Exponential backoff in Azure Data Factory: step by step

This guide shows how to implement a retry pattern with exponential backoff in Azure Data Factory to retry an activity (e.g.: Copy) without failing the pipeline immediately. Exponential backoff helps reduce pressure on external services and improves resilience in cases of transient failures.

Prerequisites

• Azure subscription with an Azure Data Factory instance.
• Permissions to create pipelines and linked services in the Data Factory.
• A test scenario: for example a Copy activity that may fail intermittently.

Step 1: Create pipeline variables

Open the new pipeline and add variables to control attempts, state and initial delay. These variables allow calculating the backoff and ending the loop when the maximum attempts is reached or when the activity succeeds.

{
  "variables": {
    "attempt": { "type": "Int", "defaultValue": 1 },
    "maxAttempts": { "type": "Int", "defaultValue": 5 },
    "delaySeconds": { "type": "Int", "defaultValue": 5 },
    "success": { "type": "Bool", "defaultValue": false }
  }
}

Step 2: Add the main activity with continueOnError

Insert the activity you want to retry (e.g.: Copy activity). Set continueOnError to true so that a failure does not immediately stop the pipeline. We will then check the result of that activity.

// Example of property used in the JSON (UI does this automatically)
"activities": [
  {
    "name": "CopyMain",
    "type": "Copy",
    "policy": { "timeout": "7.00:00:00", "retry": 0 },
    "typeProperties": { /* connection and mapping */ },
    "continueOnError": true
  }
]

Step 3: Check success with If Condition

After CopyMain, add an If Condition that evaluates whether the activity succeeded. The expression uses the activity object: activity('CopyMain').Status. If it is 'Succeeded', set the success variable to true; otherwise, proceed to the retry logic.

// If Activity condition
@equals(activity('CopyMain').Status, 'Succeeded')

// Set Variable (success = true) on the True branch
// On the False branch we will increment attempt and prepare the Wait

Step 4: Increment attempt and calculate exponential backoff

On the False branch of the If Condition do two things: increment the attempt variable and calculate the new delaySeconds (for example, double it). Use Set Variable with Data Factory expressions.

// Increment attempt
Set Variable 'attempt' -> @add(variables('attempt'), 1)

// Calculate new delaySeconds (exponential)
Set Variable 'delaySeconds' -> @mul(variables('delaySeconds'), 2)

Step 5: Wait with Wait and wrap in Until

After updating the variables, add a Wait activity that uses delaySeconds. To repeat the process until success or until reaching maxAttempts, wrap everything inside an Until activity with an appropriate exit condition.

// Until condition: exit when success = true OR attempt >= maxAttempts
@or(equals(variables('success'), true), greaterOrEquals(variables('attempt'), variables('maxAttempts')))

// Wait property (use expression for seconds)
"waitTimeInSeconds": "@variables('delaySeconds')"

Logical structure of the pipeline (simplified): Until -> [ CopyMain (continueOnError:true) -> If Condition { True: Set success=true; False: Set attempt++, Set delaySeconds*=2, Wait } ]

Verify the result

Run the pipeline with a scenario where the activity fails a few times. In the Azure Data Factory Monitor pane view the pipeline runs and activity Runs. Check the variables at the end of the execution (in the Output section of the pipeline run) and confirm that:

• The Until repeated attempts until success or until maxAttempts.
• The wait times increased (delaySeconds) as expected.
• The final state (success) is consistent with what you observed in the activities.

Conclusion

With this pattern you use Until, If, Set Variable and Wait to implement an exponential backoff in Azure Data Factory without relying on the native retry. Next steps: add a maximum cap to the delay, introduce jitter (randomness) to avoid thundering herd and log metrics in Monitor for alerts. Tip: prefer using Execute Pipeline to isolate the retry logic and reuse it — want to try that pattern in a separate pipeline?