Monitoring and alerts in Azure Data Factory

Pipelines fail in production — what matters is being notified immediately and having enough diagnostics to resolve issues quickly. Azure Monitor integrated into ADF offers alerts, dashboards and detailed log analysis.

Enable diagnostic logs

// Portal: ADF → Diagnostic Settings → Add diagnostic setting
Logs:
  ✓ PipelineRuns
  ✓ ActivityRuns
  ✓ TriggerRuns
Destination:
  ✓ Send to Log Analytics Workspace: law-bconcepts-prod

Automatic alerts via Azure Monitor

// Monitor → Alerts → Create Alert Rule
Scope: ADF instance
Condition:
  Signal: Failed pipeline runs (metric)
  Threshold: > 0 (any failure)
  Evaluation period: last 5 minutes

Action Group:
  Email: dados@bconcepts.pt
  SMS:   +351 9XX XXX XXX
  Logic App: (optional) create a ticket in the support system

Query logs in Log Analytics

// KQL — failed pipelines in the last 24h
ADFPipelineRun
| where TimeGenerated > ago(24h)
| where Status == "Failed"
| project PipelineName, Start, End, Status, ErrorMessage
| order by Start desc

// Average duration per pipeline (last 2 weeks)
ADFPipelineRun
| where TimeGenerated > ago(14d)
| where Status == "Succeeded"
| summarize avg_duration_min = avg(End - Start) / 1m by PipelineName
| order by avg_duration_min desc

// Slow activities
ADFActivityRun
| where TimeGenerated > ago(7d)
| where Status == "Succeeded"
| summarize p95_duration = percentile(End - Start, 95) by ActivityName
| order by p95_duration desc

Custom workbook

Create an Azure Workbook in Monitor with the KQL queries above as table and chart visualizations. Share the URL with the team — it is the pipeline health dashboard.

Conclusion

Monitoring is not optional in production. Configure alerts from the first deployment and enable diagnostic logs — the cost of Log Analytics is marginal compared to the time saved diagnosing silent failures.