How to detect dynamic schemas in Azure Data Factory

João Barros 29 de September de 2026 4 min read

This tutorial shows how to create a pipeline in Azure Data Factory that detects schema changes in data (for example, new fields in a CSV/Parquet) and triggers conditional transformations. Detecting dynamic schemas prevents load failures and allows pipelines to adapt automatically.

Prerequisites

Azure account with permission to create resources (Data Factory, Storage).
Azure Data Factory (v2) created and access to the ADF portal or Git integration.
Azure Blob Storage or ADLS Gen2 with sample files (CSV/Parquet).
Basic knowledge of pipelines, activities and Linked Services in ADF.

Step 1: Concept and strategy

The approach is explained: use the Get Metadata activity to read the file's schema/columns, compare it with a stored reference schema (in a JSON file in Storage or in a SQL table) and, if there are differences, mark the event (for example with a control file or Trigger). This strategy avoids transformation failures and allows manual or automatic reviews.

Step 2: Prepare the reference schema

Create a simple JSON file in Storage that contains the list of expected columns. That file will be the point of comparison.

{
  "columns": ["id","name","date","amount"]
}

Place this file, for example, in the 'control' container as reference_schema.json.

Step 3: Pipeline - get metadata from the data file

Create a pipeline in ADF with a Get Metadata activity pointing to the dataset of the data file (CSV or Parquet). Configure the Field list for 'structure' or 'columnCount' and 'childItems' depending on the format. The goal is to obtain the list of detected columns.

// Example of expected output from the Get Metadata activity (pseudo-JSON)
{
  "structure": [
    {"name":"id","type":"String"},
    {"name":"name","type":"String"},
    {"name":"date","type":"String"},
    {"name":"amount","type":"Decimal"},
    {"name":"new_col","type":"String"}
  ]
}

Step 4: Read the reference schema

Add a Lookup activity that reads the reference_schema.json file from the same Storage. Configure the Lookup for 'First row only' = false if it returns an array. The output will have the list of expected columns.

// Example of Lookup output
{
  "value": [{"columns": ["id","name","date","amount"]}]
}

Step 5: Compare schemas with an If Condition activity

Add an If Condition activity that compares the column arrays. Use an expression that checks if there is any column in the detected schema that is not in the reference schema, or vice versa. To simplify, convert lists to ordered strings and compare or use contains functions.

// Example expression (ADF expression language)
@greater(length(arrayExcept(activity('GetMetadata').output.structure[*].name, activity('Lookup').output.value[0].columns)), 0)

// arrayExcept does not exist natively; alternative: use an Azure Function/Databricks for comparison when needed.

Note: if you need more complex logic (differences including types), call an Azure Function or a Databricks Notebook from the pipeline to compare and return a boolean and list of differences.

Step 6: Actions depending on the result

On the true branch (difference found) perform one of these actions: create an alert file in Storage, send an email via Logic App, or write the difference to a SQL table for analysis. On the false branch, continue with the normal ETL.

// Example: Web activity to call Logic App (simplified body)
{
  "differences": "@string(activity('Compare').output)"
}

Step 7: Automate and version the reference schema

Automate the update of the reference_schema.json file when changes are approved: for example, have a separate pipeline that, after manual validation, updates the file. Keep history with timestamps for auditing.

Verify the result

To confirm detection works: place a file with a new column in Storage and run the pipeline. Check the run in the ADF Monitor; confirm that the If Condition activity took the difference branch and that the chosen action (alert file, email, SQL record) executed. Review the outputs of the Get Metadata and Lookup activities in the output pane to see the lists of columns.

Conclusion

Detecting dynamic schemas in Azure Data Factory protects pipelines from failures and facilitates data governance. Next steps: implement richer logic (types, renames, required columns) using Azure Function or Databricks for comparison, and add automated tests. Tip: always log the detected schema to ease debugging; what is the first schema change you expect to encounter?