How to do incremental merge of Parquet files in Azure Data Factory

João Barros 02 de November de 2026 4 min read

This tutorial explains how to implement an incremental merge process for Parquet files in Azure Data Factory (ADF) to keep a Lakehouse up to date. It is useful to avoid duplicates and apply updates or deletions to analytical data without rebuilding everything.

Prerequisites

Azure account with permissions to create resources.
Azure Data Factory v2 created.
Azure Data Lake Storage Gen2 (ADLS Gen2) or container with source and destination Parquet files.
Basic knowledge of Parquet, SQL and Copy/Data Flow activity in ADF.

Step 1: Concept and incremental merge strategy

Before creating the pipeline, define the strategy: we will identify source Parquet files that represent new records or updates, and apply a merge to update the destination table stored as Parquet. A common pattern is to use a natural key column (id) and a version/control column/updated_at.

Step 2: Prepare folders and naming

Organize the files in ADLS Gen2 into two folders: /bronze/incremental/ for input files and /silver/tables/<name>/ for the destination. Use timestamped names for easier tracking.

Step 3: Create Linked Services and Datasets in ADF

Configure a Linked Service for ADLS Gen2 with Managed Identity or Service Principal. Then create two Parquet Datasets: one for the incremental source and one for the destination table.

// Exemplo conceptual de JSON do dataset (simplificado)
{
  "name": "ParquetDatasetSource",
  "properties": {
    "linkedServiceName": { "referenceName": "LS_ADLSGen2", "type": "LinkedServiceReference" },
    "type": "Parquet",
    "typeProperties": { "location": { "type": "AzureBlobFSLocation", "fileSystem": "data", "folderPath": "bronze/incremental" } }
  }
}

Step 4: Use Mapping Data Flow to perform the merge

The most practical component is a Mapping Data Flow that reads the incremental source and the destination and performs a join by key to decide whether to insert, update, or keep. Data Flow allows MERGE-like operations using conditional split transformations and a sink with update/insert.

// Fluxo simplificado de transformações no Data Flow
1. SourceA: leitor da origem incremental (Parquet)
2. SourceB: leitor da tabela destino (Parquet)
3. Join: left join SourceA.id == SourceB.id
4. DerivedColumn: calcular action = iif(isNull(SourceB.id), 'insert', iif(SourceA.updated_at > SourceB.updated_at, 'update', 'skip'))
5. FilterInsert: action == 'insert'
6. FilterUpdate: action == 'update'
7. SinkInsert: modo insert para destino Parquet (append)
8. SinkUpdate: modo update com chave id (modo upsert se suportado) ou escrever para staging e substituir

Step 5: Configure the Sink for upsert or replace

Parquet does not have native transactions; for upsert I recommend two options: (A) use a format that supports Upsert (e.g., Delta Lake) if available; (B) implement the pattern: write the updated full result to a temporary directory and then replace the destination directory. In ADF Data Flow you can write the updated file to /silver/tables/<name>_tmp and, afterwards, a copy/rename step to replace.

Step 6: ADF Pipeline for orchestration

Create a pipeline that orchestrates: (1) validate new files (Get Metadata or Lookup), (2) run the Mapping Data Flow, (3) move processed files to archive and (4) replace destination if needed. Parameterize with datasetPath and runId for reuse.

// Exemplo de sequência no pipeline (pseudocode)
- Get Metadata (folder bronze/incremental) -> outputs.list
- If length(outputs.list)>0 then
    - Data Flow activity (mergeFlow) with param sourcePath
    - If write to tmp, então Copy Activity para mover tmp -> silver e Remove antiga pasta
    - Move source files -> /archive/yyyyMMdd

Verify the result

Confirm there are no duplicates: open the /silver/tables/<name> folder and validate count by id (use Notebook, Synapse or Power BI). Check updated_at timestamps and compare some records that should have been updated. Also confirm that the files in the bronze folder were moved to archive.

Conclusion

With this incremental merge pattern in Azure Data Factory you can keep a Lakehouse updated without full rebuilds. Next steps: adapt to Delta Lake for native upserts, add versioning and alerts. Tip: test with small datasets and validate before applying to production — what is the biggest challenge you have in your scenario?