How to load Avro files to ADLS Gen2 with Azure Data Factory

This tutorial shows how to load Avro files to ADLS Gen2 using Azure Data Factory, converting them to Parquet to optimize storage and read performance. It is useful when you receive Avro dumps from external systems and want to prepare data for analysis in Power BI or processing in Synapse.

Prerequisites

• Azure account with permission to create resources.
• Storage account with ADLS Gen2 (hierarchical namespace) or target OneLake.
• Azure Data Factory (V2) created.
• Avro files available in Blob Storage, HTTP endpoint or FTP.
• Basic knowledge of pipelines in Azure Data Factory.

Step 1: Create Linked Services for source and sink

First, create two Linked Services in Azure Data Factory: one for the source where the Avro files are stored (Blob Storage, HTTP or FTP) and another for the target ADLS Gen2. Using Managed Identity is recommended for security reasons.

// Exemplo JSON mínimo de Linked Service para Azure Blob Storage (a interface de authoring do ADF faz isto automaticamente)  {
  "name": "LS_BlobSource",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "connectionString": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"
    }
  }
}

Step 2: Create Avro and Parquet Datasets

Create a Dataset representing the Avro files (source) and another Dataset for Parquet (sink). In the Avro Dataset select the Avro format; in the Parquet choose the Parquet format and set the output folder in ADLS Gen2.

// Dataset Avro (exemplo de schema opcional)  {
  "name": "DS_Avro_Source",
  "properties": {
    "type": "AzureBlob",
    "typeProperties": {
      "format": {
        "type": "Avro"
      }
    }
  }
}

Step 3: Pipeline with Copy activity for Avro→Parquet conversion

Add a Copy activity to the pipeline. Set the source to the Avro Dataset and the sink to the Parquet Dataset. In Mapping choose Auto mapping (or define a manual mapping if you want to rename fields). Configure performance and parallelism according to the file sizes.

// Exemplo simplificado de activity Copy JSON (ADF pipeline)  {
  "name": "Copy_Avro_to_Parquet",
  "type": "Copy",
  "inputs": [{ "referenceName": "DS_Avro_Source", "type": "DatasetReference" }],
  "outputs": [{ "referenceName": "DS_Parquet_Sink", "type": "DatasetReference" }],
  "typeProperties": {
    "enableStaging": false,
    "source": { "type": "AvroSource" },
    "sink": { "type": "ParquetSink" }
  }
}

Step 4: Handle large files and performance

If the Avro files are large, configure MaxConcurrentConnections and use PolyBase or parallel copy when the sink is Azure SQL/Synapse. For ADLS Gen2, increase the copy degree of parallelism and adjust the Parquet block size to improve subsequent reads.

// Exemplo: propriedades de performance na actividade Copy
"typeProperties": {
  "parallelCopies": 8,
  "dataIntegrationUnit": 32
}

Step 5: Deal with inconsistent schemas (common errors)

Common errors: schema drift in Avro, optional fields or differing types between files. Use Mapping Data Flows if you need logic to normalize fields, or add a Get Metadata + If Condition activity to validate the schema before copying.

// Pseudocódigo: verificar existência de campo via Get Metadata + If Condition  Get Metadata -> schema
If schema contém campoX -> executar Copy
Else -> enviar alerta via Web Activity

Step 6: Automate with trigger and logging

Create a scheduled or event-based trigger (BlobCreated) to fire the pipeline when new Avro files arrive. Configure diagnostics/logging in ADF to capture failures and run times, and use Azure Monitor for alerts.

// Exemplo: trigger tipo Event Grid para BlobCreated (a UI do ADF cria este JSON)  {
  "name": "Trigger_OnBlobCreated",
  "type": "BlobEvents",
  "properties": {
    "events": [ "Microsoft.Storage.BlobCreated" ]
  }
}

Verify the result

Check the target folder in ADLS Gen2: you should see Parquet files with the expected size and count. Open a Parquet file with tools like Azure Storage Explorer or use Synapse/Databricks to read and validate the schema and data samples. In ADF Monitor view the pipeline run details for the Copy activity to confirm rowsCopied and duration.

Conclusion

By converting Avro to Parquet in Azure Data Factory you obtain columnar files ready for analysis and optimize costs and performance. Next steps: add data quality validation (Data Flows) or integrate with Synapse for table catalogs. Tip: testing with small files first helps tune parallelism without high costs — have you tried using schema drift with Mapping Data Flows?