How to copy and transform JSON to Parquet in Azure Data Factory

This tutorial shows how to copy JSON files from an Azure Blob Storage container to Parquet files in a Data Lake using Azure Data Factory, with a light transformation to normalize fields. Converting JSON to Parquet reduces costs and improves analytical performance.

Prerequisites

• Azure account with permissions to create resources
• Azure Data Factory already created
• Azure Blob Storage with some sample JSON files
• Azure Data Lake Storage Gen2 or another Blob destination for Parquet

Step 1: Create Linked Services for source and destination

You need to connect Azure Data Factory to the source storage (Blob) and the destination (ADLS Gen2 or another Blob). Use Managed Identity or Service Principal to avoid hardcoded credentials.

In the Azure Data Factory portal: Manage > Linked services > New.

// Exemplo JSON para criar Linked Service via ARM/REST (sintaxe simplificada)
{
  "name": "LS_Blob_Storage",
  "properties": {
    "type": "AzureBlobStorage",
    "typeProperties": {
      "connectionString": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=..."
    }
  }
}

Step 2: Create source JSON and destination Parquet Datasets

The source Dataset points to the container and the path with JSON files. The destination Dataset should have Parquet format. Configure the schema or let the schema be inferred in Mapping Data Flow/Copy Activity.

// Dataset exemplo (simplificado)
{
  "name": "DS_Json_Incoming",
  "properties": {
    "type": "AzureBlob",
    "typeProperties": {
      "format": { "type": "JsonFormat", "filePattern": "setOfObjects" },
      "fileName": "*.json",
      "folderPath": "input/json"
    }
  }
}

// Dataset destino Parquet (simplificado)
{
  "name": "DS_Parquet_Output",
  "properties": {
    "type": "AzureBlob",
    "typeProperties": {
      "format": { "type": "ParquetFormat" },
      "folderPath": "output/parquet"
    }
  }
}

Step 3: Decide Copy Activity vs Mapping Data Flow

If you only need format conversion (JSON → Parquet) without complex schema transformation, use Copy Activity to be cheaper and simpler. To normalize fields, unroll arrays or join complex structures, use Mapping Data Flow.

Step 4: Example with Copy Activity (simple conversion)

Create a pipeline with a Copy activity. In the Source, select the JSON Dataset and configure the file pattern; in the Sink, select DS_Parquet_Output. Check "Enable schema drift" if you have variable schemas.

// Copy Activity JSON->Parquet (pseudocódigo de pipeline)
{
  "name": "CopyJsonToParquet",
  "type": "Copy",
  "inputs": [{ "referenceName": "DS_Json_Incoming" }],
  "outputs": [{ "referenceName": "DS_Parquet_Output" }],
  "typeProperties": {
    "enableStaging": false
  }
}

Step 5: Example with Mapping Data Flow (normalize arrays)

If the JSONs contain arrays that you need to "explode" into rows, create a Mapping Data Flow with Source (JSON format), a Flatten/Unroll transformation and then a Sink to Parquet. Define the schema in the Sink for Parquet.

// Flow simplificado:
Source(JSON) -> Select(columns) -> Flatten(unrollBy: items) -> DerivedColumn(normalize campos) -> Sink(Parquet)

In the Data Flow, configure the column that contains arrays in Unroll By. Use Derived Column to combine or format fields (e.g.: concatenate id + subid).

Step 6: Run and parameterize for multiple files

To process multiple files or folders, parameterize the Dataset (folderPath, fileName) and pass the parameter in the pipeline. Use ForEach if you need per-file logic.

// Exemplo de parâmetro no pipeline
pipeline.parameters.folder = 'input/json'
// Passa para o Dataset: @pipeline().parameters.folder

Verify the result

Validate at the destination: enter the destination container and confirm .parquet files were generated. You can open a Parquet file with Azure Storage Explorer or use Azure Synapse/Databricks to read. In ADF Monitor check the pipeline run and copy logs for common errors (schema mismatch, permissions, corrupted files).

Conclusion

Converting JSON to Parquet in Azure Data Factory improves performance and reduces storage costs. Start with Copy Activity for simple scenarios and use Mapping Data Flow for normalizations/arrays. Next step: test reading the Parquet in Power BI or in Azure Synapse to verify the gains. Tip: start with a reduced set of files to validate schemas before processing at scale — do you have questions about arrays or schema drift in your JSON?