(+351) 21 24 10006  ·  info@bconcepts.pt
Carnaxide, Lisbon

How to copy SFTP files to ADLS Gen2 in Azure Data Factory: step by step

João Barros 20 de November de 2026 4 min read

This tutorial shows how to copy files from an SFTP server to an ADLS Gen2 account using Azure Data Factory, preserving the folder structure, recording metadata and handling common failures. It is useful to automate the secure ingestion of external files for further processing.

Prerequisites

  • Azure account with permissions to create resources.
  • Azure Data Factory (v2) created.
  • Access to an SFTP server (host, port, user, password or SSH key).
  • ADLS Gen2 account with a container and permissions (Storage Blob Data Contributor).
  • Optional Azure Key Vault to store credentials (recommended).

Step 1: Create Linked Service for SFTP

A Linked Service describes how to connect to the SFTP. If using a password or SSH key, prefer storing it in Azure Key Vault. In the ADF portal, go to Manage > Linked services > New and choose SFTP. Configure the host, port, authentication and test the connection.

{
  "type": "Sftp",
  "typeProperties": {
    "host": "sftp.exemplo.com",
    "port": 22,
    "authenticationType": "Basic", /* ou "SSHPublicKey" */
    "userName": "meu_utilizador",
    "password": "@Microsoft.KeyVault(\'https://meu-kv.vault.azure.net/secrets/sftp-pwd\')"
  }
}

Step 2: Create Linked Service for ADLS Gen2

Create the Linked Service for the ADLS Gen2 storage account. Use the ADF Managed Identity to avoid credentials in the configuration. In the portal, choose Azure Data Lake Storage Gen2 and select Authorization via Managed Identity.

{
  "type": "AzureDataLakeStorageGen2",
  "typeProperties": {
    "url": "https://meuaccount.dfs.core.windows.net"
  },
  "connectVia": null
}

Step 3: Create SFTP DataSet and ADLS Gen2 DataSet

Define DataSets that point to the source path on the SFTP and to the destination folder in ADLS Gen2. Configure wildcardPaths if you want to copy multiple files or the entire tree.

{
  "name": "SftpFiles",
  "properties": {
    "type": "FileShare",
    "linkedServiceName": {"referenceName":"LS_SFTP","type":"LinkedServiceReference"},
    "typeProperties": {"folderPath":"/incoming/{{year}}/{{month}}"}
  }
}

{
  "name": "ADLSFiles",
  "properties": {
    "type": "AzureBlobFS",
    "linkedServiceName": {"referenceName":"LS_ADLS","type":"LinkedServiceReference"},
    "typeProperties": {"folderPath":"/raw/sftp/{{year}}/{{month}}"}
  }
}

Step 4: Create Pipeline with Copy activity

Create a pipeline with the Copy activity. To preserve the folder structure use the Preserve Hierarchy option in the Copy behaviour. Configure fault tolerance and retry to handle transient failures.

{
  "name": "CopyFromSftpToADLS",
  "properties": {
    "activities": [
      {
        "name": "CopySftpToADLS",
        "type": "Copy",
        "typeProperties": {
          "source": {"type":"BinarySource"},
          "sink": {"type":"BinarySink"},
          "enableStaging": false
        },
        "inputs": [{"referenceName":"SftpFiles","type":"DatasetReference"}],
        "outputs": [{"referenceName":"ADLSFiles","type":"DatasetReference"}],
        "policy": {"retry":3,"timeout":"00:30:00","retryIntervalInSeconds":60}
      }
    ]
  }
}

Step 5: Record metadata with Lookup/Stored Procedure activity

After the copy, record metadata (source, destination, timestamp, size) in a JSON file or in a metadata table. You can use a Lookup activity to get file details and then a Stored Procedure to write to a database or a Copy to a log file.

{
  "name": "WriteMetadata",
  "type": "Copy",
  "inputs": [{"referenceName":"MetadataSource","type":"DatasetReference"}],
  "outputs": [{"referenceName":"ADLSMetadata","type":"DatasetReference"}]
}

Step 6: Handle common errors and clean up incomplete files

Common errors: SFTP authentication failure, timeouts, locked files. Use retry policies, file size validation and a status column in the metadata. For partially transferred files, copy to a temporary folder and only move to the final folder after validation.

// Example logic: move only after_success
1. Copy to /raw/sftp/_staging/{{...}}
2. Verify size/MD5
3. If OK, move to /raw/sftp/{{year}}/{{month}}/
4. If NOK, move to /raw/sftp/_quarantine/

Verify the result

Confirm that the files arrived: open the ADLS Gen2 container and check the preserved folder structure and files with correct sizes. Check the pipeline logs in the Azure Data Factory Monitor for success and failure entries. Also confirm that the metadata table/file contains records for each processed file.

Conclusion

You now have an ADF pipeline that copies files from an SFTP server to ADLS Gen2, preserving hierarchy, handling failures and storing metadata. Next steps: automate with a trigger, encrypt in transit/at rest and integrate with an ETL/Mapping Data Flows process. Tip: try Managed Identity + Key Vault for credential security — have you tested that combination yet?