Advanced parameterization in ADF: generic, reusable pipelines

The most common mistake in Azure Data Factory is creating a separate pipeline per table. Parameterization lets you create generic pipelines that receive configuration at runtime, drastically reducing the number of artifacts to maintain.

Parameter types in ADF

Pipeline Parameters:  defined on the pipeline, passed at runtime
Dataset Parameters:   make datasets dynamic (schema, table, path)
Linked Service Params: dynamic credentials per environment
Global Parameters:    constants shared by all pipelines
Variables:            internal state during pipeline execution

Parameterized dataset

// Dataset: DS_SQL_Generic
Parameters:
  p_schema: string = "dbo"
  p_table:  string

Properties → Table:
  Schema: @{dataset().p_schema}
  Table:  @{dataset().p_table}

// Use in the Copy Activity:
Source Dataset: DS_SQL_Generic
  p_schema: "sales"
  p_table:  @{pipeline().parameters.source_table}

Metadata-driven pipeline

-- Configuration table
CREATE TABLE adf_config.ingest_tables (
    id              INT IDENTITY PRIMARY KEY,
    source_schema   VARCHAR(50),
    source_table    VARCHAR(200),
    sink_path       VARCHAR(500),
    watermark_col   VARCHAR(100),
    is_active       BIT DEFAULT 1
);

-- ADF Pipeline:
1. Lookup → SELECT * FROM adf_config.ingest_tables WHERE is_active = 1
2. ForEach → items: @{activity('LookupConfig').output.value}
3.   Execute Pipeline: PL_Copy_Generic
       p_source_schema: @{item().source_schema}
       p_source_table:  @{item().source_table}
       p_sink_path:     @{item().sink_path}
       p_wm_column:     @{item().watermark_col}

Useful dynamic expressions

// Today's date for partitioning
@{formatDateTime(pipeline().TriggerTime, 'yyyy/MM/dd')}

// Dynamic file name
@{concat(pipeline().parameters.table_name, '_', formatDateTime(utcNow(),'yyyyMMdd_HHmmss'), '.parquet')}

// Incremental filter condition
@{concat('updated_at > ', string(activity('LookupWatermark').output.firstRow.last_wm))}

Conclusion

A metadata-driven pipeline with 1 pipeline + 1 dataset + 1 configuration table replaces 50 individual pipelines. It is the difference between an ADF of 300 unmaintainable artifacts and one of 20 elegant ones.