(+351) 21 24 10006  ·  info@bconcepts.pt
Carnaxide, Lisbon

How to create and use Delta Live Tables in Databricks: step by step

João Barros 01 de November de 2026 4 min read

This tutorial shows how to create and use Delta Live Tables in Databricks to build a declarative ETL/ELT pipeline with data quality and observability. Learning to use Delta Live Tables is useful to automate transformations, manage dependencies between tables and detect data quality issues earlier.

Prerequisites

  • Databricks account with permissions to create Delta Live Tables pipelines.
  • Cluster with a runtime that supports Delta Live Tables (compatible DBR).
  • Source CSV or Parquet file in DBFS or in a mounted storage.
  • Basic knowledge of SQL and Python/PySpark.

Step 1: Prepare source data in DBFS

Place a sample file (for example orders.csv) in DBFS to test. This makes initial ingestion easier before connecting external sources.

# Exemplo para correr num notebook Databricks (Python)
dbutils.fs.mkdirs('/tmp/dlt_demo')
orders_csv = '''order_id,customer_id,amount,order_date
1,100,49.99,2023-10-01
2,101,19.50,2023-10-02
3,100,5.00,2023-10-02
4,103,,2023-10-03
'''
with open('/dbfs/tmp/dlt_demo/orders.csv','w') as f:
    f.write(orders_csv)
print('Ficheiro de teste criado em /tmp/dlt_demo/orders.csv')

Step 2: Create a declarative Delta Live Tables pipeline

Delta Live Tables uses Python or SQL scripts to define sources, transformations and tables with quality rules. Create a new pipeline in the UI and choose a notebook or script as the source. Below is a minimal Python example using the dlt library.

# Exemplo de notebook Python para Delta Live Tables
import dlt
from pyspark.sql.functions import col, to_timestamp

@dlt.table(
    name='raw_orders',
    comment='Dados brutos importados do CSV'
)
def raw_orders():
    return (
        spark.read.option('header','true')
             .option('inferSchema','true')
             .csv('/tmp/dlt_demo/orders.csv')
    )

@dlt.table(name='clean_orders', comment='Orders limpos e validados')
@dlt.expect_or_drop('valid_amount', 'amount IS NOT NULL AND amount >= 0')
def clean_orders():
    df = dlt.read('raw_orders')
    return (
        df.withColumn('order_date', to_timestamp(col('order_date')))
          .withColumn('amount', col('amount').cast('double'))
    )

@dlt.view(name='orders_agg')
def orders_agg():
    return (
        dlt.read('clean_orders')
           .groupBy('customer_id')
           .agg({'amount':'sum'})
    )

Step 3: Configure the pipeline in the UI and run it

In the Databricks workspace create a new Delta Live Tables pipeline: select the notebook/script, set the cluster/pipeline settings and configure the target schema. Enable monitoring and choose between Development or Production runtime mode.

# Pseudopasso: na UI Databricks
# 1. Workspace > Delta Live Tables > Create pipeline
# 2. Source: seleccionar notebook com código dlt
# 3. Target schema: ex. dlt_demo
# 4. Runtime mode: Development (testes) / Production
# 5. Start pipeline

Step 4: Add quality rules and error handling

Use @dlt.expect, @dlt.expect_or_drop or @dlt.expect_all to validate data. This helps detect and log issues (e.g., null values, invalid formats) without stopping the pipeline.

@dlt.table(name='clean_orders')
@dlt.expect('non_null_customer', 'customer_id IS NOT NULL')
@dlt.expect_or_drop('valid_amount', 'amount IS NOT NULL AND amount >= 0')
def clean_orders():
    df = dlt.read('raw_orders')
    return df.withColumn('amount', col('amount').cast('double'))

Step 5: Monitor and debug the pipeline

Use the Delta Live Tables UI to view the dependency DAG, integrity metrics (expectations), ingestion rate and error logs. To debug, examine the pipeline logs and the expectations results.

# No notebook pode também consultar tabelas criadas
spark.sql('SELECT * FROM dlt_demo.clean_orders').show()
# Verificar estatísticas de expectations na UI -> Expectations

Verify the result

Confirm that the defined tables appear in the target schema (e.g.: dlt_demo.raw_orders, dlt_demo.clean_orders) and that the data was transformed according to the rules. Check the Expectations panel to see how many rows passed/were rejected and fix the code if necessary.

Conclusion

You now know how to create a basic pipeline with Delta Live Tables in Databricks, apply validations and monitor execution. Next steps: connect streaming sources, use Auto Loader for continuous ingestion and tune quality rules. Tip: start in Development mode and optimize expectations before moving to Production — which quality rule is most critical for your data?