How to create and use Delta Live Tables in Databricks: step by step
This tutorial shows how to create and use Delta Live Tables in Databricks to build a declarative ETL/ELT pipeline with data quality and observability. Learning to use Delta Live Tables is useful to automate transformations, manage dependencies between tables and detect data quality issues earlier.
Prerequisites
- Databricks account with permissions to create Delta Live Tables pipelines.
- Cluster with a runtime that supports Delta Live Tables (compatible DBR).
- Source CSV or Parquet file in DBFS or in a mounted storage.
- Basic knowledge of SQL and Python/PySpark.
Step 1: Prepare source data in DBFS
Place a sample file (for example orders.csv) in DBFS to test. This makes initial ingestion easier before connecting external sources.
# Exemplo para correr num notebook Databricks (Python)
dbutils.fs.mkdirs('/tmp/dlt_demo')
orders_csv = '''order_id,customer_id,amount,order_date
1,100,49.99,2023-10-01
2,101,19.50,2023-10-02
3,100,5.00,2023-10-02
4,103,,2023-10-03
'''
with open('/dbfs/tmp/dlt_demo/orders.csv','w') as f:
f.write(orders_csv)
print('Ficheiro de teste criado em /tmp/dlt_demo/orders.csv')
Step 2: Create a declarative Delta Live Tables pipeline
Delta Live Tables uses Python or SQL scripts to define sources, transformations and tables with quality rules. Create a new pipeline in the UI and choose a notebook or script as the source. Below is a minimal Python example using the dlt library.
# Exemplo de notebook Python para Delta Live Tables
import dlt
from pyspark.sql.functions import col, to_timestamp
@dlt.table(
name='raw_orders',
comment='Dados brutos importados do CSV'
)
def raw_orders():
return (
spark.read.option('header','true')
.option('inferSchema','true')
.csv('/tmp/dlt_demo/orders.csv')
)
@dlt.table(name='clean_orders', comment='Orders limpos e validados')
@dlt.expect_or_drop('valid_amount', 'amount IS NOT NULL AND amount >= 0')
def clean_orders():
df = dlt.read('raw_orders')
return (
df.withColumn('order_date', to_timestamp(col('order_date')))
.withColumn('amount', col('amount').cast('double'))
)
@dlt.view(name='orders_agg')
def orders_agg():
return (
dlt.read('clean_orders')
.groupBy('customer_id')
.agg({'amount':'sum'})
)
Step 3: Configure the pipeline in the UI and run it
In the Databricks workspace create a new Delta Live Tables pipeline: select the notebook/script, set the cluster/pipeline settings and configure the target schema. Enable monitoring and choose between Development or Production runtime mode.
# Pseudopasso: na UI Databricks
# 1. Workspace > Delta Live Tables > Create pipeline
# 2. Source: seleccionar notebook com código dlt
# 3. Target schema: ex. dlt_demo
# 4. Runtime mode: Development (testes) / Production
# 5. Start pipeline
Step 4: Add quality rules and error handling
Use @dlt.expect, @dlt.expect_or_drop or @dlt.expect_all to validate data. This helps detect and log issues (e.g., null values, invalid formats) without stopping the pipeline.
@dlt.table(name='clean_orders')
@dlt.expect('non_null_customer', 'customer_id IS NOT NULL')
@dlt.expect_or_drop('valid_amount', 'amount IS NOT NULL AND amount >= 0')
def clean_orders():
df = dlt.read('raw_orders')
return df.withColumn('amount', col('amount').cast('double'))
Step 5: Monitor and debug the pipeline
Use the Delta Live Tables UI to view the dependency DAG, integrity metrics (expectations), ingestion rate and error logs. To debug, examine the pipeline logs and the expectations results.
# No notebook pode também consultar tabelas criadas
spark.sql('SELECT * FROM dlt_demo.clean_orders').show()
# Verificar estatísticas de expectations na UI -> Expectations
Verify the result
Confirm that the defined tables appear in the target schema (e.g.: dlt_demo.raw_orders, dlt_demo.clean_orders) and that the data was transformed according to the rules. Check the Expectations panel to see how many rows passed/were rejected and fix the code if necessary.
Conclusion
You now know how to create a basic pipeline with Delta Live Tables in Databricks, apply validations and monitor execution. Next steps: connect streaming sources, use Auto Loader for continuous ingestion and tune quality rules. Tip: start in Development mode and optimize expectations before moving to Production — which quality rule is most critical for your data?