How to create a Delta Live Table in a Lakehouse on Microsoft Fabric
This tutorial shows how to create a Delta Live Table in a Lakehouse on Microsoft Fabric to transform and validate data in a repeatable way with versioning. A Delta Live Table enables ETL/ELT pipelines with quality rules, detection of common errors and integration with Power BI. We will explain the reason for each step and provide concrete examples that you can adapt to your scenario — for example, transforming CSV files of 10k–100k rows with ingestion latency of a few minutes.
Prerequisites
- An account with access to Microsoft Fabric and permissions to create Lakehouse and Notebooks/Jobs. Typically you need Owner or Contributor permissions in the Workspace.
- An existing Lakehouse in OneLake or permission to create a new Lakehouse. E.g.: a 50 GB Lakehouse can be sufficient for test environments.
- Simple data source: CSV or Warehouse table accessible. For this example we use a CSV with ~20,000 customer rows.
- Basic knowledge of Spark SQL and PySpark (introductory level). Familiarity with the DataFrame API helps to follow the code.
Step 1: Plan the Delta Live Table
Before writing code, decide the transformations and quality rules (e.g.: remove duplicates, validate emails using a simple regular expression, convert dates to the DATE type). Define thresholds: for example, reject batches where >5% of emails are invalid, or keep invalid records for analysis. Choose the destination Delta Table name (e.g.: lakehouse.customers) and the folder in the Lakehouse where Delta files will be stored (e.g.: /lakehouse/delta/customers). Also plan the retention and checkpointing policy — e.g., keep 30 days of Delta version history if you need rollback.
Step 2: Create a Notebook for the pipeline
Create a Notebook in the Fabric Workspace that will execute the transformations. We will use PySpark for clarity and reproducibility. The Notebook reads the source, applies transformations and writes to a Delta Table in the Lakehouse. For a 20k-row CSV, a cluster configuration with 2–4 nodes may be sufficient; for larger volumes adjust the configuration.
# Exemplo PySpark simplificado para Delta Live Table
from pyspark.sql.functions import col, to_date, lower
# Ler CSV no OneLake (caminho do Lakehouse)
input_path = 'abfss://datalake@youraccount.dfs.core.windows.net/raw/customers.csv'
df = spark.read.option('header', True).csv(input_path)
# Limpeza básica e validação simples
df_clean = (df
.withColumn('email', lower(col('email')))
.withColumn('signup_date', to_date(col('signup_date'), 'yyyy-MM-dd'))
.dropDuplicates(['customer_id'])
)
# Filtrar registos sem email
df_valid = df_clean.filter(col('email').isNotNull())
# Escrever para Delta no Lakehouse (pasta do Lakehouse)
delta_path = 'abfss://datalake@youraccount.dfs.core.windows.net/lakehouse/delta/customers'
df_valid.write.format('delta').mode('overwrite').save(delta_path)
Note: you can add schema enforcement to avoid unexpected columns causing failures, and use 'append' mode in incremental pipelines with checkpointing to avoid data loss.
Step 3: Register the Delta Table in the Lakehouse
After writing the Delta files, register the Delta Table for querying via Warehouse/Spark SQL and Power BI. This creates a named table pointing to the Delta location. Registering the table allows tools like Power BI or SQL endpoints to query the latest version directly without knowing the physical path.
-- Exemplo T-SQL/SQL no Notebook ou Warehouse
CREATE TABLE IF NOT EXISTS lakehouse.customers
USING DELTA
LOCATION 'abfss://datalake@youraccount.dfs.core.windows.net/lakehouse/delta/customers';
After creation, run a sample query and check the metadata (e.g.: information about Delta versions) to ensure the table is pointed correctly.
Step 4: Add quality rules and logging
Improve the pipeline by adding checks: total counts, percentage of nulls per column, verification of valid emails. Log results to an audit file or a monitoring table. For example, for each run store: timestamp, total, valid, invalid, invalid percentage. Configure alerts if the invalid percentage exceeds 5%.
# Exemplo simples de validação e logging
total = df.count()
valid = df_valid.count()
invalid = total - valid
audit = spark.createDataFrame([(total, valid, invalid)], ['total', 'valid', 'invalid'])
audit.write.mode('append').parquet('abfss://datalake@youraccount.dfs.core.windows.net/lakehouse/audit/customers_audit')
Also store samples of invalid records for diagnostics; e.g., limit to 1000 records per run to avoid uncontrolled growth.
Step 5: Schedule the execution as a Job
Convert the Notebook into a Job in Fabric (Spark Job/Notebook Job) with a schedule for periodic or event-triggered execution. In the Job define resources (e.g.: driver 4 cores, 16 GB; 2 workers) and alerts on failure. A practical example: daily schedule at 02:00 for batch processing, or near real-time triggering when a new file arrives, reducing latency to minutes.
# No UI do Fabric: New -> Job -> Notebook Job
# Selecionar Notebook, configurar cluster e schedule (ex.: diariamente às 02:00)
Verify the result
Confirm that Delta files exist in the Lakehouse folder and that the table was created and contains clean data. Test simple queries and open in Power BI to validate exploration. E.g.: run SELECT COUNT(*) to validate the number of rows; compare with the audit to ensure consistency. If you detect discrepancies, use Delta version history to rollback or reprocess a specific batch.
-- Verificar número de linhas
SELECT COUNT(*) FROM lakehouse.customers;
-- Verificar amostra de registos
SELECT * FROM lakehouse.customers LIMIT 10;
Conclusion
You now have a basic Delta Live Table in the Microsoft Fabric Lakehouse that applies transformations and validations, with logging and scheduling. Next steps: add advanced quality expectations, integrate with Power BI for near real-time dashboards, implement schema evolution control and optimize performance (compaction, Z-Order) for larger tables. Practical tip: start by testing with small files (1k–20k rows) and progressively increase the cluster and configuration as data volume grows.