How to create Delta table snapshots in a Lakehouse: step by step

This tutorial shows how to create periodic snapshots of a Delta table in the Lakehouse for auditing, quick backups and easy rollback. Creating snapshots helps preserve historical states without relying solely on time travel or VACUUM, and is useful for exporting versions for analysis outside the Lakehouse.

Prerequisites

• Account and workspace in Microsoft Fabric with Lakehouse available.
• Read/write permissions on the Lakehouse and on Direct Lake or OneLake.
• Environment with PySpark (notebook in Fabric or cluster that can access the Lakehouse).
• Basic knowledge of Delta Lake, PySpark and SQL.

Step 1: Choose the Delta table and the snapshot strategy

Decide which Delta table you want to snapshot and how often. Common strategies: full snapshot (entire copy) or incremental snapshot based on a date/version column. To start, we will use a daily full copy for simplicity.

Step 2: Create a snapshots directory in the Lakehouse

Create a folder inside the Lakehouse or OneLake where snapshots will be stored as new Delta tables with a date prefix. This makes management and cleanup easier.

# Exemplo PySpark para criar o diretório (se necessário apenas criar metadados)
from delta.tables import DeltaTable
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

# Paths de exemplo
lakehouse_base = 'abfss://@.dfs.core.windows.net/lakehouse'
snapshots_path = lakehouse_base + '/snapshots'
print(snapshots_path)

Step 3: Export the Delta table to a timestamped snapshot

Read the original Delta table and write it as a new Delta table with a date/time suffix. This preserves the full state at the time of the snapshot.

# Lê a tabela Delta existente e grava snapshot com timestamp
from datetime import datetime

table_path = lakehouse_base + '/tables/minha_tabela'
now = datetime.utcnow().strftime('%Y%m%d_%H%M%S')
snapshot_path = f"{snapshots_path}/minha_tabela_snapshot_{now}"

df = spark.read.format('delta').load(table_path)
# grava como nova tabela Delta
(df.write.format('delta')
   .mode('overwrite')
   .option('overwriteSchema','true')
   .save(snapshot_path))

print(f'Snapshot criado em: {snapshot_path}')

Step 4: Register the snapshot as a table in the metastore (optional)

To make queries easier via the SQL analytics endpoint and catalog, you can create an entry in the metastore that points to the snapshot.

# Exemplo SQL para criar tabela no metastore (no SQL notebook do Fabric)
CREATE TABLE IF NOT EXISTS snapshots.minha_tabela_snapshot_20240101_000000
USING DELTA
LOCATION 'abfss://@.dfs.core.windows.net/lakehouse/snapshots/minha_tabela_snapshot_20240101_000000';

Step 5: Schedule automated snapshots

Use a scheduler (Azure Data Factory, Power Automate, Azure Functions, or a Fabric job) to run the PySpark script daily. In Fabric you can use pipelines to orchestrate a notebook that runs the previous code.

# Exemplo esquemático: Azure Function (Python) invoca o notebook ou script PySpark
# Pseudocódigo para agendamento diário
# 1. Trigger Timer daily
# 2. Call Fabric notebook REST API or submit job to Spark cluster
# 3. Monitor retorno e regista sucesso/erro

Step 6: Retention and cleanup policy for snapshots

Define a policy to delete old snapshots (for example, keep 30 days). You can list snapshot directories and remove those that exceed the retention. Caution: delete only the snapshots you are certain are not needed.

# Exemplo PySpark para listar e remover snapshots antigos (pseudo)
from datetime import datetime, timedelta
import re

retention_days = 30
cutoff = datetime.utcnow() - timedelta(days=retention_days)

# lista diretórios (exemplo com dbutils/fs ou filesystem API)
files = dbutils.fs.ls(snapshots_path)
for f in files:
    m = re.search(r'minha_tabela_snapshot_(\d{8}_\d{6})', f.name)
    if m:
        ts = datetime.strptime(m.group(1), '%Y%m%d_%H%M%S')
        if ts < cutoff:
            dbutils.fs.rm(f.path, recurse=True)
            print('Removido', f.path)

Verify the result

Confirm the snapshot exists by reading it as Delta or by querying the table registered in the metastore. Check row count, schema and sample rows to ensure integrity.

# Verificação simples
snap = spark.read.format('delta').load(snapshot_path)
print('count=', snap.count())
snap.show(5)

Conclusion

You have executed a repeatable process to create snapshots of Delta tables in the Lakehouse, useful for auditing, historical analysis and rollback. Next steps: automate with a job, test retention and integrate success/error notifications. Tip: start with full snapshots and, when comfortable, evolve to partitioned incremental snapshots to reduce costs — do you want an example of an incremental snapshot?