How to version schemas in a Lakehouse: step by step

João Barros 23 de September de 2026 5 min read

This tutorial shows how to implement schema versioning in a Lakehouse to manage schema changes in Delta tables, preventing ETL pipelines from failing when new columns or different types arrive. The technique allows you to validate, apply and roll back schema changes in a controlled way.

Prerequisites

Account with access to the Microsoft Fabric/Workspace with permissions to create Lakehouses and run notebooks.
Basic knowledge of Spark/PySpark and Delta Lake.
Sample ingestion data in CSV or Parquet and access to the OneLake/Storage of the Lakehouse.

Step 1: Concept and versioning strategy

Why version: schema changes (new columns, removed columns or different types) can break consumers and jobs. The simple strategy is to keep one schema file per version (JSON), validate the input, apply a compatible transformation and write a new Delta version or create an alias/view for the current version.

Step 2: Define schemas in JSON

Create JSON files with the expected schema definition. Keep each version clear (v1, v2...). This allows programmatic comparison.

# exemplo de schema v1 (schema_v1.json)
{
  "fields": [
    {"name": "id", "type": "long", "nullable": false},
    {"name": "nome", "type": "string", "nullable": true},
    {"name": "valor", "type": "double", "nullable": true}
  ]
}

# exemplo de schema v2 (schema_v2.json) adiciona coluna 'categoria'
{
  "fields": [
    {"name": "id", "type": "long", "nullable": false},
    {"name": "nome", "type": "string", "nullable": true},
    {"name": "valor", "type": "double", "nullable": true},
    {"name": "categoria", "type": "string", "nullable": true}
  ]
}

Step 3: Load ingestion data and infer schema

Load the input file and transform types; compare with the expected schema using PySpark. If necessary, apply type coercion or add missing columns with null/default values.

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType
import json

spark = SparkSession.builder.getOrCreate()

# caminho para o CSV de ingestão
input_path = '/mnt/lakehouse/raw/ingest/input.csv'

df = spark.read.option('header', 'true').csv(input_path)

# carregar esquema esperado (v2) do JSON
with open('/dbfs/mnt/lakehouse/schemas/schema_v2.json') as f:
    schema_json = json.load(f)
    # converter para StructType simples: exemplo mínimo
    expected_fields = [(f['name'], f['type']) for f in schema_json['fields']]

# função simples de coerção/ajuste
for name, typ in expected_fields:
    if name not in df.columns:
        df = df.withColumn(name, spark.sql.functions.lit(None).cast('string'))

# coerção de tipo mínima (exemplo): cast para double quando requerido
if 'valor' in df.columns:
    df = df.withColumn('valor', df['valor'].cast('double'))

Step 4: Validate and record incompatibilities

Validate if there are extra columns, incompatible types or missing primary keys. Record issues in an audit file to allow fallback.

# detectar colunas extra
expected_names = [f[0] for f in expected_fields]
extra = [c for c in df.columns if c not in expected_names]

# exemplo de registo simples
audit = {
  'input_path': input_path,
  'expected_schema': expected_names,
  'found_columns': df.columns,
  'extra_columns': extra
}

# grava audit como JSON numa pasta de governação
import json
with open('/dbfs/mnt/lakehouse/audit/schema_audit.json', 'w') as f:
    f.write(json.dumps(audit))

Step 5: Apply compatible transformation and write as a Delta table

After adjustment, write the data to a Delta table with the schema version in the metadata (for example, table nome_versionada_v2). Keep tables per version or update a view/alias.

target_path = '/mnt/lakehouse/curated/minha_tabela_v2'

# escrever Delta
(df
 .write
 .format('delta')
 .mode('append')
 .save(target_path))

# criar/atualizar uma tabela gerida ou external table (SQL)
spark.sql(f"CREATE TABLE IF NOT EXISTS minha_tabela_v2 USING DELTA LOCATION '{target_path}'")

# opcional: atualizar uma view 'minha_tabela' para apontar para a versão corrente
spark.sql("CREATE OR REPLACE VIEW minha_tabela AS SELECT * FROM minha_tabela_v2")

Step 6: Fallback and schema rollback

If a new version fails validations upstream or downstream, use the audit and the Delta history to revert: you can point the view to the previous version or restore the table to a known snapshot using Time Travel.

# exemplo de rollback com time travel (se necessário)
spark.sql("CREATE OR REPLACE VIEW minha_tabela AS SELECT * FROM minha_tabela_v1")

# ou restaurar Delta para um commit anterior via RESTORE/Time Travel (exemplo SQL)
-- RESTORE não é universal; use Time Travel: SELECT * FROM delta.`/path` VERSION AS OF X

Verify the result

Confirm that the table/view points to the correct version and that consumers can read the data without errors. Check the audit file and inspect types/columns with DESCRIBE TABLE or spark.printSchema().

# verificar esquema
spark.table('minha_tabela').printSchema()

# ver audit
with open('/dbfs/mnt/lakehouse/audit/schema_audit.json') as f:
    print(f.read())

Conclusion

Versioning schemas in a Lakehouse reduces failures and eases controlled evolutions: keep JSON schema files, validate ingestion, apply coercions and record audits. Next steps: automate with a job/orchestrator, add schema regression tests and integrate with CI/CD. Tip: start by creating a small set of validation rules to avoid surprises in production.