How to convert Parquet to Delta Lake in Databricks

João Barros 25 de July de 2026 4 min read

How to convert Parquet to Delta Lake in Databricks is a common task when you want to take advantage of Delta's ACID transactions, Time Travel and optimizations. This hands-on guide explains why and shows step-by-step how to transform Parquet files into Delta tables, with examples in PySpark and simple checks.

Prerequisites

Databricks workspace and a cluster running a Databricks Runtime with Delta support.
Read/write permissions on the storage where the Parquet files are located (e.g.: /mnt/...).
Example Parquet files and a notebook in Python (PySpark).
Basic knowledge of Spark DataFrame and SQL.

Step 1: Inspect the Parquet file

Before converting, confirm the schema and any partition columns. Knowing the schema helps decide whether you need to do schema alignment or renames.

# caminho do ficheiro Parquet
path_parquet = '/mnt/data/sample_parquet/'

# listar ficheiros e inspecionar schema
display(dbutils.fs.ls(path_parquet))
df = spark.read.parquet(path_parquet)
df.printSchema()
df.show(5, truncate=False)

Step 2: Normalize the schema (optional)

If the Parquet files have columns with different names or inconsistent types, normalize before writing. Example: unify names and cast types.

# exemplo simples de normalização
from pyspark.sql.functions import col

# renomear coluna e ajustar tipo se necessário
df = df.withColumnRenamed('OldName', 'new_name')
df = df.withColumn('event_date', col('event_date').cast('date'))

Step 3: Write to Delta with partitioning and schema evolution

Write the DataFrame to a Delta directory. Use partitionBy for read performance, and options like mergeSchema to accept schema evolutions when appending.

path_delta = '/mnt/data/delta/sample_delta/'

# primeira gravação: criar a tabela Delta (overwrite se estiver a testar)
df.write.format('delta') 
  .mode('overwrite') 
  .option('overwriteSchema', 'true') 
  .partitionBy('year') 
  .save(path_delta)

# caso vá concatenar ficheiros com evolução de esquema, use:
# df_new.write.format('delta').mode('append').option('mergeSchema','true').save(path_delta)

Step 4: Register the Delta table in the metastore (optional but recommended)

Registering the table makes SQL queries and integration with other tools easier. You can create a managed/external table that points to the Delta location.

# criar esquema se necessário
spark.sql('CREATE DATABASE IF NOT EXISTS analytics')

# registar tabela externa que aponta para a pasta Delta
spark.sql("CREATE TABLE IF NOT EXISTS analytics.sample_delta USING DELTA LOCATION '/mnt/data/delta/sample_delta/'")

# agora pode consultar com SQL
spark.sql('SELECT COUNT(*) FROM analytics.sample_delta').show()

Step 5: Optimize and manage versions (OPTIMIZE and VACUUM)

After loading the data, use OPTIMIZE to improve reads and VACUUM to remove old files (be careful with retention). These commands are useful in production.

# OPTIMIZE (requer cluster com suporte Delta e permissões)
spark.sql('OPTIMIZE analytics.sample_delta')

# VACUUM com cuidado (padrão 7 dias); aqui um exemplo com 168 horas = 7 dias
spark.sql('VACUUM analytics.sample_delta RETAIN 168 HOURS')

Verify the result

Confirm the conversion succeeded by checking for the existence of _delta_log, querying counts and samples, and listing the table in the metastore.

# verificar diretório Delta e _delta_log
display(dbutils.fs.ls(path_delta))
display(dbutils.fs.ls(path_delta + '/_delta_log'))

# ler a tabela Delta e mostrar linhas
spark.read.format('delta').load(path_delta).show(5)

# verificar tabela no metastore
spark.sql('SHOW TABLES IN analytics').show()

Common errors: permissions on the mount, missing partition column, type conflicts when not using mergeSchema. If you see exceptions about schema incompatibility, re-evaluate schema normalization or use mergeSchema with append.

Conclusion

Converting Parquet to Delta Lake in Databricks allows you to benefit from advanced features like ACID transactions and read optimizations. Next steps: automate the process with Workflows, integrate with Unity Catalog for governance and test OPTIMIZE with ZORDER. Tip: before VACUUM, confirm retention and take a snapshot to avoid accidental data loss — want to try converting a larger dataset to measure I/O gains?