How to create a Delta Transient table in Databricks: step by step

João Barros 08 de August de 2026 4 min read

This tutorial shows how to create and use a Delta Transient table in Databricks to store temporary or intermediate data, reducing storage costs and simplifying automatic cleanup. It's useful when you need to persist occasional results without affecting long-term retention or performance.

Prerequisites

Databricks account with a workspace and running cluster.
Permissions to create Delta tables and write to a container/ADLS/DBFS.
Basic knowledge of PySpark and Delta Lake.

Step 1: Understand what a Delta Transient table is and when to use it

A Delta Transient table is a Delta table created with the property delta.isTransient=true. It indicates that the data is temporal and that long-term retention of transaction logs will not be maintained, which reduces storage cost. Use it for staging, intermediate ETL, or derived data that can be reprocessed.

Step 2: Create a small example DataFrame

Before creating the table, let's build an example DataFrame with PySpark. This step is useful for testing without external data.

from pyspark.sql import SparkSession
from pyspark.sql.functions import current_timestamp

spark = SparkSession.builder.getOrCreate()

data = [(1, 'Alice'), (2, 'Bruno'), (3, 'Carla')]
df = spark.createDataFrame(data, ['id', 'name']).withColumn('created_at', current_timestamp())
df.show()

Step 3: Write the Delta Transient table to DBFS or an ADLS path

Choose a path on DBFS or a path on ADLS/Blob. When writing with format('delta'), you can set the table property delta.isTransient. There are two ways: write as managed table (CREATE TABLE) or as Delta files to a path and register as a table.

# Example A: write to a path (Delta files) and then create table
path = '/tmp/delta_transient_example'
df.write.format('delta').mode('overwrite').save(path)

# Register as table and mark as transient
spark.sql(f"CREATE TABLE IF NOT EXISTS transient_db.delta_stage USING DELTA LOCATION '{path}' TBLPROPERTIES ('delta.isTransient' = 'true')")

Step 4: Create the Delta Transient table directly with SQL

You can also create the Delta Transient table directly with SQL, saving steps. This creates the Delta files in the metastore's default location.

spark.sql("""
CREATE TABLE IF NOT EXISTS transient_db.delta_transient_sql (
  id INT,
  name STRING,
  created_at TIMESTAMP
)
USING DELTA
TBLPROPERTIES ('delta.isTransient' = 'true')
""")

# Insert data
df.createOrReplaceTempView('tmp_df')
spark.sql('INSERT INTO transient_db.delta_transient_sql SELECT * FROM tmp_df')

Step 5: Configure log retention and cleanup

By default, transient tables don't retain long versions, but it's advisable to adjust delta.logRetentionDuration and delta.deletedFileRetentionDuration if needed. To clean up, use VACUUM. Note that VACUUM has built-in safety limits.

# Adjust table properties
spark.sql("ALTER TABLE transient_db.delta_transient_sql SET TBLPROPERTIES (
  'delta.logRetentionDuration' = 'interval 1 day',
  'delta.deletedFileRetentionDuration' = 'interval 1 day'
)")

# Run VACUUM to remove old files (example 1 day)
spark.sql("VACUUM transient_db.delta_transient_sql RETAIN 24 HOURS")

Step 6: Best practices and common mistakes

Use transient tables only for data that can be regenerated. Common mistakes: forgetting to set delta.isTransient=true, using too short a retention before VACUUM (which can delete needed state), and insufficient permissions on the storage path. Test in a development environment before production.

Verify the result

To confirm the table is created and marked as transient, check the table properties and content. You should see the delta.isTransient property set and the data inserted.

# View table properties
spark.sql("DESCRIBE EXTENDED transient_db.delta_transient_sql").show(truncate=False)

# Read the data
spark.sql('SELECT * FROM transient_db.delta_transient_sql').show()

Conclusion

Creating a Delta Transient table in Databricks is useful for staging and temporary ETL, reducing storage costs and simplifying cleanup. Next steps: integrate this table into a Databricks Workflows pipeline, test retention policies, or migrate the logic to a DLT table if you want automation. Tip: start with conservative retention and shorten it gradually as you gain confidence — what ETL scenario do you want to improve with a transient table?