How to create and manage checkpoints in Apache Spark: step by step

João Barros 23 de October de 2026 4 min read

This tutorial explains how to create and manage checkpoints in Apache Spark, a useful technique to ensure fault recovery in long-running and streaming jobs. Knowing how to configure checkpoints reduces lost work and allows you to resume processing with consistent state.

Prerequisites

Apache Spark and Python installed with PySpark configured.
Basic knowledge of DataFrame and Spark operations.
Access to a persistent file system (local or HDFS/ADLS/S3) to store checkpoints.

Step 1: Understand what a checkpoint is and when to use it

A checkpoint is saving a materialized version of an RDD/DataFrame (or the state of a streaming application) to a file system for recovery. Use checkpoints to: avoid long lineage recomputations, maintain state in Structured Streaming, or ensure consistency after failures.

Step 2: Configure the checkpoint directory

Choose a persistent directory that survives executor failures. For production, use HDFS, S3 or ADLS. In a local environment, use an absolute path.

# Exemplo PySpark: definir diretório global de checkpoint
from pyspark.sql import SparkSession

spark = SparkSession.builder 
    .appName("checkpoint-demo") 
    .getOrCreate()

# Defina um diretório no sistema de ficheiros acessível por todos os nós
checkpoint_dir = "/tmp/spark_checkpoints/my_app"
spark.sparkContext.setCheckpointDir(checkpoint_dir)

Step 3: Checkpointing RDDs to cut lineage

When an RDD has a long chain of dependencies, checkpointing stores a copy on disk and clears the lineage, speeding up recomputations.

# Criar RDD e fazer checkpoint
rdd = spark.sparkContext.parallelize(range(1000000)).map(lambda x: x * 2)
rdd = rdd.map(lambda x: x + 1)

# Marcar para checkpoint e forçar execução
rdd.checkpoint()
rdd.count()  # ação que força materialização do checkpoint

Step 4: Checkpointing in Structured Streaming

In Structured Streaming, a checkpoint is required to use state and to recover offsets in case of failure. Set checkpointLocation in the writeStream.

from pyspark.sql.functions import expr

# Exemplo simples de streaming de ficheiros CSV
input_path = "/path/to/input/csvs"
checkpoint_path = "/tmp/spark_checkpoints/stream_app"

stream_df = spark.readStream \
    .option("header", "true") \
    .csv(input_path)

# Operação com estado simples: contagem por coluna
agg = stream_df.groupBy("category").count()

query = agg.writeStream \
    .outputMode("complete") \
    .format("console") \
    .option("checkpointLocation", checkpoint_path) \
    .start()

query.awaitTermination()

Step 5: Recovering from a checkpoint

For Structured Streaming, simply start the query with the same checkpointLocation and the engine recovers offsets and state. For RDD/DataFrame, you need to load from the materialized files if you must manually restart.

# Structured Streaming: reiniciar a mesma query reutiliza o checkpoint
query = agg.writeStream \
    .outputMode("complete") \
    .format("console") \
    .option("checkpointLocation", checkpoint_path) \
    .start()

# RDD: ler ficheiro materializado (exemplo simplificado)
# Se o checkpoint guardou ficheiros parquet, podemos ler diretamente
# spark.read.parquet("/tmp/spark_checkpoints/my_app/part-*")

Step 6: Best practices and common mistakes

Best practices: use persistent directories with correct permissions, do not mix checkpoints from multiple applications in the same directory, monitor disk space. Common mistakes include: checkpointLocation inaccessible (permissions/endpoint), using a non-shared temporary directory on the cluster, and forgetting to restart streaming with the same checkpointLocation.

Verify the outcome

Confirm the checkpoint worked: for RDDs check the files created in checkpoint_dir; for Structured Streaming verify that after stopping and restarting the application with the same checkpointLocation, the application resumes without reprocessing already committed data and without losing state. Check the logs for messages about state and offsets restoration.

Conclusion

Checkpointing in Apache Spark is essential for reliability in long-running jobs and for maintaining state in streaming. Next steps: experiment with checkpoints on S3/HDFS and combine with writing state to idempotent sinks. Tip: test recovery with a small controlled failure to validate behavior before going to production — have you tested your application recovery?