How to Handle Null Values in PySpark: Step by Step

João Barros 07 de July de 2026 4 min read

Null values (NULL) show up in almost every real-world dataset: a field left blank, a join with no match, or a sensor that failed. Left untreated, they distort averages, produce misleading counts, and can even break machine learning models. Knowing how to handle null values in PySpark at scale is therefore an essential skill, and Spark offers three simple, fast approaches: drop them, fill them, or replace them with a fallback value.

Prerequisites

Python 3.8 or newer installed.
PySpark installed with pip install pyspark and Java 8, 11 or 17 on your system.
Basic knowledge of DataFrames in PySpark (how to read data and list columns).

Step 1: Create a SparkSession and a sample DataFrame

To try each technique safely, we start by creating a SparkSession and a small DataFrame with a few None values — which Spark automatically turns into NULL. Having controlled sample data makes it much easier to see the effect of each function.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("valores-nulos").getOrCreate()

dados = [
    ("Ana",   "Porto",  30),
    ("Bruno", None,     None),
    (None,    "Lisboa", 25),
    ("Rita",  None,     41),
]
df = spark.createDataFrame(dados, ["nome", "cidade", "idade"])
df.show()

Step 2: Count null values per column

Before deciding what to do, it is good practice to measure the problem. The expression below walks through each column and counts how many rows are null, giving you an instant snapshot of data quality.

from pyspark.sql.functions import col, count, when

df.select([
    count(when(col(c).isNull(), c)).alias(c)
    for c in df.columns
]).show()

The result shows the number of nulls in nome, cidade and idade. With these numbers it becomes clear which column needs the most attention and which strategy fits best.

Step 3: Drop rows with na.drop

When a row with missing data adds nothing to the analysis, the simplest option is to remove it with na.drop() (also available as dropna()). The how parameter decides whether we delete rows with any null or only those that are completely empty.

# Drop rows with ANY null value
df.na.drop(how="any").show()

# Drop only fully empty rows
df.na.drop(how="all").show()

# Keep rows with at least 2 non-null values
df.na.drop(thresh=2).show()

# Check nulls only in specific columns
df.na.drop(subset=["cidade"]).show()

By default, na.drop() uses how="any", so a single null is enough for the row to be removed. Use subset when only the critical columns matter.

Step 4: Fill nulls with na.fill

Often we want to keep every row and simply replace nulls with a default value. That is exactly what na.fill() (or fillna()) does. With a dictionary you can set a different value for each column.

# Fill null text columns with "Desconhecido"
df.na.fill("Desconhecido").show()

# A different value per column
df.na.fill({"cidade": "Sem cidade", "idade": 0}).show()

Watch out for a common error: Spark only fills columns whose type is compatible with the value you provide. A string will not fill numeric columns and vice versa, so the per-column dictionary is the safest way to avoid surprises.

Step 5: Replace nulls with another column using coalesce

The coalesce function returns the first non-null value from a list of columns or expressions. It is perfect for building a fallback value: if the first option is empty, it uses the next one.

from pyspark.sql.functions import coalesce, lit

df.withColumn(
    "cidade_final",
    coalesce(col("cidade"), lit("N/D"))
).show()

Whenever cidade is null, the new cidade_final column gets "N/D"; otherwise it keeps the original value. You can also chain several columns, such as coalesce(col("telemovel"), col("fixo"), lit("sem contacto")).

Verify the result

To confirm the cleanup worked, run the Step 2 count again on the treated DataFrame: if every column shows 0, no nulls remain. Alternatively, test a specific column with df.filter(col("idade").isNull()).count() and confirm it returns 0.

Conclusion

With na.drop, na.fill and coalesce you have the three fundamental approaches to handle null values in PySpark: drop the rows, fill them with a default, or replace them with an alternative column. The next step is to choose the strategy based on the business need — for example, filling age with the median instead of zero, so you do not skew the statistics. Which of these three techniques makes the most sense for your dataset?