How to detect and remove outliers in Apache Spark: step by step

This tutorial shows how to detect and remove outliers in Apache Spark to improve data quality and predictive model performance. We will explain why the IQR and Z-score approaches are used, when each is more appropriate, and how to apply these techniques scalably in PySpark for datasets with hundreds of thousands or millions of rows.

Prerequisites

• Spark and PySpark installation available (local or cluster). Spark 3.x is recommended for better functions and performance.
• CSV or Parquet file with numeric columns for analysis; for large datasets (e.g.: 1M–50M rows) use Parquet for faster reads and lower I/O.
• Basic knowledge of DataFrame and PySpark functions. Basic understanding of means, standard deviation and percentiles helps to interpret results.
• Resource configuration: if you have 10 executors with 4 cores each, the quantile approximation (approxQuantile) is usually fast; if working with 100M+ rows adjust the relativeError parameter.

Step 1: Load the data into PySpark

We start by reading the data into a DataFrame. Use Parquet whenever possible (faster and preserves types). If you load CSV, specify the schema to avoid costly inference. Check and handle nulls before computing statistics — for example, in many scenarios 0.1%–2% of rows may have nulls in numeric columns and you should decide whether to impute or drop them.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("outliers-example").getOrCreate()
df = spark.read.csv("/caminho/dados.csv", header=True, inferSchema=True)
# Selecionar colunas numéricas de interesse
numeric_cols = ["valor1", "valor2"]
df = df.select([col(c) for c in numeric_cols])
df.printSchema()

Practical example: on a dataset with 5M rows, df.count() can take from seconds to minutes depending on the cluster; avoid unnecessary count() calls in production.

Step 2: Detect outliers with IQR (Interquartile Range)

IQR is robust to extreme values and does not assume normality. Compute Q1 and Q3 per column, define bounds [Q1 - 1.5*IQR, Q3 + 1.5*IQR] (1.5 is the classic rule) and flag rows with values outside those bounds. For very large datasets use approxQuantile with relativeError typically between 0.01 and 0.001. For example, with 10M rows and relativeError=0.01 we usually obtain quantiles within 1%.

from pyspark.sql.functions import expr

# Calcular quantis aproximados (mais rápido para grandes dados)
quantiles = df.approxQuantile(numeric_cols, [0.25, 0.75], 0.01)
# quantiles é lista de pares [q1, q3] por coluna
bounds = {}
for i, col_name in enumerate(numeric_cols):
    q1, q3 = quantiles[i]
    iqr = q3 - q1
    lower = q1 - 1.5 * iqr
    upper = q3 + 1.5 * iqr
    bounds[col_name] = (lower, upper)

# Criar expressão para filtrar outliers por coluna
outlier_cond = " OR ".join([f"{c} < {bounds[c][0]} OR {c} > {bounds[c][1]}" for c in numeric_cols])
outliers_iqr = df.filter(expr(outlier_cond))
non_outliers_iqr = df.filter(~expr(outlier_cond))

In practice, in many cases only 0.1%–5% of records are flagged as outliers with IQR; this value depends on the domain (e.g.: sensors have more noise than financial transactions).

Step 3: Detect outliers with Z-score (assumption of normality)

Z-score is useful when the distribution is approximately normal. Compute mean and standard deviation, transform values into Z and flag points with |Z| > threshold. A common threshold is 3 (approximately 0.3% of points in a normal distribution). Note: if the standard deviation is zero, Z-score is not applicable — it typically implies a constant column or incorrect data.

from pyspark.sql.functions import mean, stddev

stats = df.agg(*[mean(c).alias(c+"_mean") for c in numeric_cols], *[stddev(c).alias(c+"_std") for c in numeric_cols]).collect()[0]

z_exprs = []
threshold = 3.0
for c in numeric_cols:
    mu = stats[c+"_mean"]
    sigma = stats[c+"_std"] if stats[c+"_std"] is not None else 0.0
    if sigma == 0.0:
        # Não é possível calcular Z-score se desvio for zero; evitar divisão por zero
        z_exprs.append(f"false")
    else:
        z_exprs.append(f"abs(({c} - {mu}) / {sigma}) > {threshold}")

outlier_cond_z = " OR ".join(z_exprs)
outliers_z = df.filter(expr(outlier_cond_z))
non_outliers_z = df.filter(~expr(outlier_cond_z))

Compare IQR vs Z-score: for heavy-tailed distributions IQR tends to flag fewer false positives; for near-Gaussian distributions Z-score is sensitive and interpretable (|Z|>3).

Step 4: Choose a strategy and remove outliers

Decide whether you want to remove all outliers identified by any method or use only one. Common strategies: (a) remove by the union of methods (more conservative), (b) remove by the intersection (only if both agree), (c) apply clipping or winsorization instead of deleting rows. For example, if you remove 2% of 1M rows you end up with 980k; that loss may be acceptable, but if you remove 20% you should review thresholds.

# Remover outliers IQR (exemplo)
df_clean = non_outliers_iqr
# Alternativa: remover união de outliers IQR e Z-score
# outliers_union = outliers_iqr.union(outliers_z).dropDuplicates()
# df_clean = df.join(outliers_union, on=numeric_cols, how='left_anti')

# Gravar resultado
df_clean.write.mode("overwrite").parquet("/caminho/dados_clean.parquet")

If you prefer not to lose records, you can apply clipping: for example truncate values below the lower bound to the lower bound and above the upper bound to the upper bound, which keeps the number of rows but reduces the impact of extremes.

Verify the result

Confirm that outliers were removed by showing counts and statistics before/after. Inspect key percentiles (1%, 50%, 99%) to confirm reduction of extremes. It is also important to validate the impact on the model: train the model before and after and compare metrics (RMSE, AUC, etc.) — sometimes removal improves performance, other times you may lose useful signal.

print("Total original:", df.count())
print("Total limpo:", df_clean.count())
# Estatísticas antes e depois
print("Estatísticas originais:")
df.describe().show()
print("Estatísticas limpas:")
df_clean.describe().show()
# Ver quantis para confirmar remoção de extremos
print("Quantis limpos:", df_clean.approxQuantile(numeric_cols, [0.01, 0.5, 0.99], 0.01))

Conclusion

You deleted or treated outliers using IQR and Z-score in Apache Spark. These techniques help improve analyses and models when applied carefully. Practical next steps: experiment with thresholds (e.g.: 1.5→3 for IQR, 3→4 for Z), test clipping vs removal, and validate impact on your model with validation data. Tip: log how many records are removed and keep a copy of the outliers for further investigation.