How to calculate and apply Z-score by group in Apache Spark: step by step

João Barros 19 de September de 2026 5 min read

Normalizing data by group helps compare values within distinct categories and detect relative anomalies. This tutorial shows how to calculate and apply the Z-score by group in Apache Spark (PySpark) to transform a numerical variable according to the mean and standard deviation of its group.

Prerequisites

PySpark installed and configured and a functional SparkSession (for example, Spark 3.x).
Basic knowledge of DataFrame, SQL and aggregation functions in Spark (groupBy, agg, join, Window).
Editor or Notebook (Jupyter, VS Code, Databricks) to run Python code and inspect results.
Preferable: understand the differences between stddev_pop and stddev_samp — this affects the interpretation of the standard deviation.

Step 1: Understand the objective and the reasons

The Z-score transforms each value x into (x - mean)/std. Doing this by group (for example, by product, store or region) allows comparing relative values within each group independently of the absolute scale. For example, sales of 110 units may be normal in a region where the mean is 100 with std ≈ 5 (z ≈ 2), but would be an outlier in a region with mean 105 and std ≈ 1 (z ≈ 5).

Practical uses: detect anomalies by product, create scaled features for machine learning models, or visualize normalized deviations between categories. It reduces bias when groups have very different scales (e.g., prices in different markets) and facilitates operational rules (for example, investigate points with |z| > 3).

Step 2: Create a minimal example dataset

Creating a small DataFrame allows validating the logic before applying it at large scale. Here we use two groups with four records each — sufficient to compute mean and stdpop. Real values will have thousands to millions of rows, but the logic is the same.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("zscore-group-example").getOrCreate()

data = [
    ("A", 10.0), ("A", 12.0), ("A", 8.0), ("A", 25.0),
    ("B", 100.0), ("B", 110.0), ("B", 95.0), ("B", 104.0),
]

df = spark.createDataFrame(data, ["group", "value"]) 
df.show()

Step 3: Calculate mean and standard deviation by group

Aggregate by group to obtain mean and std. Use built-in functions to ensure performance on clusters. Choose stddev_pop if you consider the full population of the group, or stddev_samp if you have a sample. For our example, using stddev_pop, the results are approximate:

Group A: mean ≈ 13.75, std ≈ 6.645 (calculated over 4 observations).
Group B: mean ≈ 102.25, std ≈ 5.494.

These numbers allow verifying Z-scores manually later (e.g., for A, value 25 → z ≈ (25-13.75)/6.645 ≈ 1.69).

from pyspark.sql.functions import mean, stddev_pop

stats = df.groupBy("group").agg(
    mean("value").alias("mean_value"),
    stddev_pop("value").alias("std_value")
)

stats.show()

Step 4: Join statistics to the original DataFrame

Perform a join between the original DataFrame and the per-group statistics. If the statistics (stats) are small compared to the dataset, consider a broadcast join for efficiency: broadcast(stats). For large datasets, the keyed join is the standard. Alternative: compute mean and std with Window functions (over partitionBy) to avoid an explicit join — useful in pipelines where you prefer to avoid additional shuffle steps.

df_with_stats = df.join(stats, on="group", how="left")
df_with_stats.show()

Step 5: Calculate the Z-score and handle zero standard deviation cases

Calculate (value - mean_value) / std_value. Groups with a single record or no variability will have std = 0; avoid division by zero by substituting with a defined behavior: set z_score = 0, NaN, or use a small epsilon. Another approach is to require count >= 2 to compute Z-score and flag the rest as "insufficient".

from pyspark.sql.functions import when, lit

epsilon = 1e-9

result = df_with_stats.withColumn(
    "z_score",
    when(col("std_value").isNull() | (col("std_value") == 0), lit(0.0))
    .otherwise((col("value") - col("mean_value")) / (col("std_value") + lit(epsilon)))
)

result.select("group", "value", "mean_value", "std_value", "z_score").show()

Step 6: Practical uses — detect anomalies and filter

A common threshold for anomalies is |z_score| > 3: in a normal distribution this corresponds to ~0.27% of points. If you prefer greater sensitivity, use 2.5 (~1.24%) or 2 (~5%). The threshold should reflect the cost of false positives vs false negatives in your context. For example, in an operation with 1M records per day, applying |z|>3 may return ~2700 suspicious records per day (assumption of normality), a manageable volume for manual review.

threshold = 3.0
anomalies = result.filter(abs(col("z_score")) > threshold)
anomalies.show()

Beyond detection, save the z_score column as a feature for models (normalize by group before training), or compute percentiles by group for business rules. Always test with groups of varying sizes and validate the alert rate with labeled data, if available.

Verify the result

Confirm that the means and standard deviations by group make sense using stats.show(). Manually verify some calculations (for example, the numeric examples given) and test groups with a single record to confirm the division-by-zero handling. If results are odd, review aggregations and null values.

Conclusion

Normalizing by group with Z-score in Apache Spark is a simple but powerful technique for anomaly detection and feature preparation. For production, incorporate this calculation into an ETL pipeline, validate thresholds with historical data and choose stddev_pop vs stddev_samp according to the required statistical definition. Next step: apply by time windows (e.g., Z-score by group and by month) or integrate into an automated validation step.