How to calculate percentiles and median in Apache Spark (example)

João Barros 18 de July de 2026 5 min read

How to calculate percentiles and median in Apache Spark is a common task to summarize distributions and detect outliers. In large datasets, it is not feasible to bring all data to the driver: this is where distributed functions like percentile_approx and utilities like DataFrame.stat.approxQuantile come in. This practical guide explains the rationale behind each option and shows concrete examples in PySpark, including performance tips and result verification.

Prerequisites

Python 3.7+ and PySpark installed (version compatible with your infrastructure).
Basic knowledge of Apache Spark DataFrame: creating SparkSession, using groupBy and agg.
Local environment or cluster with sufficient memory to experiment. For example, for datasets of 1–10 million rows, at least a few GB of memory per executor is recommended.

Step 1: Prepare session and data

Start by creating a SparkSession and an example DataFrame. We use a small set to observe behavior; in production you may have millions of rows. This example allows testing both approxQuantile (executed on the driver) and percentile_approx (distributed).

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, percentile_approx

spark = SparkSession.builder.appName("QuantilesExample").getOrCreate()

# Dados de exemplo: (group, value)
data = [(1, 10.0),(1, 20.0),(1, 30.0),(2, 5.0),(2, 7.0),(2, 100.0)]
df = spark.createDataFrame(data, ["group","value"]) 
df.show()

With these data: for group 1 the expected median is 20.0; for group 2 the median is 7.0. Overall, the distribution has an outlier (100.0) that affects quartiles and mean but not the median as severely.

Step 2: Calculate median (50th percentile) with approxQuantile

The function DataFrame.stat.approxQuantile is simple to use when you want quantiles of individual columns. It executes an operation that returns results to the driver — therefore it requires sufficient driver memory when applied to columns with many distinct values. The relativeError parameter controls tolerance: 0.01 means relative error up to 1% (typical), 0.001 gives higher precision but increases computational cost.

# approxQuantile(col, probabilities, relativeError)
quantiles = df.stat.approxQuantile("value", [0.5], 0.01)
print("Median (approxQuantile):", quantiles[0])

# Multiple quantiles: 25%, 50%, 75%
q = df.stat.approxQuantile("value", [0.25, 0.5, 0.75], 0.01)
print("Quartiles:", q)

In large datasets (for example, 10M+ rows) avoid using approxQuantile for many columns simultaneously without ensuring driver memory; use sampling or percentile_approx for aggregations.

Step 3: Use percentile_approx for aggregations and by group

percentile_approx is implemented as a distributed aggregation function — ideal for groupBy. It accepts a single percentile or a list and returns an array when you request multiple percentiles. The third argument (accuracy) controls the granularity of the internal summary: typical values range from 100 to 10000; the larger the value, the higher the precision and the cost.

from pyspark.sql.functions import percentile_approx

# Global median with percentile_approx
df.agg(percentile_approx(col("value"), 0.5).alias("median")).show()

# Global quartiles (returns array)
df.agg(percentile_approx("value", [0.25, 0.5, 0.75]).alias("quartis")).show()

# Quartiles by group
quartis_por_grupo = df.groupBy("group").agg(
    percentile_approx("value", [0.25, 0.5, 0.75]).alias("quartis")
)
quartis_por_grupo.show()

# Separate quartiles into readable columns
from pyspark.sql.functions import col
quartis_por_grupo.select(
    col("group"),
    col("quartis").getItem(0).alias("q1"),
    col("quartis").getItem(1).alias("q2"),
    col("quartis").getItem(2).alias("q3")
).show()

Practical example: in a set with 6 values, requesting [0.25, 0.5, 0.75] returns an array with three entries. For highly unbalanced groups (for example, one group with 10M rows and another with 10 rows) you may see differences in precision between groups; consider adjusting accuracy per case.

Step 4: Handle nulls, types and precision

Before calculating percentiles, handle nulls and ensure numeric types. Cast to DoubleType if integers and decimals are mixed. To increase precision in percentile_approx, specify a higher accuracy value (e.g., 1000 or 10000), but this increases CPU and memory usage.

# Remove nulls
df_clean = df.filter(col("value").isNotNull())

# Cast type (example)
df_clean = df_clean.withColumn("value", col("value").cast("double"))

# Example with higher precision (accuracy = 10000)
df_clean.agg(percentile_approx("value", 0.5, 10000).alias("median_preciso")).show()

If you work with financial data, consider using DecimalType to preserve precision; however, some functions may require casting to double.

Verify the result

Validate results by comparing methods: approxQuantile (list on the driver) versus percentile_approx (DataFrame/array). For small datasets, compute the exact median with local Python to confirm. On large datasets, start with relativeError ~0.01 or accuracy ~1000 and only increase if tests indicate significant bias.

# Exact median on small data (for verification only)
vals = [r[0] for r in df.select("value").rdd.flatMap(lambda x: x).collect()]
vals.sort()
from statistics import median
print("Exact median (sample):", median(vals))

Conclusion

You calculated percentiles and median in Apache Spark using approxQuantile for single columns and percentile_approx for aggregations and groupBy. In summary: use approxQuantile when you can bring a summary to the driver and need quick quantiles for a column; use percentile_approx for distributed and group operations. Adjust relativeError and accuracy according to the trade-off between precision and cost. Next steps: experiment with multiple percentiles on real data (e.g., 1M–100M rows), measure time and memory, and document the precision you consider acceptable for your case.