How to detect and fix skew in joins in Apache Spark: step by step
How to detect and fix skew in joins in Apache Spark to speed up large joins. Practical techniques: analysis, broadcast,…
4 min
30 Jul 2026
How to calculate percentiles and median in Apache Spark (example)
How to calculate percentiles and median in Apache Spark: practical guide with examples in PySpark using approxQuantile …
5 min
18 Jul 2026
How to Pivot in PySpark: Turn Rows into Columns
Learn how to pivot in PySpark with groupBy, pivot and agg: turn rows into columns, handle nulls, and reverse the pivot …
5 min
13 Jul 2026
How to Read and Write Parquet in PySpark: Step by Step
Learn how to read and write Parquet files in PySpark step by step: partitionBy, write modes, compression, and how to av…
4 min
11 Jul 2026
How to use explode in PySpark: expand arrays into rows
Learn to use the explode function in PySpark to turn arrays into several rows, with a practical example and how to avoi…
4 min
09 Jul 2026
How to Handle Null Values in PySpark: Step by Step
Learn to handle null values (NULL) in PySpark with na.drop, na.fill and coalesce. A step-by-step guide with practical e…
4 min
07 Jul 2026
How to Use Window Functions in PySpark: Step by Step
Learn to use Window functions in PySpark with row_number, running sum and lag. A hands-on, step-by-step example to buil…
4 min
06 Jul 2026
How to Filter Rows in PySpark with filter and where
Learn how to filter rows in PySpark with filter() and where(): simple and combined conditions plus handy filters like i…
4 min
05 Jul 2026
How to add columns with withColumn in PySpark
Learn how to add columns to a DataFrame with withColumn in PySpark: calculated, constant and conditional columns plus t…
4 min
04 Jul 2026
How to remove duplicates in PySpark: step by step
Learn to remove duplicates in PySpark with dropDuplicates and distinct, across all columns or by key, and keep the righ…
4 min
04 Jul 2026
How to aggregate data with groupBy in PySpark
Learn to aggregate data with groupBy in PySpark: count, sum and average by group with simple examples and a result you …
4 min
04 Jul 2026
How to join DataFrames in PySpark: step by step
Learn how to join DataFrames in PySpark step by step: inner and left joins, choosing the right join type and avoiding d…
4 min
04 Jul 2026