How to use explode in PySpark: expand arrays into rows

João Barros 09 de July de 2026 4 min read

Many data sources store several values inside a single cell as an array — for example, the list of products in each order or the tags on an article. While the data stays "trapped" inside the array, it is hard to count, filter, or join it to other tables. In PySpark, the explode function turns each element of the array into its own row, making it one of the most useful tools for "flattening" nested structures before aggregating or filtering. It is a very common operation with data coming from APIs or JSON files, where repeated fields almost always arrive as lists inside a single column.

Prerequisites

An environment with PySpark installed (Databricks, Microsoft Fabric, or local Spark).
An active SparkSession (in notebooks it already exists as spark).
Basic DataFrame knowledge: creating one, viewing it with show(), and selecting columns.

Step 1: Create a DataFrame with an array column

For this example, we will create a small DataFrame where each row represents an order and the produtos column holds a list of items. This lets us see the effect of explode without relying on external data.

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode

spark = SparkSession.builder.getOrCreate()

dados = [
    (1, ["teclado", "rato", "monitor"]),
    (2, ["portátil"]),
    (3, []),
]
df = spark.createDataFrame(dados, ["encomenda_id", "produtos"])
df.show(truncate=False)

The result shows three rows, each with an array in the produtos column. Notice that order 3 has an empty array — we will see later why that matters.

Step 2: Apply explode to expand the array

The explode function creates one row per element in the array. We use it inside a select (or withColumn) to generate a new column with the individual values.

from pyspark.sql.functions import explode

resultado = df.select("encomenda_id", explode("produtos").alias("produto"))
resultado.show(truncate=False)

Now each product appears in its own row, keeping the matching encomenda_id. Order 1, which had three products, now spans three rows. Notice that the other columns in the select are repeated for each element — that is how explode preserves the context of every value.

Step 3: Keep rows with empty or null arrays

There is an important detail: explode drops rows whose array is empty or null. That is why order 3 disappeared from the result. If you need to keep those rows (with a null value in the new column), use explode_outer.

from pyspark.sql.functions import explode_outer

df.select("encomenda_id", explode_outer("produtos").alias("produto")).show(truncate=False)

With explode_outer, order 3 stays in the result, with null in the produto column. This difference is easy to forget and often explains why records seem to be "missing" after an explode.

Tip: if you are not sure whether a column can contain nulls, use explode_outer by default. It is safer than discovering later that you lost rows.

Step 4: Also get the position of each element

Sometimes we want to know the order in which each value appeared in the array. The posexplode function returns two columns: the position (starting at 0) and the value.

from pyspark.sql.functions import posexplode

df.select("encomenda_id", posexplode("produtos").alias("posicao", "produto")).show(truncate=False)

This is useful, for example, to identify the first product of each order (position 0) or to rebuild the original order later. Just like explode, there is also a posexplode_outer variant to keep empty or null rows.

Verify the result

To confirm the expansion worked, compare the number of rows before and after. The sum of the array sizes should match the total rows after explode (with explode_outer, you also add the empty or null rows).

print("Linhas originais:", df.count())
print("Linhas apos explode:", resultado.count())
resultado.groupBy("encomenda_id").count().show()

If the numbers add up and each encomenda_id has the expected number of products, the operation is correct.

Conclusion

With explode, explode_outer, and posexplode you can now flatten arrays into rows and choose whether to keep empty records or not. The natural next step is to combine explode with groupBy to count how many times each product appears, or to apply the same idea to more complex structures. Have you thought about how you would use explode_outer to avoid losing orders without products in your reports?