How to Read and Write Parquet in PySpark: Step by Step

João Barros 11 de July de 2026 4 min read

Parquet is the file format most used in Spark data projects: it stores information by columns, takes up far less space than a CSV, and lets you read only the columns you need. Learning how to read and write Parquet in PySpark is therefore one of the essential steps to building fast, cheap pipelines in a Lakehouse. Let's do it with minimal examples you can copy and run right away.

Prerequisites

An environment with PySpark: Databricks, Microsoft Fabric, Azure Synapse, or pip install pyspark on your machine.
Python 3.8 or later and Java 11 (only needed if you run Spark locally).
Basic DataFrame knowledge: select, filter, and show.
A folder you can write files to (local disk or cloud storage).

Step 1: Create the Spark session and a sample DataFrame

To write Parquet in PySpark you need a SparkSession and some data. In Databricks, Fabric, or Synapse the spark variable already exists — in that case skip the first two lines.

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("parquet-demo").getOrCreate()

dados = [
    ("Ana", "Lisboa", 2025, 1200.0),
    ("Bruno", "Porto", 2025, 890.5),
    ("Carla", "Lisboa", 2026, 1540.0),
    ("Diogo", "Faro", 2026, 730.25),
]
vendas = spark.createDataFrame(dados, ["vendedor", "cidade", "ano", "valor"])
vendas.show()

Step 2: Write the DataFrame to Parquet

You write with write.parquet(). One detail confuses many people at first: the path you provide is not a file, it is a folder. Spark creates several .parquet files inside it, one per partition it holds in memory, plus a few control files.

vendas.write.mode("overwrite").parquet("/tmp/vendas_parquet")

mode defines what happens when the folder already exists:

overwrite: deletes the previous content and writes again.
append: adds new files to the ones already there.
error (default): fails with the path already exists error.
ignore: writes nothing if the folder already exists.

Step 3: Partition the data with partitionBy

If you almost always filter by the same column (year, country, date), store the data partitioned. Spark creates one subfolder per value and, when reading, skips the folders that do not matter — this is called partition pruning and it is what makes queries fast.

(vendas.write
    .mode("overwrite")
    .partitionBy("ano")
    .parquet("/tmp/vendas_por_ano"))

On disk it looks like this:

/tmp/vendas_por_ano/ano=2025/part-00000-....snappy.parquet
/tmp/vendas_por_ano/ano=2026/part-00000-....snappy.parquet

Tip: partition by columns with few distinct values (tens or hundreds). Partitioning by a column with thousands of values creates thousands of tiny files and makes everything slower, not faster.

Step 4: Read Parquet files

To read, just point to the folder. Unlike CSV, you do not need to declare the schema or use inferSchema: column names and types are stored inside the Parquet file itself.

df = spark.read.parquet("/tmp/vendas_por_ano")
df.printSchema()

# apenas as linhas de 2026: o Spark lê só a subpasta ano=2026
df_2026 = df.filter(df.ano == 2026)
df_2026.show()

Because Parquet is columnar, you can also ask for only the columns you need — Spark never even reads the others from disk:

df.select("vendedor", "valor").show()

Step 5: Compression and number of files

By default Spark compresses with snappy, the best balance between size and speed. If you need even smaller files, use gzip. And when the result is small, merge everything into a single file with coalesce(1) to avoid the small files problem.

(vendas
    .coalesce(1)
    .write
    .mode("overwrite")
    .option("compression", "gzip")
    .parquet("/tmp/vendas_pequeno"))

Verify the result

Read back what you wrote and check three things: the row count, the column types, and the partitions created.

lido = spark.read.parquet("/tmp/vendas_por_ano")

print(lido.count())               # deve devolver 4
lido.printSchema()                # valor: double, ano: integer
lido.groupBy("ano").count().show()  # 2 linhas por ano

If count() returns 4 and the schema shows valor as double, everything is fine — the types survived the round trip to disk, which does not happen with CSV. The most common error at this stage is Unable to infer schema for Parquet: it almost always means the path is wrong or the folder is empty.

Conclusion

With write.parquet(), partitionBy(), and read.parquet() you already have the essentials to swap your CSVs for a format that saves both space and read time. The natural next step is moving to Delta Lake tables, which add ACID transactions, version history, and MERGE on top of these same Parquet files. Before that, run an experiment: write the same DataFrame to CSV and to Parquet and compare the folder sizes — how big was the difference in your case?