When a team starts building a data lake, the conversation usually revolves around tools: which processing engine, which orchestrator, which cloud platform. But there is a quieter decision that ends up determining half of the platform's performance and cost — the format in which the data is stored on disk. This is where names like Parquet, Avro and ORC come in.
These three open formats solve the same problem — storing large volumes of data efficiently — but they do it in different ways, designed for different workloads. Choosing badly does not break anything immediately: the data is still there. What happens is more subtle. Queries get slower, the storage and compute bill goes up, and no one quite understands why.
This article explains how each format works, where it shines and where it stumbles, so that the choice stops being a detail inherited from a tutorial and becomes a conscious decision.
Row or column: the decision that changes everything
Before the names, the underlying idea. A data file can organise information in two ways: by row or by column.
In a row-oriented format, the values of one record are kept together: a customer's name, age, city and date are written in sequence, and the next customer follows. It is great for writing one record at a time and for reading whole records — the way a transactional application works.
In a column-oriented format, it is the opposite: all the names are kept together, then all the ages, then all the cities. Reading an entire table becomes more work, but answering "what is the average age of customers in Lisbon" gets much faster, because the engine only reads the age and city columns and ignores everything else. In analytics, where queries touch few columns but millions of rows, this difference is enormous.
Keeping this distinction in mind resolves 80% of the doubts: Parquet and ORC are columnar; Avro is row-oriented. The rest are details — important ones, but details.
Parquet: the columnar format that dominates analytics
Apache Parquet has become the default format of practically the entire modern data ecosystem. If you use Spark, a lakehouse or a query engine like Trino, the data is almost certainly in Parquet — and there are good reasons for that.
By storing data by column, Parquet lets the engine read only the necessary columns (so-called projection pushdown). Because values in the same column are similar to each other — many similar dates, many repeated codes — compression is far more effective than in a row format. On top of that, Parquet stores per-block statistics (minimum, maximum, null count), which lets the engine skip entire blocks that are irrelevant to the query (predicate pushdown). Less data read means faster and cheaper queries.
The flip side: writing a single record at a time in Parquet is inefficient. The format was designed to write many records at once and read many times — not for constant updates of individual rows.
ORC: the columnar cousin from the Hive world
Apache ORC (Optimized Row Columnar) was born in the Hive ecosystem and solves the same problem as Parquet, also in a columnar way. In practice, the two are more alike than different: both compress well, both store statistics, both accelerate analytical queries.
The differences are matters of tuning. ORC tends to achieve slightly better compression ratios in some scenarios and brings lightweight indexes and robust support for ACID transactions when used inside Hive. Parquet, in turn, has more universal support outside the Hadoop world — libraries, engines and cloud services speak Parquet natively.
The rule of thumb: if your environment is heavily Hive/Hadoop, ORC feels at home. If you want the common denominator that works everywhere, Parquet is the safe bet. Choosing between the two rarely decides the success of a platform — using a columnar format instead of a row one, that does make a difference.
Avro: when the row and the schema rule
Apache Avro plays in a different league. It is row-oriented, which makes it a poor candidate for analytical queries that scan few columns — but excellent for something else: moving data and writing records as they arrive.
Avro stores the schema alongside the data, in JSON, and takes schema evolution seriously: you can add a new field, give it a default value and keep reading old files without breaking anything. That is why it is the natural format for event streams — it is ubiquitous in Kafka pipelines — and for the layer where data lands raw before being transformed.
Think of Avro as the format for transport and landing, and Parquet/ORC as the format for the analytical warehouse. Many architectures use both: they receive events in Avro and, further down the pipeline, rewrite them in Parquet for querying.
Compression, schema and evolution
All three formats support compression, but with nuances. In columnar formats, because similar values sit side by side, algorithms like Snappy (fast) or ZSTD (more compact) pay off a lot. Choosing the codec is a classic trade-off: compressing more saves storage but spends more CPU on reading and writing.
As for the schema, all of them store it with the data — which avoids the nightmare of CSV files without a reliable header. The difference is in the culture of evolution: Avro was designed around it, with clear rules of forward and backward compatibility. Parquet and ORC also evolve schema, but the discipline sits more on the table side (for example, through table formats like Delta Lake or Iceberg, which manage metadata on top of the files).
The small files problem
There is a trap that catches almost everyone, and it does not depend on the format: small files. A data lake with millions of tiny files — the result, for instance, of streaming writes every few seconds — is slow regardless of whether it is Parquet, ORC or Avro. Each file has a fixed cost of opening and reading metadata; multiplied by millions, that cost dominates everything.
- Symptom: simple queries take forever and the engine spends more time listing files than reading data.
- Cause: frequent ingestion that generates many small files, without consolidation.
- Cure: compaction processes that merge small files into larger ones (ideally in the hundreds of MB), and sensible partitioning that does not fragment the data excessively.
Choosing the right format does not save a platform full of small files. Both problems have to be solved together.
How to choose without overcomplicating
Reducing it all to the essentials, the decision fits in three questions:
- Will you query few columns from large tables? Use a columnar format — Parquet as the default choice, ORC if you live in the Hive world.
- Will you move events or land raw data that changes shape? Avro, for schema evolution and record-by-record writing.
- Do you need transactions, time travel or updates? The answer is no longer just the file format, but a table format (Delta Lake, Iceberg, Hudi) on top of Parquet or ORC.
In most platforms, the winning combination is mundane: Avro (or JSON) at the entry, Parquet in the analytical warehouse. You do not need more exoticism than that to get it right.
Mini-case: a retailer tidying up the data lake
A retailer with a physical and online store kept everything — sales, clicks, stock — in compressed JSON files, written hourly. Daily reports took around 40 minutes and the compute bill kept growing. The team thought it needed a more powerful engine; in fact, it needed to tidy up the house.
They made two changes. First, they started rewriting historical data in Parquet partitioned by date, with ZSTD compression. Second, they introduced a nightly compaction step that consolidated the thousands of small files from the day into a few large ones. They kept Avro only in the entry layer, where the events arrive.
The result: reports that took 40 minutes started running in about 6, the volume of data read per query dropped to a fraction of the previous one, and storage shrank by nearly 60% thanks to better compression. No new engine, no rewriting of the application — just the right format, in the right place, with no small files getting in the way.
In practice
File formats are not the most glamorous topic of a data platform, but they are among those that most quietly decide its cost and its speed. The good news is that the decision is simple once you understand the logic: columnar (Parquet or ORC) to query, row (Avro) to transport and land.
Before switching engines or increasing resources because of slow queries, it is worth looking down, at the format and the size of the files. Often, the biggest gain is right there — and it does not cost a new licence, only a well-made engineering decision.
