Blog

Best Answer: What is Apache Parquet?

Key Takeaways

  • Apache Parquet is an open-source columnar file format suited to analytical queries that read selected columns across many rows.
  • Files organize rows into row groups, columns into column chunks, and chunks into pages. Metadata helps readers locate data and skip irrelevant portions.
  • Encodings and compression codecs are different layers; neither guarantees that every dataset becomes smaller or every query becomes faster.
  • Parquet doesn't require Hadoop or HDFS, and it doesn't provide database transactions or table management by itself.

Apache Parquet is a binary file format that stores data by column within groups of rows. Its layout lets analytical engines read selected columns without decoding every field, while metadata can help them skip data that cannot match a filter.

Parquet grew out of the Hadoop ecosystem, but it isn't tied to HDFS: readers can access files on local filesystems, distributed filesystems, and object storage. Its efficiency depends on your data, writer settings, storage system, and query engine—not just the .parquet extension.

How Parquet Works

Columnar Storage Within Row Groups

A row-oriented representation keeps a record's fields together. Parquet instead groups values from the same column together within each row group; an entire column need not occupy one uninterrupted region across the file.

The Parquet layout terminology describes three levels:

  • Row group: a horizontal slice of the table containing a set of rows.
  • Column chunk: the contiguous data for one leaf column within a row group.
  • Page: a subdivision of a column chunk used for encoding and compression. Chunks contain data pages and may include a dictionary page.

The Anatomy of a Parquet File

This simplified layout shows a conventional, unencrypted, self-contained file. Row-group labels describe logical organization rather than separate row-group headers.

PAR1                                      leading magic bytes
Row group 0
  Column A chunk: [dictionary?] [data page] [data page] ...
  Column B chunk: [dictionary?] [data page] ...
Row group 1
  Column A chunk: [dictionary?] [data page] ...
  Column B chunk: [dictionary?] [data page] ...
...
Optional auxiliary structures             e.g. page indexes
File metadata                             schema, row groups,
                                          column offsets and sizes;
                                          statistics when present
Metadata length                           4 bytes, little-endian
PAR1                                      trailing magic bytes

The file format specification explains the read path: readers first locate footer metadata, then use column-chunk offsets to find the data they need. The trailing metadata length makes that metadata locatable from the end of the file.

Projection and Predicate Pushdown

Projection means reading only the columns a query needs. Predicate pushdown means passing a filter down to the reader so it can avoid unnecessary work before returning rows.

Imagine an events table with 40 columns. Your query needs country and amount_cents, filtered to dates on or after October 1, 2026. Without an external index, a CSV scan must traverse records containing every column, even if its parser avoids converting unused fields.

With Parquet, a capable reader can:

  1. Read metadata to identify the relevant column chunks.
  2. Skip an entire row group whose recorded maximum date is September 30, 2026.
  3. Read the projected columns and the filter column in the remaining groups, then evaluate the filter.
  4. Avoid decoding unrelated payload columns.

Optional page indexes can enable finer-grained page skipping. Bloom filters can rule out chunks for some equality searches; a positive result means a value might exist, not that it definitely does.

These are capabilities, not promises. Statistics or indexes must be present and usable, and the reader must support them. Metadata and surviving data still need to be read; buffering and read-ahead can also fetch extra bytes. Parquet pruning isn't a general-purpose point-lookup index.

Parquet Encodings Versus Compression Codecs

Encodings change how values are represented. Dictionary encoding replaces values with IDs into a dictionary; run-length encoding represents repeated runs compactly, including runs of dictionary IDs or nesting levels.

Compression codecs, such as Snappy, Zstandard, and Gzip, compress page data after encoding. A dictionary-encoded column can also use Zstandard compression: these aren't competing settings.

Grouping similar values can improve compression, but high-cardinality values, already-compressed payloads, and small datasets may benefit less. Benchmark your codec and encoding choices against storage size, write throughput, and read CPU rather than assuming one setting wins everywhere.

Interoperability, Schemas, and Nested Data

Parquet distinguishes physical storage types from logical type annotations. For example, BYTE_ARRAY with a STRING annotation represents UTF-8 text; INT32 with a DATE annotation represents a date rather than an ordinary integer.

Nested schemas support structures, lists, and maps. Definition and repetition levels preserve their structure: definition levels identify which optional values or ancestors are present, while repetition levels identify where repeated structures continue or restart.

Apache Spark, Hive, Drill, Impala, and PyArrow support Parquet, but that doesn't mean every reader supports every codec, logical type, or optional index. Check the implementation support matrix and test representative files, especially timestamps, decimals, nulls, and nested values.

Converting Avro, Thrift, or Protocol Buffers data requires an application or library to map its schema and values into Parquet. The format doesn't automatically perform those conversions. For the processing-engine context, see what Apache Spark does.

A Runnable PyArrow Example

In a Python virtual environment, install PyArrow:

python -m pip install pyarrow

Save this as parquet_example.py and run python parquet_example.py. It creates an explicit schema, writes two deliberately tiny row groups, and reads selected columns with a date filter. The temporary directory is cleaned up afterward.

from datetime import date
from tempfile import TemporaryDirectory

import pyarrow as pa
import pyarrow.parquet as pq

schema = pa.schema([
    ("event_date", pa.date32()),
    ("country", pa.string()),
    ("amount_cents", pa.int64()),
    ("request_id", pa.string()),
])

table = pa.table(
    {
        "event_date": [date(2026, 9, 30)] * 3
                      + [date(2026, 10, 1)] * 3,
        "country": ["US", "CA", "US", "US", "CA", "US"],
        "amount_cents": [100, 200, 300, 400, 500, 600],
        "request_id": ["r1", "r2", "r3", "r4", "r5", "r6"],
    },
    schema=schema,
)

with TemporaryDirectory() as directory:
    path = f"{directory}/events.parquet"
    pq.write_table(
        table,
        path,
        compression="zstd",
        use_dictionary=["country"],
        row_group_size=3,
        write_statistics=True,
    )
    result = pq.read_table(
        path,
        columns=["country", "amount_cents"],
        filters=[("event_date", ">=", date(2026, 10, 1))],
    )
    print(result.to_pylist())

The result contains the three October rows and only the two requested output columns. Date statistics allow the first row group to be skipped; request_id isn't needed for either projection or filtering. This demonstrates layout and pruning, not a performance benchmark.

In write_table(), row_group_size is a row count, not a byte target. The read_table() options control projection and filtering; Arrow's Parquet guide explains statistics-based row-group skipping. Three-row groups are for illustration, not production sizing.

Why Use Parquet—and When Not To

Parquet is a strong candidate for OLAP—analytical processing that scans and aggregates many rows—especially when queries use only part of a wide schema. That's different from the small, frequently updated records typical of transaction processing; see OLTP versus OLAP.

FormatOrganizationChoose it when
ParquetTyped column chunks within row groupsYou repeatedly scan selected columns and can benefit from pruning.
CSVText records with delimited fieldsHuman inspection or simple interchange matters more than typed columnar scans.
AvroSchema-based binary recordsYou serialize and process whole records rather than prioritize columnar scans.

The CSV format description and Avro specification explain those representations. For a closer comparison, read Avro versus Parquet.

Read Efficiency Has a Write Cost

Parquet writers organize column data, build encodings and metadata, and optionally compress pages. That work consumes CPU and buffering resources, but it doesn't establish a universal write-speed ranking. Favor Parquet when repeated analytical reads justify the write work; measure carefully for tiny files, full-record scans, or latency-sensitive ingestion.

A File Format Isn't a Table Format

Parquet describes files, not how concurrent writers commit a table or how readers select a consistent table snapshot. A table format such as Apache Iceberg adds table-level metadata and commit semantics around data files, including Parquet files. A directory of Parquet files alone doesn't provide those guarantees.

Production Checklist

Use Arrow's dataset guidance as a starting point, then tune against your workload:

  • Size row groups deliberately. Larger groups can amortize metadata and improve compression, but increase buffering and make pruning coarser. Very small groups add metadata overhead.
  • Control file counts. Many tiny files add discovery, file-open, and metadata costs. Compact small outputs, while keeping enough independent work for parallel readers.
  • Partition for common filters. Partitioning can eliminate whole files before their data is read. Avoid high-cardinality partition keys that fragment output; Hive-style partitioning explains the directory convention.
  • Consider sorting. Clustering related filter values can narrow row-group min/max ranges, but sorting adds write work.
  • Validate compatibility. Test schema changes, logical types, codecs, and optional indexes with every important reader.
  • Measure actual work. Compare bytes read, files and row groups skipped, CPU, memory, and end-to-end latency—not just final file size.

FAQ

What is Apache Parquet used for?

Apache Parquet is used to store typed analytical datasets for column-selective scans and aggregations. It works with local files, object storage, and distributed filesystems; Hadoop and HDFS aren't required.

What is the difference between a Parquet encoding and a compression codec?

An encoding changes how values are represented, such as replacing repeated strings with dictionary IDs. A compression codec such as Zstandard or Snappy compresses page data after encoding, so both mechanisms can be used together.

Can Parquet schemas evolve across files?

Files can have different schemas, but the reader or table layer must reconcile compatible differences. Adding a nullable column isn't the same as safely renaming a field or changing its type; test the exact evolution with your consumers.

How large should Parquet files and row groups be?

There is no universally optimal size: row width, memory, storage latency, parallelism, and filter selectivity all matter. Avoid tiny files and excessively small row groups, and distinguish row-count settings from byte-size targets.

Is Parquet a database or a file format?

Parquet is a file format, not a database or transaction protocol. A query engine reads the files, while a table format such as Iceberg can add table metadata, snapshots, and atomic commits.

Further Reading

Written by Sam Erickson

Software engineer. I write every article on StackChief myself. More about me

Related reading