Parquet

Apache Parquet is a column-oriented binary storage format optimized for analytical workloads. Originally developed within the Apache Hadoop ecosystem, Parquet provides efficient compression and encoding schemes for large-scale data processing.

Binary Layout

Section	Internal Name	Description	Possible Values / Format
File Header	magic	4-byte magic number identifying Parquet files	ASCII: PAR1 (hex: 50 41 52 31)
Row Group	row_group_metadata	Metadata for each row group	Contains column chunk metadata and statistics
	column_chunk	Data for each column in the row group	Compressed and encoded column data
File Footer	metadata	File-level metadata including schema and row groups	Thrift-encoded metadata structure
	metadata_length	Length of metadata section	4-byte little-endian integer
	magic	Footer magic number	ASCII: PAR1 (hex: 50 41 52 31)

Column Storage Example

Row-based storage (traditional):

id,name,last_name,age
1,John,Buck,35
2,Jane,Doe,27
3,Joe,Dane,42

Column-based Storage (Parquet):

id: [1, 2, 3]
name: [John, Jane, Joe]
last_name: [Buck, Doe, Dane]
age: [35, 27, 42]

Encoding Types

Encoding	Internal Name	Description	Use Case
Plain	PLAIN	Values encoded back to back without compression	Default fallback for all data types
Dictionary	RLE_DICTIONARY	Values replaced with dictionary indices using RLE	Repeated string values, low-cardinality columns
Run Length / Bit-Packing Hybrid	RLE	Combination of bit-packing and run length encoding	Repetition/definition levels, dictionary indices, booleans
Delta Binary Packed	DELTA_BINARY_PACKED	Delta encoding with binary packing for integers	INT32, INT64 with sequential or clustered values
Delta Length Byte Array	DELTA_LENGTH_BYTE_ARRAY	Delta-encoded lengths followed by concatenated data	Variable-length byte arrays
Delta Byte Array	DELTA_BYTE_ARRAY	Incremental/front compression storing prefix lengths	BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY with common prefixes
Byte Stream Split	BYTE_STREAM_SPLIT	Scatters bytes to separate streams for better compression	FLOAT, DOUBLE, INT32, INT64 (added in Parquet 2.8)

Deprecated Encodings

Encoding	Internal Name	Description	Replacement
Plain Dictionary	PLAIN_DICTIONARY	Legacy dictionary encoding in data pages	Use RLE_DICTIONARY in data pages
Bit Packed	BIT_PACKED	Fixed-width bit-packing without padding	Use RLE hybrid encoding

Compression Codecs

Codec	Description	Best For
UNCOMPRESSED	No compression applied	Testing or very small files
SNAPPY	Fast compression/decompression	General-purpose, balanced performance
GZIP	Higher compression ratio	Storage-constrained environments
LZO	Fast decompression	Read-heavy workloads
BROTLI	Modern compression algorithm	High compression ratio needs
LZ4	Extremely fast compression	Low-latency applications
ZSTD	Zstandard compression with configurable levels	Best balance of speed and compression ratio