Repository navigation
perf(parquet): stream page-filtered row group decoding - #563
Merged
Merged
Conversation
lxy-9602
reviewed
Nov 17, 2026
SteNicholas
force-pushed
the
PAIMON-560
branch
from
November 17, 2026 12:47
f1816d2 to
c34d489
Compare
Page-filtered Parquet reads decoded all selected rows before returning the first batch, retaining decoded columns for an entire row group. This delayed first-batch delivery and made decoded-output memory grow with the selected row count. Incremental decoding bounds that output by the requested batch size while preserving selected-row alignment. - Keep field decoders and per-leaf selection cursors across ReadNext calls to decode one bounded output window at a time, addressing apache#560. - Assemble RecordBatches directly for single-chunk columns and align chunked output within each window. Preserve projected schemas, dictionaries, nested rows, file row IDs and returned-batch ownership. - Cache decoding errors, release decoder state on EOF or close and reset active filtered readers when read ranges or projections change. - Measure and document first-batch latency and peak tracked Arrow memory alongside throughput and storage bytes. Compare partial row groups across batch sizes with page-index filtering enabled and disabled.
SteNicholas
force-pushed
the
PAIMON-560
branch
from
November 17, 2026 13:33
c34d489 to
9090426
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #560
Page-filtered Parquet reads decoded every selected row of every projected field before returning
any batch. The reader retained decoded columns for the entire filtered row group, delaying the
first batch and making decoded-output memory grow with the selected row count.
The change follows the stateful batching approach used by Velox's Parquet row reader, retaining
field decoders and advancing them by the requested batch size:
PageFilteredRowGroupReader: keeps per-field decoders, per-leaf selection cursors andskip/read progress across
ReadNextcalls. Each call decodes a bounded window of selectedlogical rows, including nested leaves whose page boundaries and compressed row coordinates
differ. Existing page pruning, pre-buffering and missing-index fallback remain in use.
RecordBatchoutput directly when every column has one chunk.Columns split into multiple chunks are aligned within the current decoding window, with
array offsets normalized. The reader exposes the projected schema before reading data and
preserves dictionary output, nested rows and ownership of returned batches.
keeps previously returned batches valid.
FileReaderWrapper::PrepareForReadingdiscardsthe active filtered reader when read ranges or projections change; file row IDs continue
to refer to the original selected rows across batches and row groups.
alongside throughput and storage bytes. A partial-row-group benchmark compares 64, 1024
and 4096 rows per batch with page-index filtering enabled and disabled.
The memory bound applies to decoded output for the active window. Pre-buffered input, required
page/decoder state and batches retained by the caller remain separate memory costs. Generic
late-materialization probe/payload streaming is outside this change.
Tests
UT:
src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp:MultiColumnMultiBatchStreaming: consolidates overlapping batch/offset tests and checksinteger/string column alignment, exact batch sizes, contents and normalized offsets.
NestedStructSubColumnProjectionandNestedColumnsWithNullsMisalignedPagesMultiRowGroup: projected nested schemas, nulls,misaligned struct/list/map pages, mixed full/partial row groups, original file row IDs and
returned-array validity after reader destruction, with and without page indexes.
StreamingPreservesDictionaryValuesAcrossSelectionsandDictionaryEmptySelectionDoesNotReadPages: dictionary schemas and values across batches,invalid batch sizes, repeated EOF and empty selections without dictionary/data-page reads.
StreamingBoundsDecodedMemoryAndPreservesReturnedBatches: bounded tracked memory duringfirst-batch and complete reads, early/repeated close, decoder cleanup and returned-batch
ownership after reader destruction.
StreamingDefersLaterPageDecodeErrors: a corrupt later page does not prevent earlierbatches from being returned; subsequent reads preserve the failure and prior batches.
DirectOffsetIndexJumpDoesNotReadUnselectedPageHeaders,DirectOffsetIndexJumpSupportsDictionaryFallbackToPlainandDirectOffsetIndexJumpDataPageV2AdjacentPages: small-batch streaming preserves page pruning,dictionary-to-plain transitions and DataPageV2 page-boundary handling.
MissingOffsetIndexFallsBackToSequentialReadandInvalidOffsetIndexFallsBackWithoutChangingSelectedRows: fallback preserves selected rowsacross batches, including nested fields with one valid and one missing or malformed index.
src/paimon/format/parquet/file_reader_wrapper_test.cpp:SeekBackToConsumedPageFilteredRowGroup: seeking back and restarting a partially consumedfiltered row group reconstructs the reader and returns the same rows.
PageFilteredRespectsBatchSize: exact output sizes and original row positions acrossseveral batch sizes, including an empty column projection.
ApplyReadRangesWiderSecondCallandPrepareForReading: changing ranges or projectionsduring streaming resets the active reader, preserves retained batches and works with
immediate and lazy preparation.
IT:
No standalone integration tests are added. The format-layer tests exercise the end-to-end
ParquetFileBatchReaderpath, including bitmap selection and file row ID translation.API and Format
No public API under
include/, stored file format or table protocol changes. The existingParquet batch-size setting now bounds incremental decoding in the page-filtered path.
Documentation
docs/source/examples/benchmark.rstdescribes the first-batch latency and tracked-memorycounters, their measurement boundaries and the filtered batch-size comparison. It also explains
how to use storage-byte counters to distinguish decoding changes from page pruning.
Generative AI tooling
Generated-by: Codex (GPT-6)