Skip to content

perf(parquet): stream page-filtered row group decoding - #563

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-560
Nov 18, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-560

Conversation

@SteNicholas

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #560

Page-filtered Parquet reads decoded every selected row of every projected field before returning
any batch. The reader retained decoded columns for the entire filtered row group, delaying the
first batch and making decoded-output memory grow with the selected row count.

The change follows the stateful batching approach used by Velox's Parquet row reader, retaining
field decoders and advancing them by the requested batch size:

  • PageFilteredRowGroupReader: keeps per-field decoders, per-leaf selection cursors and
    skip/read progress across ReadNext calls. Each call decodes a bounded window of selected
    logical rows, including nested leaves whose page boundaries and compressed row coordinates
    differ. Existing page pruning, pre-buffering and missing-index fallback remain in use.
  • Batch assembly: creates RecordBatch output directly when every column has one chunk.
    Columns split into multiple chunks are aligned within the current decoding window, with
    array offsets normalized. The reader exposes the projected schema before reading data and
    preserves dictionary output, nested rows and ownership of returned batches.
  • Reader lifecycle: caches decoding errors, releases decoder state at EOF or close and
    keeps previously returned batches valid. FileReaderWrapper::PrepareForReading discards
    the active filtered reader when read ranges or projections change; file row IDs continue
    to refer to the original selected rows across batches and row groups.
  • Parquet benchmarks: report mean first-batch latency and peak tracked Arrow memory
    alongside throughput and storage bytes. A partial-row-group benchmark compares 64, 1024
    and 4096 rows per batch with page-index filtering enabled and disabled.

The memory bound applies to decoded output for the active window. Pre-buffered input, required
page/decoder state and batches retained by the caller remain separate memory costs. Generic
late-materialization probe/payload streaming is outside this change.

Tests

UT:

  • src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp:
    • MultiColumnMultiBatchStreaming: consolidates overlapping batch/offset tests and checks
      integer/string column alignment, exact batch sizes, contents and normalized offsets.
    • NestedStructSubColumnProjection and
      NestedColumnsWithNullsMisalignedPagesMultiRowGroup: projected nested schemas, nulls,
      misaligned struct/list/map pages, mixed full/partial row groups, original file row IDs and
      returned-array validity after reader destruction, with and without page indexes.
    • StreamingPreservesDictionaryValuesAcrossSelections and
      DictionaryEmptySelectionDoesNotReadPages: dictionary schemas and values across batches,
      invalid batch sizes, repeated EOF and empty selections without dictionary/data-page reads.
    • StreamingBoundsDecodedMemoryAndPreservesReturnedBatches: bounded tracked memory during
      first-batch and complete reads, early/repeated close, decoder cleanup and returned-batch
      ownership after reader destruction.
    • StreamingDefersLaterPageDecodeErrors: a corrupt later page does not prevent earlier
      batches from being returned; subsequent reads preserve the failure and prior batches.
    • DirectOffsetIndexJumpDoesNotReadUnselectedPageHeaders,
      DirectOffsetIndexJumpSupportsDictionaryFallbackToPlain and
      DirectOffsetIndexJumpDataPageV2AdjacentPages: small-batch streaming preserves page pruning,
      dictionary-to-plain transitions and DataPageV2 page-boundary handling.
    • MissingOffsetIndexFallsBackToSequentialRead and
      InvalidOffsetIndexFallsBackWithoutChangingSelectedRows: fallback preserves selected rows
      across batches, including nested fields with one valid and one missing or malformed index.
  • src/paimon/format/parquet/file_reader_wrapper_test.cpp:
    • SeekBackToConsumedPageFilteredRowGroup: seeking back and restarting a partially consumed
      filtered row group reconstructs the reader and returns the same rows.
    • PageFilteredRespectsBatchSize: exact output sizes and original row positions across
      several batch sizes, including an empty column projection.
    • ApplyReadRangesWiderSecondCall and PrepareForReading: changing ranges or projections
      during streaming resets the active reader, preserves retained batches and works with
      immediate and lazy preparation.

IT:

No standalone integration tests are added. The format-layer tests exercise the end-to-end
ParquetFileBatchReader path, including bitmap selection and file row ID translation.

API and Format

No public API under include/, stored file format or table protocol changes. The existing
Parquet batch-size setting now bounds incremental decoding in the page-filtered path.

Documentation

docs/source/examples/benchmark.rst describes the first-batch latency and tracked-memory
counters, their measurement boundaries and the filtered batch-size comparison. It also explains
how to use storage-byte counters to distinguish decoding changes from page pruning.

Generative AI tooling

Generated-by: Codex (GPT-6)

@lxy-9602
lxy-9602 requested a review from lucasfang November 16, 2026 00:56
lxy-9602
lxy-9602 previously approved these changes Nov 17, 2026

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Page-filtered Parquet reads decoded all selected rows before returning
the first batch, retaining decoded columns for an entire row group.
This delayed first-batch delivery and made decoded-output memory grow
with the selected row count. Incremental decoding bounds that output
by the requested batch size while preserving selected-row alignment.

- Keep field decoders and per-leaf selection cursors across ReadNext
  calls to decode one bounded output window at a time, addressing apache#560.
- Assemble RecordBatches directly for single-chunk columns and align
  chunked output within each window. Preserve projected schemas,
  dictionaries, nested rows, file row IDs and returned-batch ownership.
- Cache decoding errors, release decoder state on EOF or close and
  reset active filtered readers when read ranges or projections change.
- Measure and document first-batch latency and peak tracked Arrow memory
  alongside throughput and storage bytes. Compare partial row groups
  across batch sizes with page-index filtering enabled and disabled.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 69d14e5 into apache:main Nov 18, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Stream decoding of page-filtered Parquet row groups

2 participants