Repository navigation
feat(reader): bound reader preparation across splits - #576
Merged
Merged
Conversation
SteNicholas
force-pushed
the
PAIMON-561
branch
2 times, most recently
from
November 18, 2026 12:18
639eb52 to
080101e
Compare
lxy-9602
reviewed
Nov 18, 2026
Append/raw reads prepared up to four files per split. Opening every split before the first batch made the number of live readers grow with the plan, even when the caller stopped after a few rows. Deferring split opening also requires keeping planned real-time views available after their tickets are consumed. - Add the C++ read.preparation-window option (default 4) to bound prepared and in-flight file readers independently of build parallelism, batch prefetch and warmup, addressing apache#561. - Open append-only and primary-key splits in plan order, closing each before opening the next. Capture the delete-row policy and retain the read state so the reader can outlive its TableRead. - Validate disk split metadata and pin real-time views before consuming tickets. Rejected plans preserve valid tickets for retry. Accepted plans retain their views across expiry and committed-state refreshes. - Return deferred failures in file order and preserve status details when adding split context. Drain pending builds before collecting final metrics and release unused factories on EOF, close or error.
SteNicholas
force-pushed
the
PAIMON-561
branch
from
November 18, 2026 20:59
080101e to
ebe1e01
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
#564 limited file preparation inside a single append/raw split. A scan over
Sraw splits still opened every split up front and could retain up to4 × Sfile readers before the first batch. This follow-up for #561 makes the window configurable and opens splits on demand. Window sizing discussion · Split initialization discussionTableRead.Tests
API and Format
The C++ option
Options::READ_PREPARATION_WINDOW(read.preparation-window) accepts a positive 32-bit integer and defaults to4. It bounds prepared and in-flight file readers. A value of1prepares only the current file. Executor parallelism, batch prefetch and warmup are configured independently.CreateReader(split)reports file-open and build errors from the initial window at creation. WithCreateReader(splits),NextBatchorNextBatchWithBitmapreports a terminal error when it reaches the affected split or file. Both paths use the existing table and serialized split formats.Documentation
The prefetch guide includes a configuration example and describes the window, reader lifetime, error timing and real-time ticket handling. Sequential file preparation
Generative AI tooling
Generated-by: Codex (GPT-6)