Skip to content

feat(reader): bound reader preparation across splits - #576

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-561
Nov 19, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-561

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Purpose

#564 limited file preparation inside a single append/raw split. A scan over S raw splits still opened every split up front and could retain up to 4 × S file readers before the first batch. This follow-up for #561 makes the window configurable and opens splits on demand. Window sizing discussion · Split initialization discussion

  • Append-only and primary-key readers open splits in plan order and close each before opening the next. Closing before the first batch opens no data files. A real-time split can group multiple disk inputs, and merge-on-read opens the inputs it needs together.
  • Creating a reader for a split list validates disk split metadata and pins real-time views before consuming any tickets. A rejected plan leaves valid tickets available for retry. The reader retains its pinned views and the selected delete-row policy, so deferred reads can outlive the TableRead.
  • A deferred error in a split list names the affected split and preserves the underlying status. Closing drains pending builds before collecting final metrics and releases unopened split factories.

Tests

  • Option and reader tests cover window bounds, concurrent preparation, ordered refill, next-file warmup and cancellation. A counting file system checks how many streams open before the first batch and how many remain live after early close.
  • Raw and multi-split tests check file order, bitmap selection, deletion vectors, row kinds, row IDs and metrics. They also exercise returned batches after close and deferred readers after their creator is destroyed.
  • Real-time integration tests cover malformed disk and indexed splits, ticket reuse after validation failure, and reads across ticket expiry and committed-state refreshes. Nested projection and aggregation tests check failures when reading reaches the affected split.

API and Format

The C++ option Options::READ_PREPARATION_WINDOW (read.preparation-window) accepts a positive 32-bit integer and defaults to 4. It bounds prepared and in-flight file readers. A value of 1 prepares only the current file. Executor parallelism, batch prefetch and warmup are configured independently.

CreateReader(split) reports file-open and build errors from the initial window at creation. With CreateReader(splits), NextBatch or NextBatchWithBitmap reports a terminal error when it reaches the affected split or file. Both paths use the existing table and serialized split formats.

Documentation

The prefetch guide includes a configuration example and describes the window, reader lifetime, error timing and real-time ticket handling. Sequential file preparation

Generative AI tooling

Generated-by: Codex (GPT-6)

@SteNicholas
SteNicholas force-pushed the PAIMON-561 branch 2 times, most recently from 639eb52 to 080101e Compare November 18, 2026 12:18
Comment thread include/paimon/defs.h Resolved
Append/raw reads prepared up to four files per split. Opening every
split before the first batch made the number of live readers grow with
the plan, even when the caller stopped after a few rows. Deferring split
opening also requires keeping planned real-time views available after
their tickets are consumed.

- Add the C++ read.preparation-window option (default 4) to bound
  prepared and in-flight file readers independently of build
  parallelism, batch prefetch and warmup, addressing apache#561.
- Open append-only and primary-key splits in plan order, closing each
  before opening the next. Capture the delete-row policy and retain the
  read state so the reader can outlive its TableRead.
- Validate disk split metadata and pin real-time views before consuming
  tickets. Rejected plans preserve valid tickets for retry. Accepted
  plans retain their views across expiry and committed-state refreshes.
- Return deferred failures in file order and preserve status details
  when adding split context. Drain pending builds before collecting
  final metrics and release unused factories on EOF, close or error.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit c8ac78d into apache:main Nov 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants