Skip to content

feat(reader): lazily initialize sequential file readers - #564

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-561
Nov 17, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-561

Conversation

@SteNicholas

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #561

Sequential append/raw reads constructed every file reader before returning, retaining open
streams and format metadata for the entire split. Splits with many small files delayed the
first batch and paid setup costs for files that early-stopping readers never consumed.

The change prepares an ordered, bounded window of readers while retaining concurrent
construction and next-file warmup:

  • ConcatBatchReader: adds lazy construction with a window of at most four reader stacks,
    including in-flight builds. It releases exhausted readers, retains their metrics and refills
    the window as consumption advances. Ready lookahead readers are warmed without blocking
    reads from the current file; skipped and empty files preserve the split's file order.
  • RawFileSplitRead: uses the lazy path for sequential append/raw reads and retains an
    independent creator with its context, copied partition, file metadata, schema inputs,
    filesystem and memory pools. Per-file predicates, deletion vectors, row kinds and file-row
    IDs retain their existing behavior. Merge-on-read keeps its eager input-construction helpers.
  • Reader lifecycle: validates plans eagerly and reports initial-window construction errors
    from CreateReader. Later file-open/build failures become terminal read errors when the
    affected file is reached. Close, destruction and failure cancel queued builds, drain running
    work and release prepared readers without scheduling the unopened tail. Returned batches
    remain valid after their readers are released.
  • Sequential-reader benchmarks: compare eager and lazy construction over 128 or 1024 small
    files, for full scans and first-batch termination. Counters cover first-batch latency,
    file opens, initial footer/setup reads, peak retained streams, tracked pool allocation,
    bytes read and throughput, with an optional per-open delay to simulate remote setup latency.

The construction bound applies to reader stacks, including work in flight. Per-file metadata
and owned creation inputs still scale with the split's file count; batches retained by callers
remain a separate memory cost. The construction window is independent of batch prefetch and
warmup settings.

Tests

UT:

  • src/paimon/common/reader/concat_batch_reader_test.cpp:
    • TestSimple and TestSimpleWithBitmap: reuse the existing data and bitmap cases for eager
      and lazy readers, including empty inputs and empty files across several batch sizes.
    • TestWarmupLooksOneReaderAhead: checks eager and lazy warmup ordering, including refilled
      windows, without warming beyond the next reader.
    • LazyWindowPreservesOrderBitmapAndMetrics: checks bounded live readers, skipped and empty
      files, ordered output, both read APIs, accumulated metrics and creator release at EOF.
    • LazyEarlyCloseAndDestructionReleaseCreators: verifies bounded first-batch construction,
      early/repeated close, destruction, creator cleanup and returned-batch validity.
    • LazyInitialFailureDrainsPartialWindow and LazyDeferredFailuresAreTerminal: cover partial
      initialization cleanup and terminal build, read and bitmap-conversion failures.
    • LazyRefillOverlapsReadingAndCloseCancelsQueuedBuilds: gates a refill to verify that reading
      continues while construction is in flight; close and destruction drain running work and
      prevent queued factories from opening additional files.
  • src/paimon/core/operation/raw_file_split_read_test.cpp:
    • LazyInitialWindowBuildsConcurrentlyAndEarlyCloseStopsOpening: uses a gated counting
      filesystem to check concurrent initial preparation, bounded first-batch opens and cleanup
      without opening the tail after early close.
    • LazyReaderOutlivesCreatorWithPerFilePredicatesDvAndRowIds: reads across multiple windows
      after releasing caller-owned state, checking per-file predicate ownership, deletion vectors,
      row kinds, row IDs, retained output and bounded resources.
    • LazyValidationAndFileOpenFailures: consolidates invalid-plan checks, including the null
      predicate-provider case, and verifies that validation opens no data files. It also checks
      initial and deferred missing-file errors, repeated failure propagation and stream cleanup.

IT:

No standalone integration tests are added. The raw-reader tests exercise real ORC files through
RawFileSplitRead, including filesystem opens, filtering, deletion vectors and row tracking.

API and Format

No public API under include/, stored file format or table protocol changes. File-open/build
failures beyond the initial preparation window are now reported through NextBatch or
NextBatchWithBitmap when the affected file is reached, instead of during CreateReader.

Documentation

docs/source/user_guide/prefetch.rst describes bounded sequential preparation, deferred-error
semantics, ownership and cleanup, and its relationship to next-file warmup.
docs/source/examples/benchmark.rst documents the eager/lazy comparison, full-scan and early-stop
cases, counters and measurement boundaries. It distinguishes tracked pool allocation from
process RSS and simulated open latency from a complete remote-storage model.

Generative AI tooling

Generated-by: Codex (GPT-6)

@SteNicholas
SteNicholas force-pushed the PAIMON-561 branch 3 times, most recently from 598d55e to ca24f9e Compare November 16, 2026 07:32
Sequential append/raw reads constructed every file reader before
returning, retaining open streams and format metadata for the entire
split. Splits with many small files delayed the first batch and paid
setup costs for files that early-stopping readers never consumed.

- Prepare an ordered window of at most four reader stacks, including
  in-flight builds, for sequential append/raw reads, addressing apache#561.
  Preserve concurrent construction and next-file warmup.
- Release exhausted readers, retain their metrics and refill the window
  as files are read. Preserve file order, filtering, deletion vectors
  and row tracking; keep merge-on-read input construction eager.
- Own deferred creation state beyond the caller's lifetime. Validate
  plans eagerly and report later file-open/build failures as terminal
  read errors when the affected file is reached. Cancel queued builds
  and drain running work on close, destruction or failure.
- Cover bounded construction, deferred failures and creator lifetimes.
  Compare eager and lazy full scans and early stops with small-file
  benchmarks for first-batch latency, setup I/O and peak resources.

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lxy-9602

Copy link
Copy Markdown
Member

Non-blocking: the four-reader window is scoped to each split. CreateReader(splits) still eagerly creates a reader for every split, so with S splits containing at least four files each, it can retain up to 4 × S file readers before returning the first batch. Early termination also pays the initial preparation cost for splits that are never consumed.

Could we consider lazy initialization at the split level as a follow-up? If that requires substantial changes, leaving a TODO near the multi-split creation loop would be sufficient for this PR.

@SteNicholas

Copy link
Copy Markdown
Member Author

Thanks @lxy-9602. Your observation is correct: the four-reader bound is per raw split. CreateReader(splits) still constructs every split reader eagerly, so S raw splits with at least four files each can retain up to 4 × S file readers before the first batch. The shared executor limits concurrent builds, not the total number of retained readers, and early termination still pays the initial preparation cost for later splits.

Agreed that lazy split initialization belongs in a follow-up. It needs to preserve split order and ownership of the creation state, define validation and deferred-error behavior, and test that early close leaves later splits unopened. A TODO near the multi-split creation loop would be a useful pointer to that remaining work. This PR addresses the per-split bound; the cross-split limitation remains.

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 360fa70 into apache:main Nov 17, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Lazily initialize readers for sequential multi-file splits

3 participants