Repository navigation
feat(reader): lazily initialize sequential file readers - #564
Conversation
598d55e to
ca24f9e
Compare
Sequential append/raw reads constructed every file reader before returning, retaining open streams and format metadata for the entire split. Splits with many small files delayed the first batch and paid setup costs for files that early-stopping readers never consumed. - Prepare an ordered window of at most four reader stacks, including in-flight builds, for sequential append/raw reads, addressing apache#561. Preserve concurrent construction and next-file warmup. - Release exhausted readers, retain their metrics and refill the window as files are read. Preserve file order, filtering, deletion vectors and row tracking; keep merge-on-read input construction eager. - Own deferred creation state beyond the caller's lifetime. Validate plans eagerly and report later file-open/build failures as terminal read errors when the affected file is reached. Cancel queued builds and drain running work on close, destruction or failure. - Cover bounded construction, deferred failures and creator lifetimes. Compare eager and lazy full scans and early stops with small-file benchmarks for first-batch latency, setup I/O and peak resources.
ca24f9e to
31535f8
Compare
|
Non-blocking: the four-reader window is scoped to each split. Could we consider lazy initialization at the split level as a follow-up? If that requires substantial changes, leaving a TODO near the multi-split creation loop would be sufficient for this PR. |
|
Thanks @lxy-9602. Your observation is correct: the four-reader bound is per raw split. Agreed that lazy split initialization belongs in a follow-up. It needs to preserve split order and ownership of the creation state, define validation and deferred-error behavior, and test that early close leaves later splits unopened. A TODO near the multi-split creation loop would be a useful pointer to that remaining work. This PR addresses the per-split bound; the cross-split limitation remains. |
Purpose
Linked issue: close #561
Sequential append/raw reads constructed every file reader before returning, retaining open
streams and format metadata for the entire split. Splits with many small files delayed the
first batch and paid setup costs for files that early-stopping readers never consumed.
The change prepares an ordered, bounded window of readers while retaining concurrent
construction and next-file warmup:
ConcatBatchReader: adds lazy construction with a window of at most four reader stacks,including in-flight builds. It releases exhausted readers, retains their metrics and refills
the window as consumption advances. Ready lookahead readers are warmed without blocking
reads from the current file; skipped and empty files preserve the split's file order.
RawFileSplitRead: uses the lazy path for sequential append/raw reads and retains anindependent creator with its context, copied partition, file metadata, schema inputs,
filesystem and memory pools. Per-file predicates, deletion vectors, row kinds and file-row
IDs retain their existing behavior. Merge-on-read keeps its eager input-construction helpers.
from
CreateReader. Later file-open/build failures become terminal read errors when theaffected file is reached. Close, destruction and failure cancel queued builds, drain running
work and release prepared readers without scheduling the unopened tail. Returned batches
remain valid after their readers are released.
files, for full scans and first-batch termination. Counters cover first-batch latency,
file opens, initial footer/setup reads, peak retained streams, tracked pool allocation,
bytes read and throughput, with an optional per-open delay to simulate remote setup latency.
The construction bound applies to reader stacks, including work in flight. Per-file metadata
and owned creation inputs still scale with the split's file count; batches retained by callers
remain a separate memory cost. The construction window is independent of batch prefetch and
warmup settings.
Tests
UT:
src/paimon/common/reader/concat_batch_reader_test.cpp:TestSimpleandTestSimpleWithBitmap: reuse the existing data and bitmap cases for eagerand lazy readers, including empty inputs and empty files across several batch sizes.
TestWarmupLooksOneReaderAhead: checks eager and lazy warmup ordering, including refilledwindows, without warming beyond the next reader.
LazyWindowPreservesOrderBitmapAndMetrics: checks bounded live readers, skipped and emptyfiles, ordered output, both read APIs, accumulated metrics and creator release at EOF.
LazyEarlyCloseAndDestructionReleaseCreators: verifies bounded first-batch construction,early/repeated close, destruction, creator cleanup and returned-batch validity.
LazyInitialFailureDrainsPartialWindowandLazyDeferredFailuresAreTerminal: cover partialinitialization cleanup and terminal build, read and bitmap-conversion failures.
LazyRefillOverlapsReadingAndCloseCancelsQueuedBuilds: gates a refill to verify that readingcontinues while construction is in flight; close and destruction drain running work and
prevent queued factories from opening additional files.
src/paimon/core/operation/raw_file_split_read_test.cpp:LazyInitialWindowBuildsConcurrentlyAndEarlyCloseStopsOpening: uses a gated countingfilesystem to check concurrent initial preparation, bounded first-batch opens and cleanup
without opening the tail after early close.
LazyReaderOutlivesCreatorWithPerFilePredicatesDvAndRowIds: reads across multiple windowsafter releasing caller-owned state, checking per-file predicate ownership, deletion vectors,
row kinds, row IDs, retained output and bounded resources.
LazyValidationAndFileOpenFailures: consolidates invalid-plan checks, including the nullpredicate-provider case, and verifies that validation opens no data files. It also checks
initial and deferred missing-file errors, repeated failure propagation and stream cleanup.
IT:
No standalone integration tests are added. The raw-reader tests exercise real ORC files through
RawFileSplitRead, including filesystem opens, filtering, deletion vectors and row tracking.API and Format
No public API under
include/, stored file format or table protocol changes. File-open/buildfailures beyond the initial preparation window are now reported through
NextBatchorNextBatchWithBitmapwhen the affected file is reached, instead of duringCreateReader.Documentation
docs/source/user_guide/prefetch.rstdescribes bounded sequential preparation, deferred-errorsemantics, ownership and cleanup, and its relationship to next-file warmup.
docs/source/examples/benchmark.rstdocuments the eager/lazy comparison, full-scan and early-stopcases, counters and measurement boundaries. It distinguishes tracked pool allocation from
process RSS and simulated open latency from a complete remote-storage model.
Generative AI tooling
Generated-by: Codex (GPT-6)