Repository navigation
Description
Search before asking
- I searched the existing issues and open pull requests and found no issue tracking a bounded reader-construction window for sequential multi-file splits. Related work: [Feature] Open data file streams and build readers concurrently on the read path #333 parallelizes file opening/building; [Feature] Warm up next data file in ConcatBatchReader #419 warms up the next already-built reader; [Feature] Decouple raw warmup and read-ahead caching from batch prefetch #512 / feat(read): decouple raw warmup and read-ahead caching from prefetch #513 decouple raw warmup from batch prefetch.
Motivation
AbstractSplitRead::CreateRawFileReadersWithMetaImpl schedules construction for every data file and waits for all results before returning. RawFileSplitRead then passes the fully built list to ConcatBatchReader for sequential consumption.
The bounded build executor from #333 limits concurrent construction, but the number of completed reader stacks retained before the first batch still grows with the entire split. Each stack can hold streams, parsed file metadata, and format/prefetch state. Large small-file splits pay setup for files that may never be consumed if the caller stops early.
ConcatBatchReader already releases readers at EOF and warms one file ahead. Those lifecycle improvements can be retained while making construction itself incremental.
Solution
Introduce lazy reader construction for paths that concatenate independent files:
- Keep ordered per-file descriptors or owned creator state and construct only the current file plus a bounded lookahead window.
- Retain bounded concurrent setup within that window and the existing next-file warmup, so a full scan still overlaps remote setup and reading.
- Release completed reader stacks and refill the window as consumption advances.
- Start with append/raw sequential paths. Keep the eager helpers available to callers that genuinely require all active inputs, including merge-on-read.
- Ensure deferred creator state owns or safely retains its context, schema/mapping inputs, filesystem, and memory pool.
- Preserve plan-level validation. Document that file-open/build failures outside the prepared window are reported when that window is reached, and propagate asynchronous preparation failures through the read API.
No storage-format change is required. This should build on #333 and #419 rather than replace their concurrency and warmup improvements.
Anything else?
Acceptance and validation:
- Counting/gated filesystem tests show that construction and the first batch open only a bounded window of an N-file split.
- Early close does not schedule unopened tail files; already submitted work is drained and released safely.
- Outstanding reader resources stay bounded by the preparation window rather than total file count.
- Preserve file order, predicate/bitmap/DV behavior, row kinds, file-row IDs, accumulated metrics, empty-file handling, and error propagation.
- Verify safe lifetimes when the returned reader outlives its creator, plus cancellation and partial initialization failures.
- Benchmark large small-file splits with full consumption and early termination. Compare first-batch latency, file-open/footer-read counts, peak memory/resources, and full-scan throughput.
This issue concerns construction lifetime, independently of raw warmup scheduling in #512 / #513. It does not propose changing merge-on-read input requirements.
Are you willing to submit a PR?
- I'm willing to submit a PR!
Activity
feat(reader): lazily initialize sequential file readers
feat(reader): lazily initialize sequential file readers
feat(reader): lazily initialize sequential file readers
feat(reader): lazily initialize sequential file readers
feat(reader): open splits on demand and bound file preparation