Skip to content

[Feature] Lazily initialize readers for sequential multi-file splits #561

Closed

Description

@SteNicholas

Search before asking

Motivation

AbstractSplitRead::CreateRawFileReadersWithMetaImpl schedules construction for every data file and waits for all results before returning. RawFileSplitRead then passes the fully built list to ConcatBatchReader for sequential consumption.

The bounded build executor from #333 limits concurrent construction, but the number of completed reader stacks retained before the first batch still grows with the entire split. Each stack can hold streams, parsed file metadata, and format/prefetch state. Large small-file splits pay setup for files that may never be consumed if the caller stops early.

ConcatBatchReader already releases readers at EOF and warms one file ahead. Those lifecycle improvements can be retained while making construction itself incremental.

Solution

Introduce lazy reader construction for paths that concatenate independent files:

  • Keep ordered per-file descriptors or owned creator state and construct only the current file plus a bounded lookahead window.
  • Retain bounded concurrent setup within that window and the existing next-file warmup, so a full scan still overlaps remote setup and reading.
  • Release completed reader stacks and refill the window as consumption advances.
  • Start with append/raw sequential paths. Keep the eager helpers available to callers that genuinely require all active inputs, including merge-on-read.
  • Ensure deferred creator state owns or safely retains its context, schema/mapping inputs, filesystem, and memory pool.
  • Preserve plan-level validation. Document that file-open/build failures outside the prepared window are reported when that window is reached, and propagate asynchronous preparation failures through the read API.

No storage-format change is required. This should build on #333 and #419 rather than replace their concurrency and warmup improvements.

Anything else?

Acceptance and validation:

  • Counting/gated filesystem tests show that construction and the first batch open only a bounded window of an N-file split.
  • Early close does not schedule unopened tail files; already submitted work is drained and released safely.
  • Outstanding reader resources stay bounded by the preparation window rather than total file count.
  • Preserve file order, predicate/bitmap/DV behavior, row kinds, file-row IDs, accumulated metrics, empty-file handling, and error propagation.
  • Verify safe lifetimes when the returned reader outlives its creator, plus cancellation and partial initialization failures.
  • Benchmark large small-file splits with full consumption and early termination. Compare first-batch latency, file-open/footer-read counts, peak memory/resources, and full-scan throughput.

This issue concerns construction lifetime, independently of raw warmup scheduling in #512 / #513. It does not propose changing merge-on-read input requirements.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions