Repository navigation
Conversation
lucasfang
left a comment
There was a problem hiding this comment.
Self-review by OpenAI Codex of commit dd993eb. This is an author-side review, not an independent maintainer approval.
No blocking correctness findings found in the final implementation.
Reviewed cache ownership and once-per-generation registration; weak late-range callbacks; thread join/discard ordering on schema reset, Close and destruction; deferred and terminal read errors; adaptive/format fallback; no-executor synchronous reads; Parquet buffer handoff; bitmap/filter and file-row-ID forwarding; preservation of other format plugins' plain read path; and cache/I/O metrics lifetime.
Corrected stale callback-lifetime and default-read comments during this review. These final edits only change comments; the implementation was validated by all four complete common/core/Parquet/ORC executables (4,846 passed, 2 conditional skips), and the final source passed read-only formatting/lint and git diff --check.
Remaining validation limits: no local sanitizer, repository-wide unittest or integration-executable runs. Default synchronous reads now incur the explicitly requested cache/RAW memory and I/O behavior; RAW on synchronous filesystems retains its documented latency limitation.
There was a problem hiding this comment.
@lucasfang, the focused Parquet and ORC cases passed. I found failures under allocation pressure and with caching disabled, along with memory growth on synchronous scans. PTAL.
BREAKING CHANGE: WarmupLevel::DECODED is removed. Use RAW for byte warmup or NONE to disable it.
lucasfang
left a comment
There was a problem hiding this comment.
Follow-up self-review by OpenAI Codex of commit 4ddc508. This is an author-side review, not an independent maintainer approval.
The final scope is NONE/RAW warmup only. DECODED and the synchronous wrapper's decode thread, pending Arrow batch and join/discard machinery have been removed. Prefetch's background decode loop is still started by actual batch reads, not Warmup().
No blocking correctness findings found. Checked:
- Raw byte warmup remains independent of prefetch, including format/adaptive fallback.
- Warmup never calls the batch decoder; synchronous reads stay on the caller thread.
- Cache generation/reset, weak late-range callbacks, deferred/terminal errors, row IDs, and early close/destruction remain covered.
- All callers and docs were migrated away from WarmupLevel::DECODED; the public enum removal is explicitly documented as a source API change.
- The format guard is now named FormatSupportsReadAhead to describe its byte-cache role.
Validation: all four complete common/core/Parquet/ORC executables passed (4,842 passed, 2 conditional cache-disabled skips). Focused checks passed (139 common tests plus 2 skips, 141 context/system/format-table tests). Build, read-only formatting/lint and git diff --check passed.
Repository-wide unittest, integration executables and sanitizer builds were not run locally. RAW retains its documented synchronous-filesystem limitation; NONE disables early fetching.
Detect synchronous byte-range support from reader capabilities and build once. Share the internal cache context with LM and ORC instead of range callbacks. Bound synchronous buffer residency, validate ranges before splitting, and discover ORC stripe ranges on demand while preserving Parquet buffer handoff. Cover lifecycle, adaptive fallback, payload warmup and high file positions. BREAKING CHANGE: remove PrefetchFileBatchReader::SetPreBufferRangeCallback and PreBufferRangeCallback. Plugins must remove overrides and rebuild.
There was a problem hiding this comment.
@lucasfang, I have reproduced the three issues below against b755d585. PTAL.
| uint64_t next_row = reader_->GetNextRowToRead(); | ||
| if (read_ahead_ && (!pre_buffered_row_range_ || next_row < pre_buffered_row_range_->first || | ||
| next_row >= pre_buffered_row_range_->second)) { | ||
| PAIMON_RETURN_NOT_OK(read_ahead_->PrepareRanges(this)); |
There was a problem hiding this comment.
Preparing the next stripe only in NextBatch() is too late when reading involves a seek. The prefetch loop calls EnsureReaderPosition() first, and ORC's seekToRow() can initialize the target stripe, seek within a row group, and read payload before this code runs. In a real three-stripe ORC probe, after warming stripe 0, seeking into stripe 2 synchronously read all 7 bytes of that stripe's payload; the following NextBatch() then fetched the same 7 bytes again through the cache. Please prepare the target stripe from the requested row before the underlying seek performs I/O. A regression should route the format reads through CacheInputStream and assert physical reads/cache hits around the seek. DiscoverStripeRangesOnDemand currently passes a separate raw CountingStream directly to ORC, so its increasing cache-I/O counter does not demonstrate that decoding consumes the fetched bytes.
There was a problem hiding this comment.
Thanks for the detailed reproduction. You're right about both the seek ordering and the test's inability to prove that decoding consumes the cached bytes.
In 5a70ca5, I removed the ORC per-stripe warmup optimization and its dedicated tests from this PR to keep the scope focused on decoupling RAW warmup and read-ahead caching from batch prefetch. ORC now retains its existing all-stripe PreBufferRange() implementation, with no new range-preparation hooks in NextBatch() or SeekToRow(). This removes the late per-stripe preparation path identified here. The existing cost of discovering all stripe ranges is documented; any future per-stripe optimization should cover seek-triggered I/O and verify physical reads/cache hits as you suggested.
The real-ORC decoupling tests remain, covering synchronous reads, parallel prefetch, adaptive fallback, and late materialization. The five affected test targets passed locally (5,228 passed, 2 conditional skips).
Forward synchronous batch reads through the matching underlying API so sparse bitmap compaction keeps the format reader's file-row mapping and bounds checks. Avoid opening a cache stream when LM wraps a reader without byte-range support. Prepare the target ORC stripe before seekToRow can read payload bytes. Add sparse-bitmap, LM capability and physical seek I/O regression coverage, and route on-demand ORC stripe tests through CacheInputStream. Validation: 5,430 tests passed with 2 conditional skips across seven complete test executables; changed-file pre-commit checks and git diff --check passed.
There was a problem hiding this comment.
@lucasfang, thanks for updates. I left only comment for updates. PTAL.
There was a problem hiding this comment.
@lucasfang, thanks for update. I left only one comment for the remaining issue reproduced below: predicate-driven ORC stripe skipping bypasses range preparation and causes duplicate payload I/O. PTAL.
There was a problem hiding this comment.
@lucasfang, I reproduced the five issues below at 5a70ca5 using the local Parquet fixture, a range-capable test plugin, and a cache with a pinned buffer. PTAL.
| input streams. It is not a whole-query or whole-table I/O total. All counters accumulate for the | ||
| ``PrefetchIoMetrics`` describes only I/O that passes through the instrumented | ||
| file input streams, including synchronous reads with batch prefetch disabled. | ||
| It is not a whole-query or whole-table I/O total. All counters accumulate for the |
There was a problem hiding this comment.
Not fully sure I understand—if we’ve already disabled prefetch, why does PrefetchIoMetrics still show I/O?
| cache context. Format readers expose their initial ranges | ||
| through ``PreBufferRange()``; the public prefetch reader has no range callback | ||
| setter. Plugins overriding the former callback setter must remove that override | ||
| and rebuild against the updated interface. |
There was a problem hiding this comment.
Do we need to document this in the code? Otherwise the next person will only see a static piece of documentation and won’t necessarily understand the rationale behind this change.
| /// WarmupLevel::NONE when the cache is off (see SetReadAheadCacheEnabled()). | ||
| /// WarmupLevel::DECODED does not depend on the cache. | ||
| /// RAW also works with EnablePrefetch(false) and never starts batch decoding. | ||
| /// @see WarmupLevel |
There was a problem hiding this comment.
I feel like the places we’re saying warmup can be supported even when prefetch isn’t enabled might be a bit too broad—doesn’t it feel a little forced?
| } | ||
| mutable uint32_t open_count = 0; | ||
| bool fail_cache_open = false; | ||
| }; |
There was a problem hiding this comment.
There seems to be too much CountingFileSystem usage in the code right now. Please refactor these into the test utilities/tools.
| embedded_bytes.size()); | ||
| EXPECT_OK_AND_ASSIGN(FileStatus file_status, | ||
| file_system->GetFileStatus(path + "bucket-0/" + file_name)); | ||
| return std::make_shared<DataFileMeta>( |
There was a problem hiding this comment.
Is this change related to warmup in any way?
| resident_ranges_.erase(current); | ||
| } | ||
| return budget_->resident_bytes.load(std::memory_order_acquire) <= limit - size; | ||
| } |
There was a problem hiding this comment.
[P2] Normalize overlapping ranges before merging
MergeRangesLocked() assumes that new_ranges is internally disjoint, but CoalesceByteRanges() can retain partially overlapping ranges when their combined span exceeds the size limit. For example, with range_size_limit=32 and hole_size_limit=0, registering {{10,10},{30,10}} followed by {{0,25},{5,30}} produces [0,10), [10,20), [20,25), [5,30), [30,40). This breaks the ordering required by the binary searches, and MakeRoomLocked() can subsequently fail its offset assertion during eviction. Please normalize the new ranges into a disjoint union before splitting and merging, and add coverage for overlapping ranges whose combined span exceeds the limit.
| }); | ||
| } | ||
|
|
||
| ReadAheadCache::ReadAheadCache(const std::shared_ptr<InputStream>& stream, |
There was a problem hiding this comment.
[P2] Make prefetch publication safe under allocation failure
Not blocking. PreBuffer() publishes entries incrementally but dispatches their I/O only after preparing the entire window. If a later buffer allocation throws std::bad_alloc, DispatchFetches() is never reached, and destroying the pending promises leaves previously published entries with broken promises. The allocation exception escapes Warmup(), while subsequent reads can throw std::future_error instead of returning a Status. The new resident_ranges_.insert() introduces another allocation failure point after publication. Please convert allocation failures to Status::OutOfMemory and either roll back undispatched entries or complete their futures with an error, keeping the resident index consistent.
|
|
||
| /// A synchronous file reader with independent byte caching and raw warmup. | ||
| class WarmupFileBatchReader : public FileBatchReader { | ||
| public: |
There was a problem hiding this comment.
Please add reader layout comments for the WarmupFileBatchReader in the SplitRead class header. Now it’s hard to tell how each reader is called.
| for (uint32_t i = 0; i < reader_count; i++) { | ||
| futures.push_back(RunOrSubmit( | ||
| executor.get(), | ||
| [&builder, &cache, &streams, i, next_stream, io_metrics, |
There was a problem hiding this comment.
[P2] Wait for all submitted tasks before unwinding Open()
CollectAll() exits immediately if a future.get() throws; it does not drain the remaining futures on that path. These tasks capture builder, cache, and streams by reference, so an allocation failure in one task can unwind Open() and destroy its locals while another task is still accessing them, causing a use-after-free. An exception while submitting a later task has the same problem. Destroying these promise-backed futures does not wait for the tasks to finish. Please ensure exception-safe cleanup waits for all accepted tasks before destroying the objects they reference.
| Result<bool> ResolvePreBufferEnabled(const std::map<std::string, std::string>& options, | ||
| const std::optional<ReadHints>& hints) { | ||
| if (hints.has_value() && hints->prefetch_enabled && hints->read_ahead_cache_enabled) { | ||
| if (hints.has_value() && hints->read_ahead_cache_enabled) { |
There was a problem hiding this comment.
Shared caching sets pre_buffer=false, but the page-filtered path still calls DispatchPreBuffer() → PreBufferRanges() unconditionally. Bitmap point lookups and late-materialization payload reads taking this path can therefore buffer the same raw bytes in both Paimon’s ReadAheadCache and Parquet’s range cache. Please honor the shared-cache setting here, or at least add an TODO documenting the duplicate buffering and the required follow-up.
Purpose
Linked issue: close #512
With batch prefetch disabled, reads previously bypassed the shared read-ahead cache and RAW warmup. This change lets readers that expose byte ranges, including Parquet and ORC, use caching and RAW warmup on the synchronous path, including when the adaptive strategy disables prefetch.
The existing defaults now take effect together: batch prefetch off, read-ahead cache on, and RAW warmup. NONE disables early fetching while retaining caching during normal reads; RAW fetches bytes through the cache and does nothing when caching is disabled. Warmup never decodes batches. When batch prefetch is enabled, the first actual read starts parallel decoding.
FileReadAheadcontext. Detect synchronous read-ahead support from reader capabilities, without a format whitelist or rebuilding the reader.CacheInputStreamso late materialization can prepare payload ranges before its first payload batch, without the public range callback.Tests
API and Format
Breaking C++ API changes:
WarmupLevel::DECODED. Callers must use RAW or NONE; warmup no longer starts decoding.PrefetchFileBatchReader::SetPreBufferRangeCallbackandPreBufferRangeCallback. Plugins overriding the setter must remove that override and rebuild against the changed virtual interface. Initial byte ranges remain available throughPreBufferRange(); late-materialization preparation uses the internal shared context.The remaining warmup enum values, configuration defaults and
ReadHintsfields are retained.ReadHints::read_ahead_cache_enableddescribes the active shared cache independently of batch prefetch. Cache and optional I/O metrics are available on synchronous reads as well. No configuration options, storage-format changes or protocol changes are introduced.Documentation
Update the prefetch/warmup guide, I/O metrics guide and API comments to explain independent cache and warmup behavior, synchronous buffer limits and migration from the removed APIs. Document that RAW overlaps I/O only when the filesystem provides genuinely asynchronous reads, and that early termination can leave warmed bytes unused and must wait for in-flight fetches.
Generative AI tooling
Generated-by: OpenAI Codex (codex-cli 0.166.0)