Repository navigation
[core] Stream Avro data reads with bounded buffering - #1449
Conversation
0a7d4c0 to
952581d
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
The 56 existing Avro tests passed locally. The inline comments cover full-scan costs, executor compatibility, and cleanup on cancellation or error.
0bb6a96 to
b7600be
Compare
|
Could we start by increasing the initial read-ahead window from 64 KiB to 512 KiB? This should reduce range GETs for near-full scans, such as when a deletion vector excludes just one row. |
Avro scans fetched the entire file and retained every decoded record before yielding the first Arrow batch. Even a short LIMIT paid the full I/O and decoding cost, and row selection needed a file-wide mask. - Parse the writer schema once per file and feed selected Avro blocks to the same reader as the consumer requests batches, addressing apache#1444. - Start with 512 KiB of read-ahead. Fetch known payloads in requests of up to 8 MiB and grow the window to that size after the first batch of an unrestricted scan. - Apply inclusive ranges to physical row positions. Skip unselected block payloads while checking sync markers, stop after the last selected row and keep predicate columns for exact residual filtering. - Decode one batch of physical rows at a time on Tokio's blocking pool or inline on other executors. Release exhausted read buffers, abort queued work on drop and free scan state on completion or error.
b7600be to
17eca70
Compare
|
@JingsongLi Addressed your read-ahead suggestion in 17eca70d. The initial window is now 512 KiB, including for explicit row ranges. Unrestricted scans still grow to 8 MiB after the first batch. Extended the existing request-count test to cover small-block full scans and a near-full scan excluding one row, checking the initial window, subsequent request counts and returned row order. Updated the README and PR description; the benchmark figures remain explicitly attributed to the earlier 64 KiB revisions. |
Purpose
Closes #1444.
Avro data-file scans fetched the entire file and decoded every record before yielding the first Arrow batch. Read OCF blocks on demand so a short LIMIT avoids whole-file I/O and decoding, and scan memory depends on the active block, output batch and bounded read-ahead.
Brief change log
apache_avro::Reader.Tests
Local validation at
17eca70dwith Rust 1.98.0 and the default Paimon features:cargo +1.98.0 test --locked --offline -p paimon --lib -- --quiet: 4,990 passed, 7 ignored.cargo +1.98.0 test --locked --offline -p paimon --lib avro: 83 passed.cargo +1.98.0 fmt --all -- --check: passed.cargo +1.98.0 clippy --locked --offline -p paimon --lib --tests -- -D warnings: passed.The storage-request test covers full scans with roughly 64 KiB and 1 MiB blocks, plus a near-full scan whose row ranges exclude one row. It checks the 512 KiB initial request, subsequent request counts, physical row order and reader release. Prefix/drop tests cover bounded first-batch I/O and backpressure.
Other coverage includes all six supported codecs; cross-block and empty ranges; header-only files and zero-width records; missing/reordered fields, numeric widening, nullable unions, nested/logical values and predicates on unprojected columns. Corrupt payloads, truncated framing, excessive record counts and injected storage failures are checked while retaining the errored stream to verify resource release.
Cancellation tests cover pending and ready reads, sparse decoding, and queued blocking work. Non-Tokio tests use memory-backed FileIO with
futures::executor::block_on. Related cases share fixtures and assertions.Full-scan and LIMIT measurements
These are historical measurements. Both streaming revisions used a 64 KiB initial read-ahead window; the revised reader measured here is
0bb6a961. They have not been rerun at17eca70d, which starts with 512 KiB. The 128 KiB LIMIT input figures below therefore do not describe the current window.On a 15.61 MiB file, the larger full-scan window reduced range reads from 250 to 4 and raised median peak RSS from 19.0 to 28.6 MiB relative to the original streaming revision. Both streaming revisions served
LIMIT 32with 128 KiB of input.Fixture and measurement method
AvroFormatReaderfrom the eager base (ddcad6bd), the original streaming revision (952581de), and the revised reader (0bb6a961). Use identical release compiler settings, requested fields, batch size, andFileReadimplementation for all three. These timings are for a warm local file on arm64 macOS 26.6.2 with Rust 1.98.0.apache_avro::Writer, the null codec, a 65,536-byte block target, and a sync marker of sixteen0x07bytes. The schema containsid(long),payload(eight-digit zero-padded ID followed by 128xcharacters), and 96 long fields namedcol_2throughcol_97, each containing the ID. The file contains 250 blocks and 16,367,638 bytes.FileRead::readand perform a seek and exact read on that file handle. Measure from before stream construction through stream drop. Record the first batch when it reaches the consumer and usewait4for process peak RSS. Warm up each reader and workload once, then rotate reader order across five fresh processes per combination. The table gives medians, with min–max for elapsed time.API and Format
The
FormatFileReaderinterface and Avro OCF encoding remain unchanged. I/O and decoding errors now surface while polling the stream. Batches end at block boundaries and may shrink after selection or filtering.Memory includes the header/schema, compressed and decompressed block, output/conversion buffers, row ranges and bounded read-ahead. A single large block, header or value can dominate memory. An unrestricted scan may read ahead up to 8 MiB after its first batch, including when an upstream LIMIT needs another block. Dropping the stream cancels queued work; a running task may finish its current decompression and physical batch. Skipped payloads and undecoded suffix records are not validated.
Documentation
benchmarks/avro/README.mddocuments block traversal, row selection, the 512 KiB initial window, memory bounds, backpressure and cancellation, and links to the historical measurements above. The fixture recipe and measurement method are retained in this description.🤖 Assisted with Codex.