Skip to content

[core] Stream Avro data reads with bounded buffering - #1449

Merged
JingsongLi merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1444
Nov 20, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
SteNicholas:PAIMON-1444

Conversation

@SteNicholas

@SteNicholas SteNicholas commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #1444.

Avro data-file scans fetched the entire file and decoded every record before yielding the first Arrow batch. Read OCF blocks on demand so a short LIMIT avoids whole-file I/O and decoding, and scan memory depends on the active block, output batch and bounded read-ahead.

Brief change log

  • Read the header lazily, parse the writer schema once per file, and feed successive selected blocks to the same apache_avro::Reader.
  • Start with 512 KiB of read-ahead, including for explicit row ranges, to reduce range GETs for near-full scans. Fetch known payloads in requests of up to 8 MiB. After the first batch, unrestricted scans grow their read-ahead window to 8 MiB; explicit ranges retain 512 KiB to allow payload skipping.
  • Apply inclusive ranges to original physical row positions without a file-sized mask. Skip unselected block payloads while checking framing and sync markers, stop after the final selected row, and preserve projection widening and exact residual filtering.
  • Run schema parsing, decompression, datum decoding, Arrow conversion and filtering on Tokio's blocking pool, with an inline fallback for other executors. Count discarded rows toward each decode task's batch limit and yield between immediately ready reads.
  • Abort queued decode tasks and release their captured input on drop. Release exhausted storage windows and drop scan state before yielding a terminal error or the batch that exhausts the file or selection.

Tests

Local validation at 17eca70d with Rust 1.98.0 and the default Paimon features:

  • cargo +1.98.0 test --locked --offline -p paimon --lib -- --quiet: 4,990 passed, 7 ignored.
  • cargo +1.98.0 test --locked --offline -p paimon --lib avro: 83 passed.
  • cargo +1.98.0 fmt --all -- --check: passed.
  • cargo +1.98.0 clippy --locked --offline -p paimon --lib --tests -- -D warnings: passed.

The storage-request test covers full scans with roughly 64 KiB and 1 MiB blocks, plus a near-full scan whose row ranges exclude one row. It checks the 512 KiB initial request, subsequent request counts, physical row order and reader release. Prefix/drop tests cover bounded first-batch I/O and backpressure.

Other coverage includes all six supported codecs; cross-block and empty ranges; header-only files and zero-width records; missing/reordered fields, numeric widening, nullable unions, nested/logical values and predicates on unprojected columns. Corrupt payloads, truncated framing, excessive record counts and injected storage failures are checked while retaining the errored stream to verify resource release.

Cancellation tests cover pending and ready reads, sparse decoding, and queued blocking work. Non-Tokio tests use memory-backed FileIO with futures::executor::block_on. Related cases share fixtures and assertions.

Full-scan and LIMIT measurements

These are historical measurements. Both streaming revisions used a 64 KiB initial read-ahead window; the revised reader measured here is 0bb6a961. They have not been rerun at 17eca70d, which starts with 512 KiB. The 128 KiB LIMIT input figures below therefore do not describe the current window.

On a 15.61 MiB file, the larger full-scan window reduced range reads from 250 to 4 and raised median peak RSS from 19.0 to 28.6 MiB relative to the original streaming revision. Both streaming revisions served LIMIT 32 with 128 KiB of input.

Workload Reader Elapsed ms, median (min–max) First batch ms Consumed rows/s Range reads Bytes read Peak RSS MiB
Full scan Eager 143.87 (131.67–152.51) 77.12 278,023 1 16,367,638 418.8
Full scan Original streaming 140.00 (125.78–147.47) 1.09 285,705 250 16,367,638 19.0
Full scan Revised streaming 113.87 (102.51–116.52) 1.06 351,286 4 16,367,638 28.6
LIMIT 32 Eager 121.17 (114.86–128.81) 81.52 264 1 16,367,638 418.5
LIMIT 32 Original streaming 1.12 (1.09–1.22) 1.09 28,605 2 131,072 18.3
LIMIT 32 Revised streaming 1.04 (1.01–1.16) 1.00 30,859 2 131,072 18.3
Fixture and measurement method
  • Compare the production AvroFormatReader from the eager base (ddcad6bd), the original streaming revision (952581de), and the revised reader (0bb6a961). Use identical release compiler settings, requested fields, batch size, and FileRead implementation for all three. These timings are for a warm local file on arm64 macOS 26.6.2 with Rust 1.98.0.
  • Generate 40,000 records for IDs 0 through 39,999 using apache_avro::Writer, the null codec, a 65,536-byte block target, and a sync marker of sixteen 0x07 bytes. The schema contains id (long), payload (eight-digit zero-padded ID followed by 128 x characters), and 96 long fields named col_2 through col_97, each containing the ID. The file contains 250 blocks and 16,367,638 bytes.
  • Project all 98 fields with a 1,024-row batch target, no predicates, and no row ranges. Consume all batches for a full scan. For the prefix, count 32 rows and drop the stream immediately. The first batch held 1,024 rows for the eager reader and 217 for each streaming reader.
  • Open the file before timing. Count one range for each FileRead::read and perform a seek and exact read on that file handle. Measure from before stream construction through stream drop. Record the first batch when it reaches the consumer and use wait4 for process peak RSS. Warm up each reader and workload once, then rotate reader order across five fresh processes per combination. The table gives medians, with min–max for elapsed time.

API and Format

The FormatFileReader interface and Avro OCF encoding remain unchanged. I/O and decoding errors now surface while polling the stream. Batches end at block boundaries and may shrink after selection or filtering.

Memory includes the header/schema, compressed and decompressed block, output/conversion buffers, row ranges and bounded read-ahead. A single large block, header or value can dominate memory. An unrestricted scan may read ahead up to 8 MiB after its first batch, including when an upstream LIMIT needs another block. Dropping the stream cancels queued work; a running task may finish its current decompression and physical batch. Skipped payloads and undecoded suffix records are not validated.

Documentation

benchmarks/avro/README.md documents block traversal, row selection, the 512 KiB initial window, memory bounds, backpressure and cancellation, and links to the historical measurements above. The fixture recipe and measurement method are retained in this description.

🤖 Assisted with Codex.

@SteNicholas
SteNicholas force-pushed the PAIMON-1444 branch 2 times, most recently from 0a7d4c0 to 952581d Compare November 18, 2026 07:19

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 56 existing Avro tests passed locally. The inline comments cover full-scan costs, executor compatibility, and cleanup on cancellation or error.

Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
Comment thread crates/paimon/src/arrow/format/avro.rs Resolved
@JingsongLi

Copy link
Copy Markdown
Contributor

Could we start by increasing the initial read-ahead window from 64 KiB to 512 KiB? This should reduce range GETs for near-full scans, such as when a deletion vector excludes just one row.

Avro scans fetched the entire file and retained every decoded record
before yielding the first Arrow batch. Even a short LIMIT paid the
full I/O and decoding cost, and row selection needed a file-wide mask.

- Parse the writer schema once per file and feed selected Avro blocks
  to the same reader as the consumer requests batches, addressing apache#1444.
- Start with 512 KiB of read-ahead. Fetch known payloads in requests of
  up to 8 MiB and grow the window to that size after the first batch of
  an unrestricted scan.
- Apply inclusive ranges to physical row positions. Skip unselected
  block payloads while checking sync markers, stop after the last
  selected row and keep predicate columns for exact residual filtering.
- Decode one batch of physical rows at a time on Tokio's blocking pool
  or inline on other executors. Release exhausted read buffers, abort
  queued work on drop and free scan state on completion or error.
@SteNicholas

SteNicholas commented Nov 19, 2026 •

Copy link
Copy Markdown
Member Author

@JingsongLi Addressed your read-ahead suggestion in 17eca70d. The initial window is now 512 KiB, including for explicit row ranges. Unrestricted scans still grow to 8 MiB after the first batch.

Extended the existing request-count test to cover small-block full scans and a near-full scan excluding one row, checking the initial window, subsequent request counts and returned row order. Updated the README and PR description; the benchmark figures remain explicitly attributed to the earlier 64 KiB revisions.

@JingsongLi
JingsongLi merged commit 885f1db into apache:main Nov 20, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Stream Avro data reads with bounded buffering

2 participants