Skip to content

fix(blob): support legacy Python CRCs and configurable verification - #1467

Draft
XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix-legacy-python-blob-crc
Draft

XiaoHongbo-Hope wants to merge 1 commit into
apache:mainfrom
XiaoHongbo-Hope:fix-legacy-python-blob-crc

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Closes #1466. Intact BLOB files written by Python before apache/paimon#9426 fail Rust reads because their CRC excludes the trailing entry length.

Brief change log

  • Accept standard and historical Python CRCs using one payload scan, without additional I/O.
  • Add blob.read.verify-crc (default true) to scalar/array/map BLOB reads, including Data Evolution fallback reads. false skips CRC calculation only; magic, length and range checks remain enabled.
  • Keep writers and descriptor-only reads unchanged.

Tests

Historical Python-generated scalar/array/map fixtures, mixed standard/legacy entries, empty/null entries, corruption rejection, option parsing, and disabled verification through format-reader and Data Evolution paths.

cargo test -p paimon --lib blob --locked: 275 passed.
cargo fmt --all -- --check and git diff --check: passed.
cargo clippy -p paimon --lib --locked -- -D warnings: passed.

API and Format

New optional read setting; no writer or on-disk format changes. Disabling CRC means payload corruption detectable only by the checksum is no longer rejected. This setting does not control external-object or Python-side descriptor fetching.

Documentation

Added the option and its scope to the Python binding guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BLOB reads reject historical Python CRCs

1 participant