[python] Cache decoded BLOB indexes - #9547
Open
YannByron wants to merge 1 commit into
Open
Conversation
JingsongLi
reviewed
Sep 2, 2026
| self.blob_lengths = blob_lengths | ||
| self.blob_offsets = blob_offsets | ||
| # Readers own mutable lists; the cached immutable index is shared. | ||
| blob_lengths, blob_offsets = _decode_blob_index(index_bytes) |
Contributor
There was a problem hiding this comment.
Please use file_path as the cache key. Paimon BLOB files are immutable and have unique paths, so the path already identifies the index. Keying the cache by index_bytes forces every reader to read the complete index before it can check the cache, hashes and compares a potentially large byte string, and retains the compressed bytes alongside the decoded tuples. A path-keyed cache can avoid all of these costs.
Cache immutable decoded indexes by BLOB file path so repeated readers skip index I/O and offset reconstruction. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-5.6-sol AI-Contributed/Feature: 45/45 AI-Contributed/UT: 47/47
YannByron
force-pushed
the
perf/python-blob-index-cache
branch
from
September 2, 2026 14:45
922acfc to
4103c3b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Creating a
FormatBlobReadercurrently decompresses the same immutable BLOB index and rebuilds all offsets every time. Workloads that repeatedly open the same BLOB files pay this CPU cost on each reader creation.Cache decoded indexes by their exact compressed bytes with a bounded 16-entry LRU. Cached values remain immutable, while every reader receives its own mutable lists. This change is extracted from #9466 so the generic reader optimization can be reviewed independently.
A local paired ACT benchmark on an Apple M2 Pro showed Paimon batch-fetch throughput improving from 148.70 to 172.56 samples/s (+16.05%). With the same HDF5 baseline, the Paimon/HDF5 ratio improved from 87.88% to 101.98%; tensor fingerprints and three rounds of train/validation loss remained identical.
Tests
PYTHONPATH=. python -m pytest pypaimon/tests/blob_test.py -q(118 passed, 1 skipped)ruff check pypaimon/read/reader/format_blob_reader.py pypaimon/tests/blob_test.pyflake8 --config=dev/cfg.ini pypaimon/read/reader/format_blob_reader.py pypaimon/tests/blob_test.pygit diff --check origin/master..HEAD