Repository navigation
feat(cache): cache typed objects and reuse vector index readers - #483
Merged
Merged
Conversation
lxy-9602
reviewed
Nov 9, 2026
ljyss9
force-pushed
the
codex/read-object-cache
branch
2 times, most recently
from
November 11, 2026 10:03
24ba783 to
d2b9df9
Compare
lxy-9602
reviewed
Nov 12, 2026
ljyss9
force-pushed
the
codex/read-object-cache
branch
2 times, most recently
from
November 12, 2026 09:09
f9cf573 to
9f4da89
Compare
lxy-9602
reviewed
Nov 12, 2026
Look up externally populated readers by immutable index URI, column and actual index type. A miss reuses the open format reader. Use the requested pool for Lumina searches and preserve it when replacing pre-filters. Include charged memory usage in cache value equality so replacing an entry with a different weight updates LRU accounting and eviction. BREAKING CHANGE: CacheValue is abstract. Use MemorySegmentCacheValue or ObjectCacheValue<T> and CacheValue::Get<T>. FileIndexFormat::Reader implementations must implement GetIndexTypes. Recompile downstream users. Generated-by: OpenAI Codex (GPT-6)
ljyss9
force-pushed
the
codex/read-object-cache
branch
from
November 12, 2026 10:42
9f4da89 to
0e73d77
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Reuse opened vector File Index readers across reads through the existing user-provided cache. Cache values now support typed objects as well as memory segments, and external index readers are keyed by index URI and column. Embedded indexes and reads without a cache retain their existing behavior.
Keep query filters and search options independent for each call, and give Lumina searches a per-call session memory resource. Migrate existing page, manifest, snapshot and Parquet cache users to the typed value API.
Tests
Built the existing Release configuration with tests and Lumina enabled:
Executed the complete test binaries with the repository test fixtures:
paimon-common-test: 1,797 passed, 2 skipped.paimon-core-test: 2,504 passed.paimon-parquet-format-test: 233 passed.paimon-lumina-index-test: 42 passed.Coverage includes typed payload lifetime after eviction, memory accounting, cache key separation, reader reuse, disabled/embedded caching, failed loads, oversized objects, externally populated readers, independent concurrent Lumina queries, and eviction during a query.
git diff --checkpassed. The full pre-commit runner was not available locally.API and Format
This changes the public cache API and requires downstream migration/recompilation:
CacheValuebecomes an abstract base class; useMemorySegmentCacheValuefor byte segments andObjectCacheValue<T>for objects.GetSegment()withCacheValue::Get<const MemorySegment>(value);Get<T>()retains the cache value owner and returns null for a type mismatch.CacheKind::INDEX_READERandCacheKey::ForIndexReader(uri, column_name).No storage format or protocol changes. Reader cache identity is the immutable index URI and column; construction options are not part of the key.
Documentation
Update the manifest cache documentation to use the typed segment accessor.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-6)