Skip to content

feat(cache): cache typed objects and reuse vector index readers - #483

Merged
lxy-9602 merged 1 commit into
apache:mainfrom
ljyss9:codex/read-object-cache
Nov 12, 2026
Merged

lxy-9602 merged 1 commit into
apache:mainfrom
ljyss9:codex/read-object-cache

Conversation

@ljyss9

@ljyss9 ljyss9 commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Purpose

Reuse opened vector File Index readers across reads through the existing user-provided cache. Cache values now support typed objects as well as memory segments, and external index readers are keyed by index URI and column. Embedded indexes and reads without a cache retain their existing behavior.

Keep query filters and search options independent for each call, and give Lumina searches a per-call session memory resource. Migrate existing page, manifest, snapshot and Parquet cache users to the typed value API.

Tests

Built the existing Release configuration with tests and Lumina enabled:

cmake --build /tmp/paimon-bitmap-first-build --target paimon-common-test paimon-core-test paimon-parquet-format-test paimon-lumina-index-test -j 8

Executed the complete test binaries with the repository test fixtures:

  • paimon-common-test: 1,797 passed, 2 skipped.
  • paimon-core-test: 2,504 passed.
  • paimon-parquet-format-test: 233 passed.
  • paimon-lumina-index-test: 42 passed.

Coverage includes typed payload lifetime after eviction, memory accounting, cache key separation, reader reuse, disabled/embedded caching, failed loads, oversized objects, externally populated readers, independent concurrent Lumina queries, and eviction during a query.

git diff --check passed. The full pre-commit runner was not available locally.

API and Format

This changes the public cache API and requires downstream migration/recompilation:

  • CacheValue becomes an abstract base class; use MemorySegmentCacheValue for byte segments and ObjectCacheValue<T> for objects.
  • Replace GetSegment() with CacheValue::Get<const MemorySegment>(value); Get<T>() retains the cache value owner and returns null for a type mismatch.
  • Add CacheKind::INDEX_READER and CacheKey::ForIndexReader(uri, column_name).

No storage format or protocol changes. Reader cache identity is the immutable index URI and column; construction options are not part of the key.

Documentation

Update the manifest cache documentation to use the typed segment accessor.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

Comment thread include/paimon/cache/cache.h Resolved
@ljyss9
ljyss9 force-pushed the codex/read-object-cache branch 2 times, most recently from 24ba783 to d2b9df9 Compare November 11, 2026 10:03
Comment thread include/paimon/cache/cache.h Resolved
Comment thread src/paimon/common/io/cache/cache.cpp Resolved
@ljyss9
ljyss9 force-pushed the codex/read-object-cache branch 2 times, most recently from f9cf573 to 9f4da89 Compare November 12, 2026 09:09
Look up externally populated readers by immutable index URI, column and actual index type. A miss reuses the open format reader. Use the requested pool for Lumina searches and preserve it when replacing pre-filters.

Include charged memory usage in cache value equality so replacing an entry with a different weight updates LRU accounting and eviction.

BREAKING CHANGE: CacheValue is abstract. Use MemorySegmentCacheValue or ObjectCacheValue<T> and CacheValue::Get<T>. FileIndexFormat::Reader implementations must implement GetIndexTypes. Recompile downstream users.

Generated-by: OpenAI Codex (GPT-6)
@ljyss9
ljyss9 force-pushed the codex/read-object-cache branch from 9f4da89 to 0e73d77 Compare November 12, 2026 10:42

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 56283e9 into apache:main Nov 12, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants