Skip to content

feat(realtime): support precise scalar filtering with vector and full-text search - #546

Merged
SteNicholas merged 6 commits into
apache:mainfrom
lxy-9602:rt-precise-scalar
Nov 18, 2026
Merged

SteNicholas merged 6 commits into
apache:mainfrom
lxy-9602:rt-precise-scalar

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158

Enable precise index evaluation for real-time append and deduplicate reads, combining scalar predicates with vector and full-text search.

Scalar predicates and pinned deletion visibility are applied before TopK, so filtered or deleted rows do not consume result slots. Queries read each segment within the intersection of its pinned bounds and the requested offset range, excluding committed prefixes and later writes.

  • Reuse predicate evaluation, selectivity ordering, search filtering, and score completion between disk and real-time readers.
  • Introduce shared index reader capabilities and optional segment-level real-time index sources.
  • Support offset selection and deletion filtering when materializing real-time results.
  • Validate the pinned store schema before reading.

Real-time primary-key merge remains unsupported in precise mode. Lumina real-time search integration is deferred to a separate PR; search behavior is covered with mocks.

Tests

Added and updated unit tests for predicate evaluation, search capabilities, score alignment, offset bounds, deletion visibility, and pinned-source lifetime.

Added six integration tests covering:

  1. Exact scalar filtering before vector TopK, including unsupported and missing scalar indexes.
  2. FullTextSearch and mixed FullTextMatch/scalar/vector predicates with separate score columns.
  3. Reads across multiple disk files, sealed segments, and building segments.
  4. In-memory deduplication updates and deletes before TopK.
  5. Committed disk deletion vectors combined with real-time deletions across prepare, commit, and refresh.
  6. Pinned plans and readers retaining their offset bounds and deletion visibility across writes and refresh.

API and Format

Public C++ API changes include:

  • IndexReaderCapabilities, shared by file and real-time index readers.
  • Real-time full-text/vector search methods and offset-based search result types.
  • Optional RealtimeIndexedReadView and RealtimeIndexReadSource interfaces.
  • Offset selection, offset deletions, and scalar-index pruning control in RealtimeQueryContext.

The reader inheritance and query-context layout change. No storage format or serialized protocol changes.

Documentation

Updated API comments for supported table modes, row-ID domains, pinned visibility, and search filtering requirements.

Generative AI tooling

Generated-by: OpenAI Codex

@lxy-9602 lxy-9602 changed the title feat(realtime): support precise scalar filtering with vector and full… feat(realtime): support precise scalar filtering with vector and full-text search Nov 10, 2026

@lszskye lszskye left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit dcd25ab into apache:main Nov 18, 2026
31 of 32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants