Skip to content

feat(file-index): track result exactness and support global top-k - #573

Merged
lucasfang merged 6 commits into
apache:mainfrom
lszskye:multi_file_index_optimize
Nov 20, 2026
Merged

lucasfang merged 6 commits into
apache:mainfrom
lszskye:multi_file_index_optimize

Conversation

@lszskye

@lszskye lszskye commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #452

Scalar index evaluation previously rechecked indexed rows without distinguishing exact results from potentially incomplete results. Root search limits were also applied per file, so reading multiple files could return more than the requested TopK.

This change:

  • Adds result-level exactness to FileIndexResult and defines its propagation through And/Or.
  • Uses the first exact scalar index result. If none is available, scans the original candidate scope to recover matches that an inexact index may omit.
  • Passes eligible rows between index queries and pushes scalar predicates into fallback data reads.
  • Adds TopKBatchReader to select and order root search results across all files and supplied splits. Internal score columns are removed when not projected.
  • Normalizes dictionary-encoded columns before combining candidates, allowing files with different physical string encodings to participate in the same TopK.
  • Global TopK with fallback branches is rejected.
  • Root vector searches require an explicit distance type.

Tests

Coverage includes:

  • Result exactness and And/Or behavior.
  • First-exact index selection and candidate propagation.
  • Timestamp precision loss, recovery of missing rows, and avoidance of unnecessary scalar rechecks.
  • Global TopK across files and splits, score projection, metric ordering, ties, batching, and deletion vectors.

API and Format

  • Adds FileIndexResult::IsExact() and makes And/Or pure virtual. Custom subclasses must initialize exactness and implement both methods.
  • Extends bitmap/scored result construction with exactness.
  • Adds FileIndexReader::Evaluate(predicate, candidates).
  • Requires distance_type for root vector searches.
  • Defines root search limits and ordering across all splits supplied to TableRead::CreateReader.

Documentation

Updates the file-index guide and public API comments for result exactness.

Generative AI tooling

Generated-by: OpenAI Codex

@lszskye
lszskye force-pushed the multi_file_index_optimize branch from a02b111 to f7a4f54 Compare November 16, 2026 13:26
@lszskye
lszskye force-pushed the multi_file_index_optimize branch from f7a4f54 to 9be4b86 Compare November 19, 2026 08:17
Preserve optional root search field names and reject missing names when a root search is present. Remove nullable normalization from TopKBatchReader and disable three affected realtime integration tests with TODOs pending a reader schema fix.
lxy-9602
lxy-9602 previously approved these changes Nov 19, 2026

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lucasfang lucasfang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lucasfang
lucasfang merged commit bab9db0 into apache:main Nov 20, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants