Skip to content

perf(global-index): materialize bitmap results eagerly and batch unions - #552

Merged
SteNicholas merged 3 commits into
apache:mainfrom
lxy-9602:global-result
Nov 12, 2026
Merged

SteNicholas merged 3 commits into
apache:mainfrom
lxy-9602:global-result

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #551 .

Global index bitmap results retain suppliers and their captured objects until first access. Merging multiple results through repeated pairwise OR operations also repeatedly copies growing intermediate bitmaps.

This change:

  • Computes bitmap results during creation and propagates supplier errors immediately, allowing results to outlive their readers without retaining them.
  • Adds a shared UnionResults helper that uses FastUnion for bitmap results across sorted-file readers, union readers, and OR predicate evaluation. Other result types retain their virtual Or behavior.
  • Computes bitmap intersections by copying the operand with the smaller storage size and applying &=, avoiding an extra result copy.

Existing handling of unavailable index results, predicate field tracking, and scored-result semantics is preserved.

Tests

Added or updated coverage for:

  • Immediate supplier execution, error propagation, and release of captured objects.
  • Result validity after reader closure and destruction.
  • AND, OR, and offset results remaining valid after their operands are released.
  • Batch unions with empty, single, overlapping, and 64-bit row-id inputs.
  • Scored and mixed-result unions, including duplicate-row validation.

API and Format

The public BitmapGlobalIndexResult constructor now accepts a RoaringBitmap64. Supplier-based callers use BitmapGlobalIndexResult::Create(), which executes the supplier immediately and returns any error.

No storage format or protocol changes.

Documentation

Updated public API comments to describe result creation and bitmap access.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@lszskye lszskye left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit 3a33280 into apache:main Nov 12, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Improve global index bitmap result materialization and aggregation

3 participants