Skip to content

perf(global-index): reduce bitmap result memory retention - #544

Merged
SteNicholas merged 1 commit into
apache:mainfrom
lxy-9602:perf-bitmap
Nov 10, 2026
Merged

SteNicholas merged 1 commit into
apache:mainfrom
lxy-9602:perf-bitmap

Conversation

@lxy-9602

Copy link
Copy Markdown
Member

Purpose

Repeated lazy bitmap OR operations retain intermediate unions through supplier chains, increasing memory usage and creating deep recursion when querying many index files.

  • Materialize bitmap-to-bitmap OR eagerly and return a cached union without retaining its operands.
  • Return the existing result for zero offsets, avoiding bitmap reconstruction.
  • Optimize run containers after rebuilding a bitmap for a nonzero offset.

Tests

  • Verify that bitmap OR propagates supplier errors immediately.
  • Verify that zero-offset operations reuse the existing result without invoking its supplier.
  • Cover run optimization across 32-bit boundaries, preserving row IDs, reducing encoded size, repeated optimization, and empty bitmaps.

API and Format

  • Add the public RoaringBitmap64::RunOptimize() method.
  • Bitmap-to-bitmap OR now evaluates suppliers and reports errors during Or().

Documentation

Update API comments to describe eager bitmap OR, cached results, and the new run optimization method.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@lszskye lszskye left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@SteNicholas SteNicholas left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@SteNicholas
SteNicholas merged commit d94394b into apache:main Nov 10, 2026
18 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants