Skip to content

feat(realtime): add Lumina vector index support - #583

Open
lxy-9602 wants to merge 3 commits into
apache:mainfrom
lxy-9602:lumina-rt
Open

lxy-9602 wants to merge 3 commits into
apache:mainfrom
lxy-9602:lumina-rt

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: #158
Enable Lumina vector search over uncommitted data in append and append deduplicate modes.

  • Register the Lumina real-time index plugin with per-column configuration for bruteforce and flatnsw.
  • Apply scalar filters and committed or pending deletion vectors before source-local TopK.
  • Allocate search scratch memory from the query pool and track retained index memory separately.
  • Preserve the existing read-and-write commit path, with a TODO for exporting memory indexes using offset-to-row-ID remapping.

Tests

  • Add unit coverage for configuration, input validation, offset bounds, pre-filtering, memory accounting, reader lifetime, and concurrent reads and writes.
  • Add integration coverage using real Lumina indexes with indexed and unindexed scalar predicates across memory, disk, deletion vectors, and commit transitions.

API and Format

Documentation

Document real-time Lumina configuration, query setup, and limitations in realtime.rst.

Generative AI tooling

Generated-by: OpenAI Codex

@lxy-9602
lxy-9602 marked this pull request as draft November 19, 2026 07:27
@lxy-9602 lxy-9602 changed the title feat(realtime)!: add Lumina vector index support feat(realtime): add Lumina vector index support Nov 19, 2026
@lxy-9602
lxy-9602 marked this pull request as ready for review November 19, 2026 08:18
realtime.index.lumina.embedding.distance.metric = l2
realtime.index.lumina.embedding.encoding.type = rawf32

For disk indexes, repeat these settings with the ``file-index.lumina`` prefix,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not understand with the doc, if I need use the realtime vector index, I need to config "realtime.index.lumina.columns = xxx" and "file-index.lumina.columns = xxx" together and use the same param/encoding value?

return Status::Invalid("Lumina realtime index requires [offset, LIST<FLOAT>]");
}
const std::map<std::string, std::string> lumina_options =
OptionsUtils::FetchOptionsWithPrefix(LuminaDefines::kOptionKeyPrefix, options);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why + kOptionKeyPrefix at factory first, and then - kOptionKeyPrefix at here? why not direct use the options without + & - kOptionKeyPrefix

class LuminaRealtimeIndexReader;
class LuminaMemoryPool;

/// Append-only vector index with one writer and concurrent readers. IDs are realtime offsets.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does IDs mean SearchResult.topk.id?

if (vectors->IsNull(row)) {
continue;
}
const Status status = LuminaToPaimonStatus(streamer_->Insert(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

flatnsw defaults to 1,048,576 non-null vectors per column per segment. If the next vector goes into the same segment, the insert fails and invalidates the shared RealtimeContext. Writes to other buckets sharing that context also stop, and the writer can no longer seal or prepare data it already accepted. Recovery requires a new writer and context and replay from the last committed offset.

continue;
}
if (vectors->value_length(row) != static_cast<int64_t>(index_info_.dimension)) {
return Status::Invalid("Lumina realtime vector dimension mismatch");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In append deduplicate mode, WriteAndLookup() adds the old row's offset to the pending deletion vector before Lumina validates the replacement. If the replacement has the wrong dimension, the store releases its lock with the deletion still recorded. A concurrent CreatePlan() can pin that deletion before the writer invalidates the RealtimeContext. That plan can omit the key entirely because the deletion hides the old row and the replacement write failed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants