Repository navigation
Conversation
| realtime.index.lumina.embedding.distance.metric = l2 | ||
| realtime.index.lumina.embedding.encoding.type = rawf32 | ||
|
|
||
| For disk indexes, repeat these settings with the ``file-index.lumina`` prefix, |
There was a problem hiding this comment.
I'm not understand with the doc, if I need use the realtime vector index, I need to config "realtime.index.lumina.columns = xxx" and "file-index.lumina.columns = xxx" together and use the same param/encoding value?
| return Status::Invalid("Lumina realtime index requires [offset, LIST<FLOAT>]"); | ||
| } | ||
| const std::map<std::string, std::string> lumina_options = | ||
| OptionsUtils::FetchOptionsWithPrefix(LuminaDefines::kOptionKeyPrefix, options); |
There was a problem hiding this comment.
why + kOptionKeyPrefix at factory first, and then - kOptionKeyPrefix at here? why not direct use the options without + & - kOptionKeyPrefix
| class LuminaRealtimeIndexReader; | ||
| class LuminaMemoryPool; | ||
|
|
||
| /// Append-only vector index with one writer and concurrent readers. IDs are realtime offsets. |
There was a problem hiding this comment.
does IDs mean SearchResult.topk.id?
| if (vectors->IsNull(row)) { | ||
| continue; | ||
| } | ||
| const Status status = LuminaToPaimonStatus(streamer_->Insert( |
There was a problem hiding this comment.
flatnsw defaults to 1,048,576 non-null vectors per column per segment. If the next vector goes into the same segment, the insert fails and invalidates the shared RealtimeContext. Writes to other buckets sharing that context also stop, and the writer can no longer seal or prepare data it already accepted. Recovery requires a new writer and context and replay from the last committed offset.
| continue; | ||
| } | ||
| if (vectors->value_length(row) != static_cast<int64_t>(index_info_.dimension)) { | ||
| return Status::Invalid("Lumina realtime vector dimension mismatch"); |
There was a problem hiding this comment.
In append deduplicate mode, WriteAndLookup() adds the old row's offset to the pending deletion vector before Lumina validates the replacement. If the replacement has the wrong dimension, the store releases its lock with the deletion still recorded. A concurrent CreatePlan() can pin that deletion before the writer invalidates the RealtimeContext. That plan can omit the key entirely because the deletion hides the old row and the replacement write failed.
Purpose
Linked issue: #158
Enable Lumina vector search over uncommitted data in append and append deduplicate modes.
bruteforceandflatnsw.Tests
API and Format
Documentation
Document real-time Lumina configuration, query setup, and limitations in
realtime.rst.Generative AI tooling
Generated-by: OpenAI Codex