Repository navigation
[vector] Support MaxSim search on range-sharded multivector columns - #11290
Open
JingsongLi wants to merge 5 commits into
Open
JingsongLi wants to merge 5 commits into
JingsongLi wants to merge 5 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A Data Evolution row can contain multiple embeddings, for example one per document chunk. Spark SQL accepts a matrix in
vector_search, Flink SQL accepts a JSON matrix in thequery_vectorargument ofsys.vector_search, and Java acceptswithMultiVector(float[][]). A query matrix ranks each row once by summing the best cosine match for each query vector. A single query vector ranks the row by its best child match.The five native range-sharded indexes write each vector with the row ID of its source list. Populated shards use the native index format directly. As with single-vector range indexes, the writer skips ranges with no embeddings. Search keeps the best hit per row for each query vector and requests additional candidates when repeated labels fill the native top K. A matrix query unions candidate rows and reranks their complete embedding lists from the query snapshot. Candidate retrieval is approximate. List indexes default to cosine similarity. The SQL build procedures reject an explicitly configured non-cosine metric before looking for shards, including on an empty table. Each list column can have its own index and dimension.
Tests
Added Spark SQL and Flink SQL end-to-end tests for
ivf-flat,ivf-pq,ivf-sq,ivf-rq, anddiskann. Each checks the ranked row IDs and exact MaxSim scores for a document that wins through two separate child matches. Flink SQL also checks that a detail-mode scan and an indexed search return the same rows and scores. Additional tests cover typed dimensions, independent queries in a batch, distinct parent hits, deletes, filtering, incomplete writes, reading repeated 64-bit row IDs directly from a native file, and all-empty ranges. Spark SQL verifies an all-empty range produces no index entry and returns no rows in both fast and detail search modes.The latest run passed 93 vector module tests, 29 Flink 1.20 SQL tests, and 10 Spark 3 tests on JDK 8.