Skip to content

[vector] Support MaxSim search on range-sharded multivector columns - #11290

Open
JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/range-multivector-maxsim
Open

JingsongLi wants to merge 5 commits into
apache:masterfrom
JingsongLi:codex/range-multivector-maxsim

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

A Data Evolution row can contain multiple embeddings, for example one per document chunk. Spark SQL accepts a matrix in vector_search, Flink SQL accepts a JSON matrix in the query_vector argument of sys.vector_search, and Java accepts withMultiVector(float[][]). A query matrix ranks each row once by summing the best cosine match for each query vector. A single query vector ranks the row by its best child match.

The five native range-sharded indexes write each vector with the row ID of its source list. Populated shards use the native index format directly. As with single-vector range indexes, the writer skips ranges with no embeddings. Search keeps the best hit per row for each query vector and requests additional candidates when repeated labels fill the native top K. A matrix query unions candidate rows and reranks their complete embedding lists from the query snapshot. Candidate retrieval is approximate. List indexes default to cosine similarity. The SQL build procedures reject an explicitly configured non-cosine metric before looking for shards, including on an empty table. Each list column can have its own index and dimension.

Tests

Added Spark SQL and Flink SQL end-to-end tests for ivf-flat, ivf-pq, ivf-sq, ivf-rq, and diskann. Each checks the ranked row IDs and exact MaxSim scores for a document that wins through two separate child matches. Flink SQL also checks that a detail-mode scan and an indexed search return the same rows and scores. Additional tests cover typed dimensions, independent queries in a batch, distinct parent hits, deletes, filtering, incomplete writes, reading repeated 64-bit row IDs directly from a native file, and all-empty ranges. Spark SQL verifies an all-empty range produces no index entry and returns no rows in both fast and detail search modes.

The latest run passed 93 vector module tests, 29 Flink 1.20 SQL tests, and 10 Spark 3 tests on JDK 8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant