diff --git a/docs/docs/flink/procedures/indexes.md b/docs/docs/flink/procedures/indexes.md index 5b9715d2481f..acf92e4d3b74 100644 --- a/docs/docs/flink/procedures/indexes.md +++ b/docs/docs/flink/procedures/indexes.md @@ -200,7 +200,7 @@ To perform vector similarity search on a table with a global vector index. Retur - `vector_column` (required): the name of the vector column to search. -- `query_vector` (required): comma-separated float values representing the query vector, e.g. '1.0,2.0,3.0'. +- `query_vector` (required) accepts comma-separated coordinates for one vector, such as `'1.0,2.0,3.0'`. For MaxSim search on a list of vectors in a Data Evolution table, pass a JSON matrix such as `'[[1,0],[0,1]]'`. - `top_k` (required): the number of nearest neighbors to return. diff --git a/docs/docs/multimodal-table/global-index/vector.mdx b/docs/docs/multimodal-table/global-index/vector.mdx index 2bf2960f2690..6b455b4b294d 100644 --- a/docs/docs/multimodal-table/global-index/vector.mdx +++ b/docs/docs/multimodal-table/global-index/vector.mdx @@ -311,15 +311,15 @@ result = ( ``` Exact scalar global-index answers pre-filter the vector search. If scalar indexes cannot -evaluate the predicate, each shard runs ANN once and checks only its candidates against the +evaluate the predicate, each shard retrieves ANN candidates and checks them against the filter columns of the same pinned snapshot. The coordinator merges the filtered results; it does not perform a full-table scalar scan or a raw scan to fill missing matches. `vector-index.post-filter.candidates-factor` is a positive integer, defaulting to `10`. The candidate count is `max(original Top-K * factor, required reranking limit)`, capped by the shard size and the backend integer limit. For example, Top-K 10 normally requests 100 -candidates. Batch execution may split large inputs into memory-bounded chunks; every query -still runs ANN once. Post-filtering may return fewer than Top-K rows, including zero even +candidates. Batch execution may split large inputs into memory-bounded chunks. On single-vector +columns, each query runs ANN once per shard. Post-filtering may return fewer than Top-K rows, including zero even when matches exist outside the candidates. Increasing the factor trades more ANN/scalar work for more candidates. Post-filtering does not retry or scan raw data to fill Top-K. Raw work outside vector coverage still follows the configured index search modes. When a @@ -344,6 +344,58 @@ A search selects top-K row IDs; reading those rows through a regular table scan does not imply score order. See [Read Scored Results](./manage-indexes#read-scored-results) for SQL ordering and Java/Python score access. +### Search a List of Vectors + +On a Data Evolution table, Spark SQL, Flink SQL and Java can search a column containing multiple +embeddings per row. The native `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq`, and `diskann` +indexes support these columns with range sharding and cosine similarity. +Omitting the metric selects `cosine`. Index creation rejects any other metric. +The column can be `ARRAY>` or `ARRAY>`. For the latter, +specify the inner dimension when building the index. Each row can contain a different +number of vectors. You can build an independent index for each such column. + +```sql +-- Spark SQL +CALL sys.create_global_index( + table => 'db.documents', index_column => 'chunks', index_type => 'ivf-flat', + options => 'ivf-flat.dimension=2,ivf-flat.distance.metric=cosine' +); + +SELECT id, __paimon_search_score +FROM vector_search('db.documents', 'chunks', array(array(1.0f, 0.0f), array(0.0f, 1.0f)), 10) +ORDER BY __paimon_search_score DESC, id; +``` + +A single query vector ranks rows by their best matching embedding. A query matrix +uses MaxSim to rank rows. For every query vector, take the highest cosine similarity +among a row's embeddings, then sum those similarities. Each row can occupy one +place in the result. Null embeddings are skipped. A row with no embeddings has no score. +A range with no embeddings produces no index file. +Query matrices require at least one vector, fixed inner dimensions and finite coordinates. +Spark SQL accepts a matrix in an uncorrelated `vector_search` call. A correlated +`LATERAL vector_search` call takes one query vector per outer row. +In Flink SQL, pass the matrix as a JSON string to the search procedure. The procedure +executes matrix queries locally even when `vector-search.distribute.enabled` is enabled. + +```sql +CALL sys.vector_search( + `table` => 'db.documents', vector_column => 'chunks', + query_vector => '[[1,0],[0,1]]', top_k => 10, + projection => 'id,__paimon_search_score' +); +``` + +For a matrix query, Paimon retrieves candidate rows for each query vector and scores the +union against their complete lists from the query snapshot. Candidate retrieval is approximate. +Each indexed vector carries the row ID of its source list. When several matches share that +ID, the reader keeps their highest score and can request more distinct candidate rows +before applying the post-filter. +By default, it requests up to four times the final limit per query vector per shard. +Set `refine_factor` to select that multiplier for any of the five indexes. Uncovered rows +included by the search mode are scored with the same MaxSim formula. Java exposes matrix queries through +`newVectorSearchBuilder().withMultiVector(float[][])`. For independent top-K results +per vector, use `newBatchVectorSearchBuilder().withVectors(float[][])`. + ## Search Options Search-time options are passed with each vector search request: @@ -353,8 +405,8 @@ Search-time options are passed with each vector search request: | `ivf.nprobe` | Automatic | Explicit number of IVF clusters to probe. When omitted, paimon-vindex derives the width from the index, `top_k`, and filter selectivity. | | `centroid.nprobe` | Automatic | Number of centroid shards to search per row-ID range. Paimon chooses this width when omitted. Within each selected shard, `ivf.nprobe` controls the IVF lists to search and `diskann.l_search` controls the DiskANN graph search width. | | `ivf.max_initial_filter_expansion_factor` | Disabled | Positive integer limiting filter-driven expansion of the initial automatic IVF probe width. A factor of `1` disables initial filter expansion. Progressive retries may still probe more clusters when fewer than `top_k` filtered results are found. | -| `ivf.refine_factor` | Disabled | Retrieves `top_k * refine_factor` IVF candidates and reranks them with the original vectors stored in the Paimon table. It is most useful for compressed indexes such as `ivf-pq`, `ivf-sq`, and `ivf-rq` when recall is more important than latency. | -| `ivf_pq.batch_table_reuse` | `auto` | IVF-PQ batch search distance-table reuse mode: `auto`, `on`, or `off`. Other index types and scalar searches ignore it. | +| `refine_factor` | List columns use `4` for a matrix query and `1` for a single query vector. Disabled for single-vector columns. | Retrieves `top_k * refine_factor` candidates per query vector and reranks them with the original vectors stored in the Paimon table. For single-vector columns it is most useful with compressed indexes such as `ivf-pq`, `ivf-sq`, and `ivf-rq`. | +| `ivf_pq.batch_table_reuse` | `auto` | Distance-table reuse for native batched IVF-PQ searches on single-vector columns. Accepts `auto`, `on`, or `off`. | | `ivf_pq.batch_table_reuse.max_bytes` | 512 MiB | Positive long integer limiting the memory used by IVF-PQ batch distance-table reuse. Search falls back to direct table construction when the reusable tables exceed the budget. | | `diskann.l_search` | Automatic | paimon-vindex DiskANN graph candidate width. The automatic value uses calibration when available, otherwise `max(100, 2 * top_k)`. | @@ -368,9 +420,11 @@ reduce Recall@K compared with uncapped automatic search. Progressive expansion o fewer than `top_k` valid results are returned; if the capped initial search already fills `top_k`, probing stops. -`ivf.refine_factor` can also be configured with `refine_factor`, `rerank_factor`, and hyphenated -spellings such as `ivf.refine-factor`. Setting `ivf.refine_factor=1` still performs the raw-vector -rerank for the indexed candidates; leaving it unset skips the rerank stage. +`refine_factor` also accepts the alias `rerank_factor` and hyphenated spellings such as +`refine-factor`. The four IVF indexes also accept `ivf.refine_factor`. Setting `refine_factor=1` +still performs the raw-vector rerank for the indexed candidates. Single-vector columns skip +reranking when the option is unset. +List columns always score candidate rows against their complete embedding lists. ## Build Options @@ -380,8 +434,8 @@ Build options for native vector indexes: |---|---|---| | `.sharding` | `range` | Use `centroid` to train an outer router for each row-ID range when building a Spark Data Evolution index. | | `.centroid.nlist` | Automatic | Number of outer centroids per contiguous row-ID range with centroid sharding. | -| `.dimension` | `128` | Vector dimension for `ARRAY` columns. Ignored for `VECTOR` columns. | -| `.distance.metric` | `inner_product` | Distance metric. Supported values: `l2`, `cosine`, `inner_product`. | +| `.dimension` | `128` for `ARRAY` | Required for `ARRAY>`. Columns with a fixed-size vector type supply their own dimension. | +| `.distance.metric` | `inner_product` for single-vector columns, `cosine` for lists | Single-vector columns support `l2`, `cosine`, and `inner_product`. List columns support `cosine`. | | `.ivf.coarse-assignment` | `auto` | Build-time list assignment for all IVF types. `auto` uses Vamana when `dimension × nlist ≥ 1,000,000` and exact assignment otherwise; `exact` always uses exact assignment. | | `.ivf.train.max-points-per-centroid` | `256` | Positive coarse K-means training limit for all IVF types: at most `nlist × value` vectors. | | `.train.sample-ratio` | `1.0` | Ratio of vectors sampled for native index training. Must be greater than `0` and less than or equal to `1`. Lower values reduce training memory and build cost, but may reduce index quality. | diff --git a/paimon-api/src/main/java/org/apache/paimon/options/VectorIndexOptions.java b/paimon-api/src/main/java/org/apache/paimon/options/VectorIndexOptions.java index 80f1c4e7c258..03fea5a0180c 100644 --- a/paimon-api/src/main/java/org/apache/paimon/options/VectorIndexOptions.java +++ b/paimon-api/src/main/java/org/apache/paimon/options/VectorIndexOptions.java @@ -55,6 +55,7 @@ public Options options() { return options; } + /** Returns the normalized metric, defaulting to inner product for single-vector searches. */ public String metric() { return metric; } @@ -110,14 +111,20 @@ private static String legacyIndexType(Options options, String column) { return type == null ? null : type.toLowerCase(Locale.ROOT).trim(); } + /** Resolves single-vector options, including the default metric for a known index type. */ public static VectorIndexOptions resolve( Options options, @Nullable String type, String column) { - return resolve(options, type, column, true); + VectorIndexOptions resolved = resolveConfigured(options, type, column); + if (resolved.type != null && resolved.options.get(resolved.type + ".metric") == null) { + metricAliases(resolved.type, column) + .forEach(key -> resolved.options.setString(key, resolved.metric)); + } + return resolved; } - /** Retains an unspecified metric when readers must obtain it from persisted index metadata. */ - public static VectorIndexOptions resolve( - Options options, @Nullable String type, String column, boolean defaultMetric) { + /** Keeps metric aliases absent from {@link #options()} when no metric was configured. */ + public static VectorIndexOptions resolveConfigured( + Options options, @Nullable String type, String column) { if (type == null) { CoreOptions core = new CoreOptions(options); List declared = @@ -140,12 +147,7 @@ public static VectorIndexOptions resolve( } String prefix = type + "."; String fieldPrefix = "fields." + column + "."; - List aliases = - Arrays.asList( - fieldPrefix + "distance.metric", - fieldPrefix + "metric", - prefix + "distance.metric", - prefix + "metric"); + List aliases = metricAliases(type, column); String metric = null; for (String key : Arrays.asList( @@ -172,11 +174,20 @@ public static VectorIndexOptions resolve( break; } } - if (metric != null || defaultMetric) { - String normalized = normalizeMetric(metric); + String normalized = normalizeMetric(metric); + if (metric != null) { aliases.forEach(key -> resolved.setString(key, normalized)); } - return new VectorIndexOptions(type, resolved, normalizeMetric(metric)); + return new VectorIndexOptions(type, resolved, normalized); + } + + private static List metricAliases(String type, String column) { + String fieldPrefix = "fields." + column + "."; + return Arrays.asList( + fieldPrefix + "distance.metric", + fieldPrefix + "metric", + type + ".distance.metric", + type + ".metric"); } private static String normalizeMetric(@Nullable String metric) { diff --git a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexer.java b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexer.java index 2a0b1c66ab29..e81027ec84d2 100644 --- a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexer.java +++ b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexer.java @@ -56,17 +56,26 @@ static GlobalIndexer create(String type, List indexFields, Options op .create(indexFields, resolveOptions(type, indexFields, options)); } + /** Opens an indexer for reading existing files using the current table options. */ + static GlobalIndexer createForRead( + String type, List indexFields, Options tableOptions) { + return GlobalIndexerFactoryUtils.load(type).createForRead(indexFields, tableOptions); + } + /** Resolve the same field and procedure options for build planning and plugin creation. */ static Options resolveOptions(String type, List indexFields, Options options) { - Options resolved = options; - if (!indexFields.isEmpty()) { - // The leading field owns the index; remaining fields are indexed payload columns. - String column = indexFields.get(0).name(); - resolved = - CoreOptions.isVectorIndexType(type) - ? VectorIndexOptions.resolve(options, type, column).options() - : new CoreOptions(options).globalIndexOptions(type, column); + if (indexFields.isEmpty()) { + return options; + } + // The leading field owns the index; remaining fields are indexed payload columns. + DataField field = indexFields.get(0); + if (CoreOptions.isVectorIndexType(type)) { + VectorIndexOptions resolved = + VectorIndexUtils.isMultiVector(field.type()) + ? VectorIndexOptions.resolveConfigured(options, type, field.name()) + : VectorIndexOptions.resolve(options, type, field.name()); + return resolved.options(); } - return resolved; + return new CoreOptions(options).globalIndexOptions(type, field.name()); } } diff --git a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexerFactory.java b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexerFactory.java index 2e2d9166e689..1a68614788d4 100644 --- a/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexerFactory.java +++ b/paimon-common/src/main/java/org/apache/paimon/globalindex/GlobalIndexerFactory.java @@ -43,6 +43,12 @@ default List selectFiles( return files; } + /** Creates an indexer for existing files. Implementations can use persisted build settings. */ + default GlobalIndexer createForRead(List indexFields, Options tableOptions) { + return create( + indexFields, GlobalIndexer.resolveOptions(identifier(), indexFields, tableOptions)); + } + /** Creates an indexer over a non-empty list of columns in index order. */ GlobalIndexer create(List indexFields, Options options); } diff --git a/paimon-common/src/main/java/org/apache/paimon/globalindex/VectorIndexUtils.java b/paimon-common/src/main/java/org/apache/paimon/globalindex/VectorIndexUtils.java index 3a332cb3d660..060c58d1c4ea 100644 --- a/paimon-common/src/main/java/org/apache/paimon/globalindex/VectorIndexUtils.java +++ b/paimon-common/src/main/java/org/apache/paimon/globalindex/VectorIndexUtils.java @@ -20,11 +20,62 @@ import org.apache.paimon.data.InternalArray; import org.apache.paimon.data.InternalVector; +import org.apache.paimon.types.ArrayType; +import org.apache.paimon.types.DataType; +import org.apache.paimon.types.FloatType; +import org.apache.paimon.types.VectorType; /** Materialization at the portable vector index boundary. */ public final class VectorIndexUtils { private VectorIndexUtils() {} + public static boolean isMultiVector(DataType type) { + if (!(type instanceof ArrayType)) { + return false; + } + DataType child = ((ArrayType) type).getElementType(); + return (child instanceof VectorType + && ((VectorType) child).getElementType() instanceof FloatType) + || (child instanceof ArrayType + && ((ArrayType) child).getElementType() instanceof FloatType); + } + + public static void validateQuery(float[][] vectors) { + if (vectors == null + || vectors.length == 0 + || vectors[0] == null + || vectors[0].length == 0) { + throw new IllegalArgumentException( + "A multivector query requires at least one non-empty vector."); + } + for (float[] vector : vectors) { + if (vector == null || vector.length != vectors[0].length) { + throw new IllegalArgumentException("Multivector query dimensions must match."); + } + for (float value : vector) { + if (!Float.isFinite(value)) { + throw new IllegalArgumentException( + "Multivector query coordinates must be finite."); + } + } + } + } + + public static void validateQuery(ArrayType columnType, float[][] vectors) { + validateQuery(vectors); + DataType child = columnType.getElementType(); + if (child instanceof VectorType) { + int dimension = ((VectorType) child).getLength(); + if (vectors[0].length != dimension) { + throw new IllegalArgumentException( + "Query vector dimension mismatch: expected " + + dimension + + ", got " + + vectors[0].length); + } + } + } + public static float[] toFloatArray(Object value) { if (value instanceof float[]) { return ((float[]) value).clone(); diff --git a/paimon-common/src/main/java/org/apache/paimon/predicate/VectorSearch.java b/paimon-common/src/main/java/org/apache/paimon/predicate/VectorSearch.java index 1114904bc6a3..2702dc732983 100644 --- a/paimon-common/src/main/java/org/apache/paimon/predicate/VectorSearch.java +++ b/paimon-common/src/main/java/org/apache/paimon/predicate/VectorSearch.java @@ -18,6 +18,7 @@ package org.apache.paimon.predicate; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.utils.RoaringNavigableMap64; import javax.annotation.Nullable; @@ -38,6 +39,7 @@ public class VectorSearch implements Serializable { private static final long serialVersionUID = 1L; private final float[] vector; + @Nullable private final float[][] multiVector; private final String fieldName; private final int limit; private final Map options; @@ -49,7 +51,16 @@ public VectorSearch(float[] vector, int limit, String fieldName) { } public VectorSearch(float[] vector, int limit, String fieldName, Map options) { - if (vector == null) { + this(vector, null, limit, fieldName, options); + } + + private VectorSearch( + float[] vector, + @Nullable float[][] multiVector, + int limit, + String fieldName, + Map options) { + if (vector == null && multiVector == null) { throw new IllegalArgumentException("Search vector cannot be null"); } if (limit <= 0) { @@ -59,6 +70,7 @@ public VectorSearch(float[] vector, int limit, String fieldName, Map(options)); } + public static VectorSearch multiVector( + float[][] vectors, int limit, String fieldName, Map options) { + VectorIndexUtils.validateQuery(vectors); + return new VectorSearch(null, vectors, limit, fieldName, options); + } + + @Nullable + public float[][] multiVector() { + return multiVector; + } + public float[] vector() { + if (multiVector != null) { + throw new IllegalStateException("Use multiVector() for a query matrix."); + } return vector; } @@ -94,7 +120,8 @@ public VectorSearch withIncludeRowIds(RoaringNavigableMap64 includeRowIds) { public VectorSearch offsetRange(long from, long to) { if (includeRowIds != null) { - VectorSearch target = new VectorSearch(vector, limit, fieldName, options()); + VectorSearch target = + new VectorSearch(vector, multiVector, limit, fieldName, options()); target.withIncludeRowIds(VectorSearchUtils.offsetRowIds(includeRowIds, from, to)); return target; } diff --git a/paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionGlobalIndexScanner.java b/paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionGlobalIndexScanner.java index c244ee1eed12..b188268e412a 100644 --- a/paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionGlobalIndexScanner.java +++ b/paimon-core/src/main/java/org/apache/paimon/globalindex/DataEvolutionGlobalIndexScanner.java @@ -503,7 +503,8 @@ private Collection createReaders( for (Map.Entry>> entry : group.metas.entrySet()) { String indexType = entry.getKey(); Map> metas = entry.getValue(); - GlobalIndexer globalIndexer = GlobalIndexer.create(indexType, indexFields, options); + GlobalIndexer globalIndexer = + GlobalIndexer.createForRead(indexType, indexFields, options); List> futures = new ArrayList<>(metas.size()); for (Map.Entry> rangeMetas : metas.entrySet()) { diff --git a/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java b/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java index a5914c5c0320..4432d3198894 100644 --- a/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java +++ b/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexBuilderUtils.java @@ -57,6 +57,7 @@ import java.util.Map; import java.util.function.BiFunction; import java.util.function.Supplier; +import java.util.stream.Collectors; import static org.apache.paimon.utils.Preconditions.checkArgument; @@ -65,6 +66,26 @@ public class GlobalIndexBuilderUtils { private static final Logger LOG = LoggerFactory.getLogger(GlobalIndexBuilderUtils.class); + /** Validates composite and multivector indexes before planning any shards to build. */ + public static void validateIndexFields(String type, List fields, Options options) { + if (fields.size() > 1) { + try { + GlobalIndexer.create(type, fields, options); + } catch (UnsupportedOperationException e) { + throw new IllegalArgumentException( + String.format( + "Index type '%s' does not support multi-column index, got columns: %s", + type, + fields.stream().map(DataField::name).collect(Collectors.toList())), + e); + } + } else if (fields.size() == 1 + && CoreOptions.isVectorIndexType(type) + && VectorIndexUtils.isMultiVector(fields.get(0).type())) { + GlobalIndexer.create(type, fields, options); + } + } + public static List toIndexFileMetas( FileIO fileIO, IndexPathFactory indexPathFactory, diff --git a/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java b/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java index f796c2394cd9..fc0a30a697ae 100644 --- a/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java +++ b/paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexQuery.java @@ -550,7 +550,8 @@ private Optional evaluateWithExecutor( supported = true; continue; } - GlobalIndexer indexer = GlobalIndexer.create(group.type, group.indexFields(), options); + GlobalIndexer indexer = + GlobalIndexer.createForRead(group.type, group.indexFields(), options); try (GlobalIndexReader reader = indexer.createReader( meta -> fileIO.newInputStream(meta.filePath()), diff --git a/paimon-core/src/main/java/org/apache/paimon/globalindex/MaxSimScorer.java b/paimon-core/src/main/java/org/apache/paimon/globalindex/MaxSimScorer.java new file mode 100644 index 000000000000..5c67438d144f --- /dev/null +++ b/paimon-core/src/main/java/org/apache/paimon/globalindex/MaxSimScorer.java @@ -0,0 +1,99 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.globalindex; + +import org.apache.paimon.data.InternalArray; +import org.apache.paimon.types.ArrayType; +import org.apache.paimon.utils.IntArrayList; + +import javax.annotation.Nullable; + +import java.util.Arrays; + +/** Scores complete embedding lists in one raw-data scan. */ +public final class MaxSimScorer { + + private final InternalArray.ElementGetter childGetter; + private final float[][] queries; + private final double[] queryNorms; + private final double[] maxima; + + public MaxSimScorer(ArrayType column, float[][] queries) { + VectorIndexUtils.validateQuery(column, queries); + this.childGetter = InternalArray.createElementGetter(column.getElementType()); + this.queries = queries; + this.queryNorms = new double[queries.length]; + this.maxima = new double[queries.length]; + for (int i = 0; i < queries.length; i++) { + queryNorms[i] = norm(queries[i]); + } + } + + /** Returns negative infinity when the row contains no embeddings. */ + public float score(InternalArray children) { + bestMatches(children, null, maxima); + double sum = 0; + for (double value : maxima) { + sum += value; + } + return (float) sum; + } + + /** Writes each selected query's best cosine in selection order. Null selects all queries. */ + public void bestMatches( + InternalArray children, @Nullable IntArrayList selected, double[] scores) { + int count = selected == null ? queries.length : selected.size(); + Arrays.fill(scores, 0, count, Double.NEGATIVE_INFINITY); + for (int j = 0; j < children.size(); j++) { + if (children.isNullAt(j)) { + continue; + } + float[] stored = + VectorIndexUtils.toFloatArray(childGetter.getElementOrNull(children, j)); + if (stored.length != queries[0].length) { + throw new IllegalArgumentException( + "Query vector dimension mismatch: expected " + + stored.length + + ", got " + + queries[0].length); + } + double storedNorm = norm(stored); + for (int pos = 0; pos < count; pos++) { + int i = selected == null ? pos : selected.get(pos); + double dot = 0; + for (int d = 0; d < stored.length; d++) { + dot += (double) queries[i][d] * stored[d]; + } + double denominator = queryNorms[i] * storedNorm; + scores[pos] = Math.max(scores[pos], denominator == 0 ? 0 : dot / denominator); + } + } + } + + private static double norm(float[] vector) { + double squared = 0; + for (float value : vector) { + if (!Float.isFinite(value)) { + throw new IllegalArgumentException("Multivector coordinates must be finite."); + } + squared += (double) value * value; + } + return Math.sqrt(squared); + } +} diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataEvolutionVectorRead.java b/paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataEvolutionVectorRead.java index 1a94edebf6de..8867ea05c225 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataEvolutionVectorRead.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/AbstractDataEvolutionVectorRead.java @@ -30,9 +30,11 @@ import org.apache.paimon.globalindex.GlobalIndexIOMeta; import org.apache.paimon.globalindex.GlobalIndexReader; import org.apache.paimon.globalindex.GlobalIndexer; +import org.apache.paimon.globalindex.MaxSimScorer; import org.apache.paimon.globalindex.OffsetGlobalIndexReader; import org.apache.paimon.globalindex.ScoredGlobalIndexResult; import org.apache.paimon.globalindex.VectorGlobalIndexer; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.globalindex.VectorSearchMetric; import org.apache.paimon.globalindex.io.GlobalIndexFileReader; import org.apache.paimon.index.GlobalIndexMeta; @@ -48,6 +50,7 @@ import org.apache.paimon.reader.RecordReader; import org.apache.paimon.table.FileStoreTable; import org.apache.paimon.table.SpecialFields; +import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataTypeRoot; import org.apache.paimon.types.RowType; @@ -94,6 +97,7 @@ public abstract class AbstractDataEvolutionVectorRead implements Serializable { protected final int limit; protected final DataField vectorColumn; protected final Map options; + @Nullable protected final float[][] queryMatrix; private final int candidatesFactor; // Serialized with Spark/Flink worker closures; reset for each new read. private boolean postFiltering; @@ -119,6 +123,18 @@ protected AbstractDataEvolutionVectorRead( int limit, DataField vectorColumn, @Nullable Map options) { + this(table, partitionFilter, filter, limit, vectorColumn, options, null); + } + + protected AbstractDataEvolutionVectorRead( + FileStoreTable table, + @Nullable PartitionPredicate partitionFilter, + @Nullable Predicate filter, + int limit, + DataField vectorColumn, + @Nullable Map options, + @Nullable float[][] queryMatrix) { + this.queryMatrix = queryMatrix; this.table = table; this.partitionFilter = partitionFilter; this.filter = filter; @@ -175,7 +191,7 @@ protected GlobalIndexer createGlobalIndexer(List splits) } protected GlobalIndexer createGlobalIndexer(String indexType) { - return GlobalIndexer.create( + return GlobalIndexer.createForRead( indexType, Collections.singletonList(vectorColumn), table.coreOptions().toConfiguration()); @@ -185,7 +201,7 @@ private GlobalIndexer createGlobalIndexer(String indexType, GlobalIndexMeta meta if (meta.extraFieldIds() == null) { return createGlobalIndexer(indexType); } - return GlobalIndexer.create( + return GlobalIndexer.createForRead( indexType, meta.getIndexedFields(table.rowType()), table.coreOptions().toConfiguration()); @@ -356,6 +372,26 @@ protected CompletableFuture> eval( int searchLimit, @Nullable RoaringNavigableMap64 includeRowIds, ExecutorService executor) { + if (queryMatrix != null) { + return evalBatch( + globalIndexer, + indexPathFactory, + rowRangeStart, + rowRangeEnd, + vectorIndexFiles, + queryMatrix, + searchLimit, + includeRowIds, + executor) + .thenApply( + perToken -> { + List candidates = new ArrayList<>(); + for (Optional hits : perToken) { + hits.ifPresent(candidates::add); + } + return Optional.of(mergeIndexedCandidates(candidates)); + }); + } if (vectorIndexFiles.isEmpty()) { return CompletableFuture.completedFuture(Optional.empty()); } @@ -529,7 +565,22 @@ protected ScoredGlobalIndexResult withRawSearch( } protected int indexedSearchLimit(String indexType) { - return VectorSearchRefineOptions.searchLimit(limit, configuredRefineFactor(indexType)); + int factor = configuredRefineFactor(indexType); + // Several moderately good token matches can outrank each token's individual winner. + return VectorSearchRefineOptions.searchLimit( + limit, queryMatrix != null && factor == 0 ? 4 : factor); + } + + protected ScoredGlobalIndexResult mergeIndexedCandidates(List hits) { + if (queryMatrix == null) { + return ScoredGlobalIndexResult.merge(hits); + } + // Only row IDs survive token retrieval. The full lists supply all MaxSim scores. + RoaringNavigableMap64 rows = new RoaringNavigableMap64(); + for (ScoredGlobalIndexResult hit : hits) { + rows.or(hit.results()); + } + return ScoredGlobalIndexResult.create(rows, id -> 0f); } protected ScoredGlobalIndexResult maybeRerankIndexedResult( @@ -537,7 +588,12 @@ protected ScoredGlobalIndexResult maybeRerankIndexedResult( String indexType, @Nullable GlobalIndexer globalIndexer, float[] queryVector) { - if (configuredRefineFactor(indexType) == 0 || result.results().isEmpty()) { + if (queryMatrix != null && !result.results().isEmpty()) { + return readRawRefineSearch(result.results(), globalIndexer, queryVector); + } + if ((!VectorIndexUtils.isMultiVector(vectorColumn.type()) + && configuredRefineFactor(indexType) == 0) + || result.results().isEmpty()) { return result; } ScoredGlobalIndexResult candidates = result.topK(indexedSearchLimit(indexType)); @@ -625,31 +681,31 @@ private ScoredGlobalIndexResult readRawSearch( String metric, float[] queryVector, boolean includeFilter) { - RowType readType = rawSearchReadType(includeFilter); rawRowRanges = filteredRawRowRanges(rawRowRanges, preFilter); if (rawRowRanges.isEmpty()) { return ScoredGlobalIndexResult.createEmpty(); } - TableScan.Plan plan = - newRawReadBuilder(readType, false).withRowRanges(rawRowRanges).newScan().plan(); - ReadBuilder readBuilder = newRawReadBuilder(readType, includeFilter); - int vectorIndex = readType.getFieldIndex(vectorColumn.name()); - int rowIdIndex = readType.getFieldIndex(SpecialFields.ROW_ID.name()); PriorityQueue topKHeap = new PriorityQueue<>(Math.max(1, limit + 1), WEAKEST_SCORE_FIRST); - - try (RecordReader reader = - readBuilder.newRead().executeFilter().createReader(plan)) { - reader.forEachRemaining( - row -> { - long rowId = row.getLong(rowIdIndex); - if (scoreCandidates != null && !scoreCandidates.contains(rowId)) { - return; - } - if (row.isNullAt(vectorIndex)) { - return; + MaxSimScorer scorer = + VectorIndexUtils.isMultiVector(vectorColumn.type()) + ? new MaxSimScorer( + (ArrayType) vectorColumn.type(), + queryMatrix == null ? new float[][] {queryVector} : queryMatrix) + : null; + + readRawVectorRows( + rawRowRanges, + scoreCandidates, + includeFilter, + (rowId, row, vectorIndex) -> { + if (scorer != null) { + float score = scorer.score(row.getArray(vectorIndex)); + if (score != Float.NEGATIVE_INFINITY) { + offerScore(topKHeap, limit, rowId, score); } + } else { float[] stored = getVector(row, vectorIndex); checkVectorDimension(queryVector, stored); offerScore( @@ -657,11 +713,8 @@ private ScoredGlobalIndexResult readRawSearch( limit, rowId, VectorSearchMetric.computeScore(queryVector, stored, metric)); - }); - } catch (IOException e) { - throw new RuntimeException("Failed to read raw vectors for vector search.", e); - } - + } + }); return scoredResult(topKHeap); } @@ -674,16 +727,32 @@ protected Map readRawVectors( List rawRowRanges, @Nullable RoaringNavigableMap64 candidates, boolean includeFilter) { - RowType readType = rawSearchReadType(includeFilter); - rawRowRanges = filteredRawRowRanges(rawRowRanges, null); + Map rawVectors = new HashMap<>(); + readRawVectorRows( + rawRowRanges, + candidates, + includeFilter, + (rowId, row, vectorIndex) -> rawVectors.put(rowId, getVector(row, vectorIndex))); + return rawVectors; + } + + @FunctionalInterface + interface RawVectorConsumer { + void accept(long rowId, InternalRow row, int vectorIndex); + } + + void readRawVectorRows( + List rawRowRanges, + @Nullable RoaringNavigableMap64 candidates, + boolean includeFilter, + RawVectorConsumer consumer) { if (rawRowRanges.isEmpty()) { - return Collections.emptyMap(); + return; } - + RowType readType = rawSearchReadType(includeFilter); TableScan.Plan plan = newRawReadBuilder(readType, false).withRowRanges(rawRowRanges).newScan().plan(); ReadBuilder readBuilder = newRawReadBuilder(readType, includeFilter); - Map rawVectors = new HashMap<>(); int vectorIndex = readType.getFieldIndex(vectorColumn.name()); int rowIdIndex = readType.getFieldIndex(SpecialFields.ROW_ID.name()); @@ -698,14 +767,11 @@ protected Map readRawVectors( if (row.isNullAt(vectorIndex)) { return; } - float[] stored = getVector(row, vectorIndex); - rawVectors.put(rowId, stored); + consumer.accept(rowId, row, vectorIndex); }); } catch (IOException e) { throw new RuntimeException("Failed to read raw vectors for vector search.", e); } - - return rawVectors; } private List filteredRawRowRanges( @@ -871,6 +937,9 @@ private float[] getVector(InternalRow row, int vectorIndex) { } protected String rawSearchMetric(@Nullable GlobalIndexer globalIndexer) { + if (VectorIndexUtils.isMultiVector(vectorColumn.type())) { + return "cosine"; + } String metric = null; if (globalIndexer != null) { if (!(globalIndexer instanceof VectorGlobalIndexer)) { @@ -899,12 +968,12 @@ private String configuredRawSearchMetric(Map options) { if (options.containsKey(fieldPrefix + "index-options")) { CoreOptions configured = new CoreOptions(new Options(table.options(), options)); VectorIndexOptions resolved = - VectorIndexOptions.resolve( - configured.toConfiguration(), null, vectorColumn.name(), false); + VectorIndexOptions.resolveConfigured( + configured.toConfiguration(), null, vectorColumn.name()); if (resolved.type() == null) { resolved = - VectorIndexOptions.resolve( - configured.toConfiguration(), "vector", vectorColumn.name(), false); + VectorIndexOptions.resolveConfigured( + configured.toConfiguration(), "vector", vectorColumn.name()); } String configuredMetric = resolved.options().get(resolved.type() + ".metric"); if (configuredMetric != null) { diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionBatchVectorRead.java b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionBatchVectorRead.java index b53f82caa547..2abd2b111325 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionBatchVectorRead.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionBatchVectorRead.java @@ -21,13 +21,17 @@ import org.apache.paimon.globalindex.GlobalIndexReadThreadPool; import org.apache.paimon.globalindex.GlobalIndexResult; import org.apache.paimon.globalindex.GlobalIndexer; +import org.apache.paimon.globalindex.MaxSimScorer; import org.apache.paimon.globalindex.ScoreGetter; import org.apache.paimon.globalindex.ScoredGlobalIndexResult; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.index.IndexPathFactory; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.Predicate; import org.apache.paimon.table.FileStoreTable; +import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataField; +import org.apache.paimon.utils.IntArrayList; import org.apache.paimon.utils.Range; import org.apache.paimon.utils.RoaringNavigableMap64; import org.apache.paimon.utils.RowRangeIndex; @@ -38,6 +42,7 @@ import java.util.ArrayList; import java.util.Comparator; import java.util.Deque; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Optional; @@ -65,6 +70,9 @@ public DataEvolutionBatchVectorRead( @Nullable Map options) { super(table, partitionFilter, filter, limit, vectorColumn, options); this.vectors = vectors; + if (vectorColumn != null && VectorIndexUtils.isMultiVector(vectorColumn.type())) { + VectorIndexUtils.validateQuery((ArrayType) vectorColumn.type(), vectors); + } } @Override @@ -132,7 +140,7 @@ private ScoredGlobalIndexResult[] readIndexedBatch( int parallelism = table.coreOptions().toConfiguration().get(GLOBAL_INDEX_THREAD_NUM); ExecutorService executor = GlobalIndexReadThreadPool.getExecutorService(parallelism); - boolean refine = configuredRefineFactor(indexType) != 0; + boolean refine = requiresRefinement(indexType); // A refined group contributes only the rows that survive reranking. Until then its // duplicate tracking is independent of previous groups. RoaringNavigableMap64[] seen = refine ? overlapBitmaps(n, overlaps) : indexedRows; @@ -233,7 +241,7 @@ private static RowRangeIndex overlappingRanges( protected ScoredGlobalIndexResult[] maybeRerankIndexedBatchResults( ScoredGlobalIndexResult[] results, String indexType, GlobalIndexer globalIndexer) { - if (configuredRefineFactor(indexType) == 0) { + if (!requiresRefinement(indexType)) { return results; } @@ -250,6 +258,9 @@ protected ScoredGlobalIndexResult[] maybeRerankIndexedBatchResults( return candidates; } + if (VectorIndexUtils.isMultiVector(vectorColumn.type())) { + return rerankMultiVectors(candidates, unionCandidates); + } Map rawVectors = readRawVectors(unionCandidates, false); String metric = rawSearchMetric(globalIndexer); ScoredGlobalIndexResult[] reranked = new ScoredGlobalIndexResult[n]; @@ -259,4 +270,40 @@ protected ScoredGlobalIndexResult[] maybeRerankIndexedBatchResults( } return reranked; } + + private boolean requiresRefinement(String indexType) { + return VectorIndexUtils.isMultiVector(vectorColumn.type()) + || configuredRefineFactor(indexType) != 0; + } + + private ScoredGlobalIndexResult[] rerankMultiVectors( + ScoredGlobalIndexResult[] candidates, RoaringNavigableMap64 union) { + Map queriesByRow = new HashMap<>(); + List> heaps = new ArrayList<>(vectors.length); + for (int i = 0; i < vectors.length; i++) { + heaps.add(new PriorityQueue<>(WEAKEST_SCORE_FIRST)); + for (long rowId : candidates[i].results()) { + queriesByRow.computeIfAbsent(rowId, ignored -> new IntArrayList(1)).add(i); + } + } + MaxSimScorer scorer = new MaxSimScorer((ArrayType) vectorColumn.type(), vectors); + double[] scores = new double[vectors.length]; + readRawVectorRows( + union.toRangeList(), + union, + false, + (rowId, row, vectorIndex) -> { + IntArrayList queries = queriesByRow.get(rowId); + scorer.bestMatches(row.getArray(vectorIndex), queries, scores); + for (int i = 0; i < queries.size(); i++) { + if (scores[i] != Double.NEGATIVE_INFINITY) { + offerScore(heaps.get(queries.get(i)), limit, rowId, (float) scores[i]); + } + } + }); + for (int i = 0; i < vectors.length; i++) { + candidates[i] = scoredResult(heaps.get(i)); + } + return candidates; + } } diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorRead.java b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorRead.java index ee5b14aa10aa..c22d6e4d3784 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorRead.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorRead.java @@ -22,10 +22,12 @@ import org.apache.paimon.globalindex.GlobalIndexResult; import org.apache.paimon.globalindex.GlobalIndexer; import org.apache.paimon.globalindex.ScoredGlobalIndexResult; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.index.IndexPathFactory; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.Predicate; import org.apache.paimon.table.FileStoreTable; +import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataField; import org.apache.paimon.utils.RoaringNavigableMap64; @@ -55,8 +57,25 @@ public DataEvolutionVectorRead( DataField vectorColumn, float[] vector, @Nullable Map options) { - super(table, partitionFilter, filter, limit, vectorColumn, options); + this(table, partitionFilter, filter, limit, vectorColumn, vector, options, null); + } + + public DataEvolutionVectorRead( + FileStoreTable table, + @Nullable PartitionPredicate partitionFilter, + @Nullable Predicate filter, + int limit, + DataField vectorColumn, + float[] vector, + @Nullable Map options, + @Nullable float[][] queryMatrix) { + super(table, partitionFilter, filter, limit, vectorColumn, options, queryMatrix); this.vector = vector; + if (vectorColumn != null && VectorIndexUtils.isMultiVector(vectorColumn.type())) { + VectorIndexUtils.validateQuery( + (ArrayType) vectorColumn.type(), + queryMatrix == null ? new float[][] {vector} : queryMatrix); + } } @Override @@ -128,6 +147,6 @@ protected ScoredGlobalIndexResult readIndexed( } } return maybeRerankIndexedResult( - ScoredGlobalIndexResult.merge(results), indexType, globalIndexer, vector); + mergeIndexedCandidates(results), indexType, globalIndexer, vector); } } diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorScan.java b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorScan.java index 267c09371eaf..28e0e352e65b 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorScan.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/DataEvolutionVectorScan.java @@ -349,8 +349,8 @@ private static List addAll(List left, List right) { @Nullable private String configuredVectorIndexType() { - return VectorIndexOptions.resolve( - new Options(table.options(), options), null, vectorColumn.name(), false) + return VectorIndexOptions.resolveConfigured( + new Options(table.options(), options), null, vectorColumn.name()) .type(); } diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilder.java b/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilder.java index e33f1028669a..a35000ded120 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilder.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilder.java @@ -43,6 +43,12 @@ public interface VectorSearchBuilder extends Serializable { /** The vector to search. */ VectorSearchBuilder withVector(float[] vector); + /** Rank each multivector row once using the sum of its best cosine matches. */ + default VectorSearchBuilder withMultiVector(float[][] vectors) { + throw new UnsupportedOperationException( + "Multivector queries are not supported by this builder."); + } + /** Option for vector indexes. */ default VectorSearchBuilder withOption(String key, String value) { throw new UnsupportedOperationException( diff --git a/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilderImpl.java b/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilderImpl.java index 200a8d593fe2..eb496db63340 100644 --- a/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilderImpl.java +++ b/paimon-core/src/main/java/org/apache/paimon/table/source/VectorSearchBuilderImpl.java @@ -20,6 +20,7 @@ import org.apache.paimon.Snapshot; import org.apache.paimon.catalog.TableQueryAuthResult; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.options.VectorIndexOptions; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.Predicate; @@ -52,6 +53,7 @@ public class VectorSearchBuilderImpl implements VectorSearchBuilder { protected int limit; protected DataField vectorColumn; protected float[] vector; + @Nullable protected float[][] multiVector; protected Map options = new HashMap<>(); @Nullable private Snapshot pinnedSnapshot; @@ -110,10 +112,19 @@ public VectorSearchBuilder withVectorColumn(String name) { @Override public VectorSearchBuilder withVector(float[] vector) { + this.multiVector = null; this.vector = vector; return this; } + @Override + public VectorSearchBuilder withMultiVector(float[][] vectors) { + VectorIndexUtils.validateQuery(vectors); + this.multiVector = vectors; + this.vector = vectors[0]; + return this; + } + @Override public VectorSearchBuilder withOptions(Map options) { if (options != null) { @@ -130,7 +141,7 @@ public VectorSearchBuilder withOption(String key, String value) { @Override public VectorScan newVectorScan() { - rejectUnderQueryAuth(); + validateSearch(); if (isPrimaryKeyVectorSearch()) { return new PrimaryKeyVectorScan( table, vectorColumn.id(), partitionFilter, filter, pinnedSnapshot); @@ -141,13 +152,13 @@ public VectorScan newVectorScan() { @Override public VectorRead newVectorRead() { - rejectUnderQueryAuth(); + validateSearch(); checkNotNull(vector, "vector must be set via withVector()"); if (isPrimaryKeyVectorSearch()) { return new PrimaryKeyVectorRead(table, vectorColumn, vector, limit, options, filter); } return new DataEvolutionVectorRead( - table, partitionFilter, filter, limit, vectorColumn, vector, options); + table, partitionFilter, filter, limit, vectorColumn, vector, options, multiVector); } protected boolean isPrimaryKeyVectorSearch() { @@ -162,6 +173,17 @@ public VectorSearchBuilderImpl withSnapshot(Snapshot snapshot) { return this; } + protected void validateSearch() { + rejectUnderQueryAuth(); + if (multiVector != null + && (!table.coreOptions().dataEvolutionEnabled() + || vectorColumn == null + || !VectorIndexUtils.isMultiVector(vectorColumn.type()))) { + throw new IllegalArgumentException( + "A query matrix requires a multivector column on a Data Evolution table."); + } + } + protected void rejectUnderQueryAuth() { TableQueryAuthResult.rejectSearchUnderQueryAuth(table); } diff --git a/paimon-core/src/test/java/org/apache/paimon/GlobalIndexOptionsTest.java b/paimon-core/src/test/java/org/apache/paimon/GlobalIndexOptionsTest.java index cf9acfff7898..227470e18735 100644 --- a/paimon-core/src/test/java/org/apache/paimon/GlobalIndexOptionsTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/GlobalIndexOptionsTest.java @@ -123,10 +123,7 @@ void testColumnDeclarationsAndLegacyFallback(String type, String legacyFamily) { if ("ivf-pq".equals(type)) { assertThat(VectorIndexOptions.columns(options.toConfiguration())) .containsExactly("first", "second"); - assertThat( - VectorIndexOptions.resolve( - options.toConfiguration(), null, "first", false) - .type()) + assertThat(VectorIndexOptions.resolve(options.toConfiguration(), null, "first").type()) .isEqualTo("ivf-pq"); } } @@ -226,6 +223,31 @@ void testTypedFullTextFieldOptionsUseNativeEncoding() { assertThat(resolved.get("full-text.stop-words")).isEqualTo("paimon;lake"); } + @Test + void testPluginReceivesSingleVectorDefaultAndExplicitListMetric() { + DataField vector = new DataField(0, "embedding", DataTypes.VECTOR(2, DataTypes.FLOAT())); + DataField list = new DataField(0, "embedding", DataTypes.ARRAY(vector.type())); + Options options = new Options(); + + assertThat( + GlobalIndexer.resolveOptions( + "ivf-flat", Collections.singletonList(vector), options) + .get("ivf-flat.metric")) + .isEqualTo("inner_product"); + assertThat( + GlobalIndexer.resolveOptions( + "ivf-flat", Collections.singletonList(list), options) + .get("ivf-flat.metric")) + .isNull(); + + options.setString("fields.embedding.index-options", "{\"metric\":\"INNER-PRODUCT\"}"); + assertThat( + GlobalIndexer.resolveOptions( + "ivf-flat", Collections.singletonList(list), options) + .get("ivf-flat.metric")) + .isEqualTo("inner_product"); + } + @Test void testUnrelatedFieldOptionsDoNotRequestDefaultMetric() { CoreOptions options = @@ -233,8 +255,8 @@ void testUnrelatedFieldOptionsDoNotRequestDefaultMetric() { Collections.singletonMap( "fields.embedding.index-options", "{\"nlist\":64}")); assertThat( - VectorIndexOptions.resolve( - options.toConfiguration(), "ivf-pq", "embedding", false) + VectorIndexOptions.resolveConfigured( + options.toConfiguration(), "ivf-pq", "embedding") .options() .get("ivf-pq.metric")) .isNull(); diff --git a/paimon-core/src/test/java/org/apache/paimon/globalindex/GlobalIndexQueryTest.java b/paimon-core/src/test/java/org/apache/paimon/globalindex/GlobalIndexQueryTest.java index 23bf51366c1f..e22a7ff4554a 100644 --- a/paimon-core/src/test/java/org/apache/paimon/globalindex/GlobalIndexQueryTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/globalindex/GlobalIndexQueryTest.java @@ -172,7 +172,7 @@ void workerReceivesRoutedFilesAfterSplitSerialization() throws Exception { return reader; }); GlobalIndexerFactory factory = mock(GlobalIndexerFactory.class); - when(factory.create(anyList(), any())).thenReturn(indexer); + when(factory.createForRead(anyList(), any())).thenReturn(indexer); try (MockedStatic factories = mockStatic(GlobalIndexerFactoryUtils.class)) { factories.when(() -> GlobalIndexerFactoryUtils.load("ivf-pq")).thenReturn(factory); @@ -268,7 +268,7 @@ void testBoundedRangeUsesSingleScan(boolean lowerInclusive, boolean upperInclusi GlobalIndexer indexer = mock(GlobalIndexer.class); when(indexer.createReader(any(), anyList(), eq(100L), anyList(), any())).thenReturn(reader); GlobalIndexerFactory factory = mock(GlobalIndexerFactory.class); - when(factory.create(anyList(), any(Options.class))).thenReturn(indexer); + when(factory.createForRead(anyList(), any(Options.class))).thenReturn(indexer); try (MockedStatic factories = mockStatic(GlobalIndexerFactoryUtils.class)) { factories.when(() -> GlobalIndexerFactoryUtils.load("btree")).thenReturn(factory); @@ -464,7 +464,7 @@ void testUnsupportedPredicateFailsAtRuntime(String indexType) { when(indexer.createReader(any(), anyList(), anyLong(), anyList(), any())) .thenReturn(reader); GlobalIndexerFactory factory = mock(GlobalIndexerFactory.class); - when(factory.create(anyList(), any(Options.class))).thenReturn(indexer); + when(factory.createForRead(anyList(), any(Options.class))).thenReturn(indexer); try (MockedStatic factories = mockStatic(GlobalIndexerFactoryUtils.class)) { factories.when(() -> GlobalIndexerFactoryUtils.load(indexType)).thenReturn(factory); @@ -534,7 +534,7 @@ public List toRangeList() { when(indexer.createReader(any(), anyList(), anyLong(), anyList(), any())) .thenReturn(reader); GlobalIndexerFactory factory = mock(GlobalIndexerFactory.class); - when(factory.create(anyList(), any(Options.class))).thenReturn(indexer); + when(factory.createForRead(anyList(), any(Options.class))).thenReturn(indexer); RowType rowType = RowType.of(DataTypes.INT()); GlobalIndexQuery plan = @@ -609,7 +609,7 @@ void testPushesLocalRangesToIndexer(String indexType) throws Exception { GlobalIndexer indexer = mock(GlobalIndexer.class); when(indexer.createReader(any(), anyList(), eq(100L), anyList(), any())).thenReturn(reader); GlobalIndexerFactory factory = mock(GlobalIndexerFactory.class); - when(factory.create(anyList(), any(Options.class))).thenReturn(indexer); + when(factory.createForRead(anyList(), any(Options.class))).thenReturn(indexer); FileIO fileIO = mock(FileIO.class); List first = Collections.singletonList(new Range(110, 119)); List second = Collections.singletonList(new Range(130, 139)); diff --git a/paimon-core/src/test/java/org/apache/paimon/globalindex/VectorSearchMetricTest.java b/paimon-core/src/test/java/org/apache/paimon/globalindex/VectorSearchMetricTest.java index e13e7c66e257..7394d74e0b38 100644 --- a/paimon-core/src/test/java/org/apache/paimon/globalindex/VectorSearchMetricTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/globalindex/VectorSearchMetricTest.java @@ -18,6 +18,10 @@ package org.apache.paimon.globalindex; +import org.apache.paimon.data.GenericArray; +import org.apache.paimon.types.ArrayType; +import org.apache.paimon.types.FloatType; + import org.junit.jupiter.api.Test; import static org.assertj.core.api.Assertions.assertThat; @@ -26,6 +30,30 @@ /** Tests for {@link VectorSearchMetric}. */ class VectorSearchMetricTest { + @Test + void testMaxSimResetsNegativeScoresAndValidatesEmbeddingLists() { + ArrayType column = new ArrayType(new ArrayType(new FloatType())); + MaxSimScorer scorer = new MaxSimScorer(column, new float[][] {{Float.MAX_VALUE}, {1}}); + GenericArray negative = new GenericArray(new float[] {-Float.MAX_VALUE}); + GenericArray positive = new GenericArray(new float[] {Float.MIN_VALUE}); + assertThat(scorer.score(new GenericArray(new Object[] {positive, positive}))).isEqualTo(2f); + assertThat(scorer.score(new GenericArray(new Object[] {null, negative}))).isEqualTo(-2f); + assertThat(scorer.score(new GenericArray(new Object[0]))) + .isEqualTo(Float.NEGATIVE_INFINITY); + assertThatIllegalArgumentException() + .isThrownBy(() -> new MaxSimScorer(column, new float[0][])); + assertThatIllegalArgumentException() + .isThrownBy(() -> new MaxSimScorer(column, new float[][] {{1}, {1, 2}})); + assertThatIllegalArgumentException() + .isThrownBy( + () -> + scorer.score( + new GenericArray( + new Object[] { + new GenericArray(new float[] {Float.NaN}) + }))); + } + @Test void testNormalizeAndSupportedMetrics() { assertThat(VectorSearchMetric.normalize("INNER-PRODUCT")).isEqualTo("inner_product"); diff --git a/paimon-core/src/test/java/org/apache/paimon/index/pkvector/PrimaryKeyVectorIndexOptionsTest.java b/paimon-core/src/test/java/org/apache/paimon/index/pkvector/PrimaryKeyVectorIndexOptionsTest.java index 76585bcd6637..926b97742411 100644 --- a/paimon-core/src/test/java/org/apache/paimon/index/pkvector/PrimaryKeyVectorIndexOptionsTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/index/pkvector/PrimaryKeyVectorIndexOptionsTest.java @@ -79,9 +79,7 @@ void testIndexTypeMustBeFieldScoped() { options.put("pk-vector.index.columns", "embedding"); options.put("pk-vector.index.type", "ivf-pq"); - assertThat( - VectorIndexOptions.resolve(new Options(options), null, "embedding", false) - .type()) + assertThat(VectorIndexOptions.resolve(new Options(options), null, "embedding").type()) .isNull(); } diff --git a/paimon-core/src/test/java/org/apache/paimon/options/VectorIndexOptionsTest.java b/paimon-core/src/test/java/org/apache/paimon/options/VectorIndexOptionsTest.java index 6dbd5bf27886..d460b1cc45f1 100644 --- a/paimon-core/src/test/java/org/apache/paimon/options/VectorIndexOptionsTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/options/VectorIndexOptionsTest.java @@ -59,7 +59,7 @@ void testResolutionReturnsMatchingTypeAndOptions(String declaration) { assertThat(values).isEqualTo(original); VectorIndexOptions persisted = - VectorIndexOptions.resolve(new Options(values), "ivf-flat", "embedding", false); + VectorIndexOptions.resolve(new Options(values), "ivf-flat", "embedding"); assertThat(persisted.type()).isEqualTo("ivf-flat"); assertThat(persisted.options().get("ivf-flat.nlist")).isEqualTo("64"); assertThat(persisted.metric()).isEqualTo("cosine"); @@ -73,8 +73,7 @@ void testLegacyRegistrationIsOnlyACompatibilityFallback(String suffix) { input.setString("fields.old.pk-vector.index.type", "ivf-pq"); input.setString("fields.other_old.pk-vector.index.type", "diskann"); assertThat(VectorIndexOptions.columns(input)).containsExactly("old", "other_old"); - assertThat(VectorIndexOptions.resolve(input, null, "old", false).type()) - .isEqualTo("ivf-pq"); + assertThat(VectorIndexOptions.resolve(input, null, "old").type()).isEqualTo("ivf-pq"); assertThat(new CoreOptions(input).vectorIndexColumns("ivf-pq")).containsExactly("old"); assertThat(new CoreOptions(input).vectorIndexColumns("diskann")) .containsExactly("other_old"); @@ -85,10 +84,8 @@ void testLegacyRegistrationIsOnlyACompatibilityFallback(String suffix) { input.setString("global-index.diskann." + suffix, "video"); assertThat(VectorIndexOptions.columns(input)) .containsExactly("video", "embedding", "other"); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).type()) - .isEqualTo("ivf-pq"); - assertThat(VectorIndexOptions.resolve(input, null, "video", false).type()) - .isEqualTo("diskann"); + assertThat(VectorIndexOptions.resolve(input, null, "embedding").type()).isEqualTo("ivf-pq"); + assertThat(VectorIndexOptions.resolve(input, null, "video").type()).isEqualTo("diskann"); assertThat(new CoreOptions(input).vectorIndexColumns("ivf-pq")) .containsExactly("embedding", "other"); assertThat(new CoreOptions(input).vectorIndexColumns("diskann")).containsExactly("video"); @@ -106,14 +103,14 @@ void testLegacyRegistrationIsOnlyACompatibilityFallback(String suffix) { }) void testRemovedTypeAliasesAreIgnored(String key) { Options input = Options.fromMap(Collections.singletonMap(key, "ivf-pq")); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).type()).isNull(); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).options().toMap()) + assertThat(VectorIndexOptions.resolve(input, null, "embedding").type()).isNull(); + assertThat(VectorIndexOptions.resolveConfigured(input, null, "embedding").options().toMap()) .isEqualTo(input.toMap()); input.setString("fields.embedding.pk-vector.index.type", "diskann"); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).type()) + assertThat(VectorIndexOptions.resolve(input, null, "embedding").type()) .isEqualTo("diskann"); input.setString("global-index.ivf-flat.index-columns", "embedding"); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).type()) + assertThat(VectorIndexOptions.resolve(input, null, "embedding").type()) .isEqualTo("ivf-flat"); } @@ -124,13 +121,15 @@ void testLegacyPrimaryKeyTypeResolvesFieldParameters() { values.put("ivf-pq.dimension", "768"); values.put("fields.embedding.index-options", "{\"nlist\":64,\"metric\":\"COSINE\"}"); Options input = new Options(values); - assertThat(VectorIndexOptions.resolve(input, null, "embedding", false).type()) - .isEqualTo("ivf-pq"); - Options resolved = VectorIndexOptions.resolve(input, null, "embedding", false).options(); + assertThat(VectorIndexOptions.resolve(input, null, "embedding").type()).isEqualTo("ivf-pq"); + Options resolved = VectorIndexOptions.resolveConfigured(input, null, "embedding").options(); assertMetric(resolved, "cosine"); assertThat(resolved.get("ivf-pq.dimension")).isEqualTo("768"); assertThat(resolved.get("ivf-pq.nlist")).isEqualTo("64"); - assertThat(VectorIndexOptions.resolve(resolved, null, "embedding", false).options().toMap()) + assertThat( + VectorIndexOptions.resolveConfigured(resolved, null, "embedding") + .options() + .toMap()) .isEqualTo(resolved.toMap()); assertThat(input.toMap()).isEqualTo(values); } @@ -141,7 +140,7 @@ void testAmbiguousDeclarationsDoNotFallBackToHistoricalHint() { options.setString("global-index.ivf-pq.index-columns", "embedding"); options.setString("global-index.diskann.index-column", "embedding"); options.setString("index_type", "ivf-flat"); - assertThat(VectorIndexOptions.resolve(options, null, "embedding", false).type()).isNull(); + assertThat(VectorIndexOptions.resolve(options, null, "embedding").type()).isNull(); } @ParameterizedTest @@ -201,7 +200,7 @@ void testReaderKeepsUnspecifiedMetricAndWriterUsesDefault() { new Options( Collections.singletonMap( "fields.embedding.index-options", "{\"nlist\":64}")); - Options read = VectorIndexOptions.resolve(input, "ivf-pq", "embedding", false).options(); + Options read = VectorIndexOptions.resolveConfigured(input, "ivf-pq", "embedding").options(); assertThat(read.get("ivf-pq.nlist")).isEqualTo("64"); assertThat(read.get("ivf-pq.metric")).isNull(); assertThat(read.get("fields.embedding.distance.metric")).isNull(); diff --git a/paimon-core/src/test/java/org/apache/paimon/table/source/VectorSearchBuilderTest.java b/paimon-core/src/test/java/org/apache/paimon/table/source/VectorSearchBuilderTest.java index 30900d54a4d1..506c99802a4f 100644 --- a/paimon-core/src/test/java/org/apache/paimon/table/source/VectorSearchBuilderTest.java +++ b/paimon-core/src/test/java/org/apache/paimon/table/source/VectorSearchBuilderTest.java @@ -912,6 +912,39 @@ public void testVectorSearchRefineFactorReranksIndexCandidates() throws Exceptio assertThat(fieldBatch.get(1).results()).containsExactly(2L); } + @Test + public void testTypedMultiVectorValidatesQueryDimensionsOnEmptyTable() throws Exception { + Identifier id = identifier("empty_multi_vector"); + catalog.createTable( + id, + Schema.newBuilder() + .column("chunks", DataTypes.ARRAY(DataTypes.VECTOR(3, DataTypes.FLOAT()))) + .option("bucket", "-1") + .option("row-tracking.enabled", "true") + .option("data-evolution.enabled", "true") + .build(), + false); + FileStoreTable table = getTable(id); + float[][] wrong = {{1, 0}, {0, 1}}; + VectorSearchBuilder search = + table.newVectorSearchBuilder().withVectorColumn("chunks").withLimit(1); + assertThatThrownBy(() -> search.withMultiVector(wrong).executeLocal()) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("expected 3, got 2"); + assertThatThrownBy(() -> search.withVector(wrong[0]).executeLocal()) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("expected 3, got 2"); + assertThatThrownBy( + () -> + table.newBatchVectorSearchBuilder() + .withVectorColumn("chunks") + .withLimit(1) + .withVectors(wrong) + .executeBatchLocal()) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("expected 3, got 2"); + } + @Test public void testSharedVectorDeclarationKeepsDataEvolutionReads() throws Exception { catalog.createTable( @@ -998,6 +1031,100 @@ public void testBatchRefineReadsUnionCandidatesOnceAndScoresPerQuery() { assertThat(reranked[1].results()).containsExactly(2L); } + @Test + public void testBatchReranksMultiVectorCandidatesInOneRawReadByDefault() throws Exception { + Identifier id = identifier("multi_vector_refine"); + catalog.createTable( + id, + withVectorSchemaOptions( + Schema.newBuilder() + .column( + "chunks", + DataTypes.ARRAY( + DataTypes.ARRAY(DataTypes.FLOAT())))) + .build(), + false); + FileStoreTable table = getTable(id); + GenericArray x = new GenericArray(new float[] {1, 0}); + GenericArray y = new GenericArray(new float[] {0, 1}); + write( + table, + GenericRow.of(new GenericArray(new Object[] {x})), + GenericRow.of(new GenericArray(new Object[] {y})), + GenericRow.of(new GenericArray(new Object[] {x, y}))); + + AtomicInteger rawReads = new AtomicInteger(); + DataEvolutionBatchVectorRead read = + new DataEvolutionBatchVectorRead( + table, + null, + null, + 2, + table.rowType().getField("chunks"), + new float[][] {{1, 0}, {0, 1}}, + null) { + @Override + protected GlobalIndexer createGlobalIndexer( + List splits) { + return null; + } + + @Override + protected CompletableFuture>> evalBatch( + GlobalIndexer indexer, + IndexPathFactory paths, + long from, + long to, + List files, + float[][] queries, + int k, + RoaringNavigableMap64 included, + ExecutorService executor) { + // Row 2 is shared. Each other row is a candidate for only its *opposite* + // query. + return CompletableFuture.completedFuture( + Arrays.asList( + Optional.of( + VectorSearchBuilderTest.scoredResult(0.5f, 1L, 2L)), + Optional.of( + VectorSearchBuilderTest.scoredResult( + 0.5f, 0L, 2L)))); + } + + @Override + void readRawVectorRows( + List ranges, + RoaringNavigableMap64 candidates, + boolean includeFilter, + RawVectorConsumer consumer) { + rawReads.incrementAndGet(); + assertThat(candidates).containsExactly(0L, 1L, 2L); + super.readRawVectorRows(ranges, candidates, includeFilter, consumer); + } + }; + IndexFileMeta file = + new IndexFileMeta( + "ivf-pq", + "part", + 0, + 3, + null, + null, + new GlobalIndexMeta(0, 2, 0, null, null)); + IndexVectorSearchSplit split = + new IndexVectorSearchSplit( + 0, 2, Collections.singletonList(file), Collections.emptyList()); + List results = read.readBatch(() -> Collections.singletonList(split)); + assertThat(rawReads.get()).isEqualTo(1); + for (int i = 0; i < results.size(); i++) { + ScoredGlobalIndexResult result = (ScoredGlobalIndexResult) results.get(i); + long opposite = 1 - i; + assertThat(result.results()).containsExactly(opposite, 2L); + assertThat(result.scoreGetter().score(opposite)).isZero(); + assertThat(result.scoreGetter().score(2L)).isEqualTo(1f); + } + } + @Test public void testVectorSearchRefineFactorValidation() throws Exception { createTableDefault(); diff --git a/paimon-flink/paimon-flink-common/pom.xml b/paimon-flink/paimon-flink-common/pom.xml index 131fdbc92fe1..a985dcc6f1e2 100644 --- a/paimon-flink/paimon-flink-common/pom.xml +++ b/paimon-flink/paimon-flink-common/pom.xml @@ -262,6 +262,13 @@ under the License. test + + org.apache.paimon + paimon-vector + ${project.version} + test + + org.xerial sqlite-jdbc diff --git a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java index 6c40f0937115..c0bfd3a29481 100644 --- a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java +++ b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java @@ -20,7 +20,7 @@ import org.apache.paimon.flink.globalindex.GenericIndexTopoBuilder; import org.apache.paimon.flink.globalindex.SortedIndexTopoBuilder; -import org.apache.paimon.globalindex.GlobalIndexer; +import org.apache.paimon.globalindex.GlobalIndexBuilderUtils; import org.apache.paimon.options.Options; import org.apache.paimon.partition.PartitionPredicate; import org.apache.paimon.predicate.Predicate; @@ -117,20 +117,9 @@ public String[] call( Options userOptions = createUserOptions(table, options); indexType = indexType.toLowerCase(Locale.ROOT).trim(); - if (indexColumns.size() > 1) { - // Fail fast before submitting the job: index types that do not support multi-column - // throw from GlobalIndexerFactory#create, which happens before any indexer side effect. - List indexFields = - indexColumns.stream().map(rowType::getField).collect(Collectors.toList()); - try { - GlobalIndexer.create(indexType, indexFields, userOptions); - } catch (UnsupportedOperationException e) { - throw new IllegalArgumentException( - String.format( - "Index type '%s' does not support multi-column index, got columns: %s", - indexType, indexColumns)); - } - } + List indexFields = + indexColumns.stream().map(rowType::getField).collect(Collectors.toList()); + GlobalIndexBuilderUtils.validateIndexFields(indexType, indexFields, userOptions); try { if (SortedIndexTopoBuilder.supports(indexType)) { SortedIndexTopoBuilder.buildIndexAndExecute( diff --git a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/VectorSearchProcedure.java b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/VectorSearchProcedure.java index 5368835437c5..3b799dbc693c 100644 --- a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/VectorSearchProcedure.java +++ b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/VectorSearchProcedure.java @@ -39,16 +39,20 @@ import org.apache.paimon.table.source.ReadBuilder; import org.apache.paimon.table.source.TableScan; import org.apache.paimon.table.source.VectorScan; -import org.apache.paimon.table.source.VectorSearchBuilder; import org.apache.paimon.table.source.VectorSearchBuilderImpl; import org.apache.paimon.table.source.snapshot.TimeTravelUtil; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataTypes; import org.apache.paimon.types.RowType; import org.apache.paimon.utils.InternalRowUtils; +import org.apache.paimon.utils.JsonSerdeUtil; import org.apache.paimon.utils.Pair; import org.apache.paimon.utils.StringUtils; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.DeserializationFeature; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.JsonNode; +import org.apache.paimon.shade.jackson2.com.fasterxml.jackson.databind.node.ArrayNode; + import org.apache.flink.table.annotation.ArgumentHint; import org.apache.flink.table.annotation.DataTypeHint; import org.apache.flink.table.annotation.ProcedureHint; @@ -139,7 +143,13 @@ public String[] call( "Table column '%s' conflicts with vector-search metadata.", SEARCH_SCORE); - float[] queryVector = parseVector(queryVectorStr); + VectorSearchBuilderImpl builder = newVectorSearchBuilder(procedureContext, fileStoreTable); + builder.withVectorColumn(vectorColumn).withLimit(topK).withOptions(optionsMap); + if (queryVectorStr.trim().startsWith("[")) { + builder.withMultiVector(parseQueryMatrix(queryVectorStr)); + } else { + builder.withVector(parseVector(queryVectorStr)); + } Predicate filter = parseFilter(where, tableType); PartitionPredicate partitionFilter = parsePartitions(partitions, fileStoreTable); Projection parsedProjection = Projection.parse(projection, tableType, filter); @@ -147,17 +157,9 @@ public String[] call( validatePrimaryKeyFilter(fileStoreTable, vectorColumn, filterParts); Snapshot snapshot = TimeTravelUtil.tryTravelOrLatest(fileStoreTable); - if (snapshot == null) { - return new String[0]; + if (snapshot != null) { + builder.withSnapshot(snapshot); } - - VectorSearchBuilder builder = - newVectorSearchBuilder(procedureContext, fileStoreTable) - .withSnapshot(snapshot) - .withVector(queryVector) - .withVectorColumn(vectorColumn) - .withLimit(topK) - .withOptions(optionsMap); if (filter != null) { builder.withFilter(filter); } @@ -165,7 +167,11 @@ public String[] call( builder.withPartitionFilter(partitionFilter); } - VectorScan.Plan vectorPlan = builder.newVectorScan().scan(); + VectorScan scan = builder.newVectorScan(); + if (snapshot == null) { + return new String[0]; + } + VectorScan.Plan vectorPlan = scan.scan(); GlobalIndexResult result = builder.newVectorRead().read(vectorPlan); ReadBuilder readBuilder = fileStoreTable.newReadBuilder(); @@ -222,6 +228,32 @@ private static float[] parseVector(String vectorStr) { return vector; } + private static float[][] parseQueryMatrix(String json) { + ArrayNode rows; + try { + rows = + JsonSerdeUtil.OBJECT_MAPPER_INSTANCE + .readerFor(ArrayNode.class) + .with(DeserializationFeature.FAIL_ON_TRAILING_TOKENS) + .readValue(json); + } catch (IOException e) { + throw new IllegalArgumentException("query_vector must be a JSON matrix of numbers", e); + } + float[][] matrix = new float[rows.size()][]; + for (int i = 0; i < matrix.length; i++) { + JsonNode row = rows.get(i); + checkArgument(row.isArray(), "query_vector matrix rows must be arrays"); + matrix[i] = new float[row.size()]; + for (int j = 0; j < row.size(); j++) { + JsonNode coordinate = row.get(j); + checkArgument( + coordinate.isNumber(), "query_vector matrix coordinates must be numbers"); + matrix[i][j] = coordinate.floatValue(); + } + } + return matrix; + } + @Nullable private static Predicate parseFilter(@Nullable String where, RowType tableType) throws Exception { diff --git a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkDataEvolutionVectorRead.java b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkDataEvolutionVectorRead.java index 192bd54002ba..ac89947b92a9 100644 --- a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkDataEvolutionVectorRead.java +++ b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkDataEvolutionVectorRead.java @@ -50,6 +50,7 @@ import java.io.IOException; import java.util.ArrayList; +import java.util.Collection; import java.util.List; import java.util.Map; import java.util.Optional; @@ -96,11 +97,8 @@ public GlobalIndexResult read(VectorScan.Plan plan) { return GlobalIndexResult.createEmpty(); } - GlobalIndexer globalIndexer = - !indexSplits.isEmpty() && !rawSplits.isEmpty() - ? createGlobalIndexer(indexSplits) - : null; - if (!indexSplits.isEmpty() && !rawSplits.isEmpty()) { + Collection> groups = groupByIndexType(indexSplits); + if (groups.size() == 1 && !rawSplits.isEmpty()) { int parallelism = flinkParallelism(); List rawRowRanges = rawRowRanges(rawSplits); if (indexSplits.size() >= parallelism * 2L @@ -109,19 +107,23 @@ && rawRowCount(rawRowRanges) >= parallelism * 2L) { indexSplits, rawSplits, rawRowRanges, - globalIndexer, + createGlobalIndexer(indexSplits), rawPreFilter(rawSplits), parallelism); } } - ScoredGlobalIndexResult indexed = - indexSplits.isEmpty() - ? ScoredGlobalIndexResult.createEmpty() - : readIndexSplitsInFlink(indexSplits, globalIndexer); - ScoredGlobalIndexResult raw = - readRawSplitsInFlink(rawSplits, globalIndexer, rawPreFilter(rawSplits)); - return indexed.or(raw).topK(limit); + List results = new ArrayList<>(); + GlobalIndexer rawIndexer = null; + for (List group : groups) { + GlobalIndexer indexer = createGlobalIndexer(group); + results.add(readIndexSplitsInFlink(group, indexer)); + if (rawIndexer == null) { + rawIndexer = indexer; + } + } + results.add(readRawSplitsInFlink(rawSplits, rawIndexer, rawPreFilter(rawSplits))); + return ScoredGlobalIndexResult.merge(results).topK(limit); } protected ScoredGlobalIndexResult readIndexSplitsInFlink( diff --git a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkVectorSearchBuilderImpl.java b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkVectorSearchBuilderImpl.java index d4bf9748634f..ca9e3bf25255 100644 --- a/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkVectorSearchBuilderImpl.java +++ b/paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/vectorsearch/FlinkVectorSearchBuilderImpl.java @@ -40,7 +40,10 @@ public FlinkVectorSearchBuilderImpl(InnerTable table, StreamExecutionEnvironment @Override public VectorRead newVectorRead() { - rejectUnderQueryAuth(); + if (multiVector != null) { + return super.newVectorRead(); + } + validateSearch(); checkNotNull(vector, "vector must be set via withVector()"); if (isPrimaryKeyVectorSearch()) { return new FlinkPrimaryKeyVectorRead( diff --git a/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/VectorSearchProcedureITCase.java b/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/VectorSearchProcedureITCase.java index a874a5030fc6..827378e194a6 100644 --- a/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/VectorSearchProcedureITCase.java +++ b/paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/procedure/VectorSearchProcedureITCase.java @@ -53,6 +53,8 @@ import org.apache.flink.table.api.config.TableConfigOptions; import org.apache.flink.types.Row; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; import java.util.Arrays; import java.util.Collections; @@ -70,6 +72,97 @@ public class VectorSearchProcedureITCase extends CatalogITCaseBase { private static final String VECTOR_FIELD = "vec"; private static final int DIMENSION = 2; + @ParameterizedTest + @CsvSource({ + "ivf-flat, ivf-flat.nlist=1", + "ivf-pq, ivf-pq.nlist=1;ivf-pq.pq.m=1", + "ivf-sq, ivf-sq.nlist=1", + "ivf-rq, ivf-rq.nlist=1", + "diskann, diskann.pq.m=1" + }) + public void testRangeIndexRanksRowsByMaxSim(String indexType, String indexOptions) { + createMultiVectorTable(); + tEnv.getConfig().set(TableConfigOptions.TABLE_DML_SYNC, true); + // Supply distinct training vectors for the quantized indexes in a low-scoring row. + String trainingVectors = + IntStream.range(0, 256) + .mapToObj(i -> String.format("ARRAY[-1,-1,%d,%d]", i % 17, i % 31)) + .collect(Collectors.joining(",")); + sql( + "INSERT INTO DOCS VALUES (1, ARRAY[ARRAY[10,0,0,0]]), " + + "(2, ARRAY[ARRAY[0,10,0,0]]), (3, ARRAY[ARRAY[3,4,0,0]]), " + + "(4, ARRAY[ARRAY[2,0,0,0], ARRAY[0,3,0,0]]), (5, ARRAY[%s])", + trainingVectors); + String matrix = "[[1,0,0,0],[0,1,0,0]]"; + String projection = "id,__paimon_search_score"; + assertThat(search("DOCS", matrix, 2, projection, null)).isEmpty(); + List expected = + Arrays.asList( + Row.of("{\"id\":\"4\",\"__paimon_search_score\":\"2.0\"}"), + Row.of("{\"id\":\"3\",\"__paimon_search_score\":\"1.4\"}")); + assertThat(search("DOCS", matrix, 2, projection, "vector-index.search-mode=detail")) + .isEqualTo(expected); + + buildMultiVectorIndex(indexType, indexType + ".dimension=4;" + indexOptions); + assertThat(search("DOCS", matrix, 2, projection, "vector-search.distribute.enabled=true")) + .isEqualTo(expected); + } + + @Test + public void testDistributedListSearchWithDifferentIndexTypes() { + createMultiVectorTable(); + tEnv.getConfig().set(TableConfigOptions.TABLE_DML_SYNC, true); + sql("INSERT INTO DOCS VALUES (1, ARRAY[ARRAY[1,0]]), (2, ARRAY[ARRAY[0,1]])"); + for (String type : Arrays.asList("ivf-flat", "ivf-sq")) { + buildMultiVectorIndex( + type, + type + ".dimension=2;" + type + ".nlist=1;global-index.row-count-per-shard=1"); + } + assertThat(sql("SELECT index_type, COUNT(*) FROM `DOCS$table_indexes` GROUP BY index_type")) + .containsExactlyInAnyOrder(Row.of("ivf-flat", 2L), Row.of("ivf-sq", 2L)); + assertThat( + search( + "DOCS", + "1,0", + 1, + "id", + "vector-search.distribute.enabled=true;global-index.thread-num=1")) + .containsExactly(Row.of("{\"id\":\"1\"}")); + } + + @Test + public void testMultiVectorIndexRejectsNonCosineMetricOnEmptyTable() { + createMultiVectorTable(); + for (String metric : Arrays.asList("l2", "inner_product")) { + assertThatThrownBy(() -> buildMultiVectorIndex("ivf-flat", "ivf-flat.metric=" + metric)) + .hasStackTraceContaining( + "require cosine similarity, got metric '" + metric + "'"); + } + } + + @Test + public void testRejectsInvalidMaxSimQueryMatrix() { + createMultiVectorTable(); + assertThatThrownBy(() -> search("DOCS", "[[1,null]]", 1, "id", null)) + .hasStackTraceContaining("matrix coordinates must be numbers"); + assertThatThrownBy(() -> search("DOCS", "[[1,0],[1]]", 1, "id", null)) + .hasStackTraceContaining("dimensions must match"); + } + + private void createMultiVectorTable() { + sql( + "CREATE TABLE DOCS (id INT, vec ARRAY>) WITH (" + + "'bucket'='-1', 'row-tracking.enabled'='true', 'data-evolution.enabled'='true', " + + "'vector-index.search-mode'='fast')"); + } + + private void buildMultiVectorIndex(String indexType, String options) { + sql( + "CALL sys.create_global_index(`table` => 'default.DOCS', index_column => 'vec', " + + "index_type => '%s', options => '%s')", + indexType, options); + } + @Test public void testPrimaryKeyVectorSearch() throws Exception { createPrimaryKeyVectorTable("PK_T"); @@ -864,6 +957,8 @@ public void testVectorSearchValidation() { .hasStackTraceContaining("top_k must be positive"); assertThatThrownBy(() -> search("VALIDATION_T", "NaN,0.0", 1, "id", null)) .hasStackTraceContaining("query_vector values must be finite"); + assertThatThrownBy(() -> search("VALIDATION_T", "[[1,0]]", 1, "id", null)) + .hasStackTraceContaining("query matrix requires a multivector column"); assertThatThrownBy(() -> search("VALIDATION_T", "0.0,0.0", 1, "missing", null)) .hasStackTraceContaining("Unknown projection column"); assertThatThrownBy(() -> search("VALIDATION_T", "0.0,0.0", 1, "id,id", null)) diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java index 1d8e8b6fcd25..fb67f24cf551 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java @@ -18,6 +18,7 @@ package org.apache.paimon.spark.procedure; +import org.apache.paimon.globalindex.GlobalIndexBuilderUtils; import org.apache.paimon.globalindex.GlobalIndexer; import org.apache.paimon.options.Options; import org.apache.paimon.partition.PartitionPredicate; @@ -171,19 +172,8 @@ public InternalRow[] call(InternalRow args) { Options userOptions = createUserOptions(table, optionString); - if (indexColumns.size() > 1) { - // Fail fast before submitting the job: index types that do not support - // multi-column throw from GlobalIndexerFactory#create, which happens - // before any indexer side effect. - try { - GlobalIndexer.create(indexType, indexFields, userOptions); - } catch (UnsupportedOperationException e) { - throw new IllegalArgumentException( - String.format( - "Index type '%s' does not support multi-column index, got columns: %s", - indexType, indexColumns)); - } - } + GlobalIndexBuilderUtils.validateIndexFields( + indexType, indexFields, userOptions); GlobalIndexTopologyBuilder topoBuilder = GlobalIndexTopologyBuilderUtils.createTopoBuilder( diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkDataEvolutionVectorRead.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkDataEvolutionVectorRead.java index 7037f19bcac3..8d47dcab9509 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkDataEvolutionVectorRead.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkDataEvolutionVectorRead.java @@ -38,7 +38,6 @@ import java.io.IOException; import java.util.ArrayList; -import java.util.Collections; import java.util.List; import java.util.Map; import java.util.Optional; @@ -63,7 +62,19 @@ public SparkDataEvolutionVectorRead( DataField vectorColumn, float[] vector, @Nullable Map options) { - super(table, partitionFilter, filter, limit, vectorColumn, vector, options); + this(table, partitionFilter, filter, limit, vectorColumn, vector, options, null); + } + + public SparkDataEvolutionVectorRead( + FileStoreTable table, + @Nullable PartitionPredicate partitionFilter, + @Nullable Predicate filter, + int limit, + DataField vectorColumn, + float[] vector, + @Nullable Map options, + @Nullable float[][] queryMatrix) { + super(table, partitionFilter, filter, limit, vectorColumn, vector, options, queryMatrix); } @Override @@ -99,11 +110,7 @@ protected ScoredGlobalIndexResult readIndexed( List> splitGroups = splitGroups(serializedSplits, parallelism); SerializableFunction, byte[]> task = group -> { - GlobalIndexer taskGlobalIndexer = - GlobalIndexer.create( - indexType, - Collections.singletonList(vectorColumn), - table.coreOptions().toConfiguration()); + GlobalIndexer taskGlobalIndexer = createGlobalIndexer(indexType); IndexPathFactory indexPathFactory = table.store().pathFactory().globalIndexFileFactory(); @@ -134,8 +141,12 @@ protected ScoredGlobalIndexResult readIndexed( results.add(next.get()); } } + ScoredGlobalIndexResult merged = mergeIndexedCandidates(results); ScoredGlobalIndexResult result = - ScoredGlobalIndexResult.merge(results).topK(searchLimit); + queryMatrix == null + ? merged.topK(searchLimit) + : maybeRerankIndexedResult( + merged, indexType, taskGlobalIndexer, vector); if (result.results().isEmpty()) { return null; } @@ -149,6 +160,9 @@ protected ScoredGlobalIndexResult readIndexed( List remoteResults = mapInSpark(splitGroups, task, splitGroups.size()); + if (queryMatrix != null) { + return mergeRemoteResults(remoteResults); + } return maybeRerankIndexedResult( mergeRemoteResults(remoteResults, searchLimit), indexType, globalIndexer, vector); } diff --git a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkVectorSearchBuilderImpl.java b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkVectorSearchBuilderImpl.java index 3c8eedc3a580..b4faeec1eec5 100644 --- a/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkVectorSearchBuilderImpl.java +++ b/paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/read/SparkVectorSearchBuilderImpl.java @@ -38,12 +38,12 @@ public SparkVectorSearchBuilderImpl(InnerTable table) { @Override public VectorRead newVectorRead() { - rejectUnderQueryAuth(); + validateSearch(); if (isPrimaryKeyVectorSearch()) { return new SparkPrimaryKeyVectorRead( table, vectorColumn, vector, limit, options, filter); } return new SparkDataEvolutionVectorRead( - table, partitionFilter, filter, limit, vectorColumn, vector, options); + table, partitionFilter, filter, limit, vectorColumn, vector, options, multiVector); } } diff --git a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala index ad277d5b09fb..7a7a9ba8f192 100644 --- a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala +++ b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonBaseScan.scala @@ -220,10 +220,14 @@ object PaimonBaseScan { table.newVectorSearchBuilder() } val vectorBuilder = vectorSearchBuilder - .withVector(vectorSearch.vector()) .withVectorColumn(vectorSearch.fieldName()) .withLimit(vectorSearch.limit()) .withOptions(vectorSearch.options()) + if (vectorSearch.multiVector() == null) { + vectorBuilder.withVector(vectorSearch.vector()) + } else { + vectorBuilder.withMultiVector(vectorSearch.multiVector()) + } if (pushedPartitionFilters.nonEmpty) { vectorBuilder.withPartitionFilter(PartitionPredicate.and(pushedPartitionFilters.asJava)) } diff --git a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala index 9176ca6720aa..6b53cd099dec 100644 --- a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala +++ b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/catalyst/plans/logical/PaimonTableValuedFunctions.scala @@ -35,9 +35,10 @@ import org.apache.spark.sql.catalyst.analysis.TableFunctionRegistry.TableFunctio import org.apache.spark.sql.catalyst.expressions.{Alias, And, Attribute, AttributeSet, CreateArray, CreateMap, CreateNamedStruct, Expression, ExpressionInfo, ExprId, Literal, OuterReference} import org.apache.spark.sql.catalyst.plans.{Inner, JoinType} import org.apache.spark.sql.catalyst.plans.logical.{Filter, LeafNode, LogicalPlan, Project, SubqueryAlias, UnaryNode} -import org.apache.spark.sql.catalyst.util.MapData +import org.apache.spark.sql.catalyst.util.{ArrayData, MapData} import org.apache.spark.sql.connector.catalog.{Identifier, Table, TableCatalog} import org.apache.spark.sql.paimon.shims.SparkShimLoader +import org.apache.spark.sql.types.{ArrayType, DataType, DoubleType, FloatType} import org.apache.spark.sql.util.CaseInsensitiveStringMap import org.apache.spark.unsafe.types.UTF8String @@ -47,6 +48,17 @@ import scala.util.control.NonFatal object PaimonTableValuedFunctions { + private[spark] def readQueryVector(values: ArrayData, elementType: DataType): Array[Float] = { + require( + elementType == FloatType || elementType == DoubleType, + "Query vectors require FLOAT or DOUBLE coordinates.") + Array.tabulate(values.numElements()) { + i => + require(!values.isNullAt(i), "Query coordinates cannot be null") + if (elementType == FloatType) values.getFloat(i) else values.getDouble(i).toFloat + } + } + private case class DynamicVectorSearchExtraction( relation: DynamicVectorSearchRelation, projectList: Seq[Expression], @@ -473,7 +485,7 @@ case class IncrementalToAutoTag(override val args: Seq[Expression]) * Usage: vector_search(table_name, column_name, query_vector, limit[, options]) * - table_name: the Paimon table to search * - column_name: the vector column name - * - query_vector: array of floats representing the query vector + * - query_vector: a single vector, or a matrix of query vectors for a multivector column * - limit: the number of top results to return * - options: optional options as a map or semicolon-separated key-value string * @@ -503,7 +515,7 @@ case class VectorSearchQuery(override val args: Seq[Expression]) s"Column $columnName does not exist in table ${innerTable.name()}" ) } - val queryVector = extractQueryVector(argsWithoutTable(1)) + val query = argsWithoutTable(1) val limit = parsePositiveLimit(argsWithoutTable(2).eval()) val options: Map[String, String] = if (argsWithoutTable.size == 4) { @@ -511,14 +523,27 @@ case class VectorSearchQuery(override val args: Seq[Expression]) } else { Map.empty[String, String] } - new VectorSearch(queryVector, limit, columnName, options.asJava) + query.dataType match { + case ArrayType(ArrayType(elementType, _), _) + if elementType == FloatType || elementType == DoubleType => + val rows = query.eval().asInstanceOf[ArrayData] + require(rows != null, "Query matrix cannot be null") + val vectors = Array.tabulate(rows.numElements()) { + i => + require(!rows.isNullAt(i), "Query vectors cannot be null") + readQueryVector(rows.getArray(i), elementType) + } + VectorSearch.multiVector(vectors, limit, columnName, options.asJava) + case ArrayType(_: ArrayType, _) => + throw new IllegalArgumentException("A query matrix requires FLOAT or DOUBLE coordinates.") + case _ => new VectorSearch(extractQueryVector(query), limit, columnName, options.asJava) + } } def extractQueryVector(expr: Expression): Array[Float] = { expr match { - case Literal(arrayData, _) if arrayData != null => - val arr = arrayData.asInstanceOf[org.apache.spark.sql.catalyst.util.ArrayData] - arr.toFloatArray() + case Literal(values: ArrayData, ArrayType(elementType, _)) => + readQueryVector(values, elementType) case CreateArray(elements, _) if elements != null => elements.map { case Literal(v: Float, _) => v @@ -629,10 +654,20 @@ case class VectorSearchQuery(override val args: Seq[Expression]) } else { Map.empty[String, String] } + val query = argsWithoutTable(1) + query.dataType match { + case ArrayType(FloatType | DoubleType, _) => + case ArrayType(_: ArrayType, _) => + throw new UnsupportedOperationException( + "LATERAL vector_search requires a single query vector per outer row.") + case _ => + throw new IllegalArgumentException( + "LATERAL vector_search requires an array of FLOAT or DOUBLE.") + } DynamicVectorSearchRelation( innerTable, columnName, - argsWithoutTable(1), + query, limit, options, toAttributes(SparkTypeUtils.fromPaimonRowType(innerTable.rowType()))) diff --git a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala index 0717322646cc..d045281c5acb 100644 --- a/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala +++ b/paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/execution/PaimonStrategy.scala @@ -57,6 +57,7 @@ import org.apache.spark.sql.execution.exchange.ShuffleExchangeLike import org.apache.spark.sql.execution.joins.{BroadcastHashJoinExec, BroadcastNestedLoopJoinExec} import org.apache.spark.sql.execution.shim.{PaimonCreateTableAsSelectStrategy, PaimonReplaceTableAsSelectStrategy, PaimonReplaceTableStrategy} import org.apache.spark.sql.paimon.shims.SparkShimLoader +import org.apache.spark.sql.types.ArrayType import scala.collection.JavaConverters._ import scala.collection.mutable.ArrayBuffer @@ -808,7 +809,9 @@ case class LateralVectorSearchExec( private def toFloatArray(value: Any): Option[Array[Float]] = { value match { case null => None - case arrayData: ArrayData => Some(arrayData.toFloatArray()) + case arrayData: ArrayData => + val elementType = queryVectorExpr.dataType.asInstanceOf[ArrayType].elementType + Some(PaimonTableValuedFunctions.readQueryVector(arrayData, elementType)) case _ => throw new RuntimeException(s"Cannot extract query vector from expression value: $value") } diff --git a/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/MultiVectorSearchTest.scala b/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/MultiVectorSearchTest.scala new file mode 100644 index 000000000000..ff99b98e9988 --- /dev/null +++ b/paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/MultiVectorSearchTest.scala @@ -0,0 +1,270 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.paimon.spark.sql + +import org.apache.paimon.catalog.Identifier +import org.apache.paimon.globalindex.ScoredGlobalIndexResult +import org.apache.paimon.predicate.PredicateBuilder +import org.apache.paimon.schema.Schema +import org.apache.paimon.spark.PaimonSparkTestBase +import org.apache.paimon.types.DataTypes + +/** End-to-end tests for row-ranked multivector search. */ +class MultiVectorSearchTest extends PaimonSparkTestBase { + + test("reject an explicit non-cosine metric when creating a list index on an empty table") { + withTable("T") { + spark.sql("""CREATE TABLE T (chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + |'data-evolution.enabled'='true')""".stripMargin) + val error = intercept[RuntimeException] { + spark + .sql( + """CALL sys.create_global_index(table => 'test.T', index_column => 'chunks', + |index_type => 'ivf-flat', options => 'ivf-flat.metric=inner_product')""".stripMargin) + .collect() + } + assert(error.getCause.isInstanceOf[IllegalArgumentException]) + assert( + error.getCause.getMessage.contains("require cosine similarity, got metric 'inner_product'")) + } + } + + test("skip a range with no embeddings when building a list index") { + withTable("T") { + spark.sql( + """CREATE TABLE T (chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + |'data-evolution.enabled'='true', 'vector-index.search-mode'='detail')""".stripMargin) + spark.sql("""INSERT INTO T VALUES (NULL), (array()), (array(CAST(NULL AS ARRAY)))""") + spark + .sql("""CALL sys.create_global_index(table => 'test.T', index_column => 'chunks', + |index_type => 'ivf-flat', options => 'ivf-flat.dimension=2')""".stripMargin) + .collect() + assert( + spark + .sql("SELECT * FROM `T$table_indexes` WHERE index_type = 'ivf-flat'") + .collect() + .isEmpty) + val query = "SELECT * FROM vector_search('T', 'chunks', array(array(1f, 0f)), 1)" + assert(spark.sql(query).collect().isEmpty) + spark.sql("ALTER TABLE T SET TBLPROPERTIES ('vector-index.search-mode'='fast')") + assert(spark.sql(query).collect().isEmpty) + } + } + + test("reject matrix queries with unsupported SQL shapes during analysis") { + withTable("T") { + spark.sql("""CREATE TABLE T (id INT, chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + |'data-evolution.enabled'='true')""".stripMargin) + val correlated = intercept[UnsupportedOperationException] { + spark + .sql("""SELECT r.id FROM T AS q, LATERAL ( + | SELECT id FROM vector_search('T', 'chunks', q.chunks, 1) + |) AS r""".stripMargin) + .queryExecution + .analyzed + } + assert(correlated.getMessage.contains("single query vector per outer row")) + val integers = intercept[IllegalArgumentException] { + spark + .sql("SELECT id FROM vector_search('T', 'chunks', array(array(1, 2)), 1)") + .queryExecution + .analyzed + } + assert(integers.getMessage.contains("FLOAT or DOUBLE coordinates")) + } + } + + Seq("ivf-flat", "ivf-pq", "ivf-sq", "ivf-rq", "diskann").foreach { + kind => + test(s"$kind range index ranks rows by MaxSim in Spark SQL") { + withTable("T") { + spark.sql("""CREATE TABLE T (id INT, chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + |'data-evolution.enabled'='true', 'vector-index.search-mode'='fast') + |""".stripMargin) + // Rows 1 and 2 win the individual queries. Row 4 wins their combined score. + // Row 5 supplies distinct training vectors for the quantized indexes. + spark.sql("""INSERT INTO T SELECT cast(id AS INT), CASE id + |WHEN 1 THEN array_repeat(array(1f, 0f, 0f, 0f), 32) + |WHEN 2 THEN array(array(0f, 1f, 0f, 0f)) + |WHEN 3 THEN array(array(1f, 1f, 0f, 0f)) + |WHEN 4 THEN array(array(3f, 0f, 1f, 0f), array(0f, 4f, 1f, 0f)) + |ELSE transform(sequence(1, 256), i -> + | array(-1f, -1f, cast(i % 17 AS FLOAT), cast(i % 31 AS FLOAT))) + |END FROM range(1, 6, 1, 1)""".stripMargin) + val query = """SELECT id, __paimon_search_score FROM vector_search('T', 'chunks', + | array(array(1f, 0f, 0f, 0f), array(0f, 1f, 0f, 0f)), 2) + |ORDER BY __paimon_search_score DESC, id""".stripMargin + assert(spark.sql(query).collect().isEmpty) + + val options = Seq(s"$kind.sharding=range", s"$kind.dimension=4") ++ (kind match { + case "diskann" => Seq("diskann.pq.m=1") + case "ivf-pq" => Seq("ivf-pq.nlist=1", "ivf-pq.pq.m=1") + case _ => Seq(s"$kind.nlist=1") + }) + spark + .sql(s"""CALL sys.create_global_index(table => 'test.T', index_column => 'chunks', + |index_type => '$kind', options => '${options.mkString(",")}')""".stripMargin) + .collect() + val rows = spark.sql(query).collect() + assert(rows.map(_.getInt(0)).toSeq == Seq(4, 3)) + assert(math.abs(rows(0).getFloat(1) - (3 / math.sqrt(10) + 4 / math.sqrt(17))) < 1e-5) + assert(math.abs(rows(1).getFloat(1) - math.sqrt(2)) < 1e-5) + } + } + } + + test("rank complete lists by MaxSim across indexed and uncovered rows") { + withTable("T") { + spark.sql(""" + |CREATE TABLE T (id INT, chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + | 'deletion-vectors.enabled'='true', + | 'data-evolution.enabled'='true', 'vector-index.search-mode'='detail') + |""".stripMargin) + spark.sql(""" + |INSERT INTO T VALUES + |(1, array_repeat(array(1f, 0f), 32)), (2, array(array(0f, 1f))), + |(3, array(null, array(1f, 1f))), (4, array()), (5, null) + |""".stripMargin) + val matrix = "array(array(1f, 0f), array(0f, 1f))" + def query(q: String, k: Int): Seq[(Int, Float)] = + spark + .sql(s"""SELECT id, __paimon_search_score FROM vector_search('T', 'chunks', $q, $k) + |ORDER BY __paimon_search_score DESC, id""".stripMargin) + .collect() + .map(row => (row.getInt(0), row.getFloat(1))) + .toSeq + + val exact = query(matrix, 3) + assert(exact.map(_._1) == Seq(3, 1, 2)) + assert(math.abs(exact.head._2 - math.sqrt(2)) < 1e-5) + spark + .sql("""CALL sys.create_global_index(table => 'test.T', index_column => 'chunks', + |index_type => 'ivf-flat', options => 'ivf-flat.dimension=2,ivf-flat.nlist=1, + |ivf-flat.distance.metric=cosine')""".stripMargin.replace("\n", "")) + .collect() + assert(query(matrix, 1) == exact.take(1)) + assert(query("array(1f, 0f)", 3).map(_._1) == Seq(1, 3, 2)) + val table = loadTable("T") + val qs = Array(Array(1f, 0f), Array(0f, 1f)) + val predicate = new PredicateBuilder(table.rowType()).equal(0, Int.box(2)) + val filtered = table + .newVectorSearchBuilder() + .withVectorColumn("chunks") + .withMultiVector(qs) + .withLimit(1) + .withFilter(predicate) + .executeLocal() + .asInstanceOf[ScoredGlobalIndexResult] + assert(filtered.results().getLongCardinality == 1) + assert(filtered.scoreGetter().score(filtered.results().toArray(1).head) == 1f) + val batch = table + .newBatchVectorSearchBuilder() + .withVectorColumn("chunks") + .withVectors(qs) + .withLimit(1) + .withOption("ivf.refine_factor", "2") + .executeBatchLocal() + assert(batch.size() == 2) + assert(batch.get(0).results().toArray(1).head != batch.get(1).results().toArray(1).head) + + spark.sql("INSERT INTO T VALUES (6, array(array(1f, 0f), array(0f, 1f)))") + assert(query(matrix, 2) == Seq((6, 2f), exact.head)) + spark.sql("DELETE FROM T WHERE id = 3") + assert(query(matrix, 3).map(_._1) == Seq(6, 1, 2)) + } + } + + test("lateral search decodes DOUBLE and FLOAT query vectors by their element type") { + withTable("T", "Q") { + spark.sql( + """CREATE TABLE T (id INT, chunks ARRAY>) + |TBLPROPERTIES ('bucket'='-1', 'row-tracking.enabled'='true', + |'data-evolution.enabled'='true', 'vector-index.search-mode'='detail')""".stripMargin) + spark.sql("INSERT INTO T VALUES (1, array(array(1f, 0f))), (2, array(array(0f, 1f)))") + spark.sql("CREATE TABLE Q (id INT, v ARRAY)") + spark.sql("INSERT INTO Q VALUES (1, array(1D, 0D)), (2, array(0D, 1D))") + def search(expr: String) = + s"""SELECT q.id, r.id FROM Q AS q, LATERAL ( + | SELECT id FROM vector_search('T', 'chunks', $expr, 1)) AS r ORDER BY q.id""".stripMargin + Seq("q.v", "CAST(q.v AS ARRAY)").foreach { + expr => + assert( + spark.sql(search(expr)).collect().map(r => (r.getInt(0), r.getInt(1))).toSeq == + Seq((1, 1), (2, 2))) + } + val error = intercept[IllegalArgumentException] { + spark.sql(search("CAST(q.v AS ARRAY)")).queryExecution.analyzed + } + assert(error.getMessage.contains("FLOAT or DOUBLE")) + spark.sql("INSERT INTO Q VALUES (3, array(CAST(NULL AS DOUBLE), 1D))") + val nullCoordinate = intercept[Exception] { + spark.sql(search("q.v")).collect() + } + assert( + Iterator + .iterate[Throwable](nullCoordinate)(_.getCause) + .takeWhile(_ != null) + .exists(e => Option(e.getMessage).exists(_.contains("Query coordinates cannot be null")))) + } + } + + test("two indexed list columns select their own dimensions in distributed search") { + withTable("docs") { + val schema = Schema + .newBuilder() + .column("id", DataTypes.INT()) + .column("chunks", DataTypes.ARRAY(DataTypes.ARRAY(DataTypes.FLOAT()))) + .column("images", DataTypes.ARRAY(DataTypes.VECTOR(3, DataTypes.FLOAT()))) + .option("bucket", "-1") + .option("data-evolution.enabled", "true") + .option("row-tracking.enabled", "true") + .option("ivf-flat.metric", "l2") + .option("ivf-flat.sharding", "centroid") + .option("global-index.row-count-per-shard", "1") + .option("global-index.thread-num", "1") + .option("vector-search.distribute.enabled", "true") + .build() + paimonCatalog.createTable(Identifier.create(dbName0, "docs"), schema, false) + spark.sql("""INSERT INTO docs VALUES + |(1, array(array(1f, 0f)), array(array(0f, 0f, 1f))), + |(2, array(array(0f, 1f)), array(array(1f, 0f, 0f)))""".stripMargin) + Seq("chunks" -> "ivf-flat.dimension=2,", "images" -> "").foreach { + case (column, dimension) => + spark + .sql(s"""CALL sys.create_global_index(table => 'test.docs', index_column => '$column', + |index_type => 'ivf-flat', options => '${dimension}ivf-flat.nlist=1,ivf-flat.metric=cosine,ivf-flat.sharding=range') + |""".stripMargin) + .collect() + } + def winner(column: String, q: String): Int = + spark + .sql(s"SELECT id FROM vector_search('docs', '$column', array($q, $q), 1)") + .collect() + .head + .getInt(0) + assert(winner("chunks", "array(1f, 0f)") == 1) + assert(winner("images", "array(1f, 0f, 0f)") == 2) + } + } +} diff --git a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java index 2929bd07649b..e00bddbe8def 100644 --- a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java +++ b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexReader.java @@ -25,6 +25,7 @@ import org.apache.paimon.globalindex.AbstractVectorGlobalIndexReader; import org.apache.paimon.globalindex.GlobalIndexIOMeta; import org.apache.paimon.globalindex.ScoredGlobalIndexResult; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.globalindex.io.GlobalIndexFileReader; import org.apache.paimon.index.vector.VectorIndexInput; import org.apache.paimon.index.vector.VectorIndexMetadata; @@ -39,6 +40,7 @@ import org.apache.paimon.types.FloatType; import org.apache.paimon.types.VectorType; import org.apache.paimon.utils.IOUtils; +import org.apache.paimon.utils.Range; import org.apache.paimon.utils.RoaringNavigableMap64; import java.io.IOException; @@ -79,17 +81,19 @@ public class NativeVectorGlobalIndexReader extends AbstractVectorGlobalIndexRead private final DataType fieldType; private final ExecutorService executor; private final String expectedIndexType; + private final boolean multiVector; + private final long totalRowCount; private volatile VectorIndexMetadata nativeMeta; private volatile VectorIndexReader vectorReader; - private SeekableInputStream openStream; + private volatile SeekableInputStream openStream; public NativeVectorGlobalIndexReader( GlobalIndexFileReader fileReader, List ioMetas, DataType fieldType, ExecutorService executor) { - this(fileReader, ioMetas, fieldType, executor, null); + this(fileReader, ioMetas, fieldType, executor, null, -1); } NativeVectorGlobalIndexReader( @@ -98,12 +102,24 @@ public NativeVectorGlobalIndexReader( DataType fieldType, ExecutorService executor, String expectedIndexType) { + this(fileReader, ioMetas, fieldType, executor, expectedIndexType, -1); + } + + NativeVectorGlobalIndexReader( + GlobalIndexFileReader fileReader, + List ioMetas, + DataType fieldType, + ExecutorService executor, + String expectedIndexType, + long totalRowCount) { checkArgument(ioMetas.size() == 1, "Expected exactly one index file per shard"); this.executor = executor; this.fileReader = fileReader; this.ioMeta = ioMetas.get(0); this.fieldType = fieldType; this.expectedIndexType = expectedIndexType; + this.multiVector = VectorIndexUtils.isMultiVector(fieldType); + this.totalRowCount = totalRowCount; } boolean hasJointLayout() throws IOException { @@ -123,8 +139,9 @@ public CompletableFuture> visitVectorSearch( private Optional searchShard(VectorSearch query) { try { ensureLoaded(); + SearchScope scope = SearchScope.of(query.includeRowIds(), query.limit()); return Optional.ofNullable( - search(query, false, SearchScope.of(query.includeRowIds(), query.limit()))); + multiVector ? searchRows(query, scope) : search(query, false, scope)); } catch (IOException e) { throw new RuntimeException( String.format( @@ -155,6 +172,14 @@ List> searchShardBatch( } try { ensureLoaded(); + if (multiVector) { + List> results = + new ArrayList<>(batch.vectorCount()); + for (int i = 0; i < batch.vectorCount(); i++) { + results.add(Optional.ofNullable(searchRows(batch.forIndex(i), scope))); + } + return results; + } return searchBatch(batch, joint, scope); } catch (IOException e) { throw new RuntimeException( @@ -244,6 +269,72 @@ private ScoredGlobalIndexResult search( : found.orElse(null); } + private ScoredGlobalIndexResult searchRows(VectorSearch query, SearchScope scope) + throws IOException { + validateSearchVector(query.vector()); + + RoaringNavigableMap64 remaining = query.includeRowIds(); + long vectors = nativeMeta.totalVectors(); + int requested = (int) Math.min(scope.effectiveK, vectors); + if (requested == 0) { + return null; + } + float[] normalized = new float[query.vector().length]; + NativeVectorGlobalIndexer.normalizeCosine(query.vector(), normalized); + byte[] filter = scope.filterBytes; + Map scores = new HashMap<>(); + RoaringNavigableMap64 rows = new RoaringNavigableMap64(); + while (requested > 0) { + VectorSearchParams params = searchParams(query.options(), requested); + VectorSearchResult found = + filter == null + ? vectorReader.search(normalized, params) + : vectorReader.search(normalized, params, filter); + long[] ids = found.ids(); + float[] distances = found.distances(); + int hits = 0; + for (int i = 0; i < ids.length; i++) { + if (ids[i] >= 0) { + rows.add(ids[i]); + scores.merge(ids[i], 1.0f - distances[i], Math::max); + hits++; + } + } + if (scores.size() >= query.limit() || hits < requested || hits == vectors) { + break; + } + // A full Roaring range needs a container per 65536 row IDs. For an unknown range + // or one with fewer vectors than containers, widen the unfiltered search instead. + if (remaining == null && (totalRowCount < 0 || (totalRowCount >>> 16) > vectors)) { + int next = (int) Math.min(Math.min(vectors, Integer.MAX_VALUE), 2L * requested); + if (next == requested) { + break; + } + requested = next; + continue; + } + // Excluding a matching row skips all its vectors in the next search. + if (remaining == query.includeRowIds()) { + remaining = new RoaringNavigableMap64(); + if (query.includeRowIds() == null) { + remaining.addRange(new Range(0, totalRowCount - 1)); + } else { + remaining.or(query.includeRowIds()); + } + } + remaining.andNot(rows); + requested = + (int) Math.min(query.limit() - scores.size(), remaining.getLongCardinality()); + if (requested > 0) { + filter = remaining.serialize(); + } + } + if (scores.isEmpty()) { + return null; + } + return ScoredGlobalIndexResult.create(rows, scores::get).topK(query.limit()); + } + private static boolean incomplete(Optional found, int limit) { return !found.isPresent() || found.get().results().getLongCardinality() < limit; } @@ -413,30 +504,36 @@ private void validateSearchVector(Object vector) { } else if (fieldType instanceof ArrayType) { validFieldType = ((ArrayType) fieldType).getElementType() instanceof FloatType; } - if (!validFieldType) { + if (!validFieldType && !multiVector) { throw new IllegalArgumentException( "Vector index requires VectorType or ArrayType, but field type is: " + fieldType); } int queryDim = ((float[]) vector).length; - if (queryDim != nativeMeta.dimension()) { + int dimension = nativeMeta.dimension(); + if (multiVector) { + VectorIndexUtils.validateQuery(new float[][] {(float[]) vector}); + } + if (queryDim != dimension) { throw new IllegalArgumentException( String.format( "Query vector dimension mismatch: index expects %d, but got %d", - nativeMeta.dimension(), queryDim)); + dimension, queryDim)); } } private void ensureLoaded() throws IOException { - if (vectorReader == null) { + if (openStream == null) { synchronized (this) { - if (vectorReader == null) { + if (openStream == null) { SeekableInputStream in = fileReader.getInputStream(ioMeta); VectorIndexReader reader = null; try { - VectorIndexInput input = new SeekableStreamVectorIndexInput(in); - reader = new VectorIndexReader(input); + reader = new VectorIndexReader(new SeekableStreamVectorIndexInput(in)); nativeMeta = reader.metadata(); + if (multiVector && !"cosine".equals(nativeMeta.metric())) { + throw new IOException("Multivector index requires cosine similarity."); + } if (expectedIndexType != null && !expectedIndexType.equals( nativeMeta.indexType().replace('_', '-'))) { diff --git a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java index 6531054af0ac..ef160121702d 100644 --- a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java +++ b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexWriter.java @@ -25,6 +25,7 @@ import org.apache.paimon.fs.PositionOutputStreamWrapper; import org.apache.paimon.globalindex.GlobalIndexSingleColumnWriter; import org.apache.paimon.globalindex.ResultEntry; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.globalindex.io.GlobalIndexFileWriter; import org.apache.paimon.index.vector.CentroidRouter; import org.apache.paimon.index.vector.VectorIndexMetadata; @@ -82,6 +83,7 @@ public class NativeVectorGlobalIndexWriter implements GlobalIndexSingleColumnWri private final int dim; private final double trainSampleRatio; private final boolean centroidShard; + private final InternalArray.ElementGetter childGetter; private File tempVectorFile; private FileChannel writeChannel; @@ -127,8 +129,21 @@ public NativeVectorGlobalIndexWriter( this.centroidShard = centroidShard; this.fileWriter = fileWriter; this.identifier = identifier; - validateFieldType(fieldType); + boolean multiVector = VectorIndexUtils.isMultiVector(fieldType); + if (multiVector) { + NativeVectorGlobalIndexerFactory.validateMultiVectorOptions( + centroidShard, options.get("metric")); + } + DataType childType = multiVector ? ((ArrayType) fieldType).getElementType() : fieldType; + childGetter = multiVector ? InternalArray.createElementGetter(childType) : null; + validateFieldType(childType); this.nativeOptions = options; + if (multiVector && !options.containsKey("dimension")) { + throw new IllegalArgumentException( + "An ARRAY> index requires an explicit " + + identifier + + ".dimension."); + } this.dim = Integer.parseInt(options.get("dimension")); if (Double.isNaN(trainSampleRatio) || Double.isInfinite(trainSampleRatio) @@ -192,15 +207,38 @@ public void write(Object fieldData, long rowId) { rowCount++; return; } + if (childGetter != null) { + InternalArray children = (InternalArray) fieldData; + try { + for (int i = 0; i < children.size(); i++) { + if (!children.isNullAt(i)) { + writeVector(childGetter.getElementOrNull(children, i), rowId); + } + } + } catch (RuntimeException e) { + // A partially written list cannot be published. + close(); + throw e; + } + } else { + writeVector(fieldData, rowId); + } + rowCount++; + } - if (centroidShard && rowId < 0) { + private void writeVector(Object fieldData, long rowId) { + if ((centroidShard || childGetter != null) && rowId < 0) { throw new IllegalArgumentException( - "Centroid shards require nonnegative relative row IDs."); + "Vector index labels require nonnegative relative row IDs."); } if (spillBuffer == null) { openSpillFile(); } float[] src = materializeAndValidate(fieldData, rowId); + if (childGetter != null) { + NativeVectorGlobalIndexer.normalizeCosine(src, vectorBuf); + src = vectorBuf; + } if (spillBuffer.remaining() < recordSizeInBytes) { flushWriteBuffer(); @@ -215,7 +253,6 @@ public void write(Object fieldData, long rowId) { for (int i = 0; i < dim; i++) { spillBuffer.putFloat(src[i]); } - rowCount++; count++; } diff --git a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java index 3833a1b4af51..fbae543edc3c 100644 --- a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java +++ b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexer.java @@ -72,6 +72,18 @@ public NativeVectorGlobalIndexer( this.trainSampleRatio = trainSampleRatio; } + /** Normalize nonzero vectors before native cosine computes their squared norm in float. */ + static void normalizeCosine(float[] source, float[] target) { + double squared = 0; + for (float coordinate : source) { + squared += (double) coordinate * coordinate; + } + double scale = squared == 0 ? 1 : 1 / Math.sqrt(squared); + for (int i = 0; i < source.length; i++) { + target[i] = (float) (source[i] * scale); + } + } + @Override public GlobalIndexWriter createWriter(GlobalIndexFileWriter fileWriter) { return new NativeVectorGlobalIndexWriter( @@ -90,7 +102,7 @@ public GlobalIndexReader createReader( fileReader, files, fieldType, executor, identifier); } return new NativeVectorGlobalIndexReader( - fileReader, files, fieldType, executor, identifier); + fileReader, files, fieldType, executor, identifier, totalRowCount); } @Override diff --git a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java index b04c5def5bc1..f4812565b767 100644 --- a/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java +++ b/paimon-vector/src/main/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactory.java @@ -21,7 +21,9 @@ import org.apache.paimon.globalindex.CentroidVectorGlobalIndexer; import org.apache.paimon.globalindex.GlobalIndexer; import org.apache.paimon.globalindex.GlobalIndexerFactory; +import org.apache.paimon.globalindex.VectorIndexUtils; import org.apache.paimon.options.Options; +import org.apache.paimon.types.ArrayType; import org.apache.paimon.types.DataField; import org.apache.paimon.types.DataType; import org.apache.paimon.types.VectorType; @@ -55,6 +57,15 @@ public GlobalIndexer create(List indexFields, Options options) { : new NativeVectorGlobalIndexer(field.type(), nativeOptions, identifier, ratio); } + @Override + public GlobalIndexer createForRead(List indexFields, Options tableOptions) { + if (indexFields.size() == 1 && VectorIndexUtils.isMultiVector(indexFields.get(0).type())) { + // List shards carry their dimension and metric in the native header. + return create(indexFields, new Options()); + } + return GlobalIndexerFactory.super.createForRead(indexFields, tableOptions); + } + static Map nativeOptions( DataType fieldType, Options tableOptions, String identifier, String fieldName) { boolean centroid = CentroidVectorGlobalIndexer.centroidSharding(identifier, tableOptions); @@ -69,12 +80,33 @@ static Map nativeOptions( fieldName, true); nativeOptions.put("index.type", identifier.replace('-', '_')); - nativeOptions.put( - "dimension", String.valueOf(dimension(fieldType, nativeOptions, identifier))); - nativeOptions.putIfAbsent("metric", NativeVectorGlobalIndexer.DEFAULT_METRIC); + boolean multiVector = VectorIndexUtils.isMultiVector(fieldType); + DataType vectorType = multiVector ? ((ArrayType) fieldType).getElementType() : fieldType; + if (!multiVector + || vectorType instanceof VectorType + || nativeOptions.containsKey("dimension")) { + nativeOptions.put( + "dimension", String.valueOf(dimension(vectorType, nativeOptions, identifier))); + } + if (multiVector) { + nativeOptions.putIfAbsent("metric", "cosine"); + validateMultiVectorOptions(centroid, nativeOptions.get("metric")); + } else { + nativeOptions.putIfAbsent("metric", NativeVectorGlobalIndexer.DEFAULT_METRIC); + } return nativeOptions; } + static void validateMultiVectorOptions(boolean centroid, String metric) { + if (centroid) { + throw new IllegalArgumentException("Multivector indexes require range sharding."); + } + if (!"cosine".equals(metric)) { + throw new IllegalArgumentException( + "Multivector indexes require cosine similarity, got metric '" + metric + "'."); + } + } + static double trainSampleRatio(Options tableOptions, String identifier, String fieldName) { Map source = tableOptions.dynamicOptions(); String key = diff --git a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java index c876e3167919..93aea84863e1 100644 --- a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java +++ b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexTest.java @@ -18,16 +18,21 @@ package org.apache.paimon.vector.index; +import org.apache.paimon.data.BinaryVector; +import org.apache.paimon.data.GenericArray; import org.apache.paimon.fs.FileIO; import org.apache.paimon.fs.Path; import org.apache.paimon.fs.PositionOutputStream; +import org.apache.paimon.fs.SeekableInputStream; import org.apache.paimon.fs.local.LocalFileIO; import org.apache.paimon.globalindex.GlobalIndexIOMeta; +import org.apache.paimon.globalindex.GlobalIndexReader; import org.apache.paimon.globalindex.ResultEntry; import org.apache.paimon.globalindex.ScoredGlobalIndexResult; import org.apache.paimon.globalindex.io.GlobalIndexFileReader; import org.apache.paimon.globalindex.io.GlobalIndexFileWriter; import org.apache.paimon.index.vector.IvfPqBatchTableReuseMode; +import org.apache.paimon.index.vector.VectorIndexReader; import org.apache.paimon.index.vector.VectorSearchParams; import org.apache.paimon.options.Options; import org.apache.paimon.predicate.BatchVectorSearch; @@ -55,6 +60,7 @@ import java.util.UUID; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicInteger; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -113,6 +119,246 @@ public void cleanup() throws IOException { // =================== Tests that do NOT need native library ===================== + @Test + public void testMultiVectorArrayRequiresExplicitBuildDimension() { + DataType type = new ArrayType(new ArrayType(new FloatType())); + String kind = "ivf-flat"; + Map options = + NativeVectorGlobalIndexerFactory.nativeOptions( + type, new Options(), kind, fieldName); + assertThatThrownBy( + () -> + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, options, kind)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("ivf-flat.dimension"); + } + + @Test + public void testIncompleteMultiVectorRowCannotBePublished() { + DataType type = new ArrayType(new ArrayType(new FloatType())); + Options options = new Options(); + options.setInteger("ivf-flat.dimension", 2); + Map nativeOptions = + NativeVectorGlobalIndexerFactory.nativeOptions( + type, options, "ivf-flat", fieldName); + try (NativeVectorGlobalIndexWriter writer = + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, nativeOptions, "ivf-flat")) { + GenericArray invalidRow = + new GenericArray( + new Object[] { + new GenericArray(new float[] {1f, 0f}), + new GenericArray(new float[] {1f}) + }); + assertThatThrownBy(() -> writer.write(invalidRow, 7)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("dimension mismatch"); + assertThatThrownBy(writer::finish).isInstanceOf(IllegalStateException.class); + } + } + + @Test + public void testNativeMultiVectorFileStoresRowIds() throws Exception { + Assumptions.assumeTrue(isNativeAvailable()); + DataType type = new ArrayType(new VectorType(2, new FloatType())); + Options options = new Options(); + options.setInteger("ivf-flat.nlist", 1); + Map nativeOptions = + NativeVectorGlobalIndexerFactory.nativeOptions( + type, options, "ivf-flat", fieldName); + float[] q = {1f, 0f}; + Object x = BinaryVector.fromPrimitiveArray(q); + Object y = BinaryVector.fromPrimitiveArray(new float[] {0f, 1f}); + long rowId = (1L << 32) + 7; + List files; + try (NativeVectorGlobalIndexWriter writer = + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, nativeOptions, "ivf-flat")) { + writer.write(new GenericArray(new Object[] {y, null, x, x}), rowId); + writer.write(new GenericArray(new Object[] {y}), 19); + writer.write( + new GenericArray( + new Object[] {BinaryVector.fromPrimitiveArray(new float[] {-1f, 0f})}), + 42); + files = writer.finish(); + } + GlobalIndexIOMeta meta = toIOMetas(files, indexPath).get(0); + try (SeekableInputStream in = createFileReader(indexPath).getInputStream(meta); + VectorIndexReader reader = + new VectorIndexReader( + new NativeVectorGlobalIndexReader.SeekableStreamVectorIndexInput( + in))) { + assertThat(reader.metadata().totalVectors()).isEqualTo(5); + assertThat(reader.search(q, VectorSearchParams.automatic(5)).ids()) + .containsExactlyInAnyOrder(rowId, rowId, rowId, 19L, 42L); + } + try (GlobalIndexReader reader = + new NativeVectorGlobalIndexer(type, nativeOptions, "ivf-flat") + .createReader( + createFileReader(indexPath), + Collections.singletonList(meta), + rowId + 1, + null, + executor)) { + ScoredGlobalIndexResult hits = + reader.visitVectorSearch(new VectorSearch(q, 2, fieldName)).get().get(); + assertThat(hits.results().toArray(2)).containsExactly(19, rowId); + assertThat(hits.scoreGetter().score(rowId)).isEqualTo(1f); + RoaringNavigableMap64 filter = new RoaringNavigableMap64(); + filter.add(rowId); + ScoredGlobalIndexResult filtered = + reader.visitVectorSearch( + new VectorSearch(q, 2, fieldName).withIncludeRowIds(filter)) + .get() + .get(); + assertThat(filtered.results().toArray(2)).containsExactly(rowId); + assertThat(filtered.scoreGetter().score(rowId)).isEqualTo(1f); + } + } + + @Test + public void testEmptyMultiVectorRangeDoesNotCreateAFile() throws Exception { + DataType type = new ArrayType(new VectorType(2, new FloatType())); + String kind = "ivf-flat"; + Map options = + NativeVectorGlobalIndexerFactory.nativeOptions( + type, new Options(), kind, fieldName); + try (NativeVectorGlobalIndexWriter writer = + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, options, kind)) { + writer.write(null, 0); + writer.write(new GenericArray(new Object[0]), 1); + writer.write(new GenericArray(new Object[] {null}), 2); + assertThat(writer.finish()).isEmpty(); + assertThat(fileIO.listStatus(indexPath)).isEmpty(); + } + } + + @Test + public void testMultiVectorCosineSearchAtExtremeScales() throws Exception { + Assumptions.assumeTrue(isNativeAvailable()); + DataType type = new ArrayType(new VectorType(2, new FloatType())); + Options options = new Options(); + options.setInteger("ivf-flat.nlist", 1); + Map nativeOptions = + NativeVectorGlobalIndexerFactory.nativeOptions( + type, options, "ivf-flat", fieldName); + for (float storedScale : new float[] {1e-30f, 1e30f, 1f}) { + List files; + try (NativeVectorGlobalIndexWriter writer = + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, nativeOptions, "ivf-flat")) { + for (int i = 0; i < 8; i++) { + float x = i == 7 ? storedScale : -storedScale; + writer.write( + new GenericArray( + new Object[] { + BinaryVector.fromPrimitiveArray(new float[] {x, 0f}) + }), + i); + } + files = writer.finish(); + } + try (GlobalIndexReader reader = + new NativeVectorGlobalIndexer(type, nativeOptions, "ivf-flat") + .createReader( + createFileReader(indexPath), + toIOMetas(files, indexPath), + 8, + null, + executor)) { + for (float queryScale : new float[] {1e-30f, 1e30f, 1f}) { + float[] query = {queryScale, 0f}; + ScoredGlobalIndexResult result = + reader.visitVectorSearch(new VectorSearch(query, 1, fieldName)) + .get() + .get(); + assertThat(result.results().toArray(1)).containsExactly(7); + assertThat(result.scoreGetter().score(7)).isEqualTo(1f); + assertThat(query).containsExactly(queryScale, 0f); + } + } + } + } + + @Test + public void testMultiVectorParentFilter() throws Exception { + Assumptions.assumeTrue(isNativeAvailable()); + DataType type = new ArrayType(new VectorType(2, new FloatType())); + String kind = "ivf-flat"; + Options options = new Options(); + options.setInteger("ivf-flat.nlist", 1); + Map nativeOptions = + NativeVectorGlobalIndexerFactory.nativeOptions(type, options, kind, fieldName); + assertThat(nativeOptions.get("dimension")).isEqualTo("2"); + float[] q = {1f, 0f}; + try (NativeVectorGlobalIndexWriter writer = + new NativeVectorGlobalIndexWriter( + createFileWriter(indexPath), type, nativeOptions, kind)) { + writer.write(null, 7); + writer.write(new GenericArray(new Object[] {null}), 19); + writer.write(new GenericArray(new Object[0]), 0); + Object x = BinaryVector.fromPrimitiveArray(q); + Object y = BinaryVector.fromPrimitiveArray(new float[] {0f, 1f}); + Object opposite = BinaryVector.fromPrimitiveArray(new float[] {-1f, 0f}); + writer.write(new GenericArray(new Object[] {x, x, x, null}), 23); + writer.write(new GenericArray(new Object[] {y, y, y}), 5); + writer.write(new GenericArray(new Object[] {opposite}), 11); + List files = writer.finish(); + assertThat(files).hasSize(1); + assertThat(files.get(0).rowCount()).isEqualTo(6); + try (GlobalIndexReader reader = + new NativeVectorGlobalIndexer(type, nativeOptions, kind) + .createReader( + createFileReader(indexPath), + toIOMetas(files, indexPath), + 24, + null, + executor)) { + Optional hits = + reader.visitVectorSearch(new VectorSearch(q, 3, fieldName)).get(); + assertThat(hits.get().results().toArray(3)).containsExactly(5, 11, 23); + RoaringNavigableMap64 filter = new RoaringNavigableMap64(); + filter.add(5); + ScoredGlobalIndexResult filtered = + reader.visitVectorSearch( + new VectorSearch(q, 1, fieldName).withIncludeRowIds(filter)) + .get() + .get(); + assertThat(filtered.results().toArray(1)).containsExactly(5); + assertThat(filtered.scoreGetter().score(5)) + .isCloseTo(0f, org.assertj.core.data.Offset.offset(0.00001f)); + filter.add(23); + filter.add(19); // Null list inside the range. + AtomicInteger encodings = new AtomicInteger(); + RoaringNavigableMap64 batchFilter = + new RoaringNavigableMap64() { + @Override + public byte[] serialize() throws IOException { + encodings.incrementAndGet(); + return super.serialize(); + } + }; + batchFilter.or(filter); + List> batch = + reader.visitBatchVectorSearch( + new BatchVectorSearch( + new float[][] {q, {0f, 1f}}, 3, fieldName) + .withIncludeRowIds(batchFilter)) + .get(); + assertThat(batch).hasSize(2); + for (Optional result : batch) { + assertThat(result.get().results().toArray(3)).containsExactly(5, 23); + } + assertThat(encodings.get()).isEqualTo(1); + assertThat(batch.get(0).get().scoreGetter().score(23)).isEqualTo(1f); + assertThat(batch.get(1).get().scoreGetter().score(5)).isEqualTo(1f); + assertThat(batchFilter.toArray(3)).containsExactly(5, 19, 23); + } + } + } + @Test public void testDimensionMismatch() { Options options = createDefaultOptions(64); diff --git a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java index 0ccb7f6cf181..6672a8195c81 100644 --- a/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java +++ b/paimon-vector/src/test/java/org/apache/paimon/vector/index/NativeVectorGlobalIndexerFactoryTest.java @@ -77,6 +77,26 @@ void shardingSelectsTheLayoutAndKeepsTheLocalAlgorithm(String algorithm) { .hasMessageContaining("Can't find global index"); } + @Test + void multiVectorIndexesRequireRangeShardingAndCosine() { + String kind = "diskann"; + List fields = + Collections.singletonList( + new DataField(0, "chunks", new ArrayType(new ArrayType(new FloatType())))); + Options options = new Options(); + options.setString("diskann.sharding", "centroid"); + assertThatThrownBy(() -> GlobalIndexer.create(kind, fields, options)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("range sharding"); + options.setString("diskann.sharding", "range"); + for (String metric : new String[] {"l2", "inner_product"}) { + options.setString("diskann.metric", metric); + assertThatThrownBy(() -> GlobalIndexer.create(kind, fields, options)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("cosine"); + } + } + @Test void diskAnnCentroidUsesTheIvfRouterTrainingOption() throws Exception { float[] values = new float[256];