Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/flink/procedures/indexes.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ To perform vector similarity search on a table with a global vector index. Retur

- `vector_column` (required): the name of the vector column to search.

- `query_vector` (required): comma-separated float values representing the query vector, e.g. '1.0,2.0,3.0'.
- `query_vector` (required) accepts comma-separated coordinates for one vector, such as `'1.0,2.0,3.0'`. For MaxSim search on a list of vectors in a Data Evolution table, pass a JSON matrix such as `'[[1,0],[0,1]]'`.

- `top_k` (required): the number of nearest neighbors to return.

Expand Down
74 changes: 64 additions & 10 deletions docs/docs/multimodal-table/global-index/vector.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -311,15 +311,15 @@ result = (
```

Exact scalar global-index answers pre-filter the vector search. If scalar indexes cannot
evaluate the predicate, each shard runs ANN once and checks only its candidates against the
evaluate the predicate, each shard retrieves ANN candidates and checks them against the
filter columns of the same pinned snapshot. The coordinator merges the filtered results;
it does not perform a full-table scalar scan or a raw scan to fill missing matches.

`vector-index.post-filter.candidates-factor` is a positive integer, defaulting to `10`.
The candidate count is `max(original Top-K * factor, required reranking limit)`, capped by
the shard size and the backend integer limit. For example, Top-K 10 normally requests 100
candidates. Batch execution may split large inputs into memory-bounded chunks; every query
still runs ANN once. Post-filtering may return fewer than Top-K rows, including zero even
candidates. Batch execution may split large inputs into memory-bounded chunks. On single-vector
columns, each query runs ANN once per shard. Post-filtering may return fewer than Top-K rows, including zero even
when matches exist outside the candidates. Increasing the factor trades more ANN/scalar
work for more candidates. Post-filtering does not retry or scan raw data to fill Top-K.
Raw work outside vector coverage still follows the configured index search modes. When a
Expand All @@ -344,6 +344,58 @@ A search selects top-K row IDs; reading those rows through a regular table scan
does not imply score order. See [Read Scored Results](./manage-indexes#read-scored-results)
for SQL ordering and Java/Python score access.

### Search a List of Vectors

On a Data Evolution table, Spark SQL, Flink SQL and Java can search a column containing multiple
embeddings per row. The native `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq`, and `diskann`
indexes support these columns with range sharding and cosine similarity.
Omitting the metric selects `cosine`. Index creation rejects any other metric.
The column can be `ARRAY<VECTOR<FLOAT, d>>` or `ARRAY<ARRAY<FLOAT>>`. For the latter,
specify the inner dimension when building the index. Each row can contain a different
number of vectors. You can build an independent index for each such column.

```sql
-- Spark SQL
CALL sys.create_global_index(
table => 'db.documents', index_column => 'chunks', index_type => 'ivf-flat',
options => 'ivf-flat.dimension=2,ivf-flat.distance.metric=cosine'
);

SELECT id, __paimon_search_score
FROM vector_search('db.documents', 'chunks', array(array(1.0f, 0.0f), array(0.0f, 1.0f)), 10)
ORDER BY __paimon_search_score DESC, id;
```

A single query vector ranks rows by their best matching embedding. A query matrix
uses MaxSim to rank rows. For every query vector, take the highest cosine similarity
among a row's embeddings, then sum those similarities. Each row can occupy one
place in the result. Null embeddings are skipped. A row with no embeddings has no score.
A range with no embeddings produces no index file.
Query matrices require at least one vector, fixed inner dimensions and finite coordinates.
Spark SQL accepts a matrix in an uncorrelated `vector_search` call. A correlated
`LATERAL vector_search` call takes one query vector per outer row.
In Flink SQL, pass the matrix as a JSON string to the search procedure. The procedure
executes matrix queries locally even when `vector-search.distribute.enabled` is enabled.

```sql
CALL sys.vector_search(
`table` => 'db.documents', vector_column => 'chunks',
query_vector => '[[1,0],[0,1]]', top_k => 10,
projection => 'id,__paimon_search_score'
);
```

For a matrix query, Paimon retrieves candidate rows for each query vector and scores the
union against their complete lists from the query snapshot. Candidate retrieval is approximate.
Each indexed vector carries the row ID of its source list. When several matches share that
ID, the reader keeps their highest score and can request more distinct candidate rows
before applying the post-filter.
By default, it requests up to four times the final limit per query vector per shard.
Set `refine_factor` to select that multiplier for any of the five indexes. Uncovered rows
included by the search mode are scored with the same MaxSim formula. Java exposes matrix queries through
`newVectorSearchBuilder().withMultiVector(float[][])`. For independent top-K results
per vector, use `newBatchVectorSearchBuilder().withVectors(float[][])`.

## Search Options

Search-time options are passed with each vector search request:
Expand All @@ -353,8 +405,8 @@ Search-time options are passed with each vector search request:
| `ivf.nprobe` | Automatic | Explicit number of IVF clusters to probe. When omitted, paimon-vindex derives the width from the index, `top_k`, and filter selectivity. |
| `centroid.nprobe` | Automatic | Number of centroid shards to search per row-ID range. Paimon chooses this width when omitted. Within each selected shard, `ivf.nprobe` controls the IVF lists to search and `diskann.l_search` controls the DiskANN graph search width. |
| `ivf.max_initial_filter_expansion_factor` | Disabled | Positive integer limiting filter-driven expansion of the initial automatic IVF probe width. A factor of `1` disables initial filter expansion. Progressive retries may still probe more clusters when fewer than `top_k` filtered results are found. |
| `ivf.refine_factor` | Disabled | Retrieves `top_k * refine_factor` IVF candidates and reranks them with the original vectors stored in the Paimon table. It is most useful for compressed indexes such as `ivf-pq`, `ivf-sq`, and `ivf-rq` when recall is more important than latency. |
| `ivf_pq.batch_table_reuse` | `auto` | IVF-PQ batch search distance-table reuse mode: `auto`, `on`, or `off`. Other index types and scalar searches ignore it. |
| `refine_factor` | List columns use `4` for a matrix query and `1` for a single query vector. Disabled for single-vector columns. | Retrieves `top_k * refine_factor` candidates per query vector and reranks them with the original vectors stored in the Paimon table. For single-vector columns it is most useful with compressed indexes such as `ivf-pq`, `ivf-sq`, and `ivf-rq`. |
| `ivf_pq.batch_table_reuse` | `auto` | Distance-table reuse for native batched IVF-PQ searches on single-vector columns. Accepts `auto`, `on`, or `off`. |
| `ivf_pq.batch_table_reuse.max_bytes` | 512 MiB | Positive long integer limiting the memory used by IVF-PQ batch distance-table reuse. Search falls back to direct table construction when the reusable tables exceed the budget. |
| `diskann.l_search` | Automatic | paimon-vindex DiskANN graph candidate width. The automatic value uses calibration when available, otherwise `max(100, 2 * top_k)`. |

Expand All @@ -368,9 +420,11 @@ reduce Recall@K compared with uncapped automatic search. Progressive expansion o
fewer than `top_k` valid results are returned; if the capped initial search already fills `top_k`,
probing stops.

`ivf.refine_factor` can also be configured with `refine_factor`, `rerank_factor`, and hyphenated
spellings such as `ivf.refine-factor`. Setting `ivf.refine_factor=1` still performs the raw-vector
rerank for the indexed candidates; leaving it unset skips the rerank stage.
`refine_factor` also accepts the alias `rerank_factor` and hyphenated spellings such as
`refine-factor`. The four IVF indexes also accept `ivf.refine_factor`. Setting `refine_factor=1`
still performs the raw-vector rerank for the indexed candidates. Single-vector columns skip
reranking when the option is unset.
List columns always score candidate rows against their complete embedding lists.

## Build Options

Expand All @@ -380,8 +434,8 @@ Build options for native vector indexes:
|---|---|---|
| `<index-type>.sharding` | `range` | Use `centroid` to train an outer router for each row-ID range when building a Spark Data Evolution index. |
| `<index-type>.centroid.nlist` | Automatic | Number of outer centroids per contiguous row-ID range with centroid sharding. |
| `<index-type>.dimension` | `128` | Vector dimension for `ARRAY<FLOAT>` columns. Ignored for `VECTOR<FLOAT, n>` columns. |
| `<index-type>.distance.metric` | `inner_product` | Distance metric. Supported values: `l2`, `cosine`, `inner_product`. |
| `<index-type>.dimension` | `128` for `ARRAY<FLOAT>` | Required for `ARRAY<ARRAY<FLOAT>>`. Columns with a fixed-size vector type supply their own dimension. |
| `<index-type>.distance.metric` | `inner_product` for single-vector columns, `cosine` for lists | Single-vector columns support `l2`, `cosine`, and `inner_product`. List columns support `cosine`. |
| `<index-type>.ivf.coarse-assignment` | `auto` | Build-time list assignment for all IVF types. `auto` uses Vamana when `dimension × nlist ≥ 1,000,000` and exact assignment otherwise; `exact` always uses exact assignment. |
| `<index-type>.ivf.train.max-points-per-centroid` | `256` | Positive coarse K-means training limit for all IVF types: at most `nlist × value` vectors. |
| `<index-type>.train.sample-ratio` | `1.0` | Ratio of vectors sampled for native index training. Must be greater than `0` and less than or equal to `1`. Lower values reduce training memory and build cost, but may reduce index quality. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ public Options options() {
return options;
}

/** Returns the normalized metric, defaulting to inner product for single-vector searches. */
public String metric() {
return metric;
}
Expand Down Expand Up @@ -110,14 +111,20 @@ private static String legacyIndexType(Options options, String column) {
return type == null ? null : type.toLowerCase(Locale.ROOT).trim();
}

/** Resolves single-vector options, including the default metric for a known index type. */
public static VectorIndexOptions resolve(
Options options, @Nullable String type, String column) {
return resolve(options, type, column, true);
VectorIndexOptions resolved = resolveConfigured(options, type, column);
if (resolved.type != null && resolved.options.get(resolved.type + ".metric") == null) {
metricAliases(resolved.type, column)
.forEach(key -> resolved.options.setString(key, resolved.metric));
}
return resolved;
}

/** Retains an unspecified metric when readers must obtain it from persisted index metadata. */
public static VectorIndexOptions resolve(
Options options, @Nullable String type, String column, boolean defaultMetric) {
/** Keeps metric aliases absent from {@link #options()} when no metric was configured. */
public static VectorIndexOptions resolveConfigured(
Options options, @Nullable String type, String column) {
if (type == null) {
CoreOptions core = new CoreOptions(options);
List<String> declared =
Expand All @@ -140,12 +147,7 @@ public static VectorIndexOptions resolve(
}
String prefix = type + ".";
String fieldPrefix = "fields." + column + ".";
List<String> aliases =
Arrays.asList(
fieldPrefix + "distance.metric",
fieldPrefix + "metric",
prefix + "distance.metric",
prefix + "metric");
List<String> aliases = metricAliases(type, column);
String metric = null;
for (String key :
Arrays.asList(
Expand All @@ -172,11 +174,20 @@ public static VectorIndexOptions resolve(
break;
}
}
if (metric != null || defaultMetric) {
String normalized = normalizeMetric(metric);
String normalized = normalizeMetric(metric);
if (metric != null) {
aliases.forEach(key -> resolved.setString(key, normalized));
}
return new VectorIndexOptions(type, resolved, normalizeMetric(metric));
return new VectorIndexOptions(type, resolved, normalized);
}

private static List<String> metricAliases(String type, String column) {
String fieldPrefix = "fields." + column + ".";
return Arrays.asList(
fieldPrefix + "distance.metric",
fieldPrefix + "metric",
type + ".distance.metric",
type + ".metric");
}

private static String normalizeMetric(@Nullable String metric) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,17 +56,26 @@ static GlobalIndexer create(String type, List<DataField> indexFields, Options op
.create(indexFields, resolveOptions(type, indexFields, options));
}

/** Opens an indexer for reading existing files using the current table options. */
static GlobalIndexer createForRead(
String type, List<DataField> indexFields, Options tableOptions) {
return GlobalIndexerFactoryUtils.load(type).createForRead(indexFields, tableOptions);
}

/** Resolve the same field and procedure options for build planning and plugin creation. */
static Options resolveOptions(String type, List<DataField> indexFields, Options options) {
Options resolved = options;
if (!indexFields.isEmpty()) {
// The leading field owns the index; remaining fields are indexed payload columns.
String column = indexFields.get(0).name();
resolved =
CoreOptions.isVectorIndexType(type)
? VectorIndexOptions.resolve(options, type, column).options()
: new CoreOptions(options).globalIndexOptions(type, column);
if (indexFields.isEmpty()) {
return options;
}
// The leading field owns the index; remaining fields are indexed payload columns.
DataField field = indexFields.get(0);
if (CoreOptions.isVectorIndexType(type)) {
VectorIndexOptions resolved =
VectorIndexUtils.isMultiVector(field.type())
? VectorIndexOptions.resolveConfigured(options, type, field.name())
: VectorIndexOptions.resolve(options, type, field.name());
return resolved.options();
}
return resolved;
return new CoreOptions(options).globalIndexOptions(type, field.name());
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,12 @@ default List<GlobalIndexIOMeta> selectFiles(
return files;
}

/** Creates an indexer for existing files. Implementations can use persisted build settings. */
default GlobalIndexer createForRead(List<DataField> indexFields, Options tableOptions) {
return create(
indexFields, GlobalIndexer.resolveOptions(identifier(), indexFields, tableOptions));
}

/** Creates an indexer over a non-empty list of columns in index order. */
GlobalIndexer create(List<DataField> indexFields, Options options);
}
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,62 @@

import org.apache.paimon.data.InternalArray;
import org.apache.paimon.data.InternalVector;
import org.apache.paimon.types.ArrayType;
import org.apache.paimon.types.DataType;
import org.apache.paimon.types.FloatType;
import org.apache.paimon.types.VectorType;

/** Materialization at the portable vector index boundary. */
public final class VectorIndexUtils {
private VectorIndexUtils() {}

public static boolean isMultiVector(DataType type) {
if (!(type instanceof ArrayType)) {
return false;
}
DataType child = ((ArrayType) type).getElementType();
return (child instanceof VectorType
&& ((VectorType) child).getElementType() instanceof FloatType)
|| (child instanceof ArrayType
&& ((ArrayType) child).getElementType() instanceof FloatType);
}

public static void validateQuery(float[][] vectors) {
if (vectors == null
|| vectors.length == 0
|| vectors[0] == null
|| vectors[0].length == 0) {
throw new IllegalArgumentException(
"A multivector query requires at least one non-empty vector.");
}
for (float[] vector : vectors) {
if (vector == null || vector.length != vectors[0].length) {
throw new IllegalArgumentException("Multivector query dimensions must match.");
}
for (float value : vector) {
if (!Float.isFinite(value)) {
throw new IllegalArgumentException(
"Multivector query coordinates must be finite.");
}
}
}
}

public static void validateQuery(ArrayType columnType, float[][] vectors) {
validateQuery(vectors);
DataType child = columnType.getElementType();
if (child instanceof VectorType) {
int dimension = ((VectorType) child).getLength();
if (vectors[0].length != dimension) {
throw new IllegalArgumentException(
"Query vector dimension mismatch: expected "
+ dimension
+ ", got "
+ vectors[0].length);
}
}
}

public static float[] toFloatArray(Object value) {
if (value instanceof float[]) {
return ((float[]) value).clone();
Expand Down
Loading
Loading