Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion docs/docs/multimodal-table/global-index/manage-indexes.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,43 @@ sorting or guarantee lower elapsed time for every workload. Composite and
Multivalue indexes retain their existing build paths. Evaluate CPU, network and
spill costs on a representative cluster before enabling it for a workload.

## Upgrade Java Index Plugins

Support for multiple `global-index.external-path` directories changes the Java
global index writer interface from file names to allocated full paths. Plugins
compiled against the previous interface must be upgraded together with Paimon.
This applies even when the option is unset or contains only one directory.

Deploy the Paimon engine connector/runtime and Paimon-provided Java index plugins,
including `paimon-lumina`, from the same Paimon release. For snapshot or custom
Paimon builds, build them from the same source revision; matching `SNAPSHOT`
version strings alone do not ensure compatibility. Replace the old plugin JARs
in the engine's classpath or plugin directory and restart the processes that
load them.
Updating only the `lumina-jni` dependency does not update Paimon's Java adapter.

Custom Java index plugins must make these changes before being rebuilt against
the target Paimon version:

- Replace `GlobalIndexFileWriter.newFileName(prefix)` with
`newFilePath(prefix)`, which allocates and returns a full path.
- Pass that path unchanged to `newOutputStream(filePath)`. Do not prepend
another directory or select an external directory again.
- Pass the same full path as the first argument of `ResultEntry`. Its
constructor signature is unchanged, but this argument now represents a full
path rather than a file name.
- Replace `ResultEntry.fileName()` with `filePath()` where the result is read;
extract the basename only when a file name is required.

There is no compatibility adapter for the previous Java writer interface.
Using an old plugin JAR with the new runtime can fail with `NoSuchMethodError`
for `GlobalIndexFileWriter.newFileName`. Recompiling unchanged plugin source
against the new interface is insufficient; migrate its calls first.

This Java interface migration does not change index file contents or the
persisted `IndexFileMeta` format. It does not by itself require rebuilding
existing indexes.

## Shared Options

These table options affect global index build and read behavior:
Expand All @@ -475,7 +512,7 @@ These table options affect global index build and read behavior:
| `scalar-index.search-mode` | `fast` | Search mode for BTree, Bitmap, Multivalue, FM, and R-tree bbox queries. |
| `vector-index.search-mode` | `fast` | Search mode for vector queries. |
| `full-text-index.search-mode` | `fast` | Search mode for full-text queries. |
| `global-index.external-path` | Not set | Root directory for global index files. If not set, files are stored under the table index directory. |
| `global-index.external-path` | Not set | Comma-separated root directories for global index files, for example `s3://bucket-a/index,s3://bucket-b/index`. New files use round-robin selection independently of data-file external path settings. Each file's actual path is saved in metadata, so changing this option does not relocate existing files. If not set, files are stored under the table index directory. PyPaimon index builds support only a single external directory and reject comma-separated values in table or build options. |
| `global-index.column-update-action` | `THROW_ERROR` | Update policy: `THROW_ERROR`, `DROP_PARTITION_INDEX`, or `IGNORE`. See [Update Indexed Columns](#update-indexed-columns) for refresh requirements and engine differences. |
| `sorted-index.records-per-file` | `25000000` | Expected number of records per sorted global index file for BTree, Bitmap, and Multivalue builds. Falls back to `sorted-index.records-per-range`, then `btree-index.records-per-range`. |
| `sorted-index.build.max-parallelism` | `4096` | Maximum Flink or Spark parallelism for building sorted global indexes. |
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/core_configuration.html
Original file line number Diff line number Diff line change
Expand Up @@ -942,7 +942,7 @@
<td><h5>global-index.external-path</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>String</td>
<td>Global index root directory, if not set, the global index files will be stored under the &lt;table-root-directory&gt;/index.</td>
<td>Comma-separated global index root directories, new global index files are distributed in round-robin order. If not set, global index files are stored under &lt;table-root-directory&gt;/index.</td>
</tr>
<tr>
<td><h5>global-index.filter.refine-from-data</h5></td>
Expand Down
62 changes: 54 additions & 8 deletions paimon-api/src/main/java/org/apache/paimon/CoreOptions.java
Original file line number Diff line number Diff line change
Expand Up @@ -2908,7 +2908,8 @@ public String toString() {
.stringType()
.noDefaultValue()
.withDescription(
"Global index root directory, if not set, the global index files will be stored under the <table-root-directory>/index.");
"Comma-separated global index root directories, new global index files are distributed "
+ "in round-robin order. If not set, global index files are stored under <table-root-directory>/index.");

public static final ConfigOption<GlobalIndexColumnUpdateAction>
GLOBAL_INDEX_COLUMN_UPDATE_ACTION =
Expand Down Expand Up @@ -4188,17 +4189,19 @@ public boolean indexFileInDataFileDir() {
return options.get(INDEX_FILE_IN_DATA_FILE_DIR);
}

public Path globalIndexExternalPath() {
public List<Path> globalIndexExternalPaths() {
String pathString = options.get(GLOBAL_INDEX_EXTERNAL_PATH);
if (pathString == null || pathString.isEmpty()) {
return null;
return Collections.emptyList();
}
Path path = new Path(pathString);
String scheme = path.toUri().getScheme();
if (scheme == null) {
throw new IllegalArgumentException("scheme should not be null: " + path);
List<Path> paths = new ArrayList<>();
for (String value : pathString.split(",")) {
Path path = new Path(value.trim());
Comment thread
lszskye marked this conversation as resolved.
Resolved
checkArgument(path.toUri().getScheme() != null, "scheme should not be null: %s", path);
Comment thread
lszskye marked this conversation as resolved.
Resolved
paths.add(path);
}
return path;
checkArgument(!paths.isEmpty(), "global index external paths is empty");
return paths;
}

public GlobalIndexColumnUpdateAction globalIndexColumnUpdateAction() {
Expand Down Expand Up @@ -4473,6 +4476,49 @@ public String dataFileExternalPaths() {
return options.get(DATA_FILE_EXTERNAL_PATHS);
}

/** Returns all configured external data paths without applying the write strategy. */
public List<Path> dataFileExternalPathList() {
String externalPaths = dataFileExternalPaths();
if (externalPaths == null || externalPaths.isEmpty()) {
return Collections.emptyList();
}
List<Path> paths = new ArrayList<>();
for (String pathString : externalPaths.split(",")) {
paths.add(new Path(pathString.trim()));
}
return paths;
}

/** Returns external data paths eligible for writing under the configured strategy. */
public List<Path> createExternalPaths() {
String externalPaths = dataFileExternalPaths();
ExternalPathStrategy strategy = externalPathStrategy();
if (externalPaths == null
|| externalPaths.isEmpty()
|| strategy == ExternalPathStrategy.NONE) {
return Collections.emptyList();
}

String specificFS = externalSpecificFS();
List<Path> paths = new ArrayList<>();
for (Path path : dataFileExternalPathList()) {
String scheme = path.toUri().getScheme();
checkArgument(scheme != null, "scheme should not be null: %s", path);
if (strategy == ExternalPathStrategy.SPECIFIC_FS) {
checkArgument(
specificFS != null,
"External path specificFS should not be null when strategy is specificFS.");
if (scheme.equalsIgnoreCase(specificFS)) {
paths.add(path);
}
} else {
paths.add(path);
}
}
checkArgument(!paths.isEmpty(), "External paths should not be empty");
return paths;
}

public ExternalPathStrategy externalPathStrategy() {
return options.get(DATA_FILE_EXTERNAL_PATHS_STRATEGY);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,18 +18,21 @@

package org.apache.paimon.fs;

import javax.annotation.Nullable;

import java.util.List;
import java.util.concurrent.ThreadLocalRandom;

/** Provider for the round-robin external data paths. */
public class RoundRobinExternalPathProvider implements ExternalPathProvider {

private final List<Path> externalTablePaths;
private final Path relativeBucketPath;
@Nullable private final Path relativeBucketPath;

private int position;

public RoundRobinExternalPathProvider(List<Path> externalTablePaths, Path relativeBucketPath) {
public RoundRobinExternalPathProvider(
List<Path> externalTablePaths, @Nullable Path relativeBucketPath) {
this.externalTablePaths = externalTablePaths;
this.relativeBucketPath = relativeBucketPath;
this.position = ThreadLocalRandom.current().nextInt(externalTablePaths.size());
Expand All @@ -46,6 +49,10 @@ public Path getNextExternalDataPath(String fileName) {
if (position == externalTablePaths.size()) {
position = 0;
}
return new Path(new Path(externalTablePaths.get(position), relativeBucketPath), fileName);
Path parent = externalTablePaths.get(position);
if (relativeBucketPath != null) {
parent = new Path(parent, relativeBucketPath);
}
return new Path(parent, fileName);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ static boolean centroidSharding(String indexType, Options options) {

/** Persist the trained router and return the model entry with its underlying index metadata. */
ResultEntry writeRoutingModel(
GlobalIndexFileWriter io, String name, byte[] payload, String modelId)
GlobalIndexFileWriter io, String filePath, byte[] payload, String modelId)
throws IOException;

/** Open an independently trained local index for the assigned centroid. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,21 +23,22 @@
/** Write result meta. */
public class ResultEntry {

private final String fileName;
/** Full file path returned by the writer. */
private final String filePath;

/** Number of source rows covered by this index file, including null or skipped rows. */
private final long rowCount;

@Nullable private final byte[] meta;

public ResultEntry(String fileName, long rowCount, @Nullable byte[] meta) {
this.fileName = fileName;
public ResultEntry(String filePath, long rowCount, @Nullable byte[] meta) {
this.filePath = filePath;
this.rowCount = rowCount;
this.meta = meta;
}

public String fileName() {
return fileName;
public String filePath() {
return filePath;
}

public long rowCount() {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public class BitmapGlobalIndexWriter implements GlobalIndexSingleColumnWriter, C
private final RoaringNavigableMap64 nullRows;
private final RoaringNavigableMap64 nonNullRows;

private String fileName;
private String filePath;
private PositionOutputStream outputStream;
private BitmapGlobalIndexFormat.StreamingWriter streamingWriter;
private long rowCount;
Expand Down Expand Up @@ -119,7 +119,7 @@ public List<ResultEntry> finish() {
lastKey == null ? null : keySerializer.serialize(lastKey),
!nullRows.isEmpty())
.serialize();
return Collections.singletonList(new ResultEntry(fileName, rowCount, meta));
return Collections.singletonList(new ResultEntry(filePath, rowCount, meta));
}

@Override
Expand Down Expand Up @@ -149,8 +149,8 @@ private void flushCurrentBitmap() {

private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws IOException {
if (streamingWriter == null) {
fileName = fileWriter.newFileName(BitmapGlobalIndexerFactory.IDENTIFIER);
outputStream = fileWriter.newOutputStream(fileName);
filePath = fileWriter.newFilePath(BitmapGlobalIndexerFactory.IDENTIFIER);
outputStream = fileWriter.newOutputStream(filePath);
streamingWriter =
new BitmapGlobalIndexFormat.StreamingWriter(
outputStream, dictionaryBlockSize, compressionFactory);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ public class MultiValueBitmapIndexWriter implements GlobalIndexSingleColumnWrite
@Nullable private final BlockCompressionFactory compressionFactory;
private final RoaringNavigableMap64 currentBitmap = new RoaringNavigableMap64();

private String fileName;
private String filePath;
private PositionOutputStream outputStream;
private BitmapGlobalIndexFormat.StreamingWriter streamingWriter;
private BitmapGlobalIndexFormat.SerializedKey currentKey;
Expand Down Expand Up @@ -122,7 +122,7 @@ public List<ResultEntry> finish(long sourceRowCount) {
}

byte[] meta = new SortedIndexFileMeta(firstKey, lastKey, false).serialize();
return Collections.singletonList(new ResultEntry(fileName, sourceRowCount, meta));
return Collections.singletonList(new ResultEntry(filePath, sourceRowCount, meta));
}

@Override
Expand Down Expand Up @@ -150,8 +150,8 @@ private void flushCurrentBitmap() {

private BitmapGlobalIndexFormat.StreamingWriter streamingWriter() throws IOException {
if (streamingWriter == null) {
fileName = fileWriter.newFileName(MultiValueGlobalIndexerFactory.IDENTIFIER);
outputStream = fileWriter.newOutputStream(fileName);
filePath = fileWriter.newFilePath(MultiValueGlobalIndexerFactory.IDENTIFIER);
outputStream = fileWriter.newOutputStream(filePath);
streamingWriter =
new BitmapGlobalIndexFormat.StreamingWriter(
outputStream, dictionaryBlockSize, compressionFactory);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
*/
public class BTreeIndexWriter implements GlobalIndexSingleColumnWriter, Closeable {

private final String fileName;
private final String filePath;
private final PositionOutputStream out;
private boolean closed;

Expand Down Expand Up @@ -127,8 +127,8 @@ public BTreeIndexWriter(
|| fileVersion > BTreeFileFooter.MAX_SUPPORTED_VERSION) {
throw new IllegalArgumentException("Unsupported BTree index version: " + fileVersion);
}
this.fileName = indexFileWriter.newFileName(BTreeGlobalIndexerFactory.IDENTIFIER);
this.out = indexFileWriter.newOutputStream(this.fileName);
this.filePath = indexFileWriter.newFilePath(BTreeGlobalIndexerFactory.IDENTIFIER);
this.out = indexFileWriter.newOutputStream(this.filePath);
try {
this.keySerializer = keySerializer;
this.comparator = keySerializer.createComparator();
Expand Down Expand Up @@ -230,7 +230,7 @@ public List<ResultEntry> finish() {
lastKey == null ? null : keySerializer.serialize(lastKey),
nullBitmap.initialized())
.serialize();
return Collections.singletonList(new ResultEntry(fileName, rowCount, metaBytes));
return Collections.singletonList(new ResultEntry(filePath, rowCount, metaBytes));
}

@Nullable
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ public class FMGlobalIndexWriter implements GlobalIndexSingleColumnWriter, Close

private CharBuilder text = new CharBuilder();
private boolean[] nullRows = new boolean[128];
@Nullable private String fileName;
@Nullable private String filePath;
@Nullable private PositionOutputStream stream;
@Nullable private DataOutputStream output;
private long firstRowId;
Expand Down Expand Up @@ -131,7 +131,7 @@ public List<ResultEntry> finish() {
return Collections.emptyList();
}
Preconditions.checkState(
stream != null && output != null && fileName != null,
stream != null && output != null && filePath != null,
"FM index container output is missing.");
byte[] indexMeta = FMIndexFile.writeIndexMeta(firstRowId, totalRowCount, partitions);
FMIndexFile.BlockInfo directory =
Expand All @@ -140,7 +140,7 @@ public List<ResultEntry> finish() {
FMIndexFile.writeContainerFooter(
output, directory, firstRowId, totalRowCount, partitions.size());
closeOutput();
return Collections.singletonList(new ResultEntry(fileName, totalRowCount, indexMeta));
return Collections.singletonList(new ResultEntry(filePath, totalRowCount, indexMeta));
} catch (IOException e) {
closeOutputQuietly();
throw new RuntimeException("Failed to finish FM global index container.", e);
Expand Down Expand Up @@ -290,8 +290,8 @@ private void ensureOutput() throws IOException {
if (output != null) {
return;
}
fileName = fileWriter.newFileName("fm");
stream = fileWriter.newOutputStream(fileName);
filePath = fileWriter.newFilePath("fm");
stream = fileWriter.newOutputStream(filePath);
output = new DataOutputStream(stream);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@
/** Global index writer io operators. */
public interface GlobalIndexFileWriter {

String newFileName(String prefix);
/** Allocates and returns the full path for a new index file. */
String newFilePath(String prefix);
Comment thread
lszskye marked this conversation as resolved.
Resolved

PositionOutputStream newOutputStream(String fileName) throws IOException;
/** Opens the allocated file path without selecting another location. */
PositionOutputStream newOutputStream(String filePath) throws IOException;
}
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@ private void flush() {
firstRow,
lastRow,
bounds);
String name = files.newFileName("rtree");
try (PositionOutputStream output = files.newOutputStream(name)) {
String filePath = files.newFilePath("rtree");
try (PositionOutputStream output = files.newOutputStream(filePath)) {
output.write(meta.bytes);
List<RTreeIndexFile.Entry> level = entries;
int depth = 0;
Expand Down Expand Up @@ -175,7 +175,7 @@ private void flush() {
finished = true;
throw new UncheckedIOException("Failed to write R-tree index.", e);
}
results.add(new ResultEntry(name, meta.rowCount(), meta.bytes));
results.add(new ResultEntry(filePath, meta.rowCount(), meta.bytes));
entries.clear();
nulls.clear();
nonNulls.clear();
Expand Down
Loading
Loading