Skip to content

[core] Support multiple external paths for global index - #11248

Open
lszskye wants to merge 7 commits into
apache:masterfrom
lszskye:support_multi_global_index_external_path
Open

lszskye wants to merge 7 commits into
apache:masterfrom
lszskye:support_multi_global_index_external_path

Conversation

@lszskye

@lszskye lszskye commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Allow global-index.external-path to configure multiple root directories,
for example s3://bucket-a/index,s3://bucket-b/index. New index files are
distributed across these directories in round-robin order.

  • Separate new-file path allocation from existing-file path resolution.
  • Allocate each file's full path once and carry it through writing,
    ResultEntry, and commit metadata generation.
  • Preserve round-robin state across file rotation and synchronize shared
    path allocation.
  • Update index implementations, adapters, tests, and documentation to use
    the full-path contract consistently.

Compatibility

The Java global-index writer SPI now uses allocated full paths instead of basenames.
Java index plugins, including paimon-lumina, must be migrated, rebuilt, and deployed
together with the target Paimon runtime. This also applies when
global-index.external-path is unset or contains a single directory.
Use Paimon-provided plugins from the same release; snapshot or custom Paimon builds
must use the same source revision. Custom plugins must migrate their writer calls
and ResultEntry values before rebuilding against the target Paimon version.
Replace old plugin JARs and restart the processes that load them.

This PR uses coordinated plugin upgrades and does not retain a compatibility
adapter for old plugin binaries. The Java SPI migration does not change index file
contents or the persisted IndexFileMeta format and does not itself require
rebuilding existing indexes.

See the Java index plugin upgrade guide.

Tests

  • Configuration parsing, round-robin distribution, and concurrent allocation.
  • Allocated-path preservation through writing and commit metadata generation,
    including reads after external-path configuration changes.
  • Targeted Bitmap, BTree, FMIndex, primary-key index, Spark, and ES index tests.
  • Multiple external-directory cleanup in FileSystemCatalogTest and
    CachingCatalogTest.
  • Normal Maven verify for paimon-lumina and its reactor dependencies: all 18
    modules succeeded; six targeted writer and native index build/query tests passed
    on JDK 11 with no skips. The generated plugin JAR references newFilePath.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spreading index files across several storage roots has end-to-end value. The PR head passed 220 selected Java tests on JDK 11 with normal Maven verify. Compiling its merge with current master failed in the R-tree writer. I also reproduced compatibility failures in catalog cleanup and a real PyPaimon index build. The inline comments have the inputs and observed results.

Wait for pending test compaction before temporary-directory cleanup,
migrate R-tree writers and fixtures to allocated file paths, and retain
Python process diagnostics when pytest exits without a summary.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Distributing the index files has end-to-end value. The latest head passed 119 selected Core, vector and Lumina writer tests. The additional Java plugin upgrade documentation covers the intended coordinated JAR replacement. The code still conflicts with current master in NativeVectorGlobalIndexWriter.java after the centroid sharding change.

[P1] Complete the engine paths before the multi-root option is usable. I rebuilt a B-tree from PyPaimon on a real Parquet Data Evolution table with two file: roots. The persisted path used the whole comma-separated value as one directory, and the file was outside both configured roots. Please support multi-root placement in the Python builder or reject it before artifacts are written. Java drop also still parses whitespace differently from the builder. Earlier drop/readback findings

[P1] The copied index metadata must use the actual destination. The latest code still passes only targetIndexFilePath.getName() to CopyFilesUtil.toNewIndexFileMeta, which computes the external path from the source parent. The two-root sys.copy failure I previously reproduced remains applicable after the documentation commit. Please use the allocated destination Path and target path factory when persisting the metadata, and include the append-after-copy check. Earlier reproduction

Reuse path allocators across index shards, unify external path parsing, and reject unsupported multi-root PyPaimon builds. Preserve allocated destination paths in copied index metadata and add regression coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants