Repository navigation
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Spreading index files across several storage roots has end-to-end value. The PR head passed 220 selected Java tests on JDK 11 with normal Maven verify. Compiling its merge with current master failed in the R-tree writer. I also reproduced compatibility failures in catalog cleanup and a real PyPaimon index build. The inline comments have the inputs and observed results.
Wait for pending test compaction before temporary-directory cleanup, migrate R-tree writers and fixtures to allocated file paths, and retain Python process diagnostics when pytest exits without a summary.
JingsongLi
left a comment
There was a problem hiding this comment.
Distributing the index files has end-to-end value. The latest head passed 119 selected Core, vector and Lumina writer tests. The additional Java plugin upgrade documentation covers the intended coordinated JAR replacement. The code still conflicts with current master in NativeVectorGlobalIndexWriter.java after the centroid sharding change.
[P1] Complete the engine paths before the multi-root option is usable. I rebuilt a B-tree from PyPaimon on a real Parquet Data Evolution table with two file: roots. The persisted path used the whole comma-separated value as one directory, and the file was outside both configured roots. Please support multi-root placement in the Python builder or reject it before artifacts are written. Java drop also still parses whitespace differently from the builder. Earlier drop/readback findings
[P1] The copied index metadata must use the actual destination. The latest code still passes only targetIndexFilePath.getName() to CopyFilesUtil.toNewIndexFileMeta, which computes the external path from the source parent. The two-root sys.copy failure I previously reproduced remains applicable after the documentation commit. Please use the allocated destination Path and target path factory when persisting the metadata, and include the append-after-copy check. Earlier reproduction
Reuse path allocators across index shards, unify external path parsing, and reject unsupported multi-root PyPaimon builds. Preserve allocated destination paths in copied index metadata and add regression coverage.
Purpose
Allow
global-index.external-pathto configure multiple root directories,for example
s3://bucket-a/index,s3://bucket-b/index. New index files aredistributed across these directories in round-robin order.
ResultEntry, and commit metadata generation.path allocation.
the full-path contract consistently.
Compatibility
The Java global-index writer SPI now uses allocated full paths instead of basenames.
Java index plugins, including
paimon-lumina, must be migrated, rebuilt, and deployedtogether with the target Paimon runtime. This also applies when
global-index.external-pathis unset or contains a single directory.Use Paimon-provided plugins from the same release; snapshot or custom Paimon builds
must use the same source revision. Custom plugins must migrate their writer calls
and
ResultEntryvalues before rebuilding against the target Paimon version.Replace old plugin JARs and restart the processes that load them.
This PR uses coordinated plugin upgrades and does not retain a compatibility
adapter for old plugin binaries. The Java SPI migration does not change index file
contents or the persisted
IndexFileMetaformat and does not itself requirerebuilding existing indexes.
See the Java index plugin upgrade guide.
Tests
including reads after external-path configuration changes.
FileSystemCatalogTestandCachingCatalogTest.verifyforpaimon-luminaand its reactor dependencies: all 18modules succeeded; six targeted writer and native index build/query tests passed
on JDK 11 with no skips. The generated plugin JAR references
newFilePath.