Skip to content

feat: support weighted data paths and round-robin global index paths - #545

Merged
lxy-9602 merged 5 commits into
apache:mainfrom
lszskye:optimize_external_path
Nov 12, 2026
Merged

lxy-9602 merged 5 commits into
apache:mainfrom
lszskye:optimize_external_path

Conversation

@lszskye

@lszskye lszskye commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #543

Support weighted placement of data files and distribute index files and
checkpoints across multiple external directories.

  • Add data-file.external-paths.weights and the weighted strategy.
    Weights apply only to this strategy; empty weights fall back to round robin.
  • Allow comma-separated roots in global-index.external-path, with
    independent round-robin selection for index files and checkpoints.
  • Recover checkpoints by scanning all configured task directories and
    selecting the largest checkpoint ID. Continue ID allocation from that
    maximum and clean up checkpoints across all configured roots.
  • Allocate full index paths once and retain them for writing, size queries,
    metadata, and cleanup, without a filename-to-path cache.
  • Place newly created DV files in the configured global external directories
    when index-file-in-data-file-dir=false.

Tests

  • Add option tests for weight parsing, ignoring weights under other
    strategies, and parsing multiple global index roots.
  • Add checkpoint tests for recovery across roots with duplicate older IDs,
    continued task cleanup after a deletion failure, and deletion error
    propagation.
  • Update path factory tests and index writer tests for full-path allocation
    and the revised factory interfaces.

API and Format

  • Add Options::DATA_FILE_EXTERNAL_PATHS_WEIGHTS.
  • Breaking API change: replace GlobalIndexFileWriter::NewFileName() with
    NewPath() and remove its ToPath() method. Callers must retain the
    returned full path and pass it to NewOutputStream() and GetFileSize().
  • Update built-in index writers and their callers to use full paths.
  • No storage format or protocol changes. External index locations use the
    existing metadata field, and checkpoint filenames retain their format.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

@lszskye
lszskye force-pushed the optimize_external_path branch from 3c972d8 to 18afb70 Compare November 10, 2026 11:35
Comment thread include/paimon/defs.h Resolved
Comment thread src/paimon/core/core_options.cpp Resolved

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 3d3bc00 into apache:main Nov 12, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants