Skip to content

feat(file-index): support attaching indexes to existing data files - #582

Open
lxy-9602 wants to merge 4 commits into
apache:mainfrom
lxy-9602:file-index-task
Open

lxy-9602 wants to merge 4 commits into
apache:mainfrom
lxy-9602:file-index-task

Conversation

@lxy-9602

@lxy-9602 lxy-9602 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Purpose

Linked issue: close #581

Add FileIndexWrite::PrepareCommit to attach caller-provided serialized index payloads to existing data files without rewriting the data.

The API resolves target files from a specified snapshot, merges supplied field/type pairs with existing indexes, writes one external index container per data file, and returns commit messages grouped by partition and bucket.

  • Support fixed-bucket append tables and V1/V2 index containers.
  • Preserve unrelated indexes, auxiliary files, and historical index files.
  • Follow Java's index filename generation rules.
  • Publish file indexes via UUID temporary files and rename, reusing existing targets without rewriting them while still preparing commit messages.
  • Include index metadata in manifest entry identifiers so metadata-only rewrites survive merging and stale metadata is detected.
  • Preserve reused physical paths during abort, including compaction level upgrades.

Backfill commits must be serialized with compaction/overwrite per partition and bucket.

Tests

Added coverage for:

  • Index merging, replacement, empty payloads, and embedded/external source containers.
  • Direct commits, snapshot selection, branch isolation, original field names after schema changes, and deletion vectors.
  • Optional dependencies, invalid inputs, short reads, failure cleanup, and repeated backfills.
  • Manifest identifiers, metadata rewrite conflicts, and abort cleanup of reused paths.
  • Cover complete temporary files before rename, early reuse without accessing input streams, commit messages for reused targets, publication conflicts, and failure cleanup.

API and Format

Add the public FileIndexInput, DataFileIndexInput, and FileIndexWrite APIs.

Inputs use original data-file field names and physical row positions, including DV-deleted rows. Null filesystem, memory pool, and executor arguments select internal defaults.

Reuse existing index container and commit message formats; no storage format or protocol changes.

Documentation

Add API references and a user-guide example covering payload inputs, commit usage, and concurrency assumptions.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support attaching file indexes to existing data files

1 participant