Skip to content

fix(core): read data-evolution files of an incremental window that are not column slices as stored - #1422

Open
jackylee-ch wants to merge 3 commits into
apache:mainfrom
jackylee-ch:fix/data-evolution-column-slices
Open

jackylee-ch wants to merge 3 commits into
apache:mainfrom
jackylee-ch:fix/data-evolution-column-slices

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: N/A (bug fix)

Reading the window of incremental-between-timestamp of a data-evolution table fails when a compaction merged the files of two inserts and an update then covered the merged rows: "All data files in a field merge split should have the same row id range". The window holds the two inserts' files and the update's file, which are not column slices of one row range, and the reader tried to merge them into one group.

Brief change log

  • DataEvolutionReader reads the files of a streaming (incremental) group that is not a column slice of one row range as they are stored, as it does for raw-convertible groups.
  • New is_data_evolution_column_slice_group decides this: every normal file covers the same rows and every BLOB or vector file lies inside them. Real column slices (an insert and an update of its rows) still merge, and ordinary reads are unchanged.

Tests

  • Added data_evolution_window_test.rs: a window of an insert and a column update of its rows merges into the updated rows; a window of two inserts, a compaction and an update of every row reads the eight stored rows instead of failing. The second test fails on main and passes with this change.
  • Added a unit test for is_data_evolution_column_slice_group.
  • cargo test -p paimon passes; cargo fmt --check is clean.

API and Format

No API or storage format change.

Documentation

No documentation change.

…e not column slices as stored

A window of an incremental-between-timestamp read can hold the files of two inserts and the file of an update of the rows a compaction merged from them. These are not column slices of one row range, and merging them failed the read with 'All data files in a field merge split should have the same row id range'. Read the files of such a streaming group as they are stored. Groups of ordinary reads and real column slices are unchanged.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS at 6e45635.

The incremental-window range mismatch is a real end-to-end problem, but the newly enabled raw-event path still returns incorrect results under ordinary projection and predicates. Reproduced with public write/compact/scan/read APIs in an isolated worktree; details inline. This is a completeness issue in the new supported path: the baseline full-width read throws the range-mismatch error.

Rust 1.98.0: relevant Data Evolution library tests and existing window, incremental, streaming, system-column and ROW-sidecar integration tests pass. Two independent review-only regression tests fail: full-width 8 events become 4 with an id-only projection, and value=1 incorrectly returns no events. In both tests, reusing the unfiltered full-width plan with the same projected/filtered reader returns the expected result, isolating the issue to planning before this reader branch. Head CI is green but does not cover these cases.

// split that are not column slices of one row range are
// independent changes and are read as they are stored.
if is_raw_convertible(&files)
|| (split.is_streaming() && !is_data_evolution_column_slice_group(&files))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Apply physical-event semantics before scan pruning

Selecting raw events only here is too late: TableScan has already applied merged-row statistics and read-field pruning. On this PR's own two inserts (values 1..4) → compaction → value-only update to 50 fixture, the full timestamp window returns 8 physical events. With with_projection(&["id"]), its scan drops the update-only file and returns 4 rows; with with_filter(value = 1), latest-provider stats (min=max=50) drop the entire group and return zero instead of the original (1,1) event. Both are public-API reproductions on this exact head. Reading the original unfiltered/full-width plan through the same projected or filtered reader returns 8 or 1 respectively.

Please classify these physical-event groups before planner statistics/field pruning and use event-safe per-file pruning (preserving events whose projected fields are absent as NULL), or conservatively disable the merged-group optimizations for them. Add projection and predicate regressions alongside the two existing unfiltered window tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 373f19b: incremental non-column-slice groups now bypass merged statistics and projection pruning before the raw read, with projection and predicate regressions verified on the old and new heads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants