Repository navigation
fix(core): read data-evolution files of an incremental window that are not column slices as stored - #1422
fix(core): read data-evolution files of an incremental window that are not column slices as stored#1422jackylee-ch wants to merge 3 commits into
Conversation
…e not column slices as stored A window of an incremental-between-timestamp read can hold the files of two inserts and the file of an update of the rows a compaction merged from them. These are not column slices of one row range, and merging them failed the read with 'All data files in a field merge split should have the same row id range'. Read the files of such a streaming group as they are stored. Groups of ordinary reads and real column slices are unchanged.
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 6e45635.
The incremental-window range mismatch is a real end-to-end problem, but the newly enabled raw-event path still returns incorrect results under ordinary projection and predicates. Reproduced with public write/compact/scan/read APIs in an isolated worktree; details inline. This is a completeness issue in the new supported path: the baseline full-width read throws the range-mismatch error.
Rust 1.98.0: relevant Data Evolution library tests and existing window, incremental, streaming, system-column and ROW-sidecar integration tests pass. Two independent review-only regression tests fail: full-width 8 events become 4 with an id-only projection, and value=1 incorrectly returns no events. In both tests, reusing the unfiltered full-width plan with the same projected/filtered reader returns the expected result, isolating the issue to planning before this reader branch. Head CI is green but does not cover these cases.
| // split that are not column slices of one row range are | ||
| // independent changes and are read as they are stored. | ||
| if is_raw_convertible(&files) | ||
| || (split.is_streaming() && !is_data_evolution_column_slice_group(&files)) |
There was a problem hiding this comment.
[P1] Apply physical-event semantics before scan pruning
Selecting raw events only here is too late: TableScan has already applied merged-row statistics and read-field pruning. On this PR's own two inserts (values 1..4) → compaction → value-only update to 50 fixture, the full timestamp window returns 8 physical events. With with_projection(&["id"]), its scan drops the update-only file and returns 4 rows; with with_filter(value = 1), latest-provider stats (min=max=50) drop the entire group and return zero instead of the original (1,1) event. Both are public-API reproductions on this exact head. Reading the original unfiltered/full-width plan through the same projected or filtered reader returns 8 or 1 respectively.
Please classify these physical-event groups before planner statistics/field pruning and use event-safe per-file pruning (preserving events whose projected fields are absent as NULL), or conservatively disable the merged-group optimizations for them. Add projection and predicate regressions alongside the two existing unfiltered window tests.
There was a problem hiding this comment.
Fixed in 373f19b: incremental non-column-slice groups now bypass merged statistics and projection pruning before the raw read, with projection and predicate regressions verified on the old and new heads.
Purpose
Linked issue: N/A (bug fix)
Reading the window of
incremental-between-timestampof a data-evolution table fails when a compaction merged the files of two inserts and an update then covered the merged rows: "All data files in a field merge split should have the same row id range". The window holds the two inserts' files and the update's file, which are not column slices of one row range, and the reader tried to merge them into one group.Brief change log
DataEvolutionReaderreads the files of a streaming (incremental) group that is not a column slice of one row range as they are stored, as it does for raw-convertible groups.is_data_evolution_column_slice_groupdecides this: every normal file covers the same rows and every BLOB or vector file lies inside them. Real column slices (an insert and an update of its rows) still merge, and ordinary reads are unchanged.Tests
data_evolution_window_test.rs: a window of an insert and a column update of its rows merges into the updated rows; a window of two inserts, a compaction and an update of every row reads the eight stored rows instead of failing. The second test fails onmainand passes with this change.is_data_evolution_column_slice_group.cargo test -p paimonpasses;cargo fmt --checkis clean.API and Format
No API or storage format change.
Documentation
No documentation change.