Skip to content

Spark 4.1: Fix row lineage in vectorized ORC reads - #17635

Draft
joyhaldar wants to merge 1 commit into
apache:mainfrom
joyhaldar:orc-vectorized-row-lineage
Draft

Spark 4.1: Fix row lineage in vectorized ORC reads#17635
joyhaldar wants to merge 1 commit into
apache:mainfrom
joyhaldar:orc-vectorized-row-lineage

Conversation

@joyhaldar

@joyhaldar joyhaldar commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

VectorizedSparkOrcReaders.StructConverter checked idToConstant before any metadata column, so _row_id matched the generic constant branch and every row received first_row_id. Stored per-row values were ignored and the position offset was never applied. _last_updated_sequence_number had the same problem.

_row_id for a file with first_row_id = 100. Rows 0 and 2 store explicit values (555, 557), rows 1 and 3 store null and should inherit 100 + position.

row expected before
0 555 100
1 101 100
2 557 100
3 103 100
  • Adds RowIdColumnVector and LastUpdatedSeqColumnVector, mirroring RowIdReader and LastUpdatedSeqReader in OrcValueReaders.
  • Moves the metadata-column checks ahead of idToConstant, matching the row-based reader.
  • Passes the ORC schema into StructConverter to look up stored _row_id and _last_updated_sequence_number columns by field id.

Row based ORC got lineage support in #15776 and #16534, but the vectorized reader was not added.

Found while adding vectorized coverage to the format model TCK (#17610).

Co-authored-by: Joy Haldar <joy.haldar@target.com>
@github-actions github-actions Bot added the spark label Aug 13, 2026
@joyhaldar

Copy link
Copy Markdown
Contributor Author

cc: @pvary, @Guosmilesmile, please take a look when you have some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant