Skip to content

[python] Use the target branch schema for latest reads - #11298

Open
skyyyy955 wants to merge 2 commits into
apache:masterfrom
skyyyy955:fix/python-branch-copy-latest-schema
Open

skyyyy955 wants to merge 2 commits into
apache:masterfrom
skyyyy955:fix/python-branch-copy-latest-schema

Conversation

@skyyyy955

Copy link
Copy Markdown
Contributor

Purpose

Follow-up to #10748, which fixed schema selection for branch time travel. With no snapshot or tag selector, main.copy({"branch": "dev"}) could still read data from dev using the fields from main. In a Parquet table where only dev widened count from INT to BIGINT, a stored value of 3000000000 came back as -1294967296. A column added on dev was also missing from the result. Original discussion

When a copy switches branches for a latest read, it now loads the target branch's schema and reapplies the caller's explicit option overrides and removals. REST tables use the catalog response for the selected branch. Snapshot and tag reads continue to use their historical schemas.

Tests

All four new regression cases failed on 7a30ef580e and passed with the fix. They cover reading the widened value, a schema change with no new snapshot, writes through the branch copy, option overrides, and REST metadata for managed and external tables.

On Windows with Python 3.10.11 and PyArrow 19.0.1, all 191 tests in the ten targeted files passed on rerun. The first run had one intermittent failure in test_rename_branch_basic. In isolation it hit Windows error 5 once in eight runs on both the base and patched code. Flake8 passed for both changed Python files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant