GH-50910: [Format] Tolerate unrecognized logical/physical type combinations when reading - #50909
GH-50910: [Format] Tolerate unrecognized logical/physical type combinations when reading#50909divjotarora wants to merge 1 commit into
Conversation
|
Thanks for opening a pull request! This pull request has been automatically converted to a draft because its title doesn't match Arrow's required format. If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or After updating the title, you can mark the pull request as ready for review. See also: |
|
|
4dc0ee3 to
6017b0e
Compare
There was a problem hiding this comment.
Changes in this file are due to pinning to a parquet-testing commit that contains the new file from apache/parquet-testing#122. The only file we actually use from this set is data/int32_with_uuid_logical_type.parquet.
6017b0e to
7062580
Compare
Reranko05
left a comment
There was a problem hiding this comment.
Could you also add Component of the PR in the title, it will be easier for maintainers to review it.
GH-<Issue Number>: [<Component>] <Title>
Rationale for this change
See apache/parquet-format#607 for rationale.
What changes are included in this PR?
This PR gracefully handles unrecognized logical/physical type combinations by dropping the logical type during the read and dropping any associated statistics for the relevant columns. Note that unrecognized logical types are already handled gracefully and no changes were required.
Are these changes tested?
Yes, via unit tests and an e2e test that reads a real file that contains an invalid type combination.
Are there any user-facing changes?
No