[Parquet]: GH-563: Make path_in_schema optional - #3470
Conversation
# Which issue does this PR close? none # Rationale for this change This is a proof of concept implementation for apache/parquet-format#563 # What changes are included in this PR? Since version 57.0.0, this crate has been tolerant of a missing `path_in_schema`. This PR adds options to cease writing the field as well. The option defaults to continuing to write the field. See related discussion on parquet mailing list: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt # Are these changes tested? Yes # Are there any user-facing changes? No, this only adds an optional behavior change that defaults to no change # Related PRs - apache/parquet-format#563 - apache/parquet-format#564 - apache/parquet-java#3470
# Which issue does this PR close? none # Rationale for this change This is a proof of concept implementation for apache/parquet-format#563 # What changes are included in this PR? Since version 57.0.0, this crate has been tolerant of a missing `path_in_schema`. This PR adds options to cease writing the field as well. The option defaults to continuing to write the field. See related discussion on parquet mailing list: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt # Are these changes tested? Yes # Are there any user-facing changes? No, this only adds an optional behavior change that defaults to no change # Related PRs - apache/parquet-format#563 - apache/parquet-format#564 - apache/parquet-java#3470
|
@etseidl What are the plans for this PR? Can we merge it as-is? If not, can we split out the reader change into a separate PR as it's an orthogonal improvement to stop using |
@divjotarora this PR is in limbo until the parquet versioning debate is settled. The issue with making only the read side change is that the thrift generated parser will error out until the IDL can be changed to make |
Rationale for this change
Proof of concept for apache/parquet-format#564
What changes are included in this PR?
Makes changes to
ParquetMetadataConverterto allow for a missingpath_in_schemain the Parquet footer. Also adds a writer option controlling whether or not to write the field. The option defaults totrueto avoid breaking old readers that expect the field to be present.Are these changes tested?
Yes, a test has been added, but likely more will be needed.
Are there any user-facing changes?
Adds a new configuration option.