Skip to content

GH-50142: [C++][Python] Honor Parquet time coercion and format version - #53161

Open
gitedmond wants to merge 1 commit into
apache:mainfrom
gitedmond:gh-50142-time-coercion
Open

gitedmond wants to merge 1 commit into
apache:mainfrom
gitedmond:gh-50142-time-coercion

Conversation

@gitedmond

@gitedmond gitedmond commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Fixes #50142. Writing time64[ns] with Parquet version 1.0 or 2.4 currently emits TIME(NANOS), and explicit coerce_timestamps options leave time-of-day fields unchanged. For example, a value of 1,000 nanoseconds remains TIME(NANOS) instead of becoming 1 microsecond under version 2.4.

Credit to fenfeng9's analysis for separating the compatibility bug from the existing timestamp-only option semantics. This follows the agreed direction to reuse the existing coercion options, rather than adding time-specific options. The footer's reported format_version is not changed.

What changes are included in this PR?

  • Select time annotations and physical widths from the requested coercion unit and format version.
  • Convert values to the selected on-disk units, preserving nulls and rejecting precision loss or integer overflow. Milliseconds use INT32; microseconds and nanoseconds use INT64.
  • Update the existing older-version roundtrip fixture, add focused regressions, and document time-of-day coercion.

Are these changes tested?

  • parquet-arrow-reader-writer-test: 834 passed; two unsupported dictionary cases skipped and one existing large-string test disabled.
  • parquet-arrow-metadata-test: 49 passed.
  • Python Parquet test_datetime.py, with Parquet and pandas groups enabled, against locally built PyArrow: 48 passed, including 29 focused time cases.
  • Fail-before checks: the new C++ coercion test and corrected datetime roundtrip fail against unmodified main; 25 of the 29 Python regression cases fail against released PyArrow 25.0.1.
  • Clang-format 18.1.8, cpplint 1.6.1, configured Python flake8 checks, and git diff --check pass.

Are there any user-facing changes?

coerce_timestamps and allow_truncated_timestamps now also apply to Arrow time32/time64 fields. Without explicit coercion, nanosecond times are converted to microseconds for versions 1.0/2.4 and preserved for 2.6. Lossy conversion raises an error unless an explicit coercion target and allow_truncated_timestamps=True are both provided.

Was AI used for this PR?

Codex assisted with the investigation, implementation, tests, documentation and PR description, and reviewed the changes before submission.

PR code and description written by:

  • Human
  • AI

Reviewed before submission by:

  • Human
  • AI
  • Not reviewed

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50142 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting review Awaiting review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Python][Parquet] pyarrow.parquet.write_table ignores version and coerce_timestamps arguments for Time64 fields

1 participant