[Python][Parquet] pyarrow.parquet.write_table ignores version and coerce_timestamps arguments for Time64 fields

Describe the bug, including details regarding any error messages, version, and platform.

OS: Ubuntu 24.04 WSL
Python: 3.12.3
pyarrow: 24.0.0

Suppose we have a pyarrow table with a nanosecond resolution time column:

import pyarrow as pa
t = pa.table({'A': [40984E9]}, schema=pa.schema({'A': pa.time64('ns')}))

I want to save this to an older Parquet version, to support older readers such as Azure Synapse, which doesn't support nanosecond time resolution.

According to the pyarrow.parquet.write_table docs, if coerce_timestamps is not specified:

defaults are chosen depending on version. For version='1.0' and version='2.4', nanoseconds are cast to microseconds (‘us’)

So, I should be able to save this to either version '1.0' or version '2.4', and it should automatically cast nanoseconds to microseconds:

Parquet 2.4

pa.parquet.write_table(t, 'test-pyarrow-2.4.parquet', version='2.4')

However, inspecting the metadata, we can immediately see that the version argument has been ignored:

pa.parquet.ParquetFile('test-pyarrow-2.4.parquet').metadata
# <pyarrow._parquet.FileMetaData object at 0x7fef9110b9c0>
#   created_by: parquet-cpp-arrow version 24.0.0
#   num_columns: 1
#   num_rows: 1
#   num_row_groups: 1
#   format_version: 2.6
#   serialized_size: 383

Additionally, we can see that the time field has not been coerced to microseconds:

pa.parquet.ParquetFile('test-pyarrow-2.4.parquet').schema
# <pyarrow._parquet.ParquetSchema object at 0x7fefa95ee4c0>
# required group field_id=-1 schema {
#   optional int64 field_id=-1 A (Time(isAdjustedToUTC=false, timeUnit=nanoseconds));
# }

If we add the (optional) argument coerce_timestamps='us', the effect is the same:

pa.parquet.write_table(t, 'test-pyarrow-2.4-coerce.parquet', version='2.4', coerce_timestamps='us')
pa.parquet.ParquetFile('test-pyarrow-2.4-coerce.parquet').metadata
# <pyarrow._parquet.FileMetaData object at 0x7fef07178310>
#   created_by: parquet-cpp-arrow version 24.0.0
#   num_columns: 1
#   num_rows: 1
#   num_row_groups: 1
#   format_version: 2.6
#   serialized_size: 383

pa.parquet.ParquetFile('test-pyarrow-2.4-coerce.parquet').schema
# <pyarrow._parquet.ParquetSchema object at 0x7fef882d27c0>
# required group field_id=-1 schema {
#   optional int64 field_id=-1 A (Time(isAdjustedToUTC=false, timeUnit=nanoseconds));
# }

Parquet 1.0

If we try setting the Parquet version to '1.0', we are similarly unable to coerce the timestamps to microseconds:

pa.parquet.write_table(t, 'test-pyarrow-1.0.parquet', version='1.0')
pa.parquet.ParquetFile('test-pyarrow-1.0.parquet').metadata
# <pyarrow._parquet.FileMetaData object at 0x7fef0712bce0>
#   created_by: parquet-cpp-arrow version 24.0.0
#   num_columns: 1
#   num_rows: 1
#   num_row_groups: 1
#   format_version: 1.0
#   serialized_size: 382

pa.parquet.ParquetFile('test-pyarrow-1.0.parquet').schema
# <pyarrow._parquet.ParquetSchema object at 0x7fef0cc69e80>
# required group field_id=-1 schema {
#   optional int64 field_id=-1 A (Time(isAdjustedToUTC=false, timeUnit=nanoseconds));
# }

This time, we at least get the correct Parquet version. However, the time field still has nanosecond resolution.
Similarly, the coerce_timestamps='us' argument has the same result.

Component(s)

Python, Parquet

Just my personal reading of this issue:

I think there are two separate points here.

First, time32 / time64 are Parquet TIME logical types, not TIMESTAMP logical types:

https://parquet.apache.org/docs/file-format/types/logicaltypes/#time

So I am not sure that coerce_timestamps is currently documented as applying to Arrow time32 / time64. From the current implementation, this option seems to be applied to Arrow TimestampType only.

Second, I think there is still a real issue with version='1.0' / version='2.4'. NANOS is treated as a Parquet 2.6 feature, so writing time64[ns] as TIME(NANOS) when an older version is requested looks wrong to me.

About metadata.format_version: I don't think this is strong evidence by itself. The Parquet footer does not store the exact writer option like 2.4. It stores the Parquet metadata version field, which is an integer value such as 1 or 2. PyArrow reads version 2 as the latest supported 2.x version, currently 2.6. So format_version: 2.6 does not necessarily mean that version='2.4' was ignored.

@pitrou

@pitrou Sorry to ping you directly. Could you help take a look at this issue when you have time? I am not sure who else to ping here.

@fenfeng9

@fenfeng9 I think your analysis above is right.

@pitrou

Second, I think there is still a real issue with version='1.0' / version='2.4'. NANOS is treated as a Parquet 2.6 feature, so writing time64[ns] as TIME(NANOS) when an older version is requested looks wrong to me.

@pitrou Thanks for your reply!
Do we still need to handle this case? If so, do you have any suggestions?
For example, we could reuse coerce_timestamps/allow_truncated_timestamps for time types, or introduce new counterparts like coerce_times and allow_truncated_times?

@daviewales

For example, we could reuse coerce_timestamps/allow_truncated_timestamps for time types, or introduce new counterparts like coerce_times and allow_truncated_times?

I think we could just reuse them.

Also, for the record, Parquet 2.6 is from 2018, I'm not sure there's a still a point in selecting an earlier version. @daviewales What is your use case for selecting Parquet 2.4?

I want to save this to an older Parquet version, to support older readers such as Azure Synapse, which doesn't support nanosecond time resolution.

This is his description in the issue.

Oops, sorry, I had missed that!

For example, we could reuse coerce_timestamps/allow_truncated_timestamps for time types, or introduce new counterparts like coerce_times and allow_truncated_times?

I think we could just reuse them.

Also, for the record, Parquet 2.6 is from 2018, I'm not sure there's a still a point in selecting an earlier version. @daviewales What is your use case for selecting Parquet 2.4?

OK, do we actually need this feature, or just need to clarify the documentation? I'm happy to address it.

Well, we may want to fix the issue anyway, since NANOS should not be emitted when a Parquet version earlier than 2.6 is selected.

Well, we may want to fix the issue anyway, since NANOS should not be emitted when a Parquet version earlier than 2.6 is selected.

Fine. I'll submit a PR for this when I get a chance. I'll reuse the coerce_timestamps and allow_truncated_timestamps to handle this, and update the docs.

@fenfeng9

@fenfeng9 I opened #53161 following your analysis and the agreed approach of reusing coerce_timestamps and allow_truncated_timestamps for time types. I reproduced the behavior, added the schema and value conversion changes with focused regressions, and credited your analysis in the PR description.

@gitedmond

@gitedmond I’ve been busy lately and forgot to follow up on this. Thanks for taking care of this issue.

Navigation Menu

Repository navigation

[Python][Parquet] pyarrow.parquet.write_table ignores version and coerce_timestamps arguments for Time64 fields #50142

Describe the bug, including details regarding any error messages, version, and platform.

Parquet 2.4

Parquet 1.0

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python][Parquet] pyarrow.parquet.write_table ignores version and coerce_timestamps arguments for Time64 fields #50142

Description

Describe the bug, including details regarding any error messages, version, and platform.

Parquet 2.4

Parquet 1.0

Component(s)

Activity

fenfeng9 commented on Jun 18, 2026

fenfeng9 commented on Jun 18, 2026

pitrou commented on Jun 23, 2026

fenfeng9 commented on Jun 23, 2026

pitrou commented on Jun 23, 2026

fenfeng9 commented on Jun 23, 2026

pitrou commented on Jun 23, 2026

fenfeng9 commented on Jun 23, 2026

pitrou commented on Jun 23, 2026

fenfeng9 commented on Jun 23, 2026

gitedmond commented on Oct 11, 2026

fenfeng9 commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions