Skip to content

fix: reject Parquet timestamp columns read as non-timestamp types - #7227

Open
ump45nose wants to merge 2 commits into
apache:mainfrom
ump45nose:fix/reject-timestamp-mismatched-reads
Open

ump45nose wants to merge 2 commits into
apache:mainfrom
ump45nose:fix/reject-timestamp-mismatched-reads

Conversation

@ump45nose

@ump45nose ump45nose commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #7214.

Rationale for this change

Spark's vectorized reader has no updater for these reads and fails with PARQUET_COLUMN_DATA_TYPE_MISMATCH, but the native scan casts the values instead:

  • an INT96 or INT64 timestamp column read as string, byte/short/int, float/double or decimal (returns text, raw numbers or NULL);
  • an INT96 column read as bigint (returns microseconds since the epoch). An INT64 timestamp read as bigint is fine in Spark and stays allowed.
  • a DATE column read as string, float or decimal (returns text or numbers). Spark's INT32 case has no string or float branch, and isDecimalTypeMatched refuses a date annotation. Reading a DATE as int stays allowed. This one isn't in Native Parquet scan reads timestamp columns as string, int, double or decimal, and INT96 as bigint, where Spark rejects them #7214; it came up in review.

What changes are included in this PR?

  • spark_has_updater rejects these reads through its existing rules: Date32 and Timestamp join the primitive types that can't be read as a string or binary, and the (Timestamp, Int8 | Int16 | Int32 | Float32 | Float64 | Decimal*) and (Date32, Float32 | Decimal*) pairs join the is_spark_rejected_conversion table. These are deferred like the other getUpdater rejections (RejectOnNonEmpty).
  • INT96 and INT64 timestamps have the same Arrow type by the time the check runs, so the INT96 bigint rule needs the Parquet type. Like TIMESTAMP(NANOS) in fix: reject TIMESTAMP(NANOS) Parquet columns read as timestamp #7089, the footer check (mark_parquet_leaves) now marks INT96 leaves with a private INT96_TIMESTAMP_KEY, and files with INT96 leaves take the deferred check path even without row filters. To carry both markers, the nanos_timestamp: bool parameter became a small ParquetLeaf enum (Other, NanosTimestamp, Int96).

How are these changes tested?

  • parquet_timestamp_read_as_non_timestamp_errors: a TIMESTAMP(MICROS) column, with and without a timezone, read as string, int, double and decimal(20,0) fails with the Spark-shaped error; read as bigint it still works.
  • parquet_date_read_as_string_float_or_decimal_errors: a DATE column read as string, float and decimal(20,0) fails with the Spark-shaped error; read as int it is still accepted.
  • int96_read_as_bigint_errors: an INT96 file read as bigint fails, with row filters on and off.
  • int96_bigint_rejection_keys_on_the_parquet_type: a marked INT96 leaf is rejected as bigint and still accepted as timestamp; an unmarked timestamp leaf is still accepted as bigint.

All of these fail without the change. I ran the parquet:: lib tests in the datafusion-comet crate (332 passed, 1 ignored) and cargo clippy -p datafusion-comet --all-targets -- -D warnings. In the full lib run, only the native UDF tests failed, because I hadn't built comet-test-udfs. I did not run the Spark/JVM suites.

AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant. The part most worth checking is the INT96 marker, which uses the same footer-marking path as TIMESTAMP(NANOS).

Spark's vectorized reader has no updater that reads an INT96 or INT64
timestamp column as a string, a narrower integer, a float or a decimal,
or an INT96 column as bigint, so it fails with a column type mismatch.
The native scan cast the values instead. Reject these conversions the
same way as the other getUpdater rejections.

Closes apache#7214.
@github-actions github-actions Bot added bug Something isn't working area:scan Parquet scan / data reading labels Oct 26, 2026

@comphead comphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the new rules against ParquetVectorUpdaterFactory in 3.4.3, 3.5.9, 4.0.4 and 4.1.3. An INT64 timestamp has only the long, timestamp and day-time interval updaters, and isDecimalTypeMatched refuses a timestamp annotation, so the rejected targets match. One point on where the rule lives, which also covers a DATE gap of the same kind.

Comment thread native/core/src/parquet/schema_adapter.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:scan Parquet scan / data reading bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native Parquet scan reads timestamp columns as string, int, double or decimal, and INT96 as bigint, where Spark rejects them

2 participants