Repository navigation
Conversation
Spark's vectorized reader has no updater that reads an INT96 or INT64 timestamp column as a string, a narrower integer, a float or a decimal, or an INT96 column as bigint, so it fails with a column type mismatch. The native scan cast the values instead. Reject these conversions the same way as the other getUpdater rejections. Closes apache#7214.
comphead
reviewed
Oct 26, 2026
comphead
left a comment
Contributor
There was a problem hiding this comment.
I checked the new rules against ParquetVectorUpdaterFactory in 3.4.3, 3.5.9, 4.0.4 and 4.1.3. An INT64 timestamp has only the long, timestamp and day-time interval updaters, and isDecimalTypeMatched refuses a timestamp annotation, so the rejected targets match. One point on where the rule lives, which also covers a DATE gap of the same kind.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #7214.
Rationale for this change
Spark's vectorized reader has no updater for these reads and fails with
PARQUET_COLUMN_DATA_TYPE_MISMATCH, but the native scan casts the values instead:string,byte/short/int,float/doubleordecimal(returns text, raw numbers or NULL);bigint(returns microseconds since the epoch). An INT64 timestamp read asbigintis fine in Spark and stays allowed.string,floatordecimal(returns text or numbers). Spark's INT32 case has no string or float branch, andisDecimalTypeMatchedrefuses a date annotation. Reading a DATE asintstays allowed. This one isn't in Native Parquet scan reads timestamp columns as string, int, double or decimal, and INT96 as bigint, where Spark rejects them #7214; it came up in review.What changes are included in this PR?
spark_has_updaterrejects these reads through its existing rules:Date32andTimestampjoin the primitive types that can't be read as a string or binary, and the(Timestamp, Int8 | Int16 | Int32 | Float32 | Float64 | Decimal*)and(Date32, Float32 | Decimal*)pairs join theis_spark_rejected_conversiontable. These are deferred like the othergetUpdaterrejections (RejectOnNonEmpty).bigintrule needs the Parquet type. LikeTIMESTAMP(NANOS)in fix: reject TIMESTAMP(NANOS) Parquet columns read as timestamp #7089, the footer check (mark_parquet_leaves) now marks INT96 leaves with a privateINT96_TIMESTAMP_KEY, and files with INT96 leaves take the deferred check path even without row filters. To carry both markers, thenanos_timestamp: boolparameter became a smallParquetLeafenum (Other,NanosTimestamp,Int96).How are these changes tested?
parquet_timestamp_read_as_non_timestamp_errors: aTIMESTAMP(MICROS)column, with and without a timezone, read asstring,int,doubleanddecimal(20,0)fails with the Spark-shaped error; read asbigintit still works.parquet_date_read_as_string_float_or_decimal_errors: a DATE column read asstring,floatanddecimal(20,0)fails with the Spark-shaped error; read asintit is still accepted.int96_read_as_bigint_errors: an INT96 file read asbigintfails, with row filters on and off.int96_bigint_rejection_keys_on_the_parquet_type: a marked INT96 leaf is rejected asbigintand still accepted astimestamp; an unmarked timestamp leaf is still accepted asbigint.All of these fail without the change. I ran the
parquet::lib tests in thedatafusion-cometcrate (332 passed, 1 ignored) andcargo clippy -p datafusion-comet --all-targets -- -D warnings. In the full lib run, only the native UDF tests failed, because I hadn't builtcomet-test-udfs. I did not run the Spark/JVM suites.AI disclosure: I wrote the change, the tests and this description with help from an AI coding assistant. The part most worth checking is the INT96 marker, which uses the same footer-marking path as
TIMESTAMP(NANOS).