Describe the bug
Comet's string→timestamp parser is a fixed list of anchored regexes (native/spark-expr/src/conversion_funcs/string.rs:1646-1660) plus an offset-suffix fallback (:1471-1475, :1567-1635). Spark's SparkDateTimeUtils.parseTimestampString is a segment scanner with per-segment digit rules (isValidDigits). They disagree in four ways, all on the default-on path, for both TIMESTAMP and TIMESTAMP_NTZ (LEGACY returns NULL, ANSI raises CAST_INVALID_INPUT):
- Month/day/hour/minute/second must be exactly 2 digits in Comet; Spark accepts 1–2 digits:
'2020-1-1', '2020-01-01 12:34:5' → Spark valid, Comet NULL / error.
- Empty fraction:
'2020-01-01 12:34:56.' → Spark valid, Comet NULL (RE_MICROSECOND requires \.\d+).
- Zone suffix on a date-only string:
'2020-10-01Z', '2020-01-01+05:30', '2020-10-01 UTC' → Spark NULL (a zone is only allowed after a time segment), Comet accepts them (midnight in that zone).
- 7-digit years: Spark's timestamp
maxDigitsYear is 6 (only stringToDate allows 7); '0002020-01-01 00:00:00' → Spark NULL, Comet 2020-01-01 00:00:00.
Steps to reproduce
CREATE TABLE t USING parquet AS SELECT * FROM VALUES
('2020-1-1'), ('2020-01-01 12:34:5'), ('2020-01-01 12:34:56.'), ('2020-10-01Z'), ('0002020-01-01 00:00:00') AS t(s);
SELECT s, CAST(s AS TIMESTAMP) FROM t;
| input |
Spark (UTC) |
Comet |
2020-1-1 |
2020-01-01 00:00:00 |
NULL |
2020-01-01 12:34:5 |
2020-01-01 12:34:05 |
NULL |
2020-01-01 12:34:56. |
2020-01-01 12:34:56 |
NULL |
2020-10-01Z |
NULL |
2020-10-01 00:00:00 |
0002020-01-01 00:00:00 |
NULL |
2020-01-01 00:00:00 |
With spark.sql.ansi.enabled=true the first three fail the whole query in Comet.
Expected behavior
The same accept/reject set as Spark.
Proposed solution
Port Spark's byte-scanning segment machine (segments 0–8 with per-segment digit-count validation; a zone suffix only after a time segment) for both the TZ and NTZ parsers. At minimum: relax month/day/hour/minute/second to \d{1,2}, allow an empty fraction, cap timestamp years at 6 digits, and only attempt suffix extraction when the remainder ends after a time segment. Add these strings to the cast suite — the fuzz alphabet 0123456789/:T contains neither - nor ., so it cannot generate them.
Additional context
Open PR #5130 re-implements the same 14 shapes as a byte classifier with bit-identical output, so it preserves these divergences. #5165 tracks sibling trimming divergences in the same parser; #3776 asks to port DateTimeUtilsSuite, which would surface some of these.
Describe the bug
Comet's string→timestamp parser is a fixed list of anchored regexes (
native/spark-expr/src/conversion_funcs/string.rs:1646-1660) plus an offset-suffix fallback (:1471-1475,:1567-1635). Spark'sSparkDateTimeUtils.parseTimestampStringis a segment scanner with per-segment digit rules (isValidDigits). They disagree in four ways, all on the default-on path, for both TIMESTAMP and TIMESTAMP_NTZ (LEGACY returns NULL, ANSI raisesCAST_INVALID_INPUT):'2020-1-1','2020-01-01 12:34:5'→ Spark valid, Comet NULL / error.'2020-01-01 12:34:56.'→ Spark valid, Comet NULL (RE_MICROSECONDrequires\.\d+).'2020-10-01Z','2020-01-01+05:30','2020-10-01 UTC'→ Spark NULL (a zone is only allowed after a time segment), Comet accepts them (midnight in that zone).maxDigitsYearis 6 (onlystringToDateallows 7);'0002020-01-01 00:00:00'→ Spark NULL, Comet2020-01-01 00:00:00.Steps to reproduce
2020-1-12020-01-01 12:34:52020-01-01 12:34:56.2020-10-01Z0002020-01-01 00:00:00With
spark.sql.ansi.enabled=truethe first three fail the whole query in Comet.Expected behavior
The same accept/reject set as Spark.
Proposed solution
Port Spark's byte-scanning segment machine (segments 0–8 with per-segment digit-count validation; a zone suffix only after a time segment) for both the TZ and NTZ parsers. At minimum: relax month/day/hour/minute/second to
\d{1,2}, allow an empty fraction, cap timestamp years at 6 digits, and only attempt suffix extraction when the remainder ends after a time segment. Add these strings to the cast suite — the fuzz alphabet0123456789/:Tcontains neither-nor., so it cannot generate them.Additional context
Open PR #5130 re-implements the same 14 shapes as a byte classifier with bit-identical output, so it preserves these divergences. #5165 tracks sibling trimming divergences in the same parser; #3776 asks to port
DateTimeUtilsSuite, which would surface some of these.