Skip to content

String-to-timestamp cast diverges from Spark's parseTimestampString: rejects 1-digit segments and empty fractions, accepts zone suffixes on date-only strings and 7-digit years #5674

Description

@peterxcli

Describe the bug

Comet's string→timestamp parser is a fixed list of anchored regexes (native/spark-expr/src/conversion_funcs/string.rs:1646-1660) plus an offset-suffix fallback (:1471-1475, :1567-1635). Spark's SparkDateTimeUtils.parseTimestampString is a segment scanner with per-segment digit rules (isValidDigits). They disagree in four ways, all on the default-on path, for both TIMESTAMP and TIMESTAMP_NTZ (LEGACY returns NULL, ANSI raises CAST_INVALID_INPUT):

  1. Month/day/hour/minute/second must be exactly 2 digits in Comet; Spark accepts 1–2 digits: '2020-1-1', '2020-01-01 12:34:5' → Spark valid, Comet NULL / error.
  2. Empty fraction: '2020-01-01 12:34:56.' → Spark valid, Comet NULL (RE_MICROSECOND requires \.\d+).
  3. Zone suffix on a date-only string: '2020-10-01Z', '2020-01-01+05:30', '2020-10-01 UTC' → Spark NULL (a zone is only allowed after a time segment), Comet accepts them (midnight in that zone).
  4. 7-digit years: Spark's timestamp maxDigitsYear is 6 (only stringToDate allows 7); '0002020-01-01 00:00:00' → Spark NULL, Comet 2020-01-01 00:00:00.

Steps to reproduce

CREATE TABLE t USING parquet AS SELECT * FROM VALUES
  ('2020-1-1'), ('2020-01-01 12:34:5'), ('2020-01-01 12:34:56.'), ('2020-10-01Z'), ('0002020-01-01 00:00:00') AS t(s);
SELECT s, CAST(s AS TIMESTAMP) FROM t;
input Spark (UTC) Comet
2020-1-1 2020-01-01 00:00:00 NULL
2020-01-01 12:34:5 2020-01-01 12:34:05 NULL
2020-01-01 12:34:56. 2020-01-01 12:34:56 NULL
2020-10-01Z NULL 2020-10-01 00:00:00
0002020-01-01 00:00:00 NULL 2020-01-01 00:00:00

With spark.sql.ansi.enabled=true the first three fail the whole query in Comet.

Expected behavior

The same accept/reject set as Spark.

Proposed solution

Port Spark's byte-scanning segment machine (segments 0–8 with per-segment digit-count validation; a zone suffix only after a time segment) for both the TZ and NTZ parsers. At minimum: relax month/day/hour/minute/second to \d{1,2}, allow an empty fraction, cap timestamp years at 6 digits, and only attempt suffix extraction when the remainder ends after a time segment. Add these strings to the cast suite — the fuzz alphabet 0123456789/:T contains neither - nor ., so it cannot generate them.

Additional context

Open PR #5130 re-implements the same 14 shapes as a byte classifier with bit-identical output, so it preserves these divergences. #5165 tracks sibling trimming divergences in the same parser; #3776 asks to port DateTimeUtilsSuite, which would surface some of these.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions