Skip to content

fix: keep timezone-aware datetimes aware and UTC-normalise naive coercion - #76

Merged
hantmac merged 1 commit into
databendlabs:mainfrom
PlaidCloud:upstream-pr/datetime-tz-aware
Oct 19, 2026
Merged

hantmac merged 1 commit into
databendlabs:mainfrom
PlaidCloud:upstream-pr/datetime-tz-aware

Conversation

@simozzy

@simozzy simozzy commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Problem

DatabendDateTime.result_processor currently coerces every non-string value with an unconditional value.replace(tzinfo=None):

else:
    return value.replace(tzinfo=None)

The driver returns tz-aware (UTC) datetimes, so this does fix the naive-column case — but it has two side effects:

  1. It clobbers timezone=True columns. A DateTime(timezone=True) / TIMESTAMP WITH TIME ZONE column loses its tzinfo, even though self.timezone is available (inherited from sqltypes.DATETIME) to distinguish the two cases.
  2. It drops tzinfo without converting to UTC first. If the driver ever returns a tz-aware value carrying a non-UTC offset, replace(tzinfo=None) truncates the offset instead of converting it — yielding the wrong naive wall-clock (e.g. a +02:00 value comes back two hours off).

Fix

if value.tzinfo is not None and not self.timezone:
    return value.astimezone(datetime.timezone.utc).replace(tzinfo=None)
return value
  • timezone=False columns still yield naive values (so the compliance suite's naive round-trips hold), but the value is normalised to UTC before tzinfo is dropped, so the wall-clock is correct regardless of the driver's offset.
  • timezone=True columns are returned unchanged, keeping their awareness.
  • None and string inputs are handled exactly as before.

Tests

tests/unit/test_databend_dialect.py — server-less, calls result_processor directly:

  • tz-aware UTC input → naive output for a timezone=False column;
  • a non-UTC (+02:00) input is normalised to UTC, not merely truncated;
  • naive passes through unchanged, None is preserved, strings still parse;
  • a timezone=True column keeps its tzinfo.

…cion

DatabendDateTime.result_processor unconditionally did
value.replace(tzinfo=None), which has two problems:

- It strips tzinfo even for timezone=True columns, so a
  DateTime(timezone=True) / TIMESTAMP WITH TIME ZONE column loses its
  awareness.
- It drops tzinfo without converting to UTC first, so a tz-aware value
  carrying a non-UTC offset yields the wrong naive wall-clock.

Only coerce to naive for timezone=False columns, and convert to UTC
before dropping tzinfo so the wall-clock is correct regardless of the
driver's offset; timezone=True columns are returned unchanged. None and
string parsing are preserved. Adds a server-less unit test.
@simozzy

simozzy commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

CI failure is unrelated to this PR — it's a Databend nightly server flake.

This PR only changes DatabendDateTime.result_processor (tz handling) plus a server-less
unit test; neither goes near the sequence code path.

Both failures are in the compliance suite's SequenceTest, on a server-side error:

[SQL: select nextval(tab_id_seq)]
APIError: [3101]UnknownSequence: 'Unknown SequenceMeta 'tab_id_seq': get_sequence' [v0.34.0]

The test creates a sequence, then nextval() reports it as unknown — the server can't find a
sequence it just created. It's non-deterministic: the sibling PR run hours earlier hit the same
SequenceTest from the opposite side (SequenceAlreadyExists on create, unknown sequence at teardown).

Root cause: CI is pinned to databend:nightly. Last green main was 2026-02-28; these runs are on
nightly v1.2.920 / driver v0.34.0, which made sequence create/get/drop racy. Every open PR inherits it.

Suggest a re-run (may pass on retry) and/or pinning CI to a known-good Databend tag. Happy to file
an upstream issue for the sequence flake.

@hantmac hantmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed result processing and SQLAlchemy DateTime/TIMESTAMP dialect adaptation. No blocking findings. For datetime inputs, timezone=False now preserves the instant by normalizing to UTC before dropping tzinfo, while timezone=True retains the driver's aware value. None, naive datetime, and existing string parsing behavior are unchanged.

Validation on Python 3.11 with databend-driver 0.34.2:

  • The added regression test passes on SQLAlchemy 1.4.54 and 2.0.55.
  • Extra checks pass on both versions for DateTime and TIMESTAMP, timezone=True/False, six offsets (including half/quarter-hour offsets and date rollover), plus None/naive/string values.
  • Combined with #82: 20 unit tests pass; the same two failures (test_view_names and test_columns) reproduce on unchanged main. No new unit-test failures.

CI caveat: the historical failed run's logs return HTTP 410, and GitHub refuses reruns older than a month, so I cannot independently confirm the author's sequence-flake diagnosis. Fresh sibling CI on the same unchanged workflow currently fails even earlier: MinIO download returns 410 and Databend fails to start, before pytest (https://github.com/databendlabs/databend-sqlalchemy/actions/runs/39038774957). No live-server datetime integration run was completed in this review. These CI/bootstrap issues need separate maintenance.

Non-blocking existing limitation: timezone=True string results still follow the old naive parser, and this PR intentionally does not change datetime bind serialization. The approval applies to the scoped datetime-object result-processing fix, not a claim of end-to-end timezone support for every input path. Approving and merging.

@hantmac
hantmac merged commit 2f0aa65 into databendlabs:main Oct 19, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants