Skip to content

fix(dlt): compare dlt version semantically for destination client#5890

Open
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:fix/dlt-version-parse
Open

fix(dlt): compare dlt version semantically for destination client#5890
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:fix/dlt-version-parse

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

The dlt destination-client selection in sqlmesh/integrations/dlt.py gated the
1.10.0 API change with a string comparison:

if dlt.__version__ >= "1.10.0":
    client = pipeline.destination_client()
else:
    client = pipeline._sql_job_client(schema)  # type: ignore

String comparison is lexicographic, not semantic, so it is wrong for any version
with a double-digit minor. For example "1.9.0" >= "1.10.0" evaluates to True
(the character '9' sorts after '1'), and likewise for "1.2.0" and
"1.5.10". The result is that every 1.x version takes the >= 1.10.0 branch,
and the pre-1.10 _sql_job_client fallback (added in #4223 for compatibility
with older dlt) is unreachable.

This is currently latent rather than crashing, because dlt versions below 1.10.0
happen to expose destination_client() as well. But the version guard is
silently incorrect and the intended fallback never runs.

The fix uses packaging.version.parse for a semantic comparison, mirroring the
idiom already used in sqlmesh/core/config/connection.py
(version.parse(...) < version.parse("...")). packaging is already a
top-level dependency, so no new dependency is introduced.

The client selection is extracted into a small _get_destination_client helper
so the version gate can be unit tested without a live dlt pipeline.

Test Plan

  • Added test_get_destination_client_version_gate in
    tests/integrations/test_dlt.py: a parametrized, pure-Python test that
    monkeypatches dlt.__version__ and asserts 1.9.0, 1.2.0, 1.5.10 select
    _sql_job_client while 1.10.0, 1.28.1, 2.0.0 select
    destination_client. It fails under the old string comparison for the pre-1.10
    cases and passes after the fix. No warehouse or live dlt pipeline required.
  • pytest tests/integrations/test_dlt.py -> 7 passed.
  • make style (ruff check, ruff format, mypy) clean.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

The dlt destination-client gate compared the version as a string
(`dlt.__version__ >= "1.10.0"`). String comparison is lexicographic, so
it wrongly evaluates "1.9.0" >= "1.10.0" (and "1.2.0", "1.5.10", ...) as
True. That sent every 1.x version down the >=1.10.0 branch and made the
pre-1.10 `_sql_job_client` fallback unreachable.

Use `packaging.version.parse` for a semantic comparison, mirroring the
existing idiom in sqlmesh/core/config/connection.py. The client selection
is extracted into a small `_get_destination_client` helper so the version
gate can be unit tested without a live dlt pipeline.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>

@StuffbyYuki StuffbyYuki left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!

Given destination_client() exists on all realistically supported dlt versions and is what everyone already hits due to the string-compare bug, would you consider simplifying to always use destination_client() (or pinning dlt>=1.10) instead of maintaining a version gate?

Let me know what you think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants