Skip to content

fix(dlt): use timestamp macros for incremental model time filter#5867

Open
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:loop/sqlmesh__003
Open

fix(dlt): use timestamp macros for incremental model time filter#5867
anxkhn wants to merge 1 commit into
SQLMesh:mainfrom
anxkhn:loop/sqlmesh__003

Conversation

@anxkhn

@anxkhn anxkhn commented Jun 30, 2026

Copy link
Copy Markdown

Description

The DLT integration generates INCREMENTAL_BY_TIME_RANGE models whose time column is cast to a timestamp:

TO_TIMESTAMP(CAST(c._dlt_load_id AS DOUBLE)) as _dlt_load_time

but it filtered that column with the categorical date macros:

WHERE
  TO_TIMESTAMP(CAST(c._dlt_load_id AS DOUBLE)) BETWEEN @start_ds AND @end_ds

@start_ds / @end_ds render bare dates (midnight on both ends), while make_inclusive (sqlmesh/utils/date.py) defines the inclusive range as 00:00:00 .. 23:59:59.999999. The timestamp form of that range is what @start_ts / @end_ts emit. On a single-day run (@start_ds == @end_ds) both bounds collapse to midnight, so any row with a non-midnight timestamp (e.g. 21:30:15) is excluded and the result is empty.

This matches the diagnosis and suggested fix in #5689: filter a timestamp column with @start_ts AND @end_ts.

Fixes #5689

Test Plan

  • Added tests/integrations/test_dlt.py: a focused unit test that calls generate_incremental_model(...) and asserts the generated WHERE uses BETWEEN @start_ts AND @end_ts (and no longer @start_ds / @end_ds). No warehouse or dlt package needed. Confirmed failing-first: reverting the source line back to @start_ds / @end_ds makes it fail; the fix makes it pass.
  • Updated the four hard-coded expected-SQL strings in tests/cli/test_cli.py (the DLT pipeline template tests). test_dlt_filesystem_pipeline, test_dlt_pipeline, test_dlt_pipeline_errors -> 3 passed (dlt 1.28.1, filesystem destination, DuckDB default, no warehouse).
  • make style (ruff format/check + mypy) clean on changed files.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

The DLT-generated INCREMENTAL_BY_TIME_RANGE model casts its time column
to a timestamp (TO_TIMESTAMP(...)) but filtered it with @start_ds AND
@end_ds. Those categorical date macros both render at midnight, so on a
single-day run any row past 00:00:00 was excluded, yielding an empty
range. Use the inclusive timestamp macros @start_ts AND @end_ts instead,
matching make_inclusive's documented contract for timestamp columns.

Fixes SQLMesh#5689

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BETWEEN @start_ds AND @end_ds results in an empty range

1 participant