fix(dlt): use timestamp macros for incremental model time filter#5867
Open
anxkhn wants to merge 1 commit into
Open
fix(dlt): use timestamp macros for incremental model time filter#5867anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
The DLT-generated INCREMENTAL_BY_TIME_RANGE model casts its time column to a timestamp (TO_TIMESTAMP(...)) but filtered it with @start_ds AND @end_ds. Those categorical date macros both render at midnight, so on a single-day run any row past 00:00:00 was excluded, yielding an empty range. Use the inclusive timestamp macros @start_ts AND @end_ts instead, matching make_inclusive's documented contract for timestamp columns. Fixes SQLMesh#5689 Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The DLT integration generates
INCREMENTAL_BY_TIME_RANGEmodels whose time column is cast to a timestamp:but it filtered that column with the categorical date macros:
@start_ds/@end_dsrender bare dates (midnight on both ends), whilemake_inclusive(sqlmesh/utils/date.py) defines the inclusive range as00:00:00 .. 23:59:59.999999. The timestamp form of that range is what@start_ts/@end_tsemit. On a single-day run (@start_ds == @end_ds) both bounds collapse to midnight, so any row with a non-midnight timestamp (e.g.21:30:15) is excluded and the result is empty.This matches the diagnosis and suggested fix in #5689: filter a timestamp column with
@start_ts AND @end_ts.Fixes #5689
Test Plan
tests/integrations/test_dlt.py: a focused unit test that callsgenerate_incremental_model(...)and asserts the generatedWHEREusesBETWEEN @start_ts AND @end_ts(and no longer@start_ds/@end_ds). No warehouse or dlt package needed. Confirmed failing-first: reverting the source line back to@start_ds/@end_dsmakes it fail; the fix makes it pass.tests/cli/test_cli.py(the DLT pipeline template tests).test_dlt_filesystem_pipeline,test_dlt_pipeline,test_dlt_pipeline_errors-> 3 passed (dlt 1.28.1, filesystem destination, DuckDB default, no warehouse).make style(ruff format/check + mypy) clean on changed files.Checklist
make styleand fixed any issuesmake fast-test)git commit -s) per the DCO