Skip to content

feat(sql): lower explicit temporal aggregates - #369

Open
zzylol wants to merge 2 commits into
mainfrom
feat/sql-temporal-aggregates
Open

feat(sql): lower explicit temporal aggregates#369
zzylol wants to merge 2 commits into
mainfrom
feat/sql-temporal-aggregates

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

ClickHouse SQL could express ordinary relational aggregates, but it could not describe PromQL-equivalent counter rate or counter increase semantics to ASAPPlanner. Rewriting these operations as sum or max changes their meaning. Applying either reducer after multiple counter series have been combined is also incorrect.

What

Adds two explicit SQL planning functions:

  • asap_rate(value, timestamp, window_ms)
  • asap_increase(value, timestamp, window_ms)

They lower to the existing canonical AggIntent::Rate and Increase forms. No QueryExpr, AggIntent, SDS, or post-ASAP wire type changes.

asap_last is deliberately unsupported: the current physical planner/runtime has no executable LastOverTime SDS path, so the SQL frontend fails closed rather than advertising an acceleration path it cannot execute.

How

The SQL function catalog registers fixed-arity pass-through UDAFs. The SQL frontend validates that:

  • value is a numeric, non-time column;
  • timestamp is the input schema's declared time index;
  • window_ms is a positive integer literal;
  • GROUP BY exactly matches a declared series identity.

The minimal series-identity contract uses existing Schema::unique_keys without changing its meaning: a row-unique key must include the declared time-index column; removing the time index yields the series key. SQL GROUP BY must match that key by resolved column ID. This rejects absent/partial identities, grouping by value/time, duplicate qualified/unqualified references to the same column, and schemas whose unique keys do not include time.

After validation, the frontend projects explicit SQL columns into (ts, value, series labels...), wraps them in the existing TimeRange, and emits the existing Reduction::PerEntity aggregate. Cross-series reduction composes in an outer query, such as SELECT sum(v) FROM (...).

Before this PR

SELECT service, asap_rate(value, ts, 300000) ... GROUP BY service failed as an unknown function. Ordinary SQL aggregate rewrites could not preserve reset-aware rate semantics or prove that samples from distinct counters stayed separate.

After this PR

Given catalog metadata time_index = ts and unique_keys = [[ts, service]], the same query lowers as:

Project(service, value AS v)
  Aggregate(PerEntity, Rate)
    TimeRange(300s)
      Project(ts AS ts, input_value AS value, service)
        Scan/Filter(samples)

ASAP-aware physical mapping produces the existing ExactAggregate(Rate) summary. asap_increase similarly produces ExactAggregate(Increase). Project, source Filter, derived-table Filter, and outer relational Aggregate nodes remain in the plan.

Evidence

The end-to-end integration test feeds both functions through ClickHouse SQL parsing, canonical lowering, and ASAP-aware mapping and observes the matching shared exact physical summary family with Reduction::PerEntity over the retained 300-second range.

Verification

  • Unit tests: cargo test -p asap-sql-function-catalog; cargo test -p asap-frontend-sql.
  • End-to-end tests: cargo test -p asap-integration-tests --test sql_to_post_asap.
  • Other checks: cargo fmt --all -- --check; cargo clippy -p asap-frontend-sql -p asap-sql-function-catalog -p asap-integration-tests --tests -- -D warnings; git diff --check.

New tests cover rate/increase lowering, complete and partial multi-series identities, empty grouping, qualified references resolved by column identity, grouping by value/time, duplicate resolved grouping columns, invalid timestamp/window arguments, mixed reducers, Project/Filter/outer-Aggregate nesting, explicit Last failure, and both functions' physical summary selection.

Architectural decisions

The bridge uses existing canonical nodes and physical families. SQL-only syntax recognition, identity proof, and column adaptation remain inside the SQL frontend.

The functions are explicit because standard ClickHouse SQL has no aggregate with PromQL-compatible reset-aware/extrapolated rate semantics. argMax(value, timestamp) remains the existing row-selector extension: it has no explicit range and is not equivalent to last_over_time.

Limitations and follow-up

  • One temporal reducer is supported per DataFusion Aggregate node. Outer relational queries compose cross-series reductions.
  • Arguments must be bare columns and the window must be a positive integer literal in milliseconds.
  • Catalogs must publish a row-unique key containing the time index. Tables without that proof fail closed.
  • These are planner bridge functions. Native ClickHouse exact fallback must execute an equivalent native SQL query rather than forwarding asap_* unchanged.
  • Last-over-time remains unavailable until its physical accumulator/readout is implemented end to end.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant