Skip to content

feat: add maintained population and temporal aggregate candidates - #404

Open
zzylol wants to merge 9 commits into
mainfrom
feat/current-series-rules
Open

feat: add maintained population and temporal aggregate candidates#404
zzylol wants to merge 9 commits into
mainfrom
feat/current-series-rules

Conversation

@zzylol

@zzylol zzylol commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

Backend issues ProjectASAP/ASAPQuery-backend#701 and ProjectASAP/ASAPQuery-backend#702 require candidate DAGs that preserve aggregate populations and expression accuracy.

What

Add language-independent maintained-population rules and typed readouts, distinct minimum state, realized temporal average, exact TopK over temporal values, and guarded relative division.

How

Planner rules produce executable typed DAGs and accuracy certificates. The backend lowers and prices these candidates without inventing the logical rewrites.

Before this PR

  • quantile(0.5, a), quantile(0.99, a), topk(1, a) and topk(5, a) did not have a Planner-owned shared current-series state contract. The backend could not consume a typed candidate describing latest-value replacement, stale removal and lookback expiry.
  • count(a) lowered to a distinct-value cardinality intent. Two live series with values 7 and 7 could therefore be represented as one distinct value instead of two series.
  • min_over_time(a[5m]) used the same exact-state kind as maximum. avg_over_time(a[5m]) and topk(5, sum_over_time(a[5m])) lacked the maintained candidate forms added here.
  • quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]) lacked a checked, whole-expression relative-error candidate; two individual quantile certificates were insufficient.
  • SQL SELECT median(latency) FROM samples and SELECT approx_percentile_cont(latency, 0.99) FROM samples could not use the current-series rule to express shared table-row state. Table rows must not inherit PromQL's latest-series membership or five-minute lookback.

After this PR

Typed candidates executable with companion backend PR #700:

  • quantile(0.5, a) and quantile(0.99, a) share one exact current-value population. Adding topk(1, a) and topk(5, a) shares the same population with a maximum-k cache of 5; changing one series replaces its old value, and stale/expired series are removed. by(job) queries share within the same grouping; different groupings or selectors retain separate populations.
  • sum(a), count(a) and avg(a) use typed current-population readouts. For two live series valued 7 and 7, count(a) is 2.
  • min_over_time(a[5m]) emits a distinct minimum accumulator; avg_over_time(a[5m]) exposes maintained sum/count states; explicitly exact topk(5, sum_over_time(a[5m])) selects the five largest finalized per-series sums.
  • For a 1% relative-error target on quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]), Planner constructs a checked DDSketch division candidate with component accuracy below approximately 0.4975%. The backend checks the execution domain and falls back for cases such as a zero denominator.

Shared SQL / PromQL rule and IR; SQL table-row deployment is not included yet:

SELECT median(latency) FROM samples;
SELECT approx_percentile_cont(latency, 0.99) FROM samples;

These SQL queries now generate MaintainPopulation(Rows) -> ReadPopulation(Quantile(q)) candidates that share the table-row producer. Likewise, SELECT * FROM samples ORDER BY latency DESC LIMIT 1 and the same query with LIMIT 5 share a maximum-k population. SQL projections/aliases are preserved, and different filters/groupings do not share state.

The table-row rule initially supports non-null Float64 value columns and single-measure aggregates. Backend #700 explicitly rejects deployment of this membership model until a row-update/deletion executor is available; existing SQL window-summary acceleration remains separate. These are candidate and correctness capabilities, not a claim of measured performance improvement.

Evidence

Companion backend PR ProjectASAP/ASAPQuery-backend#700 passes real Prometheus 3.5.0 comparison for the issue workloads, including moving windows and guarded fallback. Performance measurements and screenshots: not applicable.

Verification

  • Earlier issue-workload validation — unit/integration/doc tests: cargo +1.98.0 test --workspace passed, 1,097 tests.
  • End-to-end tests: companion backend compatibility suite passed 14 tests, with one existing Collector-schema test ignored.
  • Other checks: workspace/all-targets clippy passed with warnings denied; formatting passed.

Latest generalization checks: all affected types/mapping/SQL package tests pass; six SQL frontend regressions cover shared quantiles, scalar readouts, maximum-k, group/filter separation and malformed-state rejection. Affected all-targets clippy passes with warnings denied.

Tests verify typed population sharing, row-count intent, minimum/maximum distinction, temporal-average realization, exact TopK inputs, executable DAG legality, and whole-expression relative-error composition.

Architectural decisions

The candidate declares runtime-checked division semantics. Rank-only KLL guarantees cannot establish relative value error. Exact temporal TopK adds support for explicit exact targets while preserving existing approximate heap candidate selection.

Limitations and follow-up

Requires matching backend lowering in PR 700. The backend deploys current-series populations; table-row state requires a row-update/deletion executor and is not silently mapped onto remote write. The initial table rule supports non-null Float64 values and single-measure aggregates. Existing SQL window summaries remain supported separately. Runtime must enforce finite operands, a nonzero divisor, and a normal finite quotient. These tests establish correctness, not a measured speedup.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

@zzylol zzylol changed the title Add rules and typed IR for shared current-series quantiles and TopK Add typed current-series and temporal aggregate candidates Sep 12, 2026
@zzylol zzylol changed the title Add typed current-series and temporal aggregate candidates Add typed maintained-population and temporal aggregate candidates Sep 13, 2026
@zzylol zzylol changed the title Add typed maintained-population and temporal aggregate candidates feat: add maintained population and temporal aggregate candidates Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant