feat: add maintained population and temporal aggregate candidates - #404
Open
zzylol wants to merge 9 commits into
Open
feat: add maintained population and temporal aggregate candidates#404zzylol wants to merge 9 commits into
zzylol wants to merge 9 commits into
Conversation
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Backend issues ProjectASAP/ASAPQuery-backend#701 and ProjectASAP/ASAPQuery-backend#702 require candidate DAGs that preserve aggregate populations and expression accuracy.
What
Add language-independent maintained-population rules and typed readouts, distinct minimum state, realized temporal average, exact TopK over temporal values, and guarded relative division.
How
Planner rules produce executable typed DAGs and accuracy certificates. The backend lowers and prices these candidates without inventing the logical rewrites.
Before this PR
quantile(0.5, a),quantile(0.99, a),topk(1, a)andtopk(5, a)did not have a Planner-owned shared current-series state contract. The backend could not consume a typed candidate describing latest-value replacement, stale removal and lookback expiry.count(a)lowered to a distinct-value cardinality intent. Two live series with values7and7could therefore be represented as one distinct value instead of two series.min_over_time(a[5m])used the same exact-state kind as maximum.avg_over_time(a[5m])andtopk(5, sum_over_time(a[5m]))lacked the maintained candidate forms added here.quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m])lacked a checked, whole-expression relative-error candidate; two individual quantile certificates were insufficient.SELECT median(latency) FROM samplesandSELECT approx_percentile_cont(latency, 0.99) FROM samplescould not use the current-series rule to express shared table-row state. Table rows must not inherit PromQL's latest-series membership or five-minute lookback.After this PR
Typed candidates executable with companion backend PR #700:
quantile(0.5, a)andquantile(0.99, a)share one exact current-value population. Addingtopk(1, a)andtopk(5, a)shares the same population with a maximum-k cache of5; changing one series replaces its old value, and stale/expired series are removed.by(job)queries share within the same grouping; different groupings or selectors retain separate populations.sum(a),count(a)andavg(a)use typed current-population readouts. For two live series valued7and7,count(a)is2.min_over_time(a[5m])emits a distinct minimum accumulator;avg_over_time(a[5m])exposes maintained sum/count states; explicitly exacttopk(5, sum_over_time(a[5m]))selects the five largest finalized per-series sums.quantile_over_time(0.5, a[5m]) / quantile_over_time(0.9, a[5m]), Planner constructs a checked DDSketch division candidate with component accuracy below approximately 0.4975%. The backend checks the execution domain and falls back for cases such as a zero denominator.Shared SQL / PromQL rule and IR; SQL table-row deployment is not included yet:
These SQL queries now generate
MaintainPopulation(Rows) -> ReadPopulation(Quantile(q))candidates that share the table-row producer. Likewise,SELECT * FROM samples ORDER BY latency DESC LIMIT 1and the same query withLIMIT 5share a maximum-k population. SQL projections/aliases are preserved, and different filters/groupings do not share state.The table-row rule initially supports non-null Float64 value columns and single-measure aggregates. Backend #700 explicitly rejects deployment of this membership model until a row-update/deletion executor is available; existing SQL window-summary acceleration remains separate. These are candidate and correctness capabilities, not a claim of measured performance improvement.
Evidence
Companion backend PR ProjectASAP/ASAPQuery-backend#700 passes real Prometheus 3.5.0 comparison for the issue workloads, including moving windows and guarded fallback. Performance measurements and screenshots: not applicable.
Verification
cargo +1.98.0 test --workspacepassed, 1,097 tests.Latest generalization checks: all affected types/mapping/SQL package tests pass; six SQL frontend regressions cover shared quantiles, scalar readouts, maximum-k, group/filter separation and malformed-state rejection. Affected all-targets clippy passes with warnings denied.
Tests verify typed population sharing, row-count intent, minimum/maximum distinction, temporal-average realization, exact TopK inputs, executable DAG legality, and whole-expression relative-error composition.
Architectural decisions
The candidate declares runtime-checked division semantics. Rank-only KLL guarantees cannot establish relative value error. Exact temporal TopK adds support for explicit exact targets while preserving existing approximate heap candidate selection.
Limitations and follow-up
Requires matching backend lowering in PR 700. The backend deploys current-series populations; table-row state requires a row-update/deletion executor and is not silently mapped onto remote write. The initial table rule supports non-null Float64 values and single-measure aggregates. Existing SQL window summaries remain supported separately. Runtime must enforce finite operands, a nonzero divisor, and a normal finite quotient. These tests establish correctness, not a measured speedup.
Human review — do not complete with an agent