fix(planning): unify costed window selection and compatible sum pane sharing - #699
Merged
Conversation
…adence
A snapshot that prices no window implementation for a query got one
synthesized here, hardcoded to a lookback-wide tumbling window with a
lookback-wide pane. The query's own evaluation cadence was parsed a few
lines above, used for lifecycle costing, and then dropped.
The result is a plan whose answer only changes once per window. A
workload of two 5m-lookback quantiles evaluated every 30s planned as
window 300 / slide 300 / tumbling / Pane{300}: the state advances every
five minutes while the queries run every thirty seconds. The retained
count followed it down to 2, which is arithmetically right for a
300s pane and useless for a 30s cadence.
Derive the shape instead. When the cadence is shorter than the window
and divides it, slide by one evaluation interval and store panes of that
width; otherwise keep the previous tumbling shape. The same workload now
plans as window 300 / slide 30 / sliding / Pane{30}, retaining 11 states
-- ten panes covering the lookback plus the one still filling.
This derives a shape, never a price. `ImplementationCostEvidence` is
measured evidence: its `weighted_cost` doc puts pricing update CPU,
query-time merges, retained memory, storage, scans and network on the
evidence producer. So this does not synthesize a second candidate to
rank against the first -- a lone candidate is chosen by a `min_by` over
one element, where the cost cannot change the outcome. Ranking Pane
against FullWindow still requires a snapshot that supplies both with
their own priced evidence, and `window_candidates` already carries them
untouched when it does.
Two existing assertions encoded the old default's pane width. The
snapshot they load evaluates every 10s over a 1m lookback, so its stored
pane is now 10s; both readouts keep their 1m `readout_lookback_ms`, and
every other property those tests assert is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The derived fallback built a single candidate from `time_selection.lookback`. A query carries one range selector per operand, and they need not agree with the declared lookback or with each other. The consequence is silent. Under `hybrid_execution`, a selected state whose window has no candidate is filtered out of selection entirely, and the surviving candidates are then narrowed to the selected window before validation. So `sum(sum_over_time(a[1m])) / sum(sum_over_time(b[5m]))` under a 1m lookback kept `a` and dropped `b` to exact execution, with nothing reported and no error raised. Derive one candidate per distinct range-selector window instead, each shaped by the evaluation cadence as before. `window_implementation_id` reaches lifecycle estimates and cost manifests, so a query with several windows suffixes it per window; a single-window query keeps the snapshot's identity untouched. Three assertions changed, all of them pinning the previous artifact rather than intended behavior: - `composable_binary_binds_independent_source_windows` hand-supplied a 5m candidate so `b` would survive. The derivation now covers it, so the workaround is gone and both operands keep their own range. - `composable_binary_retains_summary_sibling_of_prometheus_filtered_subtree` asserted a filtered denominator stays a typed residual. Nothing about the filter forced that -- the missing 5m candidate did. It now gets a summary over its own filtered population, which the renamed `composable_binary_summarizes_each_prometheus_filtered_operand` pins, filter and range together. - `counter_materialization_manifest_prices_owned_state_and_distinct_native_alternative` required an `exact_backend` component. Its 5m operand landed there for the same reason and is now summarized. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sliding window has two legal physical layouts, and they trade opposite
resources. `Pane { P }` takes each sample once and composes `W / P`
states per read; `FullWindow` takes each sample into all `ceil(W / S)`
overlapping windows and is read whole. Which is cheaper depends on how
hard the source is pushing, and nothing was comparing them: the
derivation emitted only the pane form, and a lone candidate is selected
by a `min_by` over one element.
Emit both for a sliding shape and price each from the snapshot's own
lifecycle unit costs.
`ImplementationCostEvidence` is normally measured evidence, so becoming
its producer here is only sound if nothing is invented. Every unit cost
is supplied (`implementation.lifecycle_costs`), and every multiplier is
a structural count that follows from the layout's definition: states
sealed per horizon, update fanout, finalizations per read, and
`retained_state_count` -- the same function that fills
`num_aggregates_to_retain`, so quote and plan cannot disagree.
Concurrent in-flight full windows are not charged twice; the fanout term
already prices that write amplification.
The byte fields stay exactly as the snapshot supplied them. State size
needs sketch parameters that do not exist yet at this point, and
guessing them would be the fabrication this avoids. Only `cpu_cost` and
`weighted_cost` are derived, and only those two are read -- by
`validate_window_implementations` and by the ranking `min_by`.
`model_version` records that the quote is derived.
Against the planning snapshot's own evidence the crossover lands where
it should: idle, the full window wins because its fanout is free and the
pane's ten merges per read are not; at the declared 100 updates/s the
tenfold fanout dominates and panes win. Tumbling shapes are unchanged --
the validator pairs them only with `Pane`, so there is no alternative to
rank and their identity keeps the snapshot's label.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
zzylol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrates #696, #697 and #698 into this PR, targeting
main. Their commits are preserved. Depends on ProjectASAP/ASAPPlanner#403; workspace Planner dependencies and Cargo.lock pinca7546de792d74aee8231e9a1100ca893d9e86d3.Why
Queries with multiple lookbacks need appropriate, costed window candidates. Compatible SUM windows should share maintenance without changing logical readouts, and deployment must compare complete executable workload costs consistently.
What
How
Planner compares shared versus independent producer/read costs. Backend certifies physical compatibility and remaps selected producers, bindings and lifecycle consumers. Snapshot v2 deployment requires complete cost evidence; candidate discovery remains available before pricing. Synthetic fixture prices are confined to test code.
Before this PR
Different expression windows could miss candidates or create duplicate producers. Some derived summaries failed compilation, window residency was underpriced, and deployment had separate quoted/unquoted paths. Combining the original branches left 29 data-plane tests failing due to missing cost evidence; correcting the quotes exposed an inconsistent window type in one operator fixture.
After this PR
sum_over_time(a[1m]) / sum_over_time(a[10m])can select one shared pane producer through complete-cost deployment selection, with separate 1m and 10m readouts. At a one-minute cadence, the denominator merges ten 60-second panes. Different metrics, predicates and phases remain independent. Supported derived cohorts compile, and the combined library suites pass.Verification
10/55and11/65from shared pane sums.cargo +1.98.0 clippy -p control_plane -p data_plane --all-targets --locked: passed with existing warnings.8c03d7cthrough a local Cargo path override, matching CI's dependency revision; Planner uses the published Git pin without a local override.Merge and migration
Merge ProjectASAP/ASAPPlanner#403 first, then merge this integrated backend PR after CI and human review. #696, #697 and #698 are included here and need no separate merge. Deployment callers must supply snapshot v2 and complete quotes generated for the final backend/Planner revisions and workload manifests.
Reuse covers compatible selected raw SUM/MultipleSum panes. It does not enumerate every pane width, jointly reconsider selected FullWindow layouts, or merge derived/non-additive state.
Human review — do not complete with an agent