Audit ASAPQuery precompute-engine changes for backend parity

[milindsrivastava1997]

We need to audit if/which of these ASAPQuery precompute-engine changes are not in ASAPQuery-backend, then document the gaps and create follow-up work for required ports.

ASAPQuery PR inventory:

• docs(control_plane): design doc for BackendPlan wire format #389 feat(query-engine): SQL Top-K via CountMinSketchWithHeap ASAPQuery#389
• chore(promql_utilities): relocate Statistic/KeyByLabelNames to asap_types #397 refactor(query-engine): make sketch config params required in accumulator factory ASAPQuery#397
• feat(sketch_algebra): unify SketchKind onto asap_sketch::SummaryKind #400 fix(precompute): close idle/trailing windows for one-shot batch ingest (wall-clock fallback + shutdown force-close) ASAPQuery#400
• SummaryExecutor: permanent self-exclusions + upstream composition gaps leave some PromQL shapes always on the legacy/archive path #436 fix(precompute_engine): Use value_column from labels at ingest ASAPQuery#436
• backend-plan: construct BackendPlan from BackendStageConfig (Phase 1) #437 feat: add millisecond precision support for planner and query engine ASAPQuery#437
• fix(precompute): separate event and closure watermarks #483 fix(precompute): correctly parse configuration parameters from planner for CMS with heap ASAPQuery#483
• Add a shared authoritative SummaryCatalog #536 fix(precompute): keep wall-clock fallback from force-closing panes still under active ingest ASAPQuery#536
• Reference the shared SummaryCatalog from execution plans #537 test(precompute): add a wall-clock override hook to PrecomputeEngine ASAPQuery#537
• Bind physical-plan execution to the authoritative SummaryCatalog #538 test(precompute): add real-pipeline e2e regression test for #474 ASAPQuery#538
• fix: acknowledge maintenance outputs after publication #605 fix(query-engine): DeltaSetAggregator merge ordering, non-commutative fold, and get_keys() removal bug (#586) ASAPQuery#605
• Execute completed cohorts at every materialized maintenance frontier #665 fix(query-engine): honor CountMinSketch sum/count subtype ASAPQuery#665
• Compile actual finite multi-source maintenance plans #668 fix(precompute): route SetAggregator and DeltaSetAggregator, preserve deltas ASAPQuery#668
• Add offline cluster data exporter tool #704 fix(precompute): use open-closed evaluation windows ASAPQuery#704
• Lookback vs PromQL range selector: redundant window concepts (+ /query_range readout bug) #706 fix(planner,query-engine): recognize topk over temporal aggregations ASAPQuery#706
• #715 fix: reject invalid HLL precision and Min/Max subtype ASAPQuery#715
• #716 feat(query-engine): log precompute ingest throughput ASAPQuery#716
• #717 fix(query-engine): MultipleSum weights samples by 1 for count sub_type ASAPQuery#717
• #719 refactor: fold heap-CMS count_events into aggregation_sub_type ASAPQuery#719
• #721 refactor: remove Flink/Arroyo/legacy Kafka-ingest infra from asap-tools ASAPQuery#721

For each PR, identify whether its behavior already exists in ASAPQuery-backend, is intentionally superseded, or requires a backend change. Record supporting evidence and file or link follow-up issues for every required change.

[milindsrivastava1997]

Parity audit (2026-09-09)

Compared the listed ASAPQuery PRs against backend main at d5dc72f. This is a static source audit; no repository changes were made.

Already present or represented by the current backend contract

• feat(sketch_algebra): unify SketchKind onto asap_sketch::SummaryKind #400: idle wall-clock closure plus shutdown force-close.
• backend-plan: construct BackendPlan from BackendStageConfig (Phase 1) #437: millisecond window/query handling; ingest units are explicit.
• fix(precompute): separate event and closure watermarks #483: typed sketch dimensions and explicit heap weighting replace legacy parameter parsing.
• Add a shared authoritative SummaryCatalog #536: pane last-touch prevents wall-clock closure while ingest remains active.
• Reference the shared SummaryCatalog from execution plans #537/Bind physical-plan execution to the authoritative SummaryCatalog #538: deterministic fake-clock regression coverage exists at worker level.
• Add offline cluster data exporter tool #704: PromQL (start, end] behavior is present via promql_right_closed: state identity carries the convention and the worker assigns boundary samples accordingly.
• Lookback vs PromQL range selector: redundant window concepts (+ /query_range readout bug) #706: temporal Top-K and explicit heap weighting are planned and tested.
• #719: heap weighting is represented as parameters.weight_mode, rather than upstream subtype refactor.

Intentionally superseded / outside the backend contract

• chore(promql_utilities): relocate Statistic/KeyByLabelNames to asap_types #397: accumulator construction uses typed AccumulatorSpec rather than the legacy factory contract.
• fix: acknowledge maintenance outputs after publication #605 and Compile actual finite multi-source maintenance plans #668: SetAggregator/DeltaSetAggregator were retired in backend refactor(types): retire SetAggregator + DeltaSetAggregator family #264. They were a never-wired exact key-membership route for count(distinct ...); HLL is currently the sole supported approximate cardinality answerer. No key-only accumulator remains.
• #721: legacy Flink/Arroyo/Kafka infrastructure is not part of this backend supported precompute path.

Required follow-up work

1. Introduce WindowBoundaryType as the authoritative four-case contract: OpenOpen (start,end), OpenClosed (start,end], ClosedOpen [start,end), and ClosedClosed [start,end].

   • Migrate promql_right_closed: true to OpenClosed while preserving state-identity separation from existing half-open states.
   • Define tumbling/sliding, timestamp-zero, watermark-closure, and boundary-sample behavior; test every variant.

2. Port Execute completed cohorts at every materialized maintenance frontier #665/#717 count semantics for raw precompute. Bare CMS and MultipleSum currently add sample values. Add an explicit count-vs-value contract so count mode adds one per observation, as heap Top-K already does through weight_mode. THIS IS DONE.

3. Port #715 fail-fast configuration validation. Invalid HLL precision and invalid Min/Max subtype must be rejected at plan/config installation, rather than falling through to a warning/default accumulator path.

4. Port #716 ingest-throughput diagnostics. samples_ingested is maintained but not periodically reported; add a bounded periodic debug metric/log with a testable interval or metric surface.

5. Re-evaluate whether SetAggregator and DeltaSetAggregator should return. Decide whether the system needs exact, enumerable key membership / exact distinct semantics. If yes, define the supported query path, state model, chronological-merge contract for deltas, persistence/late-data semantics, and tests before restoring either accumulator.

6. Port docs(control_plane): design doc for BackendPlan wire format #389 before supporting SQL Top-K. Add SQL parsing/planning/execution for GROUP BY with COUNT(...), descending aggregate order, and LIMIT k, backed by a heap-bearing frequency sketch. This is separate from PromQL/query-plan Top-K.

7. Port SummaryExecutor: permanent self-exclusions + upstream composition gaps leave some PromQL shapes always on the legacy/archive path #436 before supporting SQL materializations. Define a typed source-column-to-ingest-value contract so a SQL materialization uses its declared value_column rather than always using the wire scalar.

No follow-up issues have been filed by this audit.

[milindsrivastava1997]

Audit update (2026-09-10)

Follow-up item 4 is complete via #578: backend ingest throughput is now exposed as periodic diagnostics and Prometheus metrics.

The merged change records monotonic accepted-sample, processed-update, and materialized-output counters; the 30-second log derives elapsed-time rates without resetting shared state. Regression coverage verifies non-destructive observations, correct interval deltas, and ticker cancellation.