Skip to content

feat: unify native/ASAP experiments and plan SQL workloads jointly - #695

Open
zzylol wants to merge 2 commits into
mainfrom
feat/unified-e2e-benefits
Open

feat: unify native/ASAP experiments and plan SQL workloads jointly#695
zzylol wants to merge 2 commits into
mainfrom
feat/unified-e2e-benefits

Conversation

@zzylol

@zzylol zzylol commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

Measure existing ASAP execution against native Prometheus, ClickHouse, and VictoriaMetrics through one reproducible experiment, and expose cross-query SQL alternatives to Planner.

What and how

  • Compile supplied workloads, start three ASAP backends, verify plan activation, ingest identical data into isolated native/fallback services, materialize summaries, and compare all six endpoints.
  • Retain endpoint failures independently; compare MetricsQL against native VictoriaMetrics and distinguish warm, hybrid, and exact fallback execution.
  • Plan SQL queries together within compatible evaluation ranges and cumulative modes, preserving query identities and physical bindings.
  • Record planning, startup, ingest, materialization, incremental batches, query CPU/latency, and backend-plus-fallback costs. Missing evidence remains null; CPU break-even is conditional on correctness and measured maintenance.
  • Add a Docker smoke runner, developer instructions, and Python checks in CI.

Before this PR

The shared fixture compared only native databases. SQL publication selected each query independently, so a grouped sum and total sum could not expose a cross-query rollup alternative. Separate replay tools did not provide unified lifecycle accounting.

After this PR

One experiment exercises all three native/ASAP pairs and retains actual plans, routes, values, and costs. Related SQL queries share a selection context; rollup candidates become visible without forcing the cost model to select them.

The live smoke compiles two sum queries, installs one SQL materialization for their workload, loads data in two batches, and queries a later SQL window. A third uninstalled window verifies exact fallback.

Verification

  • SQL regression failed before the cohort change; final ClickHouse unit suite: 8 passed.
  • Shared-workload Python suite: 15 passed, including timeout isolation, VictoriaMetrics mismatch, fallback classification, pane-aligned backfill, and refresh-cost accounting.
  • Built the real comparison compiler and data-plane binary; Rust formatting and git diff --check passed.
  • Real Docker/process smoke: 45 native/ASAP pairs equal, 30 warm, 15 expected exact fallbacks. Independent expected-value checks prevent empty responses from passing. Cleanup completed without errors.
  • Validated against backend main base 96504c7d and Planner 3be523fa, using an isolated build copy with CI-pinned sketchlib and local Collector. Immutable native images: Prometheus sha256:5ce7540c3c00ef4ab0c9d2c995c6a5b9c421f44b4a115d97a2c7af3b1c21cbb0; ClickHouse sha256:fa394da808cc53f76d0344429421d6c422a6ee85fe7450135c0e3cff4df9bcbb; VictoriaMetrics sha256:26dfc4ab90cf6fedc8115269adc22d300bcd5eab82ea1ebed3c8e22b49886ce1.

Limits and follow-up

This tiny debug-build smoke establishes functional integration and did not demonstrate a positive performance benefit. Native containers have individual resource limits; host backend processes do not have equivalent aggregate cgroup limits.

The existing SQL backfill guard rejects a later second job after first-seen metadata appears. The runner preserves that guard and performs one complete finite SQL backfill after loading all batches. SQL continuous summary maintenance and its break-even remain explicitly unavailable. Other maintenance results describe finite input batches, not sustained load. Functional smoke planning costs are declared seeds, not measured calibration or a planner-optimality claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant