Skip to content

feat(eval): offline replay foundation and planned PromQL acceptance - #689

Merged
zzylol merged 7 commits into
mainfrom
eval/query-type-accuracy
Sep 12, 2026
Merged

feat(eval): offline replay foundation and planned PromQL acceptance#689
zzylol merged 7 commits into
mainfrom
eval/query-type-accuracy

Conversation

@zzylol

@zzylol zzylol commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

Evaluate query-type sensitivity on matched data without confusing accurate fallback responses with summary acceleration or query-only cost with system cost.

What

  • Dataset-bound PromQL/ClickHouse SQL manifests: 200 cases per profile across the requested families, quantiles, windows and equality filters.
  • Prometheus-client fake gauges/resetting counters; Google mapper JSONL and Alibaba 2018 container-usage adapters, hashes and matched Remote Write/SQL loading.
  • Independent ASAP-first endpoint replay, immediate per-request journals, per-pair accuracy/route/latency verdicts and optional ASAP MetricsQL pairing. No client query deadline; native failures cannot skip ASAP or erase earlier responses.
  • Explicit offline scale plans: default benefit cell has 1,000 groups × 16 members, 100ms samples, 1h history + 30m evaluation (1,728,032,000 samples). Larger scale/million-group profiles and the 30-cell cardinality/window matrix remain opt-in. Generation and replay share the plan and reject undersized data or shortened repetition intervals.
  • Disjoint-cgroup per-request and whole-command CPU, memory, block I/O, allocated storage and optional namespace-network accounting.
  • Planned replay accepts per-series-ordered trace exports, sorting interleaved timestamps on disk while preserving samples and recording both source and replay hashes. Ordered datasets are reused directly.
  • A narrow production-planned PromQL driver: separate owned baseline/fallback stores, normal costed compiler selection/install, ingestion, finite-input drain and observed summary-readout readiness at every evaluation window. No ASAPCollector service runs (backend-local precompute only).
  • README documents actual trace fields, grouping/selection, native timestamps, cardinality limits and reproducible commands; focused manually invoked Python tests.

Evaluation and harness tests are offline/manual only: no push/PR evaluation CI workflow is added. Existing backend CI is unchanged.

How

Reuse the existing shared SQL translations and production result comparator/Remote Write encoder. Reject empty oracles, incomplete temporal history, mismatched dataset profiles, invalid samples and overlapping component scopes. Keep VM semantic differences and unavailable measurements explicit.

Before this PR

The shared sensitivity tool generated generic data expressions and checked a single baseline timestamp; dataset binding, repeated ASAP endpoint comparison and component accounting required separate manual work.

After this PR

For example, manifest --dataset synthetic emits both sum by(label_0)(rate(fake_metric_counter_total[10m])) and its SQL translation. Google uses service-grouped CPU gauges and Alibaba machine-grouped container CPU; their 60 counter cases per profile are N/A, not misleading gauge rates. Operators can load one hashed dataset and retain matched comparison/resource evidence.

Verification

  • 30 shared-workload tests and 50 existing replay tests pass locally. Regression tests reproduce and fix native-endpoint short-circuiting and invalid VM acceptance, and verify ASAP-first/no-client-deadline ordering and independent journals.
  • Real production-planned PromQL chain: 96,080 samples, sum_over_time(fake_metric[1m]), two complete windows; both warm, one summary readout each, no raw scans/exact RPCs, both accuracy comparisons pass. Normal compiler selection used measured candidate quotes, not a forced plan. In this debug/small fixture ASAP was slower (23.682ms vs direct Prometheus 9.412ms mean); this is chain evidence, not benefits evidence. See tools/shared-workload/planned-sum-evidence.md for versions, hashes and retained artifacts.
  • Real Prometheus 3.5.0 promtool import smoke: 32 samples, 16 series, 100ms timestamp span. This tests import format, not SQL query semantics.
  • Cross-series trace regression fails before the fix and passes after; focused checks also preserve the ordered-input path and reject duplicate/backwards per-series samples.
  • git diff --check passes. Visual evidence: not applicable.

Limitations / follow-up

This PR is evaluation foundation plus one real PromQL chain, not completed end-to-end benefits acceptance. SQL moving-time fixed-plan installation, MetricsQL orchestration, independent complete native/ASAP lifecycle runs, large trace measurements and all-family warm support remain unverified. Endpoint replay always leaves full-system benefit eligibility false; partial pair acceptance cannot certify VM. Replay is historical cadence, not wall-clock dashboard concurrency. TopK ties and rank-error acceptance remain follow-up. A 29,760,800-sample, 1h-window release-build experiment is being prepared separately; no outcome is asserted until it finishes.

@zzylol zzylol changed the title feat(eval): add dataset-specific accuracy and resource evaluation feat(eval): offline replay foundation and planned PromQL acceptance Sep 12, 2026
@zzylol
zzylol merged commit 52573cd into main Sep 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant