feat(eval): offline replay foundation and planned PromQL acceptance - #689
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Evaluate query-type sensitivity on matched data without confusing accurate fallback responses with summary acceleration or query-only cost with system cost.
What
Evaluation and harness tests are offline/manual only: no push/PR evaluation CI workflow is added. Existing backend CI is unchanged.
How
Reuse the existing shared SQL translations and production result comparator/Remote Write encoder. Reject empty oracles, incomplete temporal history, mismatched dataset profiles, invalid samples and overlapping component scopes. Keep VM semantic differences and unavailable measurements explicit.
Before this PR
The shared sensitivity tool generated generic
dataexpressions and checked a single baseline timestamp; dataset binding, repeated ASAP endpoint comparison and component accounting required separate manual work.After this PR
For example,
manifest --dataset syntheticemits bothsum by(label_0)(rate(fake_metric_counter_total[10m]))and its SQL translation. Google uses service-grouped CPU gauges and Alibaba machine-grouped container CPU; their 60 counter cases per profile are N/A, not misleading gauge rates. Operators can load one hashed dataset and retain matched comparison/resource evidence.Verification
sum_over_time(fake_metric[1m]), two complete windows; both warm, one summary readout each, no raw scans/exact RPCs, both accuracy comparisons pass. Normal compiler selection used measured candidate quotes, not a forced plan. In this debug/small fixture ASAP was slower (23.682ms vs direct Prometheus 9.412ms mean); this is chain evidence, not benefits evidence. Seetools/shared-workload/planned-sum-evidence.mdfor versions, hashes and retained artifacts.git diff --checkpasses. Visual evidence: not applicable.Limitations / follow-up
This PR is evaluation foundation plus one real PromQL chain, not completed end-to-end benefits acceptance. SQL moving-time fixed-plan installation, MetricsQL orchestration, independent complete native/ASAP lifecycle runs, large trace measurements and all-family warm support remain unverified. Endpoint replay always leaves full-system benefit eligibility false; partial pair acceptance cannot certify VM. Replay is historical cadence, not wall-clock dashboard concurrency. TopK ties and rank-error acceptance remain follow-up. A 29,760,800-sample, 1h-window release-build experiment is being prepared separately; no outcome is asserted until it finishes.