-
Notifications
You must be signed in to change notification settings - Fork 0
Small-sample quantile_over_time p99 diverges from Prometheus interpolation semantics #492
Copy link
Copy link
Open
Labels
area: data planeRuntime ingestion, storage, and query-serving data pathsRuntime ingestion, storage, and query-serving data pathsarea: query enginePromQL lowering, execution, result shaping, and fallback behaviorPromQL lowering, execution, result shaping, and fallback behaviorarea: summary executionWarm-summary binding, readout, composition, and result decodingWarm-summary binding, readout, composition, and result decodingbugSomething isn't workingSomething isn't workingfeature: PromQLPromQL compatibility and execution semanticsPromQL compatibility and execution semanticsfeature: sketch algorithmsSketch-family configuration, lifecycle, merge, and readout behaviorSketch-family configuration, lifecycle, merge, and readout behaviorfeature: warm summariesASAP warm-tier materialization, readiness, and servingASAP warm-tier materialization, readiness, and servingquality: accuracyExact and approximate accuracy contracts, error bounds, and routingExact and approximate accuracy contracts, error bounds, and routingtest: end-to-endProcess-boundary, differential, conformance, and full-stack testsProcess-boundary, differential, conformance, and full-stack tests
Description
Activity
Metadata
Metadata
Assignees
Labels
area: data planeRuntime ingestion, storage, and query-serving data pathsRuntime ingestion, storage, and query-serving data pathsarea: query enginePromQL lowering, execution, result shaping, and fallback behaviorPromQL lowering, execution, result shaping, and fallback behaviorarea: summary executionWarm-summary binding, readout, composition, and result decodingWarm-summary binding, readout, composition, and result decodingbugSomething isn't workingSomething isn't workingfeature: PromQLPromQL compatibility and execution semanticsPromQL compatibility and execution semanticsfeature: sketch algorithmsSketch-family configuration, lifecycle, merge, and readout behaviorSketch-family configuration, lifecycle, merge, and readout behaviorfeature: warm summariesASAP warm-tier materialization, readiness, and servingASAP warm-tier materialization, readiness, and servingquality: accuracyExact and approximate accuracy contracts, error bounds, and routingExact and approximate accuracy contracts, error bounds, and routingtest: end-to-endProcess-boundary, differential, conformance, and full-stack testsProcess-boundary, differential, conformance, and full-stack tests
Context
Found while developing/reviewing the raw-oracle process E2E in #478.
Using this deterministic raw fixture:
for:
the Prometheus-style linear interpolation oracle gives
98.4, while the production backend's DDSketch path returned approximately80.6522. Relative difference is about 18%, despite DDSketch alpha0.01.Changing the stable acceptance fixture to p50 avoids conflating interpolation definitions, but leaves the high-quantile semantic difference unresolved.
Why this matters
Prometheus clients expect
quantile_over_timesemantics. DDSketch is approximate, but the exposed accuracy envelope should have a clearly defined relationship to the Prometheus reference value. A discrete lower-rank selection that is far outside the advertised relative error is surprising for small populations.Reproduction approach
Use the process fixture in
data_plane/tests/promql_differential_process_e2e.rs, set the quantile to0.99, and retain the nine raw values above.Questions to resolve
Done when
The contract is documented and a strict process-level test compares small-N p90/p99 results against the chosen reference semantics. If Prometheus compatibility is intended, the returned value and declared accuracy must be consistent with the Prometheus oracle.