Skip to content

feat: add MetricsQL canonical frontend - #368

Merged
zzylol merged 7 commits into
mainfrom
feat/metricsql-frontend
Sep 10, 2026
Merged

feat: add MetricsQL canonical frontend#368
zzylol merged 7 commits into
mainfrom
feat/metricsql-frontend

Conversation

@zzylol

@zzylol zzylol commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why

VictoriaMetrics queries need a real MetricsQL grammar boundary that feeds the existing canonical planner without changing QueryExpr, SDS, or physical summary DAG semantics. The parser must remain consumable from the backend on stable Rust, and its compatibility with official VictoriaMetrics syntax must be measured rather than inferred from a third-party crate name.

What

  • add asap-frontend-metricsql, backed by the Apache-2.0 Rust MetricsQL port from ccollie/metricsql commit 3046709308e449a42c56bfbfd45f95af848e6768
  • vendor that pinned parser with provenance and a direct stable parser-support dependency, because Cargo root patches do not propagate to downstream consumers
  • walk the native MetricsQL AST directly into canonical unresolved QueryExpr; no source-to-source PromQL conversion and no change to shared IR semantics
  • support exact-name selectors, fixed ranges, ordinary matchers, core single-input aggregates/grouping, common rollups, and unmodified PromQL binary operators
  • fix parser compatibility for metric names in OR branches, per-branch __name__ validation, WITH string interpolation, and numeric separators
  • directly compare a representative corpus with canonical output generated by the official VictoriaMetrics Go parser v0.84.4, plus typed AST assertions for numeric and MetricsQL-only semantics
  • run every vendored parser test in CI and require each documented upstream failure name plus its normalized failure-signature hash
  • enforce exact arity for every mapped aggregate and rollup so MetricsQL multi-input aggregates fail closed instead of discarding inputs
  • expose AST-based canonical_metricsql for stable plan identity

Before this PR

ASAPPlanner had PromQL and SQL frontends. A VictoriaMetrics request had no native MetricsQL parser boundary. The initial integration also depended on a non-propagating workspace [patch], could lose metric names in foo{job="a" or job="b"}, and had no automated comparison with the official parser.

After this PR

sum by (job) (rate(http_requests_total[5m])) produces the existing canonical aggregation DAG. WITH (prefix="http_") {__name__=prefix+"requests_total"} expands to the same scan as http_requests_total. OR matcher branches retain their metric name, official numeric separators parse, and unsupported MetricsQL semantics return a typed error for exact VictoriaMetrics fallback. A separate stable Rust project can consume the frontend without Cargo patches or CPU feature flags.

Verification

  • cargo test --workspace --locked
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo test -p asap-frontend-metricsql --tests
  • bash tools/verify_metricsql_external_consumer.sh
  • python3 tools/verify_metricsql_vendored_baseline.py (21 library and 3 doctest names/signatures verified)
  • cargo fmt --all -- --check
  • git diff --check

The pinned upstream parser needs a compile-only HashSetExt import correction before its own suite runs. With that correction and its declared nightly toolchain, upstream is 230 passed/23 failed for library tests and 2 passed/3 failed for doctests. This PR fixes two numeric-separator failures; CI now verifies the exact remaining 21 library and 3 doctest failures by name and fails on any drift.

Limitations

  • implicit default_rollup(metric) and step-relative windows need runtime evaluation-step context and currently fall back
  • keep_metric_names needs metric-name lineage and currently falls back
  • MetricsQL if, ifnot, default, aggregate limits, or-delimited matchers, binary match modifiers, and multi-input aggregates parse natively but currently fall back
  • the vendored parser is an unofficial Rust port; provenance and its known upstream test baseline are recorded in UPSTREAM.md

@zzylol
zzylol marked this pull request as ready for review September 10, 2026 04:16
@zzylol
zzylol force-pushed the feat/metricsql-frontend branch from 70cc256 to da63b34 Compare September 10, 2026 05:46
@zzylol
zzylol merged commit 5f37a85 into main Sep 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant