Repository navigation
feat: add Grafana panels for lease and Redis health - #64
Merged
yashdotdev13 merged 1 commit intoOct 23, 2026
Merged
yashdotdev13 merged 1 commit into
yashdotdev13 merged 1 commit into
Conversation
Add three dashboard rows below the existing panels, using only the metrics the gateways already export: - Leases: allocation success rate, allocation failures per second and fleet-wide quota exhaustion over the last 5 minutes. - Decision latency: p50/p95/p99 and max per instance. The quantiles are client-side Micrometer quantiles, so they are shown per instance and never aggregated across instances. - Instance & Redis health: gateway up/down from Prometheus' up series, and the lease allocation failure ratio. The latter reflects global capacity exhaustion; no Redis health metric is exported yet, which the panel description states. The existing panels are unchanged. No metrics, labels or other files are added or changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expand the Grafana dashboard with panels for distributed lease health, decision latency percentiles and per-instance health, using only the metrics the gateways already export.
Related issue
Closes #40
Type of change
What changed?
The five existing panels are unchanged. Three rows are added below them in
monitoring/grafana/dashboards/traffic-control.json:sum(increase(traffic_control_quota_exhausted_total[5m]))tration_seconds{quantile=...}per instancetraffic_control_decision_duration_seconds_maxper instanceDesign decisions:
instance,quantileor `resuare added.result="failure"means Redis answered but reported insufficient global capacity, while Redisconnection errors are not counted as failures. The failure-rscribed as capacity exhaustion, not as a Redis health signal. A real Redis signal (Redis exporter or Lettuce client metrics) would be a follow-up.
Validation
.\mvnw.cmd -B verifypasses locally (not applicable: dashboard JSON only)Additionally:
python3 -m json.toolpasses on the dashboard JSONinstance/quantile/resultDistributed-systems impact
None. Dashboard JSON only: no change to Redis coordination, , concurrency, expiration/reclamation, consistency or failure handling. No application code, metrics, labels or Prometheus configuration changed.
Breaking changes
Checklist
Question for the maintainer
The existing "Decision Latency p95" panel uses
histogram_qurol_decision_duration_seconds_bucket, but the decision timer is built withpublishPercentiles(0.50, 0.95, 0.99)and no percentile histogram, so it should export summary quantiles rather than_bucketseries. Does that panel currently show data for you? If not,uantile series (as the new p50/p95/p99 panel does), or should the timer enable
publishPercentileHistogram()? I left it unchanged to keep this PR to the scope of the issue.🤖 Generated with Claude Code