Add cAdvisor container health monitoring by technowhizz · Pull Request #2599 · stackhpc/stackhpc-kayobe-config

technowhizz · 2026-09-14T16:17:30Z

PR to add more monitoring around container health statuses

coderabbitai · 2026-09-14T16:17:45Z

📝 Summary

Updates the Grafana cAdvisor dashboard with container health, CPU, and memory monitoring, modernised Prometheus queries, templates, panels, and layout. Adds a ContainerNotHealthy Prometheus alert for containers unhealthy for over 10 minutes. Adds release notes for these monitoring changes.

Walkthrough

The cAdvisor Grafana dashboard was migrated to newer Grafana and Prometheus formats. It now shows container health and resource data. Prometheus adds an alert for containers that remain unhealthy for more than 10 minutes. Release notes document these additions.

Changes

cAdvisor observability

Layer / File(s)	Summary
Dashboard modernisation `etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json`	The dashboard uses Grafana 12 metadata, Prometheus query settings, updated host and network panels, modernised variables, and a one-hour default time range.
Health and container panels `etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json`	The dashboard adds health counts, health history, and per-container CPU and memory panels.
Health alert and release note `etc/kayobe/kolla/config/prometheus/cadvisor.rules`, `releasenotes/notes/cadvisor-container-health-8e05e02c88f047ad.yaml`	Prometheus alerts when a named container remains unhealthy for 10 minutes. The release note documents the dashboard and alert changes.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🔵 Low · up to c3925

Parts of the dashboard will be blank or misleading under ordinary selections, but alerting and underlying monitoring remain unaffected. The PR is mergeable with these bounded dashboard issues acknowledged or fixed.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Warning

Some tools did not complete. Review the errors below.

🔧 Biome (2.5.10)

etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json

File contains syntax errors that prevent linting: Line 1: unexpected character %; Line 1: unexpected character %; Line 2: End of file expected; Line 1850: End of file expected; Line 1850: unexpected character %; Line 1850: unexpected character %

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 3

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 4741cbca-bec0-4ebd-9cfa-c1fcd13602de

📥 Commits

Reviewing files that changed from the base of the PR and between 362c4cf and c39256e.

📒 Files selected for processing (3)

etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json
etc/kayobe/kolla/config/prometheus/cadvisor.rules
releasenotes/notes/cadvisor-container-health-8e05e02c88f047ad.yaml

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout. (4)

GitHub Check: stackhpc/check
GitHub Check: Ansible 2.19 lint with Python 3.12
GitHub Check: Ansible 2.20 lint with Python 3.12
GitHub Check: Tox pep8 with Python 3.12

🧰 Additional context used

🪛 Biome (2.5.10)

etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json

[error] 2-1849: End of file expected

(parse)

🔇 Additional comments (2)

releasenotes/notes/cadvisor-container-health-8e05e02c88f047ad.yaml (1)

1-6: LGTM!

etc/kayobe/kolla/config/prometheus/cadvisor.rules (1)

24-24: 🎯 Functional Correctness

The configured Kolla images use the 2026.1 release. Kolla 2026.1 uses cAdvisor 0.56.2, which exports container_health_state. In this release, 1 means healthy, 0 means unhealthy, and -1 means no health check. The alert expression correctly matches unhealthy containers. The concern that this deployment predates metric support is refuted.

coderabbitai · 2026-09-14T16:24:45Z

+          "expr": "sort(sum(container_health_state{name=~\"$name\"}==-1))",
+          "format": "time_series",
+          "instant": true,
+          "legendFormat": "{{name}}",
+          "range": false,


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Fix the No Health check target in the status-history panel.

The sum() aggregation removes the name label and combines all matching -1 samples. With multiple containers, the result is -N, which does not match the -1 value mapping. instant: true also returns only the current point, so the target does not provide history across the selected time range.

Use one range series per container:

🐛 Proposed fix

- "expr": "sort(sum(container_health_state{name=~\"$name\"}==-1))", + "expr": "container_health_state{name=~\"$name\"}==-1", "format": "time_series", - "instant": true, + "instant": false, "legendFormat": "{{name}}", - "range": false, + "range": true, "refId": "No Health check"

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

"expr": "sort(sum(container_health_state{name=~\"$name\"}==-1))",

"format": "time_series",

"instant": true,

"legendFormat": "{{name}}",

"range": false,

"expr": "container_health_state{name=~\"$name\"}==-1",

"format": "time_series",

"instant": false,

"legendFormat": "{{name}}",

"range": true,

🧰 Tools

🪛 Biome (2.5.10)

[error] 2-1849: End of file expected

(parse)

coderabbitai · 2026-09-14T16:24:45Z

+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${datasource}"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "container_health_state{name=~\"$name\"}==1",
+          "format": "time_series",
+          "instant": false,
+          "legendFormat": "{{name}}",
+          "range": true,
+          "refId": "A"


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Remove the duplicate target.

The target with refId: "A" repeats the expression of the Healthy target exactly. Grafana runs both queries and renders every healthy container twice in the status history and the legend.

♻️ Proposed fix

- }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "editorMode": "code", - "exemplar": false, - "expr": "container_health_state{name=~\"$name\"}==1", - "format": "time_series", - "instant": false, - "legendFormat": "{{name}}", - "range": true, - "refId": "A" }

🧰 Tools

🪛 Biome (2.5.10)

[error] 2-1849: End of file expected

(parse)

coderabbitai · 2026-09-14T16:24:45Z

+              },
+              "unit": "percent"


🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Add the "Please select only one node" message to both new per-container panels. The node variable now defaults to All, and both new panels use the exact matcher instance="$node", so they return no data by default. The host and network panels received a noValue message for this reason, but these two panels did not.

etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json#L1159-L1160: add "noValue": "Please select only one node" to the defaults of the CPU Usage per Container panel.

etc/kayobe/kolla/config/grafana/dashboards/openstack/cadvisor.json#L1270-L1271: add the same noValue entry to the defaults of the Memory Usage per Container panel.

🧰 Tools

🪛 Biome (2.5.10)

[error] 2-1849: End of file expected

(parse)

Add cAdvisor container health monitoring

c39256e

technowhizz self-assigned this Sep 14, 2026

technowhizz requested a review from a team as a code owner September 14, 2026 16:17

github-actions Bot added the waiting-review PR is waiting for a review label Sep 14, 2026

coderabbitai Bot reviewed Sep 14, 2026

View reviewed changes

technowhizz deployed to SMS Lab September 14, 2026 16:48 — with GitHub Actions Active

technowhizz had a problem deploying to SMS Lab September 14, 2026 16:48 — with GitHub Actions Failure

technowhizz deployed to SMS Lab September 14, 2026 16:48 — with GitHub Actions Active

technowhizz had a problem deploying to SMS Lab September 14, 2026 16:48 — with GitHub Actions Failure

technowhizz had a problem deploying to SMS Lab September 14, 2026 16:49 — with GitHub Actions Failure

Navigation Menu

Uh oh!

Add cAdvisor container health monitoring - #2599

Add cAdvisor container health monitoring#2599
technowhizz wants to merge 1 commit into
stackhpc/2026.1from
cadvisor-fixes

technowhizz commented Sep 14, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 14, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Sep 14, 2026

Uh oh!

coderabbitai Bot Sep 14, 2026

Uh oh!

coderabbitai Bot Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

technowhizz commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

technowhizz commented Sep 14, 2026 •

edited

Loading

coderabbitai Bot commented Sep 14, 2026 •

edited

Loading