Enhance overcloud openbao deployment (2025.1 backport) - #2596
Enhance overcloud openbao deployment (2025.1 backport)#2596seunghun1ee wants to merge 7 commits into
Conversation
Added variable ``raft_leader_index`` to the ``secret-store-deploy-overcloud.yml`` playbook. This variable controls which controller becomes the leader when deploying OpenBao. The default is 0 (Frist controller). This variable needs to track the index of current leader controller when fixing/restarting the cluster to prevent split brain. (cherry picked from commit 2aee0d9)
(cherry picked from commit df5d15d)
(cherry picked from commit dafac0f)
(cherry picked from commit 22c4d87)
(cherry picked from commit 4367d32)
(cherry picked from commit 0fc22a2)
(cherry picked from commit c98a533)
📝 SummaryImproves 2025.1 OpenBao recovery by detecting the current Raft leader and passing its index to deployment. Adds a recovery playbook and restoration documentation. Increases the first-node unseal timeout to 10 seconds to support longer recovery operations. WalkthroughAdds leader detection for OpenBao Raft, updates deployment to use the detected leader, adds a recovery wrapper playbook, and documents automated and manual cluster restoration procedures. ChangesOpenBao recovery
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~20 minutes Severity of issue fixed: Medium Sequence Diagram(s)sequenceDiagram
participant Operator
participant FixPlaybook
participant LeaderPlaybook
participant DeployPlaybook
Operator->>FixPlaybook: Run fix-openbao-overcloud.yml
FixPlaybook->>LeaderPlaybook: Import get-current-raft-leader.yml
LeaderPlaybook-->>FixPlaybook: Set raft_leader_index
FixPlaybook->>DeployPlaybook: Import secret-store-deploy-overcloud.yml
DeployPlaybook-->>Operator: Complete leader-aware deployment
Merge Risk: 🟠 High · up to The new recovery workflow can fail to identify or unseal the actual leader, undermining the split-brain restoration feature. These issues should be fixed before merge. 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 4c9c26a6-386c-40c8-b4ad-fa41410b0aad
📒 Files selected for processing (4)
doc/source/configuration/openbao.rstetc/kayobe/ansible/secret-store/fix-openbao-overcloud.ymletc/kayobe/ansible/secret-store/get-current-raft-leader.ymletc/kayobe/ansible/secret-store/secret-store-deploy-overcloud.yml
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: stackhpc/check
- GitHub Check: Ansible 2.17 lint with Python 3.10
- GitHub Check: Ansible 2.18 lint with Python 3.12
🧰 Additional context used
🪛 Checkov (3.3.13)
etc/kayobe/ansible/secret-store/get-current-raft-leader.yml
[medium] 30-40: Ensure that HTTPS url is used with uri
(CKV2_ANSIBLE_1)
[medium] 39-49: Ensure that HTTPS url is used with uri
(CKV2_ANSIBLE_1)
🔇 Additional comments (2)
etc/kayobe/ansible/secret-store/secret-store-deploy-overcloud.yml (1)
26-34: LGTM!Also applies to: 105-107, 133-133
etc/kayobe/ansible/secret-store/fix-openbao-overcloud.yml (1)
2-6: LGTM!
| TASK [Display the index of the Raft leader] *********************************** | ||
| Monday 03 August 2026 12:15:02 +0000 (0:00:00.148) 0:00:07.797 ********* | ||
| ok: [controller-01] => | ||
| msg: 'raft_leader_index: 2' | ||
| ok: [controller-02] => | ||
| msg: 'raft_leader_index: 2' | ||
| ok: [controller-03] => | ||
| msg: 'raft_leader_index: 2' |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Correct the run_once transcript.
Display the index of the Raft leader uses run_once: true, so Ansible shows one controller result. The example shows three results. Show one result in the transcript.
| name: secret_store_keys | ||
|
|
||
| - name: Query the HA status | ||
| ansible.builtin.uri: |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Bypass proxies for controller-local status requests.
These tasks run on every controller, but uri uses a proxy by default. In a walled-garden configuration that follows the documented no_proxy entry for only the first controller, requests from other controllers can fail. ignore_errors then hides the request error and the assertion reports no leader. Set use_proxy: false on both URI tasks. (docs.ansible.com)
Also applies to: 40-40
🧰 Tools
🪛 Checkov (3.3.13)
[medium] 30-40: Ensure that HTTPS url is used with uri
(CKV2_ANSIBLE_1)
Source: MCP tools
| key: raft_leader | ||
| when: | ||
| - ha_status_query.json is defined | ||
| - ha_status_query.json.nodes is defined |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Use the OpenBao response key Nodes.
OpenBao serialises the HA member list as Nodes. The lowercase lookup is always undefined, so no controller enters raft_leader and the following assertion fails during multi-node recovery. Change both .json.nodes references to .json.Nodes. (openbao.org)
Source: MCP tools
| environment: | ||
| https_proxy: "" | ||
| run_once: true | ||
| delegate_to: "{{ groups['controllers'][raft_leader_index | int] }}" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Send the first unseal request to the selected leader.
delegate_to runs the role on the selected controller, but vault_api_addr still resolves through the original run_once host. inventory_hostname does not change during delegation. If the leader is not the first controller, this task can send the first unseal request to a non-leader. Set vault_api_addr to https://{{ raft_leader_address }}:8200. (docs.ansible.com)
Source: MCP tools
1 participant
Currently fixing split brain OpenBao cluster is not a smooth operation.
Added following to improve the cluster restoration process
Also added 10 seconds timeout on unsealing first OpenBao node to prevent the deployment playbook from failing caused by slightly longer unsealing time when fixing the cluster.