Add seed migration steps to Rocky 10 migration documentation

[owenjones]

No description provided.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 5ca59b25-9550-4cfd-8684-11ae19ca9cd3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Summary

Updated Rocky 10 migration documentation with a complete seed-host migration procedure. The procedure covers service shutdown, volume backup and restoration, seed reprovisioning, secret-store redeployment, and seed-service deployment. The Azimuth operation-status check is now optional.

Walkthrough

The Rocky 10 operations guide makes the Azimuth check conditional and replaces the seed migration TODO with procedures for service shutdown, volume backup and restoration, seed reprovisioning, and service deployment.

Changes

Rocky 10 migration guidance

Layer / File(s)	Summary
Migration checklist and seed-host procedure doc/source/operations/rocky-10.rst	The checklist marks the Azimuth operation-status check as conditional. The Seed section documents service shutdown, backup and restoration of persistent volumes, seed reprovisioning and configuration, secret-store redeployment, and seed-service deployment.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: 🔴 Critical · up to b8082

The Rocky 10 migration procedure can allow operators to destroy seed data before completing an off-seed backup, omit required Pulp storage, or lose the key material needed to restore the secret store. Merge should be blocked until the backup ordering, volume coverage, restore method, and key-file preservation requirements are explicit.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 492e5630-b335-4144-84e4-60fb94191115

📥 Commits

Reviewing files that changed from the base of the PR and between 5e0e39f and b808218.

📒 Files selected for processing (1)

• doc/source/operations/rocky-10.rst

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout. (3)

• GitHub Check: stackhpc/check
• GitHub Check: Ansible 2.18 lint with Python 3.12
• GitHub Check: Ansible 2.17 lint with Python 3.10

🔇 Additional comments (1)

doc/source/operations/rocky-10.rst (1)

20-20: LGTM!

[coderabbitai]

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- repository conventions ---'
find /tmp/coderabbit-repo-knowledge/stackhpc-stackhpc-kayobe-config-578eaa86 -maxdepth 2 -type f -name '*.md' -print
printf '%s\n' '--- operations document ---'
sed -n '450,525p' doc/source/operations/rocky-10.rst
printf '%s\n' '--- secret-store deployment references ---'
rg -n -C 8 'seed-vault-keys|seed-openbao-keys|secret_store_keys|keys_base64|secret-store-deploy-seed' .

Repository: stackhpc/stackhpc-kayobe-config

Length of output: 50387

---

🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- scoped repository learning ---'
cat /tmp/coderabbit-repo-knowledge/stackhpc-stackhpc-kayobe-config-578eaa86/learnings/etc-kayobe-ansible-secret-store.md
printf '%s\n' '--- seed deployment playbook ---'
sed -n '1,110p' etc/kayobe/ansible/secret-store/secret-store-deploy-seed.yml
printf '%s\n' '--- secret-store configuration references ---'
rg -n -C 3 'seed-(vault|openbao)-keys\.json|vault_write_keys_file|openbao_write_keys_file|stackhpc_ca_secret_store' etc/kayobe doc/source/configuration/openbao.rst

Repository: stackhpc/stackhpc-kayobe-config

Length of output: 42126

---

Preserve the secret-store key file.

If the procedure restores Vault/OpenBao volumes, it must also preserve the unchanged seed-vault-keys.json or seed-openbao-keys.json file in the Kayobe configuration. secret-store-deploy-seed.yml loads this file and uses secret_store_keys.keys_base64 to unseal the restored store. A missing or different file can prevent unsealing.

Source: MCP tools

[jackhodgkiss]

@owenjones thanks for starting this.

A few things worth mentioning from my observations upgrading the seed-vm elsewhere.

1. There is an in-place upgrade method that whilst not official supported has been used in some places such as SMSLab https://github.com/swissmakers/rocky9-to-10
2. If not using the in-place upgrade there are two other options
   a. Preserve the data partition and replace the root partition with a new image
   • This can/should be something that can quickly be performed
   • It does require that the data partition is used
   • Not all important data is stored here for example /opt/kayobe/containers/pulp
   • Steps would be
     1. Shutdown seed services
     2. Use virsh shutdown against the seed-vm
     3. Copy both the root and data image from /var/lib/libvirt/images
     4. Run kayobe seed vm deprovision
     5. Copy back the data partition to /var/lib/libvirt/images
     6. Run kayobe seed vm provision
     7. Run kayobe seed host configure
     8. Run kayobe seed service upgrade*
        b. Alternatively backup the important files and restore the contents into the seed-vm
   • This would be required if the data partition is not present within the seed-vm
   • Would be good opportunity to add seed data partition
   • Steps would be
     1. Shutdown seed services
     2. Stop docker.service and docker.socket
     3. Copy the contents of /var/lib/docker/ and /opt/kayobe/containers/pulp out of the VM
     4. Use virsh shutdown against the seed-vm
     5. Copy the root image from /var/lib/libvirt/images
     6. Run kayobe seed vm deprovision
     7. Run kayobe seed vm provision
     8. Run kayobe seed host configure
     9. Stop docker.service and docker.socket
     10. Copy the contents of /var/lib/docker/ and /opt/kayobe/containers/pulp into the VM
     11. Run kayobe seed service upgrade

* Seed service deploy might fail as the host is now Rocky 10 which means kayobe seed service upgrade will expect to see a Rocky 9 bifrost-deploy container running. The default configuration seed-vm data LVM is to mount /var/lib/docker/volumes not /var/lib/docker this means that the containers themselves will not be preserved. Running a kayobe seed service upgrade this case won't work nor will a kayobe seed service deploy as Mariadb is a different version between Rocky 9 and Rocky 10. Therefore, if the container is not preserved but the volumes are then you will have to adjust the configuration temporarily to deploy a Rocky 9 bifrost-deploy container on a Rocky 10 host after which you can revert the change and perform kayobe seed service upgrade.

Note: I am also submitting a patch to libvirt-vm role to support pulling overcloud images from Ark. This would allow for root LVM configuration.

[owenjones]

@jackhodgkiss thanks for the very detailed run down, bringing that back into the docs now.

Any idea if some of the other stray container bind mounts would need to be backed up also (e.g. there are config files for squid and openbao which end up outside the data partition)

[jackhodgkiss]

@jackhodgkiss thanks for the very detailed run down, bringing that back into the docs now.

Any idea if some of the other stray container bind mounts would need to be backed up also (e.g. there are config files for squid and openbao which end up outside the data partition)

Not aware of anything else that is important like encryption keys.

Some configuration files will be missing but that just means the container won't start.