Repository navigation
Failed to create RBD storage pool after KVM agent upgrade from 4.20 to 4.22: "org.libvirt.LibvirtException: failed to create the RBD IoCTX" #12096
Description
problem
After upgrading the KVM agent on a compute node from CloudStack 4.20 to 4.22, the agent fails to recreate or connect to the existing RBD storage pool. The error manifests in the agent logs as a LibvirtException during pool initialization, querying if the RBD pool exists (which it does on the Ceph cluster). This prevents the host from fully reconnecting and handling VM operations (e.g., volume attach/detach).
The issue appears tied to changes in libvirt (8.0+) or Ceph client libraries post-upgrade, causing IoCTX creation to fail due to temporary secret/cached state mismatches. Notably, a full reboot of the compute node resolves the issue immediately, allowing clean recreation of the pool and secret. However, this introduces unwanted downtime for running VMs on that node, which is unacceptable in production.
versions
Environment
CloudStack version: Management server upgraded to 4.22.0 (from 4.20.0)
Agent version: KVM agent upgraded from 4.20.0 to 4.22.0 on compute nodes
Hypervisor: KVM
Primary Storage: Ceph RBD (pool name: cloudstack-zone1; Ceph version: 14)
OS on compute nodes: Ubuntu 20.04
The steps to reproduce the bug
- Upgrade mgmt to 4.22
- upgrade agent to 4.22
- log error from agent.log: Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to create the RBD IoCTX. Does the pool 'cloudstack-zone1' exist?
I also do these commands on CEPH but it's still error
# ceph config set mon auth_expose_insecure_global_id_reclaim false
# ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false
# ceph config set mon auth_allow_insecure_global_id_reclaim false
Expected Behavior
The agent should successfully redefine the RBD storage pool using the existing Ceph configuration (monitors, secrets) without failure, allowing seamless host reconnection post-upgrade.
Actual Behavior
Agent logs show repeated failures to create the RBD IoCTX, followed by cleanup of the libvirt secret. Host status remains "Disconnected" or "Alert" in UI until manual intervention. Full reboot of the compute node resolves the issue immediately (it's so bad solution)
Activity
weizhouapache commentedon Nov 19, 2025
@tuanhoangth1603
did you upgrade some other packages other than cloudstack-agent and cloudstack-common during upgrade ?
tuanhoangth1603 commentedon Nov 19, 2025
Hello,
When i ran apt upgrade cloudstack-agent, many others package were automatically installed like this
weizhouapache commentedon Nov 19, 2025
@tuanhoangth1603
ok. ..
I normally use "apt install" instead of "apt upgrade"
tuanhoangth1603 commentedon Nov 19, 2025
I'll try, thank you!
weizhouapache commentedon Nov 24, 2025
@tuanhoangth1603
has the issure been resolved now ?
tuanhoangth1603 commentedon Nov 24, 2025
Yes,
apt install cloudstack-agentwork finetuanhoangth1603 commentedon Nov 26, 2025
@weizhouapache Today I just discovered a strange issue as follows, which is related to this issue.
I had to correct a wrong slave interface name inside a bond. After editing the netplan YAML and running
netplan apply, the bond came up correctly and network was fully restored and then I restart cloudstack-agent.However, immediately after that, CloudStack permanently lost the Ceph RBD primary storage mount with this repeated agent error:
and output of virsh pool-list --all showed the line:
While others agent still connected normally so I think the issue is not on the Ceph side.
Interestingly, if I reboot the entire host, the problem disappears and the RBD pool is mounted normally again.
Maybe, does a simple netplan apply (which only briefly interrupts the network) permanently break libvirt’s ability to create the RBD pool, while a full host reboot fixes it?