Repository navigation
CloudStack 4.22 – VM deployment on LINSTOR primary fails during ROOT volume population (qemu-img convert), while volume creation succeeds #12362
Description
sos_2026-01-02_15-03-28.tar.gz
###Architecture
┌──────────────────────────────────┐
│ Management Node │
│ │
│ Hostname : csmgmt01 │
│ IP : 10.50.10.100 │
│ │
│ CloudStack Mgmt : 4.22.0.0 │
│ Cloud DB (MariaDB/MySQL) │
└───────────────┬──────────────────┘
│
│ Orchestration / API / DB
│
===================================================================
|| KVM + LINSTOR CLUSTER ||
|| (All KVM nodes on 10.50.11.0/24) ||
===================================================================
┌──────────────────────────────────┐ ┌──────────────────────────────────┐ ┌──────────────────────────────────┐
│ KVM NODE 1 │ │ KVM NODE 2 │ │ KVM NODE 3 │
│ Hostname : cskvm01.poc.local │ │ Hostname : cskvm02.poc.local │ │ Hostname : cskvm03.poc.local │
│ IP : 10.50.11.101 │ │ IP : 10.50.11.102 │ │ IP : 10.50.11.103 │
│ │ │ │ │ │
│ CloudStack KVM Agent : 4.22.0.0 │ │ CloudStack KVM Agent : 4.22.0.0 │ │ CloudStack KVM Agent : 4.22.0.0 │
│ LINSTOR role : COMBINED │ │ LINSTOR role : SATELLITE │ │ LINSTOR role : SATELLITE │
│ LINSTOR Ctrl/GUI : Yes │ │ LINSTOR Ctrl/GUI : No │ │ LINSTOR Ctrl/GUI : No │
│ DRBD kernel : 9.3.0 │ │ DRBD kernel : 9.3.0 │ │ DRBD kernel : 9.3.0 │
│ Storage pool (LVMTHIN): lvm-thin-fast on each node (~223GiB free) │
└──────────────────────────────────┘ └──────────────────────────────────┘ └──────────────────────────────────┘
║═══════════════════════════════════════════════════════════════║
║ LINSTOR + DRBD replication (block storage) ║
║ CloudStack volume -> LINSTOR resource -> /dev/drbdXXXX ║
║═══════════════════════════════════════════════════════════════║
======================================================================
STORAGE (CloudStack Datastores)
(A) Primary Storage #1 (NFS Primary) [Shared NFS datastore for KVM]
┌──────────────────────────────────────────────────┐
│ NFS Primary Storage │
│ Server IP : 10.50.10.100 (csmgmt01) │
│ Export : /export/primary (example) │
│ Protocol : NFSv4.2 │
│ Used for : Primary volumes on NFS (non-Linstor) │
└──────────────────────────────────────────────────┘
(B) Primary Storage #2 (LINSTOR Primary) [Your “second primary”]
┌──────────────────────────────────────────────────────────────┐
│ LINSTOR Primary Storage (Pool name in CS: linstor-primary) │
│ CloudStack pool_type : Linstor │
│ LINSTOR controller : http://10.50.11.101:3370 │
│ Backend pools : LVM_THIN (lvm-thin-fast) on nodes │
│ Used for : Primary volumes on DRBD (/dev/drbdX) │
└──────────────────────────────────────────────────────────────┘
(C) Secondary Storage (NFS Secondary) [templates/isos/systemvms]
┌──────────────────────────────────────────────────┐
│ NFS Secondary Storage │
│ Server IP : 10.50.10.100 (csmgmt01) │
│ Export : /export/secondary │
│ Protocol : NFSv4.2 │
│ Used for : Templates / ISOs / SystemVM templates │
└──────────────────────────────────────────────────┘
problem
CloudStack 4.22 – VM deployment on LINSTOR primary fails during ROOT volume population (qemu-img convert), while volume creation succeeds
======================================================================
Deployment Topology
Management Node:
- Hostname: csmgmt01
- Role: CloudStack Management Server
- OS: Ubuntu 22.04.5 LTS
- CloudStack Version: 4.22.0.0
- Does NOT run LINSTOR or DRBD
- Manages:
- CloudStack API / UI
- Database (cloud DB)
- Storage orchestration only
KVM + LINSTOR Cluster:
- Total Nodes: 3
- Hosts:
- cskvm01.poc.local
- cskvm02.poc.local
- cskvm03.poc.local
LINSTOR Deployment Model:
- cskvm01:
- LINSTOR Controller
- LINSTOR Satellite
- DRBD
- LVM_THIN storage pool
- cskvm02:
- LINSTOR Satellite
- DRBD
- LVM_THIN storage pool
- cskvm03:
- LINSTOR Satellite
- DRBD
- LVM_THIN storage pool
All three KVM nodes:
- Registered as CloudStack KVM hosts
- Participate in LINSTOR storage
- Have identical LVM_THIN pools for DRBD-backed volumes
Storage Pools:
- LINSTOR LVM_THIN pool present and healthy on all three KVM nodes
- LINSTOR diskless pools auto-created where required
- CloudStack primary storage points to LINSTOR controller on cskvm01
Secondary Storage:
- NFSv4.2
- Exported from management-side storage
- Mounted dynamically by CloudStack agent on KVM hosts
- Used for templates and ISOs
======================================================================
Environment Summary
MANAGEMENT NODE (csmgmt01)
Role: CloudStack Management Server
OS: Ubuntu 22.04.5 LTS
CloudStack Version: 4.22.0.0
DB Schema Version: 4.22.0.0
Installed Packages:
- cloudstack-management 4.22.0.0
- cloudstack-usage 4.22.0.0
- cloudstack-common 4.22.0.0
Primary Storage (CloudStack DB):
- Name: linstor-primary
- Pool Type: Linstor
- Status: Up
- UUID: 381f423d-5c3d-4037-85bb-f704bbebaa5f
KVM HOST (example: cskvm01)
Role: CloudStack KVM Hypervisor + LINSTOR Controller
OS: Ubuntu 22.04.5 LTS
Kernel: 5.15.0-164-generic
CloudStack Agent:
- cloudstack-agent 4.22.0.0
LINSTOR:
- Controller/Satellite version: 1.33.1
- Client: 1.27.1
- Storage driver: LVM_THIN
- Controller runs only on cskvm01
- Satellites run on cskvm01, cskvm02, cskvm03
DRBD:
- Kernel module: 9.3.0
- drbd-utils: 9.33.0
- drbd-reactor: 1.10.0
- Transport: TCP
QEMU / libvirt:
- qemu-img: 6.2.0
- QEMU hypervisor: 6.2.0
- libvirtd: 8.0.0
Virtualization checks:
- Hardware virtualization (vmx/svm): Enabled
- /dev/kvm accessible
- virt-host-validate: PASS (only IOMMU warning)
======================================================================
Templates
Templates registered in CloudStack DB:
-
Ubuntu 22.04
- DB format: RAW
- DB size: ~0.64 GB
-
Ubuntu 24.04
- DB format: RAW
- DB size: ~0.58 GB
On KVM hosts, template files stored on secondary NFS and named *.raw
are detected as QCOW2 via:
qemu-img info
Example:
- file format: qcow2
- virtual size: ~2.2 GiB
- disk size: ~600–700 MiB
Service Offering
Service offering used: testlinstor
ROOT disk sizes tested:
- 10 GB
- 20 GB
ROOT volume sizes verified in CloudStack DB match the offering.
======================================================================
Observed Problem
LINSTOR primary storage is detected as UP in CloudStack and volumes can
be created successfully across the 3-node LINSTOR cluster. However,
VM deployment fails specifically during ROOT volume population
from template.
Key behavior:
- LINSTOR volume creation succeeds
- DRBD-backed block device is created on KVM host
- Failure occurs only during instance ROOT disk population
- qemu-img convert to the DRBD block device fails
- CloudStack cleans up the DRBD resource and libvirt storage pool
- VM ends in Error state
- ROOT volume is marked Destroy in CloudStack DB
======================================================================
Relevant KVM Agent Log Excerpts
INFO Linstor: Creating volume for ROOT disk
INFO Linstor: Created DRBD device: /dev/drbd1001
INFO Executing qemu-img convert to DRBD device
ERROR qemu-img convert failed: output file is smaller than input file
WARN Template copy failed, cleaning up DRBD resource
INFO Linstor: Removed DRBD device and volume as part of cleanup
CloudStack Management Log Excerpts
ERROR Unable to find ObjectInDataStore mapping for TemplateObject on Linstor storage pool
WARN Failed to create ROOT volume for VM, marking volume as Destroy
======================================================================
Database Evidence
Failed instances:
-
i-2-12-VM
- ROOT volume size: 20 GB
- state: Destroy
- service offering: testlinstor
-
i-2-13-VM
- ROOT volume size: 10 GB
- state: Destroy
- service offering: testlinstor
LINSTOR storage pool remains in state = Up throughout.
======================================================================
Key Observation
LINSTOR and DRBD are functioning correctly across all three KVM nodes:
- Storage pools are healthy
- DRBD devices are created successfully
The failure occurs only at the template-to-root-volume population stage
(qemu-img convert writing to /dev/drbdX).
This suggests an issue in CloudStack’s LINSTOR integration or template
handling during ROOT volume deployment, rather than a LINSTOR or DRBD
volume provisioning problem.
Expected Behavior
CloudStack should successfully populate the ROOT volume on LINSTOR
primary storage from the template and continue VM deployment without
cleaning up the DRBD resource.
versions
The versions of ACS, hypervisors, storage, network etc..
the first one is from nfs all other are created using linstor tag in compute offering.
linstor based volume can be deployed standalone but it can’t be used for vm creation fails.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
boring-cyborg commentedon Jan 2, 2026
Thanks for opening your first issue here! Be sure to follow the issue template!
bilalinamdar commentedon Jan 3, 2026
I am uploading the logs here.
for the kvm's and the mgmt
tar cvpzf ~/cskvm0*.logs.tar.gz /var/log/linstor-satellite/ /var/log/linstor-controller/ /var/log/cloudstack/ /var/log/libvirt/ /var/log/syslog /var/log/dmesg /var/log/kern.log
tar cvpzf ~/csmgmt01.logs.tar.gz /var/log/cloudstack/ /var/log/syslog /var/log/dmesg /var/log/kern.log
cskvm03.logs.tar.gz
cskvm02.logs.tar.gz
cskvm01.logs.tar.gz
csmgmt01.logs.tar.gz
Also some settings of linstor
root@cskvm01:~# linstor node list
linstor sp l
root@cskvm01:~# linstor sp l
Resource Groups
linstor rg l
root@cskvm01:~# linstor rg l
root@cskvm01:~#
On all
no other parameter is set for linstor drbd.
#Current Issue
due to Qemu or Template Error
But as a additional disk i can use linstor for existing vm created via nfs. it also migrated to another host without isssue live. can also create standalone volumes with linstor. I only can't create VM's with linstor before this on plain install and linstor as a only primary storage even system vm's where not making up. so this time i used nfs as primary than second primary linstor. VM spinup issue. i think this is mostly due to some template clone / raw / qcow2 issue as per some observation in logs.
Update: I tried to deploy vm with ALPINE ISO and linstor and it worked. But it is not working with the default template from the cloudstack wizards (qcow2/img)
root@cskvm01:/var/log/cloudstack/agent# linstor v l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Resource ┊ Node ┊ StoragePool ┊ VolNr ┊ MinorNr ┊ DeviceName ┊ Allocated ┊ InUse ┊ State ┊ Repl ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm01.poc.local ┊ lvm-thin-fast ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ 2.05 MiB ┊ InUse ┊ UpToDate ┊ Established(2) ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm02.poc.local ┊ lvm-thin-fast ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ 2.05 MiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm03.poc.local ┊ DfltDisklessStorPool ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ ┊ Unused ┊ TieBreaker ┊ Established(2) ┊
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm01.poc.local ┊ lvm-thin-fast ┊ 0 ┊ 1001 ┊ /dev/drbd1001 ┊ 2.05 MiB ┊ InUse ┊ UpToDate ┊ Established(2) ┊
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm02.poc.local ┊ DfltDisklessStorPool ┊ 0 ┊ 1001 ┊ /dev/drbd1001 ┊ ┊ Unused ┊ TieBreaker ┊ Established(2) ┊
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm03.poc.local ┊ lvm-thin-fast ┊ 0 ┊ 1001 ┊ /dev/drbd1001 ┊ 2.05 MiB ┊ Unused ┊ UpToDate ┊ Established(2) ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@cskvm01:/var/log/cloudstack/agent#
root@cskvm01:/var/log/cloudstack/agent#
root@cskvm01:/var/log/cloudstack/agent# linstor rd l --show-props Aux/cs-name Aux/cs-vm-name
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ ResourceGroup ┊ Layers ┊ State ┊ Aux/cs-name ┊ Aux/cs-vm-name ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ my_fast_group ┊ DRBD,STORAGE ┊ ok ┊ ROOT-24 ┊ i-2-24-VM ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ my_fast_group ┊ DRBD,STORAGE ┊ ok ┊ fff ┊ ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@cskvm01:/var/log/cloudstack/agent# linstor r l
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Layers ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm01.poc.local ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ UpToDate ┊ 2026-01-03 14:39:44 ┊
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm02.poc.local ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2026-01-03 14:39:41 ┊
┊ cs-9ae31086-91ac-40d6-9f42-790cb5e0454f ┊ cskvm03.poc.local ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-01-03 14:39:44 ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm01.poc.local ┊ DRBD,STORAGE ┊ InUse ┊ Ok ┊ UpToDate ┊ 2026-01-02 00:35:33 ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm02.poc.local ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ UpToDate ┊ 2026-01-02 00:35:33 ┊
┊ cs-60f8b37f-c085-4477-91b2-55da2d2e1c92 ┊ cskvm03.poc.local ┊ DRBD,STORAGE ┊ Unused ┊ Ok ┊ TieBreaker ┊ 2026-01-02 00:35:33 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@cskvm01:/var/log/cloudstack/agent#
rp- commentedon Jan 7, 2026
In the Linbit forums I suspected that the wrong format in the Database might be a problem for the templates,
but I see in the
qemu-img convertthat there is no source format specified, soqemu-imgshould be smart enough to do the right things.So I simply suspect, that the templates images themselves are not working? Can you try with some other templates, like simple ubuntu cloud images?
bilalinamdar commentedon Jan 7, 2026
Hi @rp- ,
I was stuck with two things while following your milan event workshop. And i am 100% sure following it correctly!
To resolve Point 2 that is deployment of template or Instance making issue i tried to convert the qcow2 image to raw didn't work than i made changes in the database of template to recognize it as raw but still failed. I tried vice versa still failed.
In the linstor forum i was blocked to reply but i think it is now operational. That's why i was unable to reply there.
If you haven't checked the log yet u can. I would suggest to check FIRST failures not last as u will find why i needed to try to fix it..
https://drive.google.com/file/d/1-RLNwX0lBIACjoB2luSxTncBFj3JHCtk/view?usp=drive_link
Logs for the csmgmt01, cskvm01,cskvm02,cskvm03
#KVM
/var/log/linstor-satellite/ /var/log/linstor-controller/ /var/log/cloudstack/ /var/log/libvirt/ /var/log/syslog /var/log/dmesg /var/log/kern.log
#MGMT
/var/log/cloudstack/ /var/log/syslog /var/log/dmesg /var/log/kern.log
Currently i have installed HCI and checking it. Its working fine out of the box. Will again erase and install the same architecture mentioned here this weekend. And this time i will keep it as is. I mean will not touch much will report out of the box experience.
rp- commentedon Jan 8, 2026
@bilalinamdar
I tried to find the first failuers, but it seems the logs only contain, were you recreated the cluster with nfs-primary first, because in the logs I only see that linstor was added later.
And then the first error is already this:
But now as I look at it, I found it more interessting that there is already an error mounting the nfs secondary share and I'm a bit surprised that it then even tried to
qemu-img convertit, so maybe we are just looking at the wrong error, and the failing nfs mount is the real problem.bilalinamdar commentedon Jan 8, 2026
@rp- Thanks for responding.
I will make the clean setup day after tommorow (weekend) and will share the virgin logs with you. Lets than analyse if the issue is there or not. This time will share screenshot, videos and logs.
Currently working with HCI as in evaluating u can say.
Please check back this weekend
rp- commentedon Jan 8, 2026
I'm also sharing here my "playbook" from Milan.
ccc-2025-playbook.sh