Repository navigation
NAS backup of large VMs fails #13883
Description
problem
When backing up up larger VMs (the ones we've tested have 2+ TB of storage), scheduled and manual backups eventually fail to NFS storage. The files are written until some sort of timeout seems to happen.
The files are available on the NFS storage but every backup they stop after a certain period of time. A side effect of this is that cloudstack doesn't apply retention to the backups in this case so they keep filling up storage.
Smaller servers (tested until 600 GB backup correctly. There is no firewall in the path. The hypervisors and NFS storage are in the same subnet.
It does seem to be a timeout issue as it seems the management server times out after an hour, but the backup continues running.
2026-08-15 06:04:06,169 DEBUG [o.a.c.b.BackupManagerImpl] (BackupPollTask:[ctx-f0772149]) (logid:34dc33a0) Scheduling 1 backup for VM (VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}) for backup schedule (BackupSchedule {"id":6,"schedule":"00:06:7","scheduleType":2,"vmId":1083}) at [2026-08-15T06:00:00+0000].
2026-08-15 06:04:06,607 DEBUG [c.c.a.t.Request] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Sending { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 06:08:39,558 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-6:[ctx-9f798002]) (logid:1d647320) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 06:13:40,517 DEBUG [o.a.c.b.B.BackupSyncTask] (BackgroundTaskPollManager-5:[ctx-55244db9]) (logid:55fc3c21) Trying to sync backups of VM [VM instance {"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}] using backup provider [nas].
2026-08-15 07:04:06,609 WARN [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Seq 10-4578471971175552676: Timed out on Seq 10-4578471971175552676: { Cmd , MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}] }
2026-08-15 07:04:06,609 ERROR [o.a.c.b.NASBackupProvider] (API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b) Operation to initiate backup timed out for VM i-10-1083-VM
Looking at the files:
$ ls -la ./cs-backup/i-10-1083-VM/2026.08.15.06.04.06/
total 996803735
drwxr-xr-x 2 nobody nogroup 8 Aug 15 07:32 .
drwxr-xr-x 4 nobody nogroup 4 Aug 15 09:56 ..
-rw------- 1 nobody nogroup 950586441728 Aug 15 07:32 datadisk.d2538e1e-7166-4e57-ac85-09cdcd97822e.qcow2
-rw-r--r-- 1 nobody nogroup 8060 Aug 15 06:04 domain-config.xml
-rw-r--r-- 1 nobody nogroup 272 Aug 15 06:04 domblklist.xml
-rw-r--r-- 1 nobody nogroup 178 Aug 15 06:04 domiflist.xml
-rw-r--r-- 1 nobody nogroup 421 Aug 15 06:04 dominfo.xml
-rw------- 1 nobody nogroup 261140578304 Aug 15 06:33 root.69d79690-5753-4972-8268-5fd38b791af7.qcow2versions
Ubuntu 24.04.4
10G networking over IPv6
Truenas NFS storage
Primary storage is Ceph
Other backup solutions for other hypervisors to the same storage have no issues with long running backups.
The steps to reproduce the bug
- Schedule or run backup
- Runs for a while
- Fails, no backups show in UI, data files seen on NFS storage
What to do about it?
No response
Activity
draggeta commentedon Aug 15, 2026
Now that I looked at it more, it does seem related to the setting:
The default is 60 minute. I don't know if it's safe to raise this and maybe giving backups their separate timeout value would be better, as I don't want all jobs to have a longer timeout if possible.
EDIT: nvm, changed the setting and started a backup and it still stopped after 60 minutes
TadiosAbebe commentedon Aug 16, 2026
@draggeta You can add a comma separated key value pairs containing timeouts for different command for the global setting
commands.timeout.For example
TakeBackupCommand=7200,RestoreBackupCommand=7200and restart your management servers. Then you will be able to see that parameter logged on the management server logs when taking or restoring a backupdraggeta commentedon Aug 16, 2026
Wow, didn't even see that option documented somewhere. I'll try it.
DaanHoogland commentedon Aug 17, 2026
we welcome any improvements to the docs, @draggeta .
draggeta commentedon Aug 17, 2026
That seems to have worked. Thank you guys.