Repository navigation
Attempt to deploy fresh 4.20.1 utilizing xcp-ng 8.3 fails to deploy system VMs #11162
Description
Hello team,
We decided to test out latest xcp-ng 8.3 hosts that are currently also visible through an XO tool. We waited for the Cloudstack version to support it and tried the required steps today. Our previous test with 4.20 was using KVM and we did not like the cluster HA delays we witnessed. XCP-ng seems a bit more promising. It seems that the process to deploy the system VMs failed because the XCP-ng hosts cannot mount the nfs share required. Attaching a log for your reference, let me know if you can help with what we are doing wrong.
Activity
DaanHoogland commentedon Jul 8, 2025
@alsko-icom , the logs are not visible.
alsko-icom commentedon Jul 8, 2025
cloudstack4.20.1.alsko.txt reuploaded file with txt extension.
DaanHoogland commentedon Jul 8, 2025
it is now zero bytes, @alsko-icom . can you create a gist or use some other upload/ftp service?
alsko-icom commentedon Jul 8, 2025
added the file to my personal nextcloud server and as a gist too.
https://gist.github.com/alsko-icom/c835b6f4ad87a27bd3ca96604dcf9c4c
and the nextcloud
https://secure.you-dream.com/index.php/s/PeYpjtBw9fJKHfm
Pearl1594 commentedon Jul 8, 2025
@alsko-icom can you please confirm if the systemvm template has been registered / seeded?
DaanHoogland commentedon Jul 8, 2025
@alsko-icom please checkt the template as @Pearl1594 suggests.
Why do you think the mount of the nfs share is the problem?
alsko-icom commentedon Jul 10, 2025
pasting here screenshot of the seed command for your reference
The team noticed some errors during deployment regarding NFS SR being unable to mount on the xcp-ng hosts. So if nfs mount failed, there was no way for the hosts to see the system vm template.
weizhouapache commentedon Jul 17, 2025
correct.
have you fixed the nfs issue ? @alsko-icom
alsko-icom commentedon Jul 23, 2025
Excuse my late reply, somehow i missed the notification of your update! Is there a problem that we have all 4 xcp-ng hosts connecting to the nfs? Not sure how to tackle. The NFS is not seen to be provisioned / mounted to the entire server pool of 4 but just to 1 server. So if the mount tries to switch to another server, the new server does not know it. Maybe we should try another approach? Also, when adding servers to the xcp-ng server pool, should they be added to the pool through cloudstack by being previously standalone or is it ok to be already added to the pool through an XO environment and the cloudstack will accept them in?
DaanHoogland commentedon Jul 23, 2025
No @alsko-icom , unless there are some setting on the nfs server preventing it, all servers should be able to share the NFS storage.
alsko-icom commentedon Jul 23, 2025
ok we will review the nfs behavior a bit more. Regarding the prep, should we add the xcp-ng hosts to the pool through cloudstack or through xen orchestrator at a lower level is fine and will not bother cloudstack?
DaanHoogland commentedon Jul 23, 2025
@alsko-icom , I think you can add it to the pool and add the pool-master. please check though.
alsko-icom commentedon Jul 28, 2025
Regarding the nfs storage it has been mounted on the xcp hosts
This is the df - h output
and during the system vm deployment attempt I get an error
Jul 28 15:35:51 hpe-xcp3 xapi: [error||5690380 |SR.probe D:31ab2dcc8c12|backtrace] sm_exec D:13bcbc145ea9 failed with exception Storage_error ([S(Backend_error);[S(SR_BACKEND_FAILURE_73);[S();S(NFS mount error [opterr=mount failed on server
10.200.30.225with return code 110]);S()]]])attaching the log file /var/log/xensource.log for 1 hour today between 15:00 and 15:59 as the deployment attempt was around 15:30.
xcp3-xensource.log
DaanHoogland commentedon Jul 29, 2025
@alsko-icom , what type of NFS server are you running and what options are turned on on it? (squash_root, nfs version, …?)
See this part of the docs for more information on primary storages.
Or see this blog for a concise explanation.
alsko-icom commentedon Jul 30, 2025
After going in the Netapp NFS configuration with the storage team to check settings you asked me, i witnessed them turning off the v4 and v4.1-4.2 support on the NFS share. So they left only v3 enabled. When enabling the zone in cloudstack, the system VMs where started successfully. There might be some issue with v4.x versions of it. So now it works!