Repository navigation
CloudStack 4.20 UI Unable to add KVM-Host on VXLAN Logic structure - Errors #10636
Description
problem
On CloudStack 4.20 with a KVM host,
modified the agent.properties private.network.device=cloudbr1 as originally have not resources available for cloudbr0
I have my VXLAN 1 on Cloudbr1 that reaches management and SS, We only have cloudbr1 created, No other bridge, and everything is BGP.
Then at the UI creating the ZONE I got errors:
We have had tryed the following Options:
Option 1: No tags
I put only [Physical Network 1] [VXLAN] Management, throws a popup error " All required traffic types should be added and with multiple physical networks each traffic type should have a label"
Option 2: No Tags, Throws and Error " All required traffic types should be added and with multiple physical networks each traffic type should have a label"
I put :
[Physical Network 1] [VXLAN] Management
[Physical Network 2] [VXLAN] Public
[Physical Network 3] [VXLAN] Guest
[Physical Network 4] [VXLAN] Storage
Option 3:
I put :
[Physical Network 1] [VXLAN] Management, Public, Guest, it goes to the next window - which is Great
Going with Option 3:
The the system throws exceptions like:
Something went wrong; please correct the following:
Unable to add the host: Cannot find the server resources at http://
At this point libvirt is always up and running.
LOGS:
Management logs shows:
Succeeded to import certificate in the keystore for agent on the KVM host: . Agent secured and trusted
The Agent Logs:
2025-03-28 16:48:42,550 DEBUG [commons.beanutils.ConvertUtils] (main:[]) (logid:) Convert string ',' to class 'java.lang.String'
2025-03-28 16:48:42,550 DEBUG [beanutils.converters.StringConverter] (main:[]) (logid:) Converting 'String' value ',' to type 'String'
2025-03-28 16:48:42,551 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [guid] has empty or null value. Using default value [null].
2025-03-28 16:48:42,552 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [developer] has empty or null value. Using default value [false].
2025-03-28 16:48:53,897 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Agent started
2025-03-28 16:48:53,902 INFO [cloud.agent.AgentShell] (main:[]) (logid:) Implementation Version is 4.20.0.0
2025-03-28 16:48:53,904 INFO [cloud.agent.AgentShell] (main:[]) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2025-03-28 16:48:53,913 DEBUG [agent.properties.AgentPropertiesFileHandler] (main:[]) (logid:) Property [host] was altered. Now using the value [10.23.123.61,10.23.123.62].
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) Setting default value: false
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) Converting 'Boolean' value 'false' to type 'Boolean'
2025-03-28 16:48:53,923 DEBUG [beanutils.converters.BooleanConverter] (main:[]) (logid:) No conversion required, value is already a Boolean
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Byte'
2025-03-28 16:48:53,925 DEBUG [beanutils.converters.ByteConverter] (main:[]) (logid:) Converted to Byte value '0'
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) Setting default value:
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) Converting 'Character' value ' ' to type 'Character'
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.CharacterConverter] (main:[]) (logid:) No conversion required, value is already a Character
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,926 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Double'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.DoubleConverter] (main:[]) (logid:) Converted to Double value '0.0'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Setting default value: 0
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Converting 'Integer' value '0' to type 'Float'
2025-03-28 16:48:53,927 DEBUG [beanutils.converters.FloatConverter] (main:[]) (logid:) Converted to Float value '0.0'
2025-03-28 16:48:53,928 DEBUG [beanutils.converters.IntegerConverter] (main:[]) (logid:) Setting default value: 0
AGENT out :
Unable to find the guid
Unable to find the guid
Unable to find the guid
Unable to find the guid
Unable to find the guid
There is nothing there
And the agent come up, and immediately goes down!!
Tata Y.
versions
ACS 4.20.0.0
Management1 Ubuntu 22.04
Management2 ubuntu 22.04
Primary Storage CEPH
Secondary Storage EMC NFS
Hypervisor: KVM UBUNTU 22.04
The steps to reproduce the bug
- Adding the first Zone Go over the process
- on network [Physical Network 1] [VXLAN] Management, Public, Guest
- Add hosts, Kvm host keeps bouncing,
- enable debug logs: - there are no additional logs -
What to do about it?
No response
Activity
Gunseerat commentedon Apr 9, 2025
Can you try entering guid from libvirt file, actually libvirt is responsible for sending this guid to cloudstack-agent.
Open /etc/libvirt/libvirtd.conf file
find host_uuid
copy this value
go to /etc/cloudstack/agent/agent.properties
in guid field which should be empty for you .... you can paste this.
Although this is sort of injecting values deliberately
weizhouapache commentedon Apr 9, 2025
@tatay188
there is log file
setup.login the same folder asagent.logyou can have a look
tatay188 commentedon Apr 9, 2025
### On the AGENT side:
Not sure if this is just related to Ubuntu systems:
there is an error on the management server every time i reload it:
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management stop
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-management defaults
DEBUG:root:Failed to execute:update-rc.d: ### error: unable to read /etc/init.d/cloudstack-management
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-management status
DEBUG:root:Failed to execute:× cloudstack-management.service - CloudStack Management Server
Loaded: loaded (/lib/systemd/system/cloudstack-management.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2025-04-02 13:13:48 UTC; 1 day 8h ago
Main PID: 537731 (code=exited, status=143)
CPU: 27min 54.476s
Obviously there is no such file: Should I create One?
ls -al /etc/init.d/
cryptdisks grub-common iwpmd nfs-common plymouth rng-tools-debian ufw
apparmor cryptdisks-early hwclock.sh keyboard-setup.sh ntp plymouth-log rpcbind unattended-upgrades
apport dbus ipmievd kmod open-iscsi procps rsync uuidd
console-setup.sh dpdk irqbalance lvm2 openipmi quota ssh x11-common
cron frr iscsid lvm2-lvmpolld openvswitch-switch
### On the management side, only when i restart the agent:
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Starting Uncomplicated firewall...
Apr 03 18:41:23 kvmvcompatl2001 systemd[1]: Finished Uncomplicated firewall.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopping Uncomplicated firewall...
Apr 03 23:30:14 kvmvcompatl2001 ufw-init[56782]: Skip stopping firewall: ufw (not enabled)
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: ufw.service: Deactivated successfully.
Apr 03 23:30:14 kvmvcompatl2001 systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:### Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:○ cloudstack-agent.service - CloudStack Agent
Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since Thu 2025-04-03 23:30:17 UTC; 31s ago
Docs: http://www.cloudstack.org/
Main PID: 56558 (code=exited, status=1/FAILURE)
CPU: 14.716s
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: Stopped CloudStack Agent.
Apr 03 23:30:17 kvmvcompatl2001 systemd[1]: cloudstack-agent.service: Consumed 14.716s CPU time.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start
### Regarding the VXLAN i found the Cloudstack does not link the Interface Loopback to the VXLAN interface,
I did manually and now it propagates the routes
CLOUDSTACK ALONE DOES NOT ADD THE PARENT Loopback interface:
vtysh -c 'show interface vxlan1000'
Interface vxlan1000 is up, line protocol is up
Link ups: 4 last: 2025/04/06 03:16:37.88
Link downs: 0 last: (never)
vrf: default
index 143 metric 0 mtu 9050 speed 0 txqlen 1000
flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 7e:05:79:2f:33:7d
Interface Type Vxlan
Interface Slave Type Bridge
VTEP IP: 10.23.133.14 VxLAN Id 1000 Access VLAN Id 1
Master interface: brvx-1000
protodown: off
Once I manually Link the parent Loopback interface:
vtysh -c 'show interface vxlan1000'
Interface vxlan1000 is up, line protocol is up
Link ups: 0 last: (never)
Link downs: 0 last: (never)
vrf: default
index 17 metric 0 mtu 9050 speed 0 txqlen 1000
flags: <UP,LOWER_UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: b6:9d:d9:66:b1:60
Interface Type Vxlan
Interface Slave Type Bridge
VTEP IP: 10.23.133.14 Link Interface lo VxLAN Id 1000 Access VLAN Id 1
Master interface: brvx-1000
protodown: off
Parent interface: lo
Is propagating the routes, and I can reach from the remote VTEP
Now the System VMs agent-state shows green
But still the System Vms' Interface name are not populated, should it be populated ? what is the normal status for this?
Please let me know.
Tata Y.