Repository navigation
Bridge interface not recognized for newly added host if they don't have a physical interface attached #10727
Description
problem
So I added a KVM hypervisor running AlmaLinux 9.5 to a Cloudstack Management Server via the UI, which failed with error Unable to add the host: Cannot find the server resources at <host>
While digging in the cloudstack management-server.log file, I noticed that my bridge br_npf0 is not found according to the management server:
2025-04-15 14:47:43,865 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) PingMap for agent: 1 will not be updated because agent is no longer in the PingMap
2025-04-15 14:47:43,866 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-7:[]) (logid:) Not processing PingRoutingCommand for agent id=0; can't find the host in the DB
2025-04-15 14:47:43,867 DEBUG [c.c.a.t.Request] (AgentManager-Handler-8:[]) (logid:) Seq 1-9217179587367141377: Processing: { Ans: , MgmtId: 90520739542428, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.CheckNetworkAnswer":{"_reconnect":"false","result":"false","details":"Can not find network: br_npf0","wait":"0","bypassHostMaintenance":"false"}}] }
2025-04-15 14:47:43,867 DEBUG [c.c.a.m.ClusteredAgentAttache] (AgentManager-Handler-8:[]) (logid:) Seq 1-9217179587367141377: No more commands found
2025-04-15 14:47:43,867 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Seq 1-9217179587367141377: Received: { Ans: , MgmtId: 90520739542428, via: 1(reacted_host_name.local), Ver: v1, Flags: 110, { CheckNetworkAnswer } }
2025-04-15 14:47:43,867 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Details from executing class com.cloud.agent.api.CheckNetworkCommand: Can not find network: br_npf0
2025-04-15 14:47:43,867 WARN [o.a.c.e.o.NetworkOrchestrator] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Unable to setup agent 1 due to Can not find network: br_npf0
2025-04-15 14:47:43,870 WARN [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) alertType=[7] dataCenterId=[1] podId=[1] clusterId=[null] message=[Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0].
2025-04-15 14:47:43,878 WARN [c.c.a.AlertManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0] and content [Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0].
2025-04-15 14:47:43,878 INFO [c.c.u.e.CSExceptionErrorCode] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
[...]
2025-04-15 14:47:43,890 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentConnectTaskPool-596:[ctx-58c65884]) (logid:c8f5bf3e) Failed to handle host connection: com.cloud.exception.ConnectionException: Incorrect Network setup on agent, Reinitialize agent after network names are setup, details : Can not find network: br_npf0
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.processConnect(NetworkOrchestrator.java:4321)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:553)
at com.cloud.agent.manager.AgentManagerImpl.sendReadyAndGetAttache(AgentManagerImpl.java:1116)
at com.cloud.agent.manager.AgentManagerImpl.handleConnectedAgent(AgentManagerImpl.java:1135)
at com.cloud.agent.manager.AgentManagerImpl$HandleAgentConnectTask.runInContext(AgentManagerImpl.java:1227)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Looking at my configuration on the KVM host, the bridge br_npf0 exists, is up, has an IP, and can ping the management server.
ip a | grep br_npf0
10: br_npf0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 10.13.37.2/24 brd 10.13.37.255 scope global noprefixroute br_npf0
28: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br_npf0 state UP group default qlen 1000
# pinging the cloudstack management server
ping cloudstack01i.npf.local -c 1
PING cloudstack01i.npf.local (10.13.37.250) 56(84) bytes of data.
64 bytes from cloudstack01i.npf.local (10.13.37.250): icmp_seq=1 ttl=64 time=0.687 ms
--- cloudstack01i.npf.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.687/0.687/0.687/0.000 ms
The management server zone is setup with that exact same bridge name:
Is there any direction to point me to ?
versions
Cloudstack 4.20 running on AlmaLinux 9.5
KVM host AlmaLinux 9.5 with bridge setup via NetworkManager
The steps to reproduce the bug
What to do about it?
No response
Activity
weizhouapache commentedon Apr 15, 2025
@deajan
can you run some commands on the kvm host ?
deajan commentedon Apr 15, 2025
Sure
Dates are recent because I disconnected / reconnected eno2 just to make sure (and yes, the management server still complains)
weizhouapache commentedon Apr 15, 2025
@deajan
looks good
can you share the agent.log ?
deajan commentedon Apr 15, 2025
Sure:
It looks like it complains about lscpu (and iscsi I haven't setup, going to use local storage for tests), but nothing about the bridges.
deajan commentedon Apr 15, 2025
@weizhouapache While digging into my problem, I am thinking of a potential issue
Do the bridges need to be physically connected ?
I have one bridge (br_bgp0) that is "fed" by a vxlan interface (not connected to a physical interface but to a wireguard instance in order to transport some public IPs into my lab) on this KVM host.
This allows me to have my public IPs directly bridged from my bgp routers into my lab host.
Is this "allowed" ?
If not, that could perhaps explain why cloudstack complains about my bridges, even though they work.
weizhouapache commentedon Apr 15, 2025
change the log level and restart cloudstack-agent ?
yes, that's ok
deajan commentedon Apr 15, 2025
Hmmm... interesting.
Looks like indeed using a bridge in cloudstack is limited to physical interfaces.
I've done a (stupid) check:
Now the agent doesn't complain anymore, since there's now a
eth*in/sys/devices/virtual/net/br_bgp0/brifI guess that indeed the bridge tests are a bit too restrictive.
I wonder if the test should be less restrictive, like checking if there is at least one non vtnet* interface in the bridge, so people would be allowed to use whatever they want behind the bridge (like vxlans, gretap or whatever).
This would also allow make cloudstack future proof for ethernet driver names. Perhaps one day we'll have igb* or whatever driver names.
I can do python and bash PRs, but I really am not java fluent.
Is there anyway this could be discussed for perhaps some next release ?
@weizhouapache Big thanks for the hints.
[-]Network interface not found for newly added host[/-][+]Bridge interface not recognized for newly added host if they don't have a physical interface attached[/+]weizhouapache commentedon Apr 15, 2025
that's strange.
in your previous reply, it showed that the
eno2is presentdeajan commentedon Apr 15, 2025
I think I mixed up br_npf0 and br_bgp0 in my tests (deleted & recreated bridges multiple times in order to diagnose my issue).
Point is, my bgp interface, the bridge br_bgp0 with a vxlan interface, wasn't accepted by Cloudstack agent since the bridge test complained that there wasn't a physical interface. There's still no physical interface yet, but I fooled the agent test by creating a dummy interface slave in that bridge and naming it ethdummy0.
I think those tests should be relaxed, as I think they only should test that there are other interfaces than vnet* connected to the bridge.
weizhouapache commentedon Apr 15, 2025
hmm, has the kvm host been added to cloudstack ?
deajan commentedon Apr 15, 2025
Yes, host is now visible and manageable in cloudstack.
Anything I need to check ?
weizhouapache commentedon Apr 16, 2025
Are the system vms Running and agent states are Up ?
weizhouapache commentedon Apr 16, 2025
@deajan
just to confirm, is br_bgp0 used as the kvm network label of a physical network ?
deajan commentedon Apr 16, 2025
@weizhouapache You mean like this ?
I don't have system VMs yet (it's a fresh cloudstack lab setup), but host is up and running according to management server.
I've redone a specific check:
Of course, this will prevent me from using cloudstack properly until I reconnect eno2, but at it validates the assumption.
weizhouapache commentedon Apr 16, 2025
I am a bit confused about br_npf0 and br_bgp0
anyway, all bridges used by cloudstack (check kvm network label) should have a physical ointerface (or virtual or dummy interface with proper name)
do you think anything we could improve or fix ?
deajan commentedon Apr 16, 2025
Sorry, as I said, at one moment, I think I have mixed both while doing tests creating/deleting bridges to find what the issue is.
For instance, br_npf has eno2 as slave interface, br_bgp0 has a vxlan as slave interface.
So basically the test to see whether there is
eth*, bond*, team*, vlan*, em*, p*p*, ens*, eno*, enp*, or enx*in/sys/devices/virtual/net/${bridge}/brifis not sufficient.I would change that check to report successful if at least one slave different than "vnet*" "or "vmbr*" is present (so we outrule running VM interfaces from uplinks).
This would allow to have bridge slaves interfaces being VXLan, GRETAP, Geneve or whatever fancy interfaces someone would like to use. It would also allow Cloudstack to be future proof if someday a new ethernet driver naming comes out.
weizhouapache commentedon Apr 16, 2025
as far as I know,
vnet*is used by vm instances on kvm host in cloudstack,vmbr*(vm bridge ?) is default bridge name of proxmox.deajan commentedon Apr 16, 2025
You are right, I actually ment
virbr*which would be NATed vm instances on kvm instead ofvmbr*... I don't use them often enough to remember the right name. Had to create one to check.