Repository navigation
packet loss from KVM to SSVM public and private ip address #9246
Description
ISSUE TYPE
- Other
COMPONENT NAME
CLOUDSTACK VERSION
CloudStack 4.19.0.1
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
I'm using nested environment over vmware. promiscuous mode and MAC address change ,Forged Transmit are enabled in port group level. SSVM and console proxy agents are up.I'm having issue with packet loss from kvm machine to SSVM /Console proxy vm both private and public ip address. also check.sh scripts works occasionally.
root@cloudstack-kvm:~# bridge link show
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 2
5: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
6: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
7: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloud0 state forwarding priority 32 cost 100
9: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
10: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master cloudbr0 state forwarding priority 32 cost 100
====================================================================================
root@cloudstack-kvm:~# ip addr show cloudbr0
3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb4:a6ca/64 scope link
valid_lft forever preferred_lft forever
========================================
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
altname enp11s0
root@cloudstack-kvm:~#
STEPS TO REPRODUCE
root@cloudstack-kvm:~# ip addr show cloudbr0
3: cloudbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
inet 10.158.65.11/25 brd 10.158.65.127 scope global cloudbr0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb4:a6ca/64 scope link
valid_lft forever preferred_lft forever
========================================
root@cloudstack-kvm:~# ip addr show ens192
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master cloudbr0 state UP group default qlen 1000
link/ether 00:50:56:b4:a6:ca brd ff:ff:ff:ff:ff:ff
altname enp11s0
root@cloudstack-kvm:~#
EXPECTED RESULTS
ping should be working fine on both private and public ips.
ACTUAL RESULTS
root@cloudstack-kvm:~# ping -O 10.158.65.25
PING 10.158.65.25 (10.158.65.25) 56(84) bytes of data.
no answer yet for icmp_seq=1
no answer yet for icmp_seq=2
no answer yet for icmp_seq=3
no answer yet for icmp_seq=4
no answer yet for icmp_seq=5
no answer yet for icmp_seq=6
no answer yet for icmp_seq=7
64 bytes from 10.158.65.25: icmp_seq=8 ttl=64 time=0.284 ms
64 bytes from 10.158.65.25: icmp_seq=9 ttl=64 time=0.285 ms
64 bytes from 10.158.65.25: icmp_seq=10 ttl=64 time=0.250 ms
64 bytes from 10.158.65.25: icmp_seq=11 ttl=64 time=0.203 ms
64 bytes from 10.158.65.25: icmp_seq=12 ttl=64 time=0.238 ms
64 bytes from 10.158.65.25: icmp_seq=13 ttl=64 time=0.162 ms
64 bytes from 10.158.65.25: icmp_seq=14 ttl=64 time=0.219 ms
64 bytes from 10.158.65.25: icmp_seq=15 ttl=64 time=0.189 ms
64 bytes from 10.158.65.25: icmp_seq=16 ttl=64 time=0.280 ms
64 bytes from 10.158.65.25: icmp_seq=17 ttl=64 time=0.180 ms
64 bytes from 10.158.65.25: icmp_seq=18 ttl=64 time=0.177 ms
64 bytes from 10.158.65.25: icmp_seq=19 ttl=64 time=0.218 ms
64 bytes from 10.158.65.25: icmp_seq=20 ttl=64 time=0.262 ms
64 bytes from 10.158.65.25: icmp_seq=21 ttl=64 time=0.229 ms
64 bytes from 10.158.65.25: icmp_seq=22 ttl=64 time=0.319 ms
64 bytes from 10.158.65.25: icmp_seq=23 ttl=64 time=0.305 ms
Activity
boring-cyborg commentedon Jun 13, 2024
Thanks for opening your first issue here! Be sure to follow the issue template!
DaanHoogland commentedon Jun 13, 2024
@dominar250 I cleanup the description a bit, please let me know if i messed it up. I do not understand your full scenario yet. you are pinging the console proxy but that does not reflect in your description.
Can you expand a bit on the scenario and the problem?
Is this initial packet drop concequently reproducible and is it true for the SSVM as well?
dominar250 commentedon Jun 13, 2024
@DaanHoogland Thanks for addressing this issue. From SSVM/consoleproxy i can able to ping my kvm machine and management server which is stable . But I can't get constant ping from kvm or management server to the public/private ip address for ssvm and console vm. ping is toggling , initial level or after couple of pings
weizhouapache commentedon Jun 13, 2024
@dominar250
it might be cause public/private IP of system vms are in the same range
can you share output of "ip a" in the system vms ?
dominar250 commentedon Jun 13, 2024
@DaanHoogland
root@s-55-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 0e:00:a9:fe:88:07 brd ff:ff:ff:ff:ff:ff
altname enp0s3
altname ens3
inet 169.254.136.7/16 brd 169.254.255.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 1e:00:f7:00:00:02 brd ff:ff:ff:ff:ff:ff
altname enp0s4
altname ens4
inet 10.158.65.24/25 brd 10.158.65.127 scope global eth1
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 1e:00:64:00:00:15 brd ff:ff:ff:ff:ff:ff
altname enp0s5
altname ens5
inet 10.158.65.13/25 brd 10.158.65.127 scope global eth2
valid_lft forever preferred_lft forever
dominar250 commentedon Jun 13, 2024
system_vm-ip.txt
weizhouapache commentedon Jun 13, 2024
@dominar250
I think it is because your systemvm have two nics/ips in the same range
typically the public and private should use different ip range and vlan/vni.
dominar250 commentedon Jun 13, 2024
@weizhouapache So cloudstack could'nt work with single ip range ?
weizhouapache commentedon Jun 13, 2024
There may be some issues with vm console and ssvm (download template/volume, etc).
Everything else should work.
It is not recommended to use single ip range. If you are just playing with cloudstack, it is ok. If you want to run a product env, you need a better design.
dominar250 commentedon Jun 13, 2024
@weizhouapache Thanks for the suggestion. Is it a good idea to run management, KVM, and NFS on a single machine? Does this setup cause routing-related issues? My plan is to use CloudStack only for Kubernetes cluster development
DaanHoogland commentedon Jun 14, 2024
sounds good @dominar250 , but still a separate ip range for the host and for the SVMs (and VMs in general) is best.