Repository navigation
console not working after system vm has been recreated on a different cluster #10050
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
Console Proxy
CLOUDSTACK VERSION
4.19.1.2
CONFIGURATION
Management server on Ubuntu 22.04 LTS
KVM hypervisor on Ubuntu 22.04 LTS
OS / ENVIRONMENT
SUMMARY
We run a multiple cluster installation. After deleting console proxy vm, Cloudstack recreated a new system vm on antoher cluster.
After that console access for instances running on a different cluster than the console proxy vm is running on, did not work anymore. But after stopping and starting an instance, console access was working again.
STEPS TO REPRODUCE
1. detroy console proxy system vm
2. let CS recreate the console proxy system vm and hope it will be created on another cluster
3. try to open a console on an instance running on a different cluster as the console proxy is running on (not working)
4. stop and start this instance and try again (working again)
5. try to open the console on an instance running on the same cluster as the console proxy is running on (working)
EXPECTED RESULTS
console should work on all instances even after CS recreated it on another cluster
ACTUAL RESULTS
console is only working on instances running on the same cluster as the console proxy system vm. After stop and start of an instance, console is working again.
Activity
yadvr commentedon Dec 9, 2024
cc @nvazquez @weizhouapache @DaanHoogland to help review/triage if it's a regression or some kind of limitation we should address in next 4.19.2/4.20.1
weizhouapache commentedon Dec 9, 2024
@sbrueseke
can you share some network information of the clusters ?
if possible, please share the output of "ip route" in the CPVM as well
DaanHoogland commentedon Dec 9, 2024
@sbrueseke are the clusters in the same pod? (I don't think it relevant, but it might)
is there a recent update involved that changed behaviour, that you know off?
sbrueseke commentedon Dec 9, 2024
Clusters are in the same pod.
Cluster A
root@pc-kvm-1:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.21
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.21
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
Cluster B
root@pc-kvm-6:~# ip route
default via 10.10.1.1 dev cloudbr0 proto static
10.10.1.0/24 dev cloudbr0 proto kernel scope link src 10.10.1.26
10.10.7.0/24 dev bond2 proto kernel scope link src 10.10.7.26
169.254.0.0/16 dev cloud0 proto kernel scope link src 169.254.0.1
weizhouapache commentedon Dec 12, 2024
@sbrueseke
looks like simple setup which should work.
any firewall rules on the kvm hosts ?
sbrueseke commentedon Dec 12, 2024
@weizhouapache no firewall on hosts
weizhouapache commentedon Dec 12, 2024
can you also check these steps ?
sbrueseke commentedon Dec 12, 2024
root@pc-kvm-1:~$ virsh domdisplay i-21-283-VM
vnc://10.10.1.21:1
root@v-529-VM:~# ping -c1 10.10.1.21
PING 10.10.1.21 (10.10.1.21): 56 data bytes
64 bytes from 10.10.1.21: icmp_seq=0 ttl=64 time=0.291 ms
--- 10.10.1.21 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.291/0.291/0.291/0.000 ms
root@v-529-VM:~# telnet 10.10.1.21 5901
Trying 10.10.1.21...
Connected to 10.10.1.21.
Escape character is '^]'.
RFB 003.008
weizhouapache commentedon Dec 12, 2024
@sbrueseke
Has the vm been running for long time, let's say more than 1 year ?
sbrueseke commentedon Dec 12, 2024
i-21-283-VM was created in June this year and last updated on July 16th. Are you able to reproduce this issue on your installation? Everything worked fine, before we recreated the console proxy VM and it was put on another cluster.
sbrueseke commentedon Dec 17, 2024
@weizhouapache are you able to reproduce this issue?
weizhouapache commentedon Dec 17, 2024
@sbrueseke
no, everything works in my testing
can you check if the hosts/vms are time-synced ?
you can enable debugging in CPVM and retry. You might get more information in /var/log/cloud.log