Skip to content

Migration was refused connection to the destination. Please check libvirt configurations compatibility and firewall  #9965

Closed
Closed
Migration was refused connection to the destination. Please check libvirt configurations compatibility and firewall #9965
Milestone
unplanned

Description

@verendrakalsi
verendrakalsi
opened on Nov 22, 2024

Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.

ISSUE TYPE
  • Other
COMPONENT NAME
CLOUDSTACK VERSION
CONFIGURATION

3 KVM hosts on Ubuntu 22.04 all are in same subnet.
Zone :- Datacenter-Linux
Pod :- Linux
Clusters :- Linux

enable automatic DRS in global settings and also on the cluster.

Firewall is disable on cloudstack and kvm hosts.

OS / ENVIRONMENT
SUMMARY

2024-11-22 17:34:27,689 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:ctx-36934979 job-484/job-485 ctx-bb524027) (logid:1ece315b) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,690 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-15:ctx-36934979 job-484/job-485 ctx-bb524027) (logid:1ece315b) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,690 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-15:ctx-36934979 job-484/job-485) (logid:1ece315b) Unable to complete AsyncJobVO: {id:485, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAAIHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAnNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345051317061, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Nov 22 17:34:26 UTC 2024, removed: null}, job origin:484
com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,698 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-15:ctx-36934979 job-484/job-485) (logid:1ece315b) Complete async job-485, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvblZNT3AAAAACAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0ALxNaWdyYXRpb24gd2FzIHJlZnVzZWQgY29ubmVjdGlvbiB0byBkZXN0aW5hdGlvbjogcWVtdSt0Y3A6Ly8xOTIuMTY4LjEwMS4xMDAvc3lzdGVtLiBQbGVhc2UgY2hlY2sgbGlidmlydCBjb25maWd1cmF0aW9uIGNvbXBhdGliaWxpdHkgYW5kIGZpcmV3YWxsIHJ1bGVzIG9uIHRoZSBzb3VyY2UgYW5kIGRlc3RpbmF0aW9uIGhvc3RzLnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFnNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ABUwADmRlY2xhcmluZ0NsYXNzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFTAAKbW9kdWxlTmFtZXEAfgAFTAANbW9kdWxlVmVyc2lvbnEAfgAFeHABAAALA3QAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAHbWlncmF0ZXBwc3EAfgAMAQAACodxAH4ADnEAfgAPcQB-ABB0ABJvcmNoZXN0cmF0ZU1pZ3JhdGVwcHNxAH4ADAEAABWJcQB-AA5xAH4AD3EAfgAQcQB-ABNwcHNxAH4ADAL____-cHQALWpkay5pbnRlcm5hbC5yZWZsZWN0Lk5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbHQAHU5hdGl2ZU1ldGhvZEFjY2Vzc29ySW1wbC5qYXZhdAAHaW52b2tlMHQACWphdmEuYmFzZXQABzExLjAuMjVzcQB-AAwCAAAAPnBxAH4AFnEAfgAXdAAGaW52b2tlcQB-ABlxAH4AGnNxAH4ADAIAAAArcHQAMWpkay5pbnRlcm5hbC5yZWZsZWN0LkRlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGx0ACFEZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsLmphdmFxAH4AHHEAfgAZcQB-ABpzcQB-AAwCAAACNnB0ABhqYXZhLmxhbmcucmVmbGVjdC5NZXRob2R0AAtNZXRob2QuamF2YXEAfgAccQB-ABlxAH4AGnNxAH4ADAEAAABpcQB-AA50ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JwcHNxAH4ADAEAABXqcQB-AA5xAH4AD3EAfgAQcQB-ACZwcHNxAH4ADAEAAABmcQB-AA50ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnBwc3EAfgAMAQAAAo5xAH4ADnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHBwc3EAfgAMAQAAADBxAH4ADnQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVucHBzcQB-AAwBAAAAN3EAfgAOdABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxscHBzcQB-AAwBAAAAZnEAfgAOdABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA2dAAPY2FsbFdpdGhDb250ZXh0cHBzcQB-AAwBAAAANHEAfgAOcQB-ADlxAH4ANnQADnJ1bldpdGhDb250ZXh0cHBzcQB-AAwBAAAALXEAfgAOdAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ADJxAH4AM3Bwc3EAfgAMAQAAAlpxAH4ADnEAfgAtcQB-AC5xAH4AM3Bwc3EAfgAMAgAAAgNwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ADdxAH4AGXEAfgAac3EAfgAMAgAAAQhwdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAzcQB-ABlxAH4AGnNxAH4ADAIAAARocHQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2VycQB-ABlxAH4AGnNxAH4ADAIAAAJ0cHQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ASHEAfgAzcQB-ABlxAH4AGnNxAH4ADAIAAAM9cHQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAzcQB-ABlxAH4AGnNyAB9qYXZhLnV0aWwuQ29sbGVjdGlvbnMkRW1wdHlMaXN0ergXtDynnt4CAAB4cHgAABCadwgAAAAAAAAAAHg
2024-11-22 17:34:27,726 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-6:ctx-b5d1c146 job-484) (logid:1ece315b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
com.cloud.utils.exception.CloudRuntimeException: Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts.
2024-11-22 17:34:27,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-b5d1c146 job-484) (logid:1ece315b) Complete async job-484, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Migration was refused connection to destination: qemu+tcp://192.168.101.100/system. Please check libvirt configuration compatibility and firewall rules on the source and destination hosts."}

STEPS TO REPRODUCE
EXPECTED RESULTS
ACTUAL RESULTS

Activity

  1. btzq commented on Nov 24, 2024

    @btzq
    btzq
    on Nov 24, 2024

    @verendrakalsi can you explain bit more instead of just pasting the logs.

  2. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    not sure why all the details are not showing up, i have updated it.

    Anyways.

    I have 3 KVM hosts in same subnet and cloudstack mgmt on a different subnet but there is communication between KVM hosts & cloudstack mgmt. There is no firewall in between.

    I have enabled DRS at global & cluster level and also they are on same storage as well. However either DRS runs automatic or manual, the VMs doesnt move and i get this error

  3. slavkap commented on Nov 25, 2024

    @slavkap
    slavkap
    on Nov 25, 2024
    Contributor

    @verendrakalsi, are libvirt TCP connections allowed?
    The problem is that the connection was refused. This is for RHEL but probably could help in solving the issue unable to connect to server

  4. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    Actually i am using ubuntu 22.04 as base OS not RHEL and yes Libvirt TCP connection is allowed

  5. weizhouapache commented on Nov 25, 2024

    @weizhouapache
    weizhouapache
    on Nov 25, 2024
    Member

    @verendrakalsi
    can you check /etc/default/libvirtd if it has the following config ?

    LIBVIRTD_ARGS="--listen"

  6. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    Here is the screenshot from one of the server and it is same for all of them.

    image

  7. weizhouapache commented on Nov 25, 2024

    @weizhouapache
    weizhouapache
    on Nov 25, 2024
    Member

    Here is the screenshot from one of the server and it is same for all of them.

    can you remove the line LIBVIRTD_ARGS="" and restart libvirtd ?

    if everything works, the output should be same as

    # ps -ef|grep libvirtd
root         969       1  0 Oct31 ?        00:20:07 /usr/sbin/libvirtd -l

  8. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    i have commented the line and i get this output after running the ps command.

    image

  9. slavkap commented on Nov 25, 2024

    @slavkap
    slavkap
    on Nov 25, 2024
    Contributor

    @verendrakalsi, can you check that TCP port for libvirt is open?

    netstat -lntp | grep libvirtd

    this is in my lab

    
[root@cs-dev-kvm1-4 packages]# cat /etc/libvirt/libvirtd.conf | grep listen_tcp
#listen_tcp = 1
listen_tcp=0
[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
error: failed to connect to the hypervisor
error: unable to connect to server at '127.0.0.1:16509': Connection refused

    after changing to listen_tcp=1

    [root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

  10. weizhouapache commented on Nov 25, 2024

    @weizhouapache
    weizhouapache
    on Nov 25, 2024
    Member

    @verendrakalsi, can you check that TCP port for libvirt is open?

    netstat -lntp | grep libvirtd

    this is in my lab

    
[root@cs-dev-kvm1-4 packages]# cat /etc/libvirt/libvirtd.conf | grep listen_tcp
#listen_tcp = 1
listen_tcp=0
[root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
error: failed to connect to the hypervisor
error: unable to connect to server at '127.0.0.1:16509': Connection refused

    after changing to listen_tcp=1

    [root@cs-dev-kvm1-4 packages]# virsh -c qemu+tcp://127.0.0.1/system
setlocale: No such file or directory
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

    for Secure host, it uses tls

    listen_tcp=0
    listen_tls=1

  11. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    As per the documentation https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#kvm-installation-overview
    The both should be '0'
    listen_tcp=0
    listen_tls=0

    However i changed it to this and now migration is working.
    listen_tcp=1
    listen_tls=0

    I am going to load more VMs and see if even automatic is working or not. mostly it should.

  12. weizhouapache commented on Nov 25, 2024

    @weizhouapache
    weizhouapache
    on Nov 25, 2024
    Member

    As per the documentation https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#kvm-installation-overview The both should be '0' listen_tcp=0 listen_tls=0

    However i changed it to this and now migration is working. listen_tcp=1 listen_tls=0

    I am going to load more VMs and see if even automatic is working or not. mostly it should.

    did you add the host via cloudstack UI ?

    it looks like your host is unsecure. you can go to host details and click "provision host security keys" which will secure cloudstack-agent and libvirtd with certificates.
    please refer to https://docs.cloudstack.apache.org/en/latest/adminguide/hosts.html#securing-process

  13. verendrakalsi commented on Nov 25, 2024

    @verendrakalsi
    verendrakalsi
    on Nov 25, 2024
    Author

    yes i did added it via cloudstacck UI and yes it was unsecure.
    I just secured it on all the hosts.
    Thank you for this information

  14. apache
    locked and limited conversation to collaborators on Nov 27, 2024
  15. DaanHoogland
    converted this issue into a discussion #9987 on Nov 27, 2024
  16. DaanHoogland
    added this to the unplanned milestone on Sep 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions