Repository navigation
Network isolation creation error #11743
Description
problem
I get an error when I try to create an isolated network.
2025-09-23 10:03:48,650 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Allocating the VR with id=5337 in datacenter Zone {"id": "3", "name": "PVCL.TEH.IR", "uuid": "d04f1ed7-09e9-4beb-853b-b4f4adbce9bd"} with the hypervisor type VMware
2025-09-23 10:03:48,679 DEBUG [c.c.n.r.NetworkHelperImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) VMware won't support system vm, skip it
2025-09-23 10:03:48,690 DEBUG [o.a.c.n.r.d.RouterDeploymentDefinition] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Lock is released for network [id: 210] (Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115}) as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(3)-Pod(null)-Cluster(null)-Host(null)-Storage()]
2025-09-23 10:03:48,698 WARN [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Failed to implement network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} elements and resources as a part of network restart due to com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:3] is unreachable: Can't find all necessary running routers!
at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:250)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1753)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1678)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3719)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2955)
at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2977)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at jdk.proxy3/jdk.proxy3.$Proxy200.restartNetwork(Unknown Source)
at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:97)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-09-23 10:03:48,699 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623, ctx-6af3b517]) (logid:efdae8f3) Network Network {"id": 210, "name": "amin-test", "uuid": "ae4e93f1-0536-4d65-a0a0-dccfa2a3ed9f", "networkofferingid": 115} failed to restart.
2025-09-23 10:03:48,733 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Complete async job-52623, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to restart network"}
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Publish async job-52623 complete on message bus
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs related to job-52623
2025-09-23 10:03:48,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Update db status for job-52623
2025-09-23 10:03:48,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Wake up jobs joined with job-52623 and disjoin all subjobs created from job- 52623
2025-09-23 10:03:48,751 DEBUG [c.c.a.ApiServer] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Retrieved cmdEventType from job info: NETWORK.RESTART
2025-09-23 10:03:48,775 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-52623
2025-09-23 10:03:48,775 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-11:[ctx-0fcb64cf, job-52623]) (logid:efdae8f3) Remove job-52623 from job monitoring
2025-09-23 10:03:49,217 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 routers to update status.
2025-09-23 10:03:49,222 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 1 VPC's to update Redundant State.
2025-09-23 10:03:49,224 DEBUG [c.c.n.r.V.CheckRouterTask] (RouterStatusMonitor-1:[ctx-38f7c6c5]) (logid:c3bcc01d) Found 0 networks to update RvR status.
2025-09-23 10:03:49,459 DEBUG [c.c.s.S.DbCollector] (StatsCollector-4:[ctx-deea6087]) (logid:40ac84bb) DbCollector is running...
2025-09-23 10:03:49,580 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-82351737]) (logid:070d357e) HA health check task is running...
2025-09-23 10:03:49,802 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) No inactive management server node found
2025-09-23 10:03:49,802 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-b8e48fac]) (logid:c375191b) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 2ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
versions
Cloudstack Management 4.21
Vmware 8.0.3
Activity
kiranchavala commentedon Sep 29, 2025
@Hossein4185 Looks like cloudstack was not able to reach the virtual router on vmware
Could check if the virtual router is running on vcenter and is reachable from management server
Hossein4185 commentedon Sep 29, 2025
@kiranchavala
Running on host but unreachable on cloudstack.
2025-09-29 17:43:27,305 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-0415c95b, ctx-61551cf1]) (logid:de53d393) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:27,479 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) ===START=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,480 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca]) (logid:3a198344) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:27,497 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:27,503 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:27,504 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:27,504 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:27,516 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-040112ca, ctx-042aa1bd]) (logid:3a198344) ===END=== 172.17.0.247 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:27,775 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:27,775 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-d7297205]) (logid:e90196b4) Peer scan is finished. profiler: Done. Duration: 7ms , profilerQueryActiveList: Done. Duration: 4ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:28,158 DEBUG [o.a.c.h.H.HAManagerBgPollTask] (BackgroundTaskPollManager-5:[ctx-d2786022]) (logid:e4d9f092) HA health check task is running...
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:29,276 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:29,276 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-e055b2aa]) (logid:8d29f66a) Peer scan is finished. profiler: Done. Duration: 5ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 2ms
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,420 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648]) (logid:e83af821) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,435 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,443 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,444 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,444 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,458 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-b990b648, ctx-dccb201e]) (logid:e83af821) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) ===START=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:29,961 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73]) (logid:e197f43d) Two factor authentication is already verified for the user 9, so skipping
2025-09-29 17:43:29,977 DEBUG [c.c.a.ApiServer] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) CIDRs from which account 'Account [{"accountName":"a.bahri","id":15,"uuid":"95be6584-640d-434d-8a1f-7df726a918ae"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:29,985 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) Account for user id e33a25d0-28c7-48df-b19f-131956f501a4 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:29,985 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:29,985 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:29,997 DEBUG [c.c.a.ApiServlet] (qtp1404565079-13333:[ctx-d1bd9e73, ctx-2f052f28]) (logid:e197f43d) ===END=== 172.17.2.60 -- GET managementserverid=9c427c44-5065-45ac-9480-c9e804ba168d&command=readyForShutdown&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) ===START=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,376 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c]) (logid:76b7253c) Two factor authentication is already verified for the user 12, so skipping
2025-09-29 17:43:30,381 DEBUG [c.c.a.m.D.Task] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Response Received:
2025-09-29 17:43:30,381 DEBUG [c.c.a.t.Request] (DirectAgent-458:[ctx-b9d84a2f]) (logid:e73a213c) Seq 16-4644900065679250305: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"Stop VM r-5387-VM Succeed","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,382 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250305: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,382 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Details from executing class com.cloud.agent.api.StopCommand: Stop VM r-5387-VM Succeed
2025-09-29 17:43:30,382 ERROR [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Failed to start instance VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2025-09-29 17:43:30,392 DEBUG [c.c.a.ApiServer] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) CIDRs from which account 'Account [{"accountName":"ho.karami","id":9,"uuid":"aa8fee67-15f6-41c8-a5e6-4cf64bd0878e"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2025-09-29 17:43:30,394 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Cleaning up resources for the vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,401 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) Account for user id 13a153b4-c31b-4a3a-833f-d895c29928c3 is Root Admin or Domain Admin, all APIs are allowed.
2025-09-29 17:43:30,401 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2025-09-29 17:43:30,401 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) API rate limiting is disabled. We will not use ApiRateLimitService.
2025-09-29 17:43:30,433 DEBUG [c.c.a.ApiServlet] (qtp1404565079-795:[ctx-3615110c, ctx-e743e658]) (logid:76b7253c) ===END=== 172.17.0.247 -- GET jobId=e73a213c-53bd-4f8c-a05f-3d771e7f9ee3&command=queryAsyncJobResult&response=json&
2025-09-29 17:43:30,439 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Wait time setting on com.cloud.agent.api.StopCommand is 1800 seconds
2025-09-29 17:43:30,445 DEBUG [c.c.a.m.ClusteredDirectAgentAttache] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Routed from 345050399512
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Sending { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,445 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Executing: { Cmd , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":"false","checkBeforeCleanup":"false","controlIp":"172.17.26.167","forceStop":"false","vlanToPersistenceMap":{"1000":"true"},"expungeVM":"false","volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,446 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:6f34189c) Seq 16-4644900065679250306: Executing request
2025-09-29 17:43:30,446 DEBUG [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Executing resource command StopCommand: [{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"172.17.26.167","forceStop":false,"vlanToPersistenceMap":{"1000":true},"expungeVM":false,"volumesToDisconnect":[],"vmName":"r-5387-VM","executeInSequence":true,"wait":0,"bypassHostMaintenance":false}].
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) find VM r-5387-VM on host
2025-09-29 17:43:30,467 INFO [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM not found in host cache
2025-09-29 17:43:30,467 DEBUG [c.c.h.v.m.BaseMO] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) load VM cache on host
2025-09-29 17:43:30,565 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-59:[ctx-b1ed20e4, esxi03.pvcl.afracloud.local, job-53846/job-53849, cmd: StopCommand]) (logid:e73a213c) VM r-5387-VM is already in stopped state
2025-09-29 17:43:30,566 DEBUG [c.c.a.m.D.Task] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Response Received:
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (DirectAgent-59:[ctx-b1ed20e4]) (logid:e73a213c) Seq 16-4644900065679250306: Processing: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":"true","details":"VM r-5387-VM is already in stopped state","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-29 17:43:30,566 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Seq 16-4644900065679250306: Received: { Ans: , MgmtId: 345050399512, via: 16(esxi03.pvcl.afracloud.local), Ver: v1, Flags: 10, { StopAnswer } }
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [c.c.n.g.ControlNetworkGuru] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Released nic: NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} for vm VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"}
2025-09-29 17:43:30,704 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) The nic Nic {"broadcastUri":null,"deviceId":1,"iPv4Address":"172.17.26.101","id":55319,"instanceId":5387,"reservationId":"1772dbf2-0235-4112-99ac-e84ae568fb8d","uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b"} on NicProfile {"broadcastUri":null,"deviceId":null,"iPv4Address":null,"id":55319,"reservationId":null,"uuid":"7d4b2a25-d59c-4844-85cc-1965c520337b","vmId":5387} was released according to VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} by guru com.cloud.network.guru.ControlNetworkGuru@1909b900, now updating record.
2025-09-29 17:43:30,728 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released network resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,749 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully released storage resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,750 DEBUG [c.c.v.ClusteredVirtualMachineManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Successfully cleaned up resources for the VM VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Starting","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} in Starting state
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Found 1 inactive management server node based on timestamp
2025-09-29 17:43:30,771 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) management server node msid: 345050417779, name: clst02.pvcl.afracloud.local, service ip: 172.24.48.134, version: 4.21.0.0
2025-09-29 17:43:30,771 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:[ctx-2b0964fe]) (logid:7567fe59) Peer scan is finished. profiler: Done. Duration: 4ms , profilerQueryActiveList: Done. Duration: 1ms, , profilerSyncClusterInfo: Done. Duration: 1ms, profilerInvalidatedNodeList: Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms,, profilerNewList: Done. Duration: 0ms, profilerInactiveList: Done. Duration: 1ms
2025-09-29 17:43:30,787 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) VM instance {"id":5387,"instanceName":"r-5387-VM","state":"Stopped","type":"DomainRouter","uuid":"c5fc54ed-81f9-4743-8bdb-48dc6379f9f8"} state transited from [Starting] to [Stopped] with event [OperationFailed]. VM's original host: null, new host: null, host before state transition: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total CPU: 301728 and CPU after applying overprovisioning: 301728
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Hosts's actual total RAM: (1023.66 GB) 1099147345920 and RAM after applying overprovisioning: (1023.66 GB) 1099147378688
2025-09-29 17:43:30,808 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release cpu from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: 6500, reserved: 0, actual total: 301728, total with overprovisioning: 301728; new used: 6000,reserved:0; movedfromreserved: false,moveToReservered: false
2025-09-29 17:43:30,810 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) release mem from host: Host {"id":16,"name":"esxi03.pvcl.afracloud.local","type":"Routing","uuid":"766179c0-1126-4ff1-9aa9-1542d965cacd"}, old used: (4.50 GB) 4831838208, reserved: (0 bytes) 0, total: (1023.66 GB) 1099147378688; new used: (4.00 GB) 4294967296, reserved: (0 bytes) 0; movedfromreserved: false, moveToReservered: false
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Invocation exception, caused by: com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849, ctx-783e2a35]) (logid:e73a213c) Rethrow exception com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
2025-09-29 17:43:30,820 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Done with run of VM work job: com.cloud.vm.VmWorkStart for VM 5387, job origin: 53846
2025-09-29 17:43:30,820 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Unable to complete AsyncJob {"accountId":9,"cmd":"com.cloud.vm.VmWorkStart","cmdInfo":"rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAJAAAAAAAAAAwAAAAAAAAVC3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AA5SZXN0YXJ0TmV0d29ya3QAP3JPMEFCWE55QUJGcVlYWmhMbXhoYm1jdVFtOXZiR1ZoYnMwZ2NvRFZuUHJ1QWdBQldnQUZkbUZzZFdWNGNBRXhw","cmdVersion":0,"completeMsid":null,"created":"Mon Sep 29 17:40:32 IRST 2025","id":53849,"initMsid":345050399512,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":12,"uuid":"cd1036aa-8b98-4906-b35f-f6b3503f4ace"}, job origin: 53846 com.cloud.exception.AgentUnavailableException: Resource [Host:16] is unreachable: Host 16: Unable to start instance due to Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1637)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5827)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5951)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:689)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:637)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.utils.exception.ExecutionException: Unable to start VM:c5fc54ed-81f9-4743-8bdb-48dc6379f9f8 due to error in finalizeStart, not retrying
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1601)
... 20 more
2025-09-29 17:43:30,827 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Complete async job-53849, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC1jb20uY2xvdWQuZXhjZXB0aW9uLkFnZW50VW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAFwIAAHhyADBjb20uY2xvdWQuZXhjZXB0aW9uLlJlc291cmNlVW5hdmFpbGFibGVFeGNlcHRpb25WTU9wAAAAHwIAAkoAA19pZEwABl9zY29wZXQAEUxqYXZhL2xhbmcvQ2xhc3M7eHIAImNvbS5jbG91ZC5leGNlcHRpb24uQ2xvdWRFeGNlcHRpb2556I4b8eRExwIAAkwAC2NzRXJyb3JDb2RldAATTGphdmEvbGFuZy9JbnRlZ2VyO0wABmlkTGlzdHQAFUxqYXZhL3V0aWwvQXJyYXlMaXN0O3hyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3IALGNvbS5jbG91ZC51dGlscy5leGNlcHRpb24uRXhlY3V0aW9uRXhjZXB0aW9uVk1PcAAAAA4CAAFJAAtjc0Vycm9yQ29kZXhxAH4ABnEAfgAOdABkVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAFXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIACEIABmZvcm1hdEkACmxpbmVOdW1iZXJMAA9jbGFzc0xvYWRlck5hbWVxAH4ACUwADmRlY2xhcmluZ0NsYXNzcQB-AAlMAAhmaWxlTmFtZXEAfgAJTAAKbWV0aG9kTmFtZXEAfgAJTAAKbW9kdWxlTmFtZXEAfgAJTAANbW9kdWxlVmVyc2lvbnEAfgAJeHABAAAGQXQAA2FwcHQAJmNvbS5jbG91ZC52bS5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsdAAeVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbC5qYXZhdAAQb3JjaGVzdHJhdGVTdGFydHBwc3EAfgASAQAAFsNxAH4AFHEAfgAVcQB-ABZxAH4AF3Bwc3EAfgASAv____5wdAAtamRrLmludGVybmFsLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2UwdAAJamF2YS5iYXNldAAHMTcuMC4xNnNxAH4AEgIAAABNcHEAfgAacQB-ABt0AAZpbnZva2VxAH4AHXEAfgAec3EAfgASAgAAACtwdAAxamRrLmludGVybmFsLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAI5cHQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHQAImNvbS5jbG91ZC52bS5WbVdvcmtKb2JIYW5kbGVyUHJveHl0ABpWbVdvcmtKb2JIYW5kbGVyUHJveHkuamF2YXQAD2hhbmRsZVZtV29ya0pvYnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9icHBzcQB-ABIBAAACsXEAfgAUdAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0cHBzcQB-ABIBAAAAMXEAfgAUdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5wcHNxAH4AEgEAAAA4cQB-ABR0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxwcHNxAH4AEgEAAABncQB-ABR0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ADp0AA9jYWxsV2l0aENvbnRleHRwcHNxAH4AEgEAAAA1cQB-ABRxAH4APXEAfgA6dAAOcnVuV2l0aENvbnRleHRwcHNxAH4AEgEAAAAucQB-ABR0ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4ANnEAfgA3cHBzcQB-ABIBAAACfXEAfgAUcQB-ADFxAH4AMnEAfgA3cHBzcQB-ABICAAACG3B0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AO3EAfgAdcQB-AB5zcQB-ABICAAABCHB0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ADdxAH4AHXEAfgAec3EAfgASAgAABHBwdAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJxAH4AHXEAfgAec3EAfgASAgAAAntwdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBMcQB-ADdxAH4AHXEAfgAec3EAfgASAgAAA0hwdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ADdxAH4AHXEAfgAec3IAH2phdmEudXRpbC5Db2xsZWN0aW9ucyRFbXB0eUxpc3R6uBe0PKee3gIAAHhweAAAAAB0ALBSZXNvdXJjZSBbSG9zdDoxNl0gaXMgdW5yZWFjaGFibGU6IEhvc3QgMTY6IFVuYWJsZSB0byBzdGFydCBpbnN0YW5jZSBkdWUgdG8gVW5hYmxlIHRvIHN0YXJ0ICBWTTpjNWZjNTRlZC04MWY5LTQ3NDMtOGJkYi00OGRjNjM3OWY5ZjggZHVlIHRvIGVycm9yIGluIGZpbmFsaXplU3RhcnQsIG5vdCByZXRyeWluZ3VxAH4AEAAAABVzcQB-ABIBAAAGZXEAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIBAAAWw3EAfgAUcQB-ABVxAH4AFnEAfgAXcHBzcQB-ABIC_____nBxAH4AGnEAfgAbcQB-ABxxAH4AHXEAfgAec3EAfgASAgAAAE1wcQB-ABpxAH4AG3EAfgAgcQB-AB1xAH4AHnNxAH4AEgIAAAArcHEAfgAicQB-ACNxAH4AIHEAfgAdcQB-AB5zcQB-ABICAAACOXBxAH4AJXEAfgAmcQB-ACBxAH4AHXEAfgAec3EAfgASAQAAAGZxAH4AFHEAfgAocQB-AClxAH4AKnBwc3EAfgASAQAAFz9xAH4AFHEAfgAVcQB-ABZxAH4AKnBwc3EAfgASAQAAAGNxAH4AFHEAfgAtcQB-AC5xAH4AL3Bwc3EAfgASAQAAArFxAH4AFHEAfgAxcQB-ADJxAH4AM3Bwc3EAfgASAQAAADFxAH4AFHEAfgA1cQB-ADZxAH4AN3Bwc3EAfgASAQAAADhxAH4AFHEAfgA5cQB-ADpxAH4AO3Bwc3EAfgASAQAAAGdxAH4AFHEAfgA9cQB-ADpxAH4APnBwc3EAfgASAQAAADVxAH4AFHEAfgA9cQB-ADpxAH4AQHBwc3EAfgASAQAAAC5xAH4AFHEAfgBCcQB-ADZxAH4AN3Bwc3EAfgASAQAAAn1xAH4AFHEAfgAxcQB-ADJxAH4AN3Bwc3EAfgASAgAAAhtwcQB-AEVxAH4ARnEAfgA7cQB-AB1xAH4AHnNxAH4AEgIAAAEIcHEAfgBIcQB-AElxAH4AN3EAfgAdcQB-AB5zcQB-ABICAAAEcHBxAH4AS3EAfgBMcQB-AE1xAH4AHXEAfgAec3EAfgASAgAAAntwcQB-AE9xAH4ATHEAfgA3cQB-AB1xAH4AHnNxAH4AEgIAAANIcHEAfgBRcQB-AFJxAH4AN3EAfgAdcQB-AB5xAH4AVHhzcgARamF2YS5sYW5nLkludGVnZXIS4qCk94GHOAIAAUkABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAQvXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeAAAAAAAAAAQdnIAE2NvbS5jbG91ZC5ob3N0Lkhvc3S4fD08IdIeQAIAAHhw
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Publish async job-53849 complete on message bus
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Wake up jobs related to job-53849
2025-09-29 17:43:30,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-22:[ctx-95638da3, job-53846/job-53849]) (logid:e73a213c) Update db status for job-53849
weizhouapache commentedon Sep 29, 2025
I suspect if the systemvm template is Ready
please also check if the global setting
router.template.vmwareis set correctlyHossein4185 commentedon Sep 30, 2025
@weizhouapache
The global settings are as follows:
I can use the shared network as shown in the image below, but I am having problems with Isoleted Network.