Repository navigation
KVMHAMonitor getting initialized without host ha enabled #8682
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
HA
CLOUDSTACK VERSION
4.19.0
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
Even when HA has been disabled on cluster and/or host level, KVMHAMonitor getting initialized on KVM hosts.
It looks like code does not check if HA is enabled or not:
https://github.com/apache/cloudstack/blob/8f6721ed4c4e1b31081a951c62ffbe5331cf16d4/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
STEPS TO REPRODUCE
1) disable HA on cluster level via UI
2) login to kvm host
3) take a look at agent.log
EXPECTED RESULTS
When HA is disabled I would expect that the management server adds the following parameter to agent.properties file of each host:
reboot.host.and.alert.management.on.heartbeat.timeout=false
or
KVMHAMonitor will not getting initialized at all.
ACTUAL RESULTS
Nothing happens when disabling HA on cluster and/or host level. KVMHAMonitor is getting initialized and will perform checks of the host. In some situations this will lead to an automatic reboot of the host because of KVMHAMonitor is not able to write heartbeat for pool to primary storage:
Feb 19 12:01:08 kvm-2 java[9465]: WARN [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; try: 5 of 5.
Feb 19 12:01:08 kvm-2 java[9465]: WARN [kvm.resource.KVMHAMonitor] (Thread-1:) (logid:) Write heartbeat for pool [71c272d3-b180-4b18-a0fc-cfc1dc5b86c9] failed: Down; stopping cloudstack-agent.
Feb 19 12:02:08 kvm-2 heartbeat: kvmspheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.
Activity
[-]KVMHAMonitor getting initalized without host ha enabled[/-][+]KVMHAMonitor getting initialized without host ha enabled[/+]DaanHoogland commentedon Mar 4, 2024
@sbrueseke are you addressing (designing/implementing a solution to) this bug?
sbrueseke commentedon Mar 4, 2024
I'm not deep enough into the topic to suggest a solution. We noticed the bug because we were carrying out some tests with Linstor in our lab and suddenly all the KVM hosts restarted. By evaluating the logs, we saw that it was due to the KVMHAMonitor. However, we did not have HA enabled, not on the cluster and not on the hosts.
The question is why the KVMHAMonitor is running on the hosts even if HA is not configured. Maybe there is a reason, but I don't know it. But what, in my opinion, needs to be prevented is that KVMHAMonitor restarts hosts despite HA being deactivated.
One solutions for this would be to set reboot.host.and.alert.management.on.heartbeat.timeout=false in the agent.properties by the management server if HA is not configured or deactivated.
But as I sad before, I'm not deep into that topic and I do not know if this is a valid solution or if there is a much better one.
adidiborg commentedon Mar 12, 2024
@sbrueseke, We are also facing similar issue, but as per our logs ( on all KVM ) we are only seeing below log lines -
and not seeing this one which you have mentioned
dseralathan commentedon Mar 12, 2024
heartbeat[13962]: kvmheartbeat.sh will reboot system because it was unable to write the heartbeat to the storage.
I received the same error when my Primary NFS Storage was in accessible .
DaanHoogland commentedon Apr 5, 2024
Hey @sbrueseke , I finally got a chance to look at this again.If you are using NFS this is how it is supposed to work , I think. If you are using LinStor, see the merged PR for 4.19 (#8670). Is this still an issue?
sbrueseke commentedon Apr 8, 2024
@DaanHoogland now I am confused! Why is KVMHAMonitor rebooting hosts when the host is unable to write a heartbeat file to any storage when any HA setting is disabled in the UI? Do you know the reason for that?
weizhouapache commentedon Apr 8, 2024
@sbrueseke
You can add a setting in
/etc/cloudstack/agent.propertiesand restart cloudstack-agentsbrueseke commentedon Apr 8, 2024
I know this workaround. My question is if this setting should be added by the management server instead of manually.
Is there any legit reason KVMHAMonitor should reboot a host when all HA settings are disabled in the UI?
weizhouapache commentedon Apr 8, 2024
@sbrueseke
as I understand, the Host HA means the HA solutions by OOBM/IPMI
sbrueseke commentedon Apr 8, 2024
Can you explain the root cause why KVMHAMonitor needs to reboot the host when a storage is read only?
weizhouapache commentedon Apr 8, 2024
@sbrueseke
it is the old behaviour introduced more than 10 years ago. I am not clear why.
there were lots of discussion in the past
#2722 #2890 #2984 #4586 and #4708
since storage issue rarely happens, it may be ok for users to change the setting in agent.properties
sbrueseke commentedon Apr 8, 2024
@weizhouapache I would suggest to take another look into that and change it default. In my opinion a host should not automatically reboot at all if no settings are configured. From what I know this looks to me like old behavior lost in the code.
weizhouapache commentedon Apr 8, 2024
I agree with you. @sbrueseke
cc @DaanHoogland @rohityadavcloud @andrijapanicsb @GutoVeronezi @wido
your opinions ?
DaanHoogland commentedon Apr 8, 2024
@sbrueseke , I beg to differ, when storage can not be reached all VMs will be running on read-only disks from there on in. This means the hosts is useless and VMs need to reboot according to VM-HA. This behaviour was implemented before Host-HA was conceived and is not related. Maybe a redesign is in place but as of now this works as designed.
andrijapanicsb commentedon Apr 8, 2024
afaik, this has 0 things to do with host HA, VM HA or anything else.
This scrip (kvmspheartbeat.sh) has been there for last 12 years (that I'm aware of) - and it only checks if NFS storage is accessible, and if not, reboots (forcefully) the host.
They only way (to my knowledge) to disable host reboots is to comment out the lines that does some echo into sys/proc...etc - at the VERY end of the script (that echo triggers forcefull reboot) - just comment out that single "echo.... line and you are good. Storage migh be unaccessible (heartbeat fail) but nothing will happen (log messages will say "I'm rebooting" but script is not doing anything due to line commented out)
slavkap commentedon Apr 8, 2024
Hi all,
KVMHAMonitor is initialized when the agent host is started (without knowing if HA is enabled/disabled). It is used either when Host HA is enabled to check the host. If there is a primary storage that supports Host HA (even if the HA is disabled) and some issue with the storage the script
kvmheartbeat.sh/kvmspheartbeat.shwill be executed.Probably the fastest and easiest solution before a redesign we should set the agent property reboot.host.and.alert.management.on.heartbeat.timeout to be disabled by default.
GutoVeronezi commentedon Apr 8, 2024
@andrijapanicsb you can also disable this behavior with the agent property
reboot.host.and.alert.management.on.heartbeat.timeout, as Wei stated in #8682 (comment).PR #4586 introduced this property with default as
trueto keep the behavior; however, I also agree with @sbrueseke. With default astrue, we are delegating to ACS the decision of rebooting the host, which might not be correct in several cases. I am inclined to make this propertyfalseby default and focus on alerting the operators about the situation, so they can take the right action over the environment.A redesign of the feature might be needed; however, we need discuss it first.