Repository navigation
Autoscale instance creation issue #9503
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
AutoScale Instance Groups
CLOUDSTACK VERSION
CloudStack 4.19.1.0
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
While creating autoscale in the parameter field when we mention the maximum number of instance as 3 the instance is created over the mentioned value. The created instance are in the state of stopped and some are in the error state. Nearly 1000 instance are created.
This issue occurs when the created instance are in stopped or in error state.
STEPS TO REPRODUCE
Create the autosccale policy with the following :
Public port
20
Private port
20
Min members
2
Max members
3
Available Instances
1
Polling interval (in sec)
60
Expunge Instance grace period (in sec)
60
Scaleup Policy:
Counter Operator
VM CPU - average percentage
Threshold
Greater than 40
ScaleDown policy:
Counter Operator
VM CPU - average percentage
Threshold
Lesser than 40
EXPECTED RESULTS
The instance that is created should be equal to the given maximum value. Even though the instance is not in the running state.
ACTUAL RESULTS
The instance is keep on creating beyond the given maximum value of instance should be created.
The disable option is also not working for the created autoscale policy.
Activity
boring-cyborg commentedon Aug 8, 2024
Thanks for opening your first issue here! Be sure to follow the issue template!
[-]Autoscale instance creation [/-][+]Autoscale instance creation issue[/+]btzq commentedon Aug 8, 2024
Hey @hari1822 , my team faced this issue before resulting in 30K Autoscale VMs created.
Our findings was that this happens when the VM is unable to be created sucessfully from the VM Template.
But, Cloudstack tries really hard to start up a VM, so it keeps retrying on loop, forever.
I created a ticket here reporting the issue, suggesting better handling to avoid infinite loops:
#9318
Anyways, in our case, what was causing the issue was either:
We were using Linstor as the SDS Storage, but Linbit managed to resolve the issue for us and we were able to create Autoscale VMs without issues ever since.
What stroage are you using?
hari1822 commentedon Aug 9, 2024
When this problem arises, we are unable to delete or disable the auto scale group, and the scaling occurs within the given interval.
btzq commentedon Aug 9, 2024
@hari1822 , okay thats new for us.
When we encountered this issue, we were able to disable the Autoscale Group.
We then did either 1 of the 2 options:
But note, in Option 1, we encountered the UI crashing a few times and DB going 100%.
In this Option 2, we felt it was okay because the VM itself was not created yet. Just a record of its attempt.
I have 2 Questions
hari1822 commentedon Aug 9, 2024
NFS- is used for storage.
While trying to delete the Autoscale Group : Failed to remove the load balancer rule.
If we try to delete the load balancer rule : Unable to remove the loadbalancer rule.
btzq commentedon Aug 10, 2024
@hari1822 i see
im not any good at reading logs, but it looks like your Virtual Router is stopped? If so i think thats a bigger issue you should look into first.
Do you have other VPCs and Virtual Routers working okay?
And for Autoscale, remember you need to disable the autoscale group first before being able to delete it.
If you try to delete an Autoscale Group that is still enabled, it will throw and error.
Only when an Autoscale Group is disabled can you delete it, or make changes to load balancer etc
hari1822 commentedon Aug 12, 2024
@btzq Will look into it
DaanHoogland commentedon Jan 22, 2025
@hari1822 , any updates?