Repository navigation
Failures while assigning public IP via enableStaticNat #10313
Description
problem
enableStaticNat starts failing just after several uses.
Actually I see that IP is assigned but the call still fails with timeout just after 3-4 uses
For my test I created a number of VMs, and then tried to assign public IP to all of them sequentially:
for i in `cloudstack listVirtualMachines | jq ".virtualmachine[].id"`
> do
> ip=`cloudstack associateIpAddress networkId=_my-network-id_ | jq ".ipaddress.id"`
> time cloudstack enableStaticNat ipaddressid=${ip} virtualmachineid=${i}
> done
{
"success": true
}
real 0m9.272s
user 0m0.259s
sys 0m0.052s
{
"success": true
}
real 0m10.097s
user 0m0.274s
sys 0m0.033s
{
"success": true
}
real 0m10.301s
user 0m0.260s
sys 0m0.040s
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 534, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 448, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 337, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=10)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/cloudstack", line 11, in <module>
load_entry_point('cs==2.7.1', 'console_scripts', 'cs')()
File "/usr/lib/python3/dist-packages/cs/__init__.py", line 104, in main
response = getattr(cs, command)(fetch_result=fetch_result,
File "/usr/lib/python3/dist-packages/cs/client.py", line 213, in handler
return self._request(command, **kwargs)
File "/usr/lib/python3/dist-packages/cs/client.py", line 273, in _request
response = session.send(prepped,
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=10)
real 0m10.357s
user 0m0.281s
sys 0m0.068s
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 446, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 441, in _make_request
httplib_response = conn.getresponse()
...
I tried different configurations of network and VR, and got it always failing, in the best case after 6-7 successful assignments.
large VR with 4 CPUS and several GB memory did not help neither.
time for all failing ones shows real 0m10.350s or slightly more.
Started from #10184
versions
CloudStack 4.20.0.0 with https://github.com/apache/cloudstack/pull/10254/files applied (PR does not help with this)
Ubuntu 22.04.5 LTS
libvirt 8.0.0-1ubuntu7.10
isolated network over VLAN
about 1000 public IPs in /20
The steps to reproduce the bug
- Create number of VMs (I used 100, but looks like just several should be enough
Repeat 2-3 for VMs in 1. till it starts failing (just after about 3-4 cycles in my case) - Use
associateIpAddressto get public IP ID - Use
enableStaticNatwith VM ID and IP ID
InitiallyenableStaticNattakes 9 seconds then increases to 10, and then just starts failing with timeout
Here is a cycle doing the above
for i in `cloudstack listVirtualMachines | jq ".virtualmachine[].id"`; do
ip=`cloudstack associateIpAddress networkId=${YOUR-NETWORK-ID-HERE} | jq ".ipaddress.id"`
time cloudstack enableStaticNat ipaddressid=${ip} virtualmachineid=${i}
done
What to do about it?
Looks like the call is taking too much time to return. Should be optimized.
Activity
[-]Failures while assiging public IP via enableStaticNat[/-][+]Failures while assigning public IP via enableStaticNat[/+]akrasnov-drv commentedon Feb 3, 2025
Just a small update.
After a day I tried to assign another public IP and got the same error. So the problem is not because of some temporary overload, it persists.
Another maybe interesting thing, when I tried to add public IP to VM that already had public IP, I immediately got corresponding error.
431 - is already associated with another public ipI mean api response was very fast:
akrasnov-drv commentedon Feb 3, 2025
Release IP was also very slow, though I did it from UI, and it did not fail. From the allocated list IPs disappeared fast enough, but notifications about IP release started appearing in UI just after maybe some 40 seconds. Release of all 16 IPs I had took about 2 minutes.
What's interesting, after releasing all IPs I was able to assign IPs again, but even with delays between executions it started failing after 3 assigned IP, each time
enableStaticNattook longer:then failed with
akrasnov-drv commentedon Feb 3, 2025
Maybe anybody can suggest java parameters to improve this?
akrasnov-drv commentedon Feb 3, 2025
I found this warning in management log, seems it's relevant
The part of log corresponding to
enableStaticNat(execution-failure) is attachedmgmt.log.gz
DaanHoogland commentedon Feb 4, 2025
@akrasnov-drv , do you have webhooks enabled? it seems like the issue is not in enableStaticNat itself.
weizhouapache commentedon Feb 4, 2025
@akrasnov-drv
have you increased the setting
workersand restart mgmt server ?shwstppr commentedon Feb 4, 2025
@DaanHoogland I don't think webhooks is the cause here. The logs shared are for the API call that finished in around 10s. I see multiple agent-server Command-Answer communication taking few seconds. Also, multiple errors like,
Maybe the API needs optimization or there is underlying network issue
@akrasnov-drv do you also have logs where API calls started failing with timeout?
weizhouapache commentedon Feb 4, 2025
it looks @akrasnov-drv uses redundant VRs, the error happens in the BACKUP VR which do not have default route as the public nic is DOWN.
akrasnov-drv commentedon Feb 4, 2025
First of all thanks for the attention and care.
@DaanHoogland I tried using webhooks in the past, but when started getting different issues I recreated the cluster without webhooks. Here is my agent config
Nevertheless (I believe I reported it before) there is
/etc/libvirt/hooks/qemubrought by cloudstack-agent packageand I had some doubts about it. Though I do not see how it can be related to the current nat issue. I created 100 VMs via api without a problem, only this call fails with timeout.
@weizhouapache I have workers in global config set to 50, but as you see above, agent has it set to 5, and it's not something I set. I can increase that, no problem, but I really doubt number of workers should be relevant to a failure in single api call.
As to VR HA - yes, currently I have a pair, but I believe it failed also with a single one. I'll change setup to make sure and provide info on that.
@shwstppr I'll clean the env and start it again to provide a wider log, covering both successful executions and then failure.
In the meantime, has anybody tried my flow? Did it work (and then the failure is just in my env)?
DaanHoogland commentedon Feb 5, 2025
@akrasnov-drv enabling hundred(s) static nat adresses in not a usual case. But it should work. We'll have to investigate what might go wrong. Do you have a clean environment to experiment in?
If so, let's try to create it in a none redundant env to see how fast it occurs there.
Next step can be with redundant VR to see if this is the issue.
I doubt either is the culprit but we'll have to start simple.
akrasnov-drv commentedon Feb 5, 2025
I removed all VMs and network and recreated with standard isolated network offering with static nat with single VR.
Each call execution time was increasing with the last one almost at timeout:
It managed to assign about 40 IPs on the way before failing.
So, there are 2 outcomes from my tests
Attaching logs from management and from VR for the time of above.
cloud.log.gz
management-server.log.gz
akrasnov-drv commentedon Feb 5, 2025
@DaanHoogland it was not our intention. We are just trying to use CloudStack fleet in Jenkins. The only plugin supporting CloudStack is jcloud plugin, and it requires public IP and static nat.
I opened Jira ticket for this https://issues.jenkins.io/browse/JENKINS-74928 but have not got single reply for months.
So we are trying to use what we have. And in the end we'll need hundreds of nodes, maybe even about thousand.
Though, should not be a problem from iptables side...
Besides, I think to try changing jenkins plugin, but that's another story.
DaanHoogland commentedon Feb 6, 2025
Noted @akrasnov-drv , this is added to the backlog and has to be investigated. I am afraid I don't have a workaround off the top of my head.