createAccount fails with "unable to verify user credentials and/or request signature" #11394

problem

I don't know what other category to sort this under, so I selected "Bug". I just need some help figuring out where the issue lies. Our Development team runs integration testing of the CRM with CloudStack. For this, they will do the following using a domain admin account:

• listDomainChildren
• listDomains
• createDomain
• createAccount
• listUsers
• getuserKeys
• registerUserKeys
• listTemplates
• listNetworks
• deployVirtualMachine
• queryAsyncJobResult

Now the issue we encounter is that around once every 2-3 weeks the ONLY call that fails in this workflow is createAccount. It errors with "unable to verify user credentials and/or request signature" and in the management-server.log I can see User signature [xxxx] is not equaled to computed signature [yyyy]. However, now the interesting part is that when we generate a new API key and secret, and use these, the createAccount call will succeed. The code to make all the API calls and generate the signature remain unchanged. I just need some help to rack my brain and to help determine where/how it goes wrong, as the API keys are still shown to be the same for this user.

Notes from our Dev team:

1. Every API request has the format Base URL+API Path+Command String+Signature.
2. Make sure all spaces are encoded as "%20" rather than "+".
3. For each field-value pair (as separated by a '&') in the Command String, URL encode each value
4. Sort it alphabetically via the field for each field-value pair.
5. Take the sorted Command String and run it through the HMAC SHA-1 hashing algorithm (most programming languages offer a utility method to do this) with the user's Secret Key. Base64 encode the resulting byte array in UTF-8 so that it can be safely transmitted via HTTP.

versions

4.19.3 but the issue has been persistent on older versions

The steps to reproduce the bug

Obscure, unclear how or when to trigger it. It occurs at random about every 3 weeks.

What to do about it?

Help me figure out how to resolve this.