[base] Reduce likeliness of being throttled when uploading blobs#195
[base] Reduce likeliness of being throttled when uploading blobs#195ao-picterra wants to merge 1 commit into
Conversation
| if not resp.ok: | ||
| logger.error("Error when uploading to blobstore %s" % upload_url) | ||
| raise APIError(resp.text) | ||
| time.sleep(2) |
There was a problem hiding this comment.
Something feels weird about this fix. This adds an unconditional sleep to the upload unless I'm missing something, which to me is a sign we should fix our throttling instead of the code here.
Like adding a sleep in a retry is fine, but we shouldn't need sleep in the happy path - this just means our throttling limit is too low.
(Maybe I'm misunderstanding the fix / this isn't ready yet)
There was a problem hiding this comment.
Ah sorry I didn't notice I assigned to you (I guess GH did it on his own), indeed the issue is on the server side and I'm fixing it there
There was a problem hiding this comment.
Ah no all good, you didn't assign it but I still get notif (I'm repo owner or something I guess). So all good if its just wip/debug stuff :)
2 participants
We don't retry on POST and PUTs, so uploading file to blobstore are subject to incur in "burst" throttling, this tries to reduce the likeliness.