Fix ECR BLOB_UPLOAD_INVALID by forcing new session on push retry#792
Fix ECR BLOB_UPLOAD_INVALID by forcing new session on push retry#792am-saksham wants to merge 1 commit into
Conversation
|
@am-saksham thanks a lot for the quick PR. I was thinking we should revisit the whole Distribution Spec implementation. We might want to implement resume upload with PATCH as well. Let's see what everyone thinks about that. |
|
Thanks @sadysnaat! I completely agree that moving to the full Distribution Spec with chunked I put this Let me know if the team decides to merge this as a temporary bridge, or if you'd prefer to hold off for a complete |
Incorporate the fresh-session approach from open apple#792 while limiting it to blob uploads, honoring disabled retry policy, and retrying only transport failures, server failures, or ECR BLOB_UPLOAD_INVALID responses. Add local registry tests that prove a 416 retry uses a second upload UUID and that a client without retry options performs only one POST and PUT. Upstream-Ref: apple#792 Bug-Ref: apple#790 Bug-Ref: apple/container#1895 Release-Highlight: Incorporates apple#792 for apple#790 and apple/container#1895: image pushes now recover from ECR BLOB_UPLOAD_INVALID by restarting the blob in a fresh upload session without inventing retries for retry-disabled clients.
Record the tested apple/container#1757 launchd identity fix and the narrowed apple/containerization#792 implementation for apple/containerization#790 and apple/container#1895. Include exact source and shared-test-support commits so either change can be reviewed for Apple without pushing it there.
Add the 5xx restart and unrelated 416 regression coverage to the current apple/containerization#792 handoff and track the standalone test commit.
Fixes #790
Resolves apple/container#1895
Motivation
When pushing large, multi-architecture images to AWS ECR, network interruptions cause the push to fail with a
416 Range Not SatisfiableandBLOB_UPLOAD_INVALIDerror.This occurs because the generic retry loop in
RegistryClient.swiftcatches the failure, resets the data stream to byte 0, and blindly retries the exact same upload session URL. Because ECR expects the retry to resume from the last committed byte, it rejects the byte 0 payload.Modifications
This PR decouples blob upload retries from the generic
RegistryClientretry loop to allow for proper session regeneration.RegistryClient.swift: Added aretryOptionsOverrideparameter to the genericrequest()signature. This allows specific operations to opt out of the default infinite retry loop while maintaining the client's standard behavior for auth and manifests.RegistryClient+Push.swift: Completely disabled the generic retry for blobPOSTandPUTrequests by passingRetryOptions(maxRetries: 0, retryInterval: 0).RegistryClient+Push.swift: Wrapped the blob upload sequence in a dedicated, smart retry loop. If aPUTdrops mid-upload, the loop now catches the error locally, requests a fresh session UUID via a newPOST, and starts the byte 0 upload on that clean session.Result
Interrupted pushes to AWS ECR now successfully recover by abandoning the stale session and initiating a fresh upload, eliminating the 416 error.