Repository navigation
Conversation
Docker Hub has a `POST /v2/<dst>/_docker/copy/image` endpoint (basically a blob mount, but for a manifest) that copies a manifest, all of its child manifests, and all of their blobs server side. Try it before copying client side, both for the top level of `CopyManifest` and for each child manifest in `EnsureManifest`, and fall back to the existing client side copy on any failure. A 12 platform index with 15 layers each went from 243 requests (~12.4s) to 27 requests (~2s) against a local instance of Hub's internal registry implementation with 40ms of added latency, which also matters for the 300 requests per minute budget Hub gets in `rate-limits.go`. The endpoint is feature flagged by namespace, so remember per host when it does not exist (a non-JSON 404 or a 405) and per source/target namespace pair when it is not enabled (a 400 `MANIFEST_BLOB_UNKNOWN`), to avoid wasting a request on every manifest. Check whether the target already has the manifest first (the endpoint redoes the whole push instead of no-oping), and send the request through the same client transport (authentication, rate limiting, User-Agent) as everything else. Note that `CopyManifest` on the server side path only knows the digest of what it copied, so `MediaType` and `Size` on the returned descriptor are only filled in when a lookup was needed anyhow. Add tests that run `EnsureManifest` and `CopyManifest` against a fake Docker Hub (an in-memory registry plus the copy endpoint and each of its failure modes). Assisted-By: "claude my eyes right out"
Member
Author
|
Gonna incorporate this into my personal deployment and see how it goes ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docker Hub has a
POST /v2/<dst>/_docker/copy/imageendpoint (basically a blob mount, but for a manifest) that copies a manifest, all of its child manifests, and all of their blobs server side. Try it before copying client side, both for the top level ofCopyManifestand for each child manifest inEnsureManifest, and fall back to the existing client side copy on any failure. A 12 platform index with 15 layers each went from 243 requests (~12.4s) to 27 requests (~2s) against a local instance of Hub's internal registry implementation with 40ms of added latency, which also matters for the 300 requests per minute budget Hub gets inrate-limits.go.The endpoint is feature flagged by namespace, so remember per host when it does not exist (a non-JSON 404 or a 405) and per source/target namespace pair when it is not enabled (a 400
MANIFEST_BLOB_UNKNOWN), to avoid wasting a request on every manifest. Check whether the target already has the manifest first (the endpoint redoes the whole push instead of no-oping), and send the request through the same client transport (authentication, rate limiting, User-Agent) as everything else.Note that
CopyManifeston the server side path only knows the digest of what it copied, soMediaTypeandSizeon the returned descriptor are only filled in when a lookup was needed anyhow.Add tests that run
EnsureManifestandCopyManifestagainst a fake Docker Hub (an in-memory registry plus the copy endpoint and each of its failure modes).Assisted-By: "claude my eyes right out"