From 94866709e506b2a96b2413ab3a3554e1cfbe806f Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 17:27:32 +0000 Subject: [PATCH 1/9] Document BuildKit fork updates before CVE remediation Co-authored-by: Codex --- cve-mitigation/SKILL.md | 14 +++ cve-mitigation/references/buildkit.md | 156 ++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) create mode 100644 cve-mitigation/references/buildkit.md diff --git a/cve-mitigation/SKILL.md b/cve-mitigation/SKILL.md index 5a565700a4e5b3..24be7a4c2ea829 100644 --- a/cve-mitigation/SKILL.md +++ b/cve-mitigation/SKILL.md @@ -70,6 +70,19 @@ When a CVE is reported: 4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change exported APIs. These require code migration, not just `go get`. +## BuildKit fork updates + +BuildKit binaries in Bob's base image must be fixed in `gitpod-io/buildkit` +first. Follow the [BuildKit fork update procedure](references/buildkit.md) to +bring `custom-mirror` up to date with upstream while preserving Gitpod's patches, +validate the fork, and manually publish its image to +`eu.gcr.io/gitpod-core-dev/build/buildkit`. Only then update Bob's Dockerfile to +consume the published image and validate it in a preview. + +Do not replace the fork's binaries with ad hoc rebuilds inside Bob's Dockerfile +or substitute upstream BuildKit. If the fork update or publication cannot be +completed, report the remaining findings and defer the BuildKit remediation. + ## Remediation (Go dependencies) ### Standard update flow @@ -205,5 +218,6 @@ done ## Reference files +- [BuildKit fork updates](references/buildkit.md) — Upstream patch replay, manual image publication, and Bob promotion. - `references/ci-scanning.md` — CI pipeline details for vulnerability scanning. - `references/go-modules.md` — List of all Go modules and their scan-relevant groupings. diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md new file mode 100644 index 00000000000000..0854a14e954ddf --- /dev/null +++ b/cve-mitigation/references/buildkit.md @@ -0,0 +1,156 @@ +# Updating the Gitpod BuildKit fork + +Remediate BuildKit in `gitpod-io/buildkit` before changing its consumer in this +repository. The order is: update the fork, validate it, authenticate to Google +Cloud, manually build and push an image to +`eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob's `FROM` reference. + +Gitpod's customizations live on `custom-mirror`, the head branch of the +long-running [fork PR #1](https://github.com/gitpod-io/buildkit/pull/1). Updates +belong on that branch; do not merge PR #1 into the fork's default branch, +`master`. Inspect its current state and comments rather than assuming a tag or +the default branch contains the intended Gitpod changes. + +## Evidence from previous updates + +- The [February 2024 instructions in PR #1](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-1922138187) + describe syncing the fork's default branch, exporting Gitpod's patches, + selecting an upstream release tag, replaying the patches with `git am --3way`, + and updating `custom-mirror`. The tag and patch-base SHA in that comment are + historical examples, not values to reuse. +- The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845) + records syncing the fork, exporting five customization patches, applying them + to upstream `v0.20.1`, and then updating image-builder-bob for preview testing. + It also records that builds then published to GHCR and previously used Core Dev. +- On November 12, 2025, [Gero reported rebuilding this branch and pushing + `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498) + to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4) + resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the + `custom-mirror` head inspected for this documentation. His later + [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's + base image from `gitpod.3` to `gitpod.4` and records image-build and workspace + start/stop/restart checks in a preview. +- A later example has [Gero's gRPC update in the fork](https://github.com/gitpod-io/buildkit/commit/aa03c4117c98110495de9d5d840e1c8cb106a221) + followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1) + moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory. + +These records establish fork build/publication before consumer promotion. The +2025 consumer used GHCR; it does not prove that particular image was pushed to +Core Dev. The required destination for this procedure is now Core Dev. Its +`build/buildkit` path also appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5). +Do not assume the fork's current GitHub Actions publisher targets Core Dev. + +## Bring custom-mirror up to date with upstream + +First inspect both repositories' history, the current Bob image, its source tag, +and PR #1's comments. Verify the upstream base immediately before Gitpod's +customization commits. Stop and report missing evidence if the patch series or +published image's provenance cannot be established. + +Also compare `custom-mirror` with the currently consumed release. For example, +the [gitpod.8 helper rebuild](https://github.com/gitpod-io/buildkit/commit/37c65868a73a80780ab4f2f5f302810471941d6f) +and [Git source validation backport](https://github.com/gitpod-io/buildkit/commit/3198c5119) +were on a separate release branch when this guide was written. Preserve required +released fixes when bringing `custom-mirror` forward; starting from its head +alone can omit them. + +For a dependency-only fix, branch directly from `origin/custom-mirror` and open +a PR targeting `custom-mirror`. When updating the upstream baseline, use the +patch-replay process documented in PR #1. The following is an adapted recipe +that prepares a review branch instead of immediately rewriting the shared branch: + +```bash +git clone https://github.com/gitpod-io/buildkit.git +cd buildkit +git remote add upstream https://github.com/moby/buildkit.git +git fetch origin --tags +git fetch upstream --tags + +# Set these after inspecting history; do not reuse old comment values. +: "${BUILDKIT_UPSTREAM_RELEASE:?Select the upstream release tag}" +: "${BUILDKIT_PATCH_BASE:?Verify the commit before Gitpod customizations}" +: "${BUILDKIT_UPDATE_BRANCH:?Choose a new review branch name}" + +BUILDKIT_PATCH_DIR=$(mktemp -d) +git branch backup/custom-mirror-before-update origin/custom-mirror +git format-patch --output-directory "$BUILDKIT_PATCH_DIR" \ + "${BUILDKIT_PATCH_BASE}..origin/custom-mirror" +git switch -c "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_UPSTREAM_RELEASE" +git am --3way "$BUILDKIT_PATCH_DIR"/*.patch + +# Review conflicts and restore any required release-only fixes identified above. +git diff --check +git push --set-upstream origin "$BUILDKIT_UPDATE_BRANCH" +``` + +Sync the fork's default branch from upstream as described in PR #1 when needed; +that sync is separate from publishing Gitpod's customized branch. Review the +replayed patches and open a PR targeting `custom-mirror`. Keep the fork's mirror +behavior and previously released security fixes. Do not replace this process +with a blanket copy of another branch's module and vendor tree. + +The original comment rewrites `custom-mirror` with a force push. Updating that +shared branch is a maintainer action after review; if a history rewrite is +necessary, use an explicitly authorized `--force-with-lease`, preserving a backup. +Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork. + +## Validate, build, and publish manually + +Run the fork's relevant tests on native Linux amd64 and inspect the resulting +BuildKit and bundled helper binaries. A Go pin does not rebuild downloaded CNI +archives. Check the actual Go/dependency metadata and image scan; source module +scans alone do not establish that the shipped binaries are fixed. + +After the fork change is reviewed and present on `custom-mirror`, use a clean +checkout of that branch. Select a new, unused Gitpod image tag and record its +source commit. Docker Buildx, the Google Cloud CLI, Grype, and permission to push +to Core Dev are prerequisites. The login command is `gcloud auth login`. + +```bash +set -euo pipefail +git switch custom-mirror +git pull --ff-only origin custom-mirror +test -z "$(git status --porcelain)" +git rev-parse HEAD + +: "${BUILDKIT_IMAGE_TAG:?Choose a new unused Gitpod image tag}" +BUILDKIT_IMAGE="eu.gcr.io/gitpod-core-dev/build/buildkit:${BUILDKIT_IMAGE_TAG}" + +gcloud auth login +gcloud auth configure-docker eu.gcr.io --quiet + +docker buildx build \ + --platform linux/amd64 \ + --target buildkit \ + --pull --load \ + --tag "$BUILDKIT_IMAGE" . + +docker run --rm "$BUILDKIT_IMAGE" --version +grype "docker:$BUILDKIT_IMAGE" --fail-on critical + +docker push "$BUILDKIT_IMAGE" +docker buildx imagetools inspect "$BUILDKIT_IMAGE" +``` + +This uses the fork's `buildkit` Dockerfile target and its committed toolchain +pins. Keep the source commit, scan results, image tag, and registry digest in +the change record. Investigate scan failures before pushing; apply existing +approved vulnerability exceptions only with evidence that they still hold. +Do not publish tags or images without authorization. + +## Promote the image into Gitpod + +Only after the image exists in Core Dev, update +`components/image-builder-bob/leeway.Dockerfile` to consume +`eu.gcr.io/gitpod-core-dev/build/buildkit:` (preferably pinned by digest). +Keep this consumer change small. Do not rebuild or replace the fork's BuildKit, +runc, or CNI binaries inside Bob's Dockerfile as a shortcut. + +Build Bob, run the CI vulnerability scan described in +[ci-scanning.md](ci-scanning.md), and validate an image build plus workspace +start/stop/restart in a preview, following PR #21142's example. Include the fork +source, published image/digest, consumer diff, scan run, and preview results in +the PR. If the fork update or manual publication is deferred, keep the current +image reference, report its remaining findings, and defer BuildKit remediation +to a follow-on PR. Do not describe an earlier scan of different binaries as +validation of the current image. From 784d9ca8f1d6ca4bb67ea0b823e19b7bd6548259 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 17:59:18 +0000 Subject: [PATCH 2/9] Clarify BuildKit fork update review and approval workflow Co-authored-by: Codex --- cve-mitigation/SKILL.md | 14 +- cve-mitigation/references/buildkit.md | 181 ++++++++++++++++++-------- 2 files changed, 130 insertions(+), 65 deletions(-) diff --git a/cve-mitigation/SKILL.md b/cve-mitigation/SKILL.md index 24be7a4c2ea829..d9e71adbdbd5b2 100644 --- a/cve-mitigation/SKILL.md +++ b/cve-mitigation/SKILL.md @@ -72,16 +72,9 @@ When a CVE is reported: ## BuildKit fork updates -BuildKit binaries in Bob's base image must be fixed in `gitpod-io/buildkit` -first. Follow the [BuildKit fork update procedure](references/buildkit.md) to -bring `custom-mirror` up to date with upstream while preserving Gitpod's patches, -validate the fork, and manually publish its image to -`eu.gcr.io/gitpod-core-dev/build/buildkit`. Only then update Bob's Dockerfile to -consume the published image and validate it in a preview. - -Do not replace the fork's binaries with ad hoc rebuilds inside Bob's Dockerfile -or substitute upstream BuildKit. If the fork update or publication cannot be -completed, report the remaining findings and defer the BuildKit remediation. +The image-builder-bob component depends on Gitpod's BuildKit fork. Follow the +[fork maintenance procedure](references/buildkit.md) to update it with upstream +while preserving Gitpod's customizations. ## Remediation (Go dependencies) @@ -218,6 +211,5 @@ done ## Reference files -- [BuildKit fork updates](references/buildkit.md) — Upstream patch replay, manual image publication, and Bob promotion. - `references/ci-scanning.md` — CI pipeline details for vulnerability scanning. - `references/go-modules.md` — List of all Go modules and their scan-relevant groupings. diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index 0854a14e954ddf..dec05196945292 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -1,9 +1,10 @@ # Updating the Gitpod BuildKit fork -Remediate BuildKit in `gitpod-io/buildkit` before changing its consumer in this -repository. The order is: update the fork, validate it, authenticate to Google -Cloud, manually build and push an image to -`eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob's `FROM` reference. +Bob depends on `gitpod-io/buildkit`. Complete the three phases below before +publishing an image or changing Bob's `FROM` reference. When publication and +consumer promotion are in scope, build and validate the approved fork commit, +publish it to `eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob. +Do not substitute upstream BuildKit for the Gitpod fork. Gitpod's customizations live on `custom-mirror`, the head branch of the long-running [fork PR #1](https://github.com/gitpod-io/buildkit/pull/1). Updates @@ -11,36 +12,18 @@ belong on that branch; do not merge PR #1 into the fork's default branch, `master`. Inspect its current state and comments rather than assuming a tag or the default branch contains the intended Gitpod changes. -## Evidence from previous updates +## Bring custom-mirror up to date with upstream -- The [February 2024 instructions in PR #1](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-1922138187) - describe syncing the fork's default branch, exporting Gitpod's patches, - selecting an upstream release tag, replaying the patches with `git am --3way`, - and updating `custom-mirror`. The tag and patch-base SHA in that comment are - historical examples, not values to reuse. -- The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845) - records syncing the fork, exporting five customization patches, applying them - to upstream `v0.20.1`, and then updating image-builder-bob for preview testing. - It also records that builds then published to GHCR and previously used Core Dev. -- On November 12, 2025, [Gero reported rebuilding this branch and pushing - `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498) - to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4) - resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the - `custom-mirror` head inspected for this documentation. His later - [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's - base image from `gitpod.3` to `gitpod.4` and records image-build and workspace - start/stop/restart checks in a preview. -- A later example has [Gero's gRPC update in the fork](https://github.com/gitpod-io/buildkit/commit/aa03c4117c98110495de9d5d840e1c8cb106a221) - followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1) - moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory. +### Phase 1: Select the version -These records establish fork build/publication before consumer promotion. The -2025 consumer used GHCR; it does not prove that particular image was pushed to -Core Dev. The required destination for this procedure is now Core Dev. Its -`build/buildkit` path also appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5). -Do not assume the fork's current GitHub Actions publisher targets Core Dev. +Before preparing the update branch, ask the user which BuildKit version to +target unless they have already specified it. Generally recommend the latest +stable BuildKit release from [upstream releases](https://github.com/moby/buildkit/releases), +checking the current release when asking. Wait for their answer, then verify the +selected upstream tag exists. Honor an explicitly requested version and do not +select a version from the historical examples in this guide. -## Bring custom-mirror up to date with upstream +### Phase 2: Prepare and compare locally, then request approval First inspect both repositories' history, the current Bob image, its source tag, and PR #1's comments. Verify the upstream base immediately before Gitpod's @@ -54,12 +37,13 @@ were on a separate release branch when this guide was written. Preserve required released fixes when bringing `custom-mirror` forward; starting from its head alone can omit them. -For a dependency-only fix, branch directly from `origin/custom-mirror` and open -a PR targeting `custom-mirror`. When updating the upstream baseline, use the +For a dependency-only fix, branch locally from `origin/custom-mirror`. +When updating the upstream baseline, use the patch-replay process documented in PR #1. The following is an adapted recipe -that prepares a review branch instead of immediately rewriting the shared branch: +that prepares a local candidate branch: ```bash +set -euo pipefail git clone https://github.com/gitpod-io/buildkit.git cd buildkit git remote add upstream https://github.com/moby/buildkit.git @@ -80,37 +64,97 @@ git am --3way "$BUILDKIT_PATCH_DIR"/*.patch # Review conflicts and restore any required release-only fixes identified above. git diff --check -git push --set-upstream origin "$BUILDKIT_UPDATE_BRANCH" ``` -Sync the fork's default branch from upstream as described in PR #1 when needed; -that sync is separate from publishing Gitpod's customized branch. Review the -replayed patches and open a PR targeting `custom-mirror`. Keep the fork's mirror -behavior and previously released security fixes. Do not replace this process -with a blanket copy of another branch's module and vendor tree. +Keep the fork's mirror behavior and previously released security fixes. Do not +replace this process with a blanket copy of another branch's module and vendor +tree. Run relevant tests and builds on native Linux amd64 before requesting +approval, and inspect the resulting BuildKit and bundled helper binaries. A Go +pin does not rebuild downloaded CNI archives. Report which checks passed and +which remain deferred; source module scans alone do not validate shipped binaries. -The original comment rewrites `custom-mirror` with a force push. Updating that -shared branch is a maintainer action after review; if a history rewrite is -necessary, use an explicitly authorized `--force-with-lease`, preserving a backup. -Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork. +After committing the local candidate, run these commands in the BuildKit clone. +Set `BUILDKIT_UPDATE_BRANCH` to its local branch name. These commands fetch the +remote branch and compare the two complete commit trees without working-tree +changes. A merge-base comparison can hide changes when the upstream baseline is +replaced: -## Validate, build, and publish manually +```bash +set -euo pipefail +: "${BUILDKIT_UPDATE_BRANCH:?Set the local candidate branch name}" +git fetch --no-tags origin \ + +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror +BUILDKIT_REVIEWED_REMOTE=$(git rev-parse --verify 'refs/remotes/origin/custom-mirror^{commit}') +BUILDKIT_REVIEWED_CANDIDATE=$(git rev-parse --verify "refs/heads/${BUILDKIT_UPDATE_BRANCH}^{commit}") +BUILDKIT_REVIEW_DIR=$(mktemp -d /tmp/buildkit-review.XXXXXX) +BUILDKIT_DIFF_FILE="$BUILDKIT_REVIEW_DIR/custom-mirror-to-candidate.diff" + +printf 'branch=%s\nremote=%s\ncandidate=%s\n' \ + "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_REVIEWED_REMOTE" \ + "$BUILDKIT_REVIEWED_CANDIDATE" > "$BUILDKIT_REVIEW_DIR/commits.txt" +git diff --no-ext-diff --no-textconv --no-color --binary --full-index \ + --no-renames --src-prefix=a/ --dst-prefix=b/ \ + "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \ + > "$BUILDKIT_DIFF_FILE" +git diff --no-color --stat --no-renames \ + "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \ + > "$BUILDKIT_REVIEW_DIR/summary.txt" +if test -s "$BUILDKIT_DIFF_FILE"; then + git apply --numstat "$BUILDKIT_DIFF_FILE" > "$BUILDKIT_REVIEW_DIR/numstat.txt" + printf 'Review diff: %s\n' "$BUILDKIT_DIFF_FILE" +else + printf 'No update: remote and candidate trees are identical.\n' +fi +``` -Run the fork's relevant tests on native Linux amd64 and inspect the resulting -BuildKit and bundled helper binaries. A Go pin does not rebuild downloaded CNI -archives. Check the actual Go/dependency metadata and image scan; source module -scans alone do not establish that the shipped binaries are fixed. +Share the resulting diff file as a clickable link, with the recorded commit IDs +and summary. Additions describe what the candidate introduces; removals describe +what it removes from remote `custom-mirror`. Binary changes are included. If the +diff is empty, report that the trees are identical instead of requesting approval +for an update. Keep the artifacts available through human review. + +Also share the local branch name, target upstream tag, patch adaptations, and +validation results or limitations. A diff summary or inline excerpts alone are +insufficient. Ask the human to approve updating or replacing remote +`custom-mirror` with this specific candidate, including any required history +rewrite. Stop and wait for their answer. Do not push branches or tags, sync the +remote default branch, or publish images during this phase. + +### Phase 3: Update remote custom-mirror after approval + +Only explicit human approval of the reviewed candidate authorizes updating or +replacing remote `custom-mirror`. A request to prepare a local candidate or edit +this skill is not approval to push. If the user prohibited pushes, wait for +explicit authorization lifting that restriction for this update. + +Fetch and recheck the remote tip and local candidate against the reviewed commit +IDs before pushing. If either changed, regenerate the comparison and obtain +approval again. Preserve the old remote tip in a backup branch. Use a normal +push for a fast-forward update; if replacement requires rewriting history, use +`--force-with-lease=refs/heads/custom-mirror:` only with +explicit approval for that rewrite. Never use an unconditional force push. +Verify the remote tip matches the approved candidate after the push. Any fork +default-branch sync is a separate remote change requiring its own authorization. + +Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork. + +## Build, validate, and publish the approved image After the fork change is reviewed and present on `custom-mirror`, use a clean -checkout of that branch. Select a new, unused Gitpod image tag and record its -source commit. Docker Buildx, the Google Cloud CLI, Grype, and permission to push -to Core Dev are prerequisites. The login command is `gcloud auth login`. +checkout of the approved commit. Continue only when image publication is in scope; +approval to update `custom-mirror` alone does not authorize publishing images. +Select a new, unused Gitpod image tag and record its source commit. Docker Buildx, +the Google Cloud CLI, Grype, and permission to push to Core Dev are prerequisites. +The login command is `gcloud auth login`. ```bash set -euo pipefail -git switch custom-mirror -git pull --ff-only origin custom-mirror +: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved candidate commit from commits.txt}" test -z "$(git status --porcelain)" +git fetch --no-tags origin \ + +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror +test "$(git rev-parse origin/custom-mirror)" = "$BUILDKIT_REVIEWED_CANDIDATE" +git switch --detach "$BUILDKIT_REVIEWED_CANDIDATE" git rev-parse HEAD : "${BUILDKIT_IMAGE_TAG:?Choose a new unused Gitpod image tag}" @@ -154,3 +198,32 @@ the PR. If the fork update or manual publication is deferred, keep the current image reference, report its remaining findings, and defer BuildKit remediation to a follow-on PR. Do not describe an earlier scan of different binaries as validation of the current image. + +## Evidence from previous updates + +- The [February 2024 instructions in PR #1](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-1922138187) + describe syncing the fork's default branch, exporting Gitpod's patches, + selecting an upstream release tag, replaying the patches with `git am --3way`, + and updating `custom-mirror`. The tag and patch-base SHA in that comment are + historical examples, not values to reuse. +- The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845) + records syncing the fork, exporting five customization patches, applying them + to upstream `v0.20.1`, and then updating image-builder-bob for preview testing. + It also records that builds then published to GHCR and previously used Core Dev. +- On November 12, 2025, [Gero reported rebuilding this branch and pushing + `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498) + to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4) + resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the + `custom-mirror` head inspected for this documentation. His later + [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's + base image from `gitpod.3` to `gitpod.4` and records image-build and workspace + start/stop/restart checks in a preview. +- A later example has [Gero's gRPC update in the fork](https://github.com/gitpod-io/buildkit/commit/aa03c4117c98110495de9d5d840e1c8cb106a221) + followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1) + moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory. + +These records establish fork build/publication before consumer promotion. The +2025 consumer used GHCR; it does not prove that particular image was pushed to +Core Dev. The required destination for this procedure is now Core Dev. Its +`build/buildkit` path also appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5). +Do not assume the fork's current GitHub Actions publisher targets Core Dev. From e153d85ffc6395301def7d999bd6d1d3b783cc20 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 18:02:55 +0000 Subject: [PATCH 3/9] Clarify when to prompt for the latest BuildKit release Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index dec05196945292..25fb9b7c493c8e 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -16,12 +16,12 @@ the default branch contains the intended Gitpod changes. ### Phase 1: Select the version -Before preparing the update branch, ask the user which BuildKit version to -target unless they have already specified it. Generally recommend the latest -stable BuildKit release from [upstream releases](https://github.com/moby/buildkit/releases), -checking the current release when asking. Wait for their answer, then verify the -selected upstream tag exists. Honor an explicitly requested version and do not -select a version from the historical examples in this guide. +If the user specifies an explicit BuildKit version, use that version without +prompting them to use the latest release. Otherwise, check the latest stable +release in [moby/buildkit releases](https://github.com/moby/buildkit/releases) +and ask whether they want to update to that specific version. Wait for their +answer before preparing the update branch. Verify the selected upstream tag +exists; do not select a version from the historical examples in this guide. ### Phase 2: Prepare and compare locally, then request approval From 7171b01467907cb156c930fbcc47deebe099b609 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 18:04:03 +0000 Subject: [PATCH 4/9] Simplify BuildKit candidate preparation guidance Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index 25fb9b7c493c8e..e0c854a90ce069 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -37,10 +37,9 @@ were on a separate release branch when this guide was written. Preserve required released fixes when bringing `custom-mirror` forward; starting from its head alone can omit them. -For a dependency-only fix, branch locally from `origin/custom-mirror`. -When updating the upstream baseline, use the -patch-replay process documented in PR #1. The following is an adapted recipe -that prepares a local candidate branch: +Prepare a local candidate from the selected upstream release and replay Gitpod's +patches using the process documented in PR #1. The following recipe keeps the +remote branches unchanged: ```bash set -euo pipefail From a37a353f527575265731480b2453c6d7d0bd0021 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 18:05:26 +0000 Subject: [PATCH 5/9] Clarify BuildKit CVE remediation within fork updates Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index e0c854a90ce069..d3d697c832dafc 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -1,6 +1,7 @@ # Updating the Gitpod BuildKit fork -Bob depends on `gitpod-io/buildkit`. Complete the three phases below before +Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on. +Complete the three phases below before publishing an image or changing Bob's `FROM` reference. When publication and consumer promotion are in scope, build and validate the approved fork commit, publish it to `eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob. @@ -37,9 +38,12 @@ were on a separate release branch when this guide was written. Preserve required released fixes when bringing `custom-mirror` forward; starting from its head alone can omit them. -Prepare a local candidate from the selected upstream release and replay Gitpod's -patches using the process documented in PR #1. The following recipe keeps the -remote branches unchanged: +Prepare a local candidate from the selected upstream release, adopting its +dependency versions as the baseline. Replay Gitpod's customization patches using +the process documented in PR #1. Review historical dependency and toolchain +patches so older pins do not undo upstream security fixes. Preserve the effect +of required release-only fixes; if upstream already contains a fix, it does not +need to be replayed. The following recipe keeps the remote branches unchanged: ```bash set -euo pipefail @@ -65,9 +69,16 @@ git am --3way "$BUILDKIT_PATCH_DIR"/*.patch git diff --check ``` -Keep the fork's mirror behavior and previously released security fixes. Do not -replace this process with a blanket copy of another branch's module and vendor -tree. Run relevant tests and builds on native Linux amd64 before requesting +Scan the candidate for remaining CVEs. If the selected upstream release still +has vulnerable dependencies, update them in this same local candidate before +requesting approval. Regenerate affected module and vendor files and update +toolchain or bundled helper builds as needed. Both the upstream update and any +additional dependency fixes must retain Gitpod's customization patches and +mirror behavior. Record the extra updates and any unresolved findings for review. +Do not replace patch replay with a blanket copy of another branch's module and +vendor tree. + +Run relevant tests and builds on native Linux amd64 before requesting approval, and inspect the resulting BuildKit and bundled helper binaries. A Go pin does not rebuild downloaded CNI archives. Report which checks passed and which remain deferred; source module scans alone do not validate shipped binaries. From 0364e724e44e57e654588522d1cec286bf970546 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 18:20:26 +0000 Subject: [PATCH 6/9] Improve local validation instructions for BuildKit fork updates Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 51 +++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index d3d697c832dafc..4a8a4810b6f33b 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -1,8 +1,8 @@ # Updating the Gitpod BuildKit fork Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on. -Complete the three phases below before -publishing an image or changing Bob's `FROM` reference. When publication and +Complete the three phases below before publishing an image or changing Bob's +`FROM` reference. When publication and consumer promotion are in scope, build and validate the approved fork commit, publish it to `eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob. Do not substitute upstream BuildKit for the Gitpod fork. @@ -45,6 +45,10 @@ patches so older pins do not undo upstream security fixes. Preserve the effect of required release-only fixes; if upstream already contains a fix, it does not need to be replayed. The following recipe keeps the remote branches unchanged: +The setup commands assume a fresh clone. Reuse an existing clean clone when +available, retaining its remotes and backups. Choose unused candidate and backup +branch names rather than overwriting previous work. + ```bash set -euo pipefail git clone https://github.com/gitpod-io/buildkit.git @@ -69,6 +73,12 @@ git am --3way "$BUILDKIT_PATCH_DIR"/*.patch git diff --check ``` +When patches conflict, adapt their intent to the selected upstream code and CI +structure. Do not restore an old workflow or dependency tree wholesale. Use +`git am --skip` only when a patch's effect is already included or superseded, +and record why. Keep a mapping of original patches to their replacements or +upstream equivalents, including mirror behavior and CI customizations. + Scan the candidate for remaining CVEs. If the selected upstream release still has vulnerable dependencies, update them in this same local candidate before requesting approval. Regenerate affected module and vendor files and update @@ -83,6 +93,43 @@ approval, and inspect the resulting BuildKit and bundled helper binaries. A Go pin does not rebuild downloaded CNI archives. Report which checks passed and which remain deferred; source module scans alone do not validate shipped binaries. +For native tests, the Git source suite requires signing fixtures. With Git, +GnuPG (including `gpg-agent`), and `ssh-keygen` installed, generate the upstream +fixtures before running the focused suite: + +```bash +set -euo pipefail +export BUILDKIT_TEST_SIGN_FIXTURES=$(mktemp -d /tmp/buildkit-sign-fixtures.XXXXXX) +for fixture_user in user1 user2; do + sh hack/fixtures/gen_gpg_test_env.sh "$fixture_user" + sh hack/fixtures/gen_ssh_test_env.sh "$fixture_user" +done +go test -mod=vendor ./util/resolver/... ./source/git/... ./util/gitutil/... +``` + +Keep the temporary signing keys outside the review artifacts. Validate the +mirror routing and localhost bypass explicitly; upstream tests alone may not +exercise Gitpod's customizations. To inspect the bundled binaries without +publishing an image, commit the candidate changes first so BuildKit's embedded +revision matches the commit being reviewed. Then use the fork's local export +target: + +```bash +set -euo pipefail +BUILDKIT_BINARY_DIR=$(mktemp -d /tmp/buildkit-binaries.XXXXXX) +docker buildx build --platform linux/amd64 --target binaries \ + --output "type=local,dest=$BUILDKIT_BINARY_DIR" . +grype "dir:$BUILDKIT_BINARY_DIR" --only-fixed --fail-on critical +``` + +Inspect Go metadata in the exported BuildKit, runc, and CNI binaries as well as +the scan results. If validation requires further changes, commit them and rebuild +the affected outputs before generating the final review diff. If an upstream +helper archive still contains vulnerable Go or +dependency versions, rebuild it from its selected source release with the fixes +in the fork's Dockerfile. Changing BuildKit's own `go.mod` cannot fix that archive. +Report image scanning and Bob preview validation separately if they are deferred. + After committing the local candidate, run these commands in the BuildKit clone. Set `BUILDKIT_UPDATE_BRANCH` to its local branch name. These commands fetch the remote branch and compare the two complete commit trees without working-tree From 8edaf87c792ab0ee96c4ab52b276d70abfab5568 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 19:19:15 +0000 Subject: [PATCH 7/9] Set CoreDev project before BuildKit image publication Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 1 + 1 file changed, 1 insertion(+) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index 4a8a4810b6f33b..29c66fdb266266 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -218,6 +218,7 @@ git rev-parse HEAD BUILDKIT_IMAGE="eu.gcr.io/gitpod-core-dev/build/buildkit:${BUILDKIT_IMAGE_TAG}" gcloud auth login +gcloud config set project gitpod-core-dev gcloud auth configure-docker eu.gcr.io --quiet docker buildx build \ From c651b6469a77fdbe76e36c95b7cfcfdac13d4917 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 21:28:23 +0000 Subject: [PATCH 8/9] Prefer published GHCR images in BuildKit CVE maintenance Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 102 +++++++++++++++++++++----- 1 file changed, 82 insertions(+), 20 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index 29c66fdb266266..eb814f0d927066 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -3,8 +3,9 @@ Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on. Complete the three phases below before publishing an image or changing Bob's `FROM` reference. When publication and -consumer promotion are in scope, build and validate the approved fork commit, -publish it to `eu.gcr.io/gitpod-core-dev/build/buildkit`, then update Bob. +consumer promotion are in scope, prefer the fork workflow's published +`ghcr.io/gitpod-io/buildkit` image, verify its source and scan results, then update +Bob. Use Core Dev only when the user explicitly requests that destination. Do not substitute upstream BuildKit for the Gitpod fork. Gitpod's customizations live on `custom-mirror`, the head branch of the @@ -195,14 +196,75 @@ default-branch sync is a separate remote change requiring its own authorization. Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork. -## Build, validate, and publish the approved image +## Publish and verify the approved GHCR image -After the fork change is reviewed and present on `custom-mirror`, use a clean -checkout of the approved commit. Continue only when image publication is in scope; -approval to update `custom-mirror` alone does not authorize publishing images. -Select a new, unused Gitpod image tag and record its source commit. Docker Buildx, -the Google Cloud CLI, Grype, and permission to push to Core Dev are prerequisites. -The login command is `gcloud auth login`. +After the approved fork change is on `custom-mirror`, prefer the image published +by `.github/workflows/buildkit.yml` to `ghcr.io/gitpod-io/buildkit`. Inspect the +current workflow before creating a tag: verify its tag trigger, registry, image +variant, architecture, and workflow state. Check whether the requested Gitpod tag +already has a published image before starting another build. Approval to update +`custom-mirror` alone does not authorize publishing tags or images. + +When publication is authorized and a new release is needed, create an unused +Gitpod release tag at the approved commit. Verify the workflow is active before +pushing the tag; a disabled workflow will not publish an image. For example: + +```bash +set -euo pipefail +: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}" +: "${BUILDKIT_RELEASE_TAG:?Choose a new unused Gitpod release tag}" +test -z "$(git status --porcelain)" +git fetch --no-tags origin \ + +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror +test "$(git rev-parse origin/custom-mirror)" = "$BUILDKIT_REVIEWED_CANDIDATE" +test -z "$(git ls-remote --tags origin "refs/tags/${BUILDKIT_RELEASE_TAG}")" +test "$(gh api repos/gitpod-io/buildkit/actions/workflows/buildkit.yml --jq .state)" = active +git tag -a "$BUILDKIT_RELEASE_TAG" "$BUILDKIT_REVIEWED_CANDIDATE" \ + -m "Gitpod BuildKit $BUILDKIT_RELEASE_TAG" +git push origin "refs/tags/${BUILDKIT_RELEASE_TAG}" +gh run list --repo gitpod-io/buildkit --workflow buildkit.yml \ + --branch "$BUILDKIT_RELEASE_TAG" +``` + +Wait for the image job to succeed. A manual dispatch on `custom-mirror` validates +the branch but does not create a tagged release; dispatch on the release tag if a +manual release run is needed. Do not create duplicate runs unnecessarily. Moving +an existing tag requires explicit authorization; never silently overwrite one. +Report image publication, draft release creation, and test results separately. +Inspect the current `needs` graph: a release can succeed while tests fail or are +still running. Do not describe successful publication as passing tests. + +Verify the published image instead of assuming it matches a local build: + +```bash +set -euo pipefail +: "${BUILDKIT_RELEASE_TAG:?Use the published Gitpod release tag}" +: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}" +BUILDKIT_IMAGE="ghcr.io/gitpod-io/buildkit:${BUILDKIT_RELEASE_TAG}" +docker buildx imagetools inspect "$BUILDKIT_IMAGE" +# Copy the verified index digest from that output; keep it in the review record. +: "${BUILDKIT_IMAGE_DIGEST:?Set the published sha256 digest}" +BUILDKIT_PINNED_IMAGE="${BUILDKIT_IMAGE}@${BUILDKIT_IMAGE_DIGEST}" +docker pull --platform linux/amd64 "$BUILDKIT_PINNED_IMAGE" +docker run --rm "$BUILDKIT_PINNED_IMAGE" --version \ + | tee /tmp/buildkit-published-version.txt +grep -F -- "$BUILDKIT_REVIEWED_CANDIDATE" /tmp/buildkit-published-version.txt +grype "docker:$BUILDKIT_PINNED_IMAGE" --fail-on critical +``` + +Verify the manifest includes Linux amd64 and that the selected image has the +rootful Alpine packaging Bob expects. Bob runs BuildKit inside workspacekit's +user namespace; that does not require switching its base to a `-rootless` image. +Keep the fork commit, workflow run, image tag/digest, and scan results in the PR. +Investigate scan failures before promoting the image; apply existing approved +vulnerability exceptions only with evidence that they still hold. + +### Core Dev publication when explicitly requested + +Core Dev is an alternative destination, not a prerequisite for using the fork's +GHCR image. If the user requests a local build and Core Dev publication, use the +approved commit, choose a new unused image tag, and run the following commands. +Docker Buildx, the Google Cloud CLI, Grype, and push permission are required. ```bash set -euo pipefail @@ -234,17 +296,16 @@ docker push "$BUILDKIT_IMAGE" docker buildx imagetools inspect "$BUILDKIT_IMAGE" ``` -This uses the fork's `buildkit` Dockerfile target and its committed toolchain -pins. Keep the source commit, scan results, image tag, and registry digest in -the change record. Investigate scan failures before pushing; apply existing -approved vulnerability exceptions only with evidence that they still hold. -Do not publish tags or images without authorization. +This uses the fork's `buildkit` target and committed toolchain pins. Record the +source commit, scan results, image tag, and registry digest. Do not publish tags +or images without authorization. ## Promote the image into Gitpod -Only after the image exists in Core Dev, update +Only after the verified image is published, update `components/image-builder-bob/leeway.Dockerfile` to consume -`eu.gcr.io/gitpod-core-dev/build/buildkit:` (preferably pinned by digest). +`ghcr.io/gitpod-io/buildkit:@sha256:`. +Use the Core Dev reference only when that destination was explicitly requested. Keep this consumer change small. Do not rebuild or replace the fork's BuildKit, runc, or CNI binaries inside Bob's Dockerfile as a shortcut. @@ -252,7 +313,7 @@ Build Bob, run the CI vulnerability scan described in [ci-scanning.md](ci-scanning.md), and validate an image build plus workspace start/stop/restart in a preview, following PR #21142's example. Include the fork source, published image/digest, consumer diff, scan run, and preview results in -the PR. If the fork update or manual publication is deferred, keep the current +the PR. If the fork update or image publication is deferred, keep the current image reference, report its remaining findings, and defer BuildKit remediation to a follow-on PR. Do not describe an earlier scan of different binaries as validation of the current image. @@ -282,6 +343,7 @@ validation of the current image. These records establish fork build/publication before consumer promotion. The 2025 consumer used GHCR; it does not prove that particular image was pushed to -Core Dev. The required destination for this procedure is now Core Dev. Its -`build/buildkit` path also appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5). -Do not assume the fork's current GitHub Actions publisher targets Core Dev. +Core Dev. GHCR is the preferred destination for this procedure. Core Dev's +`build/buildkit` path appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5) +and remains an option when explicitly requested. Inspect the current fork +workflow to verify its actual publishing destination. From 1dd98e027a9881d0c31e19215fca6be297fbfd09 Mon Sep 17 00:00:00 2001 From: Kyle Brennan Date: Tue, 20 Oct 2026 21:39:21 +0000 Subject: [PATCH 9/9] Document automatic BuildKit publication to GHCR Co-authored-by: Codex --- cve-mitigation/references/buildkit.md | 63 ++++----------------------- 1 file changed, 8 insertions(+), 55 deletions(-) diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md index eb814f0d927066..34afd4a87d5c28 100644 --- a/cve-mitigation/references/buildkit.md +++ b/cve-mitigation/references/buildkit.md @@ -1,11 +1,9 @@ # Updating the Gitpod BuildKit fork Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on. -Complete the three phases below before publishing an image or changing Bob's -`FROM` reference. When publication and -consumer promotion are in scope, prefer the fork workflow's published -`ghcr.io/gitpod-io/buildkit` image, verify its source and scan results, then update -Bob. Use Core Dev only when the user explicitly requests that destination. +Complete the three phases below, then wait for the fork's GitHub Actions +workflow to publish the release image to `ghcr.io/gitpod-io/buildkit` before +updating Bob's `FROM` reference. No manual image push is needed. Do not substitute upstream BuildKit for the Gitpod fork. Gitpod's customizations live on `custom-mirror`, the head branch of the @@ -259,53 +257,11 @@ Keep the fork commit, workflow run, image tag/digest, and scan results in the PR Investigate scan failures before promoting the image; apply existing approved vulnerability exceptions only with evidence that they still hold. -### Core Dev publication when explicitly requested - -Core Dev is an alternative destination, not a prerequisite for using the fork's -GHCR image. If the user requests a local build and Core Dev publication, use the -approved commit, choose a new unused image tag, and run the following commands. -Docker Buildx, the Google Cloud CLI, Grype, and push permission are required. - -```bash -set -euo pipefail -: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved candidate commit from commits.txt}" -test -z "$(git status --porcelain)" -git fetch --no-tags origin \ - +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror -test "$(git rev-parse origin/custom-mirror)" = "$BUILDKIT_REVIEWED_CANDIDATE" -git switch --detach "$BUILDKIT_REVIEWED_CANDIDATE" -git rev-parse HEAD - -: "${BUILDKIT_IMAGE_TAG:?Choose a new unused Gitpod image tag}" -BUILDKIT_IMAGE="eu.gcr.io/gitpod-core-dev/build/buildkit:${BUILDKIT_IMAGE_TAG}" - -gcloud auth login -gcloud config set project gitpod-core-dev -gcloud auth configure-docker eu.gcr.io --quiet - -docker buildx build \ - --platform linux/amd64 \ - --target buildkit \ - --pull --load \ - --tag "$BUILDKIT_IMAGE" . - -docker run --rm "$BUILDKIT_IMAGE" --version -grype "docker:$BUILDKIT_IMAGE" --fail-on critical - -docker push "$BUILDKIT_IMAGE" -docker buildx imagetools inspect "$BUILDKIT_IMAGE" -``` - -This uses the fork's `buildkit` target and committed toolchain pins. Record the -source commit, scan results, image tag, and registry digest. Do not publish tags -or images without authorization. - ## Promote the image into Gitpod Only after the verified image is published, update `components/image-builder-bob/leeway.Dockerfile` to consume `ghcr.io/gitpod-io/buildkit:@sha256:`. -Use the Core Dev reference only when that destination was explicitly requested. Keep this consumer change small. Do not rebuild or replace the fork's BuildKit, runc, or CNI binaries inside Bob's Dockerfile as a shortcut. @@ -328,12 +284,12 @@ validation of the current image. - The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845) records syncing the fork, exporting five customization patches, applying them to upstream `v0.20.1`, and then updating image-builder-bob for preview testing. - It also records that builds then published to GHCR and previously used Core Dev. + It also records publication to GHCR. - On November 12, 2025, [Gero reported rebuilding this branch and pushing `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498) to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4) resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the - `custom-mirror` head inspected for this documentation. His later + `custom-mirror` head inspected when this guide was first written. His later [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's base image from `gitpod.3` to `gitpod.4` and records image-build and workspace start/stop/restart checks in a preview. @@ -341,9 +297,6 @@ validation of the current image. followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1) moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory. -These records establish fork build/publication before consumer promotion. The -2025 consumer used GHCR; it does not prove that particular image was pushed to -Core Dev. GHCR is the preferred destination for this procedure. Core Dev's -`build/buildkit` path appears in the [older Bob Dockerfile](https://github.com/gitpod-io/gitpod/blob/e2b82f69d/components/image-builder-bob/leeway.Dockerfile#L5) -and remains an option when explicitly requested. Inspect the current fork -workflow to verify its actual publishing destination. +These records establish fork build/publication before consumer promotion. +Inspect the current fork workflow to verify its publishing destination and +release trigger before preparing a release.