kylos101 · Oct 20, 2026 · Oct 20, 2026 · Oct 20, 2026 · Oct 20, 2026 · Oct 20, 2026
diff --git a/cve-mitigation/SKILL.md b/cve-mitigation/SKILL.md
@@ -70,6 +70,12 @@ When a CVE is reported:
 4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change
    exported APIs. These require code migration, not just `go get`.
 
+## BuildKit fork updates
+
+The image-builder-bob component depends on Gitpod's BuildKit fork. Follow the
+[fork maintenance procedure](references/buildkit.md) to update it with upstream
+while preserving Gitpod's customizations.
+
 ## Remediation (Go dependencies)
 
 ### Standard update flow

diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md
@@ -0,0 +1,302 @@
+# Updating the Gitpod BuildKit fork
+
+Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on.
+Complete the three phases below, then wait for the fork's GitHub Actions
+workflow to publish the release image to `ghcr.io/gitpod-io/buildkit` before
+updating Bob's `FROM` reference. No manual image push is needed.
+Do not substitute upstream BuildKit for the Gitpod fork.
+
+Gitpod's customizations live on `custom-mirror`, the head branch of the
+long-running [fork PR #1](https://github.com/gitpod-io/buildkit/pull/1). Updates
+belong on that branch; do not merge PR #1 into the fork's default branch,
+`master`. Inspect its current state and comments rather than assuming a tag or
+the default branch contains the intended Gitpod changes.
+
+## Bring custom-mirror up to date with upstream
+
+### Phase 1: Select the version
+
+If the user specifies an explicit BuildKit version, use that version without
+prompting them to use the latest release. Otherwise, check the latest stable
+release in [moby/buildkit releases](https://github.com/moby/buildkit/releases)
+and ask whether they want to update to that specific version. Wait for their
+answer before preparing the update branch. Verify the selected upstream tag
+exists; do not select a version from the historical examples in this guide.
+
+### Phase 2: Prepare and compare locally, then request approval
+
+First inspect both repositories' history, the current Bob image, its source tag,
+and PR #1's comments. Verify the upstream base immediately before Gitpod's
+customization commits. Stop and report missing evidence if the patch series or
+published image's provenance cannot be established.
+
+Also compare `custom-mirror` with the currently consumed release. For example,
+the [gitpod.8 helper rebuild](https://github.com/gitpod-io/buildkit/commit/37c65868a73a80780ab4f2f5f302810471941d6f)
+and [Git source validation backport](https://github.com/gitpod-io/buildkit/commit/3198c5119)
+were on a separate release branch when this guide was written. Preserve required
+released fixes when bringing `custom-mirror` forward; starting from its head
+alone can omit them.
+
+Prepare a local candidate from the selected upstream release, adopting its
+dependency versions as the baseline. Replay Gitpod's customization patches using
+the process documented in PR #1. Review historical dependency and toolchain
+patches so older pins do not undo upstream security fixes. Preserve the effect
+of required release-only fixes; if upstream already contains a fix, it does not
+need to be replayed. The following recipe keeps the remote branches unchanged:
+
+The setup commands assume a fresh clone. Reuse an existing clean clone when
+available, retaining its remotes and backups. Choose unused candidate and backup
+branch names rather than overwriting previous work.
+
+```bash
+set -euo pipefail
+git clone https://github.com/gitpod-io/buildkit.git
+cd buildkit
+git remote add upstream https://github.com/moby/buildkit.git
+git fetch origin --tags
+git fetch upstream --tags
+
+# Set these after inspecting history; do not reuse old comment values.
+: "${BUILDKIT_UPSTREAM_RELEASE:?Select the upstream release tag}"
+: "${BUILDKIT_PATCH_BASE:?Verify the commit before Gitpod customizations}"
+: "${BUILDKIT_UPDATE_BRANCH:?Choose a new review branch name}"
+
+BUILDKIT_PATCH_DIR=$(mktemp -d)
+git branch backup/custom-mirror-before-update origin/custom-mirror
+git format-patch --output-directory "$BUILDKIT_PATCH_DIR" \
+  "${BUILDKIT_PATCH_BASE}..origin/custom-mirror"
+git switch -c "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_UPSTREAM_RELEASE"
+git am --3way "$BUILDKIT_PATCH_DIR"/*.patch
+
+# Review conflicts and restore any required release-only fixes identified above.
+git diff --check
+```
+
+When patches conflict, adapt their intent to the selected upstream code and CI
+structure. Do not restore an old workflow or dependency tree wholesale. Use
+`git am --skip` only when a patch's effect is already included or superseded,
+and record why. Keep a mapping of original patches to their replacements or
+upstream equivalents, including mirror behavior and CI customizations.
+
+Scan the candidate for remaining CVEs. If the selected upstream release still
+has vulnerable dependencies, update them in this same local candidate before
+requesting approval. Regenerate affected module and vendor files and update
+toolchain or bundled helper builds as needed. Both the upstream update and any
+additional dependency fixes must retain Gitpod's customization patches and
+mirror behavior. Record the extra updates and any unresolved findings for review.
+Do not replace patch replay with a blanket copy of another branch's module and
+vendor tree.
+
+Run relevant tests and builds on native Linux amd64 before requesting
+approval, and inspect the resulting BuildKit and bundled helper binaries. A Go
+pin does not rebuild downloaded CNI archives. Report which checks passed and
+which remain deferred; source module scans alone do not validate shipped binaries.
+
+For native tests, the Git source suite requires signing fixtures. With Git,
+GnuPG (including `gpg-agent`), and `ssh-keygen` installed, generate the upstream
+fixtures before running the focused suite:
+
+```bash
+set -euo pipefail
+export BUILDKIT_TEST_SIGN_FIXTURES=$(mktemp -d /tmp/buildkit-sign-fixtures.XXXXXX)
+for fixture_user in user1 user2; do
+  sh hack/fixtures/gen_gpg_test_env.sh "$fixture_user"
+  sh hack/fixtures/gen_ssh_test_env.sh "$fixture_user"
+done
+go test -mod=vendor ./util/resolver/... ./source/git/... ./util/gitutil/...
+```
+
+Keep the temporary signing keys outside the review artifacts. Validate the
+mirror routing and localhost bypass explicitly; upstream tests alone may not
+exercise Gitpod's customizations. To inspect the bundled binaries without
+publishing an image, commit the candidate changes first so BuildKit's embedded
+revision matches the commit being reviewed. Then use the fork's local export
+target:
+
+```bash
+set -euo pipefail
+BUILDKIT_BINARY_DIR=$(mktemp -d /tmp/buildkit-binaries.XXXXXX)
+docker buildx build --platform linux/amd64 --target binaries \
+  --output "type=local,dest=$BUILDKIT_BINARY_DIR" .
+grype "dir:$BUILDKIT_BINARY_DIR" --only-fixed --fail-on critical
+```
+
+Inspect Go metadata in the exported BuildKit, runc, and CNI binaries as well as
+the scan results. If validation requires further changes, commit them and rebuild
+the affected outputs before generating the final review diff. If an upstream
+helper archive still contains vulnerable Go or
+dependency versions, rebuild it from its selected source release with the fixes
+in the fork's Dockerfile. Changing BuildKit's own `go.mod` cannot fix that archive.
+Report image scanning and Bob preview validation separately if they are deferred.
+
+After committing the local candidate, run these commands in the BuildKit clone.
+Set `BUILDKIT_UPDATE_BRANCH` to its local branch name. These commands fetch the
+remote branch and compare the two complete commit trees without working-tree
+changes. A merge-base comparison can hide changes when the upstream baseline is
+replaced:
+
+```bash
+set -euo pipefail
+: "${BUILDKIT_UPDATE_BRANCH:?Set the local candidate branch name}"
+git fetch --no-tags origin \
+  +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror
+BUILDKIT_REVIEWED_REMOTE=$(git rev-parse --verify 'refs/remotes/origin/custom-mirror^{commit}')
+BUILDKIT_REVIEWED_CANDIDATE=$(git rev-parse --verify "refs/heads/${BUILDKIT_UPDATE_BRANCH}^{commit}")
+BUILDKIT_REVIEW_DIR=$(mktemp -d /tmp/buildkit-review.XXXXXX)
+BUILDKIT_DIFF_FILE="$BUILDKIT_REVIEW_DIR/custom-mirror-to-candidate.diff"
+
+printf 'branch=%s\nremote=%s\ncandidate=%s\n' \
+  "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_REVIEWED_REMOTE" \
+  "$BUILDKIT_REVIEWED_CANDIDATE" > "$BUILDKIT_REVIEW_DIR/commits.txt"
+git diff --no-ext-diff --no-textconv --no-color --binary --full-index \
+  --no-renames --src-prefix=a/ --dst-prefix=b/ \
+  "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \
+  > "$BUILDKIT_DIFF_FILE"
+git diff --no-color --stat --no-renames \
+  "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \
+  > "$BUILDKIT_REVIEW_DIR/summary.txt"
+if test -s "$BUILDKIT_DIFF_FILE"; then
+  git apply --numstat "$BUILDKIT_DIFF_FILE" > "$BUILDKIT_REVIEW_DIR/numstat.txt"
+  printf 'Review diff: %s\n' "$BUILDKIT_DIFF_FILE"
+else
+  printf 'No update: remote and candidate trees are identical.\n'
+fi
+```
+
+Share the resulting diff file as a clickable link, with the recorded commit IDs
+and summary. Additions describe what the candidate introduces; removals describe
+what it removes from remote `custom-mirror`. Binary changes are included. If the
+diff is empty, report that the trees are identical instead of requesting approval
+for an update. Keep the artifacts available through human review.
+
+Also share the local branch name, target upstream tag, patch adaptations, and
+validation results or limitations. A diff summary or inline excerpts alone are
+insufficient. Ask the human to approve updating or replacing remote
+`custom-mirror` with this specific candidate, including any required history
+rewrite. Stop and wait for their answer. Do not push branches or tags, sync the
+remote default branch, or publish images during this phase.
+
+### Phase 3: Update remote custom-mirror after approval
+
+Only explicit human approval of the reviewed candidate authorizes updating or
+replacing remote `custom-mirror`. A request to prepare a local candidate or edit
+this skill is not approval to push. If the user prohibited pushes, wait for
+explicit authorization lifting that restriction for this update.
+
+Fetch and recheck the remote tip and local candidate against the reviewed commit
+IDs before pushing. If either changed, regenerate the comparison and obtain
+approval again. Preserve the old remote tip in a backup branch. Use a normal
+push for a fast-forward update; if replacement requires rewriting history, use
+`--force-with-lease=refs/heads/custom-mirror:<reviewed-remote-sha>` only with
+explicit approval for that rewrite. Never use an unconditional force push.
+Verify the remote tip matches the approved candidate after the push. Any fork
+default-branch sync is a separate remote change requiring its own authorization.
+
+Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork.
+
+## Publish and verify the approved GHCR image
+
+After the approved fork change is on `custom-mirror`, prefer the image published
+by `.github/workflows/buildkit.yml` to `ghcr.io/gitpod-io/buildkit`. Inspect the
+current workflow before creating a tag: verify its tag trigger, registry, image
+variant, architecture, and workflow state. Check whether the requested Gitpod tag
+already has a published image before starting another build. Approval to update
+`custom-mirror` alone does not authorize publishing tags or images.
+
+When publication is authorized and a new release is needed, create an unused
+Gitpod release tag at the approved commit. Verify the workflow is active before
+pushing the tag; a disabled workflow will not publish an image. For example:
+
+```bash
+set -euo pipefail
+: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}"
+: "${BUILDKIT_RELEASE_TAG:?Choose a new unused Gitpod release tag}"
+test -z "$(git status --porcelain)"
+git fetch --no-tags origin \
+  +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror
+test "$(git rev-parse origin/custom-mirror)" = "$BUILDKIT_REVIEWED_CANDIDATE"
+test -z "$(git ls-remote --tags origin "refs/tags/${BUILDKIT_RELEASE_TAG}")"
+test "$(gh api repos/gitpod-io/buildkit/actions/workflows/buildkit.yml --jq .state)" = active
+git tag -a "$BUILDKIT_RELEASE_TAG" "$BUILDKIT_REVIEWED_CANDIDATE" \
+  -m "Gitpod BuildKit $BUILDKIT_RELEASE_TAG"
+git push origin "refs/tags/${BUILDKIT_RELEASE_TAG}"
+gh run list --repo gitpod-io/buildkit --workflow buildkit.yml \
+  --branch "$BUILDKIT_RELEASE_TAG"
+```
+
+Wait for the image job to succeed. A manual dispatch on `custom-mirror` validates
+the branch but does not create a tagged release; dispatch on the release tag if a
+manual release run is needed. Do not create duplicate runs unnecessarily. Moving
+an existing tag requires explicit authorization; never silently overwrite one.
+Report image publication, draft release creation, and test results separately.
+Inspect the current `needs` graph: a release can succeed while tests fail or are
+still running. Do not describe successful publication as passing tests.
+
+Verify the published image instead of assuming it matches a local build:
+
+```bash
+set -euo pipefail
+: "${BUILDKIT_RELEASE_TAG:?Use the published Gitpod release tag}"
+: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}"
+BUILDKIT_IMAGE="ghcr.io/gitpod-io/buildkit:${BUILDKIT_RELEASE_TAG}"
+docker buildx imagetools inspect "$BUILDKIT_IMAGE"
+# Copy the verified index digest from that output; keep it in the review record.
+: "${BUILDKIT_IMAGE_DIGEST:?Set the published sha256 digest}"
+BUILDKIT_PINNED_IMAGE="${BUILDKIT_IMAGE}@${BUILDKIT_IMAGE_DIGEST}"
+docker pull --platform linux/amd64 "$BUILDKIT_PINNED_IMAGE"
+docker run --rm "$BUILDKIT_PINNED_IMAGE" --version \
+  | tee /tmp/buildkit-published-version.txt
+grep -F -- "$BUILDKIT_REVIEWED_CANDIDATE" /tmp/buildkit-published-version.txt
+grype "docker:$BUILDKIT_PINNED_IMAGE" --fail-on critical
+```
+
+Verify the manifest includes Linux amd64 and that the selected image has the
+rootful Alpine packaging Bob expects. Bob runs BuildKit inside workspacekit's
+user namespace; that does not require switching its base to a `-rootless` image.
+Keep the fork commit, workflow run, image tag/digest, and scan results in the PR.
+Investigate scan failures before promoting the image; apply existing approved
+vulnerability exceptions only with evidence that they still hold.
+
+## Promote the image into Gitpod
+
+Only after the verified image is published, update
+`components/image-builder-bob/leeway.Dockerfile` to consume
+`ghcr.io/gitpod-io/buildkit:<gitpod-release-tag>@sha256:<verified-index-digest>`.
+Keep this consumer change small. Do not rebuild or replace the fork's BuildKit,
+runc, or CNI binaries inside Bob's Dockerfile as a shortcut.
+
+Build Bob, run the CI vulnerability scan described in
+[ci-scanning.md](ci-scanning.md), and validate an image build plus workspace
+start/stop/restart in a preview, following PR #21142's example. Include the fork
+source, published image/digest, consumer diff, scan run, and preview results in
+the PR. If the fork update or image publication is deferred, keep the current
+image reference, report its remaining findings, and defer BuildKit remediation
+to a follow-on PR. Do not describe an earlier scan of different binaries as
+validation of the current image.
+
+## Evidence from previous updates
+
+- The [February 2024 instructions in PR #1](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-1922138187)
+  describe syncing the fork's default branch, exporting Gitpod's patches,
+  selecting an upstream release tag, replaying the patches with `git am --3way`,
+  and updating `custom-mirror`. The tag and patch-base SHA in that comment are
+  historical examples, not values to reuse.
+- The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845)
+  records syncing the fork, exporting five customization patches, applying them
+  to upstream `v0.20.1`, and then updating image-builder-bob for preview testing.
+  It also records publication to GHCR.
+- On November 12, 2025, [Gero reported rebuilding this branch and pushing
+  `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498)
+  to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4)
+  resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the
+  `custom-mirror` head inspected when this guide was first written. His later
+  [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's
+  base image from `gitpod.3` to `gitpod.4` and records image-build and workspace
+  start/stop/restart checks in a preview.
+- A later example has [Gero's gRPC update in the fork](https://github.com/gitpod-io/buildkit/commit/aa03c4117c98110495de9d5d840e1c8cb106a221)
+  followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1)
+  moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory.
+
+These records establish fork build/publication before consumer promotion.
+Inspect the current fork workflow to verify its publishing destination and
+release trigger before preparing a release.