diff --git a/cve-mitigation/SKILL.md b/cve-mitigation/SKILL.md index 5a565700a4e5b3..d9e71adbdbd5b2 100644 --- a/cve-mitigation/SKILL.md +++ b/cve-mitigation/SKILL.md @@ -70,6 +70,12 @@ When a CVE is reported: 4. **Check for breaking API changes.** Some upgrades (e.g., `runc` v1.1.x → v1.2.x) change exported APIs. These require code migration, not just `go get`. +## BuildKit fork updates + +The image-builder-bob component depends on Gitpod's BuildKit fork. Follow the +[fork maintenance procedure](references/buildkit.md) to update it with upstream +while preserving Gitpod's customizations. + ## Remediation (Go dependencies) ### Standard update flow diff --git a/cve-mitigation/references/buildkit.md b/cve-mitigation/references/buildkit.md new file mode 100644 index 00000000000000..34afd4a87d5c28 --- /dev/null +++ b/cve-mitigation/references/buildkit.md @@ -0,0 +1,302 @@ +# Updating the Gitpod BuildKit fork + +Mitigate BuildKit CVEs in `gitpod-io/buildkit`, the fork Bob depends on. +Complete the three phases below, then wait for the fork's GitHub Actions +workflow to publish the release image to `ghcr.io/gitpod-io/buildkit` before +updating Bob's `FROM` reference. No manual image push is needed. +Do not substitute upstream BuildKit for the Gitpod fork. + +Gitpod's customizations live on `custom-mirror`, the head branch of the +long-running [fork PR #1](https://github.com/gitpod-io/buildkit/pull/1). Updates +belong on that branch; do not merge PR #1 into the fork's default branch, +`master`. Inspect its current state and comments rather than assuming a tag or +the default branch contains the intended Gitpod changes. + +## Bring custom-mirror up to date with upstream + +### Phase 1: Select the version + +If the user specifies an explicit BuildKit version, use that version without +prompting them to use the latest release. Otherwise, check the latest stable +release in [moby/buildkit releases](https://github.com/moby/buildkit/releases) +and ask whether they want to update to that specific version. Wait for their +answer before preparing the update branch. Verify the selected upstream tag +exists; do not select a version from the historical examples in this guide. + +### Phase 2: Prepare and compare locally, then request approval + +First inspect both repositories' history, the current Bob image, its source tag, +and PR #1's comments. Verify the upstream base immediately before Gitpod's +customization commits. Stop and report missing evidence if the patch series or +published image's provenance cannot be established. + +Also compare `custom-mirror` with the currently consumed release. For example, +the [gitpod.8 helper rebuild](https://github.com/gitpod-io/buildkit/commit/37c65868a73a80780ab4f2f5f302810471941d6f) +and [Git source validation backport](https://github.com/gitpod-io/buildkit/commit/3198c5119) +were on a separate release branch when this guide was written. Preserve required +released fixes when bringing `custom-mirror` forward; starting from its head +alone can omit them. + +Prepare a local candidate from the selected upstream release, adopting its +dependency versions as the baseline. Replay Gitpod's customization patches using +the process documented in PR #1. Review historical dependency and toolchain +patches so older pins do not undo upstream security fixes. Preserve the effect +of required release-only fixes; if upstream already contains a fix, it does not +need to be replayed. The following recipe keeps the remote branches unchanged: + +The setup commands assume a fresh clone. Reuse an existing clean clone when +available, retaining its remotes and backups. Choose unused candidate and backup +branch names rather than overwriting previous work. + +```bash +set -euo pipefail +git clone https://github.com/gitpod-io/buildkit.git +cd buildkit +git remote add upstream https://github.com/moby/buildkit.git +git fetch origin --tags +git fetch upstream --tags + +# Set these after inspecting history; do not reuse old comment values. +: "${BUILDKIT_UPSTREAM_RELEASE:?Select the upstream release tag}" +: "${BUILDKIT_PATCH_BASE:?Verify the commit before Gitpod customizations}" +: "${BUILDKIT_UPDATE_BRANCH:?Choose a new review branch name}" + +BUILDKIT_PATCH_DIR=$(mktemp -d) +git branch backup/custom-mirror-before-update origin/custom-mirror +git format-patch --output-directory "$BUILDKIT_PATCH_DIR" \ + "${BUILDKIT_PATCH_BASE}..origin/custom-mirror" +git switch -c "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_UPSTREAM_RELEASE" +git am --3way "$BUILDKIT_PATCH_DIR"/*.patch + +# Review conflicts and restore any required release-only fixes identified above. +git diff --check +``` + +When patches conflict, adapt their intent to the selected upstream code and CI +structure. Do not restore an old workflow or dependency tree wholesale. Use +`git am --skip` only when a patch's effect is already included or superseded, +and record why. Keep a mapping of original patches to their replacements or +upstream equivalents, including mirror behavior and CI customizations. + +Scan the candidate for remaining CVEs. If the selected upstream release still +has vulnerable dependencies, update them in this same local candidate before +requesting approval. Regenerate affected module and vendor files and update +toolchain or bundled helper builds as needed. Both the upstream update and any +additional dependency fixes must retain Gitpod's customization patches and +mirror behavior. Record the extra updates and any unresolved findings for review. +Do not replace patch replay with a blanket copy of another branch's module and +vendor tree. + +Run relevant tests and builds on native Linux amd64 before requesting +approval, and inspect the resulting BuildKit and bundled helper binaries. A Go +pin does not rebuild downloaded CNI archives. Report which checks passed and +which remain deferred; source module scans alone do not validate shipped binaries. + +For native tests, the Git source suite requires signing fixtures. With Git, +GnuPG (including `gpg-agent`), and `ssh-keygen` installed, generate the upstream +fixtures before running the focused suite: + +```bash +set -euo pipefail +export BUILDKIT_TEST_SIGN_FIXTURES=$(mktemp -d /tmp/buildkit-sign-fixtures.XXXXXX) +for fixture_user in user1 user2; do + sh hack/fixtures/gen_gpg_test_env.sh "$fixture_user" + sh hack/fixtures/gen_ssh_test_env.sh "$fixture_user" +done +go test -mod=vendor ./util/resolver/... ./source/git/... ./util/gitutil/... +``` + +Keep the temporary signing keys outside the review artifacts. Validate the +mirror routing and localhost bypass explicitly; upstream tests alone may not +exercise Gitpod's customizations. To inspect the bundled binaries without +publishing an image, commit the candidate changes first so BuildKit's embedded +revision matches the commit being reviewed. Then use the fork's local export +target: + +```bash +set -euo pipefail +BUILDKIT_BINARY_DIR=$(mktemp -d /tmp/buildkit-binaries.XXXXXX) +docker buildx build --platform linux/amd64 --target binaries \ + --output "type=local,dest=$BUILDKIT_BINARY_DIR" . +grype "dir:$BUILDKIT_BINARY_DIR" --only-fixed --fail-on critical +``` + +Inspect Go metadata in the exported BuildKit, runc, and CNI binaries as well as +the scan results. If validation requires further changes, commit them and rebuild +the affected outputs before generating the final review diff. If an upstream +helper archive still contains vulnerable Go or +dependency versions, rebuild it from its selected source release with the fixes +in the fork's Dockerfile. Changing BuildKit's own `go.mod` cannot fix that archive. +Report image scanning and Bob preview validation separately if they are deferred. + +After committing the local candidate, run these commands in the BuildKit clone. +Set `BUILDKIT_UPDATE_BRANCH` to its local branch name. These commands fetch the +remote branch and compare the two complete commit trees without working-tree +changes. A merge-base comparison can hide changes when the upstream baseline is +replaced: + +```bash +set -euo pipefail +: "${BUILDKIT_UPDATE_BRANCH:?Set the local candidate branch name}" +git fetch --no-tags origin \ + +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror +BUILDKIT_REVIEWED_REMOTE=$(git rev-parse --verify 'refs/remotes/origin/custom-mirror^{commit}') +BUILDKIT_REVIEWED_CANDIDATE=$(git rev-parse --verify "refs/heads/${BUILDKIT_UPDATE_BRANCH}^{commit}") +BUILDKIT_REVIEW_DIR=$(mktemp -d /tmp/buildkit-review.XXXXXX) +BUILDKIT_DIFF_FILE="$BUILDKIT_REVIEW_DIR/custom-mirror-to-candidate.diff" + +printf 'branch=%s\nremote=%s\ncandidate=%s\n' \ + "$BUILDKIT_UPDATE_BRANCH" "$BUILDKIT_REVIEWED_REMOTE" \ + "$BUILDKIT_REVIEWED_CANDIDATE" > "$BUILDKIT_REVIEW_DIR/commits.txt" +git diff --no-ext-diff --no-textconv --no-color --binary --full-index \ + --no-renames --src-prefix=a/ --dst-prefix=b/ \ + "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \ + > "$BUILDKIT_DIFF_FILE" +git diff --no-color --stat --no-renames \ + "$BUILDKIT_REVIEWED_REMOTE" "$BUILDKIT_REVIEWED_CANDIDATE" -- \ + > "$BUILDKIT_REVIEW_DIR/summary.txt" +if test -s "$BUILDKIT_DIFF_FILE"; then + git apply --numstat "$BUILDKIT_DIFF_FILE" > "$BUILDKIT_REVIEW_DIR/numstat.txt" + printf 'Review diff: %s\n' "$BUILDKIT_DIFF_FILE" +else + printf 'No update: remote and candidate trees are identical.\n' +fi +``` + +Share the resulting diff file as a clickable link, with the recorded commit IDs +and summary. Additions describe what the candidate introduces; removals describe +what it removes from remote `custom-mirror`. Binary changes are included. If the +diff is empty, report that the trees are identical instead of requesting approval +for an update. Keep the artifacts available through human review. + +Also share the local branch name, target upstream tag, patch adaptations, and +validation results or limitations. A diff summary or inline excerpts alone are +insufficient. Ask the human to approve updating or replacing remote +`custom-mirror` with this specific candidate, including any required history +rewrite. Stop and wait for their answer. Do not push branches or tags, sync the +remote default branch, or publish images during this phase. + +### Phase 3: Update remote custom-mirror after approval + +Only explicit human approval of the reviewed candidate authorizes updating or +replacing remote `custom-mirror`. A request to prepare a local candidate or edit +this skill is not approval to push. If the user prohibited pushes, wait for +explicit authorization lifting that restriction for this update. + +Fetch and recheck the remote tip and local candidate against the reviewed commit +IDs before pushing. If either changed, regenerate the comparison and obtain +approval again. Preserve the old remote tip in a backup branch. Use a normal +push for a fast-forward update; if replacement requires rewriting history, use +`--force-with-lease=refs/heads/custom-mirror:` only with +explicit approval for that rewrite. Never use an unconditional force push. +Verify the remote tip matches the approved candidate after the push. Any fork +default-branch sync is a separate remote change requiring its own authorization. + +Do not merge PR #1 into `master` or publish an upstream image as the Gitpod fork. + +## Publish and verify the approved GHCR image + +After the approved fork change is on `custom-mirror`, prefer the image published +by `.github/workflows/buildkit.yml` to `ghcr.io/gitpod-io/buildkit`. Inspect the +current workflow before creating a tag: verify its tag trigger, registry, image +variant, architecture, and workflow state. Check whether the requested Gitpod tag +already has a published image before starting another build. Approval to update +`custom-mirror` alone does not authorize publishing tags or images. + +When publication is authorized and a new release is needed, create an unused +Gitpod release tag at the approved commit. Verify the workflow is active before +pushing the tag; a disabled workflow will not publish an image. For example: + +```bash +set -euo pipefail +: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}" +: "${BUILDKIT_RELEASE_TAG:?Choose a new unused Gitpod release tag}" +test -z "$(git status --porcelain)" +git fetch --no-tags origin \ + +refs/heads/custom-mirror:refs/remotes/origin/custom-mirror +test "$(git rev-parse origin/custom-mirror)" = "$BUILDKIT_REVIEWED_CANDIDATE" +test -z "$(git ls-remote --tags origin "refs/tags/${BUILDKIT_RELEASE_TAG}")" +test "$(gh api repos/gitpod-io/buildkit/actions/workflows/buildkit.yml --jq .state)" = active +git tag -a "$BUILDKIT_RELEASE_TAG" "$BUILDKIT_REVIEWED_CANDIDATE" \ + -m "Gitpod BuildKit $BUILDKIT_RELEASE_TAG" +git push origin "refs/tags/${BUILDKIT_RELEASE_TAG}" +gh run list --repo gitpod-io/buildkit --workflow buildkit.yml \ + --branch "$BUILDKIT_RELEASE_TAG" +``` + +Wait for the image job to succeed. A manual dispatch on `custom-mirror` validates +the branch but does not create a tagged release; dispatch on the release tag if a +manual release run is needed. Do not create duplicate runs unnecessarily. Moving +an existing tag requires explicit authorization; never silently overwrite one. +Report image publication, draft release creation, and test results separately. +Inspect the current `needs` graph: a release can succeed while tests fail or are +still running. Do not describe successful publication as passing tests. + +Verify the published image instead of assuming it matches a local build: + +```bash +set -euo pipefail +: "${BUILDKIT_RELEASE_TAG:?Use the published Gitpod release tag}" +: "${BUILDKIT_REVIEWED_CANDIDATE:?Use the approved fork commit}" +BUILDKIT_IMAGE="ghcr.io/gitpod-io/buildkit:${BUILDKIT_RELEASE_TAG}" +docker buildx imagetools inspect "$BUILDKIT_IMAGE" +# Copy the verified index digest from that output; keep it in the review record. +: "${BUILDKIT_IMAGE_DIGEST:?Set the published sha256 digest}" +BUILDKIT_PINNED_IMAGE="${BUILDKIT_IMAGE}@${BUILDKIT_IMAGE_DIGEST}" +docker pull --platform linux/amd64 "$BUILDKIT_PINNED_IMAGE" +docker run --rm "$BUILDKIT_PINNED_IMAGE" --version \ + | tee /tmp/buildkit-published-version.txt +grep -F -- "$BUILDKIT_REVIEWED_CANDIDATE" /tmp/buildkit-published-version.txt +grype "docker:$BUILDKIT_PINNED_IMAGE" --fail-on critical +``` + +Verify the manifest includes Linux amd64 and that the selected image has the +rootful Alpine packaging Bob expects. Bob runs BuildKit inside workspacekit's +user namespace; that does not require switching its base to a `-rootless` image. +Keep the fork commit, workflow run, image tag/digest, and scan results in the PR. +Investigate scan failures before promoting the image; apply existing approved +vulnerability exceptions only with evidence that they still hold. + +## Promote the image into Gitpod + +Only after the verified image is published, update +`components/image-builder-bob/leeway.Dockerfile` to consume +`ghcr.io/gitpod-io/buildkit:@sha256:`. +Keep this consumer change small. Do not rebuild or replace the fork's BuildKit, +runc, or CNI binaries inside Bob's Dockerfile as a shortcut. + +Build Bob, run the CI vulnerability scan described in +[ci-scanning.md](ci-scanning.md), and validate an image build plus workspace +start/stop/restart in a preview, following PR #21142's example. Include the fork +source, published image/digest, consumer diff, scan run, and preview results in +the PR. If the fork update or image publication is deferred, keep the current +image reference, report its remaining findings, and defer BuildKit remediation +to a follow-on PR. Do not describe an earlier scan of different binaries as +validation of the current image. + +## Evidence from previous updates + +- The [February 2024 instructions in PR #1](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-1922138187) + describe syncing the fork's default branch, exporting Gitpod's patches, + selecting an upstream release tag, replaying the patches with `git am --3way`, + and updating `custom-mirror`. The tag and patch-base SHA in that comment are + historical examples, not values to reuse. +- The [March 2025 update](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-2742603845) + records syncing the fork, exporting five customization patches, applying them + to upstream `v0.20.1`, and then updating image-builder-bob for preview testing. + It also records publication to GHCR. +- On November 12, 2025, [Gero reported rebuilding this branch and pushing + `v0.20.1-gitpod.4`](https://github.com/gitpod-io/buildkit/pull/1#issuecomment-3521030498) + to pick up Alpine 3.22.2. That [fork tag](https://github.com/gitpod-io/buildkit/tree/v0.20.1-gitpod.4) + resolves to `93509e16a383c7e9baf619c704ae632b864196f1`, also the + `custom-mirror` head inspected when this guide was first written. His later + [Gitpod PR #21142](https://github.com/gitpod-io/gitpod/pull/21142) changes Bob's + base image from `gitpod.3` to `gitpod.4` and records image-build and workspace + start/stop/restart checks in a preview. +- A later example has [Gero's gRPC update in the fork](https://github.com/gitpod-io/buildkit/commit/aa03c4117c98110495de9d5d840e1c8cb106a221) + followed by a [Gitpod consumer commit](https://github.com/gitpod-io/gitpod/commit/e9642e9aa3c1172ed57d74d2d0aac6d84b62a9d1) + moving Bob from `gitpod.5` to `gitpod.6` for that gRPC advisory. + +These records establish fork build/publication before consumer promotion. +Inspect the current fork workflow to verify its publishing destination and +release trigger before preparing a release.