feat(ingress): record source metadata in the image - #121
Merged
Merged
Conversation
Add OCI labels, manifest annotations and /etc/dstack-ingress/build-info pointing back to the source repository, commit, version and pinned base image, so a published digest can be traced to the code that built it. All values derive from the git checkout (commit, the new VERSION file, the Dockerfile FROM line), so rebuilding a clean checkout of the same commit still yields the same digest. The build script warns and marks the revision -dirty when the tree has uncommitted or untracked changes; CI passes --require-clean and checks that the release tag matches VERSION. The entrypoint prints the build info as its first log line. Also add a .dockerignore so untracked local artifacts (oci.tar, __pycache__, .pytest_cache) neither inflate the build context nor leak into the image.
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
Only minor documentation/comment wording adjustments are suggested; the functional changes appear consistent and complete.
Pull request overview
This PR makes the custom-domain/dstack-ingress image self-describing by embedding standard OCI source metadata into (1) image config labels, (2) manifest annotations, and (3) an in-image /etc/dstack-ingress/build-info file, while preserving reproducible digests between CI and local builds.
Changes:
- Add a committed
VERSIONfile and enforce tag ↔ VERSION consistency in the release workflow. - Extend
build-image.shto generate consistent OCI labels/manifest annotations and emit the same key/value set into.BUILD_INFOcopied into the image. - Print the build identification line on container startup and document how to inspect the metadata.
File summaries
| File | Description |
|---|---|
| custom-domain/dstack-ingress/VERSION | Introduces committed version source for reproducible builds and release tagging. |
| custom-domain/dstack-ingress/scripts/entrypoint.sh | Prints build identification from /etc/dstack-ingress/build-info early at startup. |
| custom-domain/dstack-ingress/README.md | Documents OCI metadata locations/keys and reproducible build workflow. |
| custom-domain/dstack-ingress/Dockerfile | Copies generated .BUILD_INFO into the image as /etc/dstack-ingress/build-info. |
| custom-domain/dstack-ingress/build-image.sh | Generates metadata, enforces clean-tree behavior (optional), and applies labels/annotations deterministically. |
| custom-domain/dstack-ingress/.gitignore | Ignores the generated .BUILD_INFO artifact. |
| custom-domain/dstack-ingress/.dockerignore | Prevents large/local artifacts from entering build context and impacting digest. |
| .github/workflows/dstack-ingress-release.yml | Enforces VERSION matching and builds releases with --require-clean, plus improved release notes. |
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The version gate parsed the version out of GITHUB_REF unconditionally, so a workflow_dispatch run took the literal string 'refs/heads/main' as the version and only failed later at the registry push. Derive the version from the committed VERSION file, which is what the image records, and require the ref to be a dstack-ingress-v* tag that matches it.
The reproduce instructions dropped the prerequisite line the v2.4 notes carried. The pinned toolchain only reproduces the published digest on a native linux/amd64 host, so someone following the steps on arm64 would get a mismatch with nothing explaining why.
dstack-ingress-v2.5 was tagged and published on 2026-09-03, so 2.5 is taken: docker.io/dstacktee/dstack-ingress:2.5 already exists at sha256:97285855. Releasing 2.5 again would either fail on the existing tag or overwrite a published digest that users may have pinned.
The exclude list enumerated junk, and .pytest_cache/ was anchored to the context root while its sibling **/__pycache__/ was not. scripts/ holds test_*.py and the repo has no pytest rootdir anchor, so `cd scripts/tests && pytest` writes scripts/tests/.pytest_cache/, which the Dockerfile then bind-mounts into /scripts and the digest changes. --require-clean cannot catch that: pytest drops a .gitignore containing `*` into its cache, so git reports a clean tree while the context is not. Any other tool that ships a self-ignoring cache would slip through the same way. Turn the list around. The image needs .BUILD_INFO, pinned-packages.txt, requirements.txt and scripts/; excluding everything else by default makes the property hold by construction instead of by enumeration.
Every run step in the job operates on custom-domain/dstack-ingress, but the version gate had no working-directory and so spelled the path out three more times, in the read and in both error messages. A job-level default covers all of them, and the gate can name the VERSION file the same way the docs do.
The base.name and base.digest labels are what someone verifying the supply chain checks the image against, and they were taken from the first FROM in the Dockerfile. That is right today, with one stage, and silently wrong the day a builder stage is added ahead of it: the image would claim a base it was never built on, and nothing would say so. Require exactly one FROM. Adding a stage now fails the build with a message naming the candidates, instead of shipping a wrong label.
Three sed invocations over the same six-line file, one per field. One awk pass produces the same line, and a value containing '=' survives it.
The release steps stopped at the tag, which is exactly where 2.4 and 2.5 stopped: both were built and pushed, neither was ever pinned, so the compose files and README snippets people copy still deploy 2.3 and the repository holds no record mapping those two digests to a commit. Spell out the whole sequence -- VERSION, tag, pin -- and give a grep that lists every place the examples name the image, so the last step cannot be the one nobody remembers.
scripts/ is re-included wholesale, so the whitelist only held outside it: scripts/.mypy_cache/ or scripts/.ruff_cache/ still reached the image, and those tools self-ignore in git exactly like pytest's cache, so the dirty warning and --require-clean stayed quiet about a changed digest. Exclude the dot directories anywhere under scripts/, plus __pycache__, which is the one of these that does not start with a dot.
pinned-packages.txt is both an input to the build -- it is bind-mounted in to pin apt -- and the record of what the build installed, and the script rewrote it in place after the cleanliness check had already run. So a stale committed file was invisible: the release built against the old pins and published them as the record, while anyone following the reproduce recipe got a dirty tree on the second run and an image stamped -dirty for a commit that is not. Compare instead. A release (--require-clean) now fails, before the push, with the diff that explains why; a local build regenerates the file and says the image it just produced used the previous pins.
The mismatch was only fatal under --require-clean, so the documented local path -- ./build-image.sh --push repo:tag -- warned, rewrote the file and pushed anyway: the published image was built from the stale pins and the record went out with it. Gate on --push as well. Print the whole diff while doing so. Truncating at 40 lines makes a snapshot bump that moves most of the 136 pins look like a small and complete one, and the file is short enough that there is nothing to protect the log from. The release steps now also say to regenerate the file before tagging, which is where the failure would otherwise land.
The local mismatch path regenerated the pin file, warned, and then printed the ready-to-paste skopeo copy line anyway. That tarball was built with the pins that had just been replaced, and the dirty check ran before the rewrite, so it carries a clean revision label: pasting the command publishes exactly the self-contradicting image this check exists to stop, by the route the script itself suggests. Say why instead.
kvinwang
force-pushed
the
feat/ingress-image-metadata
branch
from
September 20, 2026 02:01
bff597b to
8e854f4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A published
dstacktee/dstack-ingress@sha256:…currently carries nothing that points back to the code that built it (configLabels: null, no manifest annotations;/etc/GIT_REVexisted but nothing read or documented it). This PR makes the image self-describing while keeping the digest reproducible between CI and a local./build-image.sh.The same key/value set is recorded in three places:
skopeo inspect docker://<ref> | jq .Labels,docker inspectskopeo inspect --raw docker://<ref> | jq .annotations/etc/dstack-ingress/build-info(replaces/etc/GIT_REV)catinside the container; also printed as the first line of the container logKeys are the standard OCI ones:
source,revision,version,url,documentation,title,description,licenses,base.name,base.digest.Reproducibility
Every value is derived from the git checkout only:
git rev-parse HEAD, the new committedVERSIONfile, the DockerfileFROMdigest, and a hard-coded canonicalSOURCE_URL(overridable for forks). No build time, CI run id or runner identity goes in, so the digest of a clean checkout is stable.build-image.shwarns and marksrevisionas<sha>-dirtywhen the tree has uncommitted or untracked changes. CI passes--require-clean.VERSIONis a committed file rather thangit describe, sogit checkout <sha> && ./build-image.shreproduces the digest even without tags fetched. Bumping the version is now a source change before tagging (set to2.5for the next release).Bonus: fixes a silent digest divergence
scripts/__pycache__is gitignored, and the Dockerfile copiesscripts/wholesale into/scripts. So any developer who had run the test suite baked stray.pycfiles into their image — with no dirty-tree warning and a digest quietly different from CI's. This is the same class of bug as thechmod 0644 requirements.txtcaveat in the v2.4 release notes. The new.dockerignorefixes it (and keeps the 120 MBoci.tarout of the build context).Release workflow
The version gate now derives the version from
VERSIONand requires the ref to be a matchingdstack-ingress-v*tag. Previously aworkflow_dispatchrun parsed the version as the literal stringrefs/heads/mainand only failed much later at the registry push. The release body gained the source commit, how to read the metadata, and the reproduce commands.Note: only
manifest:annotations are used. BuildKit v0.20.2 rejectsindex:/index-descriptor:annotations for single-platform OCI export, andmanifest-descriptor:only lands in the localindex.json, whichskopeo copydoes not push.Testing
Determinism and the
.dockerignorefixTwo full builds at
a66b950, the second with a simulated developer tree (gitignoredscripts/__pycache__/*.pyc,scripts/tests/__pycache__/*.pyc,.pytest_cache/present):sha256:bb0a3aa8f5dbd1ea9221770b7c82723ffda67546c575025fb73e2b247f03fc24sha256:bb0a3aa8f5dbd1ea9221770b7c82723ffda67546c575025fb73e2b247f03fc24Identical, which covers both determinism and the
.dockerignorefix. Reproduce on a native linux/amd64 host:Metadata survives the push to a registry
Docker v2s2 manifests have no
annotationsfield, so this had to be verified rather than assumed. Pushed the built image through a local registry with the exact command CI uses (skopeo copy --insecure-policy oci-archive:./oci.tar docker://…):sha256:d4ce223a…sha256:d4ce223a…(unchanged)application/vnd.oci.image.manifest.v1+jsonThe digest being unchanged also matters for the existing attestation step, which takes
subject-digestfrom the localoci.tar. Cross-checked against the real world: the publisheddstacktee/dstack-ingress:2.4manifest is OCI format and its registry digest matches the v2.4 release notes exactly, so skopeo already preserves the format today.Other checks
revisionequalsHEAD,versionis2.5__pycache__under/scriptsin the resulting image/etc/dstack-ingress/build-infomatches; container log starts withdstack-ingress 2.5 revision … source https://github.com/Dstack-TEE/dstack-examples--require-cleanwith an untracked file underscripts/fails before building and leaves no.BUILD_INFObehindpinned-packages.txtunchanged)dstack-ingress-v2.5(pass),dstack-ingress-v9.9(fail, mismatch),refs/heads/main(fail, not a tag)--annotationneeds Buildx ≥ 0.12; ubuntu-24.04 runners ship 0.36.1Follow-up
docker-compose.yaml/docker-compose.multi.yamlstill pin2.3; they get bumped to the real digest after the release build, as in dc34a5d.