From ad298eb0684c1c5c699baf131f791d7663781af3 Mon Sep 17 00:00:00 2001 From: Sebastian Mendel Date: Mon, 21 Sep 2026 12:18:55 +0200 Subject: [PATCH 1/2] docs(release): a repo without a release workflow gets one, not a manual create The skill described "tag-only repositories" as a flow: create the release by hand with --verify-tag and let release-published workflows add provenance. The build then runs in the repository, which is SLSA Build Level 2 at best, and netresearch/timetracker stayed there. Its template for projects shipped as source (python-release.yml with a caller build-cmd plus attest-release-files.yml) is Level 2 for the same reason. - ci-workflow-templates.md: the default for a source-tree project is a tag-push caller of netresearch/.github's release-source-archive.yml plus verify-release.yml. It builds inside the reusable (Level 3) and creates the release. The python-release.yml shape stays as the Level 2 alternative for releases that need their own build command. - SKILL.md, release-process.md, AGENTS.md: a missing release workflow is a gap to close; the manual --verify-tag create is the step for a release that cannot wait for it. - Eval 41 also expects the lasting fix to be named. Assisted-by: claude-code:claude-opus-5 Agent-Session: https://claude.ai/code/session_01Cn9YYfLWrmSH1t13kGqe4K Agent-Host: 0493f0 Signed-off-by: Sebastian Mendel --- AGENTS.md | 2 +- skills/github-release/SKILL.md | 4 +- skills/github-release/evals/evals.json | 6 +- .../references/ci-workflow-templates.md | 57 ++++++++++--------- .../references/release-process.md | 2 +- 5 files changed, 40 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8b94943..515c857 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,4 +24,4 @@ commands/ /release, /release-prepare, /release-status 3. Create signed tag: `git tag -s vX.Y.Z -m "vX.Y.Z"` 4. Push: `git push origin main vX.Y.Z` -**NEVER use `gh release create` without `--verify-tag`.** Where a release workflow exists it handles GitHub release creation. Where none does, publish against the already-pushed signed tag: `gh release create vX.Y.Z --verify-tag --notes-file ` — `--verify-tag` aborts rather than creating a lightweight tag. +**NEVER use `gh release create` without `--verify-tag`.** Where a release workflow exists it handles GitHub release creation. Where none does, the fix is to add one that calls the org reusable for the artefact (`release-source-archive.yml` for a source tree; SLSA Build Level 3, see `skills/github-release/references/ci-workflow-templates.md`). Until then, publish against the already-pushed signed tag: `gh release create vX.Y.Z --verify-tag --notes-file ` — `--verify-tag` aborts rather than creating a lightweight tag. diff --git a/skills/github-release/SKILL.md b/skills/github-release/SKILL.md index 5416193..c6b9e91 100644 --- a/skills/github-release/SKILL.md +++ b/skills/github-release/SKILL.md @@ -18,7 +18,7 @@ allowed-tools: Bash(gh:*) Bash(git:*) Read Write Edit Glob Grep Blocked by hooks. Immutable releases (GA Oct 2025) make tag names permanent — a bare `gh release create` creates the tag when it is missing, that tag is lightweight, and the name is burned forever. `--verify-tag` aborts unless the tag already exists on the remote, so the invocation can only publish a tag somebody pushed on purpose; the guard allows it for that reason (`--verify-tag=false` stays blocked). -Where the repository has a workflow that publishes the release, that workflow does it and you do not run `gh release create` at all. Where it does not — see *Tag-only repositories* below — publishing by hand against the pushed signed tag is the correct step, not a workaround. +Where the repository has a workflow that publishes the release, that workflow does it and you do not run `gh release create` at all. Where it does not, **add one** — for a project shipped as its source tree, a tag-push caller of `netresearch/.github`'s `release-source-archive.yml` (`references/ci-workflow-templates.md`). Publishing by hand against the pushed signed tag (*Tag-only repositories* below) is the step for a release that cannot wait for that workflow, not a flow to keep. **`gh release edit` is allowed ONLY for `--notes` / `--notes-file`** to overhaul the release description after CI publishes. All other `gh release edit` flags are blocked. @@ -45,6 +45,8 @@ checks the published body. ### Tag-only repositories +**This state is a gap to close, not a flow to keep.** A repository in it gets its provenance from a workflow that runs after a hand-made release, so the build happens in the repository itself — SLSA Build Level 2 at best, because GitHub grants Level 3 only when the build runs inside a reusable the repository cannot edit. The fix is a release workflow on the tag push that calls the org reusable for the artefact: `release-source-archive.yml` (source tree), `release-go-app.yml` (Go), `release-typo3-extension.yml` (TYPO3). It builds, generates SBOMs, signs, attests and creates the release in one run; see `references/ci-workflow-templates.md`. Until that workflow exists: + Some repositories have no workflow that creates the release object. Their supply-chain workflows listen on `release: published` instead — provenance attestation, SBOM upload, registry publish — and that event never fires for a release created with `GITHUB_TOKEN` (`references/typo3-ter-publishing.md` has the mechanism). The release therefore has to be created by a human credential, which is `gh` on your machine: ```bash diff --git a/skills/github-release/evals/evals.json b/skills/github-release/evals/evals.json index 5a18513..6aeeb54 100644 --- a/skills/github-release/evals/evals.json +++ b/skills/github-release/evals/evals.json @@ -1110,7 +1110,7 @@ { "id": 41, "prompt": "v6.4.0 is tagged with git tag -s and pushed. This repository has no workflow that creates the GitHub release - slsa-provenance.yml and the SBOM upload both trigger on `release: published`. Publish the release.", - "expected_output": "Recognise that no workflow creates the release object, and that a release created with GITHUB_TOKEN would not fire release: published, so the release must be created with a human credential. Run gh release create v6.4.0 --verify-tag with the notes, then verify the downstream jobs ran - gh attestation verify on the published archive.", + "expected_output": "Recognise that no workflow creates the release object, and that a release created with GITHUB_TOKEN would not fire release: published, so the release must be created with a human credential. Run gh release create v6.4.0 --verify-tag with the notes, then verify the downstream jobs ran - gh attestation verify on the published archive. Then state that the lasting fix is a tag-push release workflow calling netresearch/.github's release-source-archive.yml, which builds the archive inside the reusable (SLSA Build Level 3) and creates the release itself, so no release needs creating by hand again.", "expectations": [ "Identifies that no workflow publishes the release", "Uses --verify-tag", @@ -1135,6 +1135,10 @@ { "type": "must_not", "pattern": "(?i)git tag (-d|-f|--delete)" + }, + { + "type": "content", + "pattern": "(?i)(release-source-archive|release workflow)" } ], "samples": { diff --git a/skills/github-release/references/ci-workflow-templates.md b/skills/github-release/references/ci-workflow-templates.md index 5cdbb39..81eb0bf 100644 --- a/skills/github-release/references/ci-workflow-templates.md +++ b/skills/github-release/references/ci-workflow-templates.md @@ -18,49 +18,52 @@ Netresearch maintains org-level reusable workflows: ## GitHub Release Only (No Package Registry) -A project installed straight from its repository (a herdr plugin, for example) publishes nothing to PyPI, npm, Packagist or TER; the release carries the tagged source tree as an archive plus a checksum file. The org `python-release.yml` covers this with `publish-pypi: false`. Pattern from [netresearch/herdr-bg-activity](https://github.com/netresearch/herdr-bg-activity/blob/main/.github/workflows/release.yml): +A project installed straight from its repository — an application, a plugin, anything shipped as its tagged source tree — publishes nothing to PyPI, npm, Packagist or TER. Its release carries the source archive. Use the org reusable `release-source-archive.yml`; it is the whole release workflow: ```yaml +name: Release + on: push: - tags: ['v*.*.*'] + tags: ["v*"] permissions: {} jobs: release: - uses: netresearch/.github/.github/workflows/python-release.yml@main + uses: netresearch/.github/.github/workflows/release-source-archive.yml@main permissions: - contents: write - id-token: write - with: - publish-pypi: false - package-manager: pip - # Fail before building when the tag and the manifest disagree. - check-cmd: >- - python -c 'import os, tomllib; - v = tomllib.load(open("herdr-plugin.toml", "rb"))["version"]; - t = os.environ["GITHUB_REF_NAME"].removeprefix("v"); - assert v == t, f"tag {t} != herdr-plugin.toml version {v}"' - build-cmd: >- - mkdir -p dist && - git archive --format=tar.gz --prefix="my-plugin-${GITHUB_REF_NAME}/" - -o "dist/my-plugin-${GITHUB_REF_NAME}.tar.gz" HEAD && - (cd dist && sha256sum -- *.tar.gz > SHA256SUMS.txt) - release-files: 'dist/*' - - # python-release.yml uploads dist/ as the `dist` artifact but attests nothing. - attest: + contents: write # create the release + id-token: write # Sigstore OIDC identity + attestations: write # file the attestations with GitHub + + # The release is created with GITHUB_TOKEN, so no `release` event starts + # anything afterwards; checks on the published release run as jobs here. + verify: needs: release - uses: netresearch/.github/.github/workflows/attest-release-files.yml@main + uses: netresearch/.github/.github/workflows/verify-release.yml@main permissions: + contents: read + attestations: read id-token: write - attestations: write with: - subject-path: 'dist/*' # same value as release-files + tag: ${{ github.ref_name }} + identity-regexp: '^https://github\.com/netresearch/\.github/\.github/workflows/release-source-archive\.yml@' + signer-workflow: netresearch/.github/.github/workflows/release-source-archive.yml ``` -Provenance is a separate reusable, `attest-release-files.yml`, rather than an input on `python-release.yml`: it needs `attestations: write`, and a called workflow's job permissions are checked at startup, so adding that scope to `python-release.yml` would fail every caller that does not grant it. The reusable workflow is the signer, so verification has to name it; `--repo` alone checks the signer against the caller repository and fails: +On the tag push it checks out the tag (refusing a lightweight one), builds `--source.tar.gz` with `git archive --format=tar | gzip -9 -n`, generates SPDX and CycloneDX SBOMs, attests build provenance and the SBOM, writes `checksums.txt`, signs every file with Cosign, and creates the release last: a draft, every file uploaded with per-asset retries, then published. Because the build runs inside a reusable the project cannot edit, the provenance is **SLSA Build Level 3** — GitHub: "Artifact attestations by itself provides SLSA v1.0 Build Level 2. Reusable workflows can provide isolation between the build process and the calling workflow, to meet SLSA v1.0 Build Level 3." The first caller is [netresearch/timetracker](https://github.com/netresearch/timetracker/blob/main/.github/workflows/release.yml). + +Verification names the reusable as the signer; `--repo` alone checks the signer against the caller repository and fails: + +```bash +gh attestation verify --source.tar.gz --repo / \ + --signer-workflow netresearch/.github/.github/workflows/release-source-archive.yml +``` + +### Level 2 alternative: your own build command + +`python-release.yml` with `publish-pypi: false` runs a `build-cmd` the caller supplies, and `attest-release-files.yml` attests its output. That is the shape [netresearch/herdr-bg-activity](https://github.com/netresearch/herdr-bg-activity/blob/main/.github/workflows/release.yml) uses. The build command lives in the calling repository, so the provenance is **Level 2**, however isolated the signer is. Use it only when the release has to carry something `release-source-archive.yml` does not build; for a plain source archive, switch to the reusable above. `attest-release-files.yml` needs `attestations: write`, which is why it is a separate reusable: a called workflow's job permissions are checked at startup, so adding that scope to `python-release.yml` would fail every caller that does not grant it. Verification names it as the signer: ```bash gh attestation verify --repo / \ diff --git a/skills/github-release/references/release-process.md b/skills/github-release/references/release-process.md index 102ea50..86d540e 100644 --- a/skills/github-release/references/release-process.md +++ b/skills/github-release/references/release-process.md @@ -17,7 +17,7 @@ The complete flow from "create a release" to "release published on GitHub." So the hooks block `gh release create` only without `--verify-tag` (and `--verify-tag=false` counts as without), and `gh release delete` always. `gh release edit` is allowed only for `--notes`/`--notes-file` flags (release description overhaul). Reading `--help` is allowed for any subcommand. -**Which of the two flows applies is a property of the repository, not a preference.** If a workflow creates the release object, that workflow is the only thing that may create it. If none does — because the supply-chain workflows listen on `release: published`, which a `GITHUB_TOKEN`-created release never fires (see `typo3-ter-publishing.md`) — then a human credential must create it, and `gh release create … --verify-tag` is that step. Establish which case you are in by reading the workflows, not by trying the command: +**Which of the two flows applies is a property of the repository, not a preference.** If a workflow creates the release object, that workflow is the only thing that may create it. If none does — because the supply-chain workflows listen on `release: published`, which a `GITHUB_TOKEN`-created release never fires (see `typo3-ter-publishing.md`) — then a human credential must create it, and `gh release create … --verify-tag` is that step. That second case is a gap to close rather than a flow to keep: the build then runs in the repository, which is SLSA Build Level 2 at best. Add a tag-push release workflow that calls the org reusable for the artefact (`release-source-archive.yml`, `release-go-app.yml`, `release-typo3-extension.yml` — see `ci-workflow-templates.md`), and the first case applies from then on. Establish which case you are in by reading the workflows, not by trying the command: ```bash grep -rln "on:" .github/workflows | xargs grep -ln "release:" ; grep -rn "gh release create\|softprops/action-gh-release\|create-release" .github/workflows From 823b7a1d85f276cd1a9903d7152a4ec5791a4ca2 Mon Sep 17 00:00:00 2001 From: Sebastian Mendel Date: Mon, 21 Sep 2026 12:39:17 +0200 Subject: [PATCH 2/2] test(evals): give eval 41's new assertion a passing sample that meets it validate-evals.sh failed: "eval[40] (id=41): assertion[4] does not match its own passing sample". The passing sample now names the lasting fix, the assertion matches `release-source-archive` only (the bare phrase "release workflow" also occurs in a failing sample), and the expectation is listed. Assisted-by: claude-code:claude-opus-5 Agent-Session: https://claude.ai/code/session_01Cn9YYfLWrmSH1t13kGqe4K Agent-Host: 0493f0 Signed-off-by: Sebastian Mendel --- skills/github-release/evals/evals.json | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/skills/github-release/evals/evals.json b/skills/github-release/evals/evals.json index 6aeeb54..fa88ab1 100644 --- a/skills/github-release/evals/evals.json +++ b/skills/github-release/evals/evals.json @@ -1117,7 +1117,8 @@ "Verifies the release: published consumers afterwards", "Does NOT: run gh release create without --verify-tag", "Does NOT: delete or move the tag", - "Does NOT: refuse on the grounds that gh release create is always forbidden" + "Does NOT: refuse on the grounds that gh release create is always forbidden", + "Names the lasting fix: a tag-push release workflow calling release-source-archive.yml" ], "assertions": [ { @@ -1138,11 +1139,11 @@ }, { "type": "content", - "pattern": "(?i)(release-source-archive|release workflow)" + "pattern": "(?i)release-source-archive" } ], "samples": { - "passing": "Nothing in .github/workflows creates the release object, and a release created with GITHUB_TOKEN would not fire release: published, so the provenance and SBOM jobs would never start. Publish it yourself against the tag that is already pushed: gh release create v6.4.0 --title \"v6.4.0\" --verify-tag --notes-file notes.md. --verify-tag makes gh abort rather than create the tag. Then confirm the downstream run landed: gh attestation verify timetracker-v6.4.0-source.tar.gz --repo netresearch/timetracker.", + "passing": "Nothing in .github/workflows creates the release object, and a release created with GITHUB_TOKEN would not fire release: published, so the provenance and SBOM jobs would never start. Publish it yourself against the tag that is already pushed: gh release create v6.4.0 --title \"v6.4.0\" --verify-tag --notes-file notes.md. --verify-tag makes gh abort rather than create the tag. Then confirm the downstream run landed: gh attestation verify timetracker-v6.4.0-source.tar.gz --repo netresearch/timetracker. For the next release, add a tag-push release workflow that calls netresearch/.github's release-source-archive.yml: it builds the archive inside the reusable (SLSA Build Level 3) and creates the release itself, so none needs creating by hand.", "failing": [ "Run gh release create v6.4.0 --title \"v6.4.0\" --notes-file notes.md and the release workflow will take it from there.", "First clear the tag so the release can recreate it: git tag -d v6.4.0 and push the deletion, then publish with --verify-tag and check the provenance once release: published has fired."