Skip to content

ci: automate release candidate staging #3983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chaokunyang wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ci: automate release candidate staging #3983

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
281 changes: 146 additions & 135 deletions .agents/skills/fory-release/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,56 +1,36 @@
---
name: fory-release
description: Prepare an Apache Fory release candidate from a clean release branch, including the version bump, RC tag, JVM staging, ASF source artifacts, SVN upload, and vote email. Use when creating or rerunning a Fory release candidate.
description: Stage or verify an Apache Fory release candidate. Use the GitHub Actions release workflow by default for ATR source staging and Nexus JVM staging; use the retained local manual workflow only when the user explicitly requests manual publishing.
---

# Apache Fory Release

Use the repository release script for the release work. Do not manually reproduce its version-bump, JVM-publication, or source-build logic, and do not add unrelated test runs.
Use `.github/workflows/stage-release-candidate.yml` for source and JVM staging
unless the user explicitly asks for a manual release. The workflow uses
`ci/release.py`; do not reproduce its source-build, JVM-publication, Nexus
closure, or artifact-verification logic in shell commands.

Do not add unrelated test runs. An invalid candidate gets a higher RC number;
never move or reuse an RC tag.

## Required Inputs

Collect these values before starting:
Collect these values:

- `release_version`: final version without `v` or an RC suffix, such as `1.7.0`.
- `rc`: RC suffix, such as `rc3`.
- `previous_version`: previous release tag version, such as `1.6.1`.
- Release discussion URL, if already known. If it is not supplied, find the
exact release thread in the Fory development-list archive as described below.
- Release discussion URL, if already known.

Load release-manager details from `.local/fory-release.env`. If it does not exist, ask for the following values once, create the ignored local file, and continue. Never commit this file.
Derive the release values:

```bash
FORY_RELEASE_MANAGER_NAME="..."
FORY_RELEASE_APACHE_EMAIL="..."
FORY_RELEASE_GPG_FINGERPRINT="..."
FORY_DIST_DEV_WC="..."
```

Load the cached values and derive the release values:

```bash
repo_root="$(git rev-parse --show-toplevel)"
release_config="$repo_root/.local/fory-release.env"
test -f "$release_config"
. "$release_config"

release_branch="releases-${release_version}"
rc_tag="v${release_version}-${rc}"
dist_version="${release_version}"
release_manager_name="${FORY_RELEASE_MANAGER_NAME:?missing release manager name}"
apache_email="${FORY_RELEASE_APACHE_EMAIL:?missing Apache email}"
gpg_fingerprint="${FORY_RELEASE_GPG_FINGERPRINT:?missing GPG fingerprint}"
svn_wc="${FORY_DIST_DEV_WC:?missing ASF Subversion working-copy path}"
release_candidate_url="https://release-test.apache.org/vote/fory/${release_version}"
```

Use the same `dist_version` in Subversion and the vote email.

### Find the release discussion

If the release discussion URL was not supplied, search the
[Fory development-list archive](https://lists.apache.org/list.html?dev@fory.apache.org)
for the exact release version and a `[DISCUSS]` subject. Use the deterministic
helper, which accepts exactly one root discussion thread:
If the discussion URL was not supplied, find the exact root discussion thread:

```bash
discussion_url="$(
Expand All @@ -59,98 +39,78 @@ discussion_url="$(
)"
```

Open the resulting URL and verify that its subject and body discuss the exact
`release_version`. If the automated search finds zero or multiple roots, use
the archive UI to search the same exact version and `[DISCUSS]`; ask the release
manager only if the result remains absent or ambiguous. When a URL is supplied,
open and verify it instead of assuming it matches this release.
Open the URL and verify that its subject and body discuss the exact version. If
the helper finds zero or multiple roots, search the Fory development-list
archive for the exact version and a `[DISCUSS]` subject. Ask the release manager
only if the result remains absent or ambiguous.

## Release Workflow
## Default CI Workflow

### 1. Create a clean release branch
### 1. Create the release branch and commit

Run from the repository root. Clean means no staged, modified, or untracked files.
Run from the repository root. Clean means no staged, modified, or untracked
files.

```bash
test -z "$(git status --porcelain)"
test "$(git remote get-url apache)" = "git@github.com:apache/fory.git"
git fetch apache main --tags
git switch -c "$release_branch" apache/main
test -z "$(git status --porcelain)"
```

Stop if the branch already exists or either cleanliness check fails. Do not remove or hide user files to make the check pass.

### 2. Bump the version

```bash
python3 ci/release.py bump_version -version "$release_version" -l all
git diff --check
git status --short
```

Review the version diff. Use this command directly; do not substitute another version-bump workflow.

### 3. Commit the release version

```bash
git add -u
git commit -m "prepare release for ${release_version}"
test -z "$(git status --porcelain)"
release_commit="$(git rev-parse HEAD)"
```

Stage only the version changes produced by the release script.

### 4. Create and push the RC tag
Stop if the branch already exists or the tree is not clean. Review the version
diff and stage only the files changed by `bump_version`.

Confirm that neither the local nor remote tag already exists. An invalid RC gets a new RC number; never move or reuse an RC tag.
### 2. Create and push the RC tag

```bash
test -z "$(git tag --list "$rc_tag")"
test -z "$(git ls-remote --tags apache "refs/tags/${rc_tag}")"
test "$(git rev-parse HEAD)" = "$release_commit"
git tag "$rc_tag" && git push apache "$rc_tag"
git tag "$rc_tag"
git push apache "$rc_tag"
test "$(git rev-parse "${rc_tag}^{commit}")" = "$release_commit"
```

The tag starts the ecosystem package-release workflows. Do not wait for them
here: start JVM publication immediately so the remote workflows and JVM staging
run in parallel. Once the tag has been pushed, any JVM or later release failure
invalidates this RC and requires a higher RC number; never move or reuse the
tag.
The tag starts the ecosystem package workflows. Once pushed, any staging or
verification failure invalidates this RC.

### 5. Publish JVM artifacts
### 3. Stage source and JVM artifacts in CI

Dispatch the workflow on the RC tag with both default jobs enabled:

```bash
python3 ci/release.py publish_jvm
gh workflow run stage-release-candidate.yml \
--repo apache/fory \
--ref "$rc_tag" \
-f source=true \
-f jvm=true
```

The command publishes from a temporary worktree at the committed `HEAD` and
removes that worktree afterward. Record the distinct Java/Kotlin and Scala
Nexus staging repository IDs from the output:
Find the newly created `workflow_dispatch` run for this exact tag, record its
run ID and URL, and wait for it with `gh run watch --exit-status`. Do not select
a run only by commit SHA because main-branch runs can share the same commit.

```bash
java_kotlin_staging_id="orgapachefory-..."
scala_staging_id="orgapachefory-..."
test -n "$java_kotlin_staging_id"
test -n "$scala_staging_id"
test "$java_kotlin_staging_id" != "$scala_staging_id"
```
The source job builds, signs, and checksum-verifies the existing source-release
archive before uploading `dist/` to Apache Trusted Release (ATR) through OIDC.
It does not use SVN credentials. The JVM job publishes from the tag commit,
identifies only the Nexus repositories created by that run, closes both in one
request, and verifies representative artifacts through anonymous downloads.

After recording both IDs, read and follow
[Nexus staging closure](references/nexus-staging.md). It contains the
credential rules, authenticated state checks, `/bulk/close` request, HTTP 201
gate, close polling, failure inspection, and anonymous artifact checks. Do not
close any repository ID that was not recorded from this publication. Keep both
repositories closed during the vote; do not promote them until the vote passes.
From the successful run log or job summary, record the distinct
`java_kotlin_staging_id` and `scala_staging_id`. Open the ATR candidate URL and
both closed Nexus repository URLs before drafting the vote.

### 6. Check the tag-triggered workflows
### 4. Check tag-triggered workflows

After JVM publication and Nexus closure, inspect the workflows that have been
running since the tag was pushed. Filter by the tag rather than only by commit
SHA so main-branch runs at the same commit are not mixed into the result. By
default, wait for every tag-triggered run and require successful conclusions:
Require all workflows triggered by the RC tag to succeed:

```bash
python3 .agents/skills/fory-release/scripts/check_tag_workflows.py \
Expand All @@ -160,70 +120,121 @@ python3 .agents/skills/fory-release/scripts/check_tag_workflows.py \
--watch
```

The helper re-queries by tag after waiting to catch later-created runs. If the
release manager explicitly waives workflow monitoring for a particular RC, run
the same command with `--allow-incomplete` instead of `--watch`, and record the
snapshot IDs, states, and reason. Do not cancel the remote workflows or report
incomplete runs as successful.
If the release manager explicitly waives workflow monitoring for this RC, use
`--allow-incomplete` instead of `--watch` and record the run states and reason.
Do not report incomplete workflows as successful.

### 7. Build the ASF source release
### 5. Draft the vote email

Start from the clean release branch. The build temporarily commits release-archive changes and resets them, so verify that it restores the original commit and clean tree.
Load release-manager identity from `.local/fory-release.env` only when drafting
the email. Never commit this ignored file.

```bash
test -z "$(git status --porcelain)"
before_build="$(git rev-parse HEAD)"
python3 ci/release.py build -v "$release_version"
test "$(git rev-parse HEAD)" = "$before_build"
test -z "$(git status --porcelain)"
test -f "dist/apache-fory-${release_version}-src.tar.gz"
test -f "dist/apache-fory-${release_version}-src.tar.gz.asc"
test -f "dist/apache-fory-${release_version}-src.tar.gz.sha512"
release_config="$(git rev-parse --show-toplevel)/.local/fory-release.env"
test -f "$release_config"
. "$release_config"
release_manager_name="${FORY_RELEASE_MANAGER_NAME:?missing release manager name}"
apache_email="${FORY_RELEASE_APACHE_EMAIL:?missing Apache email}"
gpg_fingerprint="${FORY_RELEASE_GPG_FINGERPRINT:?missing GPG fingerprint}"
```

The build command verifies the generated PGP signature and SHA-512 checksum.
If the file does not exist, ask once for the three missing values and create
the ignored local file. Do not store the GitHub or Nexus secrets in it.

### 8. Commit the source release to ASF Subversion
Read [the vote email template](assets/vote-email.txt) and produce a complete,
copyable email. Fill every placeholder from verified output, confirm that no
`${...}` placeholder remains, use an explicit UTC deadline at least 72 hours
after sending, and do not send the email unless requested.

## Explicit Manual Workflow

Only when the user explicitly requests manual publishing, read and follow
[the manual release workflow](references/manual-release.md). Do not fall back
to it automatically after a CI failure. A failed CI candidate still requires a
higher RC tag.

Use a clean, updated working copy of the ASF development distribution repository.
## Verification-Only Requests

Verification belongs to this skill; do not create or invoke a separate
release-verification workflow. For an existing candidate, inspect the exact
workflow run and tag, then download and verify the remote source artifacts:

```bash
test -d "$svn_wc/.svn" || svn checkout https://dist.apache.org/repos/dist/dev/fory "$svn_wc"
svn update "$svn_wc"
mkdir -p "$svn_wc/$dist_version"
cp dist/* "$svn_wc/$dist_version/"
svn add --force "$svn_wc/$dist_version"
svn status "$svn_wc/$dist_version"
svn commit "$svn_wc/$dist_version" -m "Prepare Apache Fory ${rc_tag}"
test -z "$(svn status "$svn_wc/$dist_version")"
svn log -l 1 "$svn_wc/$dist_version"
svn ls "https://dist.apache.org/repos/dist/dev/fory/${dist_version}/"
set -euo pipefail
: "${release_version:?missing release version}"
: "${release_candidate_url:?missing release candidate URL}"
: "${gpg_fingerprint:?missing expected GPG fingerprint}"
release_candidate_url="${release_candidate_url%/}"

archive="apache-fory-${release_version}-src.tar.gz"
verify_root="$(mktemp -d)"
verify_dist="$verify_root/dist"
verify_gnupg="$verify_root/gnupg"
mkdir -m 700 "$verify_gnupg"
mkdir "$verify_dist"
trap 'rm -rf "$verify_root"' EXIT

curl -fL "$release_candidate_url/$archive" -o "$verify_dist/$archive"
curl -fL "$release_candidate_url/$archive.asc" -o "$verify_dist/$archive.asc"
curl -fL "$release_candidate_url/$archive.sha512" \
-o "$verify_dist/$archive.sha512"
curl -fL https://downloads.apache.org/fory/KEYS -o "$verify_root/KEYS"
GNUPGHOME="$verify_gnupg" gpg --batch --import "$verify_root/KEYS"

(
cd "$verify_dist"
if command -v sha512sum >/dev/null; then
sha512sum --check "$archive.sha512"
else
shasum -a 512 -c "$archive.sha512"
fi
GNUPGHOME="$verify_gnupg" gpg --batch --verify "$archive.asc" "$archive"
)

# VALIDSIG may append the primary-key fingerprint for subkey signatures.
signer_primary_fingerprint="$(
GNUPGHOME="$verify_gnupg" gpg --batch --status-fd 1 \
--verify "$verify_dist/$archive.asc" "$verify_dist/$archive" 2>/dev/null |
awk '$2 == "VALIDSIG" { print (NF >= 12 ? $12 : $3); exit }'
)"
expected_fingerprint="$(
printf '%s' "$gpg_fingerprint" | tr -d ' ' | tr '[:lower:]' '[:upper:]'
)"
test "$signer_primary_fingerprint" = "$expected_fingerprint"
```

Inspect `svn status` before committing. The upload is complete only after `svn commit` returns a revision and the remote `svn ls` shows the three release files; local `A` status alone is not an upload.
Use the staging IDs recorded by the exact CI run and the Nexus credentials from
the release manager's secret store to repeat the existing read-only repository
state and anonymous download checks:

### 9. Draft the vote email

Read [the vote email template](assets/vote-email.txt) and produce a complete,
copyable email. Replace every placeholder from verified output, confirm that no
`${...}` placeholder remains, use an explicit UTC deadline at least 72 hours
after sending, and do not send the email unless requested.
```bash
: "${java_kotlin_staging_id:?missing Java/Kotlin staging ID}"
: "${scala_staging_id:?missing Scala staging ID}"
: "${rc_tag:?missing release-candidate tag}"
: "${NEXUS_USERNAME:?missing Nexus username}"
: "${NEXUS_PASSWORD:?missing Nexus password}"
python3 ci/release.py close_jvm_staging \
-v "$release_version" \
--rc-tag "$rc_tag" \
--java-kotlin-id "$java_kotlin_staging_id" \
--scala-id "$scala_staging_id" \
--verify-only
```

Before sending, verify the tag and commit, all URLs, both closed Maven staging repositories, the remote Subversion files, PGP fingerprint, and UTC deadline against the actual release outputs.
Finally run the tag-workflow checker from the default workflow without
publishing or changing remote state. Report verification only when the remote
checksum, signature, signer fingerprint, Nexus state/download checks, and tag
workflows all pass.

## Stop Conditions

Before pushing the tag, stop if the Git tree is dirty, a command fails, the RC
tag already exists, or the tag target would differ from the release commit.
After pushing the immutable tag, any failed JVM publication, workflow,
artifact verification, or Subversion publication invalidates that candidate;
fix the issue and create a higher RC instead of moving or reusing the tag.
Before sending the vote, require both staging repositories to be closed and
public, the Subversion commit to be remotely visible, and the tag workflows to
be successful unless the release manager explicitly waived monitoring.
Before pushing the tag, stop if the tree is dirty, the tag exists, a command
fails, or the tag target differs from the release commit. After pushing it,
stop on any CI, ATR, Nexus, signature, checksum, artifact, or tag-workflow
failure and create a higher RC after the cause is fixed. Do not promote Nexus
repositories or send the vote until the vote itself has passed.

## References

- [Apache Fory release guide](https://fory.apache.org/docs/community/how_to_release)
- [Fory development-list archive](https://lists.apache.org/list.html?dev@fory.apache.org)
- [Sonatype Nexus 2 staging REST example](https://support.sonatype.com/hc/en-us/articles/213465448-Automatically-dropping-old-staging-repositories)
- [Apache Pekko CI release workflow](https://github.com/apache/pekko/blob/main/.github/workflows/stage-release-candidate.yml)
2 changes: 1 addition & 1 deletion .agents/skills/fory-release/assets/vote-email.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The change list since Apache Fory ${previous_version}:
https://github.com/apache/fory/compare/v${previous_version}...${rc_tag}

The release candidate artifacts:
https://dist.apache.org/repos/dist/dev/fory/${dist_version}/
${release_candidate_url}

The Maven staging repositories:
Java and Kotlin:
Expand Down
Loading
Loading