Skip to content

fix(release): wait for npm before changeset publish, and decide the EQL assets from npm and the tags - #1026

Merged
auxesis merged 3 commits into
mainfrom
fix/release-publish-race
Oct 3, 2026
Merged

auxesis merged 3 commits into
mainfrom
fix/release-publish-race

Conversation

@auxesis

@auxesis auxesis commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

This pull request fixes the two faults in release.yml that Linear issue CIP-4276 describes. It also fixes a third fault that would have skipped the EQL assets on most releases. Once it merges, the next push to main builds the missing EQL 3.0.6 tag, GitHub release and image.

The release job now waits for npm before changeset publish

changeset publish is the Changesets command that publishes every workspace package whose version npm does not list yet. The publish-ffi and publish-auth jobs publish the native packages first, with npm publish. npm accepts a publish a minute or more before its package document lists the new version. The package document is npm's record of a package, with its list of versions.

So changeset publish saw some native versions as unpublished and published them again. It uses restricted access, so npm refused each one with E402 Payment Required, and the release job failed. This happened in Release JS run 36978691809 for protect-ffi 0.33.0. It happened again in Release JS run 37071820363 for @cipherstash/auth, auth-darwin-x64 and auth-win32-x64-msvc 0.44.1. Both runs were on 2 October 2026.

Now both publish jobs export a published output, which holds the same name@version list that they write to published.txt. A new step in the release job runs scripts/wait-for-npm-versions.mjs before changesets/action. The script asks npm view <name> versions for each package until npm lists every version, for up to 15 minutes. That is the same question, through the same npm, that changeset publish asks next.

A skipped publish job exports an empty string, so the script returns at once for an ordinary JavaScript release. A registry error during the wait is retried until the time limit. If the wait times out, the job fails before changeset publish, so nothing is published, and a re-run waits again.

The EQL asset jobs now ask npm and the tags

Three jobs build the EQL assets: the eql-3.0.6 tag and GitHub release with the SQL bundle, the docs bundle, and the postgres-eql image. They ran only when the release job set eql_published. A step after changeset publish set it, so a failed publish never set it, even when the publish had already reached npm. A re-run found nothing left to publish, so it did not set it either.

A new job, eql-assets, now runs scripts/eql-release-assets.mjs after the release job, whatever its result. The script reports that EQL needs its assets when npm has the tree's EQL version and the eql-<version> tag does not exist. It also counts @cipherstash/eql in this run's publishedPackages, because npm can take minutes to list a version that it just accepted. The release job exports publishedPackages straight from the Changesets step, so the output survives a failed publish.

The script builds the assets at the commit that the @cipherstash/eql@<version> tag names. changesets/action pushes that tag at the commit it published from. So a run that repairs an older release builds the source that npm shipped, not whatever main holds now. If npm has a version and neither tag nor this run says which commit published it, the script fails and asks for a manual build.

The script also reports nothing needed while EQL is a frozen publisher in scripts/release-gate.mjs. Without that check, it would fail every push to main, because a frozen EQL publishes from another repository and leaves no Changesets tag here.

The EQL jobs no longer depend on every earlier job succeeding

GitHub adds a hidden success() check to a job condition that calls no status function. That check is false when any job anywhere up the needs: chain was skipped or failed, as actions/runner issue 2205 records. publish-ffi and publish-auth are ancestors of the EQL jobs, and most releases skip at least one of them.

The new tests include a job-graph model: it decides which release.yml jobs run, given the results of the jobs before them. I ran it over main's release.yml in three cases. An EQL release with no FFI or auth release in the same run skipped all three EQL asset jobs. Only a run that also released both FFI and auth would have built them.

The four EQL jobs now start with !cancelled() and name the results they need, such as needs.eql-sql.result == 'success'. I chose !cancelled() over always() so that a person who cancels the run still stops these jobs.

The next push to main builds the EQL 3.0.6 assets

Merging this pull request is a push to main, so it starts Release JS. Unless another push lands first, that run does the following.

  1. The gate job finds nothing unpublished, so publish-ffi and publish-auth are skipped. The wait step gets two empty lists and returns at once.
  2. The release job runs changesets/action. With no changesets on main, it publishes nothing.
  3. The eql-assets job reads version 3.0.6 from packages/eql/packages/eql/package.json. It finds no eql-3.0.6 tag, finds 3.0.6 on npm, and finds @cipherstash/eql@3.0.6 at 23e9af3f. That commit is the merge of Version Packages pull request Version Packages #938. It reports needed=true and ref=23e9af3f.
  4. The eql-sql job checks out 23e9af3f and runs mise run --force build --version 3.0.6. It creates the tag eql-3.0.6 at 23e9af3f and a GitHub release that is not a prerelease. The release holds cipherstash-encrypt.sql and cipherstash-encrypt-uninstall.sql.
  5. The eql-docs job builds the docs at 23e9af3f and attaches the .zip and .tar.gz bundles to that release.
  6. The eql-image job dispatches release-postgres-eql-image.yml against the tag eql-3.0.6, with update_floating_tags=true. That workflow builds images for PostgreSQL 14, 15, 16 and 17, tagged <pg>-3.0.6 and <pg>. It then points latest and 3.0.6 at the PostgreSQL 17 image. Today the newest image tags in the registry are for 3.0.5.

I ran scripts/eql-release-assets.mjs read-only against npm and GitHub on 3 October 2026. It printed @cipherstash/eql@3.0.6 is on npm and eql-3.0.6 does not exist; building at 23e9af3f8e28c6d0495fdafcff096f58cbf0f4f7.

Between 23e9af3f and main, the three EQL workflows are identical, and packages/eql differs only in its AGENTS.md. The tag eql-bindings-v3.0.6, which release-plz made for the crate, also points at 23e9af3f. So the new tag, the npm package and the crate all name the same commit.

One risk is outside this change. AGENTS.md lists write access to ghcr.io/cipherstash/postgres-eql from this repository as an open cutover step, because the package is still linked to cipherstash/encrypt-query-language. If that access is missing, the image workflow fails when it pushes. The SQL and docs release would still exist, and a person can dispatch the image workflow again after granting access.

Tests cover each case the issue names

  • A failed changeset publish: the job-graph model runs eql-sql, eql-docs and eql-image when release fails. The script owes the assets when this run published EQL and npm does not list it yet.
  • An already-published EQL version with no tag: the script owes the assets at the commit of the Changesets tag. A process test runs the script with npm and gh replaced on PATH.
  • The wait for both the FFI and the auth packages: the wait polls each package until npm lists it, and stops asking once it does.
  • A skipped publish job: an empty list asks npm nothing, and a process test waits for the auth packages with FFI_PUBLISHED empty.

scripts/__tests__/lib/expressions.mjs now holds the GitHub expression evaluator that workflow-dispatch-job-conditions.test.mjs used. The new job-graph tests use the same evaluator, and it now models cancelled(). AGENTS.md gains a short note on each fix.

Each new test fails when the code it covers breaks

A mutation check breaks the code on purpose and confirms that a test fails. I ran 14 mutations, and each one failed at least one test. I restored the code after each one.

Mutation Result
The wait reports every version as listed 6 tests fail
The wait refuses an empty list 5 tests fail
The wait matches a version by substring 1 test fails
The wait step is removed from release 1 test fails
publish-auth exports no list 1 test fails
The script ignores this run's publish 4 tests fail
The script always builds at the run's commit 3 tests fail
The script skips the eql-<version> tag check 2 tests fail
The script ignores the frozen-publisher switch 1 test fails
eql-sql uses the hidden success() 4 tests fail
eql-assets uses the hidden success() 5 tests fail
eql-assets uses always() 1 test fails
release passes publishedPackages through a later step 1 test fails
The evaluator ignores a cancelled run 2 tests fail

Checks

  • pnpm test:scripts: 64 files and 1,193 tests pass, with 1 skipped as on main.
  • pnpm run code:check: no errors, and no new warnings.
  • actionlint 1.7.7 with shellcheck 0.11.0 on the release workflows: no findings.
  • lint:workflow-cache, lint:runners and lint:package-paths: all pass.

Linked issues

This PR fixed #1035.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PaY5xYydZUWhv8Nex9Sw8a

@auxesis
auxesis requested a review from a team as a code owner October 2, 2026 23:40
@changeset-bot

changeset-bot Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: d4a9555

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@freshtonic freshtonic left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed cb58a7b9...d7a13ff8. The change is correct, and I approve it.

What I checked

  • The publishedPackages claim is true. In changesets/action@v1.9.0, runPublish calls getExecOutput with ignoreReturnCode: true. It parses the New tag: lines, and index.ts sets published and publishedPackages before it examines the exit code. Thus a failed changeset publish still exports the packages that it published. Also, createGithubReleases is on by default, so the action pushes the @cipherstash/eql@<version> tag that eql-assets reads.
  • The wait asks the same question as changeset publish. npmVersions runs npm view <name> versions --json. It returns null on E404, and the wait retries on a 404. A re-run also exports the versions that were published before, but npm lists them at once, so they do not cause a delay.
  • No two runs can decide at the same time. concurrency: ${{ github.workflow }}-${{ github.ref }} queues the runs and does not cancel them. Thus two pushes cannot both find needed=true and both make eql-<version>.
  • latest cannot move back to an older version. eql-assets reads only the tree's EQL version, which is the newest final version on main. A repair cannot dispatch update_floating_tags=true for an older version.
  • The job graph is correct. All four EQL jobs use !cancelled() and name the results that they need. eql-docs and eql-image also require eql-sql and eql-docs to succeed. The job-graph model in the tests copies the transitive success() rule from actions/runner#2205. The "cancelled during release" case is a good test of the choice of !cancelled() instead of always().
  • Inputs to the tag and dispatch are safe. VERSION is validated before it goes into a tag name or a -f argument. ref comes from the GitHub API. eql-assets inherits contents: read and requests no id-token.
  • Changeset and skills are not necessary. The change is to repository tooling only. AGENTS.md has the two new notes.

A gap to know about (not a blocker)

The repair key is "the eql-<version> tag does not exist". _build-eql-sql.yml makes the tag and the release in one softprops/action-gh-release step. If eql-sql succeeds and eql-docs or eql-image then fails, the tag exists. No later push builds the missing docs or image. A re-run of the failed jobs in that run still works, because the eql-assets outputs stay. A later push is not a recovery path for those two jobs. This is a smaller window than the fault that this PR fixes. A short sentence in the eql-assets comment or in the AGENTS.md note can tell the next person to re-run the failed jobs, and not to wait for the next push. Write it in this PR or in a follow-up.

The PR body also identifies the open GHCR write-access risk for eql-image. I agree that it is outside this change.

auxesis and others added 3 commits October 3, 2026 10:20
Move the GitHub Actions expression evaluator out of
workflow-dispatch-job-conditions.test.mjs into lib/expressions.mjs, so a
second guard can walk the release job graph with the same reading of a
condition rather than a copy of it.

Model `cancelled()` from a run state the caller passes, false by default.
The release EQL jobs are about to be written `!cancelled() && ...`, and the
evaluator must not refuse them. `success()` and `failure()` still throw:
they depend on the whole needs chain, which these contexts do not carry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PaY5xYydZUWhv8Nex9Sw8a
…et publish

publish-ffi and publish-auth publish their seven tarballs each with
`npm publish`. The release job then runs `changeset publish`, which
publishes every version `npm info` does not list. npm lists a publish
minutes after accepting it, so changesets published some native packages
a second time, from the workspace and with restricted access, and npm
refused them with E402. That failed the release job for protect-ffi
0.33.0 (run 36978691809) and @cipherstash/auth 0.44.1 (run 37071820363).

Both publish jobs now export the name@version list they write to
published.txt. A new step in the release job, before changesets/action,
polls `npm view <name> versions` until every listed version appears, for
up to 15 minutes. A skipped publish job exports an empty list, so an
ordinary JS release does not wait. A timeout fails the job before
`changeset publish`, so nothing is published, and a re-run waits again.

Refs: CIP-4276

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PaY5xYydZUWhv8Nex9Sw8a
The SQL, docs and image jobs ran only when the release job's
`eql_published` output was true. A step after `changeset publish` set it,
so when the publish failed the step never ran, even though EQL had been
published, and a re-run found nothing to publish. That is why EQL 3.0.6
has no eql-3.0.6 tag, GitHub release or image.

A new eql-assets job runs scripts/eql-release-assets.mjs after the
release job, whatever its result. The assets are owed when npm carries
the tree's EQL version and the eql-<version> tag does not exist. This
run's publishedPackages also counts, because npm lists a new version
minutes after accepting it; the release job now exports it straight from
the changesets step. The assets are built at the commit the
@cipherstash/eql@<version> tag names, which is where npm's tarball came
from, so a later run repairs an old release from that release's source.

The four EQL jobs now use `!cancelled()` instead of the implicit
`success()`. That is false when any job up the needs chain was skipped
(actions/runner#2205), and publish-ffi and publish-auth are skipped on
most releases, so an EQL release without an FFI and an auth release in
the same run would also have skipped every EQL asset job.

Refs: CIP-4276

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PaY5xYydZUWhv8Nex9Sw8a
@auxesis
auxesis force-pushed the fix/release-publish-race branch from d7a13ff to d4a9555 Compare October 3, 2026 00:22
@auxesis
auxesis merged commit af21446 into main Oct 3, 2026
28 checks passed
@auxesis
auxesis deleted the fix/release-publish-race branch October 3, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build the missing EQL 3.0.6 GitHub release and image, and fix the release.yml race

2 participants