Add SDK canary pipeline#1950
Open
MackinnonBuck wants to merge 23 commits into
Open
Conversation
Installs an explicit @github/copilot runtime version, builds the Node SDK, and runs the Node e2e suite against it as a runtime<->SDK compat gate. No publishing. A temporary branch-push trigger validates the public path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Publishes an SDK canary to the internal copilot-canary-test feed, pinned to the exact runtime version just tested. Feed-only via publishConfig.registry, explicit --registry, and a pre-publish assertion. Runs only when all e2e matrix jobs are green (needs: [resolve, test]). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds a workflow_dispatch force_publish input that lets a human publish the SDK canary despite a non-passing e2e gate (for a known flake). A loud ::warning:: is logged whenever publish proceeds without a green gate. The bypass only skips the e2e signal; build + feed-only + read-back guards still apply. A temporary push-event bypass pairs with the temporary push trigger for branch validation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Warn step now uses always() so the bypass audit is never skipped by prior-step status; it fires whenever publish proceeds with test != success. - Add greppable 'TEMP: remove at finalize' markers on the temporary push trigger and the coupled publish.if event==push bypass clause. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Warn step runs first (before checkout), but the job default working-directory ./nodejs does not exist yet, so bash failed to cd into it and the step errored — which skipped the whole publish job the one time the bypass actually fired. Pin the step to the workspace root. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Remove the temporary push: trigger (workflow_dispatch is now the only trigger). - Remove the event==push clause from publish.if; force_publish (workflow_dispatch only) is the sole bypass. - Drop the now-dead push-fallback normalization from the resolve job. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Compute the canary base as public latest + patch bump (e.g. 1.0.7-canary.<run>.g<sha>) so canaries correlate with public releases: they sort above current public latest and below the eventual real release of that next patch. Public latest is read read-only from public npm (never the feed) with a resilient 0.0.0 fallback + warning when it can't be resolved. Runtime correlation stays carried by the pinned dependencies.@github/copilot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ses" This reverts commit b289233.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow that acts as a manual-dispatch canary pipeline to validate a specific @github/copilot runtime version against the Node SDK’s full e2e suite across OSes, and (optionally) publish a feed-only SDK canary pinned to that tested runtime.
Changes:
- Introduces
resolve+ OS-matrixtestjobs to normalize inputs, install/verify an explicit runtime version, and run Node e2e tests. - Adds a guarded
publishjob to build and publish an internal-feed-only SDK canary pinned to the tested runtime, with read-back verification. - Implements a
force_publishbypass path for workflow_dispatch runs, with an audit warning when publish proceeds without a green e2e gate.
Show a summary per file
| File | Description |
|---|---|
| .github/workflows/sdk-canary.yml | New canary workflow to gate runtime↔SDK compatibility via Node e2e tests and publish an internal-feed canary pinned to the tested runtime. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 3
- Review effort level: Low
…sses" This reverts commit 925844e.
…ion bypasses"" This reverts commit 4380414.
workflow_dispatch now takes mode: publish (default, tests must pass), publish-force (publish even on a failed gate), or tests-only (run the gate, never publish). The resolve job normalizes a PUBLISH_MODE output (only a human dispatch can pick a non-default mode; automated triggers are always 'publish'), publish.if gates on it, and the audit warning fires specifically for publish-force on a non-green gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Batch of review-response changes to .github/workflows/sdk-canary.yml: - Rename workflow to "SDK Canary Test/Publish". - Rename runtime_source value "feed" -> "internal" (option, guards, comment). - Rename test job to "E2E tests (<os>)". - Hoist the feed URL and ADO resource id into workflow-level env (FEED_URL, ADO_RESOURCE) as single sources of truth; derive .npmrc auth scopes from FEED_URL. - Add a per-ref concurrency group (cancel-in-progress: false) to avoid overlapping runs racing the feed publish. - Reuse scripts/get-version.js (current) for the public-latest lookup to stay consistent with publish.yml, keeping strict patch+1 semantics. - Tighten the semver validation regex to disallow underscores in the prerelease segment (invalid per SemVer/npm) in both the runtime-version and computed-SDK-version checks. - Verify the matched platform optional dependency version also matches the requested runtime version (not just that the package exists). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Switch the canary feed from copilot-canary-test to the final org-scoped copilot-canary feed. The feed URL lives in a single place (env.FEED_URL); the test-job .npmrc, publish-job auth .npmrc, publishConfig assertion, and read-back all derive from it, so this one change repoints every step. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Now that the pipeline is proven green against the copilot-canary feed, strip the on-branch validation scaffolding so the workflow is workflow_dispatch-only: - Remove the temporary push: trigger scoped to the feature branch. - Remove the push-event fallback case in the resolve/normalize step. - Remove the github.event_name == 'push' clause from publish.if. Version scheme, mode enum, and the copilot-canary feed are all retained. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Address PR review feedback: - Tighten the SemVer prerelease regex to require non-empty dot-separated identifiers ((-[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?), so malformed values like 1.0.7-canary..gsha are rejected at validation instead of later at npm. Applied to both the runtime-version and computed-SDK-version checks. - Write both .npmrc files with printf instead of an indented heredoc. The heredoc already produced correct column-0 keys (the YAML run block dedents to column 0, proven by the successful live publish + read-back), but printf makes the intended bytes unambiguous and avoids recurring reviewer confusion. Output is byte-identical. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the post-publish read-back (npm view against the feed) with a GITHUB_STEP_SUMMARY panel showing the runtime consumed and canary SDK produced. The read-back was not a repo standard (publish.yml trusts the npm publish exit code), was exposed to Azure Artifacts feed-propagation lag (risking false failures), and was redundant: the version is guaranteed by the publish exit code and the runtime pin is set deterministically in the same job under set -euo pipefail. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Rewrite the file header: it claimed 'No publishing happens here', but the workflow tests AND publishes an SDK canary to the internal feed. - Remove actions/checkout from the resolve job; it only normalizes dispatch inputs and validates the version, never touching repo files. - Reword the canary version-base comment to describe the mechanism (next patch of public latest) instead of a hardcoded version that rots. - Fix a doubled-word typo (read read-only -> read-only). - Note that the runtime verify targets the ./nodejs copy (the one the SDK actually spawns during e2e; the harness is a mock CAPI proxy). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The harness is the mock CAPI replay proxy: it declares @github/copilot as a devDependency but never imports or spawns it (verified: zero source imports). During e2e the SDK resolves and launches the runtime from ./nodejs's own node_modules, so overriding the runtime inside ./test/harness had no effect on what the compat gate exercises. Removing the harness override also removes the only reason to write the feed .npmrc there, shrinking the internal-source path and eliminating one place the ADO token was written to disk. Also folds in minor comment tidy-ups. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Fail the publish job (instead of falling back to a 0.0.0 canary base)
when the public SDK latest version can't be resolved, so a transient
npm failure can't publish a misleading canary for a live public package.
Also stop swallowing get-version.js stderr so failures surface in logs.
- Validate runtime_source (public|internal) in the resolve normalize step,
mirroring the existing mode validation, so a future non-dispatch trigger
can't pass an invalid source that silently behaves like public.
- Use 'npm ci --ignore-scripts' in the publish job for consistency with
the test-job installs and the canonical publish.yml.
- Give the resolve job 'permissions: {}'; it only normalizes inputs and
needs neither repo contents nor an OIDC token.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Temporarily re-adds the on-branch validation scaffolding (push trigger + resolve push case + publish.if event==push clause) to exercise the full pipeline after the recent augmentations. To be removed after the run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This reverts commit 5fd0c68.
Tighten both semver checks (runtime version and computed SDK canary version) to disallow leading zeros in major/minor/patch, e.g. 01.0.0, which are invalid SemVer and would be rejected later by npm. Uses the standard (0|[1-9][0-9]*) core pattern; the prerelease portion is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
MackinnonBuck
changed the title
Collaborator
|
Is the intent with this that:
? |
Collaborator
Author
Not necessarily. This workflow has a However, we're also going to have a nightly pipeline in the runtime which will build the latest runtime
The referenced CLI must be compatible with whichever branch the workflow is dispatched from. In the automatic runtime -> SDK dispatch, where the |
Add a repository_dispatch trigger (type runtime-canary) so the runtime's nightly canary pipeline can chain into this workflow: it re-tests and publishes the SDK against the freshly published runtime canary. The resolve job's Normalize step gains a repository_dispatch case that reads runtime_version (required), runtime_source (optional, forced to internal since a runtime canary only lives on the feed), and mode (optional, defaults publish) from client_payload via env. The runtime version is semver-validated as before, and the existing fork guard still applies. The runtime sender is deferred; only the receiver is added here. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When we bump the
@github/copilotruntime, we want to know before the bump lands whether it still works with the Node SDK. This adds a nightly-capable canary pipeline that installs a specific runtime version, exercises the full Node e2e suite against it across all supported operating systems, and, only if everything passes, publishes a matching SDK build pinned to that exact runtime.The result is an early-warning signal for runtime/SDK incompatibilities, plus a ready-to-consume SDK build for anyone who wants to try the SDK against a specific (possibly unreleased) runtime.
How it works
The pipeline runs in two phases: it first tests a chosen runtime against the SDK on Linux, macOS, and Windows, then, if that gate is green, it publishes a canary SDK exact-pinned to the runtime it was validated against. A human can trigger it manually today, choosing which runtime version to test, where to pull that runtime from, and whether to publish. It also accepts an automated trigger from the runtime's own canary pipeline, so a freshly published runtime canary can kick off a matching SDK canary on its own.
Key design decisions
Follow-ups (not in this PR)
Validation
The workflow lints cleanly and has been exercised end to end on a branch across all three operating systems, including a real internal-feed publish of a canary SDK pinned to the tested runtime. The temporary branch-only trigger used for that testing has been removed, so the net change is a single new workflow file.