Repository navigation
feat: switch installer to Claude Opus 5.5 and bump @workos/skills to 0.8.0 - #265
Merged
Merged
Conversation
The eval harness passed every tool in allowedTools, which skips installerCanUseTool. Eval agents could run commands the shipped installer denies, including curl and read-only workos CLI calls against the developer's signed-in account. allowedTools is now empty, as in production. authkit-nextjs 4.4 makes authkitMiddleware a deprecated alias of authkitProxy; the Next.js grader accepts both.
[Medium impact] The PR appears safe to merge; no actionable issues were found. SummaryChanges the installer default to
No actionable issues were found. No runtime checks were run during this review. Reviews (1) · Last reviewed commit: "test(evals): gate eval tools like produc..." · Reviewed by Greptile |
gjtorikian
approved these changes
Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
src/cli.config.ts: installer modelclaude-opus-4-5-20251101→claude-opus-5-5. The Agent SDK bump this needs landed in chore(deps): bump claude-agent-sdk to 0.3.295 for Opus 5.5 support #261.@workos/skills0.7.3 → 0.8.0. This brings the stale-API fixes and trimmed references from fix: update skills for current SDKs and frontier models skills#53, plus feat: multi-SDK expansion with auto-discovery registry #49 (SDK-aware redirect guidance) and fix: prefer existing middleware.ts over proxy.ts for Next.js 16+ #52 (accountless install).allowedTools, which skipsinstallerCanUseTool. Eval agents could therefore run commands the shipped installer denies, includingcurland read-onlyworkosCLI calls (environment list,whoami,authkit redirect-uris list, ...) against the developer's signed-in WorkOS account. I checked the transcripts from the last two runs: they contain only reads, no writes.allowedToolsis now[], as in production.authkitProxy. In authkit-nextjs 4.4,authkitMiddlewareis a deprecated alias ofauthkitProxy, and Opus 5.5 uses the current name.Decision for review: reliability vs. install time
Both runs below use production tool gating and
@workos/skills0.8.0. That covers 50 scenarios: all JS frameworks plus Python, Ruby, Go, and Elixir. PHP, Laravel, and Kotlin were skipped because their toolchains weren't installed locally.Per-framework p50 (4.5 → 5.5): nextjs 276→412s, react 166→329s, react-router 191→404s, tanstack-start 371→402s, vanilla-js 199→330s, sveltekit 216→293s, node 138→369s, python 100→467s, ruby 160→522s, go 380→535s, elixir 256→399s.
Opus 5.5 needs fewer self-corrections, but it reads about twice as many files, runs more verification commands, and sometimes spawns subagents. Typical installs take about twice as long. Earlier comparisons, run without production gating, didn't show this gap, so those numbers shouldn't be used.
This is a single run per model, so treat the deltas as directional.
Verification
bun run test: 3,476 passed. Typecheck, lint,format:check, and build are clean. The bundled skills manifest reports0.8.0.CAN_USE_TOOL_SHADOWEDwarning is gone, andworkos/curlcalls are denied.