Skip to content

feat: switch installer to Claude Opus 5.5 and bump @workos/skills to 0.8.0 - #265

Merged
nicknisi merged 2 commits into
mainfrom
feat/installer-opus-5-5
Oct 19, 2026
Merged

nicknisi merged 2 commits into
mainfrom
feat/installer-opus-5-5

Conversation

@nicknisi

Copy link
Copy Markdown
Member

What

Decision for review: reliability vs. install time

Both runs below use production tool gating and @workos/skills 0.8.0. That covers 50 scenarios: all JS frameworks plus Python, Ruby, Go, and Elixir. PHP, Laravel, and Kotlin were skipped because their toolchains weren't installed locally.

Opus 4.5 Opus 5.5
Passed (with retries) 50/50 50/50
First-attempt pass 72.0% 84.0%
Scenario duration p50 / p95 179s / 380s 364s / 534s
TTFT p50 12.3s 13.0s
Tool calls (total) 1,654 2,824
Wall time (concurrency 8) 25m 47m

Per-framework p50 (4.5 → 5.5): nextjs 276→412s, react 166→329s, react-router 191→404s, tanstack-start 371→402s, vanilla-js 199→330s, sveltekit 216→293s, node 138→369s, python 100→467s, ruby 160→522s, go 380→535s, elixir 256→399s.

Opus 5.5 needs fewer self-corrections, but it reads about twice as many files, runs more verification commands, and sometimes spawns subagents. Typical installs take about twice as long. Earlier comparisons, run without production gating, didn't show this gap, so those numbers shouldn't be used.

This is a single run per model, so treat the deltas as directional.

Verification

  • bun run test: 3,476 passed. Typecheck, lint, format:check, and build are clean. The bundled skills manifest reports 0.8.0.
  • Eval smoke under production gating: the SDK's CAN_USE_TOOL_SHADOWED warning is gone, and workos / curl calls are denied.

The eval harness passed every tool in allowedTools, which skips
installerCanUseTool. Eval agents could run commands the shipped installer
denies, including curl and read-only workos CLI calls against the
developer's signed-in account. allowedTools is now empty, as in production.

authkit-nextjs 4.4 makes authkitMiddleware a deprecated alias of
authkitProxy; the Next.js grader accepts both.
@greptile-apps

greptile-apps Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

Changes the installer default to claude-opus-5-5 and updates @workos/skills to 0.8.0.

  • The installer uses Opus 5.5 and WorkOS skills 0.8.0 for integrations.
  • Eval agents use the same tool permissions as the installer.
  • The Next.js grader recognizes authkitProxy as an AuthKit integration.

No actionable issues were found. No runtime checks were run during this review.

Reviews (1) · Last reviewed commit: "test(evals): gate eval tools like produc..." · Reviewed by Greptile

@nicknisi
nicknisi merged commit d7912fa into main Oct 19, 2026
7 checks passed
@nicknisi
nicknisi deleted the feat/installer-opus-5-5 branch October 19, 2026 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants