Repository navigation
Conversation
…missing package.json runAgentInstaller read package.json for every integration, and a missing file exits the process. Kotlin, PHP, and Laravel go through this runner, so their installs stopped right after detection with "Could not find package.json". Only JavaScript integrations read it now. Vanilla JS checked for a package named "workos", which no plain project has, so it asked "Vanilla JavaScript does not seem to be installed. Do you still want to continue?" (default No) and failed outright in CI and agent mode. It is the fallback for any JS project, so the check is skipped.
… its rules - settingSources: []. When omitted, the Agent SDK loads every settings source, including ~/.claude/settings.json: the user's own plugins, MCP servers, hooks, CLAUDE.md, and permission allow rules. Allow rules apply before installerCanUseTool, so a user setting like Bash(curl:*) bypassed the installer's Bash gate, and agents called personal MCP servers. - systemPrompt: Claude Code preset plus installer rules. Omitted, the SDK sends an empty system prompt, so the agent never got Claude Code's tool guidance (Read/Grep over cat/grep) and learned the Bash allowlist one denied command at a time (about 80% of Bash calls were denied). - Allowlist: read-only git (status/diff/log/show/ls-files/rev-parse), `cd <working dir> && <allowed>`, per-tool syntax checks (python -m py_compile/venv, ruby -c, php -l, uv venv), and dependency changes (remove/uninstall, go get, go mod tidy, composer require). Flags stay per tool because the same flag can run code elsewhere (python -c). - Non-Next.js JS prompts say the setup reference covers what the installer handles, so agents stop hunting for it. - options.onAgentMessage lets evals observe the real integration run.
…xtures
The eval sent its own prompt ("invoke the workos-authkit-<framework>
skill", a skill that no longer exists) and called runAgent directly, so
it measured a different flow from the installer: agents searched for the
skill, read reference files, and skipped production's validator. It now
calls the integration's run() with installer options, observing messages
through options.onAgentMessage and counting agent:retry corrections. The
opt-in Fizzy fixture keeps its isolated-environment path.
That surfaced two gaps the old path hid: four Next.js fixtures were on
14.2, below the installer's 15.3 minimum (now ^15.3 with React 19;
partial-install moves authkit-nextjs from 0.15 to ^4), and the
Kotlin/PHP/Laravel and vanilla-js aborts fixed in the previous commits.
Eval runs set WORKOS_TELEMETRY=false.
Brings workos/skills#55: the Next.js reference recommends authkitProxy and warns that a bare @workos-inc/node install pulls v11, outside the authkit-nextjs ^9 || ^10 peer range.
 [Medium impact] The changes reach agent setup across the installer’s supported integration workflows. Fix the shell-command bypass and the eval path that accepts rejected installs before merging. Findings
Fix with agent prompt### Issue 1
src/lib/agent-interface.ts:264-267
The new `cd` check can hide command substitution even when the project path contains no special characters. With `workingDirectory` set to `/proj`, `cd "/proj/$(id)/.." && git status` passes because `resolve()` removes the `$(id)/..` segment. The dangerous-operator check then sees only `git status`, but the returned original input makes Bash execute `id`.
Check the original command before stripping the prefix, or parse and rebuild a safe `cd` command.
> **How this was verified:** The path check removes the substitution-bearing segment before checking operators, while the allow result returns the original command for execution.
### Issue 2
tests/evals/agent-executor.ts:262-267
This catch turns an installer's blocking error into `success: false`, but `ParallelRunner` ignores that field and decides success from the grader alone. A buildable Next.js result without `/sign-in` fails the real installer but can pass `NextjsGrader`, which does not require that route. The eval can therefore report success for an install that production rejects.
Rethrow the failure or make the runner require both installer success and grader success. Add a regression test for this path.
### Issue 3
src/lib/agent-interface.ts:118
Every agent is now told that the installer will build the project and send errors back. Python, Ruby, and Elixir call `runAgent` without a `RetryConfig` and return without that feedback. The new non-Next.js note in `agent-runner.ts` also says the installer handles browser checks.
State only the checks each runner actually performs. Otherwise, agents may defer work to a follow-up that never happens.
### Issue 4
tests/evals/agent-executor.ts:242-244
Passing credentials here skips the integration's environment setup, but the eval has not written all of that setup itself. Ordinary Next.js scenarios write the API key, client ID, and generated cookie password without a default `NEXT_PUBLIC_WORKOS_REDIRECT_URI`. The real validator now requires that value, while the prompt says it is already configured.
Write the framework's callback defaults before marking setup caller-handled. Otherwise, correction counts and timings include repairing eval-only configuration gaps.
### Issue 5
tests/evals/agent-executor.ts:260
Calling real integrations concurrently now makes every scenario reset the same global debug-log path. Each `initializeAgent()` calls `initLogFile()`, which overwrites `sessionLogPath`. Later messages from all active scenarios append to whichever file was selected last.
The separate eval report remains intact, but SDK stderr and permission diagnostics become mixed together. Give each scenario its own logging context or run integrations in separate processes.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryThe PR isolates installer agents from Claude settings, states their command rules, and removes JavaScript-only manifest checks from non-JavaScript installs. Evals now call real integrations, with updated Next.js fixtures and skills.
Acknowledged limits: nicknisi states that PHP, Laravel, and Kotlin end-to-end scenarios were not run because their toolchains were unavailable; unit tests cover their early abort. nicknisi also defers the next skills bump until the backend-call-shape changes are released.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const match = command.match(/^cd\s+(?:"([^"]+)"|'([^']+)'|(\S+))\s*&&\s*(\S.*)$/s); | ||
| if (!match) return command; | ||
| const dir = match[1] ?? match[2] ?? match[3]; | ||
| return resolve(cwd, dir) === resolve(cwd) ? match[4] : command; |
There was a problem hiding this comment.
The new cd check can hide command substitution even when the project path contains no special characters. With workingDirectory set to /proj, cd "/proj/$(id)/.." && git status passes because resolve() removes the $(id)/.. segment. The dangerous-operator check then sees only git status, but the returned original input makes Bash execute id.
Check the original command before stripping the prefix, or parse and rebuild a safe cd command.
How this was verified: The path check removes the substitution-bearing segment before checking operators, while the allow result returns the original command for execution.
Knowledge Base Used: Agent execution and assets
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/agent-interface.ts
Line: 264-267
Comment:
**Directory prefixes hide shell code**
The new `cd` check can hide command substitution even when the project path contains no special characters. With `workingDirectory` set to `/proj`, `cd "/proj/$(id)/.." && git status` passes because `resolve()` removes the `$(id)/..` segment. The dangerous-operator check then sees only `git status`, but the returned original input makes Bash execute `id`.
Check the original command before stripping the prefix, or parse and rebuild a safe `cd` command.
> **How this was verified:** The path check removes the substitution-bearing segment before checking operators, while the allow result returns the original command for execution.
**Knowledge Base Used:** [Agent execution and assets](https://app.greptile.com/workos/-/custom-context/knowledge-base/workos/cli/-/docs/agent-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| } catch (error) { | ||
| return { | ||
| ...base(), | ||
| success: false, | ||
| error: error instanceof Error ? error.message : String(error), | ||
| selfCorrected: false, |
There was a problem hiding this comment.
This catch turns an installer's blocking error into success: false, but ParallelRunner ignores that field and decides success from the grader alone. A buildable Next.js result without /sign-in fails the real installer but can pass NextjsGrader, which does not require that route. The eval can therefore report success for an install that production rejects.
Rethrow the failure or make the runner require both installer success and grader success. Add a regression test for this path.
Knowledge Base Used: Build, release, and validation automation
Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/evals/agent-executor.ts
Line: 262-267
Comment:
**Failed installs pass evals**
This catch turns an installer's blocking error into `success: false`, but `ParallelRunner` ignores that field and decides success from the grader alone. A buildable Next.js result without `/sign-in` fails the real installer but can pass `NextjsGrader`, which does not require that route. The eval can therefore report success for an install that production rejects.
Rethrow the failure or make the runner require both installer success and grader success. Add a regression test for this path.
**Knowledge Base Used:** [Build, release, and validation automation](https://app.greptile.com/workos/-/custom-context/knowledge-base/workos/cli/-/docs/build-release-and-validation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| - Run one command per Bash call from the working directory. Don't use \`cd\`, \`&&\`, \`;\`, \`$(...)\`, or pipes other than \`| head\` or \`| tail\`. | ||
| - Inspect files with Read, Grep, and Glob, not Bash. | ||
| - Bash has no network access: no curl, \`npm view\`, or \`git clone\`. Use WebFetch to read documentation. The \`workos\` CLI is not available to you. | ||
| - Don't start dev servers or open a browser to test sign-in. After you finish, the installer typechecks and builds the project and sends you any errors to fix. |
There was a problem hiding this comment.
Every agent is now told that the installer will build the project and send errors back. Python, Ruby, and Elixir call runAgent without a RetryConfig and return without that feedback. The new non-Next.js note in agent-runner.ts also says the installer handles browser checks.
State only the checks each runner actually performs. Otherwise, agents may defer work to a follow-up that never happens.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/lib/agent-interface.ts
Line: 118
Comment:
**Promised checks never run**
Every agent is now told that the installer will build the project and send errors back. Python, Ruby, and Elixir call `runAgent` without a `RetryConfig` and return without that feedback. The new non-Next.js note in `agent-runner.ts` also says the installer handles browser checks.
State only the checks each runner actually performs. Otherwise, agents may defer work to a follow-up that never happens.
**Knowledge Base Used:**
- [Framework integration agents](https://app.greptile.com/workos/-/custom-context/knowledge-base/workos/cli/-/docs/framework-integration-agents.md)
- [Agent execution and assets](https://app.greptile.com/workos/-/custom-context/knowledge-base/workos/cli/-/docs/agent-execution.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| // Passing credentials marks configuration as caller-handled: no dashboard calls. | ||
| apiKey: this.credentials.workosApiKey, | ||
| clientId: this.credentials.workosClientId, |
There was a problem hiding this comment.
Passing credentials here skips the integration's environment setup, but the eval has not written all of that setup itself. Ordinary Next.js scenarios write the API key, client ID, and generated cookie password without a default NEXT_PUBLIC_WORKOS_REDIRECT_URI. The real validator now requires that value, while the prompt says it is already configured.
Write the framework's callback defaults before marking setup caller-handled. Otherwise, correction counts and timings include repairing eval-only configuration gaps.
Knowledge Base Used: Build, release, and validation automation
Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/evals/agent-executor.ts
Line: 242-244
Comment:
**Eval setup adds repair work**
Passing credentials here skips the integration's environment setup, but the eval has not written all of that setup itself. Ordinary Next.js scenarios write the API key, client ID, and generated cookie password without a default `NEXT_PUBLIC_WORKOS_REDIRECT_URI`. The real validator now requires that value, while the prompt says it is already configured.
Write the framework's callback defaults before marking setup caller-handled. Otherwise, correction counts and timings include repairing eval-only configuration gaps.
**Knowledge Base Used:** [Build, release, and validation automation](https://app.greptile.com/workos/-/custom-context/knowledge-base/workos/cli/-/docs/build-release-and-validation.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| }); | ||
|
|
||
| try { | ||
| await integration.run(options); |
There was a problem hiding this comment.
Calling real integrations concurrently now makes every scenario reset the same global debug-log path. Each initializeAgent() calls initLogFile(), which overwrites sessionLogPath. Later messages from all active scenarios append to whichever file was selected last.
The separate eval report remains intact, but SDK stderr and permission diagnostics become mixed together. Give each scenario its own logging context or run integrations in separate processes.
Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/evals/agent-executor.ts
Line: 260
Comment:
**Parallel debug logs get mixed**
Calling real integrations concurrently now makes every scenario reset the same global debug-log path. Each `initializeAgent()` calls `initLogFile()`, which overwrites `sessionLogPath`. Later messages from all active scenarios append to whichever file was selected last.
The separate eval report remains intact, but SDK stderr and permission diagnostics become mixed together. Give each scenario its own logging context or run integrations in separate processes.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.This branch has not been deployed
1 participant
Why
After #265, the Opus 5.5 installer passed every scenario but was slow and used many tool calls. Looking at the transcripts turned up structural problems in the installer itself, not just in the skills.
What
1.
fix(installer): Kotlin, PHP, and Laravel installs aborted before the agent ran.runAgentInstallerreadpackage.jsonfor every integration, and a missing file exits the process. These three integrations go through that runner, soworkos installprinted "Could not find package.json… Installer setup canceled." right after detection. Vanilla JS checked for a package namedworkos, which no plain project has, so it asked "Vanilla JavaScript does not seem to be installed. Do you still want to continue?" (default No) and failed outright in CI and agent mode.2.
fix(installer): isolate the agent from user Claude settings, and state its rules up front.settingSources: []. When this option is omitted, the Agent SDK loads~/.claude/settings.json: the user's plugins, MCP servers, hooks,CLAUDE.md, and permission allow rules. Allow rules apply beforeinstallerCanUseTool, so a user setting likeBash(curl:*)bypassed the Bash gate. Eval agents also called a personal MCP server (session history). This also made every local eval run measure the developer's personal setup.systemPrompt: the Claude Code preset plus installer rules. When omitted, the SDK sends an empty system prompt. The agent never got Claude Code's tool guidance and discovered the Bash allowlist one denial at a time: about 80% of Bash calls were denied, for both models. The appended rules state the allowlist, say to use Read/Grep/Glob for inspection, rule out network, theworkosCLI, and dev servers, and say the installer handles dashboard settings.status,diff,log,show,ls-files,rev-parse; denies-c,--output,--ext-diff, …)cd <working dir> && <allowed command>python -m py_compile/venv,ruby -c,php -l,uv venv. These stay per tool becausepython -cruns code.remove/uninstall,go get,go mod tidy,composer require3.
test(evals): the eval now runs the real installer integration. It used its own prompt, which named a skill that no longer exists, and calledrunAgentdirectly. So it measured a different flow and missed production's validator and the aborts in (1). It now calls each integration'srun(), observing messages through the newoptions.onAgentMessagehook. Four Next.js fixtures were on 14.2, below the installer's 15.3 minimum; they're now on 15.3 with React 19.4.
chore(deps):@workos/skills0.8.1 (workos/skills#55:authkitProxy, plus the@workos-inc/nodepeer-range warning).Results
50 scenarios: all JS frameworks plus Python, Ruby, Go, and Elixir, run through the real installer flow with production gating and settings isolation. The skills come from workos/skills#56 (verified backend call shapes) on top of 0.8.1. "Before" is the #265 run.
Opus 5.5 now leads on first-attempt passes and matches 4.5 on typical install time. Its slowest installs (p95) and time to first token are still higher than 4.5's.
Not covered: PHP, Laravel, and Kotlin scenarios, because their toolchains weren't installed locally. Their abort is covered by unit tests. This is one run per model.
Verification
bun run test: 3,483 passed. Typecheck, lint,format:check, and build are clean.Follow-up
After workos/skills#56 merges and is released, bump
@workos/skillsagain to ship the backend call shapes.