Conversation
Research note on Spotify's "Portal cut my Claude Code token usage by 90%" post, worked against the shipped source in spotify/portal-ai-plugins rather than the post alone. Findings: - The transferable idea is harness enforcement (PreToolUse hooks), not model routing. The post says so itself: the CLAUDE.md version failed because the rules were advisory. - The 90% figure is measured on Claude context tokens only, over four synthetic scenarios on three fixture files, with chars/4 as a token proxy. Reconstructing it with real rates (Opus 5 cache write/read vs Gemini 2.5 Flash) shows the claim survives all-in dollar accounting for its measured case: 86-91% depending on session length. - The post's stated reason is wrong even though its number is right. Repeat delegations are not free; they cost a full worker round trip each, while a resident file costs nothing marginal. Derives the crossover. - Source-level notes: a ~120KB argv payload ceiling on Linux, gate bypasses the hooks do not cover, a targeted `head -100` blocked where `Read` with a limit is allowed, deprecated hook decision schema, and lossy fence stripping in code-write. - Situates the pattern against FrugalGPT/RouteLLM, context rot, code execution with MCP, and Anthropic's own orchestrator measurement (55% cheaper, 3-7 points below best score) and lever ordering. - Names the native baseline the post does not compare against: a project `Explore` subagent pinned to a cheaper model. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
🟡 Changes recommended
The new doc includes at least one unverifiable pinned version claim and a Sources entry referencing non-existent repo artifacts, which should be corrected for traceability and long-term accuracy.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new research note documenting the “Portal/shunt delegation” pattern, focusing on harness-enforced context discipline and providing a reconstructed all-in cost model that distinguishes “tokens measured” vs “dollars paid”.
Changes:
- Introduces a layered conceptual model (hooks/scripts/skills) and identifies the transferable idea as harness enforcement via hooks.
- Analyzes what Spotify’s “90%” claim measures vs what it omits, then reconstructs savings using explicit rate assumptions.
- Summarizes source-level observations from
spotify/portal-ai-pluginsand situates the pattern among related cost/context strategies.
File summaries
| File | Description |
|---|---|
| docs/research/portal-delegation-pattern.md | New research note analyzing Portal/shunt’s delegation + enforcement pattern, measurement limits, and an all-in cost model. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Addresses two Copilot review findings on #133, both correct: - The `Explore` v2.1.198 model-inheritance claim was unsourced. Cite the Claude Code subagents reference inline, which states the boundary. - The Sources list pointed at `shared/prompt-caching.md` and `shared/cost-optimization.md` as if they were repo paths. They are files in Claude Code's bundled `claude-api` skill and unreachable to a reader. Replace with the public pricing and prompt-caching docs, which confirm the §3 rates exactly (Opus 5: $5.00 base input, $6.25 5m cache write, $0.50 cache hit; 1.25x/0.1x multipliers). The orchestrator measurement and lever ordering quoted in §5 have no public URL, so that entry now says plainly where it comes from rather than implying a repo path. It is quoted verbatim in the note so the claim stays checkable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3f9942287
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… claim Two real errors caught by Codex review on #133. The crossover inequality charged one summary cache read per turn (`$0.0002·T`) when Q delegated questions leave Q resident summaries, each re-billed on every later turn. The old linear form `Q < 7.1 + 0.53·T` grew without bound; charging both sides symmetrically gives Q < (0.0375 + 0.0030·T) / (0.0053 + 0.0002·T) which saturates at 15 — the 6000/400 token ratio at which accumulated summaries occupy as much context as the file would have. The headroom at T=10 was overstated as ~12 questions; it is ~9. The correction strengthens the section's conclusion rather than weakening it: delegation's advantage over a resident file is bounded, so the interrogation-loop inversion is sharper than first stated. Separately, the native-baseline paragraph claimed a Haiku-pinned `Explore` subagent involves "no network round trip". It is still a hosted model call. What it avoids is a second vendor and the CLI/backend/worker hops, so say that instead. Softened the matching "none of the operational cost" in Open Questions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
The Portal/shunt research surfaced three gaps in skills we already ship. No new plugin: the catalog already owns every constituent idea, and the delegation mechanism itself is largely native (an Explore subagent pinned to a cheaper model), so find-before-build says don't build it. egress-gate — delegation-for-cost is egress that does not feel like egress. The destination is a worker model rather than a named service, the payload is whole source files, and the better the optimization works the more of the repo leaves. Named as a failure mode; step 3 now names worker-model tools as unnamed destinations. eval-ladder — audit question #4 now asks whether a metric counts both sides when a change moves work rather than removing it. metric-choice.md gains the worked example: shunt's benchmark measures "Claude context tokens" only, over four scenarios on three fixtures, with chars/4 as a proxy that cannot separate a cache write from a cache read (12.5x apart in price). Two generalizable lessons: a token count is not a cost, and the quality arm is the invisible one. context-handoff — the DELEGATE step had no cost dimension at all. It now states that delegation's advantage is bounded: N questions against one corpus leave N summaries resident while the corpus is re-sent each time, so a long question-and-answer loop inverts the trade. All three kept self-contained — no references to this repo's docs/ paths, since these plugins install standalone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Demonstration — three edited skills, run on real inputRequired by 1. eval-ladder — audit question #4, run against
|
JRichlen
changed the title
|
|
Re-run outcome (promised above): reproduces identically. Job Byte-for-byte the same signature — every case in both packs, calibration control included: So this is not transient. Two independent runs, ~3 minutes apart, zero model output on every row of two packs whose prompts this diff does not alter. That closes the "flake" branch: a second failure is real, and my one re-run is spent. Where it standsThe blocker is outside what I can see or reach: the subject model returns nothing for every call, which points at I am not going to make this green by other means. Not the floor, not the path filter, not the packs, and not by reverting the three skill edits to dodge the trigger — the tier is correctly reporting that it could not evaluate, and every one of those would be tuning the gate rather than fixing the cause. What a maintainer can check
Merge postureThe research doc ( If you would rather unblock the doc, say the word and I will move the three skill edits onto their own branch and PR, leaving this one green and mergeable. I have not done that unilaterally since it changes the shape of a PR you may want kept whole. Keeping the PR watched until it is green, merged, or closed. Generated by Claude Code |
Third attempt, and a narrowed diagnosisA third routing-tier run (job I dug further rather than re-running again. Three things are now settled that weren't before. 1. The model is not the problem — ruled out
2.
|
| # | Hypothesis | Fits the evidence? |
|---|---|---|
| A | OPENROUTER_API_KEY invalid / unset / out of credit |
Yes — auth failure yields no content on every row |
| B | The model returns reasoning but empty final content | Also yes — showThinking: false strips the trace, leaving nothing to grade |
B deserves more weight than it first looks. This is a reasoning model, the config already carries a comment about fighting exactly this interaction once before (the Thinking: prefix creating a second ROUTE: line), and max_tokens is 4096 in the routing pack. If reasoning traces have grown — a provider swap under OpenRouter's routing would do it — the budget can be consumed before any final answer is emitted. Every row empty, calibration included, both packs, persisting across days is exactly what that looks like.
What separates them in one minute: the routing-results artifact on run 34414369828 carries each row's raw error. A 401/402 says A. A populated reasoning field with empty content says B.
Proposed patch if it turns out to be B
providers:
- id: openrouter:nvidia/nemotron-3-ultra-550b-a55b
config:
max_tokens: 16384 # was 4096 — reasoning trace must fit *plus* the ROUTE: line
showThinking: false(The trajectory pack is already at 8192 and fails too, so it would need the same treatment.)
I have not pushed this. I cannot run the tier to validate it, and a speculative change to a paid eval's budget, on a PR about something else, is exactly the kind of unvalidated widening that costs a cycle and reviewer trust. It is a proposal, not a fix — happy to push it the moment someone confirms B, or to open it as its own PR.
Merge posture is unchanged: the doc half is green and independently mergeable; the three skill edits stay unverified until this tier can actually evaluate.
Generated by Claude Code
…tput The routing tier has been red on this PR since 0966b4c with every row of both packs showing `<no ROUTE: line in output>` / `<no STEP: line in output>` and every slot `<missing>`. Three runs, two of them by different actors ~15h apart, all identical. Nobody could say why, because the diagnostic step prints the per-slot diff and the model's line but never the row's own error — so an auth failure, a timeout, and an empty completion are indistinguishable on the page. pass-rate.sh already separates FAULT from FAIL using `failureReason`; that distinction just never reached the log. Both diagnostic steps now print `failureReason` and a 600-char `error` slice, but only when the model line is empty — a genuine assertion failure is unchanged, so this adds no noise to the case the step was built for. Validated offline against synthetic results in promptfoo's shape, both packs, two rows each: a FAULT row now renders --- transport (empty output is a FAULT, not a verdict) --- failureReason: 2 error: API error: 401 Unauthorized - No auth credentials found and a real assertion failure renders exactly as before. Output-only step, so it cannot change any verdict. Bracketing, for whoever picks this up: the tier last genuinely evaluated and PASSED at 2026-09-09T04:00Z (run 34309062883, four minutes, artifacts uploaded), and first failed at 22:07Z the same day. The only main commit between them is 282b416, whose own routing leg ran seven seconds — a skip. So nothing in the repo changed the pack or its inputs in that window, and the cause is external: credential, credit, or provider behaviour. The pinned model is live (3 providers, 100% 3d uptime), which rules out a dead slug. This commit does not fix that; it makes the next run name it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
The transport block added in d6879e5 worked — the very next run named the cause — but it printed the provider's raw error body, and OpenRouter's 402 includes a workspace key-management URL whose path segment is a 64-char key identifier. That went into a public Actions log. It is a key identifier, not the API key, and it grants nothing without an authenticated session to that workspace. It still should not be published. The error is now passed through two substitutions before the 300-char slice: URLs become <url-redacted>, runs of 32+ hex become <hex-redacted>. Validated against the real error shape observed in the run: the useful part survives — `402 Payment Required`, `can only afford 202`, `limit_source: openrouter_credits` — and a grep for URLs or long hex over the rendered output comes back empty. Root cause this surfaced, for the record: the OpenRouter key is valid and authenticating (402, not 401); its credit or monthly cap is exhausted, to the point that it "can only afford 202" tokens against an 8192 request. No code change fixes that — the tier goes green when credit is restored. Note the earlier max_tokens hypothesis was not just unproven but backwards: OpenRouter's own remedy hint is to LOWER max_tokens to fit the balance, so the 4096 -> 16384 bump proposed in issuecomment-5620223388 would have made this worse. Left unpushed for want of validation, which turned out to be the right call for a reason I did not have at the time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Root cause found: the OpenRouter key is out of credit
402, not 401. The key is valid and authenticating fine — the credit balance or the key's monthly cap is spent, down to affording ~202 tokens against an 8192-token request. Every row of both packs fails before the model ever runs, which is why the calibration control failed too and why the signature never varied across four runs and two days. No code change fixes this. Add credits, or raise that key's monthly limit, and the tier goes green on its own. Nothing about the routing pack, the floor, the path filter, or the skill edits in this PR was ever wrong. The hypothesis I had backwardsWorth recording. issuecomment-5620223388 floated raising I left it unpushed because I couldn't validate it. That turned out to be right for a reason I didn't have at the time, which is the argument for the rule rather than for my judgement. One thing I broke and fixedThe transport block printed the provider's raw error body, and OpenRouter's 402 embeds a workspace key-management URL whose path is a 64-character key identifier. That went into a public Actions log on my push. It is an identifier, not the API key, and it grants nothing without an authenticated session to that workspace — but it should not have been published, and this is precisely the class of thing Fixed in The blind spot underneathStill worth its own issue: the routing tier is the only job in CI that touches Where this leaves the PR
The doc half is green and independently mergeable. The three skill edits still have no verification behind them, and won't until the credit is restored — that hasn't changed, but the reason is now a known quantity with a known remedy rather than an open question. Generated by Claude Code |
Correction: "out of credit" was the wrong readingMy previous comment headlined this as "the OpenRouter key is out of credit." That is wrong and I'm correcting it before it misleads anyone. The key's dashboard shows Key Limit at $32.09 / $60 monthly (53%) and Workspace Guardrail at $0.07 / $20 daily (0%). Neither budget is exhausted. I read "requires more credits" and jumped to the cap. What the error actually distinguishesThe observed facts are unchanged and quoted verbatim from run What I got wrong is which limit that names. A per-key spending cap and an account credit balance are different numbers: the cap bounds what a key may spend, the balance is the wallet it spends from. A key can sit at 53% of a $60 cap while the account balance is near zero. OpenRouter's own remedy hint points at the credits/balance page, not the key-limit page — which fits the balance reading, not the cap reading. The arithmetic fits it too: affording ~202 output tokens at this model's $2.20/M is roughly $0.0004 of headroom. That is not "$27.91 left on the cap." A discrepancy worth resolving firstThe key's dashboard reads Last Used: 13 hours ago. This PR's runs called OpenRouter at 17:46 and 17:49 UTC today. If that were the same key, "last used" should read minutes. Either 402-rejected requests don't update that field, or the key on that dashboard is not the one in the Two things worth a glance, in order:
What is not in doubtEvery row of both packs is rejected with HTTP 402 before the model runs. That is read directly from the log, not inferred, and it explains the whole shape of this failure — every row empty, calibration control included, identical across four runs and two days, unaffected by any re-run. Also unchanged: no code change in this PR fixes it, and the The two commits stand on their own merits: Generated by Claude Code |
The funding probe read /api/v1/key -> limit_remaining and called it "credit". That is the spending ceiling on one API key, not the money behind the account, and the two fail independently — the 402 body says which via metadata.limit_source. From 2026-09-10 the key cap read 53% used, comfortably healthy, while every row of every pack was refused with limit_source: openrouter_credits. PR #133 sat red for five days on a diagnosis that read the key cap and concluded funding was fine. Replayed against the old probe with that exact response shape, it prints "remaining=27.91" and exits 0: reassurance in precisely the outage it exists to catch, which is the false-green this script was written to remove. Now probes both, names both distinctly in the log, and fails closed on either. Unparseable or unreachable still warns rather than blocks — this repo does not own OpenRouter's response schema, and the pings remain the load-bearing evidence. Also drops ping-payloads.txt, a wire capture left at the repo root. It was evidence for the PR body, referenced by nothing. Verified offline with a stubbed curl across six response shapes: drained account behind a healthy key cap (fails, was green before), both healthy (passes), credits endpoint 404 (warns), credits schema renamed (warns), key cap exhausted with a funded account (fails), key endpoint down with a drained account (fails). Cheap tier 1294 passed / 0 failed, up from 1293 on this branch's head — note the PR body's "1294" predates this commit and was already one ahead of what the branch actually ran. New cheap-tier guard 19a2c is coupled four ways: remove the credits endpoint, remove the credits request, stop failing closed on the balance, or drop the key cap read, and it goes red. Its first draft passed one of those four — it anchored on the first textual mention of /api/v1/credits, which is in the probe's own comment header, so the segment swept in the key-cap block's failure. It now anchors on the request itself. Not verified from this container: the live shape of /api/v1/credits. Egress to openrouter.ai is blocked here and no key is available, so the field names come from the vendor's documented schema, not from a response observed on the wire. A rename degrades to the UNVERIFIED warning rather than a false pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd
Two parts: a research note on Spotify's Portal by Spotify cut my Claude Code token usage by 90% — worked against the shipped source in
spotify/portal-ai-pluginsrather than the post alone — and the three existing skills that research sharpened.Warning
Not mergeable as-is. The
routing tiercheck is red on0966b4cand cannot be fixed from this branch — see Verification. The doc half is green; the skill half has no verification behind it.Part 1 — the research note
docs/research/portal-delegation-pattern.md.The transferable idea is harness enforcement, not model routing. The post says so itself — the first version lived in
CLAUDE.mdand failed because "the rules were advisory, not enforced." ThePreToolUsehook is the finding; the two-model cascade is one instantiation of it.The 90% figure measures one arm of a two-arm system.
evals/benchmarks.jsonstates in its own header that it measures "Claude context tokens with vs without shunt" usingchars / 4. Four scenarios, three fixture files, worker tokens uncounted, and a proxy that cannot tell a cache write from a cache read (12.5× apart in price).Reconstructed with real rates, the claim survives anyway — 86% / 89% / 91% all-in dollar savings at 0 / 10 / 30 follow-on turns.
But the post's stated reason is wrong even though its number is right. "Re-sending the files on a follow-up is free" is true of context, false of dollars. Charging both sides symmetrically, delegation wins while
Q < (0.0375 + 0.0030·T) / (0.0053 + 0.0002·T)— saturating at 15, the6000 / 400ratio at which accumulated summaries occupy as much context as the file would have. The advantage is bounded, and it inverts in tight interrogation loops.Source-level observations: a ~120 KB
argvceiling on Linux; gate bypasses the hooks don't cover (sed -n,awk,rg,cat a.ts b.ts,Readwith anylimit); a targetedhead -100blocked whileReadwithoffset/limitis allowed; deprecated{"decision": "block"}schema;sed '/^```/d'silently corrupting any generated file containing a fence.Part 2 — three skills it sharpens
No new plugin. The catalog already owns every constituent idea, and the mechanism is largely native (an
Exploresubagent pinned to a cheaper model), sofind-before-buildsays don't build it.egress-gateeval-laddermetric-choice.mdcontext-handoffAll kept self-contained — no references to this repo's
docs/paths, since these plugins install standalone.Review corrections
Both bots found real problems; all five threads resolved.
Qquestions leaveQresident summaries390da94).T = 10headroom was overstated as ~12; it is ~9.Explorebaseline claimed "no network round trip"390da94).shared/*.mdpathsd9b1f52).v2.1.198pind9b1f52).One citation a reader still cannot follow: the §5 orchestrator measurement has no public URL. That entry says so plainly rather than attaching a URL that doesn't support it.
Verification
The routing tier is the blocker.
0966b4cis the first commit here to touch aSKILL.md, so it is the first time that tier actually evaluated — earlier greens were SKIPPED legs, which the workflow states out loud ("Green because it did not run, NOT because routing passed"). Every row of both packs returned no model output, including the must-not-fire calibration control, and it reproduced identically on one re-run. Root cause is outside this branch: the subject model returns nothing, pointing atOPENROUTER_API_KEYor the configured model's availability. Full diagnosis here and here.Nothing was relaxed to get green — not the floor, not the path filter, not the packs, and not by reverting the skill edits to dodge the trigger.
Merge posture: the doc half (
390da94and earlier) is green and independently mergeable. The skill edits rest on the cheap tier plus the demonstration until that key works. Happy to split them onto their own PR if you'd rather land the doc now.docs/testing.mdis unchanged: no eval tier, workflow job, or eval pack is added, removed, renamed, or re-scoped.🤖 Generated with Claude Code
https://claude.ai/code/session_01XHYtoYrtWhTGy59GC9uJGd