Skip to content

feat(auto-advisor): rebuild Auto Advisor on OpenCode 2.2.5 - #94

Merged
capyBearista merged 42 commits into
mainfrom
feat/auto-advisor-rebuild
Nov 1, 2026
Merged

capyBearista merged 42 commits into
mainfrom
feat/auto-advisor-rebuild

Conversation

@capyBearista

@capyBearista capyBearista commented Oct 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Rebuilds @capybearista/opencode-auto-advisor for OpenCode 2.2.5+. A second Advisor model reviews the agent's work, either when the agent asks or when a cheap Decision model spots a consequential moment. By default the Advisor is the agent's own model; set advisor.model to use a stronger one.

  • Explicit advisor() tool (no arguments). The Advisor reads the agent's system prompt, tool list and transcript, and its advice comes back as the tool result. The tool's guidance and the Advisor's prompt follow the widely used advisor-tool wording, adapted to what the plugin actually forwards.
  • Automatic reviews (on by default). On each step, a Decision model (default opencode/jev-1.13-free) classifies the moment (approach, stuck, done, routine) and scores consequence (0–4). The plugin consults the Advisor when 1 − P(routine) ≥ 0.7 and the score is ≥ 1.5. The Advisor gets the moment as a hint to check, not an instruction.
  • What the Decision model sees:
    • the last 3 real user requests, each capped at 20,000 characters by default (routing.requests, routing.requestMaxChars, where -1 means no cap). The cap applies to each request, not the total; the default total fits Jev free's 64k-token window.
    • the turn's last 12 steps, with subagent results labelled SUBAGENT and other notices NOTICE;
    • failure counts. An error result counts for any tool. A shell exit, kill or timeout notice counts only on shell/bash results, and only as a whole content part.
  • Limits: 2 automatic consults per user turn, and a 10-step cooldown across the session. A step blocked by both is logged as cooldown. Explicit calls ignore both limits. Both limits are kept in memory, so restarting OpenCode resets them.
  • Fallback chains: routing.models and advisor.model each accept an ordered list of up to 8 models (inherit allowed for the Advisor).
    • Moves to the next model: unreachable, quota or rate limit, overload, model not found, or a request the model rejects (context overflow included).
    • Stops: content filter, or a bad #variant.
    • advisor.timeoutMs defaults to −1 (no limit). OpenCode's own header and chunk timeouts end a model that goes silent, so one that is still reasoning is never cut off.
  • Cache-safe delivery. The review goes to the tail of the current request and into an auto-advisor instruction entry for later steps. It never edits the system prompt.
  • TUI:
    • a one-line Advisor widget in the sidebar; clicking it opens a scrollable markdown review panel (model, duration, time);
    • /advisor-review and the command palette open the panel too, and tui.sidebar: false hides the widget;
    • one opencode.json(c) entry loads both halves, and cli.json needs nothing.
  • Observability:
    • telemetry records the Decision input (its last 32k characters), the newest message id and each Advisor attempt;
    • bun run timeline <sessionID> renders a session's routing decisions next to its messages as one HTML page, readable only by its owner (0600). It reads each message page from a file, because OpenCode's CLI exits right after writing and cuts a large piped response short.
  • Developer tooling (not shipped):
    • bun run sandbox runs an isolated OpenCode with this package and a mock model, in the foreground until Ctrl-C. Each run gets a new private (0700) directory; inherited OPENCODE_* variables are dropped.
    • bun run typecheck now also checks scripts/.

Why 2.2.5: OpenCode 2.2.5 reworded its Code Mode catalog, and the Advisor's catalog compaction matches that text. The peers are @opencode/plugin >=2.2.5 and @opentui/* >=0.8.0. Dev pins stay at 2.2.0, because the monorepo keeps shared pins identical.

This supersedes #61.

Linked Issue

Refs #47, #50. Supersedes #61.

Plugin(s) Affected

  • opencode-agents-loader
  • opencode-double-tap-timeline
  • opencode-output-styles
  • opencode-ram-monitor
  • opencode-agent-prompt-inheritance
  • opencode-adversarial-review
  • opencode-plugins (this monorepo itself): release-channel registration, root docs
  • Other: opencode-auto-advisor (new)

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Other: _______

OpenCode Validation

  • OpenCode versions tested: @opencode/cli 2.2.0 and 2.2.5, each in an isolated HOME, with a mock provider and real OpenCode Zen for decisions. Details are in docs/auto-advisor/findings.md.
  • Runtime smoke test performed: bun run smoke, plus these live runs.
    • 2.2.5, with a probe MCP server and a probe skill:
      • the Advisor prompt had all three [context-omitted …] catalog markers;
      • a shell exit 3 counted as failed, and a read of "Exited with code 1" counted as ok;
      • the widget and panel rendered on OpenTUI 0.8.
    • Requests: a long plan prompt, then a short correction. The Decision input held the plan and the correction.
    • Advisor chain (dead port → overflow 400 → silent model → slow model sending keep-alives):
      • each failure moved to the next model;
      • the silent model was ended by the chunk timeout;
      • the keep-alive model was not cut off.
    • Decision overflow: real Jev rejected a 300k-character input, and the router moved on.
    • Labels: a synthetic <subagent> message reached the Advisor labelled SUBAGENT.
    • Real session on 2.2.5: an OpenRouter decision model (openrouter/typesafe/jev-1.14) answered every step in 150–315 ms, and the Decision input showed the user's request and a SUBAGENT result.
    • Timeline: on a 3.2 MB message page, the real CLI's piped output arrived cut short at random lengths, while file output arrived whole 10/10; the fixed script rendered the live session 5/5.
    • Sandbox: started from a path containing a space, $ and a quote; Ctrl-C, a server crash and a failing CLI at startup each left no processes behind; a tool's sleep ended with the server.
    • Not exercised live: compaction and child sessions, which unit tests cover.

Quality Checklist

  • I ran bun run typecheck && bun run lint && bun run test
  • I ran bun run build
  • This is the smallest safe root-cause fix (no unnecessary logic changes)
  • I preserved behavioral invariants and added/updated tests as needed (196 package tests; mutation-checked rules)
  • For plugin/runtime changes, I verified the built module or package export that OpenCode actually loads
  • If local .opencode config was part of testing, I verified it points at the package root or published plugin name rather than a stale dist/* artifact
  • For bug fixes, I captured the root cause and why earlier symptoms/reviewer findings were misleading (n/a: new feature)
  • I updated docs for user-facing changes
  • I added a changeset if this should ship. No changeset: the package is an unpublished 2.0.0, which the release guard publishes as-is (bun run release:changesets passes).
  • I have read the Contributing Guidelines

Additional Context

  • Privacy:
    • The Decision provider gets each step's last user requests (20,000 characters each by default), recent tool calls and truncated results. It never gets the system prompt or tool definitions.
    • The Advisor gets the transcript, up to advisor.maxTokens.
    • Local telemetry and timeline pages hold the Decision input. Both are owner-only (0600); telemetry rotates at 5 MB.
  • Cost: automatic reviews are on by default. Without advisor.model, they use the agent's own model and are billed at its rates.
  • Accepted limitations (documented in AGENTS.md and findings.md):
    • Worst-case waits: a refused connection takes about 12 s to fail over, because OpenCode retries it. A silent provider can take up to its header or chunk timeout.
    • A prompt that a compaction absorbs before any step sees it never starts its own turn.
    • If one read of the run-end event stream fails, late consults are no longer dropped.
    • Raising routing.requestMaxChars or routing.requests can overflow a small decision model's window. Sizing per model is a follow-up.
  • Follow-ups (not in this PR): per-model Decision input sizing; matching catalogs by key (metadata.experimental.instructions, 2.2.5); dev pins to 2.2.5 and OpenTUI 0.8; typechecking test files repo-wide.

github-advanced-security[bot]

This comment was marked as resolved.

Zero-argument advisor() tool plus Decision-model-triggered automatic
reviews (moment + consequence), delivered at the tail of the current
request and as an instruction entry, never by editing the system
prompt. Observe mode logs JSONL telemetry; the TUI shows review status.
One opencode.json entry loads both halves.
…pts as the request

Abort a pending consult when the run ends and skip it if the run ended
during the Decision call. Mark that outcome stale, not failed, and show
a review in the TUI only when it is new. The Decision input now uses the
latest real prompt, and observe mode catches late errors.
…en routing

Cap every transcript part at a quarter of the budget so an oversized
message can't drop the current turn or overflow the Advisor's context.
Treat a moment answer without probabilities as unanswered and never
label a consult routine. Review the request that triggered the consult,
dispose the tool on cleanup, retry a failed context-limit lookup, cap
recorded prompts, show a TUI review as new only if it changed since the
consult started, and rotate telemetry by bytes.
…babilities

The host validates decision answers against the questions and its schema
requires probabilities, so the guard was unreachable.
@capyBearista
capyBearista force-pushed the feat/auto-advisor-rebuild branch from 1b58ce9 to 9f1169a Compare October 28, 2026 03:32
github-advanced-security[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@capyBearista
capyBearista marked this pull request as ready for review October 29, 2026 05:54
devin-ai-integration[bot]

This comment was marked as resolved.

@capyBearista capyBearista changed the title feat(auto-advisor): rebuild Auto Advisor on OpenCode 2.2.0 feat(auto-advisor): rebuild Auto Advisor on OpenCode 2.2.5 Oct 31, 2026
@capyBearista
capyBearista force-pushed the feat/auto-advisor-rebuild branch from 313f05e to 99dce34 Compare October 31, 2026 16:42
…ne handle

Resolves the CodeQL file-system race alerts: the config mtime check and
read, and the telemetry size check and append, now go through the same
file handle.
Copy the log to .1 and truncate it in place instead of renaming and reopening by path. This clears the CodeQL file-system-race alert and removes a double close when the reopen failed.
Each event awaited its own status lookup, so a fast consult could send review.finished before review.started and leave the TUI footer stuck on "reviewing". Chain the emits.
advisor.model and routing.models take OpenCode's provider/model#variant syntax to pick a reasoning level. Without a variant the host applies no overlay (provider default); inherit keeps the agent's variant. Telemetry keys include the variant.
advisor.compactContext (default true) replaces OpenCode's Code Mode tool catalog, MCP server instructions and skill catalog with [context-omitted: ...] markers in the Advisor's copy of the system prompt and system updates. User and tool content and the Executor's own request are unchanged. A block is replaced only when its end precedes the next copy's start.
The Advisor's budget becomes min(advisor.maxTokens, 75% of its context window) at 3 chars per token. Default 64000; at least 1000. Over budget the existing trim drops middle turns and keeps the system prompt, the first request and the newest steps.
An invalid auto-advisor.json falls back to defaults, so advisor() silently consulted the Executor's model instead of the configured one. Return the config error instead. Document rebuilding a local checkout after pulling.
…transcript

Parts were capped individually, but one step's parallel tool results share a message. A newest message with several large parts exceeded the remaining budget and the trim loop dropped it, so the Advisor lost the latest evidence. Cap each message at a quarter of the budget too.
The Decision model was misled three ways:
- after an OpenCode restart the in-memory prompt ids were empty, so a
  background subagent result (a role "user" synthetic message) became the
  turn and the request, resetting the per-turn budget;
- delivered reviews ("moment: done") came back as system updates in its input;
- a failing shell command completes the tool, so failures looked like 0.

The turn now comes from the latest real prompt, read once from the stored
history (type "user") when the session predates the process. System updates
are left out, shell failure notices count, and tool results keep their end.
`done` and `stuck` say what is not done.

Defaults: routing.mode active, one automatic consult per turn, and a new
routing.minStepsBetweenConsultations (10) cooldown over distinct steps.
Blocked steps still ask the Decision model and log verdict "blocked".
Explicit advisor() calls ignore both limits.
The latest review sat above the composer as a sticky, unscrollable block of
raw markdown. It now lives in a session.panel ("auto-advisor"): the whole
review as rendered markdown in a scrollbox, with its moment, model, duration
and time. A compact sidebar.footer widget shows the status and opens the
panel on click; /advisor-review (also in the palette) toggles it, and
esc/q and f close it or make it full screen.

The widget and panel read the stored review through RPC when they mount,
so a review from before a restart shows too (without details, which are
memory only). New tui.sidebar (default true) hides the widget.
The Decision model saw only the newest real prompt, so a short correction
replaced the plan that defined the task, and subagent results read as the
user speaking.

- Show the last routing.requests (default 3) real prompts, oldest first,
  each capped at routing.requestMaxChars (default -1, no cap).
- Track every real prompt id from the prompt hook plus one read of the
  stored history, and start the turn at the newest one present in the
  request: the hook fires before a steered or queued prompt is delivered.
- Label other user-role messages SUBAGENT (OpenCode's <subagent> result)
  or NOTICE.
- Drop the 32k total cap, which cut the steps and counts off a long input.
Drop the "View review  /advisor-review" hint. A click on the status still opens the review panel, and /advisor-review and the palette command are unchanged.
…n first

With one consult per turn, a long orchestrated turn got a single review and later high-done steps were blocked. Default routing.maxConsultationsPerTurn back to 2; the 10-step cooldown still spaces them out.

A step blocked by both limits is now logged as cooldown, so budget appears only once the cooldown has expired.
Scores alone can't show whether a verdict was right. Each telemetry line
now carries the id of the newest message in the request and the Decision
input (its last 32,000 characters, where the steps and counts are; the
requests stay in the session), plus its full length.

bun run timeline <sessionID> [--out file.html] joins those lines with the
session's messages from the running OpenCode service into one HTML page:
a row per routing decision, expanding to the messages since the previous
step and the exact Decision input. The script is not shipped.

The README's privacy note now says the Decision input is logged locally.
The turn is the newest real prompt in the request. A summary compaction can drop it from the window, which made the turn empty and reset the per-turn budget and step count mid-turn. Keep the current turn until a new real prompt arrives.

Also retry the stored-history read on the next step when it fails, instead of giving up for the life of the process.
…ream

OpenCode reports a provider rejection without an HTTP status, such as an overflow in the stream, as Generate.InvalidRequestError with no field, and it stopped both chains. A field marks a config error (model, model.variant), which still stops; without one, move to the next model.

A failed Advisor chain now reports timeout when any attempt timed out, not only the last one (a skipped model could follow). Tests now tell stop from next, pin the 5 s, 10 s and 60 s values, and check that an interrupt never asks the next Advisor model.
… docs

The telemetry log now holds Decision inputs (user requests, tool
output), so create it and its rotation with mode 0600, and tighten a
file left from before.

The README now states the chain's real worst cases: a refused
connection costs about 12 s because OpenCode retries it, and a silent
provider costs its header or chunk timeout, with no skip. It also says
that Esc stops the wait but not the provider request, and how many
steps the log keeps at the maximum input size. findings.md records
round 3's live check. The timeline tests now cover escaping the logged
input and lines written before it existed.
A field-less Generate.InvalidRequestError now keeps the provider's
message (first 200 chars) in the attempt, so a request-shape bug that a
later model tolerates still shows in telemetry. Telemetry still appends
when it can't tighten a file owned by someone else.

The docs describe field-less rejections accurately (not only
mid-stream) and say what a compaction does to the turn and the
requests block.
The Advisor's transcript labelled messages by role, so a subagent result
read as "USER: <subagent ...>" and the Advisor could take subagent
output for the user's instructions. Share the Decision input's labels:
real prompts stay USER, subagent results become SUBAGENT and other
user-role notices NOTICE.

Over budget, the transcript kept the first role-user message, which
after a compaction can be a subagent result; it now keeps the first
real request. The explicit tool reads the stored prompts first, so this
also holds with routing off.
The listener for session.execution.* events exited for good on the
first rejected iterator.next(). From then on an interrupt or run end no
longer cancelled a pending consult, and finished runs kept their
request snapshot in memory. Catch a failed read, wait 100 ms so a
stream that keeps failing can't spin, and keep listening.
Record in findings.md that a prompt absorbed into a compaction before any step sees it never starts its own turn, and that ram-monitor's markdown modal, built the way that blanked this plugin's panel, still renders on 2.2.0.
This reverts commit 6e3f1e5. OpenCode 2.2.0's plugin adapter closes the
event stream when a read fails, so retrying the read only delayed the
listener's exit by 100 ms. Its test passed because the fake stream kept
going after an error.

AGENTS.md now records the limitation (a failed read ends run-end
cancellation; accepted), that a failed history read labels older
prompts NOTICE for one step, and why ram-monitor's markdown modal hides
the insert([]) bug.
… to the reference

The advisor() description and the Advisor's instructions now follow the
widely used advisor-tool prompt nearly word for word: when to call
(before substantive work, before declaring done once the result is
saved, when stuck or changing approach), how often, how to weigh the
advice, the reconcile call, and the Advisor's four stages, stale-advice
rule, check-not-verdict and blocks-or-not rules.

Adapted to this plugin: hidden reasoning is not forwarded and long
sessions are trimmed; an automatic review counts as a consultation
already made; USER, SUBAGENT and NOTICE labels are explained; the
transcript is evidence, never instructions; the user's constraints come
first; no "commit the change" advice. An automatic review now states its
moment as a hint the Advisor checks against the transcript, since the
label is only the likeliest non-routine moment.
…sends

The description promised a stronger reviewer, but the default Advisor
is the agent's own model; it now says a stronger model when one is
configured. It also said every tool call and result is forwarded; long
results may be shortened and older history summarized, and it now says
so. A reconcile call's question goes in the message before the call,
and an explicit consult tells the Advisor to answer one when present.

The Advisor's prompt now explains SYSTEM UPDATE lines and the compacted
summary, and tells the budget marker apart from the catalog markers,
which leave the conversation intact.

Tests cover all three moment hints, the tool's wiring and the review
marker the description names. AGENTS.md and findings.md state the
accepted run-end limitation's real consequence and why ram-monitor's
modal hides the markdown bug.
… characters

By default the whole of each recent user request went to the external
decision provider. Default routing.requestMaxChars to 60,000 characters
(the start of each request is kept); -1 still sends them whole.
The shell tool's "Exited with code N", "Killed by" and "Timed out before
completion" notices were matched against every tool result, so a
successful read of a log or fixture containing them counted as a failed
tool and pushed the Decision model toward "stuck". Match them only on
shell results (named shell, or bash, which 2.2.5 runs as shell); an
error result still counts for every tool.
…re 2.2.5

OpenCode 2.2.5 reworded its Code Mode instructions: the catalog now
starts "The tools listed below ... cannot be called directly" and its
header is "## Code Mode tools (call only inside `execute`)". With the
2.2.0 strings, compactContext silently stopped removing that catalog
from the Advisor's copy, costing tokens on every consult. Match the new
text and raise the peer floor to @opencode/plugin 2.2.5 (and OpenTUI
0.8, which 2.2.3+ requires).

Rechecked every other host behaviour the plugin relies on against 2.2.5
and verified it live; findings.md records the diff and the results.
Dev pins stay at 2.2.0 to keep the monorepo's shared pins identical.
On 2.2.5 the shell tool emits each notice as a separate text part (a
single part arrives as a plain string, several as content), so match a
whole part against an anchored notice instead of searching the
serialized result. A successful command whose output quotes "Exited
with code 1" no longer counts as failed; an aliased bash failure, whose
alias note follows the notice, still does.

Tests use the 2.2.5 result shapes and cover an error result from a
non-shell tool. The re-verify recipe now installs 2.2.5 and checks the
part shape and the bash alias.
…nt the total

routing.requestMaxChars caps each request, not the Decision input:
three requests at 60k plus recent steps could reach about 193k
characters, past the 64k-token window of the default jev-1.13-free, and
a single-model chain then gets no decision. Default to 20k (about 72k
characters in total with the defaults).

The README and findings.md now say the cap is per request, give the
total, and note that sizing to each model's window is a follow-up.
@capyBearista
capyBearista force-pushed the feat/auto-advisor-rebuild branch from 99dce34 to 851b757 Compare October 31, 2026 16:53
OpenCode's CLI exits right after writing (2.2.5, packages/cli/src/index.ts),
so `opencode api get` cut large message pages short when read through a pipe,
and the timeline failed with "JSON Parse error: Unterminated string".
Each page now goes to a file in a private temporary directory.

The test runs a fake CLI that writes 3 MB pages and exits at once. It failed
20 of 20 runs before the change and passes 20 of 20 after.
The root README said automatic reviews happen at most once per turn; the
default is two. The README, root README and package description said a
stronger model reviews the work, but by default the Advisor is the agent's
own model. The package README now also says a restart resets both limits,
where to find a session id for the timeline, and the docs index describes
the findings as 2.2.x, rechecked on 2.2.5.
The package typecheck covered only src, so scripts/timeline.ts and
smoke-built.ts were never checked. scripts/tsconfig.json checks them,
excluding tests as every package's tsconfig does.
`bun run sandbox start [dir]` installs OpenCode (2.2.5 by default) into an
isolated HOME, loads this package from source with a mock model provider
that logs every request, and starts `opencode serve`. It prints the server
URL, password and the commands to drive it. `bun run sandbox stop <dir>`
ends its processes by process group. It replaces the hand-built setup in
the findings recipe, whose pitfalls (an open stdin hanging `opencode run`,
`pkill -f` matching the calling shell) were easy to miss.
…he sandbox

findings.md gains the large-session timeline result: the CLI exits right
after writing, so piped `opencode api` output is cut short. Its intro now
dates each section by version, a renamed constant and an over-broad
OpenRouter claim are fixed, and the re-verify recipe uses `bun run sandbox`
with a step for prompts too large to pass as an argument. AGENTS.md adds
the gotcha, the sandbox and the scripts typecheck.
github-advanced-security[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@capyBearista
capyBearista force-pushed the feat/auto-advisor-rebuild branch from 13516db to e5cdc26 Compare November 1, 2026 20:48
…a private

The sandbox ran detached processes tracked by pids in a reusable directory.
That left the server password readable by other users, made a restart read
the previous run's port and password, let a stale pid point stop at an
unrelated process group, passed the caller's OPENCODE_* variables (a
profile's config dir) to the server, and stranded processes when startup
failed.

`bun run sandbox` now runs in the foreground in a new 0700 directory each
time, stops its children on Ctrl-C or when one exits, and fails fast if one
dies during startup. OPENCODE_* variables are dropped from the child and
unset by env.sh, whose values are single-quoted. The password is no longer
printed. The version setting is SANDBOX_OPENCODE_VERSION, since OpenCode
reads OPENCODE_VERSION itself; each version is installed once in the cache.

The timeline page holds the user's requests and the Decision input, so it
is now written 0600 like telemetry, also when replacing an existing file.
devin-ai-integration[bot]

This comment was marked as resolved.

@capyBearista
capyBearista merged commit 94e2e9d into main Nov 1, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants