Skip to content

feat(autopilot): Autopilot is configured in Settings and waits out plan limits - #279

Merged
hoangsnowy merged 1 commit into
mainfrom
feat/autopilot-settings-plan-limits
Oct 23, 2026
Merged

hoangsnowy merged 1 commit into
mainfrom
feat/autopilot-settings-plan-limits

Conversation

@hoangsnowy

Copy link
Copy Markdown
Owner

Change description

Autopilot is now configured from the desktop, and it does what its screen says.

1. Constants are now settings (ADR-0011, AutopilotSettings in Modules.Pipeline, tenant scope, read on every pass or call):

Key Default Range
Autopilot:MaxParallel (Tasks at once) 2 1–6
Autopilot:MaxParallelTests (Acceptance tests at once, per computer) 1 1–3
Autopilot:MaxFixRounds (Automatic fix rounds) 3 0–10
Autopilot:MaxAgentRetries (Orchestrator retries) 2 0–5
Autopilot:ChecksTimeoutMinutes 60 5–240
Autopilot:ChecksPollSeconds 60 15–600

AutopilotPlanner is still pure. The caps reach it as an AutopilotLimits parameter, and AutopilotSettings.ReadLimitsAsync clamps every stored value to its range. The acceptance-run cap used to be one global SemaphoreSlim(1). It is now a keyed RunGate, one per member's computer, whose capacity is read each time a run asks for it.

2. Settings → AI → Autopilot page (AutopilotSettingsApp.razor). It is built from SettingsFields, Panel, page-head and prefs-*, using tokens only. It has one new class, .autopilot-rules, defined in app.css. The page is registered in SettingsHub and in SettingsSearch/SettingsLinks, so Ctrl/⌘+K finds "fix rounds", "decided by" and "autopilot". For admins, Board › Autopilot now has an Autopilot settings button. Members see a line saying an admin sets these values.

3. Board copy shows live values and the real rule. It now reads "start approved tickets in order, three at a time … (up to 3 rounds per task)". Build and merge reads: "once its checks are green, the Reviewer agent approved it and the Testing agent passed it on your connected computer". The log line written when Autopilot is switched on was corrected too.

4. One fix-round counter. ProjectOrchestrator._fixRounds (kept in memory) is gone. The step-by-step path reads the persisted RemoteSessionEntity.FixRounds. A ticket's MAF flow now starts its TicketFlow.Round from the same number (TicketWorkflowRunner.Seed) and caps at TicketFlow.MaxRounds. Agent retries no longer reset the rules' rounds. The worst case is now MaxFixRounds + MaxAgentRetries tries in total, across engines and restarts. Before, it was about 3+3+2.

5. Plan limits.

  • PlanLimitDetector (in Modules.RemoteAgent, using source-generated regex) reads the tail of a runner reply. It reads a clean reply only when the reply is short. It handles:
    • Claude's You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)
    • weekly limits with a date
    • the older usage limit reached|<epoch>
    • Codex's try again in 2 hours 5 minutes
    • a reset time with no zone, which is still a limit but with an unknown reset
  • RemoteAgentLlmClient throws a typed PlanLimitException (an LlmException, so existing catches keep working). It also records the member's wait in IPlanLimits (PlanLimitTracker): reset + 1 minute, or 30 minutes when no time is given.
  • While a wait is active, ProjectOrchestrator starts nothing, retries nothing and sends nothing back. PlanLimitPause logs one line when the wait starts ("Your computer's Claude plan hit its limit — Autopilot waits until 23:50, nothing is lost.") and one when it ends. The wait is checked again after the Orchestrator agent decides, and the "agent couldn't decide → rules" fallback is skipped while a plan is limited.
  • TicketWorkflow pauses with a new stage, PlanLimited, before or after an agent step, without spending a round.
  • When a follow-up is cut short by the limit, BoardTaskService puts the change back to where it stood (its checks and findings are kept) and calls DecrementFixRoundsForTenantAsync. A first attempt that is cut short stays Failed with the CLI's text, and is not counted as stuck work for the Orchestrator agent.
  • After the reset, the planner's new ResumeTask move restarts that work plainly, so no fix round is used.
  • The Board shows the wait on the Autopilot tab (a warn-styled card) and on the Computers tab (a Warn badge reading "Claude plan limit · until 23:50"). The task failure card has a new plan-limit kind.
  • The wait is kept in memory. After a restart, the next run meets the limit again, which is an immediate answer, and the wait comes back. This is documented in CONFIG.md.

Also fixed: the analyzer warnings in the Web files this PR touches (CA2016 ×10, CA1834 ×3, CA1068, CA1001, CS8619, IDE0004). Web builds with TreatWarningsAsErrors=false, so these had been piling up there.

Docs: docs/CONFIG.md has a new Settings → Autopilot section. docs/plans/autonomy-levels.md is updated, with a new 4b for plan limits. README and design-system.md are updated for the new rail entry.

Not touched: LlmClientFactory, LlmCallScope, OfflineLlmClient, FailoverLlmClient, the Review/Acceptance/AutopilotOrchestrator/Retrospective/IssueWork agent internals, AgentProfiles/AgentSettings, PromptCatalog and InsightsApp.

Type of change

  • New feature (new agent / endpoint / capability)
  • Bugfix
  • Refactor (no behavior change)
  • Documentation / README
  • Infrastructure / CI configuration

Related issues

Autopilot audit: hard-coded caps, no Settings › Autopilot page, static Board copy, three separate fix-round counters, and plan limits that burned retries.

Checklist

  • dotnet build passes locally in Release mode. dotnet build AgentOs.slnx -c Release --no-incremental gives 0 errors. The only warning is the existing ASPIRE010 from infra/AgentOs.AppHost (AspireUseCliBundle=false), which this PR doesn't touch.
  • dotnet test --solution AgentOs.slnx -c Release: 1678 total, 0 failed (78 skipped, all env-gated E2E/lab tests).
  • No secrets committed
  • README / docs updated
  • Measurable results: plan-limit waits are logged once per wait in the Autopilot activity. Fix rounds stay the persisted metric that Insights › Delivery reads.

Test plan

Unit tests (new):

  • PlanLimitTests: detection on real-shaped strings, including resets that roll over to tomorrow or next year, the "exit N:" prefix, epoch, Codex durations, no zone, negatives, and the clean-reply length guard. Also the tracker's wait (reset + 1 min), isolation per member and per tenant, and the provider throwing PlanLimitException and recording the wait.
  • AutopilotConfigurationTests: descriptors, ranges and defaults; ReadLimitsAsync defaults and clamping; RetriesLeft with the configured budget; the pause logging once at start and once at end; CutShortByPlanLimit; RunGate cap per computer.
  • AutopilotPlannerTests: the configured slots, fix rounds and acceptance cap; WithinCaps limits; cut-short work resumes first, within the slots.
  • TicketWorkflowTests: rounds seeded from the persisted FixRounds (persisted 2 + cap 3 = exactly one round left); the configured max; the plan limit pausing the flow without spending a round, including before the first step.
  • TaskFailureTests, SettingsSearchTests: new cases.

E2E (Playwright, standalone Web): new AutopilotSettingsTests, 3/3 passed. It checks that every setting renders with its default, and that the merge and plan-limit rules are stated. It saves Tasks at once = 99 (rejected, and the reason names the bound 6), then 3 (shows "This tenant"), then resets in a finally. Ctrl+K "fix rounds" opens the page. SettingsSearchTests, PlatformSettingsTests and BoardTabsTests still pass.

In the running app (screenshots taken locally, not committed):

  • Settings › Autopilot, light and dark, in the standalone Web: the AI rail shows "Autopilot" under Prompts. Panels: Who decides (Decided by = Orchestrator agent, Each ticket runs as = MAF workflow); Limits (2 / 1 / 3 / 2, each with a "Default" badge and help text); Waiting for checks (60 / 60); Rules that always apply (merging, order, plan limits). Legible in both themes.
  • Board › Autopilot, light and dark, Web plus a throwaway Postgres container: one board, set to Build and merge. Tasks at once was set to 3 in Settings, and the Build option reads "three at a time … (up to 3 rounds per task)". The Build and merge option states the full merge rule. A warn-bordered card reads "Waiting for the AI plan — Your computer's Claude plan hit its limit — Autopilot waits until 23:50, nothing is lost." The same sentence appears exactly once at the top of the activity log. There is an "Autopilot settings" button next to Check now.
  • Board › Computers, dark: the paired computer shows "Connected" plus a yellow badge, "Claude plan limit · until 23:50".
  • The plan limit was produced end to end. A real runner (AgentOs.RemoteAgent, configured through environment variables only, nothing saved) was paired to that Web. Its CLI was a stand-in script that prints You've hit your session limit - resets 11:50pm (Asia/Ho_Chi_Minh) and exits 1. Force provider was set to RemoteAgent, and Board › New idea was triggered. The Web logged "the member's Claude plan hit its limit (resets 23:50)". The next Autopilot pass wrote the single wait line to workspaces.autopilot_events.

Not verified here: a real Claude/Codex plan limit on the lab server, and an actual resume after a real reset. Both need a used-up subscription. The resume path is covered by the planner and flow tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_018WAdQ7Rm24xss5QDSvXtJB

…an limits

- Settings › AI › Autopilot: a hub page built from the Autopilot:* descriptors
  (Decided by, Each ticket runs as, and six bounded new settings: MaxParallel,
  MaxParallelTests, MaxFixRounds, MaxAgentRetries, ChecksTimeoutMinutes,
  ChecksPollSeconds), plus the fixed rules in plain words. Found by Ctrl+K;
  Board › Autopilot links to it.
- The bounds are read per pass / per call; AutopilotPlanner stays pure and takes
  them as an AutopilotLimits parameter. The acceptance-run gate honours the cap
  per member's computer (RunGate) instead of one global semaphore.
- One fix-round counter: the persisted RemoteSessionEntity.FixRounds. The
  step-by-step path reads it (the in-memory _fixRounds is gone) and a ticket's
  MAF flow seeds TicketFlow.Round from it, so the configured max is the true
  total across engines and restarts.
- Plan limits: the RemoteAgent provider detects a Claude/Codex plan limit in the
  runner's reply (PlanLimitDetector, reset time when parseable) and raises a
  typed PlanLimitException + records the member's wait (IPlanLimits). Autopilot
  then starts nothing, retries nothing and spends no fix round until reset +1
  min, logs one plain line, shows it on Board › Autopilot and Computers, and
  resumes the cut-short work (ResumeTask). A follow-up cut short by the limit is
  restored to where it stood and its fix round is taken back.
- Board copy renders the live values and the real merge rule (green checks +
  review approved + acceptance passed on a connected computer).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018WAdQ7Rm24xss5QDSvXtJB
@hoangsnowy
hoangsnowy merged commit 52a4959 into main Oct 23, 2026
4 checks passed
hoangsnowy added a commit that referenced this pull request Oct 24, 2026
AgentOs.Web opted out of TreatWarningsAsErrors (since the 2026-05 rename), so analyzer
warnings piled up there while the rest of the solution enforced them. #279 cleared the
backlog; the Web project now builds with 0 warnings, so the opt-out goes.


Claude-Session: https://claude.ai/code/session_018WAdQ7Rm24xss5QDSvXtJB

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
hoangsnowy added a commit that referenced this pull request Oct 24, 2026
Pushed settings-autopilot (new, #279) and replaced agent-profiles, settings-prompts (#278)
and board-autopilot (#279) in the AgentOS.NET Design System project.


Claude-Session: https://claude.ai/code/session_018WAdQ7Rm24xss5QDSvXtJB

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant