diff --git a/README.md b/README.md index 97144301..17660f65 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ All shots are the standalone Web (`dotnet run --project src/AgentOs.Web`). | Boards → Tickets → Sessions | Settings | |:---:|:---:| | [![Board — connect a GitHub/ADO board and drive remote repo execution](docs/images/app-board.png)](docs/images/app-board.png) | [![Settings — agent profiles, providers, governance, platform](docs/images/app-settings.png)](docs/images/app-settings.png) | -| Connect a GitHub / Azure DevOps board, decompose an idea into tickets, drive repo-execution sessions. | One searchable rail: AI (Model keys, Routing, Agent profiles, Prompts) · Connections · Governance · People & data · Personal · System. Ctrl+K finds any setting. Changes apply on the next call. | +| Connect a GitHub / Azure DevOps board, decompose an idea into tickets, drive repo-execution sessions. | One searchable rail: AI (Model keys, Routing, Agent profiles, Prompts, Autopilot) · Connections · Governance · People & data · Personal · System. Ctrl+K finds any setting. Changes apply on the next call. | | Platform governance | Terminal — runner session feed | |:---:|:---:| @@ -319,6 +319,10 @@ in the desktop and applies on the next call: - **Settings → Platform** (`platform-admin` only) — remote auto-approve, fail-closed tool policy, allowed self-hosted Git hosts, build sandbox (mode, image, quotas), offline fallback. - **Settings → Prompts** — per-tenant system prompts for the Requirement, Coding, Testing and QA agents. +- **Settings → Autopilot** — how Autopilot runs a project (Board › Autopilot): who decides (Orchestrator agent or the + fixed rules), how each ticket runs (MAF workflow or step by step), tasks and acceptance tests at once, automatic fix + rounds (one count per task across engines and restarts), Orchestrator retries, and the checks wait. When your + computer's Claude / Codex plan hits its limit, Autopilot waits for the reset — no tries spent — and carries on. - **Settings → Audit log** — who changed what (settings, keys — never their values —, members, sign-in), CSV export. **Your repo's own rules.** When a Board ticket runs against a repo, every agent follows the repo's agent diff --git a/docs/CONFIG.md b/docs/CONFIG.md index 6fd56ea5..1a0a03e1 100644 --- a/docs/CONFIG.md +++ b/docs/CONFIG.md @@ -116,6 +116,31 @@ for models that reject sampling params; `MaxTokens` ≤ 128 000. `Pipeline:MaxIterations` (QA-loop cap when a story sets no `NMax`) — default 3, range 1–10. An unpriced model is charged at the fail-closed $15/$75 per 1M rate until added to `CostCalculator`. +### Settings → Autopilot + +How Autopilot runs a project once a person turns it on in **Board › Autopilot**. Tenant scope (admins); read on every +pass / call, so a change applies within a minute, no restart. The Board's Autopilot tab shows the live values. + +| Key | Default | Range | Notes | +|---|---|---|---| +| `Autopilot:DecidedBy` | `Orchestrator agent` | `Orchestrator agent` / `Rules` | Who picks the next moves. The agent works within the rules; `Rules` = the fixed rules alone (saves the member's plan). | +| `Autopilot:TicketEngine` | `MAF workflow` | `MAF workflow` / `Step by step` | How each started ticket runs: one Microsoft Agent Framework graph per ticket, or Autopilot driving each step every minute. | +| `Autopilot:MaxParallel` | `2` | 1–6 | Tasks running at once on a board (coding, fixing, reviewing, testing). | +| `Autopilot:MaxParallelTests` | `1` | 1–3 | Acceptance runs at once **per member's computer** (each brings the app up on its fixed ports). | +| `Autopilot:MaxFixRounds` | `3` | 0–10 | Times a change goes back to the Coding agent (red checks, conflicts, review / acceptance findings) before a person is asked. Counted **per task in total** from the persisted `FixRounds` — the same count in both engines and across restarts. | +| `Autopilot:MaxAgentRetries` | `2` | 0–5 | Extra guided retries the Orchestrator agent may give a stuck task after the fix rounds are used up. | +| `Autopilot:ChecksTimeoutMinutes` | `60` | 5–240 | How long a ticket's MAF workflow waits for a pull request's checks before handing it back to the step-by-step passes. | +| `Autopilot:ChecksPollSeconds` | `60` | 15–600 | How often it looks at them. | + +Fixed rules (not settings): *Build and merge* merges a change only when its checks are green, the Reviewer agent +approved it and the Testing agent passed it — which needs the member's computer connected. When that computer's +Claude / Codex **plan hits its limit** (e.g. `You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)`), the +RemoteAgent provider raises a typed `PlanLimitException` and records the member's wait (`IPlanLimits`); Autopilot +then starts nothing, retries nothing and spends no fix round until the reset (+1 minute; 30 minutes when the CLI gives +no time), logs one line in the Autopilot activity, shows it on Board › Autopilot and Board › Computers, and picks the +cut-short work up again afterwards. The wait is in memory: after a restart the next run meets the limit again (an +immediate answer) and the wait is back. + ### Settings → Platform (platform admins) **Run limits (per tenant)** — `Pipeline:Limits:MaxConcurrentRuns` (default `4`) and `Pipeline:Limits:RunsPerMinute` diff --git a/docs/design/design-system.md b/docs/design/design-system.md index c2e3ffa4..8540e9cc 100644 --- a/docs/design/design-system.md +++ b/docs/design/design-system.md @@ -181,7 +181,7 @@ in `app.css`; the gallery of every role × state is on `/_/components`. on focus; CSD drag/resize/maximize; grab a maximized window's titlebar to restore it under the cursor. - **Settings hub** (`SettingsHub.razor`, pinned "Settings") — every setting behind **one rail** (GNOME Settings style, never a rail inside a rail): a search field, then pages under group headings — **AI** - (Model keys, Routing, Agent profiles, Prompts) · **Connections** (GitHub, MCP servers, Plugins) · + (Model keys, Routing, Agent profiles, Prompts, Autopilot) · **Connections** (GitHub, MCP servers, Plugins) · **Governance** (Tool policy, Activity log — changes & sign-in | tool calls) · **People & data** (Users, API tokens, Data) · **Personal** (General, Appearance, Date & time, Notifications, Session) · **System** (About, Diagnostics, Platform). Deep-link with `SettingsLinks.Open(WM, "")` (e.g. `keys`; legacy `llm/keys` still diff --git a/docs/plans/autonomy-levels.md b/docs/plans/autonomy-levels.md index 2856a3c3..2c6cb2bc 100644 --- a/docs/plans/autonomy-levels.md +++ b/docs/plans/autonomy-levels.md @@ -35,25 +35,32 @@ others learned. The board shows each ticket's phase (Discovery → Coding → Te approved* starts only tickets whose dependencies are merged, at most N at a time, and re-checks when one merges. Today it starts everything at once — #12 (import) would race #11 (the module it needs). 2. ✅ (#242, Autopilot) **Self-repair on CI.** When a task's pull request goes red, run *Fix the failing checks* automatically, at most - 3 rounds, then escalate (a failure card with "This needs a person"). + *Automatic fix rounds* (Settings › Autopilot, default 3), then escalate (a failure card with "This needs a person"). + The rounds are the task's persisted `FixRounds` — one count for both engines and across restarts. 3. ✅ (#244) **Acceptance agent.** What a human reviewer did by hand on the lab: read the ticket's criteria and the project's `docs/ACCEPTANCE.md`, run the unit tests, bring the stack up and run the Playwright specs, run data checks (e.g. the CBL corpus), and post the evidence on the PR. Pass → approve; fail → *Ask for changes* with the findings. 3c. ✅ (#270) **One ticket = one MAF workflow.** The Autopilot Orchestrator agent decides which ticket starts; a Microsoft Agent Framework Workflow (`TicketWorkflow`) then runs it — Coding → checks → Reviewer → Testing → merge — - with conditional edges back to Coding (red checks, conflicts, requested changes, failed acceptance; 3 rounds). A - flow that ends without a merge, or one a restart cut short, is picked up step by step. Settings › Autopilot › *Each - ticket runs as* (MAF workflow / Step by step). + with conditional edges back to Coding (red checks, conflicts, requested changes, failed acceptance; the same fix + rounds, seeded from the task's persisted count). A flow that ends without a merge, or one a restart cut short, is + picked up step by step. Settings › Autopilot › *Each ticket runs as* (MAF workflow / Step by step); its checks wait + and poll are settings too. 3b. ✅ (#264) **Reviewer agent.** Between green CI and acceptance testing: a read-only agent reviews the diff in - context and posts its review on the PR; must-fix findings go back to the Coding agent (same 3 bounded rounds). -4. ✅ (#242, *Build and merge*: on green CI; acceptance agent next) **Merge policy.** Per project: merge on green + acceptance pass (squash, delete branch), or wait for a person. + context and posts its review on the PR; must-fix findings go back to the Coding agent (same bounded rounds). +4. ✅ (#242, #264, #244) **Merge policy.** Per project: *Build and merge* merges only on green checks + review approved + + acceptance passed (which needs the member's computer connected) — squash, delete branch — or *Build* waits for a person. +4b. ✅ **Plan limits.** When the member's computer answers with its Claude / Codex plan limit, Autopilot waits until the + reset (+1 min) — no new work, no retries, no fix rounds spent — says so once in the Autopilot activity and on the + Board (Autopilot + Computers tabs), and picks the cut-short work up again afterwards. 5. **Environment care.** The known stack landmines (stale Keycloak/Postgres volumes, orphaned containers, ports) handled by the acceptance agent's stack step, not by a person. ## What L5 adds - **Autopilot** per board: a loop that runs 1–4 whenever there is a ready ticket, inside a monthly budget and a - parallelism cap, with Stop. + parallelism cap (Settings › Autopilot: tasks at once, acceptance tests at once, fix rounds, Orchestrator retries, + checks wait), with Stop. - **Escalation** in words: a "Needs you" badge + notification with the one decision required. - **Report**: a daily summary per project — merged, in progress, blocked and why, spend. diff --git a/src/AgentOs.Modules.Pipeline/Agents/AutopilotSettings.cs b/src/AgentOs.Modules.Pipeline/Agents/AutopilotSettings.cs index d08ed17d..0787c6a6 100644 --- a/src/AgentOs.Modules.Pipeline/Agents/AutopilotSettings.cs +++ b/src/AgentOs.Modules.Pipeline/Agents/AutopilotSettings.cs @@ -1,14 +1,45 @@ -// Who decides Autopilot's next moves on a board (ADR-0011: a UI setting, per tenant). The Orchestrator agent by -// default; "Rules" keeps the fixed rules only (the agent's guardrails), e.g. to save the member's subscription. +// How Autopilot runs a board (ADR-0011: UI settings, per tenant — Settings › Autopilot). Who decides the next moves +// (the Orchestrator agent by default; "Rules" keeps the fixed rules only, e.g. to save the member's subscription), how +// each ticket runs (a MAF workflow, or step by step), and the bounds every engine keeps: tasks at once, acceptance runs +// at once, automatic fix rounds, the Orchestrator agent's extra retries, and how long to wait for a pull request's +// checks. Read per pass / per call, so a change applies to the next pass without a restart. +using System; using System.Collections.Generic; +using System.Globalization; +using System.Threading; +using System.Threading.Tasks; using AgentOs.Domain.Configuration; namespace AgentOs.Modules.Pipeline.Agents; -/// Declares Autopilot:DecidedBy. +/// The bounds Autopilot keeps on a board, as configured (). +/// Tasks running at once on a board. +/// Acceptance runs at once on a member's computer (each brings the app up on fixed ports). +/// Automatic "send it back to the Coding agent" rounds before a person is asked — the total +/// across engines and restarts (a task's persisted fix rounds count against it). +/// Extra retries the Orchestrator agent may spend on a stuck task, with guidance. +/// How long a ticket's MAF workflow waits for a pull request's checks. +/// How often it looks at them. +public sealed record AutopilotLimits(int MaxParallel, int MaxParallelTests, int MaxFixRounds, int MaxAgentRetries, TimeSpan ChecksTimeout, TimeSpan ChecksPoll) +{ + /// The built-in values (the settings' defaults). + public static AutopilotLimits Default { get; } = new( + AutopilotSettings.DefaultMaxParallel, AutopilotSettings.DefaultMaxParallelTests, AutopilotSettings.DefaultMaxFixRounds, + AutopilotSettings.DefaultMaxAgentRetries, TimeSpan.FromMinutes(AutopilotSettings.DefaultChecksTimeoutMinutes), + TimeSpan.FromSeconds(AutopilotSettings.DefaultChecksPollSeconds)); + + /// These limits with no slot caps — every legal move is offered (the Orchestrator agent picks, then + /// the caps are applied to its picks). + public AutopilotLimits Uncapped => this with { MaxParallel = int.MaxValue, MaxParallelTests = int.MaxValue }; +} + +/// Declares the Autopilot:* settings. public sealed class AutopilotSettings : ISettingsContributor { + /// The settings category (and the Settings › Autopilot page). + public const string Category = "Autopilot"; + /// The setting key. public const string DecidedByKey = "Autopilot:DecidedBy"; @@ -27,26 +58,99 @@ public sealed class AutopilotSettings : ISettingsContributor /// Autopilot drives each step itself, pass by pass. public const string StepByStep = "Step by step"; + /// Tasks running at once on a board. + public const string MaxParallelKey = "Autopilot:MaxParallel"; + + /// Acceptance runs at once on a member's computer. + public const string MaxParallelTestsKey = "Autopilot:MaxParallelTests"; + + /// Automatic fix rounds per task. + public const string MaxFixRoundsKey = "Autopilot:MaxFixRounds"; + + /// The Orchestrator agent's extra retries per stuck task. + public const string MaxAgentRetriesKey = "Autopilot:MaxAgentRetries"; + + /// How long to wait for a pull request's checks (minutes). + public const string ChecksTimeoutMinutesKey = "Autopilot:ChecksTimeoutMinutes"; + + /// How often to look at them (seconds). + public const string ChecksPollSecondsKey = "Autopilot:ChecksPollSeconds"; + + internal const int DefaultMaxParallel = 2; + internal const int DefaultMaxParallelTests = 1; + internal const int DefaultMaxFixRounds = 3; + internal const int DefaultMaxAgentRetries = 2; + internal const int DefaultChecksTimeoutMinutes = 60; + internal const int DefaultChecksPollSeconds = 60; + + /// Each bound's key, range and default — the descriptors and the typed read share it. + private static readonly (string Key, int Min, int Max, int Default)[] Bounds = + [ + (MaxParallelKey, 1, 6, DefaultMaxParallel), + (MaxParallelTestsKey, 1, 3, DefaultMaxParallelTests), + (MaxFixRoundsKey, 0, 10, DefaultMaxFixRounds), + (MaxAgentRetriesKey, 0, 5, DefaultMaxAgentRetries), + (ChecksTimeoutMinutesKey, 5, 240, DefaultChecksTimeoutMinutes), + (ChecksPollSecondsKey, 15, 600, DefaultChecksPollSeconds), + ]; + /// True when the stored value means a MAF workflow (anything but "Step by step"). - public static bool UsesMafWorkflow(string? value) => !string.Equals(value?.Trim(), StepByStep, System.StringComparison.OrdinalIgnoreCase); + public static bool UsesMafWorkflow(string? value) => !string.Equals(value?.Trim(), StepByStep, StringComparison.OrdinalIgnoreCase); /// public IEnumerable Describe() { - yield return new(TicketEngineKey, SettingKind.Choice, "Autopilot", "Each ticket runs as") + yield return new(DecidedByKey, SettingKind.Choice, Category, "Decided by") + { + DefaultValue = Agent, + Choices = [Agent, Rules], + Help = "Orchestrator agent: an AI lead reads the whole project and picks the next moves, retries stuck tasks with guidance and asks you precise questions — always within the rules. Rules: the fixed rules alone.", + }; + yield return new(TicketEngineKey, SettingKind.Choice, Category, "Each ticket runs as") { DefaultValue = MafWorkflow, Choices = [MafWorkflow, StepByStep], Help = "MAF workflow: a Microsoft Agent Framework graph runs Coding → checks → Reviewer → Testing → merge for the ticket, routing every verdict. Step by step: Autopilot drives each step itself every minute. Both use the same agents, rules and evidence.", }; - yield return new(DecidedByKey, SettingKind.Choice, "Autopilot", "Decided by") + yield return Bound(MaxParallelKey, "Tasks at once", + "How many tickets the AI works on at the same time on a project (coding, fixing, reviewing or testing). Each one uses your computer."); + yield return Bound(MaxParallelTestsKey, "Acceptance tests at once", + "How many changes the Testing agent tries at the same time on one computer. Each brings the app up on its usual ports, so more than one only works when the app can run twice."); + yield return Bound(MaxFixRoundsKey, "Automatic fix rounds", + "How many times a change goes back to the Coding agent (red checks, a conflict, review or acceptance findings) before a person is asked. Counted per task, in total — a restart grants no new rounds."); + yield return Bound(MaxAgentRetriesKey, "Orchestrator retries", + "Extra tries the Orchestrator agent may give a stuck task, with its own guidance, after the automatic fix rounds are used up."); + yield return Bound(ChecksTimeoutMinutesKey, "Wait for checks (minutes)", + "How long a ticket's MAF workflow waits for the pull request's checks before handing it back to Autopilot's step-by-step passes."); + yield return Bound(ChecksPollSecondsKey, "Look at checks every (seconds)", + "How often the MAF workflow looks at the pull request's checks while it waits."); + } + + private static SettingDescriptor Bound(string key, string label, string help) + { + var b = Array.Find(Bounds, x => x.Key == key); + return new(key, SettingKind.WholeNumber, Category, label) { - DefaultValue = Agent, - Choices = [Agent, Rules], - Help = "Orchestrator agent: an AI lead reads the whole project and picks the next moves, retries stuck tasks with guidance and asks you precise questions — always within the rules. Rules: the fixed rules alone.", + DefaultValue = b.Default.ToString(CultureInfo.InvariantCulture), + Min = b.Min, + Max = b.Max, + Help = help, }; } /// True when the stored value means the agent decides (anything but "Rules"). - public static bool AgentDecides(string? value) => !string.Equals(value?.Trim(), Rules, System.StringComparison.OrdinalIgnoreCase); + public static bool AgentDecides(string? value) => !string.Equals(value?.Trim(), Rules, StringComparison.OrdinalIgnoreCase); + + /// The tenant's Autopilot bounds, read now (each clamped to its range — a seeded value can't escape it). + public static async ValueTask ReadLimitsAsync(ISettings settings, string tenantId, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(settings); + var v = new int[Bounds.Length]; + for (var i = 0; i < Bounds.Length; i++) + { + var (key, min, max, fallback) = Bounds[i]; + v[i] = Math.Clamp(await settings.GetIntAsync(key, tenantId, fallback, ct).ConfigureAwait(false), min, max); + } + return new AutopilotLimits(v[0], v[1], v[2], v[3], TimeSpan.FromMinutes(v[4]), TimeSpan.FromSeconds(v[5])); + } } diff --git a/src/AgentOs.Modules.RemoteAgent/PlanLimit.cs b/src/AgentOs.Modules.RemoteAgent/PlanLimit.cs new file mode 100644 index 00000000..9f2d5ff0 --- /dev/null +++ b/src/AgentOs.Modules.RemoteAgent/PlanLimit.cs @@ -0,0 +1,281 @@ +// The member's own AI plan (a Claude or Codex subscription on their paired computer) said "no more for now" — e.g. +// claude -p answering "You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)". That is not a failure of +// the work: retrying it, or sending it back to the Coding agent, only burns Autopilot's bounded tries while the plan +// is closed. So the RemoteAgent provider turns such a reply into a typed signal — a PlanLimit (with the reset time +// when the CLI says it), thrown as PlanLimitException and recorded per member in IPlanLimits — and Autopilot waits +// it out (ProjectOrchestrator) instead of spending retries. Detection is pure and tested against real-shaped CLI +// lines; it only looks at the tail of a reply, and only at a short one when the CLI exited cleanly, so code or an +// answer that merely mentions a "usage limit" is never mistaken for one. + +using System; +using System.Collections.Concurrent; +using System.Globalization; +using System.Text.RegularExpressions; +using AgentOs.Domain.Llm; + +namespace AgentOs.Modules.RemoteAgent; + +/// A Claude / Codex plan limit the member's computer reported. +/// The plan, in plain words: Claude or Codex. +/// The CLI's own words. +/// When the plan takes work again; null when the CLI didn't say (or said it in a way that +/// can't be placed in time). Carries the offset of the zone the CLI named, when it named one. +/// True when is a wall-clock time in the zone the CLI named +/// (shown as is, "23:50"); false when it was computed (a duration, a timestamp) and is shown in UTC. +public sealed record PlanLimit(string Plan, string Message, DateTimeOffset? ResetsAt, bool InNamedZone = false) +{ + /// The reset time as the member reads it ("23:50", or "16:50 UTC"); null when unknown. + public string? ResetText => ResetsAt is not { } at + ? null + : InNamedZone + ? at.ToString("HH:mm", CultureInfo.InvariantCulture) + : at.ToUniversalTime().ToString("HH:mm", CultureInfo.InvariantCulture) + " UTC"; +} + +/// Autopilot's wait for a plan limit: the limit, and when work may start again. +public sealed record PlanLimitHold(PlanLimit Limit, DateTimeOffset UntilUtc) +{ + /// When Autopilot carries on, as the member reads it: the plan's reset time when known, else the time + /// Autopilot checks again. + public string WaitsUntilText => Limit.ResetText ?? UntilUtc.ToUniversalTime().ToString("HH:mm", CultureInfo.InvariantCulture) + " UTC"; + + /// The one plain line the Autopilot activity and the Board show. + public string Sentence => Limit.ResetText is not null + ? $"Your computer's {Limit.Plan} plan hit its limit — Autopilot waits until {WaitsUntilText}, nothing is lost." + : $"Your computer's {Limit.Plan} plan hit its limit — Autopilot checks again at {WaitsUntilText}, nothing is lost."; +} + +/// Reads a plan limit out of what a CLI answered. +public static partial class PlanLimitDetector +{ + /// Only the end of a reply is read: a CLI prints the limit as its last words. + internal const int TailLength = 600; + + /// A clean (exit 0) reply longer than this is real work, never a limit notice. + internal const int MaxCleanReplyLength = 600; + + /// The longest wait believed: a weekly limit, with a day to spare. + internal static readonly TimeSpan MaxWait = TimeSpan.FromDays(8); + + [GeneratedRegex(@"\b(?:hit|reached)\s+your\s+(?:(?:current|session|usage|weekly|daily|monthly|5-hour|five-hour|opus|sonnet|plan|subscription)\s+)*limit\b" + + @"|\b(?:usage|session|weekly|daily|monthly|5-hour|opus|sonnet)\s+limit\s+reached\b|\busage\s+limit\s+exceeded\b", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex HitPattern(); + + // "Claude AI usage limit reached|1761234567" — older Claude Code prints the reset as a Unix timestamp. + [GeneratedRegex(@"limit\s+reached\|(?\d{10})\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex EpochPattern(); + + // "resets 11:50pm (Asia/Ho_Chi_Minh)", "resets 7pm", "resets Oct 27, 5pm (Europe/Berlin)", "try again at 3:04 PM". + [GeneratedRegex(@"\b(?:resets?|try\s+again\s+at|available\s+again\s+at)\s+(?:at\s+)?(?:on\s+)?" + + @"(?:(?jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+(?\d{1,2})(?:st|nd|rd|th)?,?\s+(?:at\s+)?)?" + + @"(?\d{1,2})(?::(?\d{2}))?\s*(?[ap]\.?\s?m\b\.?)?(?:\s*\((?[^)\s]+)\))?", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex ClockPattern(); + + // "try again in 2 hours 5 minutes", "resets in 3h 12m", "try again in 4 days". + [GeneratedRegex(@"\b(?:try\s+again|resets?|available\s+again)\s+in\s+(?(?:\d+\s*(?:days?|d|hours?|hrs?|h|minutes?|mins?|m|seconds?|secs?|s)\b[\s,]*(?:and\s+)?)+)", + RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex DurationPattern(); + + [GeneratedRegex(@"(?\d+)\s*(?days?|d|hours?|hrs?|h|minutes?|mins?|m|seconds?|secs?|s)\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant)] + private static partial Regex DurationPart(); + + /// True when (a stored task error, a CLI reply) is a plan-limit notice. + public static bool IsPlanLimit(string? text) => !string.IsNullOrWhiteSpace(text) && HitPattern().IsMatch(TailOf(text)); + + /// The plan limit a runner's reply reports, or null. A clean reply counts only when it is short — a real + /// answer that mentions a limit is work, not a notice. + /// The CLI exited cleanly. + /// Its answer. + /// Its error text. + /// The CLI the run used ("claude", "codex"; null = the runner's default). + /// Now. + /// Resolves the zone a CLI names (null = the system's IANA zones). + public static PlanLimit? FromReply(bool ok, string? content, string? error, string? cli, DateTimeOffset nowUtc, + Func? zones = null) + { + if (ok) + { + return content is { Length: > 0 and <= MaxCleanReplyLength } ? Detect(content, cli, nowUtc, zones) : null; + } + return Detect(error, cli, nowUtc, zones); + } + + /// The plan limit reports, or null. + public static PlanLimit? Detect(string? text, string? cli, DateTimeOffset nowUtc, Func? zones = null) + { + if (string.IsNullOrWhiteSpace(text)) { return null; } + var tail = TailOf(text); + var hit = HitPattern().Match(tail); + if (!hit.Success) { return null; } + var plan = PlanOf(cli, tail); + var (resetsAt, named) = ResetOf(tail[hit.Index..], nowUtc, zones ?? SystemZone); + if (resetsAt is { } r && (r <= nowUtc || r - nowUtc > MaxWait)) { resetsAt = null; named = false; } + // The CLI's own line (an "exit 1:" prefix and earlier output stay out of it). + var start = hit.Index == 0 ? 0 : tail.LastIndexOf('\n', hit.Index - 1) + 1; + var end = tail.IndexOf('\n', hit.Index); + var message = tail[start..(end < 0 ? tail.Length : end)].Trim(); + if (message.StartsWith("exit ", StringComparison.Ordinal) && message.IndexOf(": ", StringComparison.Ordinal) is var colon and > 0 and < 12) + { + message = message[(colon + 2)..].Trim(); + } + return new PlanLimit(plan, message, resetsAt, named); + } + + private static string TailOf(string text) => text.Length <= TailLength ? text : text[^TailLength..]; + + private static string PlanOf(string? cli, string text) => + cli?.Contains("codex", StringComparison.OrdinalIgnoreCase) == true + || text.Contains("codex", StringComparison.OrdinalIgnoreCase) || text.Contains("chatgpt", StringComparison.OrdinalIgnoreCase) + ? "Codex" + : "Claude"; + + private static (DateTimeOffset? At, bool Named) ResetOf(string text, DateTimeOffset nowUtc, Func zones) + { + if (EpochPattern().Match(text) is { Success: true } e + && long.TryParse(e.Groups["epoch"].ValueSpan, NumberStyles.None, CultureInfo.InvariantCulture, out var seconds)) + { + return (DateTimeOffset.FromUnixTimeSeconds(seconds), false); + } + if (DurationPattern().Match(text) is { Success: true } d) + { + var wait = TimeSpan.Zero; + foreach (Match part in DurationPart().Matches(d.Groups["dur"].Value)) + { + var n = int.Parse(part.Groups["n"].ValueSpan, NumberStyles.None, CultureInfo.InvariantCulture); + var unit = part.Groups["u"].Value.ToUpperInvariant(); + wait += unit[0] switch + { + 'D' => TimeSpan.FromDays(n), + 'H' => TimeSpan.FromHours(n), + 'S' => TimeSpan.FromSeconds(n), + _ => TimeSpan.FromMinutes(n), + }; + } + return wait > TimeSpan.Zero ? (nowUtc + wait, false) : (null, false); + } + if (ClockPattern().Match(text) is not { Success: true } c) { return (null, false); } + var hour = int.Parse(c.Groups["h"].ValueSpan, NumberStyles.None, CultureInfo.InvariantCulture); + var minute = c.Groups["m"].Success ? int.Parse(c.Groups["m"].ValueSpan, NumberStyles.None, CultureInfo.InvariantCulture) : 0; + if (c.Groups["ap"].Success) + { + if (hour is < 1 or > 12) { return (null, false); } + var pm = c.Groups["ap"].Value.StartsWith('p') || c.Groups["ap"].Value.StartsWith('P'); + hour = hour % 12 + (pm ? 12 : 0); + } + if (hour > 23 || minute > 59) { return (null, false); } + // A wall-clock time is placed only in a zone the CLI named: the server doesn't know the computer's zone. + var zone = c.Groups["tz"].Success ? zones(c.Groups["tz"].Value) : null; + if (zone is null) { return (null, false); } + var local = TimeZoneInfo.ConvertTime(nowUtc, zone); + var date = local.Date; + var explicitDate = false; + if (c.Groups["mon"].Success && c.Groups["day"].Success) + { + var month = Array.IndexOf(Months, c.Groups["mon"].Value[..3].ToUpperInvariant()) + 1; + var day = int.Parse(c.Groups["day"].ValueSpan, NumberStyles.None, CultureInfo.InvariantCulture); + if (month < 1 || day < 1 || day > DateTime.DaysInMonth(date.Year, month)) { return (null, false); } + date = new DateTime(date.Year, month, day, 0, 0, 0, DateTimeKind.Unspecified); + explicitDate = true; + } + var wall = DateTime.SpecifyKind(date.AddHours(hour).AddMinutes(minute), DateTimeKind.Unspecified); + // "resets 11:50pm" said at 23:55 means tomorrow; a dated reset already past means next year. + if (wall <= local.DateTime) { wall = explicitDate ? wall.AddYears(1) : wall.AddDays(1); } + return (new DateTimeOffset(wall, zone.GetUtcOffset(wall)), true); + } + + private static readonly string[] Months = ["JAN", "FEB", "MAR", "APR", "MAY", "JUN", "JUL", "AUG", "SEP", "OCT", "NOV", "DEC"]; + + private static TimeZoneInfo? SystemZone(string id) + { + if (string.Equals(id, "UTC", StringComparison.OrdinalIgnoreCase) || string.Equals(id, "GMT", StringComparison.OrdinalIgnoreCase)) + { + return TimeZoneInfo.Utc; + } + try + { + return TimeZoneInfo.FindSystemTimeZoneById(id); + } + catch (Exception ex) when (ex is TimeZoneNotFoundException or InvalidTimeZoneException) + { + return TimeZoneInfo.TryConvertIanaIdToWindowsId(id, out var windowsId) && TryFind(windowsId, out var zone) ? zone : null; + } + } + + private static bool TryFind(string id, out TimeZoneInfo? zone) + { + try + { + zone = TimeZoneInfo.FindSystemTimeZoneById(id); + return true; + } + catch (Exception ex) when (ex is TimeZoneNotFoundException or InvalidTimeZoneException) + { + zone = null; + return false; + } + } +} + +/// A runner's reply was a plan-limit notice: the work didn't happen, and nothing about it failed. +public sealed class PlanLimitException : LlmException +{ + /// The limit. + public PlanLimit? Limit { get; } + + public PlanLimitException() { } + + public PlanLimitException(string message) : base(message) { } + + public PlanLimitException(string message, Exception innerException) : base(message, innerException) { } + + /// The limit , reported by . + public PlanLimitException(PlanLimit limit, string provider) : base((limit ?? throw new ArgumentNullException(nameof(limit))).Message, provider) + { + Limit = limit; + } +} + +/// Plan limits per member (their computer's plan), for Autopilot to wait out and the Board to show. +public interface IPlanLimits +{ + /// Record that 's plan hit . + void Report(string tenantId, string memberUserId, PlanLimit limit); + + /// The member's active wait, or null when their plan is open. + PlanLimitHold? Current(string tenantId, string memberUserId); +} + +/// In-memory . Singleton; thread-safe. A restart forgets a wait — the next run hits +/// the limit again (a cheap, immediate answer) and the wait is back. +public sealed class PlanLimitTracker(TimeProvider time) : IPlanLimits +{ + /// Work starts this long after the reset, so the first call isn't refused at the stroke of it. + public static readonly TimeSpan Grace = TimeSpan.FromMinutes(1); + + /// The wait when the CLI didn't say when the plan resets: check again after this long. + public static readonly TimeSpan UnknownResetWait = TimeSpan.FromMinutes(30); + + private readonly ConcurrentDictionary<(string Tenant, string Member), PlanLimitHold> _holds = new(); + + /// + public void Report(string tenantId, string memberUserId, PlanLimit limit) + { + ArgumentNullException.ThrowIfNull(tenantId); + ArgumentNullException.ThrowIfNull(limit); + var until = (limit.ResetsAt ?? time.GetUtcNow() + UnknownResetWait).ToUniversalTime() + Grace; + _holds[(tenantId, memberUserId ?? string.Empty)] = new PlanLimitHold(limit, until); + } + + /// + public PlanLimitHold? Current(string tenantId, string memberUserId) + { + var key = (tenantId ?? string.Empty, memberUserId ?? string.Empty); + if (!_holds.TryGetValue(key, out var hold)) { return null; } + if (hold.UntilUtc > time.GetUtcNow()) { return hold; } + // Only this hold: a newer report that raced in stays. + _holds.TryRemove(new System.Collections.Generic.KeyValuePair<(string, string), PlanLimitHold>(key, hold)); + return null; + } +} diff --git a/src/AgentOs.Modules.RemoteAgent/RemoteAgentLlmClient.cs b/src/AgentOs.Modules.RemoteAgent/RemoteAgentLlmClient.cs index b7bae8bc..e912ccfc 100644 --- a/src/AgentOs.Modules.RemoteAgent/RemoteAgentLlmClient.cs +++ b/src/AgentOs.Modules.RemoteAgent/RemoteAgentLlmClient.cs @@ -113,6 +113,16 @@ void OnProgress(RemoteProgress p) } stopwatch.Stop(); + // The member's Claude/Codex plan said "no more for now": a typed signal, not a failure of the work — the + // member's wait is recorded so Autopilot holds off until the reset instead of burning its tries. + if (PlanLimitDetector.FromReply(result.Ok, result.Content, result.Error, request.Cli, DateTimeOffset.UtcNow) is { } limit) + { + LlmTelemetry.RecordError(activity, "plan limit"); + _services.GetService()?.Report(target.TenantId, target.MemberUserId, limit); + _logger.LogWarning("[RemoteAgent] request {Id}: the member's {Plan} plan hit its limit (resets {Reset}).", id, limit.Plan, limit.ResetText ?? "unknown"); + throw new PlanLimitException(limit, Provider); + } + if (!result.Ok) { LlmTelemetry.RecordError(activity, result.Error ?? "remote agent failure"); diff --git a/src/AgentOs.Modules.RemoteAgent/RemoteAgentModule.cs b/src/AgentOs.Modules.RemoteAgent/RemoteAgentModule.cs index efb0a3a7..333d0735 100644 --- a/src/AgentOs.Modules.RemoteAgent/RemoteAgentModule.cs +++ b/src/AgentOs.Modules.RemoteAgent/RemoteAgentModule.cs @@ -25,6 +25,8 @@ public void AddServices(IServiceCollection services, IConfiguration configuratio // Fail-closed: deny remote/runner exec outside Development unless RemoteAgent:AutoApprove=true. // (AutoApproveRemoteExec is retained for tests + explicit dev wiring.) services.TryAddSingleton(); + // A member's Claude/Codex plan limit, as their computer reported it — Autopilot waits it out (PlanLimit.cs). + services.TryAddSingleton(_ => new PlanLimitTracker(TimeProvider.System)); services.AddSingleton( new RemoteAgentSettings(AgentOs.SharedKernel.Modularity.ModuleEnvironment.IsDevelopment(configuration))); diff --git a/src/AgentOs.Modules.Sessions/Persistence/ISessionRepository.cs b/src/AgentOs.Modules.Sessions/Persistence/ISessionRepository.cs index c1b7bd57..84773ec3 100644 --- a/src/AgentOs.Modules.Sessions/Persistence/ISessionRepository.cs +++ b/src/AgentOs.Modules.Sessions/Persistence/ISessionRepository.cs @@ -58,6 +58,9 @@ Task UpdateRepoRunResultAsync( /// Count one more round of fixes on the task (its change went back to the Coding agent). Task IncrementFixRoundsForTenantAsync(string tenantId, Guid sessionId, CancellationToken ct = default) => Task.CompletedTask; + /// Take back a round that never happened (the member's AI plan was at its limit) — never below zero. + Task DecrementFixRoundsForTenantAsync(string tenantId, Guid sessionId, CancellationToken ct = default) => Task.CompletedTask; + /// Delivery of the tenant's tasks merged since (null = all time). Task GetDeliveryForTenantAsync(string tenantId, DateTimeOffset? since = null, CancellationToken ct = default) => Task.FromResult(DeliverySummary.Empty); diff --git a/src/AgentOs.Modules.Sessions/Persistence/Repositories/SessionRepository.cs b/src/AgentOs.Modules.Sessions/Persistence/Repositories/SessionRepository.cs index 74002dcb..363d8089 100644 --- a/src/AgentOs.Modules.Sessions/Persistence/Repositories/SessionRepository.cs +++ b/src/AgentOs.Modules.Sessions/Persistence/Repositories/SessionRepository.cs @@ -214,6 +214,15 @@ public async Task IncrementFixRoundsForTenantAsync(string tenantId, Guid session await _db.SaveChangesAsync(ct).ConfigureAwait(false); } + public async Task DecrementFixRoundsForTenantAsync(string tenantId, Guid sessionId, CancellationToken ct = default) + { + var session = await _db.Sessions.IgnoreQueryFilters() + .FirstOrDefaultAsync(s => s.Id == sessionId && s.TenantId == tenantId, ct).ConfigureAwait(false); + if (session is null || session.FixRounds <= 0) { return; } + session.FixRounds--; + await _db.SaveChangesAsync(ct).ConfigureAwait(false); + } + public async Task GetDeliveryForTenantAsync(string tenantId, DateTimeOffset? since = null, CancellationToken ct = default) { var merged = await _db.Sessions.IgnoreQueryFilters() diff --git a/src/AgentOs.Web/Components/Pages/AutopilotSettingsApp.razor b/src/AgentOs.Web/Components/Pages/AutopilotSettingsApp.razor new file mode 100644 index 00000000..ebfb2cb3 --- /dev/null +++ b/src/AgentOs.Web/Components/Pages/AutopilotSettingsApp.razor @@ -0,0 +1,81 @@ +@* AutopilotSettingsApp — Settings › AI › Autopilot (ADR-0011): how Autopilot runs a project once a person turns it on + in Board › Autopilot. Who decides the next moves, how each ticket runs, the bounds every engine keeps (tasks at once, + acceptance tests at once, fix rounds, the Orchestrator agent's retries) and how long a MAF workflow waits for checks + — all rendered from the Autopilot:* descriptors (SettingsFields), read by Autopilot on its next pass. The rules that + are not settings (when it merges, what a plan limit does) are stated here in plain words, so the screen says what + Autopilot really does. Admins only: these apply to every project of the tenant. *@ + +@using Microsoft.AspNetCore.Components.Authorization +@using AgentOs.Modules.Pipeline.Agents + +@if (!_resolved) +{ + +} +else if (!_isAdmin) +{ + +} +else +{ +
+
+

Autopilot

+

How the AI works on a project by itself once Autopilot is on (Board › Autopilot). Changes apply to its next pass — within a minute, no restart.

+
+
+ + + + + + +

Every engine keeps these bounds: the Orchestrator agent, the fixed rules and a ticket's MAF workflow. Fix rounds are counted per task in total — whichever engine sent the change back, and across restarts.

+ + + + + + + + +
    +
  • Merging. With Build and merge, a change is merged only when its checks are green, the Reviewer agent approved it and the Testing agent passed it against the ticket's acceptance criteria — which needs a connected computer. Nothing untested is merged.
    • +
  • Order. Approved tickets start p0 first, then oldest — never before the tickets they depend on are done. Tickets marked for a person are never touched.
    • +
  • Plan limits. When your computer's Claude or Codex plan hits its limit, Autopilot waits until the plan resets: no new work, and no fix round or retry is spent. Work it cut short picks up again after the reset — nothing is lost.
    • +
+ +} + +@code { + [CascadingParameter] private Task? AuthState { get; set; } + + private static readonly string[] DecisionKeys = [AutopilotSettings.DecidedByKey, AutopilotSettings.TicketEngineKey]; + + private static readonly string[] LimitKeys = + [ + AutopilotSettings.MaxParallelKey, + AutopilotSettings.MaxParallelTestsKey, + AutopilotSettings.MaxFixRoundsKey, + AutopilotSettings.MaxAgentRetriesKey, + ]; + + private static readonly string[] CheckKeys = [AutopilotSettings.ChecksTimeoutMinutesKey, AutopilotSettings.ChecksPollSecondsKey]; + + private bool _resolved; + private bool _isAdmin; + private string _tenantId = string.Empty; + private string _actor = "admin"; + + protected override async Task OnInitializedAsync() + { + if (AuthState is not null) + { + var user = (await AuthState).User; + _isAdmin = user.IsAdmin(); + _tenantId = user.GetTenantId() ?? string.Empty; + _actor = user.GetUserId() ?? user.Identity?.Name ?? _actor; + } + _resolved = true; + } +} diff --git a/src/AgentOs.Web/Components/Pages/BoardApp.razor b/src/AgentOs.Web/Components/Pages/BoardApp.razor index 0aa3acc3..54e1b68d 100644 --- a/src/AgentOs.Web/Components/Pages/BoardApp.razor +++ b/src/AgentOs.Web/Components/Pages/BoardApp.razor @@ -37,6 +37,8 @@ @inject AgentOs.Web.Services.Orchestration.OrchestratorLog OrchestratorLog @inject AgentOs.Modules.Sessions.Pairing.IComputerPairingCodes PairingCodes @inject AgentOs.Modules.RemoteAgent.IRemoteAgentBroker Broker +@inject AgentOs.Modules.RemoteAgent.IPlanLimits PlanLimits +@inject AgentOs.Domain.Configuration.ISettings SettingsStore @inject AgentOs.Web.Services.Preview.PreviewService Previews @inject NavigationManager Nav @inject AgentOs.Web.Shell.Services.WindowManagerService WM @@ -619,7 +621,14 @@ REMOTE_AGENT_TOKEN=@_newRunner.Value.Token @r.Label @if (_isAdmin) { @MemberLabel(r.OwnerUserId) } - @ComputerStatus(r) + + @ComputerStatus(r) + @if (PlanLimits.Current(_tenantId, r.OwnerUserId) is { } hold) + { + @* The plan, not the computer, is the limit: shown as a warning with when it resets. *@ + @hold.Limit.Plan plan limit · until @hold.WaitsUntilText + } + @(Broker.IsConnected(r.Id) ? "now" : Ago(r.LastSeenUtc ?? r.CreatedAtUtc)) - Build — start approved tickets in order, two at a time, and fix failing checks. You merge. + Build — start approved tickets in order, @AtATime(_autopilotLimits.MaxParallel), and send red checks, conflicts and review or test findings back to the AI (up to @Rounds(_autopilotLimits.MaxFixRounds) per task). You merge. -

The Orchestrator agent reads the whole project whenever it changes and decides the next moves — within fixed rules: nothing untested is merged, no ticket starts before what it depends on. It retries stuck work with guidance, asks you precise questions on the ticket, and never touches tickets marked for a person. (Settings › Autopilot › Decided by switches to the fixed rules alone.)

+

@(_autopilotAgentDecides + ? "The Orchestrator agent reads the whole project whenever it changes and decides the next moves — within fixed rules: nothing untested is merged, no ticket starts before what it depends on. It retries stuck work with guidance, asks you precise questions on the ticket, and never touches tickets marked for a person." + : "The fixed rules decide the next moves: approved tickets in dependency order, p0 first; nothing untested is merged; tickets marked for a person are never touched.") + @(_isAdmin ? "" : " An admin sets how Autopilot runs in Settings › Autopilot.")

+ @if (PlanLimits.Current(_tenantId, b.AutopilotUserId ?? b.CreatedByUserId ?? "") is { } hold && mode != AgentOs.Web.Services.Orchestration.AutopilotMode.Off) + { +
+ Waiting for the AI plan + @hold.Sentence +
+ }
Check now + @if (_isAdmin) + { + + Autopilot settings + + }
@{ var decisions = _autopilotLog.GetValueOrDefault(b.Id) ?? []; } @@ -2411,12 +2436,29 @@ else if (_tab == Tab.Sessions) // ── Autopilot ──────────────────────────────────────────────────────────────────────────────────── private readonly Dictionary> _autopilotLog = new(); + // The tenant's Autopilot settings, read when the tab opens, so the screen says what Autopilot will really do. + private AgentOs.Modules.Pipeline.Agents.AutopilotLimits _autopilotLimits = AgentOs.Modules.Pipeline.Agents.AutopilotLimits.Default; + private bool _autopilotAgentDecides = true; + private async Task ShowAutopilot() { _tab = Tab.Autopilot; + _autopilotLimits = await AgentOs.Modules.Pipeline.Agents.AutopilotSettings.ReadLimitsAsync(SettingsStore, _tenantId); + _autopilotAgentDecides = AgentOs.Modules.Pipeline.Agents.AutopilotSettings.AgentDecides( + (await SettingsStore.GetAsync(AgentOs.Modules.Pipeline.Agents.AutopilotSettings.DecidedByKey, _tenantId)).Value); await LoadAutopilotLogAsync(); } + private static string AtATime(int n) => n switch + { + 1 => "one at a time", + 2 => "two at a time", + 3 => "three at a time", + _ => n.ToString(System.Globalization.CultureInfo.InvariantCulture) + " at a time", + }; + + private static string Rounds(int n) => n == 1 ? "1 round" : n.ToString(System.Globalization.CultureInfo.InvariantCulture) + " rounds"; + private async Task LoadAutopilotLogAsync() { foreach (var b in _boards ?? []) @@ -2436,7 +2478,7 @@ else if (_tab == Tab.Sessions) OrchestratorLog.Add(_tenantId, board.Id, mode switch { "Build" => "Autopilot on: building approved tickets in order; you merge.", - "BuildAndMerge" => "Autopilot on: building and merging when all checks pass.", + "BuildAndMerge" => "Autopilot on: building, and merging each change once its checks, review and acceptance testing pass.", _ => "Autopilot off.", }); await LoadAutopilotLogAsync(); diff --git a/src/AgentOs.Web/Components/Pages/SettingsHub.razor b/src/AgentOs.Web/Components/Pages/SettingsHub.razor index 5809ecdb..cc69c7fe 100644 --- a/src/AgentOs.Web/Components/Pages/SettingsHub.razor +++ b/src/AgentOs.Web/Components/Pages/SettingsHub.razor @@ -40,6 +40,7 @@ break; case "agents": ; break; case "prompts": ; break; + case "autopilot": ; break; case "mcp": ; break; case "plugins": ; break; case "policy": ; break; @@ -76,6 +77,7 @@ new("routing", "AI", "Routing", "lightning", Access.Admin, Host: true), new("agents", "AI", "Agent profiles", "diamond", Access.Admin), new("prompts", "AI", "Prompts", "file", Access.Admin), + new("autopilot", "AI", "Autopilot", "play", Access.Admin), new("github", "Connections", "GitHub", "git-pull-request", Access.Admin, Host: true), new("mcp", "Connections", "MCP servers", "arrow-square-out", Access.Admin), new("plugins", "Connections", "Plugins", "squares-stack", Access.Everyone), diff --git a/src/AgentOs.Web/Services/BoardTasks/BoardTaskService.cs b/src/AgentOs.Web/Services/BoardTasks/BoardTaskService.cs index 63a44d7e..154ed830 100644 --- a/src/AgentOs.Web/Services/BoardTasks/BoardTaskService.cs +++ b/src/AgentOs.Web/Services/BoardTasks/BoardTaskService.cs @@ -48,13 +48,8 @@ public sealed class BoardTaskService( TaskStopRegistry stops, IOptions pipelineOptions) { - private readonly SemaphoreSlim _testSlot = new(1, 1); - - private sealed class SlotRelease(SemaphoreSlim slot) : IDisposable - { - private int _done; - public void Dispose() { if (Interlocked.Exchange(ref _done, 1) == 0) { slot.Release(); } } - } + // Acceptance runs at once per member's computer (Settings › Autopilot › Acceptance tests at once), whoever asks. + private readonly RunGate _testGate = new(); /// A repo a new task works in (the first one is the task's primary repo). public sealed record TaskRepo(string Owner, string Repo, string DefaultBranch, Guid? WorkspaceRepoId); @@ -300,10 +295,12 @@ private async Task TestChangesAsync(string tenantId, RemoteSessionEntity session using var _identity = AgentOs.SharedKernel.Identity.AmbientIdentity.Push(tenantId, session.MemberUserId, session.Id); var sessRepo = scope.ServiceProvider.GetRequiredService(); var tester = scope.ServiceProvider.GetRequiredService(); - // One acceptance run at a time, whoever asks (a MAF ticket workflow or a step-by-step pass): each brings the - // app up on the computer's fixed ports. - await _testSlot.WaitAsync(stopLease.Token).ConfigureAwait(false); - using var _slot = new SlotRelease(_testSlot); + // Acceptance runs on one computer are capped, whoever asks (a MAF ticket workflow or a step-by-step pass): each + // brings the app up on the computer's fixed ports. The cap is the tenant's setting, read now. + var settings = scope.ServiceProvider.GetService(); + var testCap = settings is null ? AgentOs.Modules.Pipeline.Agents.AutopilotLimits.Default.MaxParallelTests + : (await AgentOs.Modules.Pipeline.Agents.AutopilotSettings.ReadLimitsAsync(settings, tenantId, stopLease.Token).ConfigureAwait(false)).MaxParallelTests; + using var _slot = await _testGate.EnterAsync(tenantId + "/" + session.MemberUserId, testCap, stopLease.Token).ConfigureAwait(false); var prs = scope.ServiceProvider.GetRequiredService(); var body = await ReadTicketBodyAsync(scope, board, tenantId, session, pat).ConfigureAwait(false) ?? ""; foreach (var change in changes) @@ -520,6 +517,26 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri foreach (var outcome in workResult.Repos) { var repoLabel = $"{outcome.Owner}/{outcome.Repo}"; + if (!outcome.Ok && AgentOs.Modules.RemoteAgent.PlanLimitDetector.IsPlanLimit(outcome.Error)) + { + // Nothing failed — the member's AI plan said "not now". A follow-up's change goes back to where it + // stood (its checks / findings intact), so the same move is made after the reset; a fresh start + // waits as cut short, and Autopilot picks it up again once the plan resets. + var before = children.FirstOrDefault(c => c.Id == outcome.SessionRepoId); + if (change is not null && before is { PrUrl.Length: > 0, Status: "Done" or "Reviewed" or "ChangesRequested" or "Accepted" or "Rejected" or "Blocked" }) + { + await sessRepo.UpdateRepoRunResultAsync(tenantId, before.Id, before.Status, before.BranchName, before.PrUrl, before.Error).ConfigureAwait(false); + } + else + { + anyFailure = true; + await sessRepo.UpdateRepoRunResultAsync(tenantId, outcome.SessionRepoId, "Failed", outcome.BranchName, + change is not null ? before?.PrUrl : null, outcome.Error).ConfigureAwait(false); + } + Publish(SessionRunEventKind.RepoDone, $"{repoLabel} — paused: your computer's AI plan hit its limit. Nothing was changed.", repoLabel, + before?.Status ?? "Failed", before?.PrUrl); + continue; + } if (!outcome.Ok) { anyFailure = true; @@ -572,6 +589,13 @@ await sessRepo.UpdateRepoRunResultAsync(tenantId, outcome.SessionRepoId, } } + // A follow-up the plan limit stopped everywhere never happened: its fix round is taken back, so the cap + // (Settings › Autopilot) counts only rounds that ran. + if (change is not null && workResult.Repos.Count > 0 + && workResult.Repos.All(o => !o.Ok && AgentOs.Modules.RemoteAgent.PlanLimitDetector.IsPlanLimit(o.Error))) + { + await sessRepo.DecrementFixRoundsForTenantAsync(tenantId, session.Id).ConfigureAwait(false); + } await sessRepo.RecomputeSessionStatusForTenantAsync(tenantId, session.Id).ConfigureAwait(false); Publish(anyFailure ? SessionRunEventKind.Failed : SessionRunEventKind.Done, anyFailure ? "Session finished with failures." : "Session complete — all repos done."); @@ -635,7 +659,7 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri if (cls != TicketClass.Greenfield) { var reason = TicketClassifier.RefusalMessage(cls); - await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, reason).ConfigureAwait(false); + await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, reason, CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Failed, reason); return; } @@ -644,7 +668,7 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri if (primary is null) { const string noRepo = "Quality run has no target repo — recreate the session from a board ticket."; - await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, noRepo).ConfigureAwait(false); + await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, noRepo, CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Failed, noRepo); return; } @@ -687,8 +711,8 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri if (result is null || result.Status == PipelineStatus.Failed) { var msg = error ?? "Quality pipeline failed — see the server log."; - await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Failed", null, null, msg).ConfigureAwait(false); - await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, msg).ConfigureAwait(false); + await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Failed", null, null, msg, CancellationToken.None).ConfigureAwait(false); + await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, msg, CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Failed, $"{repoLabel} — {msg}", repoLabel, "Failed"); return; } @@ -697,8 +721,8 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri var gate = PrGate.Evaluate(result); if (!gate.Allowed) { - await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Blocked", null, null, gate.Reason).ConfigureAwait(false); - await sessRepo.RecomputeSessionStatusForTenantAsync(tenantId, session.Id).ConfigureAwait(false); + await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Blocked", null, null, gate.Reason, CancellationToken.None).ConfigureAwait(false); + await sessRepo.RecomputeSessionStatusForTenantAsync(tenantId, session.Id, CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Failed, $"{repoLabel} — {gate.Reason}", repoLabel, "Blocked"); return; } @@ -710,15 +734,15 @@ void Publish(SessionRunEventKind kind, string message, string? repo = null, stri try { var pr = await prService.OpenPrAsync(result, descriptor, title, body, CancellationToken.None).ConfigureAwait(false); - await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Done", pr.Branch, pr.HtmlUrl, null).ConfigureAwait(false); - await sessRepo.RecomputeSessionStatusForTenantAsync(tenantId, session.Id).ConfigureAwait(false); + await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Done", pr.Branch, pr.HtmlUrl, null, CancellationToken.None).ConfigureAwait(false); + await sessRepo.RecomputeSessionStatusForTenantAsync(tenantId, session.Id, CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Done, $"{repoLabel} — PR opened.", repoLabel, "Done", pr.HtmlUrl); } catch (Exception ex) when (ex is System.Net.Http.HttpRequestException or System.Text.Json.JsonException or InvalidOperationException or ArgumentException or System.IO.IOException or TimeoutException) { - await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Failed", null, null, $"PR creation failed: {ex.Message}").ConfigureAwait(false); - await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, $"PR creation failed: {ex.Message}").ConfigureAwait(false); + await sessRepo.UpdateRepoRunResultAsync(tenantId, primary.Id, "Failed", null, null, $"PR creation failed: {ex.Message}", CancellationToken.None).ConfigureAwait(false); + await sessRepo.UpdateRunResultAsync(tenantId, session.Id, "Failed", null, $"PR creation failed: {ex.Message}", CancellationToken.None).ConfigureAwait(false); Publish(SessionRunEventKind.Failed, $"{repoLabel} — PR creation failed: {ex.Message}", repoLabel, "Failed"); } } diff --git a/src/AgentOs.Web/Services/BoardTasks/RunGate.cs b/src/AgentOs.Web/Services/BoardTasks/RunGate.cs new file mode 100644 index 00000000..5105fdd3 --- /dev/null +++ b/src/AgentOs.Web/Services/BoardTasks/RunGate.cs @@ -0,0 +1,67 @@ +// A keyed gate whose capacity is read at entry: at most N holders per key at once, N taken from the caller each time +// (the tenant's Settings › Autopilot › Acceptance tests at once, so a change applies to the next run without a +// restart). Keyed by the computer's owner — two members' computers are different machines, so they never wait on +// each other. A release wakes every waiter to re-check; holders are few (single digits), so that is cheap. + +using System; +using System.Collections.Generic; +using System.Threading; +using System.Threading.Tasks; + +namespace AgentOs.Web.Services.BoardTasks; + +/// At most N concurrent holders per key. Thread-safe. +public sealed class RunGate +{ + private readonly Lock _lock = new(); + private readonly Dictionary _held = new(StringComparer.Ordinal); + private TaskCompletionSource _released = new(TaskCreationOptions.RunContinuationsAsynchronously); + + /// Holders of right now. + public int Held(string key) + { + lock (_lock) { return _held.GetValueOrDefault(key); } + } + + /// Wait until has fewer than holders, then hold it; + /// dispose the result to let go. + public async Task EnterAsync(string key, int capacity, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(key); + var cap = Math.Max(1, capacity); + while (true) + { + Task wait; + lock (_lock) + { + var held = _held.GetValueOrDefault(key); + if (held < cap) + { + _held[key] = held + 1; + return new Lease(this, key); + } + wait = _released.Task; + } + await wait.WaitAsync(ct).ConfigureAwait(false); + } + } + + private void Exit(string key) + { + TaskCompletionSource wake; + lock (_lock) + { + var held = _held.GetValueOrDefault(key) - 1; + if (held <= 0) { _held.Remove(key); } else { _held[key] = held; } + wake = _released; + _released = new(TaskCreationOptions.RunContinuationsAsynchronously); + } + wake.TrySetResult(); + } + + private sealed class Lease(RunGate gate, string key) : IDisposable + { + private int _done; + public void Dispose() { if (Interlocked.Exchange(ref _done, 1) == 0) { gate.Exit(key); } } + } +} diff --git a/src/AgentOs.Web/Services/Orchestration/AutopilotPlanner.cs b/src/AgentOs.Web/Services/Orchestration/AutopilotPlanner.cs index f9a0d0e7..cafd73e4 100644 --- a/src/AgentOs.Web/Services/Orchestration/AutopilotPlanner.cs +++ b/src/AgentOs.Web/Services/Orchestration/AutopilotPlanner.cs @@ -6,6 +6,7 @@ using System.Collections.Generic; using System.Linq; using AgentOs.Domain.Workspaces; +using AgentOs.Modules.Pipeline.Agents; namespace AgentOs.Web.Services.Orchestration; @@ -16,7 +17,7 @@ public enum AutopilotMode Off, /// Start approved tickets in dependency order and fix failing checks; a person merges. Build, - /// Build, and merge a proposed change once all its checks pass. + /// Build, and merge a proposed change once its checks pass, review approved it and acceptance testing passed it. BuildAndMerge, } @@ -36,12 +37,20 @@ public sealed record PlanChange(Guid RepoId, int PrNumber, PrChecksState Checks, string BaseBranch = "main", ChangeReview Review = ChangeReview.Approved); /// A task as the planner sees it. -public sealed record PlanTask(Guid SessionId, int IssueNumber, string Status, IReadOnlyList OpenChanges, int FixRounds); +/// The task's persisted fix rounds (every change sent back to the Coding agent, by any engine, +/// before or after a restart) — the one counter every cap reads. +public sealed record PlanTask(Guid SessionId, int IssueNumber, string Status, IReadOnlyList OpenChanges, int FixRounds) +{ + /// Its last run was cut short by the member's plan limit (nothing failed): it resumes once the plan + /// resets, without spending a fix round or a retry. + public bool CutShortByPlanLimit { get; init; } +} /// One thing to do, and the sentence that explains it. public abstract record AutopilotAction(string Why) { public sealed record StartTicket(int Number, string Why) : AutopilotAction(Why); + public sealed record ResumeTask(Guid SessionId, int IssueNumber, string Why) : AutopilotAction(Why); public sealed record FixChecks(Guid SessionId, int IssueNumber, string Report, int Round, string Why) : AutopilotAction(Why); public sealed record UpdateBranch(Guid SessionId, int IssueNumber, Guid RepoId, int PrNumber, string Why) : AutopilotAction(Why); public sealed record RunTests(Guid SessionId, int IssueNumber, string Why) : AutopilotAction(Why); @@ -53,16 +62,6 @@ public sealed record NeedsPerson(Guid SessionId, int IssueNumber, string Reason, /// The rules. public static class AutopilotPlanner { - /// Automatic "Fix the failing checks" rounds before a person is asked. - public const int MaxFixRounds = 3; - - /// Tasks running at once on a board. - public const int MaxParallel = 2; - - /// Acceptance runs at once: a test brings the app up on fixed ports, so two at once on one computer collide - /// (seen on the lab: the second one found the ports taken and could not check its end-to-end criteria). - public const int MaxParallelTests = 1; - /// What Autopilot does next on a board, given its tickets and tasks — a pure decision, executed by the /// orchestrator. /// Off, Build, or BuildAndMerge. @@ -70,20 +69,34 @@ public static class AutopilotPlanner /// The board's open tasks and their proposed changes. /// A computer of the member is connected, so the Testing agent can run. Without one nothing /// is tested — and nothing untested is merged. - /// Slot cap. The Orchestrator agent asks for every legal move (no cap) and - /// applies the caps to the moves it picks, in its order. - /// Acceptance-run cap (see ). + /// The tenant's bounds (Settings › Autopilot; null = the defaults): slots, acceptance runs and + /// fix rounds. The Orchestrator agent asks with for every legal move and + /// applies the caps to the moves it picks, in its order. An acceptance run brings the app + /// up on fixed ports, so two at once on one computer collide (seen on the lab) — hence its own cap. public static IReadOnlyList Decide(AutopilotMode mode, IReadOnlyList tickets, IReadOnlyList tasks, - bool canTest = true, int maxParallel = MaxParallel, int maxParallelTests = MaxParallelTests) + bool canTest = true, AutopilotLimits? limits = null) { ArgumentNullException.ThrowIfNull(tickets); ArgumentNullException.ThrowIfNull(tasks); var actions = new List(); if (mode == AutopilotMode.Off) { return actions; } + var caps = limits ?? AutopilotLimits.Default; + var maxParallel = caps.MaxParallel; + var maxParallelTests = caps.MaxParallelTests; + var maxFixRounds = caps.MaxFixRounds; var running = tasks.Count(t => t.Status == "Running"); var testing = tasks.Count(t => t.OpenChanges.Any(c => c.Testing == ChangeTesting.Testing)); + // 0) Work a plan limit cut short resumes first — it was already under way, and spends no fix round or retry. + foreach (var task in tasks.Where(t => t.Status == "Failed" && t.CutShortByPlanLimit)) + { + if (running >= maxParallel) { break; } + actions.Add(new AutopilotAction.ResumeTask(task.SessionId, task.IssueNumber, + $"Picked #{task.IssueNumber} up again — your computer's AI plan is open again (no fix round used).")); + running++; + } + // 1) Proposed changes waiting on their checks: fix red ones (bounded), merge green ones (when allowed). foreach (var task in tasks.Where(t => t.Status == "Done")) { @@ -91,7 +104,7 @@ public static IReadOnlyList Decide(AutopilotMode mode, IReadOnl var conflicting = task.OpenChanges.Where(c => c.Merge == PrMergeState.Conflicts).ToList(); if (conflicting.Count > 0) { - if (task.FixRounds >= MaxFixRounds) + if (task.FixRounds >= maxFixRounds) { actions.Add(new AutopilotAction.NeedsPerson(task.SessionId, task.IssueNumber, "Still conflicts with its base after automatic fixes — needs a person.", @@ -104,25 +117,25 @@ public static IReadOnlyList Decide(AutopilotMode mode, IReadOnl $"Your branch conflicts with {conflicting[0].BaseBranch} (another change was merged first). Merge origin/{conflicting[0].BaseBranch} " + "into it, resolve every conflict keeping the intent of both changes, build, run the tests, commit and push.", conflictRound, - $"#{task.IssueNumber} conflicts with {conflicting[0].BaseBranch} — asked the AI to bring it up to date (try {conflictRound} of {MaxFixRounds}).")); + $"#{task.IssueNumber} conflicts with {conflicting[0].BaseBranch} — asked the AI to bring it up to date (try {conflictRound} of {maxFixRounds}).")); running++; continue; } var failed = task.OpenChanges.Where(c => c.Checks == PrChecksState.Failed).ToList(); if (failed.Count > 0) { - if (task.FixRounds >= MaxFixRounds) + if (task.FixRounds >= maxFixRounds) { actions.Add(new AutopilotAction.NeedsPerson(task.SessionId, task.IssueNumber, - $"Checks still fail after {MaxFixRounds} automatic fixes — needs a person.", - $"Stopped fixing #{task.IssueNumber}: its checks still fail after {MaxFixRounds} tries. It needs you.")); + $"Checks still fail after {maxFixRounds} automatic fixes — needs a person.", + $"Stopped fixing #{task.IssueNumber}: its checks still fail after {maxFixRounds} tries. It needs you.")); continue; } if (running >= maxParallel) { continue; } var round = task.FixRounds + 1; var report = string.Join("\n", failed.Select(c => c.FailedReport).Where(r => !string.IsNullOrWhiteSpace(r))); actions.Add(new AutopilotAction.FixChecks(task.SessionId, task.IssueNumber, report, round, - $"#{task.IssueNumber}'s checks failed — asked the AI to fix them (try {round} of {MaxFixRounds}).")); + $"#{task.IssueNumber}'s checks failed — asked the AI to fix them (try {round} of {maxFixRounds}).")); running++; continue; } @@ -136,18 +149,18 @@ public static IReadOnlyList Decide(AutopilotMode mode, IReadOnl var changesRequested = task.OpenChanges.Where(c => c.Review == ChangeReview.ChangesRequested).ToList(); if (changesRequested.Count > 0) { - if (task.FixRounds >= MaxFixRounds) + if (task.FixRounds >= maxFixRounds) { actions.Add(new AutopilotAction.NeedsPerson(task.SessionId, task.IssueNumber, - $"Code review still asks for changes after {MaxFixRounds} automatic fixes — needs a person.", - $"Stopped fixing #{task.IssueNumber}: code review still asks for changes after {MaxFixRounds} tries. It needs you.")); + $"Code review still asks for changes after {maxFixRounds} automatic fixes — needs a person.", + $"Stopped fixing #{task.IssueNumber}: code review still asks for changes after {maxFixRounds} tries. It needs you.")); continue; } if (running >= maxParallel) { continue; } var round = task.FixRounds + 1; actions.Add(new AutopilotAction.FixChecks(task.SessionId, task.IssueNumber, string.Join('\n', changesRequested.Select(c => c.Findings).Where(r => !string.IsNullOrWhiteSpace(r))), round, - $"Code review asked for changes on #{task.IssueNumber} — sent back to the AI (try {round} of {MaxFixRounds}).")); + $"Code review asked for changes on #{task.IssueNumber} — sent back to the AI (try {round} of {maxFixRounds}).")); running++; continue; } @@ -163,18 +176,18 @@ public static IReadOnlyList Decide(AutopilotMode mode, IReadOnl var rejected = task.OpenChanges.Where(c => c.Testing == ChangeTesting.Rejected).ToList(); if (rejected.Count > 0) { - if (task.FixRounds >= MaxFixRounds) + if (task.FixRounds >= maxFixRounds) { actions.Add(new AutopilotAction.NeedsPerson(task.SessionId, task.IssueNumber, - $"Acceptance testing still fails after {MaxFixRounds} automatic fixes — needs a person.", - $"Stopped fixing #{task.IssueNumber}: it still fails acceptance testing after {MaxFixRounds} tries. It needs you.")); + $"Acceptance testing still fails after {maxFixRounds} automatic fixes — needs a person.", + $"Stopped fixing #{task.IssueNumber}: it still fails acceptance testing after {maxFixRounds} tries. It needs you.")); continue; } if (running >= maxParallel) { continue; } var round = task.FixRounds + 1; actions.Add(new AutopilotAction.FixChecks(task.SessionId, task.IssueNumber, string.Join('\n', rejected.Select(c => c.Findings).Where(r => !string.IsNullOrWhiteSpace(r))), round, - $"#{task.IssueNumber} failed acceptance testing — sent back to the AI with the findings (try {round} of {MaxFixRounds}).")); + $"#{task.IssueNumber} failed acceptance testing — sent back to the AI with the findings (try {round} of {maxFixRounds}).")); running++; continue; } @@ -223,16 +236,17 @@ public static IReadOnlyList Decide(AutopilotMode mode, IReadOnl /// True for a move that occupies a work slot on the member's computer. public static bool TakesSlot(AutopilotAction action) => - action is AutopilotAction.StartTicket or AutopilotAction.FixChecks or AutopilotAction.RunReview or AutopilotAction.RunTests; + action is AutopilotAction.StartTicket or AutopilotAction.ResumeTask or AutopilotAction.FixChecks or AutopilotAction.RunReview or AutopilotAction.RunTests; /// /// The guardrail under the Orchestrator agent: the moves it picked, in its order, kept only while they fit — slots - /// (, counting tasks), one acceptance run at a time, and at most - /// one move per task in a pass. + /// (, counting tasks), the acceptance-run cap, + /// and at most one move per task in a pass. /// - public static IReadOnlyList WithinCaps(IEnumerable picked, int running, int testing) + public static IReadOnlyList WithinCaps(IEnumerable picked, int running, int testing, AutopilotLimits? limits = null) { ArgumentNullException.ThrowIfNull(picked); + var caps = limits ?? AutopilotLimits.Default; var kept = new List(); var touched = new HashSet(); foreach (var action in picked) @@ -241,10 +255,10 @@ public static IReadOnlyList WithinCaps(IEnumerable= MaxParallel) { continue; } + if (running >= caps.MaxParallel) { continue; } if (action is AutopilotAction.RunTests) { - if (testing >= MaxParallelTests) { continue; } + if (testing >= caps.MaxParallelTests) { continue; } testing++; } running++; @@ -257,6 +271,7 @@ public static IReadOnlyList WithinCaps(IEnumerable action switch { AutopilotAction.FixChecks a => a.SessionId, + AutopilotAction.ResumeTask a => a.SessionId, AutopilotAction.UpdateBranch a => a.SessionId, AutopilotAction.RunTests a => a.SessionId, AutopilotAction.RunReview a => a.SessionId, diff --git a/src/AgentOs.Web/Services/Orchestration/BoardTicketSteps.cs b/src/AgentOs.Web/Services/Orchestration/BoardTicketSteps.cs index 897af33b..910c2d56 100644 --- a/src/AgentOs.Web/Services/Orchestration/BoardTicketSteps.cs +++ b/src/AgentOs.Web/Services/Orchestration/BoardTicketSteps.cs @@ -15,8 +15,13 @@ namespace AgentOs.Web.Services.Orchestration; /// Ticket workflow steps on a board. -public sealed class BoardTicketSteps(BoardTaskService tasks, IServiceScopeFactory scopes, WorkspaceEntity board, string pat) : ITicketSteps +public sealed class BoardTicketSteps(BoardTaskService tasks, IServiceScopeFactory scopes, WorkspaceEntity board, string pat, + AgentOs.Modules.RemoteAgent.IPlanLimits? planLimits = null) : ITicketSteps { + /// The board's Autopilot member's plan limit, when their computer reported one that hasn't reset yet. + public string? PlanLimited(TicketFlow flow) => + planLimits?.Current(board.TenantId, board.AutopilotUserId ?? board.CreatedByUserId ?? string.Empty)?.Sentence; + public Task CodeAsync(TicketFlow flow, CancellationToken ct) => tasks.BeginStartAsync(flow.TenantId, flow.SessionId, flow.NextChange, ct: ct); @@ -33,17 +38,17 @@ public async Task> ChangesAsync(TicketFlow flow, Can } public Task<(PrChecksState Checks, PrMergeState Merge)> PrStateAsync(TicketFlow flow, TicketChange change, CancellationToken ct) => - WithPrAsync(flow, change, ct, async (prs, desc, pr) => - (await prs.GetChecksStateAsync(desc, pr, ct).ConfigureAwait(false), await prs.GetMergeStateAsync(desc, pr, ct).ConfigureAwait(false))); + WithPrAsync(flow, change, async (prs, desc, pr) => + (await prs.GetChecksStateAsync(desc, pr, ct).ConfigureAwait(false), await prs.GetMergeStateAsync(desc, pr, ct).ConfigureAwait(false)), ct); public Task FailedChecksAsync(TicketFlow flow, TicketChange change, CancellationToken ct) => - WithPrAsync(flow, change, ct, (prs, desc, pr) => prs.GetFailedChecksAsync(desc, pr, ct)); + WithPrAsync(flow, change, async (prs, desc, pr) => await prs.GetFailedChecksAsync(desc, pr, ct).ConfigureAwait(false), ct); public Task UpdateBranchAsync(TicketFlow flow, TicketChange change, CancellationToken ct) => - WithPrAsync(flow, change, ct, (prs, desc, pr) => prs.UpdateBranchAsync(desc, pr, ct)); + WithPrAsync(flow, change, (prs, desc, pr) => prs.UpdateBranchAsync(desc, pr, ct), ct); public Task MergeAsync(TicketFlow flow, TicketChange change, CancellationToken ct) => - WithPrAsync(flow, change, ct, async (prs, desc, pr) => + WithPrAsync(flow, change, async (prs, desc, pr) => { if (await prs.MergeAsync(desc, pr, ct).ConfigureAwait(false) is { } error) { return error; } using var scope = scopes.CreateScope(); @@ -51,12 +56,12 @@ public Task UpdateBranchAsync(TicketFlow flow, TicketChange change, Cancellation var repo = (await sessions.ListReposForTenantAsync(flow.TenantId, flow.SessionId, ct).ConfigureAwait(false)).First(r => r.Id == change.RepoId); await sessions.UpdateRepoRunResultAsync(flow.TenantId, repo.Id, "Merged", repo.BranchName, repo.PrUrl, null, ct).ConfigureAwait(false); await sessions.RecomputeSessionStatusForTenantAsync(flow.TenantId, flow.SessionId, ct).ConfigureAwait(false); - return (string?)null; - }); + return null; + }, ct); // Run a pull-request call for one change inside a scope (the source-control service and the change's repo). - private async Task WithPrAsync(TicketFlow flow, TicketChange change, CancellationToken ct, - Func> call) + private async Task WithPrAsync(TicketFlow flow, TicketChange change, + Func> call, CancellationToken ct) { ArgumentNullException.ThrowIfNull(flow); ArgumentNullException.ThrowIfNull(change); diff --git a/src/AgentOs.Web/Services/Orchestration/PlanLimitPause.cs b/src/AgentOs.Web/Services/Orchestration/PlanLimitPause.cs new file mode 100644 index 00000000..900d65cd --- /dev/null +++ b/src/AgentOs.Web/Services/Orchestration/PlanLimitPause.cs @@ -0,0 +1,34 @@ +// Autopilot waits out the member's Claude/Codex plan limit, board by board: while the member's computer reports one +// (IPlanLimits, recorded by the RemoteAgent provider), a pass starts nothing, retries nothing and sends nothing back +// to the Coding agent; the Autopilot activity gets ONE plain line when the wait starts ("…waits until 23:50, nothing +// is lost") and one when it ends. Work the limit cut short is picked up again by the planner (AutopilotAction.Resume). + +using System; +using System.Collections.Concurrent; +using AgentOs.Modules.RemoteAgent; + +namespace AgentOs.Web.Services.Orchestration; + +/// Per-board memory of a plan-limit wait. Thread-safe. +public sealed class PlanLimitPause +{ + private readonly ConcurrentDictionary _waiting = new(); + + /// + /// What a pass over may do, given its member's current wait (, + /// null = the plan is open): paused or not, and the line to log — only when the wait starts (or a later reset + /// replaces it) and when it ends. + /// + public (bool Paused, string? Line) Observe(Guid boardId, PlanLimitHold? hold) + { + if (hold is not null) + { + var announced = _waiting.TryGetValue(boardId, out var last) && last.UntilUtc == hold.UntilUtc; + _waiting[boardId] = hold; + return (true, announced ? null : hold.Sentence); + } + return _waiting.TryRemove(boardId, out var ended) + ? (false, $"Your computer's {ended.Limit.Plan} plan is open again — Autopilot carries on where it stopped.") + : (false, null); + } +} diff --git a/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.Agent.cs b/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.Agent.cs index f897c508..582836ff 100644 --- a/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.Agent.cs +++ b/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.Agent.cs @@ -1,7 +1,8 @@ // Autopilot's decision step. By default the Orchestrator agent decides (Settings › Autopilot › Decided by): it gets the // board's whole state and every legal move the rules allow — uncapped — picks the moves and their order, retries stuck // tasks with guidance and asks a person precise questions. The rules stay the guardrail: only legal moves can be -// picked, AutopilotPlanner.WithinCaps applies the slot caps to its picks, and retries are bounded per task. The agent +// picked, AutopilotPlanner.WithinCaps applies the slot caps to its picks, and retries are bounded per task (Settings › +// Autopilot: fix rounds + Orchestrator retries, against the task's persisted fix rounds). The agent // is asked only when the board changed; when it can't answer (no computer, timeout), that pass follows the rules. using System; @@ -25,13 +26,11 @@ namespace AgentOs.Web.Services.Orchestration; public sealed partial class ProjectOrchestrator { - /// Agent-led retries per stuck task (beyond the rules' fix rounds). - internal const int MaxAgentRetries = 2; - /// The retries the agent may still spend on a task: its persisted fix rounds (every change sent back to the - /// Coding agent, by anyone) count against the rules' rounds plus the agent's own — so a restart grants nothing new. - internal static int RetriesLeft(int fixRounds) => - Math.Clamp(AutopilotPlanner.MaxFixRounds + MaxAgentRetries - fixRounds, 0, MaxAgentRetries); + /// Coding agent, by anyone) count against the rules' rounds plus the agent's own (Settings › Autopilot) — so a + /// restart grants nothing new. + internal static int RetriesLeft(int fixRounds, AutopilotLimits limits) => + Math.Clamp(limits.MaxFixRounds + limits.MaxAgentRetries - fixRounds, 0, limits.MaxAgentRetries); private readonly ConcurrentDictionary _lastSignature = new(); private readonly ConcurrentDictionary _asked = new(); @@ -39,31 +38,34 @@ internal static int RetriesLeft(int fixRounds) => private async Task> DecideAsync(WorkspaceEntity board, AutopilotMode mode, string member, string pat, IReadOnlyList tickets, IReadOnlyList planTickets, IReadOnlyList planTasks, - IReadOnlyList sessions, ILookup repos, bool canTest, IServiceProvider sp, CancellationToken ct) + IReadOnlyList sessions, ILookup repos, bool canTest, AutopilotLimits limits, + IServiceProvider sp, CancellationToken ct) { var tenant = board.TenantId; var decidedBy = await sp.GetRequiredService().GetAsync(AutopilotSettings.DecidedByKey, tenant, ct).ConfigureAwait(false); // The agent answers on the member's computer: without one, the rules decide (and nothing gets tested anyway). if (!AutopilotSettings.AgentDecides(decidedBy.Value) || !canTest) { - return AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest); + return AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest, limits); } - var candidates = AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest, int.MaxValue, int.MaxValue); + var candidates = AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest, limits.Uncapped); var running = planTasks.Count(t => t.Status == "Running"); var testing = planTasks.Count(t => t.OpenChanges.Any(c => c.Testing == ChangeTesting.Testing)); - var stuck = await WithAnswersAsync(StuckTasks(tickets, planTickets, sessions, repos, candidates), board, pat, sessions, sp, ct).ConfigureAwait(false); - var moves = candidates.Where(a => a is not AutopilotAction.NeedsPerson) + // Work a plan limit cut short is picked up by rule, first — it isn't a decision, and must not wait on one. + var resumes = candidates.OfType().ToList(); + var stuck = await WithAnswersAsync(StuckTasks(tickets, planTickets, sessions, repos, candidates, limits), board, pat, sessions, sp, ct).ConfigureAwait(false); + var moves = candidates.Where(a => a is not AutopilotAction.NeedsPerson and not AutopilotAction.ResumeTask) .Select((a, i) => (Id: "M" + (i + 1).ToString(System.Globalization.CultureInfo.InvariantCulture), Action: a)).ToList(); var epics = await EpicsAsync(board, tenant, pat, tickets, sp, ct).ConfigureAwait(false); - if (moves.Count == 0 && stuck.Count == 0 && epics.Count == 0) { return []; } + if (moves.Count == 0 && stuck.Count == 0 && epics.Count == 0) { return AutopilotPlanner.WithinCaps(resumes, running, testing, limits); } - var freeSlots = Math.Max(0, AutopilotPlanner.MaxParallel - running); + var freeSlots = Math.Max(0, limits.MaxParallel - running - resumes.Count); // Ask only when something changed: the same picture gets the same answer (and costs the member's quota). var signature = Signature(moves.Select(m => m.Action.Why), stuck.Select(s => $"{s.IssueNumber}:{s.Situation}:{s.RetriesLeft}:{s.PersonAnswer}") .Concat(epics.Select(e => $"epic {e.IssueNumber}")) .Concat(tickets.Where(t => string.Equals(t.State, "CLOSED", StringComparison.OrdinalIgnoreCase)).Select(t => $"closed {t.Number}")), freeSlots); - if (_lastSignature.TryGetValue(board.Id, out var last) && last == signature) { return []; } + if (_lastSignature.TryGetValue(board.Id, out var last) && last == signature) { return AutopilotPlanner.WithinCaps(resumes, running, testing, limits); } var recent = (await log.RecentAsync(tenant, board.Id, 12).ConfigureAwait(false)).Select(e => e.Text).ToList(); var brief = new OrchestratorBrief(board.Id, tenant, member, board.Name, mode.ToString(), @@ -80,18 +82,20 @@ private async Task> DecideAsync(WorkspaceEntity b } if (!decision.Ran) { + // The member's plan hit its limit while the agent decided: the pass waits (StepAsync logs it), no fallback. + if (planLimits.Current(tenant, member) is not null) { return []; } if (_agentDown.TryAdd(board.Id, true)) { log.Add(tenant, board.Id, $"The Orchestrator agent couldn't decide ({decision.Error}) — following the rules until it can."); } - return AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest); + return AutopilotPlanner.Decide(mode, planTickets, planTasks, canTest, limits); } _agentDown.TryRemove(board.Id, out _); _lastSignature[board.Id] = signature; if (!string.IsNullOrWhiteSpace(decision.Reasoning)) { log.Add(tenant, board.Id, "Orchestrator: " + decision.Reasoning); } var byId = moves.ToDictionary(m => m.Id, m => m.Action, StringComparer.Ordinal); - var picked = AutopilotPlanner.WithinCaps(decision.MoveIds.Select(id => byId[id]), running, testing).ToList(); + var picked = AutopilotPlanner.WithinCaps(resumes.Concat(decision.MoveIds.Select(id => byId[id])), running, testing, limits).ToList(); running += picked.Count(AutopilotPlanner.TakesSlot); // Stuck tasks: retried with the agent's guidance (bounded), asked about, or — when it did neither — marked for a @@ -99,11 +103,11 @@ private async Task> DecideAsync(WorkspaceEntity b var retried = new HashSet(); foreach (var retry in decision.Retries) { - if (running >= AutopilotPlanner.MaxParallel) { break; } + if (running >= limits.MaxParallel) { break; } var task = stuck.First(s => s.IssueNumber == retry.IssueNumber); + // A guided retry is a change request: it counts as one more (persisted) fix round of the task. var started = await tasks.StartAsync(tenant, task.SessionId, changeRequest: retry.Guidance, ct: ct).ConfigureAwait(false); if (started != TaskStart.Started) { continue; } - _fixRounds.TryRemove(task.SessionId, out _); retried.Add(task.IssueNumber); running++; log.Add(tenant, board.Id, $"Orchestrator started #{task.IssueNumber} again with guidance: {Short(retry.Guidance)}"); @@ -265,27 +269,30 @@ private async Task CloseSupersededAsync(WorkspaceEntity board, string pat, IRead } // Tasks that stopped: couldn't finish, or blocked for a person — plus those the rules give up on this very pass. + // Work a plan limit cut short isn't stuck (it resumes by rule after the reset), so no retry is spent on it. private List StuckTasks(IReadOnlyList tickets, IReadOnlyList planTickets, - IReadOnlyList sessions, ILookup repos, IReadOnlyList candidates) + IReadOnlyList sessions, ILookup repos, IReadOnlyList candidates, + AutopilotLimits limits) { var openTickets = planTickets.Where(t => t.Open).Select(t => t.Number).ToHashSet(); var titles = tickets.Where(t => t.Number.HasValue).GroupBy(t => t.Number.GetValueOrDefault()).ToDictionary(g => g.Key, g => g.First().Title); var stuck = new List(); foreach (var s in sessions.Where(s => s.Status is "Failed" or "Blocked" && !workflows.IsRunning(s.TenantId, s.Id) - && s.IssueNumber.HasValue && openTickets.Contains(s.IssueNumber.GetValueOrDefault()))) + && s.IssueNumber.HasValue && openTickets.Contains(s.IssueNumber.GetValueOrDefault()) + && !(s.Status == "Failed" && CutShortByPlanLimit(s, repos)))) { var issue = s.IssueNumber.GetValueOrDefault(); var why = string.Join(" | ", repos[s.Id].Select(r => r.Error).Prepend(s.Error).Where(e => !string.IsNullOrWhiteSpace(e)).Distinct()); stuck.Add(new OrchestratorStuckTask(s.Id, issue, titles.GetValueOrDefault(issue, s.Title), $"{(s.Status == "Failed" ? "Couldn't finish" : "Waiting for a person")}: {(why.Length == 0 ? "no reason recorded" : why)}", - RetriesLeft(s.FixRounds))); + RetriesLeft(s.FixRounds, limits))); } foreach (var n in candidates.OfType().Where(n => stuck.All(s => s.SessionId != n.SessionId))) { var s = sessions.First(x => x.Id == n.SessionId); var findings = string.Join(" | ", repos[s.Id].Select(r => r.Error).Where(e => !string.IsNullOrWhiteSpace(e)).Distinct()); stuck.Add(new OrchestratorStuckTask(s.Id, n.IssueNumber, titles.GetValueOrDefault(n.IssueNumber, s.Title), - $"{n.Reason} {findings}".Trim(), RetriesLeft(s.FixRounds))); + $"{n.Reason} {findings}".Trim(), RetriesLeft(s.FixRounds, limits))); } return stuck; } @@ -301,7 +308,7 @@ internal static string Snapshot(IReadOnlyList tickets, IReadOnlyLis { var number = t.Number.GetValueOrDefault(); var p = plan.GetValueOrDefault(number); - sb.Append("#").Append(t.Number).Append(' ').Append(p?.Open == false ? "[closed]" : "[open]") + sb.Append('#').Append(t.Number).Append(' ').Append(p?.Open == false ? "[closed]" : "[open]") .Append(" [").Append(string.Join(", ", t.Labels)).Append(']'); if (p is { DependsOn.Count: > 0 }) { sb.Append(" depends on ").Append(string.Join(", ", p.DependsOn.Select(d => "#" + d))); } if (p is { HasLiveTask: true }) { sb.Append(" (has a task)"); } @@ -310,7 +317,7 @@ internal static string Snapshot(IReadOnlyList tickets, IReadOnlyLis sb.Append("\nTasks in progress:\n"); foreach (var task in planTasks) { - sb.Append("#").Append(task.IssueNumber).Append(' ').Append(task.Status).Append(", fix rounds used ").Append(task.FixRounds).Append('\n'); + sb.Append('#').Append(task.IssueNumber).Append(' ').Append(task.Status).Append(", fix rounds used ").Append(task.FixRounds).Append('\n'); foreach (var c in task.OpenChanges) { sb.Append(" PR #").Append(c.PrNumber).Append(": checks ").Append(c.Checks).Append(", review ").Append(c.Review) @@ -322,7 +329,7 @@ internal static string Snapshot(IReadOnlyList tickets, IReadOnlyLis } foreach (var s in sessions.Where(s => s.Status is "Failed" or "Blocked")) { - sb.Append("#").Append(s.IssueNumber).Append(' ').Append(s.Status).Append(": ").Append(Short(s.Error ?? "", 300)).Append('\n'); + sb.Append('#').Append(s.IssueNumber).Append(' ').Append(s.Status).Append(": ").Append(Short(s.Error ?? "", 300)).Append('\n'); } return sb.ToString(); } @@ -347,6 +354,7 @@ internal static string Signature(IEnumerable moves, IEnumerable private static string KindOf(AutopilotAction a) => a switch { AutopilotAction.StartTicket => "start", + AutopilotAction.ResumeTask => "resume", AutopilotAction.FixChecks => "fix", AutopilotAction.RunReview => "review", AutopilotAction.RunTests => "test", @@ -358,6 +366,7 @@ internal static string Signature(IEnumerable moves, IEnumerable private static int IssueOf(AutopilotAction a) => a switch { AutopilotAction.StartTicket x => x.Number, + AutopilotAction.ResumeTask x => x.IssueNumber, AutopilotAction.FixChecks x => x.IssueNumber, AutopilotAction.RunReview x => x.IssueNumber, AutopilotAction.RunTests x => x.IssueNumber, diff --git a/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.cs b/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.cs index befd3a0a..e93f2923 100644 --- a/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.cs +++ b/src/AgentOs.Web/Services/Orchestration/ProjectOrchestrator.cs @@ -1,7 +1,10 @@ // The project orchestrator: for a board with Autopilot on, it does what a person did by hand on the lab — reads the // board, decides (AutopilotPlanner), and acts through the same BoardTaskService the Board uses. Every decision is // logged in plain words for the Autopilot tab. One step = one pass over a board; ProjectOrchestratorWorker runs a -// step per autopilot board every minute, and "Check now" runs one on demand. +// step per autopilot board every minute, and "Check now" runs one on demand. Its bounds (tasks at once, fix rounds, +// retries) are the tenant's Settings › Autopilot, read every pass. When the member's computer reports a Claude/Codex +// plan limit (IPlanLimits), a pass does nothing until the plan resets — no new work, no retries, no fix rounds — and +// then picks up the work the limit cut short. using System; using System.Collections.Concurrent; @@ -9,9 +12,11 @@ using System.Linq; using System.Threading; using System.Threading.Tasks; +using AgentOs.Domain.Configuration; using AgentOs.Domain.Workspaces; using AgentOs.Modules.AppConfig; using AgentOs.Modules.Integration; +using AgentOs.Modules.Pipeline.Agents; using AgentOs.Modules.RemoteAgent; using AgentOs.Modules.Sessions.Persistence; using AgentOs.Modules.Sessions.Persistence.Entities; @@ -113,14 +118,18 @@ public sealed partial class ProjectOrchestrator( ILogger logger, AgentOs.Web.Services.Preview.PreviewService previews, TicketWorkflowRunner workflows, - RetrospectiveService retrospectives) + RetrospectiveService retrospectives, + IPlanLimits planLimits) : IDisposable { + /// + public void Dispose() => _oneAtATime.Dispose(); + // Tasks seen "Running" with no live run here, by pass: one pass of grace (a run registers right after it is marked). private readonly ConcurrentDictionary _orphanSeen = new(); // Boards whose testing is held for a running "Try the app" preview (logged once per hold, not every pass). private readonly ConcurrentDictionary _heldForPreview = new(); - // Automatic fix rounds per task (in memory: a restart grants at most another MaxFixRounds — bounded either way). - private readonly ConcurrentDictionary _fixRounds = new(); + // Boards waiting out their member's plan limit (logged once when the wait starts, once when it ends). + private readonly PlanLimitPause _planPause = new(); // Ticket bodies (for dependencies) change rarely; read each at most every 10 minutes. private readonly ConcurrentDictionary<(Guid, int), (DateTimeOffset At, IReadOnlyList Deps)> _deps = new(); private readonly SemaphoreSlim _oneAtATime = new(1, 1); @@ -138,6 +147,8 @@ public async Task> StepAsync(WorkspaceEntity boar try { var member = board.AutopilotUserId ?? board.CreatedByUserId ?? string.Empty; + // The member's AI plan hit its limit: nothing starts, nothing is retried, until it resets. + if (WaitsForPlan(board, member)) { return []; } using var identity = AmbientIdentity.Push(board.TenantId, member); using var scope = scopes.CreateScope(); var sp = scope.ServiceProvider; @@ -150,6 +161,7 @@ public async Task> StepAsync(WorkspaceEntity boar } var sessionsRepo = sp.GetRequiredService(); var prs = sp.GetRequiredService(); + var limits = await AutopilotSettings.ReadLimitsAsync(sp.GetRequiredService(), tenant, ct).ConfigureAwait(false); var boardDesc = new BoardDescriptor(board.Id, tenant, board.Kind, board.ProjectOwner, board.ProjectScope, board.ProjectNumber, board.ProjectNodeId, pat, board.Project); var tickets = (await sp.GetRequiredService().ListTicketsAsync(boardDesc, ct).ConfigureAwait(false)).Items @@ -164,14 +176,19 @@ public async Task> StepAsync(WorkspaceEntity boar } await CloseSupersededAsync(board, pat, sessions, repos, sp, ct).ConfigureAwait(false); - // Proposed changes: merged ones are recorded; open ones get their check state. + // Proposed changes: merged ones are recorded; open ones get their check state. Every task carries its + // persisted fix rounds — the one counter the caps read, whichever engine sent the change back. var planTasks = new List(); + foreach (var s in sessions.Where(s => s.Status is "Failed" && !workflows.IsRunning(tenant, s.Id) && CutShortByPlanLimit(s, repos))) + { + planTasks.Add(new PlanTask(s.Id, s.IssueNumber ?? 0, "Failed", [], s.FixRounds) { CutShortByPlanLimit = true }); + } foreach (var s in sessions.Where(s => s.Status is "Running" or "Done")) { // Its MAF workflow drives it: a running slot, and no step-by-step moves on it. if (workflows.IsRunning(tenant, s.Id)) { - planTasks.Add(new PlanTask(s.Id, s.IssueNumber ?? 0, "Running", [], _fixRounds.GetValueOrDefault(s.Id))); + planTasks.Add(new PlanTask(s.Id, s.IssueNumber ?? 0, "Running", [], s.FixRounds)); continue; } var open = new List(); @@ -207,7 +224,7 @@ public async Task> StepAsync(WorkspaceEntity boar var findings = testing == ChangeTesting.Rejected || review == ChangeReview.ChangesRequested ? r.Error : null; open.Add(new PlanChange(r.Id, number, checks, report, testing, findings, merge, r.DefaultBranch, review)); } - planTasks.Add(new PlanTask(s.Id, s.IssueNumber ?? 0, s.Status, open, _fixRounds.GetValueOrDefault(s.Id))); + planTasks.Add(new PlanTask(s.Id, s.IssueNumber ?? 0, s.Status, open, s.FixRounds)); } // Tickets: a ticket with any task that isn't closed is the person's or already handled — never doubled. @@ -239,7 +256,9 @@ public async Task> StepAsync(WorkspaceEntity boar { _heldForPreview.TryRemove(board.Id, out _); } - var actions = await DecideAsync(board, mode, member, pat, tickets, planTickets, planTasks, sessions, repos, canTest, sp, ct).ConfigureAwait(false); + var actions = await DecideAsync(board, mode, member, pat, tickets, planTickets, planTasks, sessions, repos, canTest, limits, sp, ct).ConfigureAwait(false); + // Deciding may itself have hit the limit (the Orchestrator agent answers on the member's computer). + if (WaitsForPlan(board, member)) { return []; } foreach (var action in actions) { await ExecuteAsync(action, board, member, pat, tickets, sp, ct).ConfigureAwait(false); @@ -290,8 +309,15 @@ private async Task ExecuteAsync(AutopilotAction action, WorkspaceEntity board, s : $"Couldn't start #{start.Number}: {started}."); break; } + case AutopilotAction.ResumeTask resume: + { + // A plain restart: no change request, so no fix round is counted. + var resumed = await tasks.StartAsync(tenant, resume.SessionId, ct: ct).ConfigureAwait(false); + log.Add(tenant, board.Id, resumed == TaskStart.Started ? resume.Why : $"Couldn't pick #{resume.IssueNumber} up again: {resumed}."); + break; + } case AutopilotAction.FixChecks fix: - _fixRounds[fix.SessionId] = fix.Round; + // The round is counted (persisted) by the restart itself: BoardTaskService increments FixRounds. var restarted = await tasks.StartAsync(tenant, fix.SessionId, "The pull request's automated checks failed. Fix the cause (not the test, unless the test itself is wrong) so they pass:\n\n" + fix.Report, ct: ct).ConfigureAwait(false); @@ -328,7 +354,6 @@ private async Task ExecuteAsync(AutopilotAction action, WorkspaceEntity board, s { await sessions.UpdateRepoRunResultAsync(tenant, repo.Id, "Merged", repo.BranchName, repo.PrUrl, null, ct).ConfigureAwait(false); await sessions.RecomputeSessionStatusForTenantAsync(tenant, merge.SessionId, ct).ConfigureAwait(false); - _fixRounds.TryRemove(merge.SessionId, out _); log.Add(tenant, board.Id, merge.Why); // Self-improvement: what this ticket taught becomes the project's practices. retrospectives.AfterMerge(tenant, member, board.Id, $"{repo.Owner}/{repo.Repo}", $"#{merge.IssueNumber}"); @@ -389,6 +414,18 @@ private async Task RecoverOrphansAsync(WorkspaceEntity board, IReadOnlyLis return changed; } + // True while the board's member waits out a plan limit (and logs the start and the end of the wait once each). + private bool WaitsForPlan(WorkspaceEntity board, string member) + { + var (paused, line) = _planPause.Observe(board.Id, planLimits.Current(board.TenantId, member)); + if (line is not null) { log.Add(board.TenantId, board.Id, line); } + return paused; + } + + // A failed task whose run the member's plan limit cut short (its error is the CLI's limit notice) — nothing failed. + internal static bool CutShortByPlanLimit(RemoteSessionEntity session, ILookup repos) => + PlanLimitDetector.IsPlanLimit(session.Error) || repos[session.Id].Any(r => r.Status == "Failed" && PlanLimitDetector.IsPlanLimit(r.Error)); + private async Task> DependenciesAsync(ISourceProvider? provider, WorkspaceEntity board, string tenant, string pat, BoardTicket t, CancellationToken ct) { if (t.Number is not { } number) { return []; } diff --git a/src/AgentOs.Web/Services/Orchestration/TicketWorkflow.cs b/src/AgentOs.Web/Services/Orchestration/TicketWorkflow.cs index 9ff6d444..1f49b661 100644 --- a/src/AgentOs.Web/Services/Orchestration/TicketWorkflow.cs +++ b/src/AgentOs.Web/Services/Orchestration/TicketWorkflow.cs @@ -1,10 +1,12 @@ // One ticket's whole agent flow as a Microsoft Agent Framework Workflow: the Autopilot Orchestrator agent decides // WHICH ticket starts; this graph then runs it — Coding → checks → Reviewer → Testing → merge — routing each verdict // along a conditional edge (red checks, conflicts, requested changes and failed acceptance go back to the Coding agent, -// bounded by the same 3 rounds the rules use). Each step is one of the existing agents (ITicketSteps — the Board's +// bounded by the same fix rounds the rules use — Settings › Autopilot, counted from the task's persisted FixRounds, so +// the cap is the true total across engines and restarts). Each step is one of the existing agents (ITicketSteps — the Board's // BoardTicketSteps in production), so governance, metrics (Insights) and evidence on the PR are unchanged. When the // graph ends without a merge — "needs a person", or an agent that couldn't run — the task is handed back to the -// Orchestrator agent and the step-by-step rules, which also pick up any flow a restart cut short. +// Orchestrator agent and the step-by-step rules, which also pick up any flow a restart cut short. A member's plan limit +// (their computer's Claude/Codex plan said "no more for now") pauses the flow before it spends a round on it. using System; using System.Collections.Generic; @@ -38,6 +40,8 @@ public enum TicketStage NeedsPerson, /// An agent couldn't run or finish — handed back to the step-by-step path. Stopped, + /// The member's AI plan hit its limit — handed back; Autopilot picks it up after the reset. + PlanLimited, } /// The state a ticket's workflow threads along every edge. @@ -49,8 +53,11 @@ public sealed class TicketFlow(string tenantId, Guid boardId, Guid sessionId, in public int IssueNumber { get; } = issueNumber; public bool MayMerge { get; } = mayMerge; public TicketStage Stage { get; set; } = TicketStage.Fix; - /// Automatic fix rounds used. + /// Automatic fix rounds used — seeded from the task's persisted fix rounds, so a flow started after a + /// restart (or after step-by-step rounds) has only what is left of the cap. public int Round { get; set; } + /// The cap on (Settings › Autopilot › Automatic fix rounds). + public int MaxRounds { get; init; } = AgentOs.Modules.Pipeline.Agents.AutopilotLimits.Default.MaxFixRounds; /// What the Coding agent is asked to change next (null = the ticket itself, first attempt). public string? NextChange { get; set; } /// Why the flow stopped (plain language), when it did. @@ -61,16 +68,16 @@ public sealed class TicketFlow(string tenantId, Guid boardId, Guid sessionId, in /// Send the change back to the Coding agent with — or stop when the rounds are used up. public TicketFlow SendBack(string what, string change) { - if (Round >= AutopilotPlanner.MaxFixRounds) + if (Round >= MaxRounds) { Stage = TicketStage.NeedsPerson; - Reason = $"{what} after {AutopilotPlanner.MaxFixRounds} automatic fixes — needs a person."; + Reason = $"{what} after {MaxRounds} automatic fixes — needs a person."; return this; } Round++; NextChange = change; Stage = TicketStage.Fix; - Trail.Add($"{what.ToLowerInvariant()} → back to Coding (try {Round} of {AutopilotPlanner.MaxFixRounds})"); + Trail.Add($"{what.ToLowerInvariant()} → back to Coding (try {Round} of {MaxRounds})"); return this; } @@ -80,6 +87,13 @@ internal TicketFlow Stop(string why) Reason = why; return this; } + + internal TicketFlow PausedForPlan(string why) + { + Stage = TicketStage.PlanLimited; + Reason = why; + return this; + } } /// A proposed change of the ticket's task, as a step reads it. @@ -108,6 +122,9 @@ public interface ITicketSteps Task UpdateBranchAsync(TicketFlow flow, TicketChange change, CancellationToken ct); /// Merge a change and record it; null on success, else the reason. Task MergeAsync(TicketFlow flow, TicketChange change, CancellationToken ct); + + /// Why no agent should run now — the member's AI plan is at its limit (plain words) — or null. + string? PlanLimited(TicketFlow flow) => null; } /// Builds and runs a ticket's MAF workflow. @@ -148,13 +165,19 @@ public static async Task RunAsync(ITicketSteps steps, TicketFlow flo return run.NewEvents.OfType().Select(e => e.Data).OfType().LastOrDefault() ?? flow; } + // The member's plan is at its limit: stop before an agent runs (and after one ran into it) — a round spent then + // would be spent on nothing. + private static TicketFlow? PlanPause(ITicketSteps steps, TicketFlow f) => steps.PlanLimited(f) is { } why ? f.PausedForPlan(why) : null; + private sealed class CodeExecutor(ITicketSteps steps) : Executor("code") { public override async ValueTask HandleAsync(TicketFlow f, IWorkflowContext context, CancellationToken ct) { + if (PlanPause(steps, f) is { } before) { return before; } var run = await steps.CodeAsync(f, ct).ConfigureAwait(false); if (run.Status != TaskStart.Started) { return f.Stop($"The Coding agent couldn't start: {run.Status}."); } await run.Completion.ConfigureAwait(false); + if (PlanPause(steps, f) is { } after) { return after; } var changes = await steps.ChangesAsync(f, ct).ConfigureAwait(false); if (changes.Count == 0 || changes.Any(c => c.Status is "Failed" or "Blocked" || c.Pr is null)) { @@ -216,9 +239,11 @@ private sealed class ReviewExecutor(ITicketSteps steps) : Executor HandleAsync(TicketFlow f, IWorkflowContext context, CancellationToken ct) { + if (PlanPause(steps, f) is { } before) { return before; } var run = await steps.ReviewAsync(f, ct).ConfigureAwait(false); if (run.Status != TaskStart.Started) { return f.Stop($"The Reviewer agent couldn't start: {run.Status}."); } await run.Completion.ConfigureAwait(false); + if (PlanPause(steps, f) is { } after) { return after; } var changes = await steps.ChangesAsync(f, ct).ConfigureAwait(false); if (changes.Any(c => c.Status == "ChangesRequested")) { @@ -239,9 +264,11 @@ private sealed class TestExecutor(ITicketSteps steps) : Executor HandleAsync(TicketFlow f, IWorkflowContext context, CancellationToken ct) { + if (PlanPause(steps, f) is { } before) { return before; } var run = await steps.TestAsync(f, ct).ConfigureAwait(false); if (run.Status != TaskStart.Started) { return f.Stop($"The Testing agent couldn't start: {run.Status}."); } await run.Completion.ConfigureAwait(false); + if (PlanPause(steps, f) is { } after) { return after; } var changes = await steps.ChangesAsync(f, ct).ConfigureAwait(false); if (changes.Any(c => c.Status == "Rejected")) { diff --git a/src/AgentOs.Web/Services/Orchestration/TicketWorkflowRunner.cs b/src/AgentOs.Web/Services/Orchestration/TicketWorkflowRunner.cs index 36bd17d7..b27d634d 100644 --- a/src/AgentOs.Web/Services/Orchestration/TicketWorkflowRunner.cs +++ b/src/AgentOs.Web/Services/Orchestration/TicketWorkflowRunner.cs @@ -3,12 +3,17 @@ // its trail goes to the Autopilot log and anything unfinished is handed back: out of rounds → marked for a person (the // Orchestrator agent may retry it with guidance or ask); an agent that couldn't run → the step-by-step path continues // from where the task stands. A restart drops the in-memory runs; the same step-by-step path picks those tasks up. +// Each flow reads the tenant's Settings › Autopilot bounds when it starts (fix rounds, checks timeout and poll), and +// its round counter starts from the task's persisted fix rounds — one count across engines and restarts. using System; using System.Collections.Concurrent; using System.Linq; using System.Threading; using System.Threading.Tasks; +using AgentOs.Domain.Configuration; +using AgentOs.Modules.Pipeline.Agents; +using AgentOs.Modules.RemoteAgent; using AgentOs.Modules.Sessions.Persistence; using AgentOs.Modules.Workspaces.Persistence.Entities; using AgentOs.Web.Services.BoardTasks; @@ -19,14 +24,9 @@ namespace AgentOs.Web.Services.Orchestration; /// Runs ticket workflows. Singleton. public sealed class TicketWorkflowRunner(IServiceScopeFactory scopes, BoardTaskService tasks, OrchestratorLog log, ILogger logger, - RetrospectiveService retrospectives) + RetrospectiveService retrospectives, IPlanLimits planLimits) { private readonly ConcurrentDictionary<(string Tenant, Guid Session), Task> _running = new(); - /// How long a workflow waits for a pull request's checks. - public TimeSpan ChecksTimeout { get; init; } = TimeSpan.FromMinutes(60); - - /// How often it looks at them. - public TimeSpan PollInterval { get; init; } = TimeSpan.FromSeconds(60); /// True while the task's workflow runs. public bool IsRunning(string tenantId, Guid sessionId) => _running.ContainsKey((tenantId, sessionId)); @@ -42,8 +42,9 @@ public bool Start(WorkspaceEntity board, string pat, Guid sessionId, int issueNu { try { - var flow = await TicketWorkflow.RunAsync(new BoardTicketSteps(tasks, scopes, board, pat), - new TicketFlow(board.TenantId, board.Id, sessionId, issueNumber, mayMerge), ChecksTimeout, PollInterval, CancellationToken.None) + var (start, limits) = await SeedAsync(board, sessionId, issueNumber, mayMerge).ConfigureAwait(false); + var flow = await TicketWorkflow.RunAsync(new BoardTicketSteps(tasks, scopes, board, pat, planLimits), + start, limits.ChecksTimeout, limits.ChecksPoll, CancellationToken.None) .ConfigureAwait(false); await FinishAsync(board, flow).ConfigureAwait(false); } @@ -61,6 +62,20 @@ public bool Start(WorkspaceEntity board, string pat, Guid sessionId, int issueNu return true; } + // The tenant's bounds now, and the flow's round counter from the task's persisted fix rounds. + private async Task<(TicketFlow Flow, AutopilotLimits Limits)> SeedAsync(WorkspaceEntity board, Guid sessionId, int issueNumber, bool mayMerge) + { + using var scope = scopes.CreateScope(); + var limits = await AutopilotSettings.ReadLimitsAsync(scope.ServiceProvider.GetRequiredService(), board.TenantId).ConfigureAwait(false); + var task = (await scope.ServiceProvider.GetRequiredService().ListForTenantAsync(board.TenantId).ConfigureAwait(false)) + .FirstOrDefault(s => s.Id == sessionId); + return (Seed(board.TenantId, board.Id, sessionId, issueNumber, mayMerge, task?.FixRounds ?? 0, limits), limits); + } + + /// A ticket's flow, its rounds counted from under . + internal static TicketFlow Seed(string tenantId, Guid boardId, Guid sessionId, int issueNumber, bool mayMerge, int persistedFixRounds, AutopilotLimits limits) => + new(tenantId, boardId, sessionId, issueNumber, mayMerge) { Round = Math.Max(0, persistedFixRounds), MaxRounds = limits.MaxFixRounds }; + private async Task FinishAsync(WorkspaceEntity board, TicketFlow flow) { var steps = flow.Trail.Count == 0 ? "" : " (" + string.Join(" → ", flow.Trail) + ")"; @@ -69,6 +84,7 @@ private async Task FinishAsync(WorkspaceEntity board, TicketFlow flow) TicketStage.Merged => $"#{flow.IssueNumber} · MAF workflow: merged{steps}.", TicketStage.ReadyToMerge => $"#{flow.IssueNumber} · MAF workflow: reviewed and tested — ready for you to merge{steps}.", TicketStage.NeedsPerson => $"#{flow.IssueNumber} · MAF workflow: {flow.Reason}{steps}", + TicketStage.PlanLimited => $"#{flow.IssueNumber} · MAF workflow paused for your computer's plan limit{steps} — Autopilot picks it up after the reset.", _ => $"#{flow.IssueNumber} · MAF workflow paused: {flow.Reason}{steps} — Autopilot continues step by step.", }; log.Add(board.TenantId, board.Id, line); diff --git a/src/AgentOs.Web/Services/SettingsLinks.cs b/src/AgentOs.Web/Services/SettingsLinks.cs index abbbadc9..d0199268 100644 --- a/src/AgentOs.Web/Services/SettingsLinks.cs +++ b/src/AgentOs.Web/Services/SettingsLinks.cs @@ -19,6 +19,9 @@ public static class SettingsLinks /// Settings → AI → Model keys: where every "no usable key" message points. public const string ModelKeys = "keys"; + /// Settings → AI → Autopilot: where Board › Autopilot points for how Autopilot runs. + public const string Autopilot = "autopilot"; + // The old two-level targets → the flat page keys. private static readonly FrozenDictionary Legacy = new Dictionary(StringComparer.Ordinal) { diff --git a/src/AgentOs.Web/Services/SettingsSearch.cs b/src/AgentOs.Web/Services/SettingsSearch.cs index d1e60010..23e2d7cd 100644 --- a/src/AgentOs.Web/Services/SettingsSearch.cs +++ b/src/AgentOs.Web/Services/SettingsSearch.cs @@ -40,6 +40,7 @@ private sealed record Page(string Target, string Title, string Path, string Icon new("routing", "Routing", Ai, "lightning", "force provider default provider routing test connection offline fallback remoteagent remote agent paired", Access.Admin), new("agents", "Agent profiles", Ai, "diamond", "agent model per agent temperature max tokens output qa loop iterations cap profile coding testing requirement orchestrator", Access.Admin), new("prompts", "Prompts", Ai, "file", "prompt system prompt workspace rules instructions version history rollback", Access.Admin), + new("autopilot", "Autopilot", Ai, "play", "autopilot orchestrator decided by rules maf workflow step by step tasks at once parallel acceptance tests fix rounds retries checks timeout poll merge plan limit", Access.Admin), new("github", "GitHub", Conn, "git-pull-request", "github token pat pull request repo owner pipeline", Access.Admin), new("mcp", "MCP servers", Conn, "arrow-square-out", "mcp model context protocol tool servers upstream", Access.Admin), new("plugins", "Plugins", Conn, "squares-stack", "plugins extensions add-ons", Access.Everyone), @@ -79,6 +80,7 @@ _ when NoEditor.Contains(d.Key) => null, { Category: "Agent profiles" or "Pipeline" } => "agents", { Category: "LLM providers" } => ModelKeySettings.Contains(d.Key) ? "keys" : "routing", { Category: "Build sandbox" or "Platform" } => "platform", + { Category: AgentOs.Modules.Pipeline.Agents.AutopilotSettings.Category } => SettingsLinks.Autopilot, _ => null, }; diff --git a/src/AgentOs.Web/Services/TaskFailure.cs b/src/AgentOs.Web/Services/TaskFailure.cs index c8346557..fc39a848 100644 --- a/src/AgentOs.Web/Services/TaskFailure.cs +++ b/src/AgentOs.Web/Services/TaskFailure.cs @@ -58,13 +58,19 @@ private static readonly (string Kind, string[] Needles, string Headline, string TaskFailureAction.TryAgain, "Try again"), ]; - private static readonly TaskFailureInfo Unknown = new("unknown", "The AI couldn't finish this task", + private static readonly TaskFailureInfo PlanLimit = new("plan-limit", "Your computer's AI plan hit its limit", + "Your Claude or Codex plan is used up for now. Nothing is lost — Autopilot picks this up again after the reset, or try again then.", + TaskFailureAction.TryAgain, "Try again"); + + private static readonly TaskFailureInfo Unknown =new("unknown", "The AI couldn't finish this task", "See the details below. Trying again often works.", TaskFailureAction.TryAgain, "Try again"); /// Explain ; null/blank → null (nothing failed). public static TaskFailureInfo? Explain(string? error) { if (string.IsNullOrWhiteSpace(error)) { return null; } + // The member's Claude/Codex plan said "not now" — nothing failed (the same detector Autopilot waits on). + if (AgentOs.Modules.RemoteAgent.PlanLimitDetector.IsPlanLimit(error)) { return PlanLimit; } foreach (var r in Rules) { foreach (var needle in r.Needles) diff --git a/src/AgentOs.Web/wwwroot/app.css b/src/AgentOs.Web/wwwroot/app.css index ac07113e..6189f534 100644 --- a/src/AgentOs.Web/wwwroot/app.css +++ b/src/AgentOs.Web/wwwroot/app.css @@ -2721,6 +2721,10 @@ button.syn-live-tag:focus-visible { outline: 2px solid var(--accent); outline-of /* Autopilot decisions — newest first, time then sentence. */ .autopilot-log { list-style: none; margin: var(--space-3) 0 0; padding: 0; display: flex; flex-direction: column; gap: var(--space-1); font-size: var(--fs-xs); max-height: 22rem; overflow: auto; } .autopilot-log li { display: flex; gap: var(--space-2); } +/* Settings › Autopilot — the rules that are not settings, in plain words. */ +.autopilot-rules { margin: 0; padding-left: var(--space-4); display: flex; flex-direction: column; gap: var(--space-2); font-size: var(--fs-sm); max-width: 72ch; } +/* A plan-limit wait is not a failure: the card reads as a warning (Board › Autopilot / Tasks). */ +.board-failure[data-kind="plan-limit"] { border-left-color: var(--warn); } /* Ask for changes — a short review in words for the open proposed change. */ .board-changes { display: flex; flex-direction: column; gap: var(--space-2); max-width: 72ch; } .board-changes textarea { width: 100%; box-sizing: border-box; resize: vertical; } diff --git a/tests/AgentOs.E2E.Tests/AgentOsUi/AutopilotSettingsTests.cs b/tests/AgentOs.E2E.Tests/AgentOsUi/AutopilotSettingsTests.cs new file mode 100644 index 00000000..e9bf65e2 --- /dev/null +++ b/tests/AgentOs.E2E.Tests/AgentOsUi/AutopilotSettingsTests.cs @@ -0,0 +1,98 @@ +// Settings → AI → Autopilot (ADR-0011): Autopilot is configured from the desktop. The page renders every Autopilot:* +// descriptor (who decides, how a ticket runs, the bounds, the checks wait) plus the fixed rules in plain words; a bound +// saves at tenant scope and resets; the range is enforced; and Ctrl+K search finds the page by what people type. + +using System.Threading.Tasks; +using Microsoft.Playwright; +using Xunit; + +namespace AgentOs.E2E.Tests.AgentOsUi; + +public sealed class AutopilotSettingsTests : IClassFixture +{ + private readonly AgentOsPageFixture _fx; + + public AutopilotSettingsTests(AgentOsPageFixture fx) => _fx = fx; + + private async Task OpenAsync() + { + await _fx.GotoDesktopAsync(); + await Assertions.Expect(_fx.Page.Locator("#components-reconnect-modal")).Not.ToBeVisibleAsync(new() { Timeout = 30_000 }); + await _fx.Page.Locator(".dock-item[title=\"Settings\"]").First.ClickAsync(); + var win = _fx.Page.Locator(".appwin.focused"); + await Assertions.Expect(win.Locator(".appwin-title")).ToHaveTextAsync("Settings"); + await win.Locator(".prefs-cat", new() { HasTextString = "Autopilot" }).ClickAsync(); + await Assertions.Expect(win.Locator(".page-title")).ToHaveTextAsync("Autopilot"); + return win; + } + + [Fact] + public async Task Autopilot_ShowsEverySetting_AndTheRealMergeRule() + { + if (!AgentOsPageFixture.IsEnabled) { Assert.Skip(AgentOsPageFixture.SkipReason); } + + var win = await OpenAsync(); + + foreach (var label in new[] + { + "Decided by", "Each ticket runs as", "Tasks at once", "Acceptance tests at once", "Automatic fix rounds", + "Orchestrator retries", "Wait for checks (minutes)", "Look at checks every (seconds)", + }) + { + await Assertions.Expect(win.Locator(".prefs-label strong").GetByText(label, new() { Exact = true })).ToBeVisibleAsync(); + } + await Assertions.Expect(win.Locator("input[aria-label=\"Tasks at once\"]")).ToHaveValueAsync("2"); + await Assertions.Expect(win.Locator("input[aria-label=\"Automatic fix rounds\"]")).ToHaveValueAsync("3"); + await Assertions.Expect(win.Locator(".autopilot-rules")).ToContainTextAsync("the Reviewer agent approved it and the Testing agent passed it"); + await Assertions.Expect(win.Locator(".autopilot-rules")).ToContainTextAsync("plan hits its limit"); + } + + [Fact] + public async Task Autopilot_ABound_SavesAtTenantScope_RejectsOutOfRange_AndResets() + { + if (!AgentOsPageFixture.IsEnabled) { Assert.Skip(AgentOsPageFixture.SkipReason); } + + var win = await OpenAsync(); + var row = win.Locator(".prefs-row:has(input[aria-label=\"Tasks at once\"])"); + var input = row.Locator("input[aria-label=\"Tasks at once\"]"); + try + { + await input.FillAsync("99"); + await input.PressAsync("Tab"); + await row.GetByRole(AriaRole.Button, new() { Name = "Save" }).ClickAsync(); + await Assertions.Expect(_fx.Page.Locator(".toast").Last).ToContainTextAsync("6"); // the range's upper bound, in the reason + await Assertions.Expect(input).ToHaveValueAsync("2"); + + await input.FillAsync("3"); + await input.PressAsync("Tab"); + await row.GetByRole(AriaRole.Button, new() { Name = "Save" }).ClickAsync(); + await Assertions.Expect(row.Locator(".badgepill").First).ToContainTextAsync("This tenant"); + await Assertions.Expect(input).ToHaveValueAsync("3"); + } + finally + { + // Zero residue: the tenant value goes back to the default. + await row.GetByRole(AriaRole.Button, new() { Name = "Reset" }).ClickAsync(); + await Assertions.Expect(row.Locator(".badgepill").First).Not.ToContainTextAsync("This tenant"); + } + await Assertions.Expect(input).ToHaveValueAsync("2"); + } + + [Fact] + public async Task CtrlK_FixRounds_OpensSettingsAutopilot() + { + if (!AgentOsPageFixture.IsEnabled) { Assert.Skip(AgentOsPageFixture.SkipReason); } + await _fx.GotoDesktopAsync(); + + await _fx.Page.Keyboard.PressAsync("Control+k"); + var input = _fx.Page.Locator(".ko-search-input"); + await Assertions.Expect(input).ToBeFocusedAsync(); + await input.PressSequentiallyAsync("fix rounds"); + var hit = _fx.Page.Locator(".ko-row", new() { HasTextString = "Settings › AI" }).First; + await Assertions.Expect(hit).ToBeVisibleAsync(); + await hit.ClickAsync(); + + var win = _fx.Page.Locator(".appwin.focused"); + await Assertions.Expect(win.Locator(".page-title")).ToHaveTextAsync("Autopilot"); + } +} diff --git a/tests/AgentOs.Tests/Llm/PlanLimitTests.cs b/tests/AgentOs.Tests/Llm/PlanLimitTests.cs new file mode 100644 index 00000000..43386a87 --- /dev/null +++ b/tests/AgentOs.Tests/Llm/PlanLimitTests.cs @@ -0,0 +1,196 @@ +// A member's Claude / Codex plan limit, read from what their computer's CLI answered — real-shaped lines (claude -p +// on the lab, 2026-10: "You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)"; older Claude Code's +// "usage limit reached|"; Codex's "try again in …") — and never from an answer that merely mentions a limit. +// The RemoteAgent provider turns a limit into PlanLimitException + a per-member wait Autopilot honours. + +using System; +using System.Globalization; +using System.Threading.Tasks; +using AgentOs.Domain.Llm; +using AgentOs.Modules.RemoteAgent; +using AgentOs.SharedKernel.Identity; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using NSubstitute; +using Shouldly; +using Xunit; + +namespace AgentOs.Tests.Llm; + +public sealed class PlanLimitTests +{ + // 17:00 in Ho Chi Minh City (UTC+7), 11:00 in Berlin (CET, UTC+1 after 25 Oct 2026). + private static readonly DateTimeOffset Now = new(2026, 10, 24, 10, 0, 0, TimeSpan.Zero); + + // The test host runs with invariant globalization (no ICU on Windows), so IANA zones are resolved here. + private static TimeZoneInfo? Zones(string id) => id switch + { + "Asia/Ho_Chi_Minh" => TimeZoneInfo.CreateCustomTimeZone(id, TimeSpan.FromHours(7), id, id), + "Europe/Berlin" => TimeZoneInfo.CreateCustomTimeZone(id, TimeSpan.FromHours(1), id, id), + _ => null, + }; + + [Fact] + public void Detect_ClaudeSessionLimit_ResetInTheNamedZone() + { + var limit = PlanLimitDetector.Detect("You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", "claude", Now, Zones); + + limit.ShouldNotBeNull(); + limit!.Plan.ShouldBe("Claude"); + limit.ResetsAt.ShouldBe(new DateTimeOffset(2026, 10, 24, 23, 50, 0, TimeSpan.FromHours(7))); + limit.InNamedZone.ShouldBeTrue(); + limit.ResetText.ShouldBe("23:50"); + limit.Message.ShouldBe("You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)"); + } + + [Fact] + public void Detect_ResetTimeAlreadyPassedToday_IsTomorrow() + { + var lateEvening = new DateTimeOffset(2026, 10, 24, 16, 55, 0, TimeSpan.Zero); // 23:55 in Ho Chi Minh City + + PlanLimitDetector.Detect("You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", null, lateEvening, Zones)! + .ResetsAt.ShouldBe(new DateTimeOffset(2026, 10, 25, 23, 50, 0, TimeSpan.FromHours(7))); + } + + // The runner reports a non-zero exit as "exit N: " after earlier output. + [Fact] + public void Detect_InARunnerError_KeepsOnlyTheCliLine() + { + var limit = PlanLimitDetector.Detect("exit 1: Cloning into 'repo'...\nYou've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", null, Now, Zones); + + limit!.Message.ShouldBe("You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)"); + PlanLimitDetector.Detect("exit 1: You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", null, Now, Zones)! + .Message.ShouldStartWith("You've hit"); + } + + [Fact] + public void Detect_WeeklyLimit_WithADate() + { + PlanLimitDetector.Detect("You've reached your weekly limit · resets Oct 27, 5pm (Europe/Berlin)", null, Now, Zones)! + .ResetsAt.ShouldBe(new DateTimeOffset(2026, 10, 27, 17, 0, 0, TimeSpan.FromHours(1))); + } + + [Fact] + public void Detect_OlderClaudeCode_EpochReset() + { + var reset = new DateTimeOffset(2026, 10, 24, 15, 0, 0, TimeSpan.Zero); + var limit = PlanLimitDetector.Detect("Claude AI usage limit reached|" + reset.ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture), null, Now); + + limit!.ResetsAt.ShouldBe(reset); + limit.InNamedZone.ShouldBeFalse(); + limit.ResetText.ShouldBe("15:00 UTC"); + } + + [Fact] + public void Detect_Codex_TryAgainInADuration() + { + var limit = PlanLimitDetector.Detect( + "■ You've hit your usage limit. Upgrade to Pro (https://openai.com/chatgpt/pricing) or try again in 2 hours 5 minutes.", "codex", Now); + + limit!.Plan.ShouldBe("Codex"); + limit.ResetsAt.ShouldBe(Now + TimeSpan.FromHours(2) + TimeSpan.FromMinutes(5)); + } + + // A wall-clock reset without a zone can't be placed (the server doesn't know the computer's zone): still a limit. + [Fact] + public void Detect_ResetWithoutAZone_IsALimitWithAnUnknownReset() + { + var limit = PlanLimitDetector.Detect("5-hour limit reached ∙ resets 3pm", null, Now, Zones); + + limit.ShouldNotBeNull(); + limit!.ResetsAt.ShouldBeNull(); + limit.ResetText.ShouldBeNull(); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + [InlineData("Approaching usage limit · resets at 3pm")] + [InlineData("Added handling for when the API rate limit reached its cap.")] + [InlineData("{\"branch\":\"issue-7-ai-fix\",\"summary\":\"Show the session limit in the footer\"}")] + public void Detect_NotALimit_IsNull(string? text) => PlanLimitDetector.Detect(text, null, Now, Zones).ShouldBeNull(); + + // A clean reply is work unless it is short: an answer that quotes a limit notice inside real output stays work. + [Fact] + public void FromReply_CleanReply_CountsOnlyWhenShort() + { + const string notice = "You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)"; + + PlanLimitDetector.FromReply(true, notice, null, null, Now, Zones).ShouldNotBeNull(); + PlanLimitDetector.FromReply(true, new string('x', 700) + notice, null, null, Now, Zones).ShouldBeNull(); + PlanLimitDetector.FromReply(false, "", "exit 1: " + new string('x', 5000) + "\n" + notice, null, Now, Zones).ShouldNotBeNull(); + PlanLimitDetector.FromReply(true, "ok", null, null, Now, Zones).ShouldBeNull(); + } + + [Fact] + public void IsPlanLimit_ReadsAStoredError() => + PlanLimitDetector.IsPlanLimit("LLM call failed: You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)").ShouldBeTrue(); + + // ── The per-member wait ── + + private sealed class ManualTime(DateTimeOffset now) : TimeProvider + { + public DateTimeOffset Now { get; set; } = now; + public override DateTimeOffset GetUtcNow() => Now; + } + + [Fact] + public void Tracker_WaitsUntilTheResetPlusAMinute_PerMember() + { + var time = new ManualTime(Now); + var tracker = new PlanLimitTracker(time); + var limit = PlanLimitDetector.Detect("You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", null, Now, Zones)!; + + tracker.Report("t1", "alice", limit); + + var hold = tracker.Current("t1", "alice"); + hold.ShouldNotBeNull(); + hold!.UntilUtc.ShouldBe(new DateTimeOffset(2026, 10, 24, 16, 51, 0, TimeSpan.Zero)); + hold.Sentence.ShouldBe("Your computer's Claude plan hit its limit — Autopilot waits until 23:50, nothing is lost."); + tracker.Current("t1", "bob").ShouldBeNull(); // another member's computer, another plan + tracker.Current("t2", "alice").ShouldBeNull(); // another tenant + + time.Now = new DateTimeOffset(2026, 10, 24, 16, 50, 30, TimeSpan.Zero); + tracker.Current("t1", "alice").ShouldNotBeNull(); // the reset, but not yet the minute after it + time.Now = new DateTimeOffset(2026, 10, 24, 16, 51, 0, TimeSpan.Zero); + tracker.Current("t1", "alice").ShouldBeNull(); + } + + [Fact] + public void Tracker_UnknownReset_ChecksAgainLater() + { + var tracker = new PlanLimitTracker(new ManualTime(Now)); + + tracker.Report("t1", "alice", new PlanLimit("Claude", "5-hour limit reached", null)); + + var hold = tracker.Current("t1", "alice")!; + hold.UntilUtc.ShouldBe(Now + PlanLimitTracker.UnknownResetWait + PlanLimitTracker.Grace); + hold.Sentence.ShouldBe("Your computer's Claude plan hit its limit — Autopilot checks again at 10:31 UTC, nothing is lost."); + } + + // ── The provider ── + + [Fact] + public async Task Client_PlanLimitReply_ThrowsTheTypedSignal_AndRecordsTheMembersWait() + { + var broker = new InProcessRemoteAgentBroker(); + using var reg = broker.RegisterRunner("conn-1", new RunnerConnection(Guid.NewGuid(), "t1", "alice")); + broker.Dispatched += d => broker.Complete(new RemoteExecResult(d.Request.Id, false, "", + "You've hit your session limit · resets 11:50pm (UTC)")); + var limits = new PlanLimitTracker(TimeProvider.System); + var ctx = Substitute.For(); + ctx.TenantId.Returns("t1"); + ctx.UserId.Returns("alice"); + var services = new ServiceCollection(); + services.AddScoped(_ => ctx); + services.AddSingleton(limits); + using var sp = services.BuildServiceProvider(); + var client = new RemoteAgentLlmClient(broker, sp, NullLogger.Instance); + + var ex = await Should.ThrowAsync(client.SendAsync(new LlmRequest("s", "u", "m"))); + + ex.ShouldBeAssignableTo(); // callers that catch provider failures keep working + ex.Limit!.Plan.ShouldBe("Claude"); + limits.Current("t1", "alice").ShouldNotBeNull(); + } +} diff --git a/tests/AgentOs.Tests/Web/AutopilotConfigurationTests.cs b/tests/AgentOs.Tests/Web/AutopilotConfigurationTests.cs new file mode 100644 index 00000000..cc85aca9 --- /dev/null +++ b/tests/AgentOs.Tests/Web/AutopilotConfigurationTests.cs @@ -0,0 +1,181 @@ +// Autopilot is configured in Settings › Autopilot (ADR-0011) and waits out a member's plan limit: +// the descriptors are bounded with today's values as defaults, the typed read clamps whatever is stored, the plan-limit +// pause logs once when it starts and once when it ends, work a limit cut short is recognised from its stored error, +// and the acceptance-run gate honours the configured cap per member's computer. + +using System; +using System.Collections.Generic; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; +using AgentOs.Domain.Configuration; +using AgentOs.Modules.Pipeline.Agents; +using AgentOs.Modules.RemoteAgent; +using AgentOs.Modules.Sessions.Persistence.Entities; +using AgentOs.Web.Services.BoardTasks; +using AgentOs.Web.Services.Orchestration; +using Shouldly; +using Xunit; + +namespace AgentOs.Tests.Web; + +public sealed class AutopilotConfigurationTests +{ + // ── Settings ── + + [Fact] + public void Describe_EveryBoundIsAWholeNumberInARange_DefaultingToTodaysValues() + { + var all = new AutopilotSettings().Describe().ToDictionary(d => d.Key); + + all.Values.ShouldAllBe(d => d.Category == AutopilotSettings.Category && d.Scope == SettingScope.Tenant && !string.IsNullOrWhiteSpace(d.Help)); + (string Key, string Default, decimal Min, decimal Max)[] expected = + [ + (AutopilotSettings.MaxParallelKey, "2", 1, 6), + (AutopilotSettings.MaxParallelTestsKey, "1", 1, 3), + (AutopilotSettings.MaxFixRoundsKey, "3", 0, 10), + (AutopilotSettings.MaxAgentRetriesKey, "2", 0, 5), + (AutopilotSettings.ChecksTimeoutMinutesKey, "60", 5, 240), + (AutopilotSettings.ChecksPollSecondsKey, "60", 15, 600), + ]; + foreach (var (key, def, min, max) in expected) + { + var d = all[key]; + d.Kind.ShouldBe(SettingKind.WholeNumber); + d.DefaultValue.ShouldBe(def); + d.Min.ShouldBe(min); + d.Max.ShouldBe(max); + } + all[AutopilotSettings.DecidedByKey].Kind.ShouldBe(SettingKind.Choice); + all[AutopilotSettings.TicketEngineKey].Kind.ShouldBe(SettingKind.Choice); + } + + private sealed class FakeSettings(Func values) : ISettings + { + public long Version => 0; + + public ValueTask GetAsync(string key, string? tenantId, CancellationToken cancellationToken = default) => + ValueTask.FromResult(new SettingValue(tenantId == "t1" ? values(key) : null, SettingSource.Tenant)); + + public ValueTask> ListAsync(string? tenantId, string? category = null, CancellationToken cancellationToken = default) => + throw new NotSupportedException(); + + public ValueTask SetAsync(string key, string? value, SettingScope scope, string? tenantId, string actor, + CancellationToken cancellationToken = default) => throw new NotSupportedException(); + } + + [Fact] + public async Task ReadLimits_Unset_IsTheDefault() + { + (await AutopilotSettings.ReadLimitsAsync(new FakeSettings(_ => null), "t1")).ShouldBe(AutopilotLimits.Default); + AutopilotLimits.Default.ShouldBe(new AutopilotLimits(2, 1, 3, 2, TimeSpan.FromMinutes(60), TimeSpan.FromSeconds(60))); + } + + [Fact] + public async Task ReadLimits_ReadsTheTenantsValues_ClampedToTheirRanges() + { + var settings = new FakeSettings(key => key switch + { + AutopilotSettings.MaxParallelKey => "4", + AutopilotSettings.MaxParallelTestsKey => "99", // a seeded value can't escape the range + AutopilotSettings.MaxFixRoundsKey => "0", + AutopilotSettings.ChecksTimeoutMinutesKey => "90", + _ => null, + }); + + var limits = await AutopilotSettings.ReadLimitsAsync(settings, "t1"); + + limits.MaxParallel.ShouldBe(4); + limits.MaxParallelTests.ShouldBe(3); + limits.MaxFixRounds.ShouldBe(0); + limits.MaxAgentRetries.ShouldBe(2); + limits.ChecksTimeout.ShouldBe(TimeSpan.FromMinutes(90)); + limits.ChecksPoll.ShouldBe(TimeSpan.FromSeconds(60)); + limits.Uncapped.MaxParallel.ShouldBe(int.MaxValue); + limits.Uncapped.MaxFixRounds.ShouldBe(0); // uncapped slots, never uncapped rounds + } + + // The Orchestrator agent's budget: the configured rounds + its own retries, against the persisted count. + [Theory] + [InlineData(0, 1, 1)] + [InlineData(1, 1, 1)] + [InlineData(2, 1, 0)] + public void RetriesLeft_UsesTheConfiguredBudget(int fixRounds, int maxAgentRetries, int expected) => + ProjectOrchestrator.RetriesLeft(fixRounds, AutopilotLimits.Default with { MaxFixRounds = 1, MaxAgentRetries = maxAgentRetries }).ShouldBe(expected); + + // ── Plan-limit pause ── + + private static PlanLimitHold Hold(int hour) => + new(new PlanLimit("Claude", "You've hit your session limit", new DateTimeOffset(2026, 10, 24, hour, 50, 0, TimeSpan.FromHours(7)), true), + new DateTimeOffset(2026, 10, 24, hour - 7, 51, 0, TimeSpan.Zero)); + + [Fact] + public void Pause_LogsOnceWhenTheWaitStarts_AndOnceWhenItEnds() + { + var pause = new PlanLimitPause(); + var board = Guid.NewGuid(); + + pause.Observe(board, null).Paused.ShouldBeFalse(); + pause.Observe(board, Hold(23)).ShouldBe((true, "Your computer's Claude plan hit its limit — Autopilot waits until 23:50, nothing is lost.")); + var later = pause.Observe(board, Hold(23)); // every later pass: still paused, nothing logged + later.Paused.ShouldBeTrue(); + later.Line.ShouldBeNull(); + pause.Observe(Guid.NewGuid(), Hold(23)).Line.ShouldNotBeNull(); // each board logs its own line + pause.Observe(board, null).ShouldBe((false, "Your computer's Claude plan is open again — Autopilot carries on where it stopped.")); + var after = pause.Observe(board, null); + after.Paused.ShouldBeFalse(); + after.Line.ShouldBeNull(); + } + + [Fact] + public void Pause_ALaterResetIsAnnounced() + { + var pause = new PlanLimitPause(); + var board = Guid.NewGuid(); + pause.Observe(board, Hold(20)); + + pause.Observe(board, Hold(23)).Line.ShouldNotBeNull(); + } + + [Fact] + public void CutShortByPlanLimit_ReadsTheStoredErrors() + { + var session = new RemoteSessionEntity { Id = Guid.NewGuid(), Status = "Failed" }; + var limited = new SessionRepoEntity { SessionId = session.Id, Status = "Failed", Error = "LLM call failed: You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)" }; + var broken = new SessionRepoEntity { SessionId = session.Id, Status = "Failed", Error = "push rejected" }; + + ProjectOrchestrator.CutShortByPlanLimit(session, new[] { limited }.ToLookup(r => r.SessionId)).ShouldBeTrue(); + ProjectOrchestrator.CutShortByPlanLimit(session, new[] { broken }.ToLookup(r => r.SessionId)).ShouldBeFalse(); + ProjectOrchestrator.CutShortByPlanLimit(new RemoteSessionEntity { Id = session.Id, Error = "You've hit your usage limit." }, + Array.Empty().ToLookup(r => r.SessionId)).ShouldBeTrue(); + } + + // ── Acceptance-run gate ── + + [Fact] + public async Task RunGate_HonoursTheCapPerComputer() + { + var gate = new RunGate(); + using var a = await gate.EnterAsync("t1/alice", 1); + using var other = await gate.EnterAsync("t1/bob", 1); // another member's computer never waits on alice's + + var second = gate.EnterAsync("t1/alice", 1); + await Task.Delay(50); + second.IsCompleted.ShouldBeFalse(); + gate.Held("t1/alice").ShouldBe(1); + + a.Dispose(); + using (await second.WaitAsync(TimeSpan.FromSeconds(5))) { gate.Held("t1/alice").ShouldBe(1); } + gate.Held("t1/alice").ShouldBe(0); + } + + [Fact] + public async Task RunGate_ARaisedCap_LetsTwoIn() + { + var gate = new RunGate(); + using var a = await gate.EnterAsync("k", 2); + using var b = await gate.EnterAsync("k", 2).WaitAsync(TimeSpan.FromSeconds(5)); + + gate.Held("k").ShouldBe(2); + } +} diff --git a/tests/AgentOs.Tests/Web/AutopilotOrchestratorTests.cs b/tests/AgentOs.Tests/Web/AutopilotOrchestratorTests.cs index 21a847c3..39641845 100644 --- a/tests/AgentOs.Tests/Web/AutopilotOrchestratorTests.cs +++ b/tests/AgentOs.Tests/Web/AutopilotOrchestratorTests.cs @@ -100,7 +100,7 @@ public void Uncapped_OffersEveryReadyTicket_TheCappedRulesOnlyTwo() PlanTicket[] tickets = [new(41, true, true, 1, [], false), new(42, true, true, 1, [], false), new(43, true, true, 2, [], false)]; AutopilotPlanner.Decide(AutopilotMode.Build, tickets, []).Count.ShouldBe(2); - AutopilotPlanner.Decide(AutopilotMode.Build, tickets, [], true, int.MaxValue, int.MaxValue).Count.ShouldBe(3); + AutopilotPlanner.Decide(AutopilotMode.Build, tickets, [], true, AutopilotLimits.Default.Uncapped).Count.ShouldBe(3); } [Fact] @@ -120,5 +120,5 @@ public void Signature_IsOrderIndependent_AndChangesWithTheState() [InlineData(5, 0)] [InlineData(9, 0)] public void RetriesLeft_ComesFromPersistedFixRounds(int fixRounds, int expected) => - ProjectOrchestrator.RetriesLeft(fixRounds).ShouldBe(expected); + ProjectOrchestrator.RetriesLeft(fixRounds, AutopilotLimits.Default).ShouldBe(expected); } diff --git a/tests/AgentOs.Tests/Web/AutopilotPlannerTests.cs b/tests/AgentOs.Tests/Web/AutopilotPlannerTests.cs index 13e8792e..314fed89 100644 --- a/tests/AgentOs.Tests/Web/AutopilotPlannerTests.cs +++ b/tests/AgentOs.Tests/Web/AutopilotPlannerTests.cs @@ -2,6 +2,7 @@ // only when the board allows it, and a person asked when the agent can't get there. using AgentOs.Domain.Workspaces; +using AgentOs.Modules.Pipeline.Agents; using AgentOs.Web.Services.Orchestration; using Shouldly; using Xunit; @@ -13,6 +14,75 @@ public sealed class AutopilotPlannerTests private static PlanTicket T(int n, bool open = true, bool ai = true, int p = 1, int[]? deps = null, bool live = false) => new(n, open, ai, p, deps ?? [], live); + // ── Settings › Autopilot bounds ── + + [Fact] + public void Limits_FromSettings_CapSlots() + { + var limits = AutopilotLimits.Default with { MaxParallel = 3 }; + + AutopilotPlanner.Decide(AutopilotMode.Build, [T(1), T(2), T(3), T(4)], [], limits: limits).Count.ShouldBe(3); + AutopilotPlanner.Decide(AutopilotMode.Build, [T(1), T(2), T(3), T(4)], []).Count.ShouldBe(2); // the default + } + + [Fact] + public void Limits_FromSettings_CapFixRounds_AndTheLogSaysTheRealNumber() + { + var limits = AutopilotLimits.Default with { MaxFixRounds = 1 }; + var red = Task(9, "Done", Pr(PrChecksState.Failed)); + + AutopilotPlanner.Decide(AutopilotMode.Build, [], [red], limits: limits).ShouldHaveSingleItem() + .ShouldBeOfType().Why.ShouldContain("try 1 of 1"); + AutopilotPlanner.Decide(AutopilotMode.Build, [], [red with { FixRounds = 1 }], limits: limits).ShouldHaveSingleItem() + .ShouldBeOfType().Why.ShouldContain("after 1 tries"); + } + + [Fact] + public void Limits_FromSettings_CapAcceptanceRuns() + { + PlanTask Green(int issue) => Task(issue, "Done", new PlanChange(Guid.NewGuid(), issue + 20, PrChecksState.Passed, null)); + var limits = AutopilotLimits.Default with { MaxParallel = 4, MaxParallelTests = 2 }; + + AutopilotPlanner.Decide(AutopilotMode.Build, [], [Green(1), Green(2), Green(3)], limits: limits) + .OfType().Count().ShouldBe(2); + } + + [Fact] + public void WithinCaps_UsesTheConfiguredCaps() + { + AutopilotAction[] picked = [new AutopilotAction.StartTicket(1, "a"), new AutopilotAction.StartTicket(2, "b"), new AutopilotAction.StartTicket(3, "c")]; + + AutopilotPlanner.WithinCaps(picked, 0, 0, AutopilotLimits.Default with { MaxParallel = 3 }).Count.ShouldBe(3); + AutopilotPlanner.WithinCaps(picked, 0, 0, AutopilotLimits.Default with { MaxParallel = 1 }).ShouldHaveSingleItem(); + } + + // ── Plan limits ── + + // Work the member's plan limit cut short picks up first once the plan resets — as a plain restart, no fix round. + [Fact] + public void CutShortByPlanLimit_ResumesFirst_WithinTheSlots() + { + var cut = Task(7, "Failed") with { CutShortByPlanLimit = true, FixRounds = 2 }; + + var actions = AutopilotPlanner.Decide(AutopilotMode.Build, [T(1), T(2)], [cut]); + + actions.Count.ShouldBe(2); + actions[0].ShouldBeOfType().IssueNumber.ShouldBe(7); + actions[1].ShouldBeOfType().Number.ShouldBe(1); + actions.OfType().ShouldBeEmpty(); + AutopilotPlanner.TakesSlot(actions[0]).ShouldBeTrue(); + } + + [Fact] + public void CutShortByPlanLimit_WaitsForASlot_AndAPlainFailureIsLeftAlone() + { + var cut = Task(7, "Failed") with { CutShortByPlanLimit = true }; + + AutopilotPlanner.Decide(AutopilotMode.Build, [], [Task(8, "Running"), Task(9, "Running"), cut]).ShouldBeEmpty(); + AutopilotPlanner.Decide(AutopilotMode.Build, [], [Task(7, "Failed")]).ShouldBeEmpty(); // the Orchestrator agent's to retry + AutopilotPlanner.Decide(AutopilotMode.Off, [], [cut]).ShouldBeEmpty(); + } + private static PlanTask Task(int issue, string status, params PlanChange[] changes) => new(Guid.NewGuid(), issue, status, changes, 0); @@ -60,7 +130,7 @@ public void FailedChecks_AreSentBackToTheAgent_WithTheReport_AndTheRound() [Fact] public void FailedChecks_AfterThreeFixes_AskAPerson() { - var task = Task(9, "Done", Pr(PrChecksState.Failed)) with { FixRounds = AutopilotPlanner.MaxFixRounds }; + var task = Task(9, "Done", Pr(PrChecksState.Failed)) with { FixRounds = AutopilotLimits.Default.MaxFixRounds }; AutopilotPlanner.Decide(AutopilotMode.BuildAndMerge, [], [task]).ShouldHaveSingleItem().ShouldBeOfType(); } @@ -92,7 +162,7 @@ public void Rejected_GoesBackToCodingWithTheFindings_ThenAPerson() var fix = AutopilotPlanner.Decide(AutopilotMode.BuildAndMerge, [], [rejected]).ShouldHaveSingleItem().ShouldBeOfType(); fix.Report.ShouldBe("- F-4: duplicate email accepted"); fix.Why.ShouldContain("failed acceptance testing"); - AutopilotPlanner.Decide(AutopilotMode.BuildAndMerge, [], [rejected with { FixRounds = AutopilotPlanner.MaxFixRounds }]) + AutopilotPlanner.Decide(AutopilotMode.BuildAndMerge, [], [rejected with { FixRounds = AutopilotLimits.Default.MaxFixRounds }]) .ShouldHaveSingleItem().ShouldBeOfType(); } diff --git a/tests/AgentOs.Tests/Web/SettingsSearchTests.cs b/tests/AgentOs.Tests/Web/SettingsSearchTests.cs index 17d36826..6c0aa338 100644 --- a/tests/AgentOs.Tests/Web/SettingsSearchTests.cs +++ b/tests/AgentOs.Tests/Web/SettingsSearchTests.cs @@ -15,7 +15,7 @@ namespace AgentOs.Tests.Web; public sealed class SettingsSearchTests { private static readonly SettingDescriptor[] Descriptors = - [.. new LlmSettings().Describe(), .. new AgentSettings().Describe()]; + [.. new LlmSettings().Describe(), .. new AgentSettings().Describe(), .. new AutopilotSettings().Describe()]; [Theory] [InlineData("gemini key", "keys")] @@ -26,6 +26,9 @@ public sealed class SettingsSearchTests [InlineData("who changed", "audit")] [InlineData("service account", "tokens")] [InlineData("qa loop", "agents")] + [InlineData("autopilot", "autopilot")] + [InlineData("fix rounds", "autopilot")] + [InlineData("decided by", "autopilot")] public void Find_Admin_TopHitIsThePageThatEditsIt(string query, string target) => SettingsSearch.Find(query, Descriptors, isAdmin: true, isPlatformAdmin: false)[0].Target.ShouldBe(target); @@ -76,6 +79,7 @@ public void TargetOf_EveryTenantDescriptorHasAPage() [InlineData("system/appearance", "appearance")] [InlineData("system", "general")] [InlineData("audit", "audit")] + [InlineData("autopilot", "autopilot")] public void Normalize_MapsLegacyTargetsToFlatPages(string target, string page) => SettingsLinks.Normalize(target).ShouldBe(page); } diff --git a/tests/AgentOs.Tests/Web/TaskFailureTests.cs b/tests/AgentOs.Tests/Web/TaskFailureTests.cs index f57fdd8d..593c60aa 100644 --- a/tests/AgentOs.Tests/Web/TaskFailureTests.cs +++ b/tests/AgentOs.Tests/Web/TaskFailureTests.cs @@ -21,6 +21,8 @@ public sealed class TaskFailureTests [InlineData("Stopped — you pressed Stop. Nothing was pushed for the unfinished parts; Try again to restart.", "stopped", TaskFailureAction.TryAgain)] [InlineData("src/A.cs(3,1): error CS1002: ; expected", "build", TaskFailureAction.TryAgain)] [InlineData("exit -1: ", "unknown", TaskFailureAction.TryAgain)] + // The member's computer answered with its Claude plan's limit (claude -p, 2026-10). + [InlineData("LLM call failed: You've hit your session limit · resets 11:50pm (Asia/Ho_Chi_Minh)", "plan-limit", TaskFailureAction.TryAgain)] public void Explain_RealErrors_ClassifiedWithOneAction(string error, string kind, TaskFailureAction action) { var info = TaskFailure.Explain(error); diff --git a/tests/AgentOs.Tests/Web/TicketWorkflowTests.cs b/tests/AgentOs.Tests/Web/TicketWorkflowTests.cs index 6a1f906f..8dd35b83 100644 --- a/tests/AgentOs.Tests/Web/TicketWorkflowTests.cs +++ b/tests/AgentOs.Tests/Web/TicketWorkflowTests.cs @@ -1,13 +1,15 @@ // The per-ticket Microsoft Agent Framework workflow, run for real (InProcessExecution) against scripted steps: every // route of the graph — green path, red checks, conflicts, a branch behind its base, requested changes, failed -// acceptance, the 3-round cap, an agent that couldn't finish, checks that never finish, and a board that leaves the -// merge to a person. +// acceptance, the 3-round cap, an agent that couldn't finish, checks that never finish, a board that leaves the +// merge to a person — plus the round counter seeded from the task's persisted fix rounds (one count across engines and +// restarts) and the member's plan limit pausing the flow before it spends a round. using System.Collections.Generic; using System.Linq; using System.Threading; using System.Threading.Tasks; using AgentOs.Domain.Workspaces; +using AgentOs.Modules.Pipeline.Agents; using AgentOs.Web.Services.BoardTasks; using AgentOs.Web.Services.Orchestration; using Shouldly; @@ -29,6 +31,11 @@ private sealed class ScriptedSteps : ITicketSteps public List CodeRequests { get; } = []; public int Updates { get; private set; } public int Merges { get; private set; } + /// Set when the member's plan hits its limit (after coding runs). + public string? Limit { get; set; } + public int LimitAfterCodeRuns { get; set; } = int.MaxValue; + + public string? PlanLimited(TicketFlow flow) => CodeRequests.Count >= LimitAfterCodeRuns ? Limit : null; private static Task Started() => Task.FromResult(new TaskRun(TaskStart.Started, Task.CompletedTask)); @@ -186,6 +193,64 @@ public async Task CodingCouldNotFinish_StopsWithTheReason() public async Task ChecksNeverFinish_StopAfterTheTimeout() => (await Run(new ScriptedSteps(), checksTimeout: TimeSpan.FromMilliseconds(30))).Stage.ShouldBe(TicketStage.Stopped); + // One counter: a flow started after step-by-step rounds (or a restart) has only what is left of the cap. + [Fact] + public async Task SeededFromPersistedFixRounds_GetsOnlyTheRoundsLeft() + { + var steps = new ScriptedSteps(); + steps.Checks.Enqueue((PrChecksState.Failed, PrMergeState.Clean)); + steps.Checks.Enqueue((PrChecksState.Failed, PrMergeState.Clean)); + var flow = TicketWorkflowRunner.Seed("t", Guid.NewGuid(), Guid.NewGuid(), 15, mayMerge: true, persistedFixRounds: 2, AutopilotLimits.Default); + + var end = await TicketWorkflow.RunAsync(steps, flow, TimeSpan.FromMinutes(1), TimeSpan.FromMilliseconds(1), CancellationToken.None); + + end.Stage.ShouldBe(TicketStage.NeedsPerson); + end.Round.ShouldBe(3); + steps.CodeRequests.Count.ShouldBe(2); // the ticket itself, then the single round that was left + end.Reason!.ShouldContain("after 3 automatic fixes"); + } + + [Fact] + public async Task ConfiguredMaxRounds_IsTheCap() + { + var steps = new ScriptedSteps(); + for (var i = 0; i < 3; i++) { steps.Checks.Enqueue((PrChecksState.Failed, PrMergeState.Clean)); } + var flow = TicketWorkflowRunner.Seed("t", Guid.NewGuid(), Guid.NewGuid(), 15, true, 0, AutopilotLimits.Default with { MaxFixRounds = 1 }); + + var end = await TicketWorkflow.RunAsync(steps, flow, TimeSpan.FromMinutes(1), TimeSpan.FromMilliseconds(1), CancellationToken.None); + + end.Stage.ShouldBe(TicketStage.NeedsPerson); + end.Round.ShouldBe(1); + end.Trail.ShouldContain("checks failed → back to Coding (try 1 of 1)"); + } + + // The member's plan hit its limit while coding: the flow stops there — no review, no test, no fix round spent. + [Fact] + public async Task PlanLimit_PausesTheFlow_WithoutSpendingARound() + { + var steps = new ScriptedSteps { Limit = "Your computer's Claude plan hit its limit — Autopilot waits until 23:50, nothing is lost.", LimitAfterCodeRuns = 1 }; + steps.Checks.Enqueue(Green); + + var flow = await Run(steps); + + flow.Stage.ShouldBe(TicketStage.PlanLimited); + flow.Round.ShouldBe(0); + flow.Reason!.ShouldContain("waits until 23:50"); + steps.CodeRequests.Count.ShouldBe(1); + steps.Merges.ShouldBe(0); + } + + [Fact] + public async Task PlanLimitBeforeTheFirstStep_StartsNothing() + { + var steps = new ScriptedSteps { Limit = "limit", LimitAfterCodeRuns = 0 }; + + var flow = await Run(steps); + + flow.Stage.ShouldBe(TicketStage.PlanLimited); + steps.CodeRequests.ShouldBeEmpty(); + } + [Fact] public async Task BoardWithoutAutoMerge_EndsReadyForAPerson() {