Subagent concurrency limiter is load-blind: parallel fleets saturate the host and freeze the CLI's own UI

[scorp508]

Describe the bug

The subagent concurrency limiter is a static counter with no awareness of machine load. When a fleet of subagents is launched, the CLI admits all of them regardless of how saturated the host already is, and it never reconsiders. On a 12-thread laptop this oversubscribes the CPU badly.

Two things make this worse than an ordinary "a parallel job is slow" complaint:

The CLI's own interactive input dies. /ask and keystrokes stopped responding for more than 17 minutes. Input is only processed between turns, so a long-running saturated turn is indistinguishable from a hung session. My first assumption was that the session had crashed.

The limiter counts the wrong thing. It counts subagents, but nearly all of the CPU came from the python and pwsh grandchildren that those agents spawned (KiCad CLI exports and evidence scripts), turning over every 1 to 2 seconds. Even a correct cap on agent count would not have bounded this.

Because the ceiling is fixed when the run starts, the run cannot self-correct. The only choices are to wait it out, or to kill the session and lose all in-flight subagent work.

Inspecting the installed app.js shows the CLI contains zero occurrences of every symbol that would be needed to observe host load. This is structural rather than a tuning problem. Occurrence counts in app.js:

os.cpus 0 occurrences loadavg 0 occurrences cpuUsage 0 occurrences freemem 0 occurrences

The admission path is a pure semaphore:

getSubAgentLimiterInfo(){ return {
  runningCount: sessionSubagentLimiterRunningCount(...),
  maxConcurrent: sessionSubagentLimiterMaxConcurrent(...)
}}
tryAcquireSubAgent(...)

It answers only the question "are fewer than N subagents running?". It admits subagent number 11 identically whether the host is idle or already at a processor queue depth of 28.

Affected version

1.0.83-0 (win32-x64)

Steps to reproduce the behavior

1. On a machine with a modest core count (12 logical processors in this case), open a session in a large repository.

2. Launch a fleet of roughly 11 subagents whose work shells out to external tools. In this case each reviewer agent ran kicad-cli exports and Python evidence scripts, so every agent continuously spawned short-lived child processes.

3. While the fleet is running, observe the host:

Get-Counter '\Processor(_Total)\% Processor Time',
            '\Processor(_Total)\% Idle Time',
            '\System\Processor Queue Length' -SampleInterval 1 -MaxSamples 3

Get-CimInstance Win32_PerfFormattedData_PerfProc_Process |
  Where-Object PercentProcessorTime -gt 0 |
  Sort-Object PercentProcessorTime -Descending | Select-Object -First 15

4. Try to use /ask, or type into the running session.

Result: all 12 logical CPUs sit at 100 percent, percent idle time is 0, the processor queue length stays between 18 and 28, and the session's input is unresponsive for more than 17 minutes. There is no warning, no throttling, and no indication anywhere in the UI that the host is saturated.

A note on step 3: a simple Get-Process CPU delta measured over a few seconds will under-report the load badly, because the worker processes spawn and exit inside the sampling window. The WMI performance counters shown above are what actually reveal where the CPU is going.

Expected behavior

1. The default concurrency ceiling is derived from the parallelism actually available on the host, using os.availableParallelism(), rather than a host-independent constant.
2. When the host is already saturated, additional subagents are queued rather than admitted, so the fleet still completes but without thrashing.
3. The interactive input path stays responsive regardless of subagent load, so the user can still query, steer, or cancel the run while a turn is in progress.
4. Saturation is visible in the UI, for example a status line reading "host 100% CPU, 4 subagents queued", so that a busy session is distinguishable from a hung one without having to resort to Get-Counter.
5. Limits can be lowered on an already-running fleet, as an escape hatch short of killing the session.

Additional context

Measurements

The numbers below use the Windows convention in which 100 percent equals one fully busy logical processor. This machine has 12 logical processors, so the whole machine is 1200 percent. A process showing 552 percent is therefore using about five and a half cores.

Each figure is an average across 10 samples taken while the fleet was running:

python, the agent worker scripts average 552 percent, peak 764 percent, about 5.5 of 12 cores

pwsh, the agent shell commands average 245 percent, peak 1164 percent, about 2.4 of 12 cores

copilot, the session process itself average 198 percent, peak 356 percent, about 2.0 of 12 cores

kicad-cli, an external tool invoked by the agents average 11 percent, peak 44 percent, about 0.1 of 12 cores

What these figures show:

• The session, and everything it spawned, accounted for roughly 84 percent of all CPU on the machine. The remaining 16 percent was ordinary background software such as Windows Defender.
• Most of the load was not the CLI itself. The copilot process used about 2 cores. The python and pwsh worker processes launched by the subagents used about 8 cores between them. Those workers are exactly what the concurrency limiter does not count.
• The CPU was doing real work rather than waiting. User time was 86 percent and privileged (kernel) time was 14 percent, so this was genuine computation and not I/O wait or driver overhead.
• The machine was oversubscribed. The processor queue length stayed between 18 and 28 while only 12 logical processors were available, meaning roughly two threads were waiting for every core. Work was queuing rather than running.
• No CPU tuning was applied anywhere. Every process ran with the default affinity mask 0xFFF, meaning all 12 processors, and at Normal priority. Nothing was pinned to a core, and no worker was given a lower priority than the interactive session.

At its peak, the session process used 3.87 GB of memory across 233 threads.

Environment

Hardware: Microsoft Surface Laptop 5
Processor: 12th Gen Intel Core i7-1265U CPU architecture x86_64 (AMD64) Cores 10 physical, 12 logical
Memory: 31.8 GB
Operating system: Windows 11 Enterprise Insider Preview, build 26310 (10.0.26310)
Terminal emulator: Standalone PowerShell 7 console window, opened from Explorer. WT_SESSION was not set, so this was not Windows Terminal. Shell PowerShell 7.6.5 (Core) Logging flags Defaults only: --log-level info, with no --log-file override

Suggested fixes

Listed roughly in order of value relative to effort:

1. Derive the default ceiling from the host. Use os.availableParallelism() with a sensible floor, for example max(2, floor(n * 0.75)), instead of a host-independent constant. This is a small change that prevents the common case.
2. Make admission load aware. Sample CPU utilization or processor queue depth inside tryAcquireSubAgent, and defer admission while the host is saturated. Subagents would then be queued rather than rejected, so the fleet still completes, just without thrashing.
3. Account for descendant processes, not only agents. Track the process tree that a subagent spawns, or cap concurrent shell and tool child processes separately. This is what actually consumed the CPU in this run.
4. Keep the input path responsive under load. Decoupling the interactive UI from turn execution, so that /ask and cancellation work in the middle of a turn, would by itself turn "the session is hung" into "the session is busy, and here is why."
5. Surface saturation in the UI. A status line indicator such as "host 100% CPU, 4 subagents queued" makes the state legible, and removes the need to diagnose it with Get-Counter.
6. Allow limits to be lowered on a running fleet. Today the ceiling is fixed once the run starts. Letting /limits reduce concurrency mid-run would provide an escape hatch short of killing the session and losing in-flight work.

Secondary issues noticed in the same run

Session log growth. This single session produced a log file that reached 1.72 GB in about 5 hours, growing at a steady 5.8 MB per minute, at ~/.copilot/logs/process-*-11268.log. Sampling the file shows that 97.4 percent of the bytes are untagged payload dumps rather than structured log lines, meaning actual tool output and message content written verbatim into the log. Line lengths are heavily skewed: the median line is 43 bytes, the 99th percentile is 5.6 KB, and the longest single line is 108 KB. Structured diagnostic records, meaning DEBUG, WARNING and INFO lines, together account for only 2.6 percent. Some form of size cap, rotation, or payload truncation would help. I have not established whether the same content is written repeatedly, so this may simply be the expected cost of verbose payload logging over a long session rather than a defect. Please treat it as a size and retention concern, not as a confirmed bug.

Auto-update renamed the running binary. Starting a second session updated the CLI, which left the first session's image named copilot.exe.old--. The running process keeps working, because Windows holds the open image handle, but a long-lived session silently continues to execute a superseded build. That also makes version reporting misleading.

Impact

This affects anyone running fleet mode, or a multi-agent factory, on a laptop or workstation. The failure mode is unhelpful in a specific way: the run gets slower because of context switching, it looks hung rather than busy, and the user's natural response, which is to kill it, destroys all in-flight subagent work.