Skip to content

Performance watchdog: detect and attribute sustained idle main-thread work - #110

Open
daiverd wants to merge 2 commits into
masterfrom
agent/issue-103-perf-watchdog
Open

Performance watchdog: detect and attribute sustained idle main-thread work#110
daiverd wants to merge 2 commits into
masterfrom
agent/issue-103-perf-watchdog

Conversation

@daiverd

@daiverd daiverd commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Fixes #103

A tab once burned ~18% of a core for three days before anyone noticed, and the condition stopped mid-investigation with no record of what had been running. This adds an always-on monitor that catches the onset, samples it while it is live, and names the culprit in one copy-pasteable line.

Design

src/diagnostics/perfWatchdog.ts runs three cheap signals and no per-event instrumentation in the steady state:

  • longtask PerformanceObserver — native, free when nothing is slow, silently absent on Firefox/Safari (the lag sampler covers those).
  • Event-loop lag sampler — one 500 ms setInterval measuring its own scheduling delay.
  • requestAnimationFrame wrapper — counts callers. It never schedules a frame of its own; the callback is passed through untouched and the native id is returned, so cancelAnimationFrame keeps working. Installing a rAF loop to measure frame rate pins the page at 60 fps and manufactures the very load it claims to observe, which is the false positive that derailed the original investigation.

Samples roll through a 10 s window (21 slots at 500 ms; the first is the baseline). The window is discarded outright on any tick that is not both idle and visible, so a trip means the whole window was quiet — background tabs throttle timers to ~1 Hz and would otherwise read as lag that is not there.

Attribution on trip: AudioContext.state via window.mudClient, connection state, open editor windows, LiveKit voice chat, inbound message and output rates sampled from useChannelHistoryStore / useOutputStore entry-id deltas (O(1), no per-event hooks), rAF callers/sec, and long-task totals. Each becomes a scored contributor; the warning leads with the top three by name.

Records are plain JSON and go to a pluggable sink (type WatchdogSink = (record: WatchdogRecord) => void), defaulting to a bounded 20-entry array plus one console.warn. The diagnostics buffer from #100 can be passed as the sink once both land — nothing here imports it.

Wiring is one line in App.tsx: useEffect(() => ensurePerfWatchdog(), []). Refcounted, so it starts once, is tied to the app rather than the connection (survives reconnects), and releases its interval, observer, three passive listeners, and the rAF wrapper on stop. EditorManager gains an openEditorCount getter for the attribution snapshot.

Thresholds and why

Constant Value Rationale
Sample interval 500 ms Issue says this is plenty; keeps the watchdog off the hot path.
Window 10 s Long enough that one route change or GC pause cannot fill it.
Idle threshold 10 s of no keydown/pointer Sustained work during active use is expected; during idle it is a bug.
Long-task busy ratio 20% of the window The observed pathology was ~18% of a core. longtask only counts tasks over 50 ms, so 20% of wall clock inside them while nobody is touching the client is unambiguous.
Long-task spread present in ≥ 4 of 20 samples A single 2 s stall is 20% of the window but is jank, not a runaway. Requiring recurrence across ≥ 2 s makes it sustained.
Lag trip average 25 ms A clean idle session measured 1 ms average, 7 ms max. 25 ms as a 10 s average is far outside that noise, and sits below the 50 ms long-task floor so it still catches death by a thousand medium tasks.
Lag spread ≥ 12 of 20 samples over 10 ms Most of the window has to be laggy, not one spike.
Episode cooldown 60 s clear One warning per episode; a new episode needs a full minute of quiet first.

Tests

src/diagnostics/perfWatchdog.test.ts, 10 cases on fake timers with an injected clock, long-task source, and probes: one warning per episode; a second episode only after the cooldown; silence while interacting; silence while hidden; the lag path tripping with zero long tasks; a record carrying subsystems, rates, ranked contributors and surviving a JSON round trip; the default sink plus console.warn fallback; and the rAF wrapper counting callers, passing through, adding zero calls of its own, and unwrapping cleanly on stop.

npx vitest run — 1071 tests across 107 files, all green. npm run typecheck clean. Biome clean on the new files (the two warnings on App.tsx are pre-existing, untouched lines).

🤖 Generated with Claude Code

daiverd and others added 2 commits August 11, 2026 19:49
A tab once burned ~18% of a core for three days before anyone noticed,
and the condition stopped mid-investigation with no record of what had
been running. This adds an always-on monitor that catches the onset,
samples it while it is live, and names the culprit.

Three cheap signals, no per-event instrumentation in the steady state:
a longtask PerformanceObserver, a 500ms interval that measures its own
scheduling delay, and a requestAnimationFrame wrapper that counts
callers. The wrapper never schedules a frame of its own — installing a
rAF loop to measure frame rate pins the page at 60fps and manufactures
the load it claims to observe.

Only trips while idle (no keydown or pointer activity for 10s) and
visible, since background tabs throttle timers to ~1Hz and sustained
work during active use is expected. Exactly one console warning per
episode; the episode ends after the condition stays clear for 60s.

On trip it records live subsystems, inbound message and output rates
sampled from store entry-id deltas, rAF callers per second, and
long-task totals, then ranks them so the warning names the top
contributors. Records are JSON and go to a pluggable sink, defaulting
to a bounded in-memory array, so the diagnostics buffer can be dropped
in later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matches the identical hunk in the diagnostics branch (#111) so the two
PRs merge cleanly in either order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
daiverd added a commit that referenced this pull request Aug 12, 2026
Both #110 and #111 add this getter at the same spot; identical hunks
let git merge whichever lands second without conflict. Also adopts the
stricter liveness check (window present and not closed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance watchdog: detect and attribute sustained main-thread work while idle

1 participant