Repository navigation
fleet: keep the log a restart destroys, and stop calling high CPU a kill - #274
Merged
devdanzin merged 1 commit intoSep 5, 2026
Merged
Conversation
Two things found while comparing a 12-instance PyPy fleet against a 5-instance
one. Neither changes what fusil fuzzes.
**`fusil.out` was truncated on every restart.** `fleet-run` redirected with `>`,
so each restart destroyed the log of the run that had just died -- which is the
only one anybody wants to read. systemd restarts an instance whenever the cgroup
OOM killer reaps it (`fusil@.service`: MemoryMax + OOMPolicy=continue +
Restart=always), so this is routine, not exceptional: one fleet restarted 10
times in 7.5 h and left a 155 KB log for a 2079-session instance, against 913 KB
for an instance that never restarted. The death reason survived only in the
remote systemd journal, which does not travel with a copied fleet directory.
Now appends; the existing "# fleet-run instance ..." line marks each run
boundary inside the file.
**`cpu_load_kills` never counted kills.** `CpuProbe.live()` only sets a score,
logs, sends `session_rename "cpu_load"` and stops monitoring -- it terminates
nothing. The Python fuzzer then sets `watch.cpu.max_score = 0` unless
`--record-high-cpu` (`fusil/python/__init__.py:938`), so the usual effect is a
name part and nothing else: the session runs to completion and is not discarded.
The counter, its schema comment ("sessions the CPU-load watcher killed") and the
report's `cpu-kills` column all said otherwise, and that misreading turned a
fleet comparison into a claim about discarded work that was simply wrong.
The sidecar key becomes `high_cpu_sessions` (schema v3) and the report column
`high-cpu`. `cpu_load_kills` is still emitted as an alias and still read on
merge, because archived sidecars carry only the old name -- verified by merging
22 real schema-2 sidecars from an existing fleet and getting the same 10002
under both keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WPBSmN87d2BqUnrDbojUT1
devdanzin
deleted the
fleet-log-truncated-on-restart-and-cpu-load-kills-misnomer
branch
September 5, 2026 22:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things found while comparing a 12-instance PyPy fleet against a 5-instance one. Neither changes what fusil fuzzes.
1.
fusil.outwas truncated on every restartfleet-runredirected with>, so each restart destroyed the log of the run that had just died — the only one anybody wants to read.This is routine, not exceptional: systemd restarts an instance whenever the cgroup OOM killer reaps it (
fusil@.service:MemoryMax+OOMPolicy=continue+Restart=always). One fleet restarted 10 times in 7.5 h and left:fusil.outThe death reason survived only in that machine's systemd journal, which doesn't travel with a copied fleet directory. Now appends; the existing
# fleet-run instance ...line already marks each run boundary inside the file.2.
cpu_load_killsnever counted killsCpuProbe.live()only sets a score, logs, sendssession_rename "cpu_load"and stops monitoring — it terminates nothing. The Python fuzzer then setswatch.cpu.max_score = 0unless--record-high-cpu(fusil/python/__init__.py:938), so the usual effect is a name part and nothing else: the session runs to completion and is not discarded.The counter, its schema comment ("sessions the CPU-load watcher killed") and the report's
cpu-killscolumn all said otherwise. That misreading turned a fleet comparison into a claim about discarded work that was simply wrong.high_cpu_sessions(schema v3), report column →high-cpucpu_load_killsis still emitted as an alias and still read on merge, since archived sidecars carry only the old nameBackward compat verified against real data: merging 22 archived schema-2 sidecars from an existing fleet gives 10002 under both keys, matching the raw sum.
Tests
Two new: both keys are emitted, and
merge()counts a pre-v3 sidecar that carries only the alias. 1300 tests pass;ruff check+ruff format --checkclean;bash -nclean.🤖 Generated with Claude Code
https://claude.ai/code/session_01WPBSmN87d2BqUnrDbojUT1