Repository navigation
fleet check: catch the two config faults that have each cost a whole fleet - #272
Merged
Merged
Conversation
…fleet Both were found by triaging fleets that had already burned their time on them, and both are one line of config to fix once you know. 1. TARGET_PYTHON must be executable by the UNPRIVILEGED FUZZ USER, not by whoever runs `fleet up`. fusil drops privileges before exec'ing it, so a target under someone's home directory is routinely executable by its owner and unreachable for the fuzz user, whose traversal fails on a path component. That surfaces once per session as a ProcessError, which kills the run; the supervisor restarts it and it fails again. Two fleets opened that way on two different target paths -- one lost 36 of 65 runs, the other 6. Checked by walking the mode bits of every path component rather than by sudo'ing to the user: same answer, needs no privilege (passwordless sudo is absent on exactly the boxes where this bites), and it can name the component that actually blocks. Both the literal path and its realpath are walked -- checking only the resolved one is a hole, because to exec a symlinked target the kernel must traverse the link's own directories first, and build trees commonly symlink the interpreter. Skipped under --unsafe/--force-unsafe, where no drop happens. 2. Discovery must run in an interpreter that HAS the modules. Without --discover-in-target fusil imports them in the RUNNER, so a runner that is not the target produces a module list and member set the target does not have. A PyPy fleet ran 43 368 sessions that way against a CPython 3.16 runner; 20% of the modules it picked do not exist on PyPy at all, and the importable remainder still exposed members it lacks. Calibrated deliberately: a different IMPLEMENTATION fails, since cpython-vs-pypy stdlibs diverge by construction and discovery in the wrong one is never right. A different VERSION of the same implementation is advisory only -- it costs coverage at the edges but does not systematically break, and a fleet that pins --modules to one installed package is unaffected. Without that split this would have blocked fleet_multidict.conf, which works and has produced findings. Verified against every fleet*.conf on this host: one newly fails (fleet_rustpython.conf, a real instance of fault 2 -- CPython runner, RustPython target, no flag), one is advisory (fleet_multidict.conf, version-only), the rest are unchanged.
devdanzin
deleted the
fleet-check-fuzz-user-exec-and-discover-in-target
branch
September 5, 2026 13:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both faults were found by triaging fleets that had already burned their time on them, and
both are one line of config to fix once you know which line.
1. The target must be executable by the fuzz user
fleet checkalready did[ -x "$TARGET_PYTHON" ]— but that asks whether you can executeit. fusil drops privileges before exec'ing the target, so a build under someone's home
directory is routinely executable by its owner and unreachable for the fuzz user, whose
traversal fails on a path component:
That surfaces once per session, kills the run, and the supervisor restarts into the same wall.
Two fleets opened that way on two different paths — one lost 36 of 65 runs, the other 6.
Checked by walking the mode bits of every path component, not by
sudo-ing to the user:same answer, needs no privilege (passwordless sudo is absent on exactly the boxes where this
bites), and it names the component that actually blocks.
Both the literal path and its realpath are walked. Checking only the resolved one is a
hole: to exec a symlinked target the kernel must traverse the link's own directories before it
resolves the link, and build trees commonly symlink the interpreter. My first version had that
hole and passed a deliberately-blocked symlinked target.
Skipped under
--unsafe/--force-unsafe, where no drop happens.2. Discovery must run in an interpreter that has the modules
Without
--discover-in-target, fusil imports the target's modules in the runner, so arunner that is not the target yields a module list and member set the target does not have. A
PyPy fleet ran 43 368 sessions that way against a CPython 3.16 runner: 20% of the modules
it picked do not exist on PyPy at all, and the importable remainder still exposed members it
lacks.
Calibrated deliberately, and this part matters:
Without that split this would have blocked
fleet_multidict.conf, which works and has producedfindings (CPython 3.14 runner, 3.16 target,
--modules=multidictso stdlib discovery is noteven in play).
Verified against every
fleet*.confon this hostfleet_rustpython.conffleet_multidict.confRegression matrix for the new logic:
--unsafe--discover-in-target--discover-in-target0700dirchmod o+X--unsafe1293 tests OK,
ruff checkandruff format --checkclean. (The change is infleet/fleet,which the Python suite does not cover; the matrix above is the verification.)