Skip to content

fleet check: catch the two config faults that have each cost a whole fleet - #272

Merged
devdanzin merged 1 commit into
mainfrom
fleet-check-fuzz-user-exec-and-discover-in-target
Sep 5, 2026
Merged

devdanzin merged 1 commit into
mainfrom
fleet-check-fuzz-user-exec-and-discover-in-target

Conversation

@devdanzin

Copy link
Copy Markdown
Owner

Both faults were found by triaging fleets that had already burned their time on them, and
both are one line of config to fix once you know which line.

1. The target must be executable by the fuzz user

fleet check already did [ -x "$TARGET_PYTHON" ] — but that asks whether you can execute
it. fusil drops privileges before exec'ing the target, so a build under someone's home
directory is routinely executable by its owner and unreachable for the fuzz user, whose
traversal fails on a path component:

ChildError: The user fusil is not allowed to execute the file /home/danzin/.local/bin/pypy3.11

That surfaces once per session, kills the run, and the supervisor restarts into the same wall.
Two fleets opened that way on two different paths — one lost 36 of 65 runs, the other 6.

Checked by walking the mode bits of every path component, not by sudo-ing to the user:
same answer, needs no privilege (passwordless sudo is absent on exactly the boxes where this
bites), and it names the component that actually blocks.

  TARGET_PYTHON is not reachable by user 'fusil': '/tmp/x/inner/python'
    blocked at: /tmp/x/inner
    -> chmod o+X /tmp/x/inner

Both the literal path and its realpath are walked. Checking only the resolved one is a
hole: to exec a symlinked target the kernel must traverse the link's own directories before it
resolves the link, and build trees commonly symlink the interpreter. My first version had that
hole and passed a deliberately-blocked symlinked target.

Skipped under --unsafe / --force-unsafe, where no drop happens.

2. Discovery must run in an interpreter that has the modules

Without --discover-in-target, fusil imports the target's modules in the runner, so a
runner that is not the target yields a module list and member set the target does not have. A
PyPy fleet ran 43 368 sessions that way against a CPython 3.16 runner: 20% of the modules
it picked do not exist on PyPy at all
, and the importable remainder still exposed members it
lacks.

Calibrated deliberately, and this part matters:

different implementation fails — cpython-vs-pypy stdlibs diverge by construction; discovery in the wrong one is never right
different version, same implementation advisory — costs coverage at the edges, does not systematically break

Without that split this would have blocked fleet_multidict.conf, which works and has produced
findings (CPython 3.14 runner, 3.16 target, --modules=multidict so stdlib discovery is not
even in play).

Verified against every fleet*.conf on this host

config before after
fleet_rustpython.conf passed fails — CPython runner, RustPython target, no flag. A real instance of fault 2.
fleet_multidict.conf passed passes, with an advisory
all others passed passes

Regression matrix for the new logic:

case result
runner == target, --unsafe PASS
PyPy target, no --discover-in-target FAIL
PyPy target, with --discover-in-target PASS
symlinked target behind a 0700 dir FAIL
… after chmod o+X PASS
… blocked again but with --unsafe PASS

1293 tests OK, ruff check and ruff format --check clean. (The change is in fleet/fleet,
which the Python suite does not cover; the matrix above is the verification.)

…fleet

Both were found by triaging fleets that had already burned their time on them, and
both are one line of config to fix once you know.

1. TARGET_PYTHON must be executable by the UNPRIVILEGED FUZZ USER, not by whoever
runs `fleet up`. fusil drops privileges before exec'ing it, so a target under
someone's home directory is routinely executable by its owner and unreachable for
the fuzz user, whose traversal fails on a path component. That surfaces once per
session as a ProcessError, which kills the run; the supervisor restarts it and it
fails again. Two fleets opened that way on two different target paths -- one lost 36
of 65 runs, the other 6.

Checked by walking the mode bits of every path component rather than by sudo'ing to
the user: same answer, needs no privilege (passwordless sudo is absent on exactly
the boxes where this bites), and it can name the component that actually blocks.
Both the literal path and its realpath are walked -- checking only the resolved one
is a hole, because to exec a symlinked target the kernel must traverse the link's
own directories first, and build trees commonly symlink the interpreter. Skipped
under --unsafe/--force-unsafe, where no drop happens.

2. Discovery must run in an interpreter that HAS the modules. Without
--discover-in-target fusil imports them in the RUNNER, so a runner that is not the
target produces a module list and member set the target does not have. A PyPy fleet
ran 43 368 sessions that way against a CPython 3.16 runner; 20% of the modules it
picked do not exist on PyPy at all, and the importable remainder still exposed
members it lacks.

Calibrated deliberately: a different IMPLEMENTATION fails, since cpython-vs-pypy
stdlibs diverge by construction and discovery in the wrong one is never right. A
different VERSION of the same implementation is advisory only -- it costs coverage
at the edges but does not systematically break, and a fleet that pins --modules to
one installed package is unaffected. Without that split this would have blocked
fleet_multidict.conf, which works and has produced findings.

Verified against every fleet*.conf on this host: one newly fails
(fleet_rustpython.conf, a real instance of fault 2 -- CPython runner, RustPython
target, no flag), one is advisory (fleet_multidict.conf, version-only), the rest are
unchanged.
@devdanzin
devdanzin merged commit d298c92 into main Sep 5, 2026
2 checks passed
@devdanzin
devdanzin deleted the fleet-check-fuzz-user-exec-and-discover-in-target branch September 5, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant