Skip to content

tests: the long-line check counts an empty end, and the fork fault returns after its leftover writes its pid (#113) - #173

Open
sjawhar wants to merge 3 commits into
FreeMixer:mainfrom
sjawhar:tests-flaky-113
Open

sjawhar wants to merge 3 commits into
FreeMixer:mainfrom
sjawhar:tests-flaky-113

Conversation

@sjawhar

@sjawhar sjawhar commented Oct 26, 2026

Copy link
Copy Markdown

Closes #113.

Two outcomes of the test job depended on timing in its fedora:44 container on a loaded host.

  • a_worker_line_too_long_to_relay_is_dropped_whole, run alone 60 times, missed the sabotage arm "the end of a worker line too long to relay is sent as a line" 9 times. The relay's buffer stops doubling at 2 MiB, the length of the test's longest line. When one read fills it with that line, the read stops just before the line's NUL. With the arm, the line's end arrives as an empty line, which the check did not count. The check now counts it. It catches the arm in 100 of 100 runs and passes 100 of 100 without it.
  • The two "left in its own session" probe tests failed in 10 of 60 runs (30 of each). The fork fault returned before its leftover process had written its pid. The runner killed the leftover before that write, so the pidfile was missing or empty. No leftover outlived a run, so the runner is not at fault. Writing the pid by rename, as test-probe flake: the fork fixture's pidfile can be read empty #113 proposes, would leave the file missing, which the test cannot tell from a fault that never forked. The fault now waits on a pipe that the leftover closes once its pid is written. After the change, 120 of 120 runs pass (60 of each). With a temporary 200 ms sleep after the leftover's setsid(), both tests fail every run before the change and pass every run after.

The steps of ci.yml's private-refs and test jobs and of fuzz.yml pass 3 of 3 runs there.

… as a piece

The sabotage arm "the end of a worker line too long to relay is sent as a line" survived 9 of 60 runs of
a_worker_line_too_long_to_relay_is_dropped_whole. The runs were in the fedora:44 CI image on a loaded host. With the
test pinned to one CPU, the arm survived 12 of 30. The relay's buffer stops doubling at 2 MiB, the length of the
test's longest line. A read that fills it with that line stops just before the line's NUL. With `s->skip = 1` gone,
the rest of that line reaches the controller as an empty line. The check counted only lines that start with "x". In
every run the arm survived, the five shorter lines had their NUL in the read that passed LINE_LIMIT, and were
dropped whole. The 2 MiB line's end arrived empty.

The check now counts an empty line as a piece. A 2 MiB line never fits the buffer with its NUL, so its start is
always dropped. With the arm, its end always reaches the controller, empty or not. The check caught the arm in 100
of 100 runs, and the test passed 100 of 100 without it.
…ritten its pid (FreeMixer#113)

The two "left in its own session" tests of test-probe failed 10 of 60 runs (30 of each). The runs were in the
fedora:44 CI image on a loaded host. They failed with "the stress plugin did not leave its process" or with the
ValueError of FreeMixer#113. The fault forked and returned at once. In 100 runs by the tests' own run() and run_lv2(), each
run took 10 to 90 ms. In 36, the pidfile was missing (18) or empty (18). The runner can kill the process left behind
before that process writes its pid. In all 100 runs, the probe's document says that the process left behind was
killed. No process with that pidfile in its environment was alive then or a second later. The runner does its job.
The fault returned too soon.

Writing the pid by rename, as FreeMixer#113 proposes, would leave the file missing instead. The test cannot tell a missing
file from a fault that never forked. In both fixtures, the fault now waits on a pipe. The process left behind closes
the pipe once it is in its own session and its pid is written. With a 200 ms sleep added after its setsid(), both
tests fail 10 of 10 runs each before this change. After it, they pass 10 of 10 each. Without the sleep, they passed
120 of 120 runs (60 of each).
@sjawhar

sjawhar commented Oct 27, 2026

Copy link
Copy Markdown
Author

The CI failure was make check-rendered: omx-contract 3.4.0 renders the _RESP lines that #196 added on main. I merged main. check-rendered, the trixie DEB build and the ci.yml steps now pass with 3.4.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test-probe flake: the fork fixture's pidfile can be read empty

1 participant