Skip to content

Make the timing tests independent of sleep accuracy - #53

Merged
wolph merged 2 commits into
developfrom
fix/timing-sensitive-tests
Oct 2, 2026
Merged

wolph merged 2 commits into
developfrom
fix/timing-sensitive-tests

Conversation

@wolph

@wolph wolph commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

The problem

The timeout tests count items against real sleeps and leave 10 to 40 ms of slack. A sleep only promises to take at least as long as requested, so the counts change with the machine.

Three failures, three measurements:

  • Blocking sleeps that overshoot. On a busy macOS machine time.sleep(0.06) took 0.11 s at the median, while asyncio.sleep(0.06) took 0.0604 s. timeout_generator(0.1, 0.06) then yields 2 items where the test and the doctest expect 3. Five cases of test_timeout_generator and the timeout_generator doctest failed in 5 out of 5 runs.
  • A coarse event loop clock. The detector tests put a 0.04 s sleep against a 0.05 s timeout. With a 15.6 ms loop clock resolution, the Windows default, both timers are due in the same loop iteration and the timeout wins. Setting loop._clock_resolution = 0.0156 reproduces i == 3 instead of 4 in test_aio_generator_timeout_detector_decorator_clean_exit on every run, which is the failure reported on Windows in Fix UniqueList membership after indexed replacement #51.
  • Async sleeps that run late. test_aio_timeout_generator expects exactly five items from five sleeps of 0.06 s against a 0.3 s timeout. It failed 3 of 25 runs on a busy machine, and fails every time when asyncio.sleep is made 20 ms late.

The change

No library code changes.

  • timeout_generator tests and the doctest run on a fake clock that only moves when it is slept on. The tests now check the requested sleeps too, so interval_multiplier and maximum_interval are verified instead of implied.
  • test_aio_timeout_generator lets asyncio.sleep advance the same fake clock.
  • The total timeout tests advance the fake clock.
  • The per-item timeout tests yield five items without waiting and then stall for 10 s against a 0.05 s timeout. The stall is cancelled when the timeout fires.
  • One test stays on the real clock. It only checks what holds for any sleep accuracy: the items are consecutive, there are not too many, and the timeout has passed.
  • The fixtures live in _python_utils_tests/clock.py and are loaded from a root conftest.py, because the doctests in python_utils need them. The sdist ships conftest.py.

This follows the fake clock that test_abatcher_timed already uses.

Verification

  • 20 out of 20 full runs pass on the machine where the old tests failed 5 out of 5.
  • The new tests pass with the 15.6 ms clock resolution and with late async sleeps. The old ones fail under both.
  • Two mutants of timeout_generator are caught: one without the interval multiplier, one that stops an item late.
  • The doctest fails on that machine when the fixture is switched off, so the fixture is what makes it pass.
  • The full suite went from about 8 s to about 1.3 s.

The timeout tests counted items against real sleeps and left 10 to 40 ms
of slack. A sleep only promises to take at least as long as requested, so
the counts changed on a busy machine and on a coarse clock:

- Blocking sleeps that overshoot by 40 ms or more made timeout_generator
  yield one item fewer, in five test cases and in its doctest.
- A 15.6 ms event loop clock resolution, the Windows default, let the
  0.05 s timeout fire together with a 0.04 s sleep, so the detector tests
  stopped at 3 instead of 4.

The sync tests and the doctest now run on a fake clock that only moves
when it is slept on, and they check the requested sleeps as well. The
total timeout tests advance the same clock. The per-item timeout tests
yield without waiting and then stall for 10 s against a 0.05 s timeout.
One test stays on the real clock and only checks what holds for any sleep
accuracy.

The fixtures are loaded from a conftest.py in the repository root so the
doctests can use them, and the sdist ships that file.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Comment thread conftest.py
root because the doctests in ``python_utils`` need them as well.
"""

pytest_plugins: tuple[str, ...] = ('_python_utils_tests.clock',)
test_aio_timeout_generator still counted items against real sleeps. The
case with five sleeps of 0.06 s against a 0.3 s timeout ends one item
short as soon as the sleeps run 15 ms late in total. It failed 3 of 25
runs on a busy machine, and fails every time when asyncio.sleep is made
20 ms late.

The test now lets asyncio.sleep advance the fake clock. The default
iterable test in test_lazy_imports uses the fake clock too, so its 0.05 s
timeout cannot end the loop before the second item.
@wolph
wolph merged commit e4408da into develop Oct 2, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants