Skip to content

Stop telemetry test servers when a test body fails - #1165

Open
MaxGhenis wants to merge 1 commit into
mainfrom
fix/telemetry-test-server-teardown
Open

MaxGhenis wants to merge 1 commit into
mainfrom
fix/telemetry-test-server-teardown

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

When a telemetry emitter test fails, pytest prints its summary and then hangs until someone interrupts it. This fixes that. It changes one test file; no production code changes.

Cause. Two tests in packages/microcosm-build/tests/engine_free/shared/test_telemetry_emitter.py start a server on a non-daemon thread and stop it only at the end of the happy path:

  • test_subprocess_exchanges_ambient_token_and_delivers_events runs ThreadingHTTPServer.serve_forever and calls shutdown() only after assert emitter.available and _process.wait(timeout=10). This is the case the NZ hub's review of microcosm#1162 hit (finding 3).
  • test_local_socket_acknowledges_after_durable_queue runs EmitterService.run, which serves until it receives a close message. Its _FakeSampler.parent_alive() always returns True, so nothing else stops it.

If an assertion fails first, the thread keeps running. Interpreter exit waits for every non-daemon thread (threading._shutdown), so the pytest process never exits.

Fix. All three tests that start a server thread now use one context manager, _serving_thread(target, stop). The third, test_token_bearing_http_post_does_not_follow_redirects, already had try/finally but used a non-daemon thread. The helper:

  • runs target on a daemon thread;
  • calls stop() and joins the thread (up to 30 s) in finally.

The HTTP servers are also entered as context managers, so server_close() always runs.

The ambient-token test also kills its emitter service subprocess in a finally if the service is still running after a failure. This is defensive: in the forced-failure runs below the service had already exited on its own both before and after the change.

Invariant. For every test in this file that starts a server thread, on every exit path of the test body (pass or fail):

  1. the thread is a daemon; and
  2. stop() is called and the thread is joined.

Either one alone keeps pytest from hanging. Two new tests force an AssertionError inside _serving_thread, one for the HTTP server and one for EmitterService. Each then asserts that the thread is a daemon and no longer alive; the emitter test also asserts the socket was removed. Three helper mutants (non-daemon thread, no finally, no stop() call) each fail both new tests (6 of 6).

Before and after. Each run used a watchdog that sends SIGINT if pytest is still alive 30 s after printing its summary (60 s for the full-file runs). The forced-failure copies differ from their source by one injected line.

Run main this branch
Whole file (failed on its own under load, see below) 1 failed, 32 passed, hung; interrupted 60.9 s after the summary 1 failed, 34 passed, exited 1.0 s after
Ambient-token test, forced startup_timeout_seconds=0 (emitter.available false) hung; interrupted 30.7 s after exited 0.7 s after
Ambient-token test, service started, forced wait(timeout=0.001) (TimeoutExpired) hung; interrupted 30.4 s after exited 0.7 s after
Socket test, forced wrong rss_bytes expectation before the close message hung; interrupted 30.4 s after exited 0.5 s after
Ambient-token test with startup_timeout_seconds=60 (success path) n/a 1 passed

Not fixed here. On this machine (load average 35–325) the ambient-token test still fails: "the local telemetry emitter service did not become ready". The service subprocess imports the microcosm.build package, which pulls in torch and pandas. That took 7–8 s against the 3 s DEFAULT_STARTUP_TIMEOUT_SECONDS. CI passes on main, presumably because its runners import faster. That is a separate, partly production issue: busy build hosts can silently lose hosted telemetry. It is queued as a follow-up. After this PR, that failure is reported and pytest exits.

Tests. Only this file: 1 failed, 34 passed (the load-caused failure above), and pytest exits 3.5 s after its summary. ruff check and ruff format --check pass. No changelog fragment, matching microcosm#1162 (test-only).

🤖 Generated with Claude Code

Two tests in test_telemetry_emitter.py start a background server on a
non-daemon thread and only stop it at the end of the happy path:

- test_subprocess_exchanges_ambient_token_and_delivers_events runs
  ThreadingHTTPServer.serve_forever and calls shutdown() only after
  `assert emitter.available` and `_process.wait(timeout=10)`;
- test_local_socket_acknowledges_after_durable_queue runs
  EmitterService.run, which serves until it receives a close message
  (its fake sampler always reports the parent alive).

If an assertion fails first, the thread keeps running. Interpreter exit
waits for every non-daemon thread (threading._shutdown), so pytest
prints its summary and then hangs until it is interrupted.

All three tests that start a server thread now use one context manager,
_serving_thread(target, stop). It runs the target on a daemon thread and
calls stop() and joins the thread in a finally block. The ambient-token
test also kills its emitter service subprocess if the service is still
running when the test body fails. Two new tests force an assertion
failure inside the helper and check that the thread was stopped and is a
daemon.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant