Keep one build's telemetry login from deciding other builds' runs

[MaxGhenis]

Problem

Every build on a host shares one telemetry spool (~/.cache/microcosm/telemetry/events.sqlite3), and every emitter service's delivery worker walked every pending run in it (CollectorDelivery.flush_once over spool.pending_runs()). For each run it read its own process's Hugging Face credential. With none, it called _make_local_only(registration, missing_huggingface_credential), which permanently sets upload_state = local_only on that run, another build's run included. The same applied to a token exchange the collector refused (401/403), a registration refused with 403 and an events batch refused with 403.

So one build started without a credential (a CI shell, a terminal with no HF_TOKEN, another login) permanently disabled hosted telemetry for every other build's queued events. An independent check on 2026-10-09 reproduced it at 7f23594: two producers share a spool, and one flush_once by build B's worker, with no credential, left ('build-a', 'local_only', 'missing_huggingface_credential').

How the collector authorizes (read for this change)

From PolicyEngine/calibration-diagnostics @ 3afb94d, telemetry-service/telemetry_collector/ (app.py, auth.py, storage.py, models.py):

• POST /v1/auth/huggingface/exchange: the HF token is checked against whoami-v2. An invalid token gets 401, a non-member of policyengine gets 403, and an unavailable HF gets 503. Success returns a JWT whose sub is the HF user id.
• POST /v1/runs: the first registration of a run id records the caller as owner_hf_id. A registration by another HF user gets 403 ("Run belongs to another Hugging Face user"). The owner with different metadata gets 409.
• POST /v1/runs/{run_id}/events: an unknown run gets 404, checked before the owner. An owner mismatch or an unregistered producer gets 403. A batch that fails the schema gets 422. A body over 1 MiB gets 413 from a middleware that runs before the route.
• RunRegistration is extra="forbid", so no local bookkeeping can ride in the stored registration (an older checkout delivering it would get 422).
• Run and producer ids match ^[A-Za-z0-9][A-Za-z0-9._:-]{0,191}$, so both may contain colons.

So a run belongs to whoever registers it first. Delivering another producer's run with your own credential has three outcomes:

• It works, if you are the same HF user.
• It gets 403, if another user registered the run.
• If the run is not registered yet, it claims the run for your user. The real producer's own registration then gets 403.

Decision: who delivers which run, with whose credential

1. A live producer's run belongs to its own service. No other service sends any request for it.
2. Another producer's run is adopted only once its service has exited. Any service may adopt it, with its own credential, one adopter at a time. Runs orphaned by dead services keep being delivered.
3. An answer about the caller's credential decides only the caller's own run. Such answers are: no credential, exchange 401/403, registration 403, events 403. An adopted run goes local-only only for an answer about the run itself: events 404/413/422 and other settled 4xx, registration 409 and other settled 4xx. Otherwise it stays pending for a service whose credential can deliver it. This service does not offer it again under the same credential, and offers it again once its login changes.

Liveness: per-producer advisory leases, no spool migration

• Each service takes an exclusive flock on <spool>.leases/<sha256(run_id NUL producer_id)>.lock when its delivery is built in main(), before the service is ready. Its run therefore never has an event without a live lease. The service holds the lease for its whole life, and the kernel drops it on any exit, SIGKILL included.
• No lease, no service. A lock attempt that fails (the lease is held, or the system refuses the lock) is repeated for up to 1 s. A service that still has no lease does not start: main() prints one line and exits 1, and the build's emitter falls back to its disabled handle, as for any service that does not become ready. The reason is that a failed attempt has already created the lease file, and a lease file nobody holds is how other services recognise an exited producer.
• Another producer's pending run is adoptable only if its lease can be taken without waiting. The adopter holds the lease only while delivering that run.
• After taking a lock, the code checks that the open file is still the path's inode. If a sweeper unlinked the file in between, it retries. If the check itself raises, the descriptor is closed first, so no lock is left held.
• No lease file means an older checkout, or a free lease that was swept. Such a run is adopted, under a lease the adopter creates, only after ORPHAN_IDLE_SECONDS (15 min) without an update. Live services append a heartbeat every 60 s by default.
• A periodic sweep (every 10 min) removes free lease files of runs with nothing pending.
• Why not a schema change: every checkout on a host shares this spool, and a new Alembic revision makes older checkouts refuse it. Leases live outside the database. upload_state values and reason strings are unchanged.

Change

• leases.py (new): ProducerLeases (hold, try_acquire, sweep), ProducerLease and OwnLeaseUnavailableError.
• collector.py:
  • CollectorDelivery(spool, registration, …) now knows its own run and holds its lease. The constructor raises OwnLeaseUnavailableError if it cannot take it (except on a platform without fcntl).
  • flush_once claims each run first (_claim): the own run always; another run only under its lease, or under the idle fallback.
  • Then, while holding the run, it checks again that the run is still pending (run_has_deliverable) before sending anything. A pass reads its list of pending runs once, then can wait seconds on the collector. Meanwhile a run's own service may have made the run local-only and exited, freeing its lease. Without this check the pass adopted that run and uploaded it under its own login. The two-process test below found this.
  • _make_local_only(…, caller_dependent=…) is the one gate. Registration 403 is now split from 409 and the other settled 4xx.
  • A refused credential is remembered as an in-memory SHA-256 fingerprint and not exchanged again.
  • A pass that has passed-over runs first notes the credential its requests would use (_note_current_credential), so a run passed over under one login is offered again once the login changes. A session the collector issued stays in use until it expires, as it does for the own run on main, so the new login counts from then.
  • The delivery's per-run memory (_registered, _passed_over, _warned_denied) is keyed by the (run_id, producer_id) pair. main keys the first and last by "run_id:producer_id", which two runs can share.
  • Warnings that say "this run" are printed only for the build's own run. An adopted run's reason is kept in the spool.
• runtime._drain waits only for the build's own run (EventSpool.run_has_deliverable). Before, every shutdown could wait out its full drain while another build had queued events, because those events are no longer this service's to deliver.
• main.py builds the delivery before the service runs and closes it afterwards. spool.py gains run_has_deliverable and last_updated.

Invariants

Each is checked by test_telemetry_foreign_runs.py, within the fake collector's model of the hosted one. Invariants 1 to 5 are also checked after every step of a Hypothesis property test (150 examples in CI) over 2–4 running builds plus up to 30 random steps:

• the steps: start, emit, flush, kill, change login, and run one service while another's collector request is in flight (that other service may exit then);
• logins: none, outsider, expired, alice, bob;
• shared run ids;
• run ids whose registration the collector answers with 409, and runs whose batches it answers with 422.

The model leaves out lock failures, ids with colons, oversized bodies (413) and real retry timing; the first two have their own tests below.

1. No foreign local-only from a caller's credential. A service never makes another producer's run local-only because of its own credential (missing, refused, or not the run's owner). In the tests, every foreign local-only is preceded, as that service's latest collector answer, by an events 422 or a registration 409 for that run.
2. No requests about a live producer's run. While a producer's service is alive and holds its lease, no other service sends any collector request about that producer's run. A service started through main() with this change holds its lease whenever it is serving, because it does not start without one. Two kinds of service have no lease and are protected only by the 15-minute idle rule, which is a heuristic: one from an older checkout, and one on a platform without fcntl.
3. Every local-only follows a justifying answer. When a service makes a run local-only, its latest collector answer is the refusal that justifies it: an events 422 or a registration 409 for another producer's run; a registration 403/409, an events 403, or an events 404/422 for its own. Two cases have no fresh answer: a missing credential, and a credential the collector refused to that service earlier, which is not exchanged again.
4. A refused credential is exchanged once per service.
5. Local-only is final. No request about a run reaches the collector after the run goes local-only, even from a service that listed the run before.
6. Orphans are delivered. Once every service has gone, every run left pending is delivered in full by a later service whose login owns it. That includes a service that was refused for the run under an earlier login, once any session from that login has expired. Two exceptions: a run id the collector answers with 409 goes local-only with run_registration_rejected, and a run whose batches get 422 goes local-only with collector_rejected_events. The property test checks this with two last services that start under generated logins, make a pass, and then become alice and bob, the only owners in the model.
7. The own run behaves as before, with one exception: a service that cannot take its lease no longer starts. Otherwise the own run goes local-only for the same reasons, with the same reason strings and warnings. The existing test_telemetry_emitter.py cases pass with two changes only: the new constructor argument, and the drain test's fake spool answering the scoped query.
8. The shutdown drain waits only for the build's own run. It loops only while that run has deliverable events. A pass it makes may still deliver an adopted run, and a pass in flight is not cut off at the deadline.

Deterministic tests:

• the e7 regression;
• a run made local-only, and its service gone, after another pass listed it is not adopted;
• a live run gets no request from a service with any of the five logins;
• an orphan waits for a credential that can deliver it, with one exchange for a refused login;
• the other-owner 403 is passed over, not made local-only;
• a passed-over orphan is offered again once the login changes (outsider and expired), with one exchange for each login;
• with a session in hand, the new login counts only once that session has expired;
• an orphan the collector refuses (422 or 409) goes local-only with no warning in this build;
• a shared run id refuses the second user only that user's own run;
• two runs whose ids join to the same text (a:b/c and a/b:c): a refusal for one does not pass over the other, and registering one does not count for the other;
• a lease-less run is adopted only once idle;
• the drain scope;
• lease exclusivity, the sweep, the replaced-inode race, and an inode check that raises leaving nothing locked;
• a lock the system refuses once at startup (ENOLCK) is retried, and no other service touches that run;
• a service that cannot take its lease (every lock refused, or the lease held) does not start: the constructor raises, and main() returns 1, prints its one line and never serves;
• a real SIGKILL releasing a lease;
• two real service processes sharing one spool (A with a login, B without). A's token exchange is held until B's first pass has made B's own run local-only, and for 1.5 s more (B's worker passes every second; the test does not check that a second pass ran). A's run must still arrive in full, B's must stay local-only and never reach the collector. Run with the service subprocesses on origin/main's code at 662fd4a's base, this test failed with build-a local-only for missing_huggingface_credential, which is the reported bug.

Review

Round 1 (GPT-6.1 Sol, at 662fd4acb) returned REQUEST_CHANGES with four should-fix findings, each executed by the reviewer. 1c9bd671b addresses them, and 615d84f89 adds two tests the microcosm-emitter port already had (an adopted run's events 403 is passed over; the lease file layout is pinned). Each finding has a test that fails on 662fd4acb and passes now (checks/r1_new_tests_fail_on_662fd4acb.txt in the evidence folder).

1. A failed own-lease lock let another service claim a live run. An injected ENOLCK on A's startup lock left a free lease file while A ran. B took the file, registered A's run under B's user, and A's later registration got 403. Now the attempt is repeated until its deadline, and a service without its lease does not start (above).
2. A replacement login never revived passed-over orphans. After a refused login made the adopter's own run local-only, switching to the owner's login left the orphan skipped: the pass-over check compared against a credential fingerprint that only a delivery refreshed, and nothing was being delivered. This is the same finding the microcosm-emitter port recorded here; the fix matches PolicyEngine/microcosm-emitter@07b2c36.
   • The reviewer also suggested dropping a cached session when the login changes. This PR does not: a session the collector issued stays in use until it expires, which is what main does for the own run, and a test pins it.
3. Distinct ids collided in the pass-over memory. ("a:b", "c") and ("a", "b:c") shared the key "a:b:c", so Bob's 403 for the first suppressed his deliverable second run. The memory is keyed by the id pair now. _registered had the same key on main, with a worse effect once runs are adopted (the second run's registration is skipped, its events get 404 and the run goes local-only), so it is fixed too.
4. An inode-check exception leaked the descriptor and the lock. _lock now closes the descriptor first.

Also from its notes:

• Local-only decisions were checked against any earlier answer. They are now checked against the deciding service's latest answer, and invariant 3 says so.
• The two-process test's 1.5 s sleep does not show that B made a second pass. The docstring and this description no longer say it did.
• The property test's liveness check used fresh rescuers, which have no pass-over memory. Its last two services now start under generated logins and make a pass first.

Mutation check

23 runtime mutants were injected through a pytest plugin that patches module globals, so no tracked file was edited. Each ran on 615d84f89 against both touched test files, and each was killed by the tests that target it: every listed test failed on an assertion or on the exception it expects, and no run had a setup or collection error. The 12 round-1 mutants are joined by 11 for the review fixes. no_gate, no_recheck, no_own_lease and drain_everything were rewritten for the code as it now is.

Mutant	What it removes	Killing tests
original_bug	both the claim and the gate	22, including the e7 regression and the stale-list test
no_gate	the _make_local_only gate (main's version)	12: orphan waits, other owner (register and events 403), login change, session wait-out, colliding ids, refused-orphan warnings, property
no_claim	lease and idle checks (deliver every pending run)	10: live run (alice, bob), other owner (register and events 403), session wait-out, colliding ids, refused lock at startup, shared run id, lease-less idle, property
gate_all_foreign	over-correction: never make a foreign run local-only	3: refused orphan (422, 409), property
conflict_caller_dependent	a registration 409 treated as an answer about the caller	2: refused orphan (409), property
no_recheck	the state check under the claim	2: the stale-list test, property
no_passed_over	memory of a 403 pass-over	4: other owner at registration and at events ([403, 403, 403] != [403]), session wait-out, colliding ids
no_credential_memo	refused-credential memo	5: orphan-wait exchange counts (outsider, expired), login-change exchange counts (outsider, expired), property
no_credential_refresh	noting the current login before pass-over filtering (finding 2)	4: login change (outsider, expired), session wait-out, property
refresh_ignores_session	over-correction: a new login counts while a session is live	session wait-out ([403, 403] != [403])
no_own_lease	the own lease	16
own_lease_optional	662fd4acb's lease code: give up on the first error, then serve lease-less (finding 1)	4: refused lock at startup, cannot start (locks refused, lease held), own lease waits out refused locks
start_without_lease	serve without a lease after waiting out errors	2: cannot start (locks refused, lease held)
hold_gives_up_on_error	lease required, but the first lock error ends the attempt	2: refused lock at startup (the service refuses to start), own lease waits out refused locks
close_keeps_lease	releasing the lease when a service goes	13
always_idle	the idle wait for lease-less runs	lease-less idle test
drain_everything	main's drain over every run	drain-scope test (it flushed for the other build's run). Two older drain tests also fail, on AttributeError, because their fake spool now answers only the scoped query
sweep_held	sweep safety	14
trust_replaced_file	the inode check	replaced-file test
unchecked_lease_leak	closing the descriptor when the inode check raises (finding 4)	unchecked-lease test
joined_pass_over_keys	pass-over memory keyed by run_id:producer_id (finding 3)	colliding ids, pass-over
joined_registered_keys	registration memory keyed the same way	colliding ids, registration
warn_for_foreign	"this run" warning scoped to the own run	refused orphan (422, 409)

The gate and the claim are independent protections. The e7 regression test fails only when both are removed (original_bug); each one alone is caught by its own tests.

Verification

Local runs are on macOS with Python 3.14.7 (free-threaded build), on a heavily loaded host. Each is labelled with the head it ran on; 615d84f89 changes only tests after 1c9bd671b.

• Touched test files (615d84f89). test_telemetry_foreign_runs.py and test_telemetry_emitter.py: 70 passed, 1 deselected. The deselected test is test_subprocess_exchanges_ambient_token_and_delivers_events: importing the service takes 5 to 6 s on this host, past that test's 3 s startup budget (unchanged by this PR; CI runs it). ruff check and ruff format --check are clean on the touched files, and tools/ci_test_plan.py verify passes.
• The new tests on the old code (662fd4acb's source). Eight cases fail there: login change (outsider, expired), session wait-out, colliding ids (pass-over and registration), refused lock at startup, unchecked lease, and the property test's second @example. Three tests were written after that run: a service that cannot take its lease does not start (two cases) and the own lease waits out refused locks. The own_lease_optional mutant restores 662fd4acb's lease code, and all three fail under it (below).
• The bug, on main's code (1c9bd671b). The two-process test ran with its service subprocesses on a224dc110's source: it fails with {'build-a': 'missing_huggingface_credential'} != {'build-a': 'pending'}. On this branch it passes.
• Deep property run (1c9bd671b; the property test is unchanged in 615d84f89). 2,000 examples passed, 0 failed, 54 invalid. Share of examples reaching each path:

Path	Share
Adopted orphan, register 201	77%
Adopted orphan, events 202	73%
Adopted orphan, register 409	14%
Adopted run local-only for that 409	14%
Adopted orphan, register 403 (another user's run)	10%
Adopted orphan, events 422	6%
Adopted run local-only for that 422	6%
Own run local-only: credential rejected	32%
Own run local-only: no credential	21%
Own run local-only: registration rejected	6%
Own run local-only: events rejected	2%

Limits and follow-ups

• Older checkouts. A service from a checkout without this change still runs the old logic on shared rows. Until those checkouts update, it can still mark other builds' runs local-only, or deliver a live build's run under its own login. New services treat an older service's run (no lease file) as live until it has been idle for 15 minutes.
• A service that cannot lock. If its lease cannot be taken within 1 s, the build runs without telemetry and says so. That is a deliberate trade: serving without the lease risks the run being uploaded under another login. On a filesystem that refuses flock altogether, no service with this change starts.
• Platforms without fcntl. There are no leases there. A service adopts only idle runs and has no exclusion. The service already needs Unix sockets, so no supported platform lacks fcntl.
• microcosm-emitter. Keep one build's telemetry login from deciding other builds' runs microcosm-emitter#5 ports this fix and already has findings 2 and 4. Findings 1 and 3 apply to it too and are recorded on that PR. The lease files are unchanged, so the two packages still recognise each other's leases.
• Overlap. Keep the telemetry emitter service alive through spool lock contention #1168, Isolate telemetry delivery failures per run #1176, Decouple the telemetry emitter's event path from spool writes #1174 and Let checkouts at different versions share the telemetry spool #1175 rewrite the same service. They are stacked on Keep the telemetry emitter service alive through spool lock contention #1168 and not yet merged. This branch keeps its collector edits to hook points (_claim, the _make_local_only gate, the credential memo and note), so whichever lands second rebases; the property test catches a dropped invariant. A rebase must keep one rule in mind: a second CollectorDelivery for the same run in one process now raises unless the first was closed.

Evidence (design note, logs, mutant plugin and results, review prompts and reports) is in a local folder, ~/reviews/microcosm-telemetry-foreign-local-only-2026-10-09/.

axiom: n/a: telemetry infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Finding from the microcosm-emitter port of this PR (PolicyEngine/microcosm-emitter#5), reproduced there; the code is identical here at 662fd4acb.

A run passed over under a login the collector refused is never offered again by the same service after its login changes.

Reproduction (two services):

1. Build A (alice) registers its run, queues events, and exits.
2. Build B starts with a login the collector refuses (outsider or expired) and makes a pass. A's run is passed over, with _passed_over[A] holding that login's fingerprint, and B's own run goes local-only.
3. B's login changes to alice.
4. B never delivers A's run. _claim compares _passed_over[key] with self._credential, which only _collector_token refreshes, and nothing calls that once every pending run is passed over and the own run is not pending.

The run is not lost, since another service can still adopt it, but this one never does. The property test misses it because its final liveness check starts fresh rescuers, which have no pass-over memory.

Fix in the emitter, commit 07b2c36: flush_once calls _note_current_credential() when _passed_over is non-empty. It leaves _credential alone while a session token is cached (requests still go out under the login that got it, as for the own run), and otherwise sets it from the current login.

The same commit has two more changes that apply here:

• leases._lock closes its descriptor if _names_open_file raises. Before, a stat error other than FileNotFoundError left the lock held on a descriptor nobody would close.
• Tests: three regression tests, a property test whose two last services start under generated logins before becoming alice and bob, and local-only decisions checked against the deciding service's latest answer.

Two claims in this PR's body were also flagged as broader than the tests: the two-process test's 1.5 s sleep does not show B made a second pass, and "right after a collector answer" was checked against any earlier answer, not the latest.

This PR should take the same fix before it merges.

🤖 Generated with Claude Code