Let checkouts at different versions share the telemetry spool

[MaxGhenis]

Stacked on #1168, which must merge first. CI only runs for pull requests into main, so this targets main; until #1168 merges, the diff also shows #1168's commits. This PR's own changes are d53bd3b9b and 3cd53f6d1, plus merges of #1168 and main (the merges resolve conflicts in collector.py, migrations.py, spool.py and the README).

Problem

Every checkout and worktree on a host opens the same spool, ~/.cache/microcosm/telemetry/events.sqlite3 (or under $XDG_CACHE_HOME; telemetry_emitter._cache_dir). Build Macs run several worktrees at different commits. With #1168, a service refuses a spool stamped with a revision missing from its checkout's history, so as soon as a newer checkout applies a second revision, every older checkout's builds run without hosted telemetry until they update. The second revision is already in flight: #1151 adds 20261008_02, one op.create_table("graph_publication_jobs", …) (#1148 adds a revision of the same name).

Options evaluated

What decides it is delivery. Every service's delivery worker delivers every pending run in its spool, not just its own (CollectorDelivery._flush_pending_runs iterates spool.pending_runs()), and event retention is enforced per file by whichever service prunes it.

	Older checkouts keep telemetry	Events queued by another version	Cost
One spool file per schema head	Yes, including checkouts from before the change	Delivered only while a service of that exact head runs; stranded once the host moves on. Nothing prunes or removes an abandoned file.	A new file, with its own 100 MB event cap, for every migration, feature branches included. Draining old files means reading and deleting rows in every historical schema.
Additive-only migrations older code tolerates (chosen)	Yes, for checkouts that include this change	One queue: a service of any version delivers them. An upgrade keeps every queued row.	Migrations are restricted, and older code must be able to tell that a newer revision descends from its head.
Older service writes to a sibling spool when the shared one is ahead	Yes	Sibling events are delivered only by services of that head, so they strand too, and each new head that falls behind adds a sibling.	Leaves migrations unconstrained, so a non-additive one still breaks older services that opened the spool before it ran: they fail mid-build, not at startup.

Per-head files have one real advantage: they also isolate checkouts that predate the change and branches whose migrations diverge. That would decide it if arbitrary historical writers had to be supported. For a queue whose point is delivery, stranding the backlog at every migration is worse, and the next migration is already additive.

Change

1. Lineage. Whenever upgrade_spool_database moves a spool to a new revision, it also records the spool's lineage (every revision in the migrating checkout's history) in spool_lineage, in the same BEGIN IMMEDIATE transaction as the Alembic upgrade.
   • The migration runner keeps this table, as Alembic keeps alembic_version, and env.py leaves it out of alembic check.
   • It is deliberately not an Alembic revision: adding one would itself be the second revision that makes every checkout older than this one refuse the spool.
2. Classification (classify_spool_revision), from the stamp, the recorded lineage and this checkout's history:
   • at head: used as is;
   • empty, or an ancestor of this head: upgraded;
   • unknown to this checkout, but the lineage records both the stamp and this head: ahead, used as is, without migrating or taking the write lock;
   • anything else (a branch with a different migration, or an unknown stamp with no or inconsistent lineage): refused with IncompatibleSpoolRevisionError. That is still one line from the service and exit 1, as in Keep the telemetry emitter service alive through spool lock contention #1168.
3. Second look under the lock. The read-only check stays (no lock in the common cases), and the spool is inspected again once BEGIN IMMEDIATE holds the lock, before Alembic runs. In Keep the telemetry emitter service alive through spool lock contention #1168, an opener that found the spool behind and then waited while a newer checkout migrated past it ran Alembic on a revision it did not know: CommandError: Can't locate revision identified by 'future' (the no_relock_check mutant reproduces exactly that).
4. Lineage repair. A checkout from before this change can move a spool's revision without recording a lineage (Publish UK Orrery graphs with the existing local build service #1151's branch today, for one). A checkout that finds the spool at its own head with no lineage, or another history's, records it under the same lock. A spool at a base revision is exempt, because every history contains it, so the spools that exist today are not written on first open.
5. Additive-only rule, stated in migrations.py and applied to every packaged revision after the initial one by test_every_packaged_revision_after_the_initial_one_is_additive. A revision may create tables. To a table that already existed it may only append ordinary columns that are nullable or defaulted, and add plain non-unique indexes. See "The additivity check" below.
6. EventSpool, upgrade_spool_database and the history helpers take an optional script_location, so tests can open one spool as checkouts with different histories.

Which checkouts this covers

Read from the code of each:

Checkout	On a spool migrated past its head	When it migrates a spool itself
Pre-Alembic (before 3d9b4678b)	Never reads a stamp; keeps writing and pruning with its own column names, and sets WAL mode	n/a
Alembic, before #1168	alembic upgrade head runs on every open, and fails on the unknown revision	Records no lineage
#1168 only	Refuses in one line, without the lock	Records no lineage
This change on	Uses it if the recorded lineage holds its head; refuses otherwise	Records the lineage; repairs a missing one at its head

Every invariant below is about the last row. The additive rule is also what keeps a pre-Alembic writer's statements working on a migrated spool, but nothing here tests that code.

Invariants (each has a test in test_telemetry_spool_versions.py)

• I1. Classification follows ancestry (Hypothesis, 300 examples over random revision trees, every head, stamp and lineage variant): at head iff stamp = head; upgrade iff the stamp is empty or a strict ancestor of the head; ahead iff the head is a strict ancestor of the stamp and the recorded lineage holds both; refused otherwise.

• I2. Real spools match a model (Hypothesis, 40 random trees of up to three revisions on top of the packaged history, real SQLite and Alembic, up to eight opens). Each open is by any checkout, with either a lineage-recording runner or one that upgrades without recording any, as Keep the telemetry emitter service alive through spool lock contention #1168's does. After every open:

  • the returned state is the model's;
  • the stamp and the recorded lineage (or its absence) are exactly the model's;
  • the schema holds exactly the tables and columns of the stamp's ancestry, so a refused checkout added nothing;
  • the database was committed to iff the model says this open upgrades or repairs a lineage (PRAGMA data_version);
  • an open that is not refused leaves every table and column that checkout's schema has in place. This holds with older runners in the mix because a lineage is trusted only when it contains the spool's stamp.

  Recorded events show every runner and state reached. In one run, 24% of examples included a refusal of a lineage-recording checkout and 20% a lineage repair.

• I3. One line of history never refuses (Hypothesis, 12 chains): lineage-recording checkouts whose heads lie on one chain, opening in any order, are never refused, and the spool ends at the newest head any of them brought.

• I4. An older checkout uses a newer spool without the lock. With another writer holding the lock, it opens well inside one busy wait, then registers, appends, batches, acknowledges and prunes. The stamp stays the newer one.

• I5. A version difference strands no event.

  • In process: events queued under the older history are delivered through the newer one, and vice versa.
  • Upgrading from the packaged head keeps every queued row byte for byte (new columns take their defaults) and its delivery, local-only runs excluded.
  • End to end: the real client starts the real service, as the older checkout, on a spool a newer history migrated. It delivers its own build from started to completed, in order, and the run the newer history queued.

  These vary the migration history under one version of the application code. What older services need from newer rows is written down in migrations.py rather than tested: upload_state == 'pending', run_id and producer_id in the registration, event_id in each event.

• I6. A live older service across a newer checkout's migration. Its open connection's statements keep working after the schema change. An append that arrives while the migration holds the lock waits and then succeeds.

• I7. Divergence is refused without the lock. Two branches each add a different revision to the same parent: the second is refused inside one busy wait and changes nothing, and the common ancestor still uses the spool. A spool stamped with a future revision is used only if its lineage holds both that revision and this head (checked with no lineage, one without the stamp, one without this head). The real service process prints exactly one line naming both revisions and exits 1.

• I8. Under the lock (deterministic): an opener that found the spool behind and then waited out a newer checkout's migration returns ahead, leaves that checkout's lineage as it was, and all three checkouts can then open the spool.

• I9. Lineage repair. After a runner without lineage migrates a spool (whether it left a stale lineage or none), an older checkout is refused; once a checkout at the new revision opens it, the lineage is that checkout's history, later opens only read (checked with the lock held elsewhere), and the older checkout works. A spool at the initial revision with no lineage is opened without a write or the lock.

• I10. Atomicity. An upgrade that fails after writing the lineage leaves the schema, the stamp and the lineage as they were.

• Lineage is not part of the compared schema: alembic check passes on a spool that records one.

A one-off run at 10× the examples (3,000 / 400 / 120 / 800) found no counterexample. A second one-off run replaced the packaged history, in the test process, with one that already has a second revision modelled on #1151's: the only tests that differed were the two that compare with a real subprocess or with the ORM models, and the packaged-revision test passed on that create_table revision.

The additivity check

It is a guard that holds a migration to a short list of operations, not a proof that older code keeps working. It is stricter than compatibility needs: it refuses some harmless changes, such as replacing an index. For each revision and each parent it builds a spool at the parent, fills the telemetry tables with the rows the real spool code stores (a pending run with events, a local-only run, a run with none), applies the revision in the production BEGIN IMMEDIATE transaction while tracing every statement SQLite executes, and requires:

• Statements: on a table that existed before, only ALTER TABLE … ADD COLUMN and CREATE INDEX. Any INSERT, UPDATE, DELETE, DROP, trigger or other ALTER on it is refused for its kind, whatever rows it would touch. A statement of a kind the check does not recognise is flagged for review.
• Definitions: each existing table's stored definition is the old one with one insertion of plain column definitions (what ADD COLUMN does). So no rebuild, even an identical one, and no changed collation, constraint or table option.
• New columns on existing tables: not generated, not in the primary key, nullable or defaulted, with no CHECK, UNIQUE or expression.
• Indexes and triggers: existing ones unchanged; new indexes on existing tables non-unique, not partial, on plain columns.
• Rows: the seeded rows unchanged, and an insert, update and delete of real rows naming only the old columns still succeeding.
• The migration itself succeeding inside that transaction.

test_the_additivity_check_flags_exactly_the_forbidden_changes (Hypothesis, 80 examples over any combination of 7 allowed and 24 forbidden changes) checks that problems are reported iff a forbidden change is present, and that on every accepted combination this checkout's spool code works with the spool stamped as a future revision. Each forbidden change alone is reported for its own reason, and nine real Alembic revisions, including batch-mode rebuilds and one that runs VACUUM, are rejected through the same path. The forbidden list includes the three changes an earlier version of this check passed, found in review: a generated column that overflows on valid JSON, DELETE … WHERE json_valid(payload_json), and a rebuild that changes a column's collation.

What it cannot see: what rows mean, and anything a migration does outside SQL on its own connection.

Mutation check

A pytest plugin swapped one behaviour per run (no tracked file edited). The baseline passed, and all 17 mutants were killed by failing assertions, with no collection or setup errors. Mutant, then the tests that killed it in the final run:

Runner behaviour:

• unknown_is_refused (the behaviour before this PR): 11 tests.
• unknown_is_ahead (no lineage needed): 8, including the diverged branch, all three refusal parameters, I1 and I2.
• ahead_needs_only_stamp (lineage need not hold this head): the diverged branch, lineage-without-this-head, I1, I2.
• ahead_needs_only_head (lineage need not hold the stamp): lineage-without-the-stamp, the stale-lineage repair test, I1.
• known_older_is_at_head (no upgrade): 7 tests.
• no_lineage: 10 tests. lineage_head_only: 9.
• incompatible_proceeds: the diverged branch, the three refusal parameters and both repair parameters ("DID NOT RAISE").
• no_lineage_repair: both repair parameters and I2.
• repair_at_base_too: test_a_spool_at_the_initial_revision_needs_no_lineage in every run, and I2 in some.
• no_relock_check: I8, CommandError … Can't locate revision identified by 'future'.
• relock_always_records_lineage: I8.
• no_lineage_filter: the alembic check test.

The additivity check's own rules, each switched off in turn:

• checker_no_statement_rule: 8 cases, among them the conditional delete and update as Alembic revisions, a replaced index, an identical rebuild and a delete through a WITH clause.
• checker_no_definition_rule: a column with a CHECK, a rebuild that adds a constraint, the collation rebuild, and the property test.
• checker_arbitrary_seed_rows (the seed rows the first version used): the conditional update, the conditional delete and the constrained generated column.
• checker_no_row_comparison: rewritten rows, deleted rows and both conditional cases.

After merging #1168's newer head, which changed the busy-timeout interface, the baseline and four mutants were run again on the merged tree 2ef0d13cb (no_relock_check, relock_always_records_lineage, no_lineage_repair, unknown_is_refused), with the same results.

Reviews

Two independent GPT-6.1 Sol reviews of the first version, a design review and a code review, both asked for changes; everything from "Lineage repair" down to the statement and definition rules came from them. The findings and what was done about each are in the comment below.

Tests run

Touched files only, with -p no:cacheprovider --basetemp=<scratch>: test_telemetry_spool_versions.py (new), 57 passed. After the first source change I also ran the migration and spool tests of the untouched test_telemetry_spool_contention.py and test_telemetry_emitter.py, and after merging main the collector tests; CI runs the rest. Ruff check and format are clean on the changed files.

Known limits

• Checkouts from before this change are not covered (table above). One that predates it still refuses a spool migrated past 20261007_01, and no in-place scheme can fix code already released. So the next spool migration (Publish UK Orrery graphs with the existing local build service #1151/Publish UK execution graphs through a general build emitter service #1148) should merge after this has reached build hosts, on a branch rebased onto it. Its create_table passes the additivity check as it stands.
• A runner without lineage can leave older checkouts refused until a lineage-recording checkout at the new revision opens the spool (I9).
• Diverging branches poison each other on a host. Once a branch's own migration has stamped the spool, a checkout with a different migration at the same point is refused until the spool is moved aside (losing its undelivered events). Never rewrite a migration that may have run on a build host; add a new revision.
• Two branches using the same revision id for different content cannot be told apart, because revision ids are not content-addressed.
• Sharing a queue shares its existing hazards across versions too. A service without credentials marks other builds' pending runs local-only, and a service pointed at a development collector delivers the whole backlog there; Keep one build's telemetry login from deciding other builds' runs #1177 addresses the first. An event sent while any writer, a migration included, holds the lock longer than SQLite's 5 s busy wait is lost; decoupling the event path from spool writes is the follow-up from Keep the telemetry emitter service alive through spool lock contention #1168.
• The 100 MB cap and seven-day retention cover telemetry event payloads. A table a later revision adds needs its own retention: older services neither prune it nor deliver from it.
• A change older code cannot use belongs in a new spool file, designed when first needed, not in this history.

axiom: n/a: telemetry infrastructure, no policy encoding

🤖 Generated with Claude Code

[MaxGhenis]

Round-1 reviews and what changed

Two independent GPT-6.1 Sol reviews of the first version (d53bd3b9b): a design review (11 findings, verdict "proceed with option 2 with named changes") and a code review (5 findings, changes requested). Responses, finding by finding; the fixes are in 3cd53f6d1.

Design review

1. Generated column passes, breaks older inserts (blocker). Fixed. A new column on an existing table that is
   generated is refused (new column is generated), as is any inserted column text containing GENERATED, AS,
   CHECK, UNIQUE or PRIMARY. Your exact column is generated-column in _FORBIDDEN_CHANGES and in
   _BREAKING_REVISIONS.
2. Row-conditional DELETE spares the seed rows (blocker). Fixed two ways. (a) The check now traces every statement
   SQLite executes during the migration (set_trace_callback) and refuses any INSERT, UPDATE or DELETE whose target is an
   existing table, whatever its WHERE clause; your DELETE … WHERE json_valid(payload_json) is conditional-delete.
   (b) The seeded rows are now the rows the real spool code stores (valid JSON, pending and local_only runs, a run
   with no events). The PR no longer claims the check certifies a migration: it is described as a guard that enforces a
   short allowed-operation list, and the module docstring says so.
3. Collation, conflict policies, table options. Fixed. An existing table's stored definition (sqlite_master.sql)
   must equal the old one with one contiguous insertion of plain column definitions, which is what ALTER TABLE … ADD COLUMN does (verified against SQLite 3.53.1: it inserts after the last column, before table constraints). Any
   rebuild, including a byte-identical one, is refused, both by that rule and by the statement rule (DROP TABLE).
   Your COLLATE RTRIM rebuild is the collation case.
4. A pre-lineage checkout leaves a compatible spool refused forever. Fixed. upgrade_spool_database now records
   the lineage, under the same BEGIN IMMEDIATE lock, when it finds the spool at its own head with no lineage or
   another history's. A spool at a base revision is exempt (every history contains it), so the spools that exist today
   are not written on first open. Tests: test_a_checkout_at_head_records_the_lineage_an_older_runner_left_out (both
   with a stale lineage and with none), test_a_spool_at_the_initial_revision_needs_no_lineage, and the tree property,
   whose opens now mix lineage-recording runners with runners that upgrade without recording any (as Keep the telemetry emitter service alive through spool lock contention #1168's does).
   Until a lineage-recording checkout at the new revision opens the spool, an older one is still refused; that cannot be
   fixed from new code, and the PR says Publish UK Orrery graphs with the existing local build service #1151 must rebase onto this.
5. Pre-Alembic writers. Scoped. Confirmed from 3d9b4678b^: that code never reads a stamp, sets WAL mode, runs
   CREATE TABLE IF NOT EXISTS and prunes on open. The PR body now distinguishes four populations and restricts every
   invariant to checkouts that include this change. Not done: tests against the historical implementations themselves
   (they cannot be imported beside the current package); the property test simulates the Keep the telemetry emitter service alive through spool lock contention #1168 runner instead.
6. One service can suppress or consume another's backlog. Scoped, not fixed here. True and pre-existing; it applies
   between two services of one version as much as across versions, and microcosm#1177 is the fix. The PR's delivery
   claim is now "no event is stranded because of a version difference", with this hazard named.
7. schema_version does not establish delivery compatibility. Scoped and documented. The module docstring now
   states the row contract older services rely on (upload_state == 'pending'; run_id/producer_id in the
   registration; event_id in each event). The PR says plainly that the tests vary migration histories under one
   application version, and that the initial revision's adoption of pre-Alembic spools marks pre-eligibility runs
   local-only.
8. Events overlapping a migration. Qualified, one test added. test_an_older_append_waits_out_a_newer_migration_in_progress
   shows an append blocked behind an uncommitted migration completing after the commit. The PR states the limit: an event
   that outlasts SQLite's 5 s busy wait is lost, as behind any long writer; decoupling the event path from spool writes
   is the existing follow-up from Keep the telemetry emitter service alive through spool lock contention #1168.
9. Integrity of the out-of-chain bookkeeping. Partly done. Added
   test_a_failed_upgrade_leaves_the_schema_the_stamp_and_the_lineage (failure after the lineage write rolls all three
   back). A stamp moved by plain Alembic is the finding-4 case and is repaired the same way. Not done: an explicit read
   transaction for the read-only look. As you note, a torn read cannot produce an unsafe proceed; the decision that
   writes is re-made under the lock.
10. Deliberate conservatism (nit). Documented. The docstring and PR say the rule is stricter than compatibility needs.
    _ROWS_FOR_OTHER_TABLES gives a constrained new table valid rows for checking later revisions, and seeding fails with
    a message naming it; new-table-with-check is an allowed case.
11. Retention overstated. Scoped. The PR and docstring now say the cap covers telemetry event payloads, and that a
    new table needs its own retention because older services neither prune it nor deliver from it.

Code review

1. Generated columns. Fixed (design finding 1); your gate column is constrained-generated-column.
2. Generic seed rows. Fixed (design finding 2); your UPDATE … WHERE upload_state='pending' is conditional-update,
   and constrained new tables are handled as in design finding 10.
3. Checker transaction semantics. Fixed. Revisions are applied on create_spool_engine(…, immediate_transactions=True),
   as production does, and a migration that raises is reported (the migration failed in its transaction). A revision
   that creates a table and then runs VACUUM is the vacuum case in both drivers.
4. Race test did not check the lineage. Fixed. It now asserts the recorded lineage is still the newer checkout's and
   reopens as all three checkouts. Mutant relock_always_records_lineage is killed by it.
5. README caveat. Fixed.