Repository navigation
Every view kind's output is pinned, and output may not depend on which query finished first - #61
Merged
Merged
Conversation
…h query finished first
test/view-snapshots.test.ts runs one gallery example per view kind (30 kinds; the oncoprint has
its own golden SVG), plus four that combine studies, through the whole engine with the warehouse
replaced by answers recorded from it, and compares four files per case in test/snapshots/:
<case>.sql each cohort's compiled SQL and each view's queries, by label and study
<case>.json what each view computed (result without timings or queries)
<case>.svg each view's figure; <case>.txt the results text MCP clients and AI tools read
A change to shared code (a SQL fragment in federation-views.ts, a figure helper such as km.ts,
stats.ts, result-text.ts) that moves another view's queries, numbers, figure or text fails in
`npm test`, named by case, with no database. A changed query has no recorded answer and fails as
"not recorded". Comparison is exact: no rounding, no tolerance.
Each case runs twice, answers delivered in the order the queries were sent and then reversed,
and the two runs must agree. That found order-dependence in four runners, each combining answers
inside a Promise.all callback as they arrived, so the method record listed cohorts or studies in
arrival order ("Colorectal" first in one run, "Pancreatic" in the next) and the stratified
log-rank took its studies in arrival order:
- survival: per-study counts; enrichment: per-site steps; diffexpr: moments, hence gene and
density-grid order; assays: per-site steps, and entity names, first-come across studies.
Each now asks through askEachSite (lib/views/shared.ts), which returns answers in the view's
order (cohorts, then studies) whatever order they finish in, and combines afterwards. Reverting any
of the four makes its case fail with "depends on the order database answers arrive"; every
combining runner moves to the helper in the per-kind reorganization, and the check covers new ones.
No other output changed. A view's queries are listed sorted in <case>.sql: independent queries
form a set; the snapshot pins which are sent, not the order they happen to be sent in.
lib/ch.ts gains setChTransport(): every statement already passes through post(); a transport can
answer it or watch it reach the database, with extra settings for that statement. Only this test
sets one. Answers are keyed by instance, format, statement text and every external table sent with
it, stored gzipped under a fixed database name, without ClickHouse's timing statistics, and pooled
across cases so any case runs alone. Recording (RECORD_SNAPSHOTS=1, .env.local) reads the public
warehouse only (the second instance is unset first) with max_threads=1: ClickHouse otherwise adds
floats across threads in varying order and returns unordered rows in varying order; two
recordings in a row are now byte-identical. The answers hold public studies' rows, patient-level
ones included (table, timeline, alterations), as the public deployment shows them.
Long arrays, and objects of more than 50 keys (a method's check values), keep their first 20 items
and stand in for the rest by count and digest, so the files stay reviewable while any change still
fails. The run's wall-clock time is left out of the text.
Verified: three replays in a row identical; npm test 1471/1471 (53 s; the snapshot file runs the
engine 68 times, ~33 s, in parallel with the rest). Size: 34 gzipped answer files and 136
snapshot files, 6.3 MB. CLAUDE.md says how to update and re-record.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CuBjiBEjYispDSEYDZDWM1
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Snapshot tests for every view kind on master, so changes to shared code (e.g. #59's edits to
federation-views.ts,figures/km.ts,stats.ts) that alter an existing view's output are caught innpm test, named by view — no database needed.What it pins
test/view-snapshots.test.tsruns one gallery example per view kind (30 kinds; the oncoprint keeps its own golden SVG) plus four multi-study examples through the whole engine, with ClickHouse replaced by answers recorded from the public warehouse. Per case, intest/snapshots/:<case>.sql— each cohort's compiled SQL and every query the view sends<case>.json— what the view computed<case>.svg/<case>.txt— its figure, and the results text MCP clients and the AI tools readComparison is exact (no rounding, no tolerance). A changed query has no recorded answer and fails as "not recorded".
Order independence, enforced
Each case runs twice — answers delivered in the order sent, then reversed — and the two runs must match. That found four runners combining answers as they arrived (the method record listed cohorts/studies in arrival order; the stratified log-rank took studies in arrival order): survival, enrichment, diffexpr, assays. Each now uses
askEachSite(lib/views/shared.ts), which returns answers in the view's order regardless of finishing order. Reverting any of the four fixes makes its case fail with "depends on the order database answers arrive". No other output changed.Recording
lib/ch.tsgainssetChTransport()(the one statement choke point; only this test uses it).RECORD_SNAPSHOTS=1with.env.localre-records from the public warehouse only, withmax_threads=1so two recordings in a row are byte-identical.UPDATE_SNAPSHOTS=1accepts intended output changes. CLAUDE.md documents both.Costs
npm test: ~53 s (was ~26 s); the snapshot file runs the engine 68 times.Verification
GATE_PORT=4111 scripts/gate.sh: exit 0 — lint, build, unit 1471/1471, postgres 47/47, access mutants, Python conformance, e2e incl. federation, all warehouse checks (cBioPortal counts, cohort logic, examples/pinned counts/method scripts, denominators, gene facets, value folding). Three replays in a row identical; two single-thread recordings identical.🤖 Generated with Claude Code
https://claude.ai/code/session_01CuBjiBEjYispDSEYDZDWM1