Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
3042607
feat(studio): SD-card stall visibility — per-trial stimulus quality, …
mbreiser Sep 13, 2026
204758f
fix(telemetry): decode sd_reads as arg << code and sd_slow_ctx as USD…
mbreiser Sep 13, 2026
4876330
docs(sd): SD-read jitter + card-stall evidence, corrections, what shi…
mbreiser Sep 13, 2026
384da89
docs: telemetry review handoff (what is logged, three-clock model + e…
mbreiser Sep 13, 2026
2f4e3b8
fix(telemetry): SET_FRAME_POSITION index from the firmware's request …
mbreiser Sep 13, 2026
f1535df
fix(telemetry): sd_reads checkpoint bit (code bit 7) — cumulative mid…
mbreiser Sep 13, 2026
6ceeafe
fix(telemetry): sd_reads checkpoint semantics in trial-quality + tele…
mbreiser Sep 13, 2026
2377a1b
fix(telemetry-report): sd_slow_ctx as USDHC error bits, sd_reads chec…
mbreiser Sep 13, 2026
7b3e619
docs(bench log): H-FAT prediction recorded before the control iterati…
mbreiser Sep 13, 2026
8446911
docs(bench log): offline FAT-cache simulation inconclusive; bench A/B…
mbreiser Sep 13, 2026
04dca6b
docs: control iteration 1 results on 3c71953 vs baseline (0 stalls vs…
mbreiser Sep 13, 2026
946e890
docs: iteration 2 results — 482k commands, 0 stalls on 3c71953; revie…
mbreiser Sep 13, 2026
94d84e4
docs(bench log): session close — causal reflash deferred, bench state…
mbreiser Sep 13, 2026
18c4ee2
feat(studio): SET_SD_DIAG 0xCE host side (encoder, GET_SD_INFO byte 2…
mbreiser Sep 13, 2026
9cd00bc
docs(sd): §8 — does the fix generalise? (no size limit under H-FAT, c…
mbreiser Sep 13, 2026
a4ced83
fix(telemetry): STATE kind 14 sd_reads_ckpt (kind 13 keeps its meanin…
mbreiser Sep 13, 2026
627d94e
docs: causal-test plan — round-3 prerequisites (bit 6, applied-mode b…
mbreiser Sep 13, 2026
b0a6ce7
feat(soak): stress-first SD campaign — soak_mode3_stress.yaml (8 MB s…
mbreiser Sep 13, 2026
c961c8a
docs: overnight soak hand-off run sheet (consolidated build, three ru…
mbreiser Sep 13, 2026
d22b720
docs: causal test result — H-FAT confirmed (4 arms on 75405ee: FAT ac…
mbreiser Sep 13, 2026
031d13b
docs: consolidation plan draft for the production-candidate hand-off …
mbreiser Sep 13, 2026
2e7b54d
docs: the one hand-off document — Mode-3 reliability, firmware + Stud…
mbreiser Sep 13, 2026
0101cef
docs: run-log format review — row types with examples, measured MB/h …
mbreiser Sep 13, 2026
9cb940f
fix(studio): whole-stack review — verdict + final drain before the ru…
mbreiser Sep 13, 2026
9f06e5e
test(trial-quality): FRAME-only slow read flags the trial; sd_slow + …
mbreiser Sep 13, 2026
71fb119
docs: whole-stack review reconciliation (web) + applied/deferred list…
mbreiser Sep 13, 2026
4c532ee
docs(run sheet): candidate e59767e, Studio v0.78, browser-free upload…
mbreiser Sep 13, 2026
4b0fdda
docs(bench log): 15:20–15:50 ET reviews, fixes, candidate e59767e, v0…
mbreiser Sep 13, 2026
484621e
docs: overnight soak test plan — timing, log completeness, recovery (…
mbreiser Sep 13, 2026
646fd1e
scripts: runlog-check.py — run-log completeness checks for the overni…
mbreiser Sep 13, 2026
3400d45
docs+scripts: runlog-check reports ring drops as info (verdicts decid…
mbreiser Sep 13, 2026
6fc19e8
docs(bench log): 16:12 switch — 1 h/286 Hz result (1.04 M commands, 0…
mbreiser Sep 13, 2026
c14ac61
docs: lab test day on Windows (no Claude) — install, flash, card, smo…
mbreiser Sep 13, 2026
d874524
docs(bench log): segment 1 — 8 MB sine at 286 Hz on e59767e: 0 stalls…
mbreiser Sep 13, 2026
0c4cfef
docs: course dashboard readiness, old course protocols on the candida…
mbreiser Sep 13, 2026
73e72c6
docs: firmware pushed — PR #55 is the merge candidate; Studio merge t…
mbreiser Sep 13, 2026
c34f8f1
docs(bench log): stress phase complete; one 19.4 ms body-phase read o…
mbreiser Sep 13, 2026
8263c12
docs: archive the day's working documents under docs/development/arch…
mbreiser Sep 13, 2026
ef4c864
docs: archive the wedge-era working notes (2026-09-08 → 09-13) under …
mbreiser Sep 13, 2026
5036f89
docs(hand-off): §6.0 test history — every bench run of the stack with…
mbreiser Sep 13, 2026
20101c1
fix(studio): session rig follows the controller's reported geometry w…
mbreiser Sep 13, 2026
4b83830
docs: candidate is feat/mode3-reliability @ 488d5b9 (PR #56 on main, …
mbreiser Sep 13, 2026
a64c7b9
feat(studio): banner + log warning when SET_SD_DIAG switches are foun…
mbreiser Sep 13, 2026
a9ee92a
protocols: one-trial Mode-3 drill (injected-stall test / pre-overnigh…
mbreiser Sep 13, 2026
d1d116d
docs(bench log): campaign stopped, 488d5b9 flashed + HIL, Studio v0.7…
mbreiser Sep 13, 2026
0419166
protocols: course-card pattern names (p3_heisenberg_ts = idx 36, sine…
mbreiser Sep 13, 2026
58c7692
feat(studio): fill pattern frame counts from GET_PATTERN_INFO after t…
mbreiser Sep 13, 2026
5d25939
docs(bench log): drill step 1 — course-card pattern names, soak-path …
mbreiser Sep 13, 2026
aa0712c
docs: the 20-frame-modulus finding (stale bridge frames) and T8 index…
mbreiser Sep 13, 2026
de30330
fix(sim): FicTrac field 22 is nanoseconds — the simulator wrote ms, s…
mbreiser Sep 13, 2026
78ca119
fix(studio): bind the stimulus-quality finalize to its run id — a pre…
mbreiser Sep 13, 2026
112c80f
fix(studio): footer stamp (a sed with -E treated \| as alternation an…
mbreiser Sep 13, 2026
b3bdc14
docs(bench log): drill exposes the overlapping-run finalize bug (fixe…
mbreiser Sep 13, 2026
8a53461
docs: telemetry & run-log reference — rows/events, three-clock model,…
mbreiser Sep 14, 2026
7e2d195
docs: fold the telemetry logging reference in — review-handoff status…
mbreiser Sep 14, 2026
d578868
docs(bench log): clean drill run — coverage 200/200, verdict correct,…
mbreiser Sep 14, 2026
ddbf721
fix(studio): link-drop fault classification also when the terminal ev…
mbreiser Sep 14, 2026
8df0b68
docs(bench log): forced watchdog drill — full recovery chain, reconne…
mbreiser Sep 14, 2026
a03d3c7
docs: USB-unplug drill (controller does not reset; Studio recovers), …
mbreiser Sep 14, 2026
9fee4c7
docs(bench log): power-cycle drill — fresh-ring recovery path works; …
mbreiser Sep 14, 2026
6f6979b
docs: overnight started 20:35 ET on 781efe2 + Studio v0.79 (stress pr…
mbreiser Sep 14, 2026
1252bf3
docs(bench log): overnight 1 h summary — 2 iterations, 40/40 pass, 0 …
mbreiser Sep 14, 2026
cd5cb40
docs(bench log): overnight 2 h summary — 5 iterations, 100/100 pass, …
mbreiser Sep 14, 2026
bcc5aa4
docs(bench log): overnight 23:45 — one 26 ms sine read, trial flagged…
mbreiser Sep 14, 2026
decdcb6
fix(runlog-check): count host-accepted 0x70 rows only when reconcilin…
mbreiser Sep 14, 2026
a4203fc
fix(runlog-check): accept both a-row hex spellings (Studio compact, h…
mbreiser Sep 14, 2026
eac5a78
docs: overnight benchmark results (29 iterations, 0 faults, 579/1/0, …
mbreiser Sep 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ test_patterns/
__pycache__/
*.pyc
*.jsonl
soak-patterns/
14 changes: 13 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,19 @@ fix flows to every page automatically; two hand-written HTML pages never will.
default `behavior_v2`); the runner asserts it via `log_control` and the bridge ACKS
the level it will actually write (`bridge.waitForLogLevelAck`) — a pre-3.0 bridge
never acks, so treat "no ack" as behavior_v1. Never write `log_format` into
`run_metadata` from anything but the acked/inferred level.
`run_metadata` from anything but the acked/inferred level. **The format authority for every
row and event (FicTrac rows, `a`, controller `cc`/`cf`/`cs`, all `{type:"log"}` events), the
three-clock model and the round-trip recipe is `docs/development/telemetry-logging-reference.md`.**
- **Controller telemetry rows + stimulus quality (v0.77, fw `sdfast`):** `cf` rows carry three OPTIONAL trailing fields
(`req_age_us`, `superseded`, `flags`) from ring-v2 firmware (0xCB flags bit 5) — readers must accept 8- or
11-element `cf` rows and STATE kinds 11–13 (`sd_layout`/`sd_slow_ctx`/`sd_reads`, reads = arg << code). `GET_SD_INFO`
0xCD is gated on 0xCB bit 5 only. Per-trial pass/flagged/unknown verdicts come from `js/trial-quality.js` (fed from
the drainer in ring order, dedup by seq; fail = any read or request age > 10 ms; `unknown` never becomes `pass`) →
`display_gap` + `trial_quality` run-log events; the Studio flags, never auto-excludes. Offline analysis:
`scripts/telemetry-report.py` (also reads the firmware repo's `scripts/sd_stall_test.py` logs). The 30–90 ms card stalls were
firmware FAT access (chain-walking seeks + `fatGet` at cluster crossings), proven by the 4-arm causal test on 2026-09-13 and
removed by the contiguous-seek fast path — never re-introduce per-seek chain walks; see
`docs/development/mode3-reliability-handoff-2026-09-14.md`.
- Bump the footer version/timestamp on every edit; never Prettier the HTML.

- **Telemetry ring — four rules from the 2026-09-12 Codex review (all tested):** (1) the ONLY gate
Expand Down
218 changes: 209 additions & 9 deletions arena_studio.html

Large diffs are not rendered by default.

16 changes: 16 additions & 0 deletions docs/development/archive/mode3-2026-09/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Archive — Mode-3 reliability working documents, 2026-09-13

Superseded by `docs/development/mode3-reliability-handoff-2026-09-14.md` (the one hand-off document). Kept for history:

- `sd-read-jitter-2026-09-13.md` — the SD-stall evidence write-up and the H-FAT explanation with diagrams (long).
- `sd-stall-causal-test-plan-2026-09-13.md` — the four-arm causal test (result: H-FAT confirmed 15:11 ET) and the campaign design.
- `consolidation-plan-2026-09-13.md` — review gates, decisions and small changes before the hand-off (all decided 17:30 ET).
- `soak-handoff-2026-09-14.md` — the first run sheet; folded into `lab-test-day-windows-2026-09-14.md`.
- `mode3-perf-handover-2026-09-13.md` — the session brief this work started from.

Wedge-era working notes (fw #50, 2026-09-08 → 09-13), superseded by the same hand-off document and the bench log in
`../../mode3-wedge-soak-plan.md` §10:

- `merge-bench-plan-2026-09-08.md`, `codex-review-2026-09-09.md`, `lab-pc-handover-2026-09-09.md` — the merge/bench plan and lab-PC handover before the wedge work.
- `mode3-wedge-status-2026-09-12.md`, `codex-review-2026-09-12-mode3-wedge.md`, `telemetry-performance-handover-2026-09-12.md` — the wedge diagnosis status, its Codex review, the telemetry-performance handover.
- `mode3-wedge-evidence-2026-09-13.md`, `mode3-wedge-night2-scan-2026-09-13.md`, `codex-review-2026-09-13-mode3-wedge-fix.md`, `mode3-wedge-upstream-note-2026-09-13.md`, `mode3-wedge-closeout-plan.md` — the evidence for the PIT race, the second night's scan, the fix review, the note for PJRC (Michael's thread), the close-out plan.
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Codex (gpt-6-astra) adversarial review — Mode-3 wedge campaign status + branch diff — reconciliation

**Date:** 2026-09-12 · **Reviewer model:** `gpt-6-astra` (standard + adversarial passes, both runs) ·
**Inputs:** `docs/development/mode3-wedge-status-2026-09-12.md` (plan/status review, run
**Inputs:** `mode3-wedge-status-2026-09-12.md` (plan/status review, run
`codex-plan-review-20260912-114033-86155`) and the branch diff `main...claude/mode3-wedge-soak`
(5,597 lines, 31 files, run `codex-diff-review-20260912-114034-86380`). Claude's independent
analyses were written first: `claude-analysis-20260912-status.md`, `claude-analysis-20260912-diff.md`.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# Consolidation plan — production candidate for the lab hand-off (draft 2026-09-13 15:40 ET, for Michael's review)

Goal: by tomorrow morning, ONE firmware build and ONE Studio version that the lab can flash, run and review, with
overnight benchmark data taken on exactly that build. Small changes only; every change re-reviewed; nothing rewritten.

## 1. What the candidate is

| piece | today | candidate |
|---|---|---|
| firmware | `feat/sd-fastpath-2x10` @ `75405ee` (31 commits on `arena-2x10-local`: health → ring → watchdog → free-running timer → SD fast path) | same branch + the small changes below, rebuilt, flashed for the overnight run |
| Studio | PR #198 (v0.76) + PR #202 (v0.77) | same PRs + small changes below (v0.78 if any HTML edit) |
| tools | `scripts/sd_stall_test.py`, `sd_upload_pat.py`, `sd_soak_campaign.sh` (firmware repo); `telemetry-report.py`, `wedge-scan.py`, `make-stress-patterns.js` (web) | committed, documented in the run sheet |

## 2. Review gates before the overnight flash (this evening)

**Michael's three criteria (15:35 ET), applied to every diff and to the reconciliation of the Codex passes:**
(a) **minimal complexity** — the simplest diff that does the job; overlapping instrumentation or dead paths get
removed, not documented around; (b) **good comments** — every non-obvious line says *why*, in the code, not in a
review report; (c) **one detailed hand-off document** explaining all the changes since 2026-09-11 together
(`docs/development/mode3-reliability-handoff-2026-09-14.md`, both repos, one narrative), not a trail across builds.


1. Whole-stack Codex adversarial reviews, running now: firmware `arena-2x10-local..HEAD`, web `main..claude/mode3-perf-sd`.
Reconcile → fix only blocking/significant items → one more Codex diff review of the fix delta → rebuild.
2. HIL on the rebuilt hex, port free: `pytest --transport=serial --port … tests/test_firmware_version.py
tests/test_health.py tests/test_telemetry.py tests/test_lab79_sd.py --pat soak-patterns/bar_200f_gs2.pat`.
3. `pixi run test` + `format-check` (web). Studio smoke on the bench origin: connect, label `freerun sdfast`, one 60 s
Mode-3 trial with the trial-quality banner (the v0.77 verdict path has not yet seen a real trial).

## 3. Decisions for Michael

- **Keep `SET_SD_DIAG` 0xCE in production?** Recommend **keep** (gated on flag bit 6, off at boot, restored by the
harness) but add a Studio banner when the readback is non-zero at connect (S-1 below). Alternative: compile it out
(`fw_sd_diag = false`) for the release build — loses the lab's ability to reproduce the stall on demand.
- **Merge target for firmware — DECIDED (Michael 17:30 ET):** the branch IS the merge candidate; pushed, PR #55
against `arena-2x10-local` (fast-forward); the lab flashes from the branch; merge after the lab day; port to
`main` (per-board `-DARENA_HW_*` after #48) as a follow-up PR that supersedes #53.
- **Web — DECIDED (Michael 17:30 ET):** merge #198 then #202 to `main` as they are, without a separate review, first
thing tomorrow morning after a clean overnight (the v0.78 export-ordering code's first bench exercise is the
night); the lab tests from main; revert is one command if anything is wrong.

## 4. Small changes proposed (only these unless the reviews find more)

| id | repo | change | why |
|---|---|---|---|
| F-1 | fw | `SET_SD_DIAG` bit0 refused (status 1) when the volume is exFAT | the arm is a silent no-op there (Codex rounds 3/4) |
| F-2 | fw | README: causal result + "production = flags 0"; `scripts/` section lists the three bench tools | hand-off |
| S-1 | web | Studio: banner "SD diagnostic switches are ON (arm N)" when `GET_SD_INFO` byte 29 ≠ 0 at connect / run start | a stray arm would silently degrade every trial |
| S-2 | web | run sheet: browser-free campaign path (upload + `sd_soak_campaign.sh`) as the default overnight tool; Studio path as the alternative | it is what ran tonight |
| S-3 | web | CLAUDE.md: one line on `sd_soak_campaign` / `sd_upload_pat` and on "stalls were FAT access — never re-introduce chain-walking seeks" | institutional memory |

### 4.1 Applied after the whole-stack reviews (15:30–16:00 ET) — see `.codex-review/report-20260913-*-fullstack.md`
Firmware (commit after `75405ee`): panels blanked at boot (a watchdog/software reset left the last frame lit),
watchdog kicked inside the ISP loops, same-index skip only after a successful DAC/LUT update, `SET_SD_DIAG` bit 0
refused on exFAT (F-1 done), harness drains the backlog to completion, script deadline/verification fixes.
Studio v0.78: verdict + final drain before the export (normal and fault runs), drain problems → `unknown`, bridge
send failure not acked, FRAME read time in verdicts, link-drop = controller fault, stress gain 0.18, comment fixes.
Deferred items are listed in the hand-off document §7.

## 5. Overnight run (on the candidate)

`scripts/sd_soak_campaign.sh PORT "<sine idx> 36" 200 <until 08:00>` — 10-min alternating segments of the 8 MB sine and
the 813 KB bar at 200 Hz, Mode 3 random walk + 90° jumps, from ~21:00 to 08:00 (~66 segments, ~8 M commands). Morning:
`telemetry-report.py soak-logs/sdstall-*-camp-200-*.jsonl` + `campaign-*.log`. Pass = 0 reads > 10 ms, 0 reboots,
req_age max < 10 ms; per-file read cost tables are the benchmark numbers for the hand-off.

## 6. Documentation to update before hand-off

**The one hand-off document** (`mode3-reliability-handoff-2026-09-14.md`): the problem (wedge + stalls), the diagnosis
path and evidence, every firmware change by layer with its wire contract and flag bit, every Studio change, how to
test (HIL, offline, bench), the overnight benchmark, known limits, and where each thing lives. Then the smaller ones:
firmware README (F-2), web release notes (S-1), `soak-handoff-2026-09-14.md` (S-2, morning results), CLAUDE.md (S-3),
`sd-read-jitter-2026-09-13.md` §7 result (done), fw #54 / web #201 closing comments, memory.
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Lab-PC handover — bench-testing the run-log v2 stack (+ optional analog), 2026-09-09

**For a fresh Claude session on the lab Windows PC.** Read this file first, then
`docs/development/merge-bench-plan-2026-09-08.md` (the full gated checklist) and
`docs/development/codex-review-2026-09-09.md` (what was just fixed and what is still open).
`merge-bench-plan-2026-09-08.md` (the full gated checklist) and
`codex-review-2026-09-09.md` (what was just fixed and what is still open).
The Mac session that wrote this may still be running; treat GitHub as the source of truth for
branch state, not this file.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ fps meter.
### Morning (desk)

- [x] ~~Codex cross-review of all pending work~~ — done 2026-09-09 morning, see
`docs/development/codex-review-2026-09-09.md`: 7 true bugs fixed and pushed (#188 strict
`codex-review-2026-09-09.md`: 7 true bugs fixed and pushed (#188 strict
mode; #190 sweep refusal + closed-loop gating, #191 rebased; fw #47 calibration guard +
mV clamp + controller_info bits); design suggestions listed for discussion.
Also rebased #186/#188/#190/#191 onto main afterwards: #193 and the two stacks all add
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,4 +83,4 @@ Both Codex passes (status/plan + branch diff) are reconciled in `.codex-review/r

**12:20 ET addendum:** the ring firmware code itself was then Codex-reviewed (`LED-Display_G6_Firmware_Arena-ring/.codex-review/report-20260912-ring.md`). Flash is go after the bounded fix commit (reference-drainer handshake, HIL test corrections, ack/incarnation guard, heap guard coverage, exact-size boot repair); dual headers / linker reservation / PSRAM frame identity are deferred to the ship-to-course gate.

**Tracking (12:35 ET):** deferred web items → #200; deferred firmware ring hardening → fw #54. Reconciliation reports committed as `docs/development/codex-review-2026-09-12-mode3-wedge.md` (this repo) and `docs/development/codex-review-2026-09-12-telemetry-ring.md` (ring branch).
**Tracking (12:35 ET):** deferred web items → #200; deferred firmware ring hardening → fw #54. Reconciliation reports committed as `codex-review-2026-09-12-mode3-wedge.md` (this repo) and `docs/development/codex-review-2026-09-12-telemetry-ring.md` (ring branch).
Loading