feat(studio): SD-stall visibility + whole-stack review fixes + controller-derived session rig — Studio v0.79 - #202
Open
mbreiser wants to merge 64 commits into
Open
feat(studio): SD-stall visibility + whole-stack review fixes + controller-derived session rig — Studio v0.79#202mbreiser wants to merge 64 commits into
mbreiser wants to merge 64 commits into
Conversation
…card identity (0xCD), ring-v2 FRAME latency fields, telemetry report — Studio v0.77 The controller's SD card stalls 23–89 ms every ~24.5k reads of a large pattern (card-internal read-count maintenance; 712/716 stalls on the 813 KB pattern, none on the 81 KB one; measured displayed-frame gap = stall + ~2 ms). Acceptable freeze: 5 ms target / 10 ms worst case. - js/arena-telemetry.js: FRAME extras from ring-v2 firmware (req_age_us u32, superseded, flags → sdRead/contiguous), STATE kinds 11 sd_layout / 12 sd_slow_ctx / 13 sd_reads, sd_slow phase byte; cf rows gain three optional trailing fields; STREAM_SCHEMA updated; drainer per-poll budget 40 → 200 chunks (background-tab 1 Hz polling could not keep up with 26 B FRAMEs at 200–286 Hz). - js/arena-wire-g6.js: 0xCB flags bit 5 sdFastPath (+ ' sdfast' label), GET_SD_INFO 0xCD encoder/decoder (CID manufacturer/product/serial/date, capacity, FAT type, cluster size, label). - js/trial-quality.js (new, dual-export, tested): per-trial pass / fail / unknown from ring records in seq order — trial = successful sd_open; fail = sd_slow read or FRAME req_age over 10 ms; coverage gaps (seq gap, reboot, ring overrun, poller paused) → unknown, never pass; dedup by seq; display_gap events; reads/commands/index-change accounting. - arena_studio.html v0.77: reads 0xCD at link-up → Studio.sdCard → run_metadata.sd_card; feeds the drainer's records to trial-quality; display_gap events + trial_quality event at run end after a final drain; banner naming flagged trials (flag only — exclusion is the experimenter's call); FRAME dump lines show age/superseded/contiguous. - scripts/telemetry-report.py (new, tested): reads accounting per pattern, SD read cost by index step class, stall clusters with spacing in commands / index changes / seconds, request→presentation percentiles, per-trial verdict table; consumes Studio logs and sd_stall_test.py logs alike. - scripts/wedge-scan.py: kinds 11–13 (sd_layout labelled contig/FRAGMENTED + spc). - tests: arena-telemetry (v1 + v2 FRAME, kinds 11–13), arena-wire (bit 5, 0xCD fixture), trial-quality (21), telemetry-report (37); pixi test task wired. Release notes v0.77; CLAUDE.md rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…HC error bits (fw round-1 review changes); CLAUDE.md telemetry/stimulus-quality rule Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…pped, card protocol, read-free path (§6); bench log entry for the 2026-09-13 performance session Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…vidence, round-trip recipe, Teensy-only timing proposals, next experiments); bench log: flash, HIL, card identity, soak restart, first look Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…bytes (after len,cmd); host-prefixed form tolerated Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-open checkpoints, largest value wins Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…metry-report (largest cumulative value per open) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…kpoint semantics; test fixtures for the checkpoint bit Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…on completes Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… is the discriminator
… 4 clusters; FAT-walk penalty gone; 0.76 reads/cmd; req_age max 4.8 ms) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…w handoff experiment 1 marked done Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…, next steps Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…9 decode, sd_layout arm bits, Studio.setSdDiag, run_metadata.sd_card.sd_diag) + causal test plan for the SD-card stalls Four arms on one firmware build, switched at soak-iteration boundaries: legacy FAT-chain seek and/or no same-index skip. The telemetry report labels each iteration's arm from the sd_layout records; the plan (docs/development/sd-stall-causal-test-plan-2026-09-13.md) gives predictions per hypothesis, the phase-byte fingerprint (H-FAT ⇒ stalls inside the seek), stopping rules and the procedure. Bench run pending. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ontiguity is the precondition; read-path, timing and memory diagrams) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…g), 0xCB bit 6 gates SET_SD_DIAG, GET_SD_INFO byte 29 bit 2 = legacy seek applied Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…yte, exFAT caveat) + §7 extended multi-hour campaign (patterns × Mode 2/3 × speeds); bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ine + bar at 286 Hz), soak_mode2_open_loop.yaml (sequential control), make-stress-patterns.js; soak driver accepts open-loop protocols (Studio v0.77); plan §7 trimmed, §5 extent-table next step, round-4 review state; bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ns, pass criteria, merge order); 2 h stress + overnight consolidated run in the plan; bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cess causes the stalls, same-index skip irrelevant); bench log 14:46–15:12 ET Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…(review gates, decisions, small changes, overnight run) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…io changes 2026-09-11 → 09-14 (problems, diagnosis path, layers, invariants, tests, evidence, open items); consolidation plan: Michael's three review criteria Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…(≈133 raw / 35–40 gz at 200 Hz), weak points (rx = drain time, a↔cc order join, hex), recommendations R1 stream-gzip → R2 schema v3 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n-log export (normal and fault runs), drain errors/refused rows/seq gaps → trial 'unknown', bridge send failure not acked, FRAME sd_load_us counted in verdicts (JS + report), link-drop classified as CONTROLLER_FAULT before the outcome is derived, stress-protocol gain 0.18, wire-comment fixes — Studio v0.78 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…its FRAME count once (25 checks) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…s in the consolidation plan and the hand-off document Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… + campaign path, index renumbering warning Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
….78, 16:12 switch plan Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…with a pre-overnight recovery drill), untested-today list, morning report Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ght test plan (metadata, schema, trial_quality ordering, cc vs a counts, drainer stats, FicTrac coverage, gap provenance) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…n the 8 MB sine (single event, no driver error) — the watch item Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ive/mode3-2026-09 (living set: hand-off, lab test day, overnight test plan, log-format review, bench log); ship plan §7.3 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…docs/development/archive/mode3-2026-09; links repointed Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… build, workload and result Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
changed the title
…hen no explicit rig is set (fresh profiles came up g6_3x10 on a 2×10 controller) — Studio v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
changed the title
…2×10 variant) — identities in the hand-off, lab plan, test plan; bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d on at connect (decision: keep the switch in production, banner it); decisions recorded in the hand-off — v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t recovery drill) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…9 connected (rig derived), drill started Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… idx 46); drill trial 180 s and run as a 1-iteration soak; lab plan + test plan updated Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…he SD listing — the closed-loop index modulus for never-previewed (large) patterns (#201) — v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…log completeness confirmed, legacy arm clean at fast-path cost (investigating) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…-coverage criterion; bench log Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o every sim log's ft was 1000× too small (found by the telemetry logging review) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…vious run's late terminal event no longer finalizes the next run's trials (bench 2026-09-13); runlog-check wording — v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d duplicated the stamp) — v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d), sim ft fix, retry Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… round-trip recipe (+ figures) One format authority for behavior_v2 run logs as built in Studio v0.79 / bridge 3.1 / fw candidate 488d5b9: every row (FicTrac, a, cc, cf 8|11, cs kinds 1–14) and event with producer citations and real example lines; timestamp merging (rx = browser drain time per block, a↔cc order join, seq rule, pair-by-order fit); the closed-loop latency recipe with the two non-derivable links; readers/tools; ranked Teensy-first instrumentation gaps. HTML companion carries the clock swimlane and latency-chain figures. Pointers added in CLAUDE.md, runlog-behavior-v2-plan.md, oscilloscope-view-spec.md; design-record header on the ring proposal. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… header, rx = browser parse time of the drain block, stream_schema cols offset; CLAUDE.md stall sentence updated to the causal result Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… no stall in this window Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ent precedes the disconnect listener (session already down) — v0.79 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ct without picker, two lit panels, outcome-classification fix Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…boot-blank fix note; lab plan test D expectation corrected Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…boot-blank retry build 781efe2 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…otocol, 10 h); drill results, panel observation, identities updated Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…stalls, req_age max 4.83 ms Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…0 stalls Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… by the Studio, soak continues Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #198 (Studio v0.76, base
claude/mode3-wedge-soak); merge #198 first, then this. Both merge cleanly ontomain(checked withgit merge-tree).pixi run test+format-checkpass. Bench-tested 2026-09-13 with firmwarefeat/sd-fastpath-2x10(3c71953on the CSHL 2×10 controller: 482k Mode-3 commands, 0 reads > 10 ms, 40/40 trials pass).What this adds — Studio v0.77
js/trial-quality.js): every trial is pass / flagged / unknown from the controller telemetry ring — any SD read or request→display age over 10 ms flags it; incomplete coverage isunknown, neverpass.display_gap+trial_qualityrun-log events, banner naming flagged trials. Flag only, never auto-exclude.GET_SD_INFO0xCD, gated on firmware flag bit 5): maker / product / serial / date / capacity / FAT type / cluster size →run_metadata.sd_card.req_age_us,superseded,flags(ring v2, 26 B FRAME; readers accept 8- or 11-fieldcfrows); STATE kinds 11–14 (sd_layout,sd_slow_ctx,sd_reads,sd_reads_ckpt); telemetry drain budget 40 → 200 chunks per poll.Studio.setSdDiag(flags)(SET_SD_DIAG0xCE, gated on flag bit 6) for the causal test of the card stalls; the arm is recorded inrun_metadata.sd_card.sd_diag.protocols/soak_mode3_stress.yaml(8 MB sine + 813 KB bar at 286 Hz),protocols/soak_mode2_open_loop.yaml(sequential control);scripts/make-stress-patterns.jsgenerates the patterns;scripts/telemetry-report.py(+41 tests) is the offline analysis;wedge-scan.pyknows the new kinds.Docs
docs/development/archive/mode3-2026-09/sd-read-jitter-2026-09-13.md— evidence and the H-FAT explanation (long; §8 has the diagrams).docs/development/archive/mode3-2026-09/sd-stall-causal-test-plan-2026-09-13.md— 4-arm causal test (~70 min) + the stress-first campaign.docs/development/telemetry-review-handoff-2026-09-13.md— brief for the telemetry review session.docs/development/mode3-wedge-soak-plan.md§10 — bench log.Firmware pairing
feat/sd-fastpath-2x10in reiserlab/LED-Display_G6_Firmware_Arena (2×10 bench branch, Codex-reviewed rounds 1–4). Everything here degrades gracefully on older firmware (all new opcodes gated onGET_FIRMWARE_VERSIONflag bits).Refs: #197, #200, #201; firmware #50, #54.
History and tests (added 2026-09-13 evening)
Read first:
docs/development/mode3-reliability-handoff-2026-09-14.md— the one document for the whole Mode-3 reliability arc (both repos): the two problems, the diagnosis path, every firmware layer and Studio change, the wire contract, §6.0 test history (every bench run with build, workload and result), the causal test table, known limits and the deferred review items. Bench log with ET timestamps:docs/development/mode3-wedge-soak-plan.md§10. What the overnight soak checks and how:docs/development/overnight-soak-test-plan-2026-09-13.md. Lab test day (Windows, no Claude):docs/development/lab-test-day-windows-2026-09-14.md. Working documents of the arc are archived underdocs/development/archive/mode3-2026-09/.Tests run on this code (summary): wedge soaks 2026-09-11/12 (fault lifecycle + post-mortem exercised on real wedges and watchdog self-resets); 24 soak iterations / 5.15 M commands on the free-running-timer firmware, 0 wedges; SD fast path: 482k commands / 40 of 40 trials pass, then the 4-arm causal test (FAT access proven to be the stall cause), 1 h at 286 Hz, and a 2.2 M-command 286 Hz campaign with the 8 MB pattern; whole-stack Codex reviews of firmware and Studio with fixes (Studio v0.78 = this PR's tip); overnight Studio soak on the final firmware to follow. Offline:
pixi run test+format-checkgreen on every commit.🤖 Generated with Claude Code