Repository navigation
eval rounds 2 and 3, round 4 pre-registration; MO-1d, refresh under a whole-value guess, Solid repin ac28157a - #14
Merged
Conversation
Plain briefs (tasks/*/brief-plain.md), scaffold --brief plain (prompt and flights.js without the race hints, .prompt written), harness/compare.mjs (Fisher's exact test), and guide/async.md examples moved off the tasks' own code. Hidden tests unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
React 0/18 defects, .rx 1/18 (small model, tracker: two .rx traps). H1 not supported (Fisher p = 1.0). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
Run against round 2's finished apps and the references to choose round 3's direction. Not a pre-registered test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
Per-row guesses (optimistic xs[i].field := v) need a row's index from an id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
…w guesses hidden-overlap.mjs per task (6 tests), score.mjs --overlap, the guide's 'Writing one row of a list' with indexOf, the .rx prompt names indexOf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
H1 not supported: React 4/18, .rx 5/18 defects (p = 1.0); 4/18 vs 2/18 without solidjs/solid#4329 (p = 0.66). All four .rx failures on the frontier model are #4329. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
ryansolid
changed the title
Three change requests on round 3's frontier tracker apps, with plain briefs, visible tests, hidden tests (the base app's as regressions) and overlap tests. References in both arms: React passes everything; .rx passes everything but the list-reload cases (solidjs/solid#4329). scaffold --base copies an existing app into the run and gives the agent its own .scratch folder. Not pre-registered yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
A declaration in a nested block may not shadow a name an import bound. In the round 4 eval draft, every .rx base app has `source filter`, which hid the prelude's `filter`: a call to it passed rxc check and failed at run time. Both checkers agree (287/287 crate programs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
A source guessed as a whole (`optimistic list := …`) lowers to a createSignal truth with a createOptimistic layer over it. `refresh list` refreshed the layer, whose compute only reads the truth, so nothing was re-asked: the guess was corrected at the body's end and the accepted row vanished until something else re-asked. Refresh now re-asks the truth (rt-web, rt and the oracle). New S-solid-35 variant; compiled, oracle and ssr suites pass, and no round 3 score changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
… lingering - ref/SOLID_NEXT and the crate's solidjs-compiler rev move to ac28157a (rc.19 plus solidjs/solid#4330). All ref suites pass with zero divergences. - eval builds against that checkout's packages. Round 3's four #4329 failures pass on it, as its README now records. - Round 4 overlap tests: when two adds or removes on one list are out, a declined one may stay until the other settles (one change, H-11; Ryan 2026-10-28: desired). The one still out must not go. - .rx references for close and comments use the plain Solid pattern: a whole-list guess, then refresh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
Freezes the round 4 hidden tests (SHA-256 in eval/PREREG-change.md). The comments test that shows solidjs/solid#4336 is run and reported alongside, outside the defect count (score.mjs 'alongside'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
ryansolid
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
guide/async.md:rxc check.optimistic todos[i].done := v), not the whole list, so that two actions on different rows don't become one change (H-11).indexOf(xs, f)(Ryan, 2026-10-27): it is in the compiler's and the reference's built-in lists and in every runtime, and spec/11's table lists it.eval/round 2 (pre-registered ineval/PREREG-plain.md, commit 9c38a3f):tasks/*/brief-plain.md) drop every sentence about timing.scaffold.mjs --brief plain.harness/compare.mjsruns Fisher's exact test.eval/round 3, overlapping writes (pre-registered ineval/PREREG-overlap.md, commit 82176b6):tasks/*/hidden-overlap.mjstests two writes in flight, and a write racing a reload.score.mjs --overlap.eval/results/overlap-2026-10-27/): React 4/18 defects, .rx 5/18 (p = 1.0). H1 is not supported. All three .rx defects on the frontier model are [2.0 / next] createOptimisticStore: a per-key guess leaves the screen when a re-ask lands while its action is still pending (D-34) solidjs/solid#4329: a per-row guess is dropped when the list reloads mid-action. Not counting #4329, the result is React 4/18, .rx 2/18 (p = 0.66). On Solid next @ ac28157a (the #4329 fix) those runs pass that test.eval/probes/: exploratory probes on round 2's apps. They are what led to round 3.eval/PREREG-change.md, commit 4ba4c71): tasks rename, close, comments on round 3's tracker apps, with references;scaffold.mjs --base,harness/prompt-change.md(a per-run.scratchfolder);score.mjsreports the [2.0 / next] createOptimistic: a guess made over one question's answer stays on screen after the question changes (D-34) solidjs/solid#4336 test alongside, outside the count. Results will come in a separate PR.SHADOWED_IMPORT(taken by default): a block declaration may not shadow an import (compiler/src/check.rs,ref/src/check.ts, spec/09, spec/99, S-module-1).refreshof a source under a whole-value guess re-asks the source's question, not the guess layer (ref/compile/rt-web.ts,rt.ts,ref/oracle/lower.ts; S-solid-35 variant).ref/SOLID_NEXT,compiler/Cargo.toml,eval/package.json).The round 3 commit message says "all four" .rx frontier failures; the correct number is three, as the README says.
🤖 Generated with Claude Code
https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX