Skip to content

eval round 4 results; STALE_WRITE_BACK check (AC-1e) for the lost update it found - #18

Open
ryansolid wants to merge 25 commits into
mainfrom
claude/project-thread-t6465s
Open

eval round 4 results; STALE_WRITE_BACK check (AC-1e) for the lost update it found#18
ryansolid wants to merge 25 commits into
mainfrom
claude/project-thread-t6465s

Conversation

@ryansolid

@ryansolid ryansolid commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What this adds

Round 4 results, in eval/results/change-2026-10-28/, as pre-registered in eval/PREREG-change.md (commit 4ba4c71):

  • 36 runs: rename, close and comments added to round 3's tracker apps; 2 arms × frontier + Haiku × 3.
  • Result: React 0/18 defects, .rx 3/18 (p = 0.23). H1 is not supported. Frontier 0/9 vs 0/9; small 0/9 vs 3/9.
  • Two of the three .rx defects are one pattern: a value read before an await and written back after it (a lost update). The third is a guess on detail with no refresh detail.
  • Reported alongside, outside the count: the [2.0 / next] createOptimistic: a guess made over one question's answer stays on screen after the question changes (D-34) solidjs/solid#4336 test failed in 4 of 6 .rx comments runs and 0 of 6 React ones.
  • Token split: measured above the shared ~37k start, the frontier model's non-docs work is 11.1k (.rx) vs 10.8k (React); the rest of the gap is reading the docs.
  • Known limits found: run agents could reach the project thread's reply tools (two posted, none read), and an owner message reached every run's context.

STALE_WRITE_BACK (06 AC-1e, taken by default 2026-10-28), a new check in both checkers:

  • It rejects a plain write after an await whose value is computed from the same source as read before that await. The fix is the functional write (x := v => …, W-6) or a guess.
  • Spec text is in 06, 07 and 99, with scenario S-infer-6.
  • It fires on 3 of the 76 .rx apps the eval's agents wrote and on none of the repo's examples. All 3 are real bugs: the 2 round 4 lost-update defects, and a latent one in frontier rename-rx-3 that passed every test (confirmed by a probe).
  • A rule for the guess-without-refresh defect was drafted and dropped, because it would reject correct programs.
  • ref suites: test, compiled, oracle 105/105; ssr 98 + 7 skipped; crate 309/309; types 145/145.

🤖 Generated with Claude Code

https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX

claude added 4 commits October 28, 2026 06:07
@claude
eval: round 4 results (change an existing app)
3a356c4 
React 0/18 defects, .rx 3/18 (p = 0.23, H1 not supported). All three .rx
defects are small-model runs; two are one pattern, a read before an await
written back after it. The solidjs/solid#4336 test is reported alongside.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 4 token split (docs vs work) and the forwarded message
820be37 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 4 README wording
0c26cf7 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
checker: STALE_WRITE_BACK (AC-1e, taken by default)
02cc763 
A plain write after an await computed from the same source as read
before it is a lost update: whatever changed the source while the run
waited is erased. Both checkers, spec 06/07/99, scenario S-infer-6.
On every .rx app the eval's agents wrote it fires on 3, all real bugs:
the 2 lost-update defects of round 4 and a latent one in a run that
passed every test (noted in the round 4 write-up).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@ryansolid ryansolid changed the title eval round 4 results: changing an existing app eval round 4 results; STALE_WRITE_BACK check (AC-1e) for the lost update it found Oct 28, 2026
claude added 8 commits October 28, 2026 08:11
@claude
eval: alongside test accepts a held selection change (solidjs/solid#4337
45f128d 
); guide: pending items beside selection-dependent data

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: .rx docs inline in the prompt (--inline-docs), one-page languag…
c917bc6 
…e guide, array methods over the prelude

guide/language.md is the working language on one page. With --inline-docs it and
guide/async.md go at the end of the .rx prompt, so reading them costs no tool calls,
and the prompt no longer lists the prelude: its filter collided with the bases'
source filter (6 of 9 failed checks in round 4's frontier .rx runs). The guide's
row lookup is todos.findIndex. Both prompts now say to use no messaging tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
guide: array methods in the async page's examples
e1d8041 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: issue desk base app (six tasks in one, tabs) with both referenc…
c9e5710 
…es; function types in type positions

- eval/tasks/desk: brief, API, 19 visible + 50 hidden + 21 overlap tests (the six tasks' tests, each on its page,
  and the tabs' own). The React reference passes all; the .rx reference all but one, solidjs/solid#4379 on next.
- parser (ref + crate): `=>` inside brackets is a function type, and anywhere in a type alias, so
  `props: { onPick: (id: number) => void }` parses. Found writing the desk.
- guide/language.md: handler props.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 5 draft: three change tasks on the desk, billed-cost scri…
63f4df1 
…pt, pre-registration draft

- tasks labels (an issue's labels), related (related help opening the help page) and count (open-issue count on the
  tab, through closes), each with visible, hidden and overlap tests over the desk's; references in both arms pass
  all (the .rx ones all but solidjs/solid#4379's test).
- harness/cost.mjs: billed cost per run from the transcripts, one-sided permutation test.
- PREREG-cost.md: draft; frozen once the #4379 fix is pinned.
- desk .rx help page: the searching probe moves inside the results' loading boundary (on a first load a probe reads
  like a read, R-9, so outside any boundary it held the page switch when the page opened mid-search).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: PREREG-cost draft: record model and effort; medians and API dol…
1816643 
…lars alongside

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: Solid 2 TSX arm for round 5
fbdc1aa 
- harness: arm `solid` builds src/App.tsx with @solidjs/compiler and mounts it with @solidjs/web's render; checks
  with tsc strict (no Solid 2 lint config on npm yet); --inline-docs puts Solid's own agent primer
  (solid-js/skills/solid-2.0/SKILL.md) at the end of the prompt, the cheatsheet readable in node_modules.
- references: Solid desk (419 lines; .rx 315, React 570) and labels/related/count, written to the .rx bases' design;
  all pass every test but the solidjs/solid#4379 one, the same test the .rx references fail.
- cost.mjs and score.mjs read the third arm; PREREG-cost.md: 54 runs, H2 = .rx cheaper than Solid TSX (secondary).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: Solid arm checks with eslint-plugin-solid's v2 config, as React…
dcabaae 
…'s does with react-hooks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@devagrawal09 devagrawal09 mentioned this pull request Oct 29, 2026
Draft
claude added 13 commits October 29, 2026 20:12
@claude
Repin Solid next to 5e68925c (2.0.0-rc.20); freeze round 5 pre-regist…
782d83e 
…ration

- ref/SOLID_NEXT and the crate's solidjs-compiler rev at rc.20: ref, oracle and compiled 105/105, ssr 98 (+7 skipped),
  types 145, crate 309, cargo tests pass. solidjs/solid#4379 is fixed there: the desk passes 90/90 in all three arms.
- eval/PREREG-cost.md frozen: pin, hidden test hashes, and the rule that runtime bugs are not waited on.
  solidjs/solid#4425 (new on rc.20) fails one related-task test in the .rx and Solid references; score.mjs keeps it
  outside round 5's defect count.
- The round-4 pilot rescored on rc.20: 0 of 9 defects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: PREREG-cost: how each run's agent gets its prompt
5656a1f 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 5 results (cost): .rx 30% above React, level with Solid; …
cac1daa 
…no defects in any arm

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
guide: effects section (async.md §6); eval: PREREG-cost-5b (frozen be…
a7aa866 
…fore runs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: PREREG-cost-5b: token count of the guide change
b8aefe2 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 5b results: effects guide closes count's gap; prompt-cach…
0c52c4c 
…e artifact found; labels gap is H-11

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: Foldkit arm (round 6): harness, references for desk and the thr…
5db7809 
…ee tasks (all tests pass), PREREG-cost-6

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 6 results: Foldkit costs most (64% over React), few turns…
d56be31 
… but most code; 2/18 defects

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 7 maintenance chains: row actions task, chain tasks (cumu…
4d6670f 
…lative briefs and tests), chain references (all pass), PREREG-chain-7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 7 results: maintenance chains reverse (React 0/6 chains w…
72cabb9 
…ith a defect, .rx 4/6, all at row actions: H-11 workarounds + createOptimisticStore error bug)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 7: the createOptimisticStore error bug is solidjs/solid#4437
60d00bd 


Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 8 pre-registration: async timing fuzzer over the rounds 5…
08228fd 
…-7 apps (harness/fuzz.mjs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
@claude
eval: round 8 addendum: two oracle fixes after a discarded partial run
cfb74e7 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ryansolid @claude