Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
3a356c4
eval: round 4 results (change an existing app)
claude Oct 28, 2026
820be37
eval: round 4 token split (docs vs work) and the forwarded message
claude Oct 28, 2026
0c26cf7
eval: round 4 README wording
claude Oct 28, 2026
02cc763
checker: STALE_WRITE_BACK (AC-1e, taken by default)
claude Oct 28, 2026
45f128d
eval: alongside test accepts a held selection change (solidjs/solid#4…
claude Oct 28, 2026
c917bc6
eval: .rx docs inline in the prompt (--inline-docs), one-page languag…
claude Oct 28, 2026
e1d8041
guide: array methods in the async page's examples
claude Oct 28, 2026
c9e5710
eval: issue desk base app (six tasks in one, tabs) with both referenc…
claude Oct 28, 2026
63f4df1
eval: round 5 draft: three change tasks on the desk, billed-cost scri…
claude Oct 28, 2026
1816643
eval: PREREG-cost draft: record model and effort; medians and API dol…
claude Oct 28, 2026
fbdc1aa
eval: Solid 2 TSX arm for round 5
claude Oct 28, 2026
dcabaae
eval: Solid arm checks with eslint-plugin-solid's v2 config, as React…
claude Oct 28, 2026
782d83e
Repin Solid next to 5e68925c (2.0.0-rc.20); freeze round 5 pre-regist…
claude Oct 29, 2026
5656a1f
eval: PREREG-cost: how each run's agent gets its prompt
claude Oct 29, 2026
cac1daa
eval: round 5 results (cost): .rx 30% above React, level with Solid; …
claude Oct 29, 2026
a7aa866
guide: effects section (async.md §6); eval: PREREG-cost-5b (frozen be…
claude Oct 29, 2026
b8aefe2
eval: PREREG-cost-5b: token count of the guide change
claude Oct 29, 2026
0c52c4c
eval: round 5b results: effects guide closes count's gap; prompt-cach…
claude Oct 29, 2026
5db7809
eval: Foldkit arm (round 6): harness, references for desk and the thr…
claude Oct 29, 2026
d56be31
eval: round 6 results: Foldkit costs most (64% over React), few turns…
claude Oct 29, 2026
4d6670f
eval: round 7 maintenance chains: row actions task, chain tasks (cumu…
claude Oct 30, 2026
72cabb9
eval: round 7 results: maintenance chains reverse (React 0/6 chains w…
claude Oct 30, 2026
60d00bd
eval: round 7: the createOptimisticStore error bug is solidjs/solid#4437
claude Oct 30, 2026
08228fd
eval: round 8 pre-registration: async timing fuzzer over the rounds 5…
claude Oct 30, 2026
cfb74e7
eval: round 8 addendum: two oracle fixes after a discarded partial run
claude Oct 30, 2026
1246c76
eval: round 8: A5 (held views) implemented as written; second run dis…
claude Oct 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion compiler/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion compiler/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ regex = "1"
serde_json = { version = "1", features = ["preserve_order"] }
# CP-1b: Solid's compiler, at the commit ref/ runs on (ref/SOLID_NEXT), and the Oxc release it is built against (CP-1c:
# the program crosses the seam, so the versions must match)
solidjs-compiler = { git = "https://github.com/solidjs/solid", rev = "ac28157aba6d03b806f0043bce70a87d515e953e", default-features = false }
solidjs-compiler = { git = "https://github.com/solidjs/solid", rev = "5e68925cc2fd216a9fb06345019e66d5da5c523c", default-features = false }
oxc_allocator = "0.144"
oxc_ast = "0.144"
oxc_span = "0.144"
Expand Down
3 changes: 2 additions & 1 deletion compiler/src/check.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2191,13 +2191,14 @@ impl<'a> Checker<'a> {
}

// ---- imperative scope (SC-2) ----
/// AC-1c, AC-1d over one imperative run (`crate::infer`).
/// AC-1c, AC-1d, AC-1e over one imperative run (`crate::infer`).
fn run_checks(&mut self, stmts: &[Stmt], action: bool, bound: &[String], inferred: bool) {
for f in run_findings(stmts, action, bound, inferred) {
let n = &f.name;
match f.code {
Code::HeldWrite => self.err("HELD_WRITE", "AC-1c", format!("{n} := … comes before an `await` in a body that is an action because it writes and awaits (AC-1b), so it is held until the body ends (AC-1): for state that shows while the work runs, `optimistic {n} := …` (a source with a question, ST-5c); for a reset that lands now, `onFlush(() => {{ {n} := … }})` (EF-8); to hold it on purpose, declare the action (`action name() {{ … }}`); or write it after the `await`")),
Code::StaleOwnRead => self.err("STALE_OWN_READ", "AC-1d", format!("{n} is read after this change wrote it: the read sees the committed value, not the write (W-4) — keep the value you wrote in a local (`let v = …; {n} := v`) and read that, or write functionally (`{n} := v => …`, W-6)")),
Code::StaleWriteBack => self.err("STALE_WRITE_BACK", "AC-1e", format!("{n} := … after an `await` is computed from {n} as it was read before that `await`: whatever changed {n} in between (another action, a reload) is overwritten — write functionally (`{n} := v => …`, W-6), which applies to {n} as it stands at the write, or guess it (`optimistic`, AC-3) and let the answer replace the guess")),
}
}
}
Expand Down
127 changes: 119 additions & 8 deletions compiler/src/infer.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
//! 06 AC-1b…AC-1d (`ref/src/infer.ts`): what the compiler reads off an imperative body. Which arrows are actions without
//! the keyword (AC-1b), and the two checks every imperative run gets: a plain write an inferred action holds across an
//! `await` (HELD_WRITE, AC-1c), and a read of a source the same change already wrote (STALE_OWN_READ, AC-1d). Solid sees
//! neither: a TypeScript body's promises and reads are not visible to it, which is why it asks for `action(function* …)`
//! and lets a stale read through.
//! 06 AC-1b…AC-1e (`ref/src/infer.ts`): what the compiler reads off an imperative body. Which arrows are actions without
//! the keyword (AC-1b), and the checks every imperative run gets: a plain write an inferred action holds across an
//! `await` (HELD_WRITE, AC-1c), a read of a source the same change already wrote (STALE_OWN_READ, AC-1d), and a write
//! after an `await` computed from the same source as read before it (STALE_WRITE_BACK, AC-1e). Solid sees none of them:
//! a TypeScript body's promises and reads are not visible to it, which is why it asks for `action(function* …)` and lets
//! a stale read or a lost update through.

use crate::ast::{Expr, ExprKind, Field, Lit, PathSeg, Stmt, StmtKind, TplPart};
use indexmap::{IndexMap, IndexSet};
Expand Down Expand Up @@ -135,6 +136,7 @@ pub fn inferred_action(stmts: &[Stmt]) -> Option<Trigger> {
pub enum Code {
HeldWrite,
StaleOwnRead,
StaleWriteBack,
}
pub struct Finding {
pub code: Code,
Expand All @@ -143,11 +145,17 @@ pub struct Finding {

/// keys from the source; "*" a computed key
type Path = Vec<String>;
/// a read of a source: its key, the path read, and how many `await`s the run had passed when it read (AC-1e)
type Dep = (String, Path, u32);
#[derive(Clone)]
struct State {
written: IndexMap<String, Vec<Path>>,
held: IndexSet<String>,
dead: bool,
/// AC-1e: the source reads each local's value was computed from
taint: IndexMap<String, Vec<Dep>>,
/// AC-1e: the `await`s passed so far on this path (the most of any path into a merge)
epoch: u32,
}
fn merge(ss: &[&State]) -> State {
let live: Vec<&&State> = ss.iter().filter(|s| !s.dead).collect();
Expand All @@ -156,8 +164,12 @@ fn merge(ss: &[&State]) -> State {
s.dead = true;
return s;
}
let mut out = State { written: IndexMap::new(), held: IndexSet::new(), dead: false };
let mut out = State { written: IndexMap::new(), held: IndexSet::new(), dead: false, taint: IndexMap::new(), epoch: 0 };
for s in live {
out.epoch = out.epoch.max(s.epoch);
for (k, v) in &s.taint {
out.taint.entry(k.clone()).or_default().extend(v.iter().cloned());
}
for (k, v) in &s.written {
out.written.entry(k.clone()).or_default().extend(v.iter().cloned());
}
Expand Down Expand Up @@ -248,6 +260,7 @@ impl Walker {
let after = self.walk(body, alive(st), &inner, local, &d);
st.written = after.written;
st.held = after.held;
st.epoch = after.epoch;
}
return;
}
Expand Down Expand Up @@ -296,6 +309,91 @@ impl Walker {
}
}
}
/// AC-1e: the source reads a value is computed from now — read directly (at this epoch) or through a local's taint.
/// A closure's body runs later; an awaited answer is the server's, not a snapshot of a source.
fn deps(&self, e: &Expr, st: &State, sh: &IndexSet<String>, out: &mut Vec<Dep>) {
let mut path: Path = vec![];
let mut x = e;
loop {
match &x.k {
ExprKind::Member { o, p, .. } => {
path.insert(0, p.clone());
x = o;
}
ExprKind::Index { o, i, .. } => {
path.insert(0, lit_key(i));
self.deps(i, st, sh, out);
x = o;
}
_ => break,
}
}
match &x.k {
ExprKind::Id(n) if sh.contains(n) => {
if let Some(t) = st.taint.get(n) {
out.extend(t.iter().cloned());
}
return;
}
ExprKind::Id(n) => {
out.push((key(n, false), path, st.epoch));
return;
}
ExprKind::Use(n) => {
out.push((key(n, true), path, st.epoch));
return;
}
_ => {}
}
if !std::ptr::eq(x, e) {
self.deps(x, st, sh, out);
return;
}
match &x.k {
ExprKind::Fn { .. } | ExprKind::ServerFn { .. } | ExprKind::Draft { .. } | ExprKind::Pending(_) | ExprKind::Latest(_) | ExprKind::Lit(_) | ExprKind::Pre { .. } | ExprKind::Peek(_) | ExprKind::StyleMap { .. } | ExprKind::Await { .. } => {}
ExprKind::Id(_) | ExprKind::Use(_) | ExprKind::Member { .. } | ExprKind::Index { .. } => {}
ExprKind::Call { f, args } => {
self.deps(f, st, sh, out);
for a in args {
self.deps(a, st, sh, out);
}
}
ExprKind::Bin { l, r, .. } => {
self.deps(l, st, sh, out);
self.deps(r, st, sh, out);
}
ExprKind::Un { e, .. } | ExprKind::Spread(e) | ExprKind::Throw(e) => self.deps(e, st, sh, out),
ExprKind::If { c, t, f } => {
self.deps(c, st, sh, out);
self.deps(t, st, sh, out);
self.deps(f, st, sh, out);
}
ExprKind::Tpl(parts) => {
for p in parts {
if let TplPart::Expr(e) = p {
self.deps(e, st, sh, out);
}
}
}
ExprKind::Arr(items) => {
for i in items {
self.deps(i, st, sh, out);
}
}
ExprKind::Rec(fields) => {
for f in fields {
match f {
Field::Kv { v, .. } => self.deps(v, st, sh, out),
Field::Spread(e) => self.deps(e, st, sh, out),
}
}
}
ExprKind::Catch { e, alt, .. } => {
self.deps(e, st, sh, out);
self.deps(alt, st, sh, out);
}
}
}
fn walk<'a>(&mut self, ss: &'a [Stmt], mut st: State, shadow: &IndexSet<String>, cl: &Closures<'a>, depth: &[*const Expr]) -> State {
let mut sh = shadow.clone();
let mut local = cl.clone();
Expand All @@ -306,7 +404,12 @@ impl Walker {
match &s.k {
StmtKind::Let { e, .. } | StmtKind::Expr(e) => {
self.read(e, &mut st, &sh, &local, depth);
let mut d = vec![];
if !stmt_await(s) {
self.deps(e, &st, &sh, &mut d);
}
if stmt_await(s) {
st.epoch += 1;
if self.action {
for h in std::mem::take(&mut st.held) {
self.report(Code::HeldWrite, h);
Expand All @@ -317,6 +420,7 @@ impl Walker {
}
if let StmtKind::Let { name, .. } = &s.k {
sh.insert(name.clone());
st.taint.insert(name.clone(), d);
match closure_of(s) {
Some(c) => {
local.insert(name.clone(), c);
Expand All @@ -338,6 +442,13 @@ impl Walker {
}
let k = key(&target.name, target.is_ctx());
let path: Path = target.path.iter().map(|p| match p { PathSeg::Name(n) => n.clone(), PathSeg::Expr(e) => lit_key(e) }).collect();
if matches!(s.k, StmtKind::Write { .. }) && !matches!(e.k, ExprKind::Fn { .. }) {
let mut d = vec![];
self.deps(e, &st, &sh, &mut d);
if d.iter().any(|(dk, dp, ep)| *dk == k && *ep < st.epoch && overlaps(dp, &path)) {
self.report(Code::StaleWriteBack, k.clone());
}
}
st.written.entry(k.clone()).or_default().push(path);
if matches!(s.k, StmtKind::Write { .. }) && self.inferred {
st.held.insert(k);
Expand Down Expand Up @@ -389,12 +500,12 @@ impl Walker {
}
}

/// AC-1c and AC-1d over one imperative run (`runFindings` in `ref/src/infer.ts`). `action`: the body is one change across
/// AC-1c, AC-1d and AC-1e over one imperative run (`runFindings` in `ref/src/infer.ts`). `action`: the body is one change across
/// its awaits (AC-1); a plain run's change ends at each `await`. `inferred`: an action by AC-1b, the only kind HELD_WRITE
/// applies to. `bound`: the run's parameters. Each finding once per name, in source order.
pub fn run_findings(stmts: &[Stmt], action: bool, bound: &[String], inferred: bool) -> Vec<Finding> {
let mut w = Walker { out: vec![], said: HashSet::new(), action, inferred };
let sh: IndexSet<String> = bound.iter().cloned().collect();
w.walk(stmts, State { written: IndexMap::new(), held: IndexSet::new(), dead: false }, &sh, &IndexMap::new(), &[]);
w.walk(stmts, State { written: IndexMap::new(), held: IndexSet::new(), dead: false, taint: IndexMap::new(), epoch: 0 }, &sh, &IndexMap::new(), &[]);
w.out
}
24 changes: 19 additions & 5 deletions compiler/src/parse.rs
Original file line number Diff line number Diff line change
Expand Up @@ -185,10 +185,17 @@ impl<'a> Cursor<'a> {
}

/// A TypeScript type, kept as text: balanced up to a depth-0 stop (`=`, `:=`, `,`, `)`, `=>`, `{` when `brace_stops`). 07 §2.
fn parse_type(c: &mut Cursor, brace_stops: bool) -> R<String> {
/// A type's text, up to where the declaration goes on. `=>` inside brackets is a function type, as it is anywhere in
/// a type alias (`arrows`); elsewhere at the top it starts a body (`fn f(): T => …`).
fn parse_type(c: &mut Cursor, brace_stops: bool, arrows: bool) -> R<String> {
let start = c.i;
let mut depth: i64 = 0;
while !c.eof() {
if c.starts_with("=>") && (depth > 0 || arrows) {
c.bump();
c.bump();
continue;
}
let ch = c.peek();
if depth == 0 && (c.starts_with("//") || c.starts_with("=>") || c.starts_with(":=") || ch == '=' || ch == ',' || ch == ')' || ch == ';' || ch == '\n' || (brace_stops && ch == '{')) {
break;
Expand Down Expand Up @@ -223,7 +230,7 @@ fn opt_type(c: &mut Cursor, brace_stops: bool) -> R<Option<String>> {
if c.peek() == ':' && c.peek_at(1) != '=' {
c.bump();
c.ws(true);
return Ok(Some(parse_type(c, brace_stops)?));
return Ok(Some(parse_type(c, brace_stops, false)?));
}
Ok(None)
}
Expand Down Expand Up @@ -377,7 +384,7 @@ pub fn parse_declarations(src: &str) -> R<Vec<JsDecl>> {
c.ws(true);
if c.eat("=") {
c.ws(true);
parse_type(&mut c, false)?;
parse_type(&mut c, false, true)?;
} else if c.starts_with("{") {
let mut depth = 0;
loop {
Expand Down Expand Up @@ -605,7 +612,7 @@ fn parse_decl_at(c: &mut Cursor) -> R<Decl> {
c.ws(true);
if c.eat("=") {
c.ws(true);
parse_type(c, false)?;
parse_type(c, false, true)?;
} else {
c.expect("{")?;
let mut depth = 1;
Expand Down Expand Up @@ -712,7 +719,7 @@ fn parse_decl_at(c: &mut Cursor) -> R<Decl> {
c.ws(true);
source = c.keyword("source");
c.ws(true);
ty = Some(parse_type(c, false)?);
ty = Some(parse_type(c, false, false)?);
}
let save = c.i;
c.ws(false);
Expand Down Expand Up @@ -1814,4 +1821,11 @@ mod recovering {
assert_eq!(errs.len(), 2);
assert!(parse_program(src).is_err());
}

#[test]
fn function_types_in_props_and_type_aliases() {
let src = "type Pick = (id: number) => void\ncomponent Row(props: { id: number; onPick: (id: number) => void; onDrop: Pick }) {\n source n: number := peek(props.id)\n <li onClick={() => { props.onPick(n) }}>{n}</li>\n}\nfn twice(x: number): number => x * 2\nexport default component App() {\n <Row id={twice(1)} onPick={id => { log(id) }} onDrop={id => { log(id) }} />\n}\n";
let p = parse_program(src).expect("parses");
assert_eq!(names(&p), ["Pick", "Row", "twice", "App"]);
}
}
64 changes: 64 additions & 0 deletions eval/PREREG-chain-7.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Round 7: maintenance chains

Pre-registration, fixed and committed before the round's first run. Builds on `PREREG-cost.md` (round 5),
`PREREG-cost-5b.md` and `PREREG-cost-6.md`; everything there holds unless this file changes it.

## Why

Rounds 1 to 6 gave each agent one change to a clean app. Frontier agents wrote the guards a race needs by hand (request
numbers, guess lists, a page id) and shipped few defects in any arm, so .rx's guarantee (a stale answer cannot
overwrite a newer one; a guess holds until its write is answered) did not show. Ryan (2026-10-29): "the agents work
around the foot guns anyway. So the benefit here isn't really being shown ... let's do the maintenance tests."

The claim tested here: hand-written guards decay when the app is changed again and again by agents who did not write
them, while in .rx there are no guards to decay. Each chain is four changes in a row, each made by a fresh agent to the
app the previous agent left, with the test suite growing as it would in a real project.

## Chains

- **Steps** (`tasks/chain1` … `chain4`, written by `harness/make-chain.mjs`): 1 labels, 2 open-issue count,
3 related help (the round 5 tasks, unchanged), 4 row actions (new: `tasks/rows`, assign and close from a list row,
with a row error; it touches the guards of the desk's assign and close and the count's).
- **Step k's run** starts from step k−1's run's `src/` (step 1 from `reference/desk/<arm>`), whatever state it was
left in. `BRIEF.md` is step k's change brief; `BASE.md` is the desk brief plus the earlier steps' change briefs, in
order. The visible tests are every step's so far (22, 24, 26, 30); the hidden and overlap tests are every step's so
far (58+26, 64+29, 70+32, 77+38), so an earlier step's hidden tests become regression tests.
- **Arms**: React and .rx, as in round 5b (React plain; .rx with the guide's two pages inline, `--inline-docs`). Same
prompt template, model and effort as rounds 5 to 6. Solid and Foldkit are not run.
- **6 chains per arm**, `chain<k>-<arm>-<n>`, n = 1..6, in `/home/claude/runs/r7`. 48 runs.
- **Batches**: each step's 12 runs (both arms) launch together, after every run of the step before has finished, so
both arms start each step in the same prompt-cache state.
- **References**: `reference/chain1..4/{react,rx}`, merged from the round 5 references plus row actions, pass every
visible, hidden and overlap test of every step (the .rx ones fail only the #4425 test, which stays alongside, as in
round 5).

## Measures

- **Defect at a step**: the run fails a visible, hidden or overlap test of its step (the #4425 test outside the count),
or does not build. Scored per run with `harness/score.mjs --overlap`.
- **New defect at a step**: a test that the previous step's app passed (or that is new at this step) fails. A test
already failing in the base is inherited, not new.
- **Regression**: a new defect in a test of an earlier step (the step's change broke something the agent was not
asked to touch).
- **Cost**: billed cost without each run's first call (`harness/cost-warm.mjs`), summed per chain.

## Hypotheses and tests

- **H7a** (primary): React chains end with a defect more often than .rx chains: chains whose step 4 app has a defect,
Fisher's exact test, one-sided, α = 0.05. With 6 chains per arm only a large gap can reach significance (e.g. 5 of 6
against 0 of 6: p = 0.008; 4 of 6 against 0 of 6: p = 0.03; 3 of 6 against 0 of 6: p = 0.09).
- **H7b** (secondary): React runs have more new defects than .rx runs over all 24 runs per arm: Fisher's exact test on
runs with at least one new defect, one-sided.
- **H7c** (secondary): React runs have more regressions: same test on runs with at least one regression.
- Reported alongside: cost per chain and per step (one-sided permutation tests both ways, 100,000 relabellings, seed
5); per step defects; which guards decayed (read from the failing tests and the diffs).

The direction is stated from the claim being tested, not from a prior that it holds: in rounds 2 to 5 React shipped
as few defects as .rx. A null or reversed result is reported as plainly.

## Limits

- Six chains per arm: power is low for anything but a large gap.
- The step order is fixed (labels, count, related, rows); row actions come last because they reuse the most guards.
- Inherited defects make later steps' bases differ between chains; the per-step "new defect" measure is there for that.
- Published as is, whatever the outcome.
Loading
Loading