From f605a4c3ca5f7b33c16d60e304f874cc01633843 Mon Sep 17 00:00:00 2001 From: Paul Hammant Date: Sat, 19 Sep 2026 16:10:02 +0100 Subject: [PATCH 1/3] asks: closure locals unified with outer-block names emit undeclared C (live on 0.696) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A closure's own locals are unified by name with same-named locals in an earlier, already-closed block of the enclosing function, promoted to heap cells, and then captured at a construction site outside the block where those cells were declared and released — so gcc rejects the generated C with "'idx' undeclared". Measured bisect, holding the downstream code byte-identical and varying only the compiler: clean on 0.668; broken on 0.675, 0.677, 0.681 and 0.696. The 0.696 row was taken by reverting only aeb's workaround rename, so it is the compiler being measured rather than the symptom. Currently invisible: aeb 6af17aa renames the colliding locals, which hides it. That is why this is worth filing anyway — the next person to reuse a name in a long function gets an undeclared-C wall in generated code they never see, naming a variable that does not exist at the reported line, with no clue it is a known compiler bug. The blast radius is the whole graph, because aeb's fan-out orchestrator is one binary for every node. Includes the generated-C evidence (cells declared inside the earlier block, released when it closes, then retained at the later construction site), and the five reductions that did NOT reproduce — nested if, tuple destructuring, interpolation over captures, closure in a while loop, and the same-name-in- earlier-block shape in a small function — so nobody re-derives that the shape alone is insufficient. The hypothesis that function size tips an inlining or scope-flattening decision is labelled as a hypothesis. Found from servirtium-vcr on CachyOS; the aeb-side workaround and the presubmit blast-radius detail came from the selenium side. Companion ask that documents the aeb-level workaround rather than the compiler defect: aeb/asks/closure-var-collides-with-function-body-name.md — both are worth keeping, they answer different questions. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_013vNUNU6AmTr5SW3hByb2QY --- ...ith-outer-block-names-emit-undeclared-c.md | 117 ++++++++++++++++++ 1 file changed, 117 insertions(+) create mode 100644 asks/closure-locals-unified-with-outer-block-names-emit-undeclared-c.md diff --git a/asks/closure-locals-unified-with-outer-block-names-emit-undeclared-c.md b/asks/closure-locals-unified-with-outer-block-names-emit-undeclared-c.md new file mode 100644 index 00000000..160da8bd --- /dev/null +++ b/asks/closure-locals-unified-with-outer-block-names-emit-undeclared-c.md @@ -0,0 +1,117 @@ +# A closure's own locals are unified with same-named locals in an earlier closed block, and emitted as captures outside their declaring block + +> **STATUS: live on ae 0.696.0.** Worked around downstream in aeb +> (`6af17aa`, renames the colliding locals), so the symptom is currently hidden +> — but the compiler bug is unfixed and will bite the next person who reuses a +> name. Verified still-broken on 0.696 by reverting only the rename. +> +> Companion ask on the aeb side: +> `aeb/asks/closure-var-collides-with-function-body-name.md`. That one describes +> the trigger as "the transform/inline path"; the mechanism below supersedes it. + +## Symptom + +gcc rejects the generated C for `aeb`'s dotnet SDK: + +``` +lib/dotnet/module.ae: In function 'dotnet_build_project': +lib/dotnet/module.ae:764:126: error: 'idx' undeclared (first use in this function) +lib/dotnet/module.ae:764:290: error: 'entry' undeclared (first use in this function) +aeb-link: FATAL — failed to link the fan-out orchestrator +``` + +**Blast radius is the whole graph, not one node.** aeb's fan-out orchestrator is +a single binary for every node, so nothing in any graph containing the offending +module can run — a full presubmit dies before executing a single test. + +Note the reported location is misleading: line 764 is +`if string.length(line_r) > 0 {`, which has no column 290. The `#line` mapping +back to `.ae` is off, so chasing that line wastes time. Chase the generated C. + +## Bisect + +One fixed aeb (dotnet SDK byte-identical across v0.311–v0.319), varying only the +compiler: + +| ae | result | +|----|--------| +| 0.668.0 | clean | +| 0.675.0 | **`idx`/`entry` undeclared** | +| 0.677.0 | **undeclared** | +| 0.681.0 | **undeclared** | +| 0.696.0 | **undeclared** (re-confirmed 2026-09-19 by reverting only the rename) | + +So it landed in **0.675** and is still present in **0.696**. + +## Mechanism, from the generated C + +In `dotnet_build_project`, the names `idx`/`entry` are assigned in **two** places: +once in an earlier nested block (a pkgrefs loop, `.ae` ~724) and once as the +*own locals* of a later closure passed to `string.seq_each` (`.ae` ~766). + +The emitted C for that one function: + +```c +/* ~12739 — inside the EARLIER block */ +int* idx = (int*)_aether_cell_new(sizeof(int)); +const char** entry = (const char**)_aether_cell_new(sizeof(const char*)); +... +/* ~12775 — that block ENDS; the cells are released */ +_aether_cell_release_str(entry); +_aether_cell_release(idx); +... +/* ~12827 — closure-construction site, OUTSIDE that block */ +_e->idx = (int*)_aether_cell_retain(idx); /* 'idx' undeclared here */ +_e->entry = (const char**)_aether_cell_retain(entry); +``` + +So codegen: + +1. unifies the closure's own locals with the enclosing function's same-named + locals purely by name, +2. concludes the pair is captured-and-mutated and promotes the **earlier** ones + to heap cells, +3. emits the capture at a construction site that sits **outside the C block + where those cells were declared and released**. + +The closure's locals are not captures at all — they are declared in the closure +body. A name assigned inside a closure should not be unified with a same-named +local of the enclosing function, least of all one in an already-closed block. + +## What did NOT reproduce it (so you don't redo this) + +Five reduced cases all compile clean on 0.677 and 0.696, so the shape alone is +not sufficient: + +1. a closure local declared inside a nested `if`; +2. the same plus tuple-destructuring assignment (`a, b = f(...)`) in the closure; +3. the same with string interpolation over captured vars; +4. the closure inside a `while` loop; +5. the same name in an earlier closed block **and** in the closure — the shape + described above, in a small function. + +The trigger needs something the small cases lack — plausibly the size of +`dotnet_build_project` (~200 lines, several closures) tipping an inlining or +scope-flattening decision. `aeb/lib/dotnet/module.ae:698-893` at the pre-`6af17aa` +revision is the reliable reproducer. + +## Confirming the diagnosis + +Renaming *only* the closure's own locals (`idx`→`vr_idx`, `entry`→`vr_entry`), +changing nothing else, makes it compile and pass. That is what shipped in aeb +`6af17aa`. It confirms name-unification as the mechanism, and is the reason the +symptom is currently invisible. + +## Why it is worth fixing rather than leaving worked around + +Two unrelated locals sharing a name in one long function is ordinary code, not a +smell anyone would flag in review. The failure is a C compile error naming a +variable that does not exist at the reported line, in generated code the author +never sees — with a whole-graph blast radius. The next occurrence will cost +someone the same day it cost here. + +--- + +Reported from `servirtium-vcr` on CachyOS (session sv-co), with the bisect and +generated-C evidence; the aeb-side workaround and the presubmit blast-radius +detail came from the selenium side (session se-co). From 055fcc7da2dcfcec102ebd905e534c93207ecb35 Mon Sep 17 00:00:00 2001 From: Paul Hammant Date: Sat, 19 Sep 2026 16:22:21 +0100 Subject: [PATCH 2/3] fix(codegen): a closure own-local shadowing a promoted capture emits valid C A closure's own body local that shared a name with a PROMOTED capture of the enclosing function was misclassified as that promoted capture, emitting undeclared C. When an enclosing function had a local captured-and-mutated by one closure (so codegen promotes it to a heap cell), a LATER closure whose own body declared a same-named local had that local treated as the promoted capture: the body emitted a cell-dereferencing write (*idx = ...) and mark_var_declared'd the name, but the closure never captured it and no prologue alias (T* idx = _env->idx;) existed -- so gcc rejected the generated C with "'idx' undeclared", in code the author never sees, with a whole-graph blast radius for a fan-out build. Landed in 0.675, live through 0.696; worked around downstream in aeb by renaming the colliding locals. Root cause in emit_closure_definitions (compiler/codegen/codegen_expr.c): the closure's body_promoted set was built from ALL parent_promoted names (filtered only by is_closure_param), and every parent_promoted name was mark_var_declared'd -- but the declaration for a promoted capture is the prologue alias, which is emitted ONLY for names in captures[]. A parent-promoted name the closure does not capture therefore had no alias. Fix: gate both the body_promoted inheritance and the mark_var_declared loop on captures[] membership -- a parent-promoted name is inherited only if genuinely captured; an uncaptured same-named name is a shadowing own local and declares normally. Verified: the minimal reproducer (earlier block's idx/entry captured+mutated -> promoted; later closure reuses the names as own locals) failed with 'idx'/'entry' undeclared before, builds+runs total=13 after. 94/94 closure/capture regression+syntax tests pass, closure-heavy tinyweb examples build clean, full make test green (410 passed). New regression test fails on the pre-fix compiler and passes after. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 16 +++++ compiler/codegen/codegen_expr.c | 32 +++++++++- ..._closure_local_shadows_promoted_capture.ae | 63 +++++++++++++++++++ 3 files changed, 109 insertions(+), 2 deletions(-) create mode 100644 tests/regression/test_closure_local_shadows_promoted_capture.ae diff --git a/CHANGELOG.md b/CHANGELOG.md index 3aa68efb..cd9cff36 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,22 @@ version number before tagging the release. ## [current] +### Fixed + +- **A closure's own body local sharing a name with a promoted capture of the + enclosing function emitted undeclared C.** When an enclosing function had a + local that one closure captured-and-mutated — so codegen promoted it to a heap + cell — a *later* closure whose own body declared a local of the **same name** + had that local misclassified as the promoted capture: the body emitted a + cell-dereferencing write (`*idx = ...`) and marked the name pre-declared, but + the closure never captured it and no prologue alias existed, so gcc rejected + the generated C with `'' undeclared` (in code the author never sees, + with a whole-graph blast radius for a fan-out build). A parent-promoted name is + now inherited into a closure's promoted-capture set only if the closure + actually captures it; an uncaptured same-named name is a shadowing own local + and declares normally. Landed in 0.675; verified fixed against a minimal + reproducer. `tests/regression/test_closure_local_shadows_promoted_capture.ae`. + ## [0.696.0] ### Added diff --git a/compiler/codegen/codegen_expr.c b/compiler/codegen/codegen_expr.c index ad1483fc..d1ba411a 100644 --- a/compiler/codegen/codegen_expr.c +++ b/compiler/codegen/codegen_expr.c @@ -2104,6 +2104,23 @@ void emit_closure_definitions(CodeGenerator* gen) { for (int p = 0; p < parent_promoted_count; p++) { if (!parent_promoted[p]) continue; if (is_closure_param(closure, parent_promoted[p])) continue; + // A parent-promoted name is a promoted capture of THIS + // closure only if the closure actually captures it — i.e. + // it has the `T* name = _env->name;` prologue alias emitted + // above. A parent-promoted name the closure does NOT capture + // is either unused here or SHADOWED by a same-named local of + // this closure's own body; inheriting it would (a) put a + // dereferencing `*name` promoted-write on that own local and + // (b) mark it pre-declared, so the local is never minted and + // the emitted C references an undeclared name. Exclude it so + // the own local declares normally. + int captured = 0; + for (int c = 0; c < cap_count; c++) { + if (captures[c] && strcmp(captures[c], parent_promoted[p]) == 0) { + captured = 1; break; + } + } + if (!captured) continue; body_promoted[body_promoted_count++] = parent_promoted[p]; } for (int p = 0; p < own_promoted_count; p++) { @@ -2123,9 +2140,20 @@ void emit_closure_definitions(CodeGenerator* gen) { // Mark promoted captures as already-declared in this local scope // so writes in the body hit the reassignment branch (emits // *name = ...) rather than trying to declare+malloc again. - // The prologue alias `T* name = _env->name;` is the declaration. + // The prologue alias `T* name = _env->name;` is the declaration — + // so only names this closure actually captures are pre-declared. + // A parent-promoted name the closure does NOT capture has no alias + // and may be shadowed by a same-named own local, which must declare + // normally (see the body_promoted filter above). for (int p = 0; p < parent_promoted_count; p++) { - if (parent_promoted[p]) mark_var_declared(gen, parent_promoted[p]); + if (!parent_promoted[p]) continue; + int captured = 0; + for (int c = 0; c < cap_count; c++) { + if (captures[c] && strcmp(captures[c], parent_promoted[p]) == 0) { + captured = 1; break; + } + } + if (captured) mark_var_declared(gen, parent_promoted[p]); } /* A closure body is its own C function — it needs the same * heap-string lifecycle as a top-level function, or heap diff --git a/tests/regression/test_closure_local_shadows_promoted_capture.ae b/tests/regression/test_closure_local_shadows_promoted_capture.ae new file mode 100644 index 00000000..1f3abc16 --- /dev/null +++ b/tests/regression/test_closure_local_shadows_promoted_capture.ae @@ -0,0 +1,63 @@ +// Regression: a closure's OWN body local must not be unified with a +// same-named PROMOTED capture of the enclosing function. +// +// closure-locals-unified-with-outer-block-names-emit-undeclared-c.md +// (live on 0.696, landed 0.675). When an enclosing function has a local +// (here `idx`/`entry`) that is captured-and-mutated by one closure — so +// codegen promotes it to a heap cell — a LATER closure whose own body +// declares locals of the SAME NAME had those locals wrongly treated as +// promoted captures. The closure body emitted `*idx = ...` / +// `_aether_str_cell_set(entry, ...)` (dereferencing a cell) and marked +// the name pre-declared, but the closure never captured it and no +// prologue alias `T* idx = _env->idx;` existed — so the generated C +// referenced an undeclared `idx`/`entry` (a gcc hard error in code the +// author never sees). The fix: a parent-promoted name is inherited into a +// closure's promoted set only if the closure actually captures it; an +// uncaptured same-named name is a shadowing own local and declares +// normally. +// +// Compiling IS the assertion (the bug was a C-compile failure); the value +// check guards that both the promoted-capture path and the own-local path +// still compute correctly. + +import std.string +import std.collections + +build(items: ptr) -> int { + total = 0 + + // Earlier block: idx/entry are captured AND mutated by `bump`, so the + // compiler promotes them to heap cells. bump: idx 0 -> 1, returns 1. + if collections.list_size(items) > 0 { + idx = 0 + entry = "seed" + bump = || { + idx = idx + 1 + entry = string.concat(entry, "!") + return idx + } + total = total + bump() + } + + // Later closure: `idx`/`entry` here are its OWN locals, NOT captures of + // the promoted names above. each: 7 + length("local")=5 => 12. + each = | it: string | { + idx = 7 + entry = "local" + return idx + string.length(entry) + } + total = total + each("x") + return total // 1 + 12 = 13 +} + +main() { + items = collections.list_new() + _a = collections.list_add(items, "a") + got = build(items) + println("got=${got}") + if got != 13 { + println("FAIL: expected 13, got ${got}") + exit(1) + } + println("PASS: closure own-local shadows promoted capture") +} From 33dee6d47607fb7b661c8844ebde06df47fecb95 Mon Sep 17 00:00:00 2001 From: Paul Hammant Date: Sat, 19 Sep 2026 16:51:14 +0100 Subject: [PATCH 3/3] test(closure): make the regression leak-minimal + record the inherent cell leak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The macOS ARM64 leaks gate failed the new regression test (4 leaks, allowed 0): it allocated a std.collections list (never freed) and a string_concat in the promoted path. Rewrote it to the minimal bug shape — an int gate instead of a list, no string_concat — which drops it to a single 20-byte leak: the promoted heap cell for the earlier block's captured-and-mutated local. That cell is INHERENT to capture promotion (a bump-only program with no name collision leaks the identical cell, independent of the shadowing bug this test guards) and is reclaimed a generation late by promotion's RCU discipline. Recorded it in tests/leaks_known.txt with an explaining comment, at 1, mirroring how the sibling test_closure_local_alloc_capture (20) is handled. Value assertion updated (got=19: bump 1+len("bumped")=7, each 7+len("local")=12). Still a genuine guard — fails on the pre-fix compiler with 'idx'/'entry' undeclared. Co-Authored-By: Claude Opus 4.8 --- tests/leaks_known.txt | 10 ++++ ..._closure_local_shadows_promoted_capture.ae | 48 +++++++++---------- 2 files changed, 33 insertions(+), 25 deletions(-) diff --git a/tests/leaks_known.txt b/tests/leaks_known.txt index 40f635dd..a61633c2 100644 --- a/tests/leaks_known.txt +++ b/tests/leaks_known.txt @@ -35,3 +35,13 @@ test_rsa_pkcs1 130 # transient-extern annotation to opt in). Bounded (does not scale with element # count), and unrelated to the value-correctness this test asserts. test_closure_local_alloc_capture 20 + +# test_closure_local_shadows_promoted_capture: the one residual leak is the +# promoted heap cell for the earlier block's captured-and-mutated local (a +# `bump` closure captures it, forcing capture promotion). That 20-byte cell is +# reclaimed a generation late by capture-promotion's RCU discipline and is +# INHERENT to any promoted-capture program — a bump-only program with no name +# collision leaks the identical cell. It is independent of the shadowing bug +# this test guards (a C-compile failure, fixed in codegen), which allocates +# nothing itself. macOS-only gate (Linux doesn't leak-check this). +test_closure_local_shadows_promoted_capture 1 diff --git a/tests/regression/test_closure_local_shadows_promoted_capture.ae b/tests/regression/test_closure_local_shadows_promoted_capture.ae index 1f3abc16..66a3871f 100644 --- a/tests/regression/test_closure_local_shadows_promoted_capture.ae +++ b/tests/regression/test_closure_local_shadows_promoted_capture.ae @@ -3,38 +3,38 @@ // // closure-locals-unified-with-outer-block-names-emit-undeclared-c.md // (live on 0.696, landed 0.675). When an enclosing function has a local -// (here `idx`/`entry`) that is captured-and-mutated by one closure — so -// codegen promotes it to a heap cell — a LATER closure whose own body -// declares locals of the SAME NAME had those locals wrongly treated as -// promoted captures. The closure body emitted `*idx = ...` / -// `_aether_str_cell_set(entry, ...)` (dereferencing a cell) and marked -// the name pre-declared, but the closure never captured it and no -// prologue alias `T* idx = _env->idx;` existed — so the generated C -// referenced an undeclared `idx`/`entry` (a gcc hard error in code the -// author never sees). The fix: a parent-promoted name is inherited into a -// closure's promoted set only if the closure actually captures it; an -// uncaptured same-named name is a shadowing own local and declares -// normally. +// (here `idx`/`entry`) captured-and-mutated by one closure — so codegen +// promotes it to a heap cell — a LATER closure whose own body declares +// locals of the SAME NAME had those locals wrongly treated as promoted +// captures. The closure body emitted a cell-dereferencing write +// (`*idx = ...`) and marked the name pre-declared, but the closure never +// captured it and no prologue alias `T* idx = _env->idx;` existed — so the +// generated C referenced an undeclared `idx`/`entry` (a gcc hard error in +// code the author never sees). The fix: a parent-promoted name is inherited +// into a closure's promoted set only if the closure actually captures it; an +// uncaptured same-named name is a shadowing own local and declares normally. // // Compiling IS the assertion (the bug was a C-compile failure); the value // check guards that both the promoted-capture path and the own-local path -// still compute correctly. +// still compute correctly. No heap collections/strings are allocated so the +// only leak is the promoted cell itself (inherent to capture promotion, +// independent of this bug — see tests/leaks_known.txt). import std.string -import std.collections -build(items: ptr) -> int { +build(n: int) -> int { total = 0 // Earlier block: idx/entry are captured AND mutated by `bump`, so the - // compiler promotes them to heap cells. bump: idx 0 -> 1, returns 1. - if collections.list_size(items) > 0 { + // compiler promotes them to heap cells. bump: idx 0 -> 1, len("bumped")=6 + // => returns 7. + if n > 0 { idx = 0 entry = "seed" bump = || { idx = idx + 1 - entry = string.concat(entry, "!") - return idx + entry = "bumped" + return idx + string.length(entry) } total = total + bump() } @@ -47,16 +47,14 @@ build(items: ptr) -> int { return idx + string.length(entry) } total = total + each("x") - return total // 1 + 12 = 13 + return total // 7 + 12 = 19 } main() { - items = collections.list_new() - _a = collections.list_add(items, "a") - got = build(items) + got = build(1) println("got=${got}") - if got != 13 { - println("FAIL: expected 13, got ${got}") + if got != 19 { + println("FAIL: expected 19, got ${got}") exit(1) } println("PASS: closure own-local shadows promoted capture")