Skip to content

emit-portable: windowed relexing for newline-mode lexers (S4, #57)#65

Merged
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-nl-s4
Jul 11, 2026
Merged

emit-portable: windowed relexing for newline-mode lexers (S4, #57)#65
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-nl-s4

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Owner

Summary

Slice 4 of #57 (portable incremental edit). Extends S3's windowed relexing (#64) from stateless lexers to newline-mode lexers (e.g. envspec):

  • Verified state analysis: of the newline lexer's five cross-token state variables (pos, pendingNl, lineStart, emittedContent, flowDepth), the first four hold constant values at every token-emission point — the only one that varies is flowDepth. So each token's AlignMeta now records the flow depth after its emission (fd), and a windowed scan can restart from any token boundary with state {pos: end(anchor), pendingNl: false, lineStart: false, emittedContent: true, flowDepth: fd(anchor)}.
  • Resync additionally requires the old token's recorded fd to equal the live scan depth — an insertion of ( that shifts downstream flow depth correctly refuses to adopt the old suffix (gate scenario dedicated to this). Zero-width NEWLINE tokens sharing an offset with their successor are disambiguated by kind during resync lookup.
  • The streamEq self-check compares fd too, so an adopted-but-wrong depth cannot survive as a latent state bomb for the next edit.
  • Drive-by parity fix: the portable newline lexer used to set pendingNl = true on LF/CR inside flow suspension; the interpreter does not (non-indent newline mode). Removed — direct token-stream comparison against the oracle lexer over flow/newline-heavy inputs is now field-identical. (Latent in S3: CST output doesn't carry nl, and the old envspec edit scenario forced full relex.)
  • regexCtx/template grammars (javascript) keep the full-relex fallback; their lexer codegen is untouched.
  • Gate: envspec grows 1 → 8 edit scenarios (flow-interior edit, depth-shift insert, comment edit, blank-line fold, large mid/tail with relexed ≤ 10, two-batch depth-inheritance chain); 21 scenarios × 3 targets total.

Validation

  • node test/portable-targets.ts — calc 8/8 + javascript 5/5 + envspec 8/8, × 3 targets; all streamEq true
  • node test/check.ts emit-parity — 5/5; node test/newline-mode.ts — 29/29; npx tsc --noEmit — clean
  • Mutation test: disabling the fd resync check makes the depth-shift scenario fail (streamEq:false + align mismatch), reverted before commit
  • Independent adversarial probe (6 attacks × 3 targets): delete flow-close/open, nested depth-2 interior edit, resync at zero-width NEWLINE neighborhood, three-batch depth-inheritance chain, edit between two flow regions — all parity with oracle-derived expectations
  • Direct oracle token-stream parity over 7 flow/newline-heavy inputs (field-by-field)

Newline-only grammars extract lexFrom with flowDepth-aware AlignMeta, resync, and streamEq checks while regex/template grammars keep full fallback.

Co-authored-by: Cursor <cursoragent@cursor.com>
@johnsoncodehk johnsoncodehk merged commit 55c2f21 into master Jul 11, 2026
3 checks passed
@johnsoncodehk johnsoncodehk deleted the portable-relex-nl-s4 branch July 11, 2026 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant