Skip to content

emit-portable: windowed relexing for template lexers (S5b, #57)#67

Merged
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-tpl-s5b
Jul 12, 2026
Merged

emit-portable: windowed relexing for template lexers (S5b, #57)#67
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-tpl-s5b

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Owner

Summary

Slice 5b of #57 (portable incremental edit). Completes windowed-relexing coverage by extending it to template-literal lexers and the combined regex-context + template case (templatejs, javascript):

  • The template lexer's only cross-token state is the interpolation brace-counter stack. Instead of recording counters, each token's AlignMeta records td — the stack depth after emission — and both restart anchors and resync points are restricted to td == 0 tokens, where the stack is provably empty. An edit inside a template therefore anchors before its head and relexes the whole literal, resyncing after the tail.
  • Resync additionally requires the old token's td == 0 and an empty live stack; adopted suffix tokens inherit their recorded td unchanged (a suffix may contain td > 0 tokens from a later template, exercised by a dedicated gate scenario).
  • javascript stacks this on top of the S5a regex-context restoration unchanged.
  • With this slice, all five lexer classes (stateless / newline / regex-context / template / regex+template) run the windowed path; only the hypothetical newline × (regex|template) combination keeps the full-relex fallback (no such grammar exists in the gate).
  • Gate: javascript loses fullRelex (+1 hole-edit scenario), templatejs gains 6 scenarios (hole edit, nested braces in hole, ${ deletion collapsing head/hole/tail into one token, middle-text edit proving deferred resync, large doc with relexed ≤ 15, suffix adoption across a second template).

Validation

  • node test/portable-targets.ts — 34 scenarios (calc 8, javascript 6, regexjs 6, templatejs 6, envspec 8) × 3 targets, all streamEq true; large-doc relexed 2–4
  • node test/check.ts emit-parity — 5/5; node test/newline-mode.ts — 29/29; npx tsc --noEmit — clean
  • Mutation tests (×2): dropping the td == 0 anchor walk-back fails the middle-edit scenario (anchor lands on a head, relexes with an empty stack, stream diverges); forcing adopted-suffix td to 0 fails the second-template scenario via the streamEq td comparison — both reverted before commit
  • Independent adversarial probe (5 attacks × 3 targets): nested template inside a hole, $ without { (stays literal), regex+template combined edit, tagged-template edit, two-batch chain re-anchoring on inherited tokens — all parity with oracle-derived expectations; edits that create unterminated templates fail identically to fresh parses on the windowed path

… S5b).

Co-authored-by: Cursor <cursoragent@cursor.com>
@johnsoncodehk johnsoncodehk merged commit 7f8a73f into master Jul 12, 2026
3 checks passed
@johnsoncodehk johnsoncodehk deleted the portable-relex-tpl-s5b branch July 12, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant