Skip to content

emit-portable: windowed relexing for regex-context lexers (S5a, #57)#66

Merged
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-rx-s5a
Jul 12, 2026
Merged

emit-portable: windowed relexing for regex-context lexers (S5a, #57)#66
johnsoncodehk merged 1 commit into
masterfrom
portable-relex-rx-s5a

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Owner

Summary

Slice 5a of #57 (portable incremental edit). Extends windowed relexing (#64, #65) to lexers with regex-vs-division context (regexCtx, no templates — e.g. regexjs):

  • The regex-context lexer carries eight cross-token state variables. Restoration strategy per class: prevText/prevKind/bpText/hasPrev/hasPrev2 are rebuilt directly from the token stream at the restart anchor; lastClose/lastBang are recorded per token (their post-emission values) and read back; the parenHead boolean stack is reconstructed backward from per-token paren depth (pd) plus a head bit (hd) recorded on ( tokens — porting the reconstructParens argument from the JS incremental lexer.
  • Resync tightens accordingly: besides the S3 token-identity conditions, the old token must match the live scan on paren depth, stack contents (element-wise), lastClose/lastBang, and previous-token text. Any mismatch keeps scanning instead of adopting the old suffix.
  • streamEq self-check now compares the recorded state fields too, so an adopted suffix with stale context state cannot survive silently.
  • javascript (regexCtx + templates) and templatejs (templates) keep the full-relex fallback — the template-stack slice is next (S5b).
  • Gate: 6 new regexjs scenarios, including both context-flip directions (deleting a turns two divisions into one regex literal; iff turns a regex back into divisions), an unbalanced-( insertion that must refuse resync, and a )-anchored append whose relex outcome hinges on the restored lastClose bit.

Validation

  • node test/portable-targets.ts — calc 8/8, javascript 5/5 (still full-relex), regexjs 6/6, envspec 8/8, × 3 targets, all streamEq true
  • node test/check.ts emit-parity — 5/5; node test/newline-mode.ts — 29/29; npx tsc --noEmit — clean
  • Mutation tests (×2): dropping the lastClose restore fails the )-anchor scenario; dropping the paren-stack comparison fails the unbalanced-( scenario — both reverted before commit
  • Independent adversarial probe (6 attacks × 3 targets): block-comment-interior edit, depth-3 paren stack, ( inside a regex body (not a real paren), division after ], non-head ) anchor with lastClose=false, two-batch state inheritance chain — all parity with oracle-derived expectations

Co-authored-by: Cursor <cursoragent@cursor.com>
@johnsoncodehk johnsoncodehk merged commit b72941e into master Jul 12, 2026
3 checks passed
@johnsoncodehk johnsoncodehk deleted the portable-relex-rx-s5a branch July 12, 2026 06:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant