Highlighter's SQL lexer (sql-highlight.js) still duplicates literal/comment rules and diverges on '' escapes

[BorisTyshkevich]

Found while shipping #139 (shared SQL span-scanner)

#139 unified the literal/comment lexer shared by src/core/sql-split.js
(splitStatements) and src/core/query-params.js (detectParams) into
src/core/sql-spans.js scanSpans. But a third SQL lexer still lives in
src/core/sql-highlight.js tokenize (the editor highlighter + its maskLiterals
mask, used for bracket matching #24, signature help #27, and auto-close), and its
string-scanning rules differ:

• tokenize (src/core/sql-highlight.js:57-66) does not honor doubled-quote
  ('') escapes — its loop stops at the first unescaped closing quote. So
  'it''s' tokenizes as 'it' (string) + s (ident) + ' (unterminated
  string), whereas scanSpans / splitStatements correctly treat 'it''s' as
  one literal.

Impact (low, pre-existing): cosmetic mis-highlighting of a string containing a
doubled single-quote, and — via maskLiterals — a slightly wrong literal mask
there, so bracket-matching / auto-close / signature-help can misbehave while the
caret is inside such a string. Not a correctness bug in query execution (split and
param detection use the correct scanSpans).

Why deferred from #139: scanSpans yields only coarse code/string/comment
spans; the highlighter needs finer token types (keyword / func / number /
operator / ident), so it can't consume scanSpans as-is. Unifying means either
extending scanSpans to a full tokenizer or layering the highlighter's
type-classification on top of the shared literal/comment pass — a bigger change
than #139's scope (which named only sql-split + query-params).

Proposal: either (a) fix tokenize's string loop to honor '' (and \)
consistently with scanSpans, or (b) rebuild the highlighter on top of the shared
span pass so all three consumers share one set of literal/comment rules. See the
docstring note in src/core/sql-spans.js for the code-vs-comment-vs-string
distinction that must be preserved.

file:line — src/core/sql-highlight.js:57-66 (tokenize string branch).

[BorisTyshkevich]

Reconciled against Phase 4 of #68 (2026-07-03). The editor is moving to CodeMirror 6 (#143 → #21), which changes how the two proposal options land:

• The mis-masking consumers die at Adopt CodeMirror 6 behind the EditorPort seam (replace the hand-rolled textarea editor) #21's cutover. Bracket matching, auto-close, and signature help all read maskLiterals/maskFromTokens, and the highlighter overlay reads tokenize — every one of those is on Adopt CodeMirror 6 behind the EditorPort seam (replace the hand-rolled textarea editor) #21's delete list (ui/editor-*, core/editor-brackets.js, core/editor-marks.js). CM6's lang-sql lexes '' escapes correctly, so option (b) ("rebuild the highlighter on the shared span pass") is superseded for the editor.
• tokenize itself survives (kept for Column-name autocompletion: FROM-driven column loading + alias/scope awareness #84's from-scope.js and completion context). So the divergence still matters after the cutover — the plan in Adopt CodeMirror 6 behind the EditorPort seam (replace the hand-rolled textarea editor) #21/Column-name autocompletion: FROM-driven column loading + alias/scope awareness #84 is that from-scope.js sits on the shared core/sql-spans.js literal/comment rules, i.e. option (b) lands naturally for the surviving consumer.
• Option (a) — the small ''/\' string-loop fix in tokenize — is still worth doing on its own anytime before the cutover: the textarea editor stays the shipped editor until Adopt CodeMirror 6 behind the EditorPort seam (replace the hand-rolled textarea editor) #21 lands, and the fix is a few lines + tests, independent of Phase 4.

Net: no change to #143 (behavior-frozen seam extraction). This issue narrows to option (a) as a standalone small fix, with (b) absorbed by #21 (deletion) + #84 (from-scope on sql-spans). Cross-referenced from #21's Step 4.