[deadstrobe5]

With the default decodeEntities: true, stateText handles every text character through one iteration of the parse loop and its state switch. With decodeEntities: false it already skips ahead with fastForwardTo. This change makes stateText scan to the next < or & (or the end of the chunk) in a local loop, so long runs of text cost one call instead of one per character. Callbacks, startIndex/endIndex and the DOM are unchanged; no callback fires during the scan, so pausing and resetting from callbacks behave as before.

The gain shows on text-heavy input such as books, articles and feeds. Markup-heavy pages spend most of their characters in tags and attribute values, so they stay flat.

Input	Before	After	Speedup
Gutenberg book, 850 KB, parseDocument	7.17 ms	4.66 ms	1.54x
same, Parser with no-op callbacks	5.50 ms	3.00 ms	1.84x
Atom feed, 100 KB, xmlMode, parseDocument	0.85 ms	0.69 ms	1.23x
NYT RSS, 57 KB, xmlMode, parseDocument	0.55 ms	0.48 ms	1.15x
Wikipedia, GitHub, HN pages			~1.0x
200 B snippet	3.7 us	3.8 us	~1.0x

Thread CPU time, median of 11 alternating rounds with both builds loaded in one process (Node 24, Ryzen 5 5600X). Text with one character between tags and a& repeated stay within noise.

I checked old against new on random documents across HTML, xmlMode and decodeEntities: false, written whole, in small chunks and one character at a time, with pauses from callbacks: callbacks, indices and DOM were identical. I added a test for text and entities split across chunks.

Summary by CodeRabbit

• Bug Fixes
  • Improved parsing of text and entities across consecutive input chunks, including entities split at the end of a chunk.
• Tests
  • Added coverage for emitted events and offsets when chunks include text, entities, and tags.

[chatgpt-codex-connector]

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-09T22:58:28.545045Z	257c76b	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

When entity decoding is enabled, Tokenizer.stateText scans ordinary characters until it reaches a tag, an entity, or the buffer end. A regression test checks emitted events and offsets for text, entities, and tags delivered across multiple input chunks.

Changes: Tokenizer text scanning

Layer / File(s)	Summary
Text scanning and regression test src/Tokenizer.ts, src/Tokenizer.spec.ts	stateText advances over ordinary text to the next <, &, or buffer end. The regression test checks event names and offsets for chunked text containing entities and tags. It also covers an entity at the end of input.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Refactor

Suggested reviewers: fb55

Merge Risk: Minimal · up to 257c7

The tokenizer change is mergeable after normal checks; no concrete remaining risk is identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the tokenizer performance change: scanning text runs in one loop when entity decoding is enabled.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 2 files.

Finishing Touches

Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit reads the tokenizer’s trail,
Through text and tags, it follows each detail.
An ampersand marks where entities start,
While chunks arrive and offsets chart.
The final token rests, its checks complete,
The rabbit thumps a happy beat.}

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 68b7e218-6a7c-41fa-a32d-44edfb4831f8

Commits

Reviewing files that changed from the base of the PR and between 8fe6e97 and 257c76b.

Files selected for processing: 2

• src/Tokenizer.spec.ts
• src/Tokenizer.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

[cubic-dev-ai]

No issues found across 2 files

_{View guided diff | Re-trigger cubic}

[greptile-apps]

Confidence Score: 5/5

[Medium impact] This PR appears safe to merge.

Summary

Speeds up ordinary text scanning when decodeEntities is enabled.

• The tokenizer scans ordinary text runs locally when entity decoding is enabled.

_{Reviews (1) · Last reviewed commit: "perf(tokenizer): skip runs of text in on..." · Reviewed by Greptile}