Repository navigation
feat(app): highlight code with @pierre/highlights instead of Shiki - #57937
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
There was a problem hiding this comment.
I found no problems that should block a merge. Both type checks and the nearest tests pass. I couldn't run the desktop app or Storybook here, so there are no screenshots from me; the ones in the description are the visual evidence.
- Tests and type checks: the
marked-parsertests inpackages/ui(56) andbun test srcinpackages/session-ui(252) pass, andtypecheckpasses inui,session-uiandapp. - Language names: Highlights accepts the usual short fence names (
ts,py,sh,yml,rs,c++,golang, …). Unsupported ones likemermaidare lexed as text but keep their ownlanguage-…class. - Streaming: I replayed the new
LiveTokenizerappend-edit logic in small deltas over TypeScript containing a block comment and a multi-line template literal. The streamed text matched the source exactly, and every character had the same colour ascodeToTokenson the whole block. - Diffs: rendering through
@pierre/diffs1.6.1 with the zed-registeredOpenCodeThemegivesvar(--syntax-*)colours, and the large-file pool'stokenizeMaxLineLengthis still passed through to the Highlights backend.
Public API is unchanged: only internal workspace modules and exports are removed (MarkedProvider/useMarked, marked-theme, getWorkerPools, workerFactory), and nothing in the SDK, server, plugins, config or CLI is touched.
No further concerns.
Fixes #57876
Replaces Shiki with
@pierre/highlightsfor all code highlighting in the desktop app: review/file diffs, markdown code blocks and streaming code blocks. Shiki,@shikijs/streamandmarked-shikiare no longer used by the GUI (packages/webkeeps its own Shiki).@pierre/diffs1.5.1 → 1.6.1 (first version withpreferredHighlighter: "highlights"). Viewers and both worker pools share onehighlightOptionsobject, so they cannot disagree.createMarkdownParser()highlights fenced code itself with amarkedrenderer. The injectedhighlightcallback andmarked-shikiare gone.LiveTokenizerreplacesShikiStreamTokenizer. Each delta is one append edit, so only the growing last line is re-lexed.OpenCodeThemeis a Zed palette whose values name the app's--syntax-*variables, so light, dark and custom themes keep working. Token types are grouped the way Pierre's ownpierre-darktheme groups them, and each group uses the app syntax role with the same meaning. Colours follow Pierre, not Shiki.MarkedProvider/useMarkedcontext,marked-theme-register, the unusedgetWorkerPools/workerFactoryexports, and Shiki'sbundledLanguagesInfo(attachment labels now use a small extension → name map).class="language-…"even without a lexer, so Mermaid and labels still work.flowchart LR subgraph Before A1[diffs pool] --> S1[Shiki + Oniguruma wasm + grammars] B1[markdown worker] --> S2[Shiki + Oniguruma wasm + grammars] C1[MarkedProvider] --> S3[Shiki via pierre shared highlighter] end subgraph After A2[diffs pool] --> H1[Highlights wasm] B2[markdown worker] --> H2[Highlights wasm] endBenchmarks on real session data
Inputs come from the last 14 days of prod OpenCode Desktop data (91 sessions, 14,085 messages): the largest review-panel diffs rebuilt from turn snapshots (picked from 5,075), files the agent wrote, edit-tool patches, assistant answers and their code blocks. One distinct file type per row.
Each engine runs the same code the app runs (
renderDiffWithHighlighter/renderFileWithHighlighterfrom@pierre/diffs1.6.1 with the pool's render options, the old vs new markdown parser, andShikiStreamTokenizervsLiveTokenizer). Every engine × suite ran in 3 fresh processes with alternating order. Values are medians of warm runs. Bun 1.4.3, Windows.Per-input results
Review panel diffs
File viewer
Edit-tool patches
Markdown answers
Streaming code blocks
Tiny streamed blocks cost a flat ~0.7 ms more, because each
LiveTokenizergets its own Wasm instance. Large blocks are 2–8× faster.Bundle (
vite build,packages/app/dist)dev)The total grows by 0.4 MiB gzip, mostly from the
@pierre/diffs1.6.1 bump. Shiki's grammar chunks are still emitted (@pierre/diffsimports its Shiki backend lazily) but are never fetched.Visuals
Real excerpts from the corpus, Shiki (left) and Highlights (right), with the app's resolved theme variables:
Storybook,
devvs this PR:Colours
Pierre decides what each character is: its lexer marks
refresh:as apropertyandMathas atype. The app theme decides the actual colours through--syntax-*. The palette connects the two by following Pierre's own groups, and the colours intentionally differ from Shiki's.pierre-darkgroupkeyword.*,preproc,tag,selector--syntax-keywordtype.*,constructor,enum, JSX components--syntax-typefunction.*,selector.id--syntax-primitivevariable,property, JSON keys--syntax-variableconstant.*,number,boolean, escapes,namespace,label--syntax-constantattribute.*,selector.class--syntax-propertyoperator,keyword.operator--syntax-operatorpunctuation.*--syntax-punctuationstring,string.regex,comment.*--syntax-string/--syntax-regexp/--syntax-commentTwo token types that share a colour in
pierre-darkalways share a role here. Where Pierre uses a shade the app has no role for, the role of the same colour family is used (numbers and escapes are cyan, like constants).