Skip to content

feat(app): highlight code with @pierre/highlights instead of Shiki - #57937

Merged
Hona merged 4 commits into
devfrom
highlights-swap
Oct 21, 2026
Merged

Hona merged 4 commits into
devfrom
highlights-swap

Conversation

@Hona

@Hona Hona commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Replaces Shiki with @pierre/highlights for all code highlighting in the desktop app: review/file diffs, markdown code blocks and streaming code blocks. Shiki, @shikijs/stream and marked-shiki are no longer used by the GUI (packages/web keeps its own Shiki).

  • Diffs: @pierre/diffs 1.5.1 → 1.6.1 (first version with preferredHighlighter: "highlights"). Viewers and both worker pools share one highlightOptions object, so they cannot disagree.
  • Markdown: Highlights is synchronous, so createMarkdownParser() highlights fenced code itself with a marked renderer. The injected highlight callback and marked-shiki are gone.
  • Streaming: LiveTokenizer replaces ShikiStreamTokenizer. Each delta is one append edit, so only the growing last line is re-lexed.
  • Theme: OpenCodeTheme is a Zed palette whose values name the app's --syntax-* variables, so light, dark and custom themes keep working. Token types are grouped the way Pierre's own pierre-dark theme groups them, and each group uses the app syntax role with the same meaning. Colours follow Pierre, not Shiki.
  • Cleanup: removed the dead MarkedProvider/useMarked context, marked-theme-register, the unused getWorkerPools/workerFactory exports, and Shiki's bundledLanguagesInfo (attachment labels now use a small extension → name map).
  • Fences keep their own language in class="language-…" even without a lexer, so Mermaid and labels still work.
flowchart LR
  subgraph Before
    A1[diffs pool] --> S1[Shiki + Oniguruma wasm + grammars]
    B1[markdown worker] --> S2[Shiki + Oniguruma wasm + grammars]
    C1[MarkedProvider] --> S3[Shiki via pierre shared highlighter]
  end
  subgraph After
    A2[diffs pool] --> H1[Highlights wasm]
    B2[markdown worker] --> H2[Highlights wasm]
  end
Loading
// packages/session-ui/src/pierre/index.ts
registerCustomTheme(OpenCodeTheme.name, () => Promise.resolve(OpenCodeTheme), "zed")
export const highlightOptions = { theme: OpenCodeTheme.name, preferredHighlighter: "highlights" } as const

Benchmarks on real session data

Inputs come from the last 14 days of prod OpenCode Desktop data (91 sessions, 14,085 messages): the largest review-panel diffs rebuilt from turn snapshots (picked from 5,075), files the agent wrote, edit-tool patches, assistant answers and their code blocks. One distinct file type per row.

Each engine runs the same code the app runs (renderDiffWithHighlighter / renderFileWithHighlighter from @pierre/diffs 1.6.1 with the pool's render options, the old vs new markdown parser, and ShikiStreamTokenizer vs LiveTokenizer). Every engine × suite ran in 3 fresh processes with alternating order. Values are medians of warm runs. Bun 1.4.3, Windows.

Surface Shiki Highlights Speedup Peak RSS (Shiki → Highlights)
Review panel diffs (10 files, 13–535 KiB) 1,820 ms 301 ms 6.1× 1,206 → 417 MiB
File viewer (10 written files) 379 ms 13.3 ms 28.5× 680 → 140 MiB
Edit-tool patches (8) 68.4 ms 4.05 ms 16.9× 284 → 112 MiB
Markdown answers (8) 47.7 ms 7.90 ms 6.0× 240 → 112 MiB
Streaming code blocks (10, 32-char deltas) 23.4 ms 10.7 ms 2.2× 193 → 90 MiB
Time to first highlighter 40–45 ms 8–12 ms ~4×
Per-input results

Review panel diffs

Input Size Shiki (ms) Highlights (ms) Speedup
session-runner.test.ts 535 KiB 787 131 6.0×
ui.rs 432 KiB 356 90.4 3.9×
index.tsx 262 KiB 359 65.8 5.4×
styles.css 133 KiB 158 6.27 25.1×
SPEC.md 106 KiB 15.1 1.48 10.2×
gpt-nano-reuses-a-cached-prompt.json 105 KiB 0.51 0.21 2.4×
Settings.swift 90 KiB 95.6 3.35 28.5×
publish.yml 63 KiB 18.6 1.11 16.7×
astro.config.mjs 17 KiB 10.5 0.65 16.1×
sdk.js 13 KiB 19.5 0.48 40.6×

File viewer

Input Size Shiki (ms) Highlights (ms) Speedup
index.html 74 KiB 147 4.45 33.1×
compare.tsx 67 KiB 59.2 2.61 22.7×
settings_window.rs 54 KiB 38.6 2.01 19.2×
v5-tail.js 40 KiB 96.7 2.38 40.6×
SPEC.md 26 KiB 4.87 0.51 9.5×
i18n_new_1.json 24 KiB 4.49 0.25 18.2×
p7_va.ts 20 KiB 4.44 0.16 27.8×
ListenActivity.java 17 KiB 13.8 0.57 24.3×
interface-mode.css 6 KiB 4.48 0.18 25.4×
inventory.py 4 KiB 4.97 0.17 29.3×

Edit-tool patches

Input Size Shiki (ms) Highlights (ms) Speedup
start.tsx 12 KiB 17.1 1.45 11.8×
ssh.rs 11 KiB 9.40 0.54 17.5×
Program.cs 9 KiB 14.3 0.92 15.5×
v5-tail.js 8 KiB 19.2 0.47 40.7×
AGENTS.md 7 KiB 1.01 0.08 13.4×
host.ts 7 KiB 4.78 0.33 14.7×
ListenActivity.java 3 KiB 2.54 0.22 11.4×
bunfig.toml 3 KiB 0.07 0.05 1.3×

Markdown answers

Input Size Shiki (ms) Highlights (ms) Speedup
answer 1 33 KiB 21.7 2.35 9.2×
answer 2 20 KiB 9.11 1.62 5.6×
answer 3 17 KiB 4.91 1.28 3.8×
answer 4 16 KiB 5.49 1.25 4.4×
answer 5 11 KiB 3.28 0.43 7.6×
answer 6 10 KiB 1.70 0.62 2.7×
answer 7 4 KiB 1.01 0.07 13.6×
answer 8 4 KiB 0.42 0.28 1.5×

Streaming code blocks

Input Size Shiki (ms) Highlights (ms) Speedup
markdown 4 KiB 2.97 1.49 2.0×
ts 3 KiB 11.4 1.44 7.9×
rust 2 KiB 4.84 1.39 3.5×
toml 1 KiB 0.52 1.00 0.5×
jsonc 1 KiB 0.29 1.23 0.2×
powershell 1 KiB 1.07 0.74 1.4×
sh 1 KiB 1.14 1.04 1.1×
nsis <1 KiB 0.31 0.84 0.4×
js <1 KiB 0.63 0.83 0.8×
mermaid <1 KiB 0.20 0.69 0.3×

Tiny streamed blocks cost a flat ~0.7 ms more, because each LiveTokenizer gets its own Wasm instance. Large blocks are 2–8× faster.

Bundle (vite build, packages/app/dist)

Shiki (dev) Highlights (this PR)
All JS/Wasm/CSS 64.2 MiB / 19.9 MiB gzip 65.2 MiB / 20.3 MiB gzip
Markdown worker loads 458 KiB + 608 KiB Oniguruma + grammars 579 KiB (Wasm inlined)
Diffs worker loads ~255 KiB + 608 KiB Oniguruma + grammars 72 KiB + 269 KiB Highlights

The total grows by 0.4 MiB gzip, mostly from the @pierre/diffs 1.6.1 bump. Shiki's grammar chunks are still emitted (@pierre/diffs imports its Shiki backend lazily) but are never fetched.

Visuals

Real excerpts from the corpus, Shiki (left) and Highlights (right), with the app's resolved theme variables:

Dark Light

Storybook, dev vs this PR:

Before After
Review diff
Markdown

Colours

Pierre decides what each character is: its lexer marks refresh: as a property and Math as a type. The app theme decides the actual colours through --syntax-*. The palette connects the two by following Pierre's own groups, and the colours intentionally differ from Shiki's.

pierre-dark group Token types App role
Keywords keyword.*, preproc, tag, selector --syntax-keyword
Types type.*, constructor, enum, JSX components --syntax-type
Functions function.*, selector.id --syntax-primitive
Variables variable, property, JSON keys --syntax-variable
Constants constant.*, number, boolean, escapes, namespace, label --syntax-constant
Attributes attribute.*, selector.class --syntax-property
Operators operator, keyword.operator --syntax-operator
Punctuation punctuation.* --syntax-punctuation
Strings / regex / comments string, string.regex, comment.* --syntax-string / --syntax-regexp / --syntax-comment

Two token types that share a colour in pierre-dark always share a role here. Where Pierre uses a shade the app has no role for, the role of the same colour family is used (numbers and escapes are cyan, like constants).

@Hona
Hona requested a review from Brendonovich as a code owner October 21, 2026 22:10
Copilot AI balanced review requested due to automatic review settings October 21, 2026 22:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found no problems that should block a merge. Both type checks and the nearest tests pass. I couldn't run the desktop app or Storybook here, so there are no screenshots from me; the ones in the description are the visual evidence.

  • Tests and type checks: the marked-parser tests in packages/ui (56) and bun test src in packages/session-ui (252) pass, and typecheck passes in ui, session-ui and app.
  • Language names: Highlights accepts the usual short fence names (ts, py, sh, yml, rs, c++, golang, …). Unsupported ones like mermaid are lexed as text but keep their own language-… class.
  • Streaming: I replayed the new LiveTokenizer append-edit logic in small deltas over TypeScript containing a block comment and a multi-line template literal. The streamed text matched the source exactly, and every character had the same colour as codeToTokens on the whole block.
  • Diffs: rendering through @pierre/diffs 1.6.1 with the zed-registered OpenCodeTheme gives var(--syntax-*) colours, and the large-file pool's tokenizeMaxLineLength is still passed through to the Highlights backend.

Public API is unchanged: only internal workspace modules and exports are removed (MarkedProvider/useMarked, marked-theme, getWorkerPools, workerFactory), and nothing in the SDK, server, plugins, config or CLI is touched.

No further concerns.

Fixes #57876

@Hona
Hona merged commit c2736fc into dev Oct 21, 2026
19 of 20 checks passed
@Hona
Hona deleted the highlights-swap branch October 21, 2026 22:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants