Skip to content

feat(scrawl): the built-in capture no longer needs the Scrawl app - #27

Merged
collissimo merged 14 commits into
collin/mainfrom
t3/scrawl-parity
Oct 31, 2026
Merged

collissimo merged 14 commits into
collin/mainfrom
t3/scrawl-parity

Conversation

@collissimo

Copy link
Copy Markdown
Owner

The native Scrawl capture from #25 still leaned on the Scrawl app for looking closer at a capture, and lacked most of what makes its text and images useful to an agent. This brings it to parity, iPhone mirroring excluded, so the Scrawl app is no longer needed.

What is new

  • Look closer without Scrawl: <T3 home>/bin/t3-scrawl.cmd runs zoom, frames, inspect and export. The app's exe decodes with Chromium; mediabunny handles the files (new dependency, MPL-2.0).
  • Ink: circle, underline, line and arrow are recognised, several marks on a keyframe are numbered, marks on one screen share a keyframe.
  • Images: each keyframe gets its own crop, and the contact sheet packs them in justified rows.
  • Context: the wheel, cursor, window log, app switches, Ctrl/Win shortcuts, flags (Alt+Shift+F) and what a click landed on (clicked button “Area”). Typed text is not logged.
  • Operating it: rebindable shortcut in Settings, command palette entries, a past captures dialog (Alt+Shift+H), pinning and Scrawl's retention presets.
  • Not losing work: video stays out of renderer memory while recording, and an interrupted capture is rebuilt at the next launch. MP4 now sets a 2-second keyframe interval: before, a crash left only a header on disk.

Sentences are still never cut at a click, and clicks the screen did not answer still read as pointed.

Behaviour changes

  • Whole captures go to the Recycle Bin after 30 days by default, like Scrawl. Before they stayed forever.
  • During a capture Alt+Shift+F is the flag, not VS Code's Format Document.
  • The first element lookup in a Chromium app turns on its accessibility tree, which can slow a heavy page.

Checked

  • vp run check:changed origin/collin/main passes (39 packages typechecked, 46 test files).
  • All 124 ink marks in 43 real captures get the shape and box of Scrawl's current sketch.rs. Sentence cuts stay at 1 of 133 lines.
  • Dev app on Windows, after merging all branches:
    • a capture with real clicks and wheel activity built a bundle with crop, full frames, window and cursor logs, and reached the composer
    • a reload during a capture rebuilt the bundle and showed the recovery toast
    • --scrawl inspect, zoom and export ran on the native bundle and exited 0

Not checked

  • A capture with clear live speech end to end, and pressing the real shortcuts: Alt+Shift+S, Alt+Shift+F and a rebound key.
  • The past captures dialog and the Settings row in the running app.
  • Element names, ink shapes and numbering in a real capture: this run drew no ink and its clicks hit T3 Code, which is deliberately not looked up.
  • The packaged build, including whether it writes and ships t3-scrawl.cmd.
  • A full app crash (only a reload was tried), a 10-minute capture's memory, and Codex-sandboxed agents running the command.
  • macOS and Linux. Most context signals are Windows-only.

The recording pill still shows in the video. No images: test recordings showed private screen content and were deleted.

Done with Claude Opus 5.5 in Claude Code, five delegated agents plus integration.

🤖 Generated with Claude Code

collissimo and others added 14 commits October 31, 2026 18:13
Native captures called every stroke group a plain "mark", so an agent could not tell a circle from
an underline, and marks on one screen each took a keyframe.

Port Scrawl's rules as pure TypeScript: shapes with from/to and the circled box, marks grouped on one
keyframe while nothing happened to the screen, numbered in drawing order in the text and scrawl.json
but not on the image. The keyframe draws all its marks' ink, which has faded from the video by then.
Whole sentences still own keyframes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eo stays on disk

The shortcut was hardcoded, a capture held its video in memory, a crash lost the result, and nobody
could see or keep what retention deletes. Settings now rebinds the shortcut and says when another
program holds it, the video and audio only live on disk, the next launch rebuilds interrupted
captures, and a history dialog finds, pins and bins captures with the date retention acts on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tified rows

Native captures used one crop box for the whole capture and a plain grid. Each keyframe now gets its
own crop from Scrawl's rules: input its line names, what a click changed, optional window and cursor
logs, edges moved off text. The sheet uses Scrawl's row planner and one chip size, the click mark is
a thin blue ring, and ordinary actions fill only the first 6 keyframes.

On the 43 real Scrawl captures the port reproduces Scrawl's crops (median overlap 1.00 with its logs,
0.91 without) and its sheet sizes exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…shortcuts, flags and what was clicked

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…th its events

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ure without the Scrawl app

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The MP4 muxer writes nothing until the next keyframe, so a capture that crashed had only a header
on disk. A 2-second keyframe interval puts a fragment there every 2 seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pture

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Oct 31, 2026
@collissimo
collissimo merged commit 2edbe2f into collin/main Oct 31, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant