feat(scrawl): the built-in capture no longer needs the Scrawl app

FORK.md

@@ -160,14 +160,58 @@ the contact sheet land in the composer on screen. Esc in T3 Code discards a capt
 
 - Desktop app only: a browser cannot see the whole screen, and mobile has no such thing. Clicks and
   ink need Windows; elsewhere a capture has voice and video.
-- While the Scrawl app runs it holds Alt+Shift+S. T3 Code logs that and takes the shortcut within
-  15 seconds of Scrawl closing. The composer button works either way.
+- **Alt+Shift+F** flags a moment while recording (`⚑`, always its own keyframe). The pill shows the
+  flag for a moment.
+- Besides clicks and ink, a capture logs the wheel, the cursor, which windows are on screen, app
+  switches, Ctrl and Win shortcuts, and what a click or stroke landed on (`clicked button “Area”`).
+  Only the cursor and flags work outside Windows. Typed text is never logged, AltGr counts as
+  typing, passwords are never read, and a window title reaches the text an agent reads only in the
+  head line, as in the Scrawl app. Fields say only that they were clicked, not what they are called.
+- This runs in two helper processes, for the length of a capture: one reads Raw Input (no system-wide
+  hook, so no program's mouse waits on it) and the window list, the other is a PowerShell asking UI
+  Automation (about 80 MB). The first lookup in a Chromium-based app turns on that app's accessibility
+  tree, which can slow a heavy page. T3 Code's own windows are not looked up, so clicks on them stay
+  unnamed. A precision touchpad's two-finger scroll may not show as Raw Input wheel events (unchecked).
+- While the Scrawl app runs it holds Alt+Shift+S. Settings > SnapShots > Screen capture says so,
+  and T3 Code takes the shortcut within 15 seconds of Scrawl closing. The composer button and the
+  command palette work either way. Rebind it in the same place.
 - Speech and pointing do not run in step, so sentences are never cut at a click: whole sentences
   own a keyframe, click bursts fold into one (`clicked ×3`), and clicks the screen did not answer
   read as `pointed`. On 32 real captures this took cuts through sentences from 59 of 127 lines to 1.
+- Ink is recognised with Scrawl's rules (`circled`, `underlined`, `drew a line`, `drew an arrow`,
+  else `marked`; `scrawl.json` has `shape`, `from`, `to`). Marks drawn on one screen share a keyframe
+  that draws all their ink, until a click or drag, a scene change, 20 s or 12 strokes. Several on a
+  keyframe are numbered in drawing order in the text and `scrawl.json` (`✎① 0:06 circled “…”`), never
+  on the image. No mark is dropped; over the keyframe budget, neighbouring groups merge. On Scrawl's
+  43 captures all 124 ink marks get the shape and box of Scrawl's current `sketch.rs`. Scrawl compares
+  the screen around the ink, but here the video shows the ink, so scene changes decide "same screen".
+  The laser and pins are not here: Scrawl's Windows build records no laser and Shift+move is the ink.
+- Each keyframe is cropped on its own to what happened around it, with Scrawl's rules and numbers:
+  the input its line names, what a click changed, and edges moved off text
+  (`frames-full/` keeps the whole screen). The contact sheet packs crops of any shape into justified
+  rows, one chip size throughout. Budget: 6 keyframes up to 40 s, one more per 20 s, at most 12;
+  ordinary actions fill only the first 6. A window or cursor log improves the crops when
+  `CaptureRecording` has one. On the 43 real Scrawl captures, given those logs, the crops reproduce
+  Scrawl's own (median overlap 1.00); without them 0.91.
 - Uses the dictation model above, with word times mapped back across the pauses it cuts out.
 - The recording pill in the corner shows in the video and keyframes; `setContentProtection` does
   not keep it out on Windows 10 22H2.
 - Bundles are in `<T3 home>/scrawl/captures` in Scrawl's format, so the Scrawl skill and the
-  `scrawl` CLI (`inspect`, `zoom`, `frames`) read them. Video and audio are deleted after 7 days.
+  `scrawl` CLI (`inspect`, `zoom`, `frames`) read them.
+- Video and audio go to disk as they are recorded and stay out of memory until the build, which reads
+  them back. A crash or reload during a capture loses at most the last seconds: the next launch
+  rebuilds it once from `unfinished.json`, `timing.json`, `audio.pcm` and the video. A capture that
+  fails is kept and offers a manual rebuild. Scene changes come from sampling the video again.
+- The recorder sets a 2-second keyframe interval on purpose: without it the MP4 muxer put nothing but
+  a header on disk for over 11 seconds, so a crash left no video. Measured in Chromium.
+- Past captures (Alt+Shift+H, palette, Settings) find, insert, pin, or bin a capture and show the
+  date retention acts on. Same presets and `.pinned` marker as Scrawl: by default video goes to the
+  Recycle Bin after 7 days and the capture after 30. The list and the hourly pass share one plan
+  (`apps/desktop/src/scrawl/captureStore.ts`), so the dates shown are what happens.
+- Agents look closer with `<T3 home>/bin/t3-scrawl.cmd` (`t3-scrawl` elsewhere), which the app writes
+  at every start and the pasted text names: `zoom <bundle> --at 0:21|"a phrase"|--mark N`, `frames
+  --from --to`, `inspect [--json|--marks]`, `export [--out] [--zoom] [--crop] [--size]`. It needs no
+  Scrawl app and no running T3 Code. The app's exe does the work with `--scrawl`: Chromium is the
+  only video decoder in the app, so plain Node (`t3.cmd`) and a shipped ffmpeg lost to a hidden page
+  running mediabunny on WebCodecs. Agents on another machine cannot reach the bundle either way.
 - Code: `apps/desktop/src/scrawl/`, `apps/web/src/scrawl/`, `packages/contracts/src/scrawl.ts`.

apps/desktop/package.json

@@ -27,6 +27,7 @@
     "electron-updater": "^6.8.9",
     "electron-webauthn": "catalog:",
     "ffi-rs": "1.3.2",
+    "mediabunny": "1.61.6",
     "playwright-core": "1.60.0",
     "react-grab": "^0.1.32",
     "tldts": "catalog:"

apps/desktop/src/boot.ts

@@ -1,2 +1,3 @@
 require("./compileCache.cjs");
-require("./main.cjs");
+// Kept off the app's main: no lock, no server, no windows. See scrawl/scrawlCli.ts.
+require(process.argv.includes("--scrawl") ? "./scrawlCli.cjs" : "./main.cjs");

apps/desktop/src/ipc/DesktopIpcHandlers.ts

@@ -12,9 +12,19 @@ import {
 import { dictationAppend, dictationCancel, dictationFinish } from "./methods/dictation.ts";
 import {
   scrawlAppendVideo,
+  scrawlBeginRebuild,
   scrawlCancel,
+  scrawlCheckShortcut,
+  scrawlGetShortcutState,
+  scrawlListCaptures,
+  scrawlListUnfinished,
+  scrawlMarkBuilt,
+  scrawlReadFile,
+  scrawlReveal,
+  scrawlSetPinned,
   scrawlStart,
   scrawlStop,
+  scrawlTrash,
   scrawlWriteFile,
 } from "./methods/scrawl.ts";
 import {
@@ -112,6 +122,16 @@ export const installDesktopIpcHandlers = Effect.fn("desktop.ipc.installHandlers"
   yield* ipc.handle(scrawlCancel);
   yield* ipc.handle(scrawlAppendVideo);
   yield* ipc.handle(scrawlWriteFile);
+  yield* ipc.handle(scrawlReadFile);
+  yield* ipc.handle(scrawlListUnfinished);
+  yield* ipc.handle(scrawlBeginRebuild);
+  yield* ipc.handle(scrawlMarkBuilt);
+  yield* ipc.handle(scrawlListCaptures);
+  yield* ipc.handle(scrawlSetPinned);
+  yield* ipc.handle(scrawlTrash);
+  yield* ipc.handle(scrawlReveal);
+  yield* ipc.handle(scrawlGetShortcutState);
+  yield* ipc.handle(scrawlCheckShortcut);
   yield* ipc.handle(dictationCancel);
   yield* ipc.handle(getLocalEnvironmentBearerToken);
 

apps/desktop/src/ipc/channels.ts

@@ -44,6 +44,16 @@ export const SCRAWL_STOP_CHANNEL = "desktop:scrawl-stop";
 export const SCRAWL_CANCEL_CHANNEL = "desktop:scrawl-cancel";
 export const SCRAWL_APPEND_VIDEO_CHANNEL = "desktop:scrawl-append-video";
 export const SCRAWL_WRITE_FILE_CHANNEL = "desktop:scrawl-write-file";
+export const SCRAWL_READ_FILE_CHANNEL = "desktop:scrawl-read-file";
+export const SCRAWL_LIST_UNFINISHED_CHANNEL = "desktop:scrawl-list-unfinished";
+export const SCRAWL_BEGIN_REBUILD_CHANNEL = "desktop:scrawl-begin-rebuild";
+export const SCRAWL_MARK_BUILT_CHANNEL = "desktop:scrawl-mark-built";
+export const SCRAWL_LIST_CAPTURES_CHANNEL = "desktop:scrawl-list-captures";
+export const SCRAWL_SET_PINNED_CHANNEL = "desktop:scrawl-set-pinned";
+export const SCRAWL_TRASH_CHANNEL = "desktop:scrawl-trash";
+export const SCRAWL_REVEAL_CHANNEL = "desktop:scrawl-reveal";
+export const SCRAWL_GET_SHORTCUT_STATE_CHANNEL = "desktop:scrawl-get-shortcut-state";
+export const SCRAWL_CHECK_SHORTCUT_CHANNEL = "desktop:scrawl-check-shortcut";
 export const GET_LOCAL_ENVIRONMENT_BEARER_TOKEN_CHANNEL =
   "desktop:get-local-environment-bearer-token";
 export const GET_CLIENT_SETTINGS_CHANNEL = "desktop:get-client-settings";

apps/desktop/src/ipc/methods/clientSettings.ts

@@ -4,6 +4,7 @@ import * as Option from "effect/Option";
 import * as Schema from "effect/Schema";
 
 import * as DesktopClientSettings from "../../settings/DesktopClientSettings.ts";
+import * as DesktopScrawl from "../../scrawl/DesktopScrawl.ts";
 import * as DesktopSnapShot from "../../snapShot/DesktopSnapShot.ts";
 import * as IpcChannels from "../channels.ts";
 import * as DesktopIpc from "../DesktopIpc.ts";
@@ -27,5 +28,6 @@ export const setClientSettings = DesktopIpc.makeIpcMethod({
     const snapShot = yield* DesktopSnapShot.DesktopSnapShot;
     yield* clientSettings.set(settings);
     yield* snapShot.configure(settings);
+    yield* (yield* DesktopScrawl.DesktopScrawl).configure(settings);
   }),
 });

apps/desktop/src/ipc/methods/scrawl.ts

@@ -1,9 +1,16 @@
 import {
   ScrawlAppendVideoInput,
   ScrawlCaptureRef,
+  ScrawlCaptureSummary,
+  ScrawlReadFileInput,
+  ScrawlSetPinnedInput,
+  ScrawlShortcutAvailability,
+  ScrawlShortcutState,
   ScrawlStartResult,
   ScrawlStopResult,
+  ScrawlUnfinished,
   ScrawlWriteFileInput,
+  SnapShotKeyChord,
 } from "@t3tools/contracts";
 import * as Effect from "effect/Effect";
 import * as Schema from "effect/Schema";
@@ -56,3 +63,93 @@ export const scrawlWriteFile = makeIpcMethod({
     yield* (yield* DesktopScrawl.DesktopScrawl).writeFile(input);
   }),
 });
+
+export const scrawlReadFile = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_READ_FILE_CHANNEL,
+  payload: ScrawlReadFileInput,
+  result: Schema.Uint8Array,
+  handler: Effect.fn("desktop.ipc.scrawl.readFile")(function* (input) {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).readFile(input);
+  }),
+});
+
+export const scrawlListUnfinished = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_LIST_UNFINISHED_CHANNEL,
+  payload: Schema.Void,
+  result: Schema.Array(ScrawlUnfinished),
+  handler: Effect.fn("desktop.ipc.scrawl.listUnfinished")(function* () {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).listUnfinished();
+  }),
+});
+
+export const scrawlBeginRebuild = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_BEGIN_REBUILD_CHANNEL,
+  payload: ScrawlCaptureRef,
+  result: ScrawlUnfinished,
+  handler: Effect.fn("desktop.ipc.scrawl.beginRebuild")(function* (input) {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).beginRebuild(input);
+  }),
+});
+
+export const scrawlMarkBuilt = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_MARK_BUILT_CHANNEL,
+  payload: ScrawlCaptureRef,
+  result: Schema.Void,
+  handler: Effect.fn("desktop.ipc.scrawl.markBuilt")(function* (input) {
+    yield* (yield* DesktopScrawl.DesktopScrawl).markBuilt(input);
+  }),
+});
+
+export const scrawlListCaptures = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_LIST_CAPTURES_CHANNEL,
+  payload: Schema.Void,
+  result: Schema.Array(ScrawlCaptureSummary),
+  handler: Effect.fn("desktop.ipc.scrawl.listCaptures")(function* () {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).listCaptures();
+  }),
+});
+
+export const scrawlSetPinned = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_SET_PINNED_CHANNEL,
+  payload: ScrawlSetPinnedInput,
+  result: Schema.Void,
+  handler: Effect.fn("desktop.ipc.scrawl.setPinned")(function* (input) {
+    yield* (yield* DesktopScrawl.DesktopScrawl).setPinned(input);
+  }),
+});
+
+export const scrawlTrash = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_TRASH_CHANNEL,
+  payload: ScrawlCaptureRef,
+  result: Schema.Void,
+  handler: Effect.fn("desktop.ipc.scrawl.trash")(function* (input) {
+    yield* (yield* DesktopScrawl.DesktopScrawl).trash(input);
+  }),
+});
+
+export const scrawlReveal = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_REVEAL_CHANNEL,
+  payload: ScrawlCaptureRef,
+  result: Schema.Void,
+  handler: Effect.fn("desktop.ipc.scrawl.reveal")(function* (input) {
+    yield* (yield* DesktopScrawl.DesktopScrawl).reveal(input);
+  }),
+});
+
+export const scrawlGetShortcutState = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_GET_SHORTCUT_STATE_CHANNEL,
+  payload: Schema.Void,
+  result: ScrawlShortcutState,
+  handler: Effect.fn("desktop.ipc.scrawl.getShortcutState")(function* () {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).shortcutState;
+  }),
+});
+
+export const scrawlCheckShortcut = makeIpcMethod({
+  channel: IpcChannels.SCRAWL_CHECK_SHORTCUT_CHANNEL,
+  payload: SnapShotKeyChord,
+  result: ScrawlShortcutAvailability,
+  handler: Effect.fn("desktop.ipc.scrawl.checkShortcut")(function* (shortcut) {
+    return yield* (yield* DesktopScrawl.DesktopScrawl).checkShortcut(shortcut);
+  }),
+});

apps/desktop/src/ipc/methods/snapShot.ts

@@ -18,6 +18,7 @@ import type * as Electron from "electron";
 
 import * as ElectronWindow from "../../electron/ElectronWindow.ts";
 import * as ElectronDialog from "../../electron/ElectronDialog.ts";
+import * as DesktopScrawl from "../../scrawl/DesktopScrawl.ts";
 import * as DesktopSnapShot from "../../snapShot/DesktopSnapShot.ts";
 import * as IpcChannels from "../channels.ts";
 import * as DesktopIpc from "../DesktopIpc.ts";
@@ -158,6 +159,8 @@ export const setSnapShotShortcutSuppressed = DesktopIpc.makeIpcMethod({
   handler: Effect.fn("desktop.ipc.snapShot.setShortcutSuppressed")(function* (suppressed, event) {
     yield* ensureTrustedSnapShotSender(event);
     yield* (yield* DesktopSnapShot.DesktopSnapShot).setShortcutSuppressed(suppressed);
+    // One recorder records both shortcuts, and a held global shortcut never reaches it.
+    yield* (yield* DesktopScrawl.DesktopScrawl).setShortcutSuppressed(suppressed);
   }),
 });
 

apps/desktop/src/preload.ts

@@ -127,6 +127,18 @@ contextBridge.exposeInMainWorld("desktopBridge", {
   scrawlCancel: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_CANCEL_CHANNEL, input),
   scrawlAppendVideo: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_APPEND_VIDEO_CHANNEL, input),
   scrawlWriteFile: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_WRITE_FILE_CHANNEL, input),
+  scrawlReadFile: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_READ_FILE_CHANNEL, input),
+  scrawlListUnfinished: () => ipcRenderer.invoke(IpcChannels.SCRAWL_LIST_UNFINISHED_CHANNEL),
+  scrawlBeginRebuild: (input) =>
+    ipcRenderer.invoke(IpcChannels.SCRAWL_BEGIN_REBUILD_CHANNEL, input),
+  scrawlMarkBuilt: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_MARK_BUILT_CHANNEL, input),
+  scrawlListCaptures: () => ipcRenderer.invoke(IpcChannels.SCRAWL_LIST_CAPTURES_CHANNEL),
+  scrawlSetPinned: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_SET_PINNED_CHANNEL, input),
+  scrawlTrash: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_TRASH_CHANNEL, input),
+  scrawlReveal: (input) => ipcRenderer.invoke(IpcChannels.SCRAWL_REVEAL_CHANNEL, input),
+  scrawlGetShortcutState: () => ipcRenderer.invoke(IpcChannels.SCRAWL_GET_SHORTCUT_STATE_CHANNEL),
+  scrawlCheckShortcut: (shortcut) =>
+    ipcRenderer.invoke(IpcChannels.SCRAWL_CHECK_SHORTCUT_CHANNEL, shortcut),
   getClientSettings: () => ipcRenderer.invoke(IpcChannels.GET_CLIENT_SETTINGS_CHANNEL),
   setClientSettings: (settings) =>
     ipcRenderer.invoke(IpcChannels.SET_CLIENT_SETTINGS_CHANNEL, settings),