Skip to content

feat: dictation built into the composer, with local Parakeet - #2

Merged
collissimo merged 5 commits into
collin/mainfrom
t3/built-in-dictation
Oct 20, 2026
Merged

feat: dictation built into the composer, with local Parakeet#2
collissimo merged 5 commits into
collin/mainfrom
t3/built-in-dictation

Conversation

@collissimo

@collissimo collissimo commented Oct 20, 2026

Copy link
Copy Markdown
Owner

T3 Voice ran as a separate Python app that drove T3 Code from the outside (UI Automation, a floating mic button, clipboard paste). This builds the same dictation into T3 Code itself, working the way T3 Voice does with Parakeet.

How it works

  • Server (apps/server/src/dictation/): Parakeet TDT 0.6b v3 through transcribe-cpp (npm, koffi, prebuilt). The .gguf from Scrawl or Hex is picked up (T3CODE_DICTATION_MODEL overrides). It loads on the first audio chunk while the user already speaks and is let go after 15 minutes unused; one model serves every client. Filler sounds are removed; ending with "Reinschrift" asks for a structured rewrite.
  • RPC (packages/contracts/src/dictation.ts): dictation.append (base64 16 kHz PCM every 0.5 s, returns a live preview of the last 20 s), dictation.finish, dictation.cancel, dictation.structure (claude --print --model haiku, prompt overridable at <T3 home>/dictation/structure-prompt.md). Scope: orchestration:operate.
  • Web: mic button next to "Attach files"; F9 start/stop, F8 (or shift-click) structured, Esc cancels a recording or inserts the plain transcript during the rewrite. Rebindable as "Composer: Dictate". AudioWorklet in public/ (desktop CSP blocks blob: worklets) resamples to 16 kHz.
  • Desktop: transcribe-cpp/koffi stay external like node-pty; their DLLs are unpacked from server.asar and the server points transcribe.cpp at the unpacked copy.

Surfaces

  • Web + desktop: yes. Remote clients work over the websocket, but the browser only allows the microphone on https/localhost, so plain http://<LAN IP> cannot dictate.
  • Mobile: unchanged, it already has on-device dictation.
  • Providers / MCP: not applicable (human input).
  • Docs: FORK.md → Fork features → Dictation.

Verified

  • vp test run apps/server/src/dictation: 41 tests (filler/codeword cases ported from T3 Voice, model ranking, engine lifecycle). Typecheck web/server/client-runtime/shared, targeted lint: clean.
  • Server with the real model + real Claude: 12 s German clip → preview, transcript, codeword, Haiku rewrite.
  • Packaged Windows build (T3-Code-0.1.0-x64.exe): Parakeet loads from server.asar.unpacked under Electron's Node.
  • Real UI (worktree dev server, headless Edge with the WAV as fake microphone): click → recording panel with live text → stop → structured prompt in the composer; F9/Esc/F9 paths work. Caught and fixed a worklet URL bug on thread routes.

🤖 Generated with Claude Code

collissimo and others added 5 commits October 20, 2026 23:39
@collissimo @claude
feat(server): dictation with a local Parakeet model
88de7f2 
The server turns streamed microphone audio into text with Parakeet
through transcribe.cpp, on its own machine. The model (found in the
Scrawl or Hex model folders, or T3CODE_DICTATION_MODEL) loads on the
first chunk while the user already speaks and is let go after 15
minutes unused. One model serves every client.

- dictation.append/finish/cancel: a recording session with a live
  preview of the last 20 seconds.
- Filler sounds are removed; a closing "Reinschrift" asks for a
  structured rewrite.
- dictation.structure: rewrites a rambling transcript into a
  structured prompt with Claude Haiku (claude --print).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@collissimo @claude
feat(web): dictate into the composer
d53bae9 
A mic button next to "Attach files" and F9 record the microphone and
insert the transcript at the caret; F8 (or shift-click, or ending with
"Reinschrift") has it rewritten into a structured prompt first. Esc
cancels a recording, or skips the rewrite and inserts the plain text.
A small panel above the button shows the live transcript and status.

Audio is captured in an AudioWorklet (served from public/, the desktop
CSP blocks blob: worklets), resampled to 16 kHz and streamed to the
server every half second. The microphone needs a secure context, so a
plain http:// LAN address cannot dictate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@collissimo @claude
build(desktop): ship transcribe.cpp in the server sidecar
f532800 
The package stays external like node-pty, so its platform package
(@transcribe-cpp/<os>-<arch>, picked by os/cpu) lands in server.asar.
Its DLLs are unpacked beside the archive, and the server points
transcribe.cpp at them because the OS loader cannot read an asar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@collissimo @claude
docs(fork): describe the built-in dictation
d40e66d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@collissimo @claude
fix(web): load the dictation worklet from the app root
9a6039b 
Relative to document.baseURI it resolved under the current thread route
and hit the SPA fallback (MIME type mismatch), so recording never
started.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Oct 20, 2026
@collissimo
collissimo merged commit 23f2b73 into collin/main Oct 20, 2026
7 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@collissimo