Skip to content

fix(preview): keep hidden desktop tabs composited while the server captures them - #1

Closed
ahalekelly wants to merge 79 commits into
mainfrom
fix/hidden-desktop-tab-snapshot
Closed

ahalekelly wants to merge 79 commits into
mainfrom
fix/hidden-desktop-tab-snapshot

Conversation

@ahalekelly

Copy link
Copy Markdown
Owner

🤖 Generated with Claude Code

Bug

preview_snapshot on a server tab the desktop renders but nobody is looking at (opened with open:false, or not the tab the user is viewing) always failed with Preview automation snapshot timed out after 15000ms, and afterwards preview_evaluate / preview_click / preview_type on that tab timed out too. Reproduced on Mac with preview_open {url, open:false} followed by preview_snapshot on a trivial page.

Root cause

  1. With the desktop attached, every server tab renders in a desktop <webview> that the server drives over CDP (ServerBrowser.connectDesktop).
  2. An inactive webview is parked at HIDDEN_BROWSER_WEBVIEW_OFFSET, where Electron stops compositing it (hostedBrowserWebviewStyle.ts). Only renderingActive moves it behind the app where it paints, and nothing set that for a server-driven capture: the client-side automation host that acquired surface activity went away when automation moved to the server (feat(preview): run the browser on the environment server pingdotgg/t3code#15328).
  3. ServerBrowserPage.captureViewport sent Page.captureScreenshot with no timeout; Chromium answers only after the page composites a frame, which never came.
  4. Agent operations run through tab.control.agent(...), a per-tab serial queue, so the hung snapshot blocked every later operation on the tab.

Fix

  • New server→desktop render command on the desktop browser channel. The server sends it while a tab needs frames (a capture, a recording, or a viewer stream) and clears it afterwards; the desktop also clears it on release, detach, and backend restart.
  • The desktop forwards it as capturing on the tab state, and HostedBrowserWebview counts it toward renderingActive, the same path picture-in-picture and client-side recordings use.
  • captureViewport gives up after 10 s with PreviewAutomationTimeoutError ("the page produced no frame ... it may not be rendering"), so a stall can no longer wedge the tab's queue.

Verification

  • tsc-rs --noEmit clean for contracts, server, desktop, web.
  • apps/server preview tests, apps/desktop preview tests, and the touched apps/web suites pass (new tests: snapshot on an unwatched desktop tab sends render true then false; a viewer keeps it on; the desktop host clears it on release).
  • ServerBrowserPage.test.ts fails locally only because Playwright's Chromium is not installed; same on main.
  • Not verified end to end in the packaged desktop app.

ahalekelly and others added 30 commits October 15, 2026 22:02
…behavior

Mobile's md4c qualified tilde openers and closers by different rules than
web's micromark, so `~$0.48 vs ~$0.65` and `~/foo and ~/bar` rendered struck
through on mobile but not web. Patch Enriched's md4c to apply emphasis-style
flanking to tilde runs, which is the rule micromark uses. This also aligns
`~~` with web: intraword `a~~b~~c` now strikes, and space-flanked `~~ b ~~`
no longer does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replays the fork's feat/voice branch onto v0.0.46-nightly.20261005.2702 as one commit:
the nightly moved dictation into VoiceInputProvider, so the original commits no
longer apply one by one. Original commits, oldest first:

- 544d6c2 feat(mobile): choose the voice transcription source, including OpenAI
- 3aab923 docs: describe OpenAI voice transcription on mobile
- fe1efd9 feat(mobile): mark dictated messages as voice transcriptions
- 71263a9 feat(mobile): send a dictation without reviewing it
- 6342064 feat(mobile): read iOS responses aloud
- 4475c65 feat(mobile): read responses with offline Pocket TTS
- d75275e feat(mobile): add speech playback controls and automatic voice replies
- 47dce2a fix(mobile): update speech runtime to prevent audio distortion
- 1abef55 feat(mobile): add offline Supertonic voice selection
- ae32053 feat(mobile): choose speech voice pace and quality
- ae4c433 fix(mobile): build voice features on the current nightly
- c8ce6c6 feat(mobile): read aloud with OpenAI and paragraph-aware native playback
- 0fb083d feat(mobile): make voice replies usable hands-free
- b7006d8 refactor(mobile): keep voice settings types out of feature and state layers
- ba9479c fix(mobile): address voice review findings
- 54fea39 feat(mobile): log speech playback diagnostics
- d3f816a feat(mobile): style the reading controls like the composer
- bd66d3e refactor(mobile): read responses aloud with OpenAI only
- 54169ac refactor(mobile): transcribe with GPT Transcribe as the only OpenAI model
- 93a2bfc refactor(mobile): keep voice changes out of upstream lines
- 441233d feat(mobile): read aloud with Gemini 3.8 Flash TTS or Flash-Lite TTS
- b91c80c feat(mobile): read aloud with Gemini Flash-Lite TTS by default
- 57f538a feat(mobile): read aloud with Gemini Flash TTS by default
- 6ac98a4 feat(mobile): tell agents replies go to text-to-speech, with unit examples
- c9b3f62 fix(mobile): clearing reading instructions sends none
- 030a823 fix(mobile): drop the reading instructions placeholder
- 2d0a07b feat(mobile): read aloud with ElevenLabs v4 or MAI-Voice-2
- 1751256 fix(mobile): stop and report recordings on audio interruptions
- a545684 feat(mobile): expose iOS background tasks from the speech module
- 9b9f01b feat(mobile): keep dictation through locks, interruptions, and failed transcriptions
- 444d887 feat(mobile): record up to 20 minutes of mono speech
- 38883e4 fix(mobile): ignore background tasks that already ended
- afc2bc6 fix(mobile): make dictation retries and background finishing reliable
- 1f52c3c fix(mobile): keep focused settings inputs above the keyboard
- a28f9d1 feat(mobile): MAI-Voice-2.1 and MAI-Transcribe-2
- 0aaaa62 fix(mobile): use West US 2 for MAI voices and transcription
- f1097aa feat(mobile): free Azure key first, paid on 403; API keys at the bottom
- 4eaac82 fix(mobile): send MAI voices parentheses instead of brackets and braces
- 96edcc1 feat(mobile): skip a spent Azure free key until its quota refills
- 7a28077 fix(mobile): without Retry-After, retry a spent Azure free key on the 2nd UTC
- 4e10bd9 feat(mobile): show stored voice API keys as their first and last four characters
- 05814e1 feat(mobile): voice API key fields take new keys only
- b73772d feat(mobile): copy stored voice API keys, and confirm before removing one
- 533092f fix(mobile): show stored voice API keys in the normal text color
- ae96ffd fix(mobile): show the first and last eight characters of stored voice API keys
- 56fca13 feat(mobile): read aloud with MAI-Voice-2.1 Flash by default
- c354cb7 feat(mobile): drop OpenAI read-aloud
The mobile app records its Effect spans in a fixed-size ring of compact OTLP
records and posts them to the connected environment's traces endpoint 5 s after
each connect, so the desktop server's trace file shows phone-side connect
timing. A failed or interrupted post keeps the spans for the next export. One
shared JS-stall monitor runs while any environment is connecting or in its
post-ready window, and ignores background suspension gaps.

The traces endpoint is now part of the typed EnvironmentHttpApi, so the export
gets typed auth errors and the DPoP refresh-and-retry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1b73fea)
The shell snapshot fetch spends ~230 ms after its HTTP span ends, with no
way to tell JSON.parse from schema decoding. Snapshot loaders now fetch the
raw response and decode a 200 body in snapshot.body, snapshot.parse, and
snapshot.decode spans with byte size and item counts. Other statuses replay
through the typed client, so validation and errors are unchanged.

The connection trace recorder gains recordSpan for spans timed outside
Effect, such as React commits.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8bec31e)
Records a react.commit span for Profiler commits of 4 ms or more (app root,
home, and thread screens), using the profiling renderer in release builds.
A client.app.resume span names the screen open at launch and each return to
the foreground, and cache.encode splits cache serialization from the SQLite
write.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8097937)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 1609bd5)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 338854b)
iOS throttles and then suspends a backgrounded app, so Profiler wall-clock
durations for commits that run around a background transition report seconds
of descheduled time as render cost. Tag each commit with AppState and record a
client.app.background marker so traces can separate real render cost.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e1d3d6c)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f4b1767)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 533e8c0)
HomeRouteScreen passed an inline onNewThreadOnBranch to every thread row, so
each home render rebuilt renderItem and re-rendered every mounted row. Make
it a stable callback and drop the HomeScreen props it never read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 9df7bc1)
A full shell snapshot replaced every thread and project object, so each
reconnect re-rendered every thread row. Reuse previous objects that are
deep-equal, and the previous lists when nothing changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 72e2612)
useEnvironmentQuery returned a fresh view object on every render, so git
status, branch lists, and every callback depending on them changed identity
whenever the thread screen rendered, defeating the memoized feed, composer,
and header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 07f2137)
The thread screen handed its memoized detail, feed, and composer a new
reconnect callback, connection tone, content presentation, and empty
pending-input drafts on every render. Each streamed event re-rendered all
of them. Share constant values and give the thread screen a reconnect
hook that does not subscribe to connection state.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit d27e0c2)
Every keystroke replaced the draft's attachments array and the whole drafts
record. The thread list under the thread screen subscribed to every draft to
find new-task drafts, and composer callbacks listed the drafts record as an
unused dependency, so each keystroke re-rendered the list and recreated the
composer's handlers. Keep the attachments array when nothing is dropped,
derive new-task drafts in their own atom, and drop the unused dependency.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 02fc755)
…med event

The thread screen refreshed git status whenever the thread shell object
changed, and the shell replaces it on every streamed update. Each refresh ran
git status on the server and invalidated the branch list, so one open thread
issued a refreshStatus and a listRefs RPC per update (36 of each, about 900
server git commands, in a 24-second turn) and re-rendered the screen each
time. Key the refresh by thread identity, and drop the git overview sheet's
extra refresh that ran on every render.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 6c378e7)
The send handler closed over the thread detail and its feed, which change on
every streamed event, so the memoized composer re-rendered on each one. Read
those values at press time instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 667d28e)
…d equivalence

Equivalence.Record(Equivalence.strictEqual()) is the same per-key reference
check the two atoms spelled out by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 2c9d44b)
…he catalog command

Use the catalog's retry command directly, as other screens do, instead of a
dedicated hook, and keep useRemoteConnections as it was: no frequently
rendering screen reads it any more.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit f7ae449)
Keep the new-task navigator mounted so repeated control launches cannot lose consumed destination parameters and return to the project picker. Start a fresh project-bound draft and remount only the composer, preserving pending-task edits. Verified with a native UI test that backgrounds a typed draft and opens the same link again.

(cherry picked from commit 0788229)
Read the initial draft destination before presentation, retain its project identity while the catalog loads, and mount the editor with the sheet. Keep the editor read-only until its draft exists. Cold launches use the same no-transition presentation as warm launches. Verified cold-start video and zero-duration keyboard events, focused navigation tests, mobile typecheck, and native compose UI checks.

(cherry picked from commit e61aaf0)
Personal builds now take T3CODE_IOS_SIGNING=personal with T3CODE_IOS_TEAM_ID
and T3CODE_IOS_BUNDLE_ID, replacing the free Personal Team and SideStore modes.
A paid team signs app groups and extensions, so widgets and the share extension
ship in every build. Push, T3's associated domains, and Sign in with Apple stay
off because they depend on T3's team. The ClerkRedirectUrl workaround for T3
Connect sign-in is gone; personal builds pair directly, for example over
Tailscale.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 068b5df)
(cherry picked from commit 29d72e2e3407a35d47053ac024c00e609a6cff8b)
…ilds

The About screen's version row reads "1.1.0 (0.0.42) · abc123def" with the
build date beneath it, so an installed Personal Team build names the
release and commit it came from.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 894ef31)
(cherry picked from commit f219cd903fbe05f86beaa1169eb67ee206e59551)
@ahalekelly
ahalekelly force-pushed the main branch 27 times, most recently from beedaa5 to a14d824 Compare October 30, 2026 09:20
@ahalekelly

Copy link
Copy Markdown
Owner Author

🤖 Generated with Claude Opus 5.5

Upstream fixed this in pingdotgg#25184, which v0.1.5 includes, so the fork's main already carries it.

@ahalekelly ahalekelly closed this Oct 30, 2026
@ahalekelly
ahalekelly deleted the fix/hidden-desktop-tab-snapshot branch October 30, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant