Skip to content

Record a task in a Bot's own browser: browser controls and the Recorder built in - #868

Merged
davidmckayv merged 500 commits into
mainfrom
david/bot-browser
Oct 29, 2026
Merged

davidmckayv merged 500 commits into
mainfrom
david/bot-browser

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Stacked on #858 (the Recorder and the OpenBot Browser), into its branch.

What this changes

A Bot's screen, opened larger from a conversation, becomes a browser a person can work in, and recording there replaces "Teach a browser workflow". A skill can now be recorded inside OpenBot, in the Bot's own browser, with no extension to install and no Connect step.

  • Browser controls. Tabs, an address bar, back, forward and reload above the Bot's screen. Live for the person holding control (keyboard included: the WAI-ARIA tabs pattern, Enter in the address bar hands the keyboard back to the page); read only while the Bot has control, showing where it is. The bar is shared with the person's own Browser page (app/src/components/browser/browser-bar.tsx).
  • The Recorder in every Bot's browser. On the managed computer, a Bot's bundled Chromium launches with the Recorder (--disable-extensions-except and --load-extension, as Playwright documents and as the hosted person's browser already loads it), kept current, driven over the browser's DevTools pipe, and idle until someone records: it injects nothing into a page and holds no connection until then.
  • Record and Stop replace "Teach a browser workflow". Name the task, Record, do it, Stop. The panel shows the name, time and step count, asks about narration before the first recording (the same account preference as the Browser page, on-device recognition only), and after Stop says what was saved to the person's Learning space. The recording lands in Intelligence exactly as the Recorder sends it from any browser.
  • Only the person is recorded, never the Bot. Record works only for the person holding control: the computer now keeps who took the wheel (heldBy), and ends a recording the moment the wheel leaves them (hand back, a request ending, a reset or restart), waiting for the recording's last events to reach Intelligence before the Recorder disconnects (bounded at twice the gateway's own timeout). A person's own take is now recorded as source: "manual", so its request text is never shown back as the Bot's reason.
  • One connection per recording. Record mints a Recorder connection (via bot-browser) for the person who pressed it and hands it to the Recorder over the DevTools launcher; Stop revokes it, and one the computer ended on its own is revoked on the next read. Settings, Browsers lists it as "'s browser" while it lives and says "recording now" for any browser that is recording (it now refreshes while open).
  • The Bot's network while recording. The Bot's pushed network policy also allows OpenBot's and the Intelligence gateway's origins (host and port only) while a recording runs in its browser, so the Recorder can reach them; the Bot's own policy returns when it ends.
  • Removed: the old demonstrations (the panel, /api/demonstrations, their store and scheduling, and the computer's gesture capture on the live screen). Skills already saved from it stay as ordinary skills; the demonstrations table stays, unread, with no migration.

Contract: shared/bot-browser.ts.

Where it runs

  • New state that outlives a request?
    • Who holds the wheel (heldBy) lives in the computer's existing control store, beside holder, on the computer that runs the Bot's browser.
    • A recording's connection is a row in the existing recorder_connections (via bot-browser, install id derived from the Bot), so any replica can find and revoke it.
    • The network widening is computed from those live rows at every policy push, the on-wake one and the periodic one alike.
    • The Recorder's report and the last stopped recording live with the browser on its computer, as the person's browser's do.
  • What happens on the second replica? Any replica serves the routes: the recording runs on the computer and its connection is in Postgres, so a recording started through one replica is stopped, narrated or read through another.
  • Anything serialised? One recording per Bot's browser, enforced by the computer that runs it (one process per browser). Minting replaces the earlier connection for the same Bot and person under The OpenBot Recorder and the OpenBot Browser: teach a Bot by doing it in your own browser #858's transaction lock and the live-install unique index.
  • Anything fanned out to a browser? No. The app polls GET /api/computers/:botId/browser once a second while the screen is open; the live screen is the existing stream.
  • New listener, port, or schedule? No listener or port. Routes /api/computers/:botId/browser/* behind the same ingress, each with a per-request timeout covering its own computer waits, so none ends in an empty reply.

Boundary and audit

  • Every acting call of the Bot still goes through the gateway. The bar and Record act for the person holding control, as the existing live-screen input does; the computer refuses them without the wheel, and the Bot's network filter applies to every request the browser makes.
  • The Bot-browser connection is audited as recorder.connected and recorder.revoked with its via and reason (stopped, ended, not-started), in the same transaction as the change.
  • The person comes from the session, the wheel's holder from the computer, and the connection is minted on the server; nobody but the person whose recording it is can stop or narrate it, or see its name and counts.

Changelog

  • CHANGELOG.md under Unreleased: "Record a task in a Bot's own browser, with nothing to install".

Proof

Run against the real product: a single-user OpenBot, its managed computer (bundled Chromium, headless on macOS, shown through the live screen), real Google Chrome driving the app, and a real local Intelligence.

  • Gate on this commit: format, lint, both typechecks, build, migrations (no drift), test:ci 10,498 pass and 0 fail, CI's real-browser computer list 115 pass and 0 fail, every Docker image built and booted. The one failure is the desktop Rust test that needs port 3001, which fails the same way on main on that machine.
  • End to end, three runs (the last on this exact commit):
    • The Bot's browser launches with the Recorder idle and disconnected, and the Bot still opens and reads pages as before.
    • With the Bot at the wheel the bar is read only and Record says to take control; taking control makes it live; addresses, back, forward, reload, new, switch and close tab, and the keyboard all work; no browser shortcut reaches Chrome.
    • A recording across an AG-UI demo chat and a public form lands in the person's Learning space with every step's targeting, the AG-UI events in CopilotKit's shape, recording.started and outcome; typed passwords and emails arrive masked; Stop answers in about 300 ms and the tail is never cut.
    • Handing control back mid-recording ends it cleanly ("The recording ended when control went back to the Bot"), and the Bot carries on from the tab the person left in front.
    • A site left out mid-visit sends nothing of itself, its address in a later from included.
    • A reload still shows "Saved to your Learning space"; Settings, Browsers shows the Bot's browser only while it records.
    • Light and dark screenshots of every state match the rest of the app.

… a policy value of the wrong type, which then reads as no restriction: a schema names exactly one type, so IT is told to check chrome://policy, and a test keeps the schema to the rules Chrome loads
…he wrong type, which then means no restriction, so IT must see status OK in chrome://policy; only a value of the right type that the Recorder cannot use is refused
…s to it in the changelog, .env.example and the docs index
…the release tree, so the check accepts the Recorder extension's package.json and still fails when a workspace's manifest is missing
… shared length, the longest that fits, so the larger body is cut first and a small response survives a large upload whole instead of being emptied before the request is touched
…er instead of the offscreen document calling chrome.runtime.getManifest, which Chrome does not give an offscreen document, so kept recordings are sent again after a reconnect or a browser restart; the chrome fake now gives each context its own chrome, the offscreen document only runtime messaging, getURL and id, and a new test runs the real service worker and document together through both paths
… still stands and end the recording through the sign-out path on a revoked or signed-out answer, so a browser revoked in OpenBot's Settings stops streaming within 30 s instead of at the next popup open or 12-hour keep-alive; an unreachable OpenBot or an error page keeps the recording going, and a restarted service worker resumes the checks
…n in ten minutes so an ordinary connect, reconnect, Disconnect and revoke run never meets it, and answer a connect over the limit as throttled with the time left instead of as busy, so Settings says to try again in so many minutes rather than that another connect is under way
… number or typed secret in spoken numbers, which speech recognition writes with commas, dashes or a currency sign, then runs the usual text masking
…he offscreen document opens the microphone and runs the browser's own speech recognition on it, each final phrase goes into the recording as a masked transcript event through the same outbox and stream, and Stop, Disconnect, a sign-out or a policy move ends it with the recording; OpenBot receives nothing of it
…itle says it is listening with the last words heard, or why it is not in a plain sentence, with a switch to mute it; the first Record opens a microphone page once to ask for the lasting grant the hidden document needs, and Record sends the page's language
…cript now redacts every spoken run of six digits or more whatever its checksum (a date said in full excepted), and every number after a word that names a secret (card, PIN, CVV, account, password and the like) however short, including in the phrase after it, since Chrome splits a phrase at a pause; the popup's interim words go through the same masking with the phrase before them, so a card number read aloud no longer reaches Intelligence or the Voice row as "4012-888888-1881" or "4012-8"
…no words: as Edge does, recognition starts, hears speech and raises no error while every final phrase comes back empty, so the row said Listening forever; now, 20 s after speech is first heard with no phrase in words, the row says this browser is not turning your speech into words and recording carries on without narration, keeps listening, and drops the sentence when a real phrase arrives
… Record records: website access is now a required host permission that the install grants (the debugger permission already implies every site, so Chrome and Edge show the same install warnings and an update is no privilege increase), and a fresh install opens one setup page that asks for the microphone as it loads, with a button to ask again and the answer in plain words, worded for Chrome's or Edge's prompt; the access page, the website-access prompts at Connect and at Record, and the microphone tab the first Record opened are gone, Record never opens a tab, and a recording without the microphone goes on while its Voice row says the microphone is not allowed and offers the setup page on a click
…s joins no Learning space: the popup said Learning may not see it whenever OpenBot could not check the Bot's space with Intelligence (Learning delivery off, its skills endpoint answering 404, or Intelligence not answering), although the pass still joins the Bot's container and the trajectory lands in it; one shared rule, joinedLearningContainer, now decides both which container the server's selector joins and when the popup warns, so a Bot with no space, a container Intelligence says does not exist, or settings OpenBot could not read still warn, and after Stop an unchecked space says the recording goes into the Bot's Learning space
…s under the sentence, as on the popup's Voice row, instead of squeezing it, the sentence keeps the muted colour the popup uses for the same state, and how to answer the prompt shows only while there is a prompt to answer
…one-time-code is now sensitive by its id, name, placeholder, label or aria-labelledby text, test id, class or short numeric shape (zip, postal, promo, coupon and country codes excepted), so its value, every attribute that echoes it and its digest are masked; verificationCode, otp, mfa_code, smsCode, authCode and like keys are credentials in bodies and URLs, and a bare code is redacted when it holds 4 to 10 digits or sits on a sign-in, verification or token request, as are code and state on an OAuth callback, so a recorded sign-in no longer sends its code to Intelligence in the input event or the request body
…s under one parent, wrapper or form (or numeric boxes side by side in one parent) is one sensitive code field though no box has a label, so every box's value is redacted in its input events, and the code the boxes make together, in order, is hashed on each input, after a paste the page spreads itself and when a box loses focus, so it is masked in any body; a lone numeric one-character box is sensitive on its own too
…harness catalogue paths and the Recorder's extension id file
…, XHR or top-frame document request that starts within a second of a click, edit, submit or key press in that tab, or that the response of such a call starts in turn (two links, three seconds at most), on both the debugger and the page-hook path; analytics, telemetry, ads and monitoring endpoints, prefetches, aborted calls, a call the tab already made (polling), and anything the page starts while the person does nothing are dropped before any body is read. The content script reports each action's time (never its detail) for this, and records a form submit and Enter in a field as their own events, masked like a click. Replayed on two real recordings of a CRM this halves the network events and drops every beacon, while the calls behind each click (a contact loading, a question sent to the assistant) stay.
…y, once per address: an iframe loading or rewriting its own address, and a page Chrome prerenders out of sight, are the page's doing and no longer recorded, and a single-page app setting the same address again and again records it once (a reload is recorded again, as is the reloaded page). On the two real CRM recordings this takes navigation events from 94 to 18 and from 47 to 11.
…rder: every recording a person makes joins their own Intelligence Learning container (user- and 32 hex digits of the SHA-256 of their OpenBot user id), made on first need with the credential that makes the openbot container (a service token on local evaluation, or a CLI login on managed Intelligence), when the browser connects and again on a capture pass, idempotently across replicas because Intelligence's container id is unique; the popup loses its Bot picker, the x-openbot-recorder-bot header, canRecordForBot, the no-Bot and Bot-forbidden refusals and /api/recorder/bots go, recorder_trajectory_bindings binds a trajectory to its person only (another person's second pass is not found), and the warning that Learning will not see a recording shows only when the person's space cannot be made or reached
…nd Settings, Learning shares them: a run gets the Bot's own learned Skills, then the person's own, then the Skills other people opened, the Bot's own winning a name clash and then the person's own, with two people's opened Skills of one name both kept as <their space>/<name>; a person's or an opened space that is missing, refusing or slow never fails a run and is left alone by runs for a minute; each person chooses only their own Bots (the default), every Bot, or the Skills they open one by one, stored in learning_skill_sharing and learning_open_skills, read by runs from an in-memory index that the saving replica refreshes at once and the others within 30 seconds, listed from the same Intelligence call delivery uses, with an unavailable line when Intelligence does not deliver them, and audited as learning.sharing_changed, learning.skill_opened and learning.skill_closed
… secret: the masking tests searched the whole serialised event for a secret's digits, so CI failed the 2 MiB form test when Date.now() read 1792433411150 and its "4111" matched the test card though the body was masked in full. Leak checks now search recordedText, the event with its timestamp, completedAt and durationMs set to 0 (and the page test's server port left out), which throws on any other clock-sized number so a new clock field cannot slip past; the tests that search events pin Date.now() to a reading holding their secrets' digits, so a check that read the clock again fails on every run
…ch click, edit, choice in a list, box ticked, form submit, Enter in a field and top-frame page change, in order, beside the narration and the outcome, so an agent can be trained to do the same clicks and enter the same data. Each step carries what an agent needs to find that element again (its accessible role and name, input type, a ranked list of stable selectors from test attributes and hand-made ids down to a short CSS path, never a generated id or a class name, its frame path inside an iframe, the page and its title, its box and the viewport), built in the page before masking so the masking covers it. No network call is recorded on its own: a step that saved something carries the one request it sent to save it (a PUT, PATCH, DELETE or GraphQL mutation, or a POST that carries what was typed or chosen or names a write; a configuration read through a POST, a query, telemetry and polling never), masked and cut to fit, the step held until its window closes or the next step starts and then sent with the request in place, steps in the order they came, on the debugger and the page-hook path alike.
…ecords each step you take (clicks, typing and the pages you open) with what it saves, and what you say; the Network row reads Capturing what each step saves (from inside the page when Chrome refused its debugger); the changelog says the same.
…hose path or operation names presence, a heartbeat, a ping, a keepalive, typing, a session refresh, telemetry, tracking or a log is never a step's save, nor is a read receipt the page posts when it shows a conversation (a PUT that marks a notification read stays the person's), so opening an assistant carries nothing; and page changes the page makes one after another, with no step of the person's between them and each within 2 s of the last (a sign-in's redirects, an app settling on its default view), are one navigate step to the final page, from the address the chain began at, while a reload or a step back is always its own.
…inside a button, link, menu item, tab, option, checkbox or another element a person acts on takes that element's role, name and selectors, keeping the box of what was clicked, and a click that meets none (a panel's empty space or text) names the clicked element by its own first 60 characters; every name read from text keeps a space between its pieces, so "Sign in" and "or create an account" no longer run together, and text a page hides from everyone is left out.
…a run's request, each event of its stream, and the stream's end, over any transport
… AG-UI, add a click's agent context, close a tab's Threads when it navigates, and stop
Every row's Disconnect dialog said OpenBot signs the Recorder in this browser out, on another
browser's row too. It now says this browser only on the row of the copy in the browser the page is
open in, and names every other by the name its row shows (your OpenBot browser for that one).
Tests and a few of Desktop's comments said "the code-owner review", "blocker 3" or "end-to-end run 1
of the review fixes" where the reason belonged. Each now says the defect it pins or what a person
saw, in the code's own terms. A guard reads the sources and tests of those parts for the phrases.
…ign out half way

Sign-out no longer fails open. Ending the person's Recorder connections is
retried three times; when it still fails, recorder.sign_out_failed goes on
the audit trail and the sign-out does not happen: the person stays signed
in and is told "Signing out did not finish. Try again." That follows the
code's own rule for ending access: a removal ends sessions and deny-lists in
one transaction, a removed person's linked accounts retire with their rows
in one transaction, and a connection ends exactly when its row is written.
A sign-out reported done while a Recorder went on recording as the person
would break it. With Better Auth the Recorders end in a before hook on
/sign-out, since the sign-out route itself answers success when deleting the
session fails; in organization mode they end before the organization is
asked to end its session. The app's Sign out now says the sentence in a
toast and stays put instead of failing silently.

Every other end of a sign-in ends the Recorders too: Better Auth's session
delete hook (signing out everywhere, ending one or the other sessions, a
password reset or change, SCIM deactivation, single logout, an account
proving the address), a person's removal (with what else a removal
retires), and SSO-required being turned on (before the sessions, and for
everybody holding a connection, so saving again finishes a failed save). In
organization mode a connection ends when the organization refuses the
session it holds a copy of, however that session ended there. An expired
session that Better Auth tidies away is not an end, and ends nothing.

The organization session cookie now carries its own expiry, set at the
handoff and enforced when it is read, and a connection's sealed copy ends
at that expiry (kept in organization_session_expires_at, never later than
the cookie's Max-Age from connecting) instead of a week from connecting.

The docs say a Recorder connection's lifetime, and that recordings cannot
be deleted from OpenBot yet and stay in Intelligence after the person is
removed, because Intelligence has no API a server credential can call for
either.
… from ending in silence

After a person took over a Bot's browser and handed back, the Bot was told only "The person
handed control back. This fresh snapshot describes the page now.", and often answered with
nothing: replaying the captured requests, 15 of 108 answers were blank, all of them after a
help request the Bot made without a word first. computer_request_help drew nothing either,
so such a turn showed nothing at all.

The tool result now says what the person did and asks the Bot to carry on, saying what it
does next first (handedBack, ahead of the page in the result). Replaying the same 36
requests three times each with it: 0 of 108 blank.

The request is drawn where it happened: while open, the Bot's screen with what it needs and
the wheel; once answered, one line saying what was asked and what came of it. If the turn
still ends there with no words, the line says so and offers Carry on, which asks the Bot to
go on as the person would. The transcript tells a drawn tool call where it sits
(ToolPlaceContext) so the card can know the turn ended on it.

Leaving a conversation while holding control left it blank on return: the waiting call
belonged to the chat that closed, so its answer and the Bot's reply went on in that chat's
turn, unseen, and the new chat offered "A browser task is paused" and Resume task. A call
that takes its answer now marks the request answered in this tab, so nothing offers to
resume it, and the card says the Bot is carrying on; a turn that ends after its chat
closed reports its reply to the roster and tells the chat open now to read the thread again
(turns-finished-away), so the reply shows without a reload.
…aving `via` for how a connection was made

`via` on Recorder rows says how a connection was made (web, desktop, openbot-browser), as `recorder.connected` and
`recorder.connect_refused` carry it. The sign-out follow-ups also put the way a person's sign-in ended (a sign-out,
sessions ended another way, the person removed, SSO required, the organization) in `via` on `recorder.revoked` and
`recorder.sign_out_failed`, so one field meant two things. That is `ending` now, through the store, the sign-out and
every caller, and the trail's descriptions say so. The organization ending a session is pinned on the trail too.
…, and the browser starts fresh

A browser's data can go wrong, so both browsers in OpenBot (a Bot's, in its screen, and the person's own Browser page)
get Clear browser data beside reload. It asks once, in one sentence: cookies, saved passwords and history are wiped,
and you'll have to sign in everywhere again. Cancel does nothing; Clear wipes it, the browser starts again fresh, and a
toast says so.

What goes is the browser's whole profile: the computer closes the browser and deletes its user data directory, which
takes cookies, cache, saved passwords, history, site storage and the Recorder's own storage with it. DevTools' clears
(Storage.clearDataForOrigin, Network.clearBrowserCookies) cover one origin's site storage or the cookies only, and
leave history, saved passwords and extension storage behind. The Bot's files on the computer stay, and so do OpenBot's
Passwords. A recording running there ends first, and the last of it is waited for, since what is unsent lives in the
storage being wiped.

A Bot's browser is cleared only by the person holding its wheel, checked by the server and again by the computer, so
it is never wiped under the Bot. The person keeps the wheel: the close is not announced as a restart, the live screen
follows the fresh browser, and the computer starts a new run so no ref from before resolves. The server then writes
computer.browser_cleared and forgets the snapshot and screenshots it kept of the browser's pages, as a reset does, and
revokes the recording's connection the clear ended. The person's own browser opens again fresh when it was open, its
Recorder connected anew, and is audited the same way.

Routes: /bot-browser/clear and /browser/clear on the computer; POST /api/computers/:botId/browser/clear and
POST /api/browser/clear on the server, each kept open for its waits.

Toasts are drawn above dialogs now: on the same layer, the Bot's screen covered a toast raised while it was open.
…rowser

Clearing the person's own browser deletes its profile, and with it the Recorder's token. A browser that was open
opened again with a new connection, which replaced the old one; a closed one left its connection live on the server
for a token nobody held any more, still listed in Settings, Browsers as connected until the next open.

So the clear now ends that browser's connection right after the wipe, open or closed, with its recorder.revoked row
in the same transaction and a reason of its own, cleared. A connection that cannot be ended is said to the person as a
clear that did not finish, and clearing again ends it.

@guidovizoso guidovizoso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core holds up well:

  • The wheel's holder is set server-side from the session, and the computer checks it too.
  • Each recording gets its own connection, audited in the same transaction.
  • Narration stays on-device and off by default.
  • The 3.7k deleted lines are the old demonstrations feature, so none of #858's safeguards were removed.
  • The tests run against a real database and a real Chromium.

Requesting changes for one blocker.

Blocker

Record works on a shared computer, where other Bots can read the person's Recorder token. The Bot's browser is mounted wherever Bots have computers (server/src/index.ts, botBrowser), and nothing on that path calls personBrowserRefusal (browser/host.ts). On Record, the Recorder in the Bot's Chromium saves the person's token with writeLocal("connection", …) (extension/src/background/connection.ts). That lands on disk under the Bot's profile, which /exec from any other Bot on a shared computer can read. A prompt-injected Bot then has a live token into that person's Learning space until Stop. A recording still unsent when the send wait runs out also stays there (bot-browser-host.ts:236). This is the SHARED_COMPUTER threat from #858. Please refuse Record the same way. This also needs to line up with #871, which changes how isolation is decided.

Should fix

  • The Bot can act again before the recording has stopped. In agent-computer/src/control.ts, release() calls syncAdmission() before onWheelLeft(), and ending the recording queues behind a Record that is still starting. If the person hands back during a start, a routine on the same Bot acts at once, and its clicks go into the person's recording. Please keep the Bot's actions closed until the recording has settled.
  • Hand-back, cancel, stop and reset can end in an empty reply. They now wait on the recording, up to the 20 s send wait. But /api/computers/:id/control/{release,cancel} and /computers/{stop,reset} aren't in botBrowserRequestSeconds (http-server.ts), so Bun closes them at 10 s. The hand-back happens, but the app reports it as failed.
  • A hand-back whose reply fails leaves the connection live. wheelReturned only runs when releaseControl succeeded (computer/routes.ts:516). After a timeout, the connection and the wider network stay until someone opens that screen. Please call it in a finally, or sweep on the policy push.
  • Someone else's counts still leak. viewFor (browser/bot-browser.ts) spreads ...common, which includes actions and phase, so a second viewer sees the recorder's step count.
  • Scope. The help-request card and hand-back reply, hiding person-filled fields from the Bot, and Clear browser are each reasonable. But together they make 11k lines hard to check against #858's guarantees. Please split them out.

Nits

  • "Too many connection codes were asked for at once" is shown after pressing Record, where the person asked for no code.
  • bot-browser-host.ts:320 passes the extension's raw error text to the person.
  • Each viewer's once-a-second read makes two computer calls and a query, and during a recording it also asks the Recorder over DevTools.
  • A Stop can take longer than the 45 s computer call timeout, and the connection is then revoked only after START_GRACE_MS.

Resolved conflicts:
- CopilotKit 1.83.0 from main (app, server, toolkit), and the lockfile regenerated.
- The resolver keeps #858's person-bound learned Skills and main's name context.
- Admin keeps the Skill sharing switch under main's reworded Network copy.
- The changelog's upgrade list is counted from what it now holds (fourteen).
- The tool-capture test helpers and both CI test fixes take main's versions.
- The assertion-shape guard points at the shifted line.
Brings in the skins (#856). The account keeps both the narration choice and
the skin, the sidebar draws the shared navigation list with the Browser button
after Connections, and PageHeader carries the page-header and page-title hooks
the skins read.
Base automatically changed from david/recorder-and-browser to main October 28, 2026 21:41
…nd keep the Bot out until a recording ends

Record hands the Recorder in the Bot's browser the person's own connection, kept in the Bot's profile
until Stop. On the shared computer any other Bot's shell can read that profile, so Record is now
refused there by the server (before a connection is minted) and by the computer, and the Bot's screen
says why in place of Record. With a supervisor or a Sandbox each Bot has its own computer and Record
works as before.

Also from the review:
- The computer admits no Bot action after a hand-back, a cancel or an interruption until the
  recording the wheel took with it has ended, including one still queued behind a Record that was
  starting.
- Hand-back, cancel, stop and reset are kept open past Bun's 10 s, since they wait for the recording
  to end; Stop gets its own 90 s call to the computer.
- A hand-back or cancel whose answer is lost still sweeps the Bot's browser: whatever the computer
  says now decides which connections are left behind.
- Someone else's running recording no longer shows its step count or microphone state.
- The Recorder's raw connect error stays in the log; the person is told in OpenBot's words. The
  too-many refusal after Record no longer talks about connection codes.
Takes #872's removal of the form-submit source test, and keeps the
demonstrations entry out of the assertion allow-list, since this branch
removes that feature and its test.
@davidmckayv

Copy link
Copy Markdown
Contributor Author

@guidovizoso thanks for the review. Addressed in 7b18393, with main merged since (CI green). Each fix has a regression test that fails before it and passes after.

Blocker: Record on a shared computer. Refused now, twice over:

  • The server refuses Record with 403 before it mints a connection, wherever Bots share one computer (botRecordRefusal in browser/host.ts, wired as recordRefused). The Bot's screen shows the reason in place of Record. The bar still works.
  • The computer refuses it too when it is the shared one (NOT_ITS_OWN_COMPUTER_RECORD in agent-computer's bot-browser-host.ts), so neither side can be talked into it alone.
  • Checked end to end on the hosted shape in a real browser. The refusal shows on the Bot's screen. A Record forced through the API gets 403 from the server and from the computer. The token never reaches the Bot's profile on disk.
  • With One computer per person, with Isolated Bots on request #871, a person's computer holds only that person's Bots' work, as is the industry standard. So once both land, Record is refused only on the computer every person shares (Helm shared). I'll line the rule up in whichever of the two merges second.

Should fix

  • The Bot acting before the recording settled. control.ts now counts the endings onWheelLeft starts and admits no Bot action until they settle (RECORDING_ENDING), including one queued behind a Record that was still starting.
  • Empty replies. control/release, control/cancel, computers/stop and computers/reset are now in botBrowserRequestSeconds.
  • A hand-back whose reply fails. Release and cancel call wheelMayHaveReturned on failure. It reads the computer and revokes every connection no recording there can be using, the same sweep a read does.
  • Someone else's counts. viewFor no longer shows another person's actions or voice.
  • Scope. Kept as one PR.

Nits

  • "Too many connection codes" after Record now reads "Too many recordings were started at once."
  • The extension's raw connect error goes to the log; the person gets OpenBot's own sentence.
  • Stop now has its own 90 s call to the computer, so a slow Recorder still answers and its connection goes at once.
  • The once-a-second read is unchanged in this round.

guidovizoso
guidovizoso previously approved these changes Oct 29, 2026

@guidovizoso guidovizoso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at b9a270b5 (fixes in 7b18393b). Approving.

Verified fixed:

  • Record on a shared computer is refused on both sides: the server returns 403 before minting a connection (botRecordRefusal), and the computer refuses Record unless it was started for one id (computerOwner in person-access.ts).
  • The Bot acting before the recording settled: control.ts counts the endings before it re-syncs admission in release and cancel, and a Bot action meanwhile gets RECORDING_ENDING.
  • Release, cancel, stop and reset have the longer request timeout. A failed release or cancel calls wheelMayHaveReturned.
  • viewFor no longer shows another person's counts. The copy and raw-error nits are fixed, and Stop has its own 90 s call.

One condition for when this and #871 meet: keep Record limited to a computer started for one Bot, i.e. isolated Bots, which is what the computer already enforces. On a person's computer under #871, every other Bot they use, a public Bot someone else wrote included, can read the Recorder token in that Bot's profile. So "refused only on Helm shared" isn't enough there. As written, botRecordRefusal would refuse every #871 provider, since per-bot no longer exists, so whichever PR merges second needs to line it up this way.

Not blocking: the once-a-second read per viewer, and the three extra features kept in this PR.

Brings in #879: a Bot's browser now also closes when the computer's memory
runs high and its idle background tabs close, sparing any browser whose wheel
a person holds, so a recording is never cut off by either.
@davidmckayv
davidmckayv merged commit 6e28921 into main Oct 29, 2026
20 checks passed
@davidmckayv
davidmckayv deleted the david/bot-browser branch October 29, 2026 17:47
davidmckayv added a commit that referenced this pull request Oct 29, 2026
Brings in a Bot's own browser with Record (#868) and closing browsers when
memory runs high (#879). Conflicts are resolved here; the Bot's browser is
moved onto per-person seats in the next commit.
davidmckayv added a commit that referenced this pull request Oct 29, 2026
… a person's own computer

A Bot's browser (#868) is now a seat's, like everything else on a computer: each
route resolves the asker's seat for the Bot and reaches that browser, on their
own computer or on an isolated Bot's computer of its own, with the person named
to the computer where people share one. Recorder connections are still filed
under the Bot, but a seat on a person's computer reads, hands back and clears
only that person's: before, Ben opening his own Bot screen ended Ada's live
recording on her computer as left behind. The Bot's network policy is pushed to
the seat's computer.

Record is refused only on the computer every person shares. A person's computer
is now told whose it is (COMPUTER_PERSON_ID, from the supervisor and the Sandbox
template), so it records in its Bots' browsers there and still refuses a
person's own browser.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants