Repository navigation
Record a task in a Bot's own browser: browser controls and the Recorder built in - #868
Conversation
… a policy value of the wrong type, which then reads as no restriction: a schema names exactly one type, so IT is told to check chrome://policy, and a test keeps the schema to the rules Chrome loads
…he wrong type, which then means no restriction, so IT must see status OK in chrome://policy; only a value of the right type that the Recorder cannot use is refused
…s to it in the changelog, .env.example and the docs index
…the release tree, so the check accepts the Recorder extension's package.json and still fails when a workspace's manifest is missing
… shared length, the longest that fits, so the larger body is cut first and a small response survives a large upload whole instead of being emptied before the request is touched
…er instead of the offscreen document calling chrome.runtime.getManifest, which Chrome does not give an offscreen document, so kept recordings are sent again after a reconnect or a browser restart; the chrome fake now gives each context its own chrome, the offscreen document only runtime messaging, getURL and id, and a new test runs the real service worker and document together through both paths
… still stands and end the recording through the sign-out path on a revoked or signed-out answer, so a browser revoked in OpenBot's Settings stops streaming within 30 s instead of at the next popup open or 12-hour keep-alive; an unreachable OpenBot or an error page keeps the recording going, and a restarted service worker resumes the checks
…n in ten minutes so an ordinary connect, reconnect, Disconnect and revoke run never meets it, and answer a connect over the limit as throttled with the time left instead of as busy, so Settings says to try again in so many minutes rather than that another connect is under way
… number or typed secret in spoken numbers, which speech recognition writes with commas, dashes or a currency sign, then runs the usual text masking
…he offscreen document opens the microphone and runs the browser's own speech recognition on it, each final phrase goes into the recording as a masked transcript event through the same outbox and stream, and Stop, Disconnect, a sign-out or a policy move ends it with the recording; OpenBot receives nothing of it
…itle says it is listening with the last words heard, or why it is not in a plain sentence, with a switch to mute it; the first Record opens a microphone page once to ask for the lasting grant the hidden document needs, and Record sends the page's language
…cript now redacts every spoken run of six digits or more whatever its checksum (a date said in full excepted), and every number after a word that names a secret (card, PIN, CVV, account, password and the like) however short, including in the phrase after it, since Chrome splits a phrase at a pause; the popup's interim words go through the same masking with the phrase before them, so a card number read aloud no longer reaches Intelligence or the Voice row as "4012-888888-1881" or "4012-8"
…no words: as Edge does, recognition starts, hears speech and raises no error while every final phrase comes back empty, so the row said Listening forever; now, 20 s after speech is first heard with no phrase in words, the row says this browser is not turning your speech into words and recording carries on without narration, keeps listening, and drops the sentence when a real phrase arrives
… Record records: website access is now a required host permission that the install grants (the debugger permission already implies every site, so Chrome and Edge show the same install warnings and an update is no privilege increase), and a fresh install opens one setup page that asks for the microphone as it loads, with a button to ask again and the answer in plain words, worded for Chrome's or Edge's prompt; the access page, the website-access prompts at Connect and at Record, and the microphone tab the first Record opened are gone, Record never opens a tab, and a recording without the microphone goes on while its Voice row says the microphone is not allowed and offers the setup page on a click
…s joins no Learning space: the popup said Learning may not see it whenever OpenBot could not check the Bot's space with Intelligence (Learning delivery off, its skills endpoint answering 404, or Intelligence not answering), although the pass still joins the Bot's container and the trajectory lands in it; one shared rule, joinedLearningContainer, now decides both which container the server's selector joins and when the popup warns, so a Bot with no space, a container Intelligence says does not exist, or settings OpenBot could not read still warn, and after Stop an unchecked space says the recording goes into the Bot's Learning space
…s under the sentence, as on the popup's Voice row, instead of squeezing it, the sentence keeps the muted colour the popup uses for the same state, and how to answer the prompt shows only while there is a prompt to answer
…one-time-code is now sensitive by its id, name, placeholder, label or aria-labelledby text, test id, class or short numeric shape (zip, postal, promo, coupon and country codes excepted), so its value, every attribute that echoes it and its digest are masked; verificationCode, otp, mfa_code, smsCode, authCode and like keys are credentials in bodies and URLs, and a bare code is redacted when it holds 4 to 10 digits or sits on a sign-in, verification or token request, as are code and state on an OAuth callback, so a recorded sign-in no longer sends its code to Intelligence in the input event or the request body
…s under one parent, wrapper or form (or numeric boxes side by side in one parent) is one sensitive code field though no box has a label, so every box's value is redacted in its input events, and the code the boxes make together, in order, is hashed on each input, after a paste the page spreads itself and when a box loses focus, so it is masked in any body; a lone numeric one-character box is sensitive on its own too
…harness catalogue paths and the Recorder's extension id file
…, XHR or top-frame document request that starts within a second of a click, edit, submit or key press in that tab, or that the response of such a call starts in turn (two links, three seconds at most), on both the debugger and the page-hook path; analytics, telemetry, ads and monitoring endpoints, prefetches, aborted calls, a call the tab already made (polling), and anything the page starts while the person does nothing are dropped before any body is read. The content script reports each action's time (never its detail) for this, and records a form submit and Enter in a field as their own events, masked like a click. Replayed on two real recordings of a CRM this halves the network events and drops every beacon, while the calls behind each click (a contact loading, a question sent to the assistant) stay.
…y, once per address: an iframe loading or rewriting its own address, and a page Chrome prerenders out of sight, are the page's doing and no longer recorded, and a single-page app setting the same address again and again records it once (a reload is recorded again, as is the reloaded page). On the two real CRM recordings this takes navigation events from 94 to 18 and from 47 to 11.
…rder: every recording a person makes joins their own Intelligence Learning container (user- and 32 hex digits of the SHA-256 of their OpenBot user id), made on first need with the credential that makes the openbot container (a service token on local evaluation, or a CLI login on managed Intelligence), when the browser connects and again on a capture pass, idempotently across replicas because Intelligence's container id is unique; the popup loses its Bot picker, the x-openbot-recorder-bot header, canRecordForBot, the no-Bot and Bot-forbidden refusals and /api/recorder/bots go, recorder_trajectory_bindings binds a trajectory to its person only (another person's second pass is not found), and the warning that Learning will not see a recording shows only when the person's space cannot be made or reached
…nd Settings, Learning shares them: a run gets the Bot's own learned Skills, then the person's own, then the Skills other people opened, the Bot's own winning a name clash and then the person's own, with two people's opened Skills of one name both kept as <their space>/<name>; a person's or an opened space that is missing, refusing or slow never fails a run and is left alone by runs for a minute; each person chooses only their own Bots (the default), every Bot, or the Skills they open one by one, stored in learning_skill_sharing and learning_open_skills, read by runs from an in-memory index that the saving replica refreshes at once and the others within 30 seconds, listed from the same Intelligence call delivery uses, with an unavailable line when Intelligence does not deliver them, and audited as learning.sharing_changed, learning.skill_opened and learning.skill_closed
… secret: the masking tests searched the whole serialised event for a secret's digits, so CI failed the 2 MiB form test when Date.now() read 1792433411150 and its "4111" matched the test card though the body was masked in full. Leak checks now search recordedText, the event with its timestamp, completedAt and durationMs set to 0 (and the page test's server port left out), which throws on any other clock-sized number so a new clock field cannot slip past; the tests that search events pin Date.now() to a reading holding their secrets' digits, so a check that read the clock again fails on every run
…ch click, edit, choice in a list, box ticked, form submit, Enter in a field and top-frame page change, in order, beside the narration and the outcome, so an agent can be trained to do the same clicks and enter the same data. Each step carries what an agent needs to find that element again (its accessible role and name, input type, a ranked list of stable selectors from test attributes and hand-made ids down to a short CSS path, never a generated id or a class name, its frame path inside an iframe, the page and its title, its box and the viewport), built in the page before masking so the masking covers it. No network call is recorded on its own: a step that saved something carries the one request it sent to save it (a PUT, PATCH, DELETE or GraphQL mutation, or a POST that carries what was typed or chosen or names a write; a configuration read through a POST, a query, telemetry and polling never), masked and cut to fit, the step held until its window closes or the next step starts and then sent with the request in place, steps in the order they came, on the debugger and the page-hook path alike.
…ecords each step you take (clicks, typing and the pages you open) with what it saves, and what you say; the Network row reads Capturing what each step saves (from inside the page when Chrome refused its debugger); the changelog says the same.
…hose path or operation names presence, a heartbeat, a ping, a keepalive, typing, a session refresh, telemetry, tracking or a log is never a step's save, nor is a read receipt the page posts when it shows a conversation (a PUT that marks a notification read stays the person's), so opening an assistant carries nothing; and page changes the page makes one after another, with no step of the person's between them and each within 2 s of the last (a sign-in's redirects, an app settling on its default view), are one navigate step to the final page, from the address the chain began at, while a reload or a step back is always its own.
…inside a button, link, menu item, tab, option, checkbox or another element a person acts on takes that element's role, name and selectors, keeping the box of what was clicked, and a click that meets none (a panel's empty space or text) names the clicked element by its own first 60 characters; every name read from text keeps a space between its pieces, so "Sign in" and "or create an account" no longer run together, and text a page hides from everyone is left out.
…a run's request, each event of its stream, and the stream's end, over any transport
… AG-UI, add a click's agent context, close a tab's Threads when it navigates, and stop
Every row's Disconnect dialog said OpenBot signs the Recorder in this browser out, on another browser's row too. It now says this browser only on the row of the copy in the browser the page is open in, and names every other by the name its row shows (your OpenBot browser for that one).
Tests and a few of Desktop's comments said "the code-owner review", "blocker 3" or "end-to-end run 1 of the review fixes" where the reason belonged. Each now says the defect it pins or what a person saw, in the code's own terms. A guard reads the sources and tests of those parts for the phrases.
…ign out half way Sign-out no longer fails open. Ending the person's Recorder connections is retried three times; when it still fails, recorder.sign_out_failed goes on the audit trail and the sign-out does not happen: the person stays signed in and is told "Signing out did not finish. Try again." That follows the code's own rule for ending access: a removal ends sessions and deny-lists in one transaction, a removed person's linked accounts retire with their rows in one transaction, and a connection ends exactly when its row is written. A sign-out reported done while a Recorder went on recording as the person would break it. With Better Auth the Recorders end in a before hook on /sign-out, since the sign-out route itself answers success when deleting the session fails; in organization mode they end before the organization is asked to end its session. The app's Sign out now says the sentence in a toast and stays put instead of failing silently. Every other end of a sign-in ends the Recorders too: Better Auth's session delete hook (signing out everywhere, ending one or the other sessions, a password reset or change, SCIM deactivation, single logout, an account proving the address), a person's removal (with what else a removal retires), and SSO-required being turned on (before the sessions, and for everybody holding a connection, so saving again finishes a failed save). In organization mode a connection ends when the organization refuses the session it holds a copy of, however that session ended there. An expired session that Better Auth tidies away is not an end, and ends nothing. The organization session cookie now carries its own expiry, set at the handoff and enforced when it is read, and a connection's sealed copy ends at that expiry (kept in organization_session_expires_at, never later than the cookie's Max-Age from connecting) instead of a week from connecting. The docs say a Recorder connection's lifetime, and that recordings cannot be deleted from OpenBot yet and stay in Intelligence after the person is removed, because Intelligence has no API a server credential can call for either.
… from ending in silence After a person took over a Bot's browser and handed back, the Bot was told only "The person handed control back. This fresh snapshot describes the page now.", and often answered with nothing: replaying the captured requests, 15 of 108 answers were blank, all of them after a help request the Bot made without a word first. computer_request_help drew nothing either, so such a turn showed nothing at all. The tool result now says what the person did and asks the Bot to carry on, saying what it does next first (handedBack, ahead of the page in the result). Replaying the same 36 requests three times each with it: 0 of 108 blank. The request is drawn where it happened: while open, the Bot's screen with what it needs and the wheel; once answered, one line saying what was asked and what came of it. If the turn still ends there with no words, the line says so and offers Carry on, which asks the Bot to go on as the person would. The transcript tells a drawn tool call where it sits (ToolPlaceContext) so the card can know the turn ended on it. Leaving a conversation while holding control left it blank on return: the waiting call belonged to the chat that closed, so its answer and the Bot's reply went on in that chat's turn, unseen, and the new chat offered "A browser task is paused" and Resume task. A call that takes its answer now marks the request answered in this tab, so nothing offers to resume it, and the card says the Bot is carrying on; a turn that ends after its chat closed reports its reply to the roster and tells the chat open now to read the thread again (turns-finished-away), so the reply shows without a reload.
…hows it on return
…requests show and end in a reply
… and what ends a Recorder's connection
…aving `via` for how a connection was made `via` on Recorder rows says how a connection was made (web, desktop, openbot-browser), as `recorder.connected` and `recorder.connect_refused` carry it. The sign-out follow-ups also put the way a person's sign-in ended (a sign-out, sessions ended another way, the person removed, SSO required, the organization) in `via` on `recorder.revoked` and `recorder.sign_out_failed`, so one field meant two things. That is `ending` now, through the store, the sign-out and every caller, and the trail's descriptions say so. The organization ending a session is pinned on the trail too.
…, and the browser starts fresh A browser's data can go wrong, so both browsers in OpenBot (a Bot's, in its screen, and the person's own Browser page) get Clear browser data beside reload. It asks once, in one sentence: cookies, saved passwords and history are wiped, and you'll have to sign in everywhere again. Cancel does nothing; Clear wipes it, the browser starts again fresh, and a toast says so. What goes is the browser's whole profile: the computer closes the browser and deletes its user data directory, which takes cookies, cache, saved passwords, history, site storage and the Recorder's own storage with it. DevTools' clears (Storage.clearDataForOrigin, Network.clearBrowserCookies) cover one origin's site storage or the cookies only, and leave history, saved passwords and extension storage behind. The Bot's files on the computer stay, and so do OpenBot's Passwords. A recording running there ends first, and the last of it is waited for, since what is unsent lives in the storage being wiped. A Bot's browser is cleared only by the person holding its wheel, checked by the server and again by the computer, so it is never wiped under the Bot. The person keeps the wheel: the close is not announced as a restart, the live screen follows the fresh browser, and the computer starts a new run so no ref from before resolves. The server then writes computer.browser_cleared and forgets the snapshot and screenshots it kept of the browser's pages, as a reset does, and revokes the recording's connection the clear ended. The person's own browser opens again fresh when it was open, its Recorder connected anew, and is audited the same way. Routes: /bot-browser/clear and /browser/clear on the computer; POST /api/computers/:botId/browser/clear and POST /api/browser/clear on the server, each kept open for its waits. Toasts are drawn above dialogs now: on the same layer, the Bot's screen covered a toast raised while it was open.
…rowser Clearing the person's own browser deletes its profile, and with it the Recorder's token. A browser that was open opened again with a new connection, which replaced the old one; a closed one left its connection live on the server for a token nobody held any more, still listed in Settings, Browsers as connected until the next open. So the clear now ends that browser's connection right after the wipe, open or closed, with its recorder.revoked row in the same transaction and a reason of its own, cleared. A connection that cannot be ended is said to the person as a clear that did not finish, and clearing again ends it.
guidovizoso
left a comment
There was a problem hiding this comment.
The core holds up well:
- The wheel's holder is set server-side from the session, and the computer checks it too.
- Each recording gets its own connection, audited in the same transaction.
- Narration stays on-device and off by default.
- The 3.7k deleted lines are the old demonstrations feature, so none of #858's safeguards were removed.
- The tests run against a real database and a real Chromium.
Requesting changes for one blocker.
Blocker
Record works on a shared computer, where other Bots can read the person's Recorder token. The Bot's browser is mounted wherever Bots have computers (server/src/index.ts, botBrowser), and nothing on that path calls personBrowserRefusal (browser/host.ts). On Record, the Recorder in the Bot's Chromium saves the person's token with writeLocal("connection", …) (extension/src/background/connection.ts). That lands on disk under the Bot's profile, which /exec from any other Bot on a shared computer can read. A prompt-injected Bot then has a live token into that person's Learning space until Stop. A recording still unsent when the send wait runs out also stays there (bot-browser-host.ts:236). This is the SHARED_COMPUTER threat from #858. Please refuse Record the same way. This also needs to line up with #871, which changes how isolation is decided.
Should fix
- The Bot can act again before the recording has stopped. In
agent-computer/src/control.ts,release()callssyncAdmission()beforeonWheelLeft(), and ending the recording queues behind a Record that is still starting. If the person hands back during a start, a routine on the same Bot acts at once, and its clicks go into the person's recording. Please keep the Bot's actions closed until the recording has settled. - Hand-back, cancel, stop and reset can end in an empty reply. They now wait on the recording, up to the 20 s send wait. But
/api/computers/:id/control/{release,cancel}and/computers/{stop,reset}aren't inbotBrowserRequestSeconds(http-server.ts), so Bun closes them at 10 s. The hand-back happens, but the app reports it as failed. - A hand-back whose reply fails leaves the connection live.
wheelReturnedonly runs whenreleaseControlsucceeded (computer/routes.ts:516). After a timeout, the connection and the wider network stay until someone opens that screen. Please call it in afinally, or sweep on the policy push. - Someone else's counts still leak.
viewFor(browser/bot-browser.ts) spreads...common, which includesactionsandphase, so a second viewer sees the recorder's step count. - Scope. The help-request card and hand-back reply, hiding person-filled fields from the Bot, and Clear browser are each reasonable. But together they make 11k lines hard to check against #858's guarantees. Please split them out.
Nits
- "Too many connection codes were asked for at once" is shown after pressing Record, where the person asked for no code.
bot-browser-host.ts:320passes the extension's raw error text to the person.- Each viewer's once-a-second read makes two computer calls and a query, and during a recording it also asks the Recorder over DevTools.
- A Stop can take longer than the 45 s computer call timeout, and the connection is then revoked only after
START_GRACE_MS.
Resolved conflicts: - CopilotKit 1.83.0 from main (app, server, toolkit), and the lockfile regenerated. - The resolver keeps #858's person-bound learned Skills and main's name context. - Admin keeps the Skill sharing switch under main's reworded Network copy. - The changelog's upgrade list is counted from what it now holds (fourteen). - The tool-capture test helpers and both CI test fixes take main's versions. - The assertion-shape guard points at the shifted line.
Brings in the skins (#856). The account keeps both the narration choice and the skin, the sidebar draws the shared navigation list with the Browser button after Connections, and PageHeader carries the page-header and page-title hooks the skins read.
…nd keep the Bot out until a recording ends Record hands the Recorder in the Bot's browser the person's own connection, kept in the Bot's profile until Stop. On the shared computer any other Bot's shell can read that profile, so Record is now refused there by the server (before a connection is minted) and by the computer, and the Bot's screen says why in place of Record. With a supervisor or a Sandbox each Bot has its own computer and Record works as before. Also from the review: - The computer admits no Bot action after a hand-back, a cancel or an interruption until the recording the wheel took with it has ended, including one still queued behind a Record that was starting. - Hand-back, cancel, stop and reset are kept open past Bun's 10 s, since they wait for the recording to end; Stop gets its own 90 s call to the computer. - A hand-back or cancel whose answer is lost still sweeps the Bot's browser: whatever the computer says now decides which connections are left behind. - Someone else's running recording no longer shows its step count or microphone state. - The Recorder's raw connect error stays in the log; the person is told in OpenBot's words. The too-many refusal after Record no longer talks about connection codes.
Takes #872's removal of the form-submit source test, and keeps the demonstrations entry out of the assertion allow-list, since this branch removes that feature and its test.
|
@guidovizoso thanks for the review. Addressed in 7b18393, with main merged since (CI green). Each fix has a regression test that fails before it and passes after. Blocker: Record on a shared computer. Refused now, twice over:
Should fix
Nits
|
guidovizoso
left a comment
There was a problem hiding this comment.
Re-reviewed at b9a270b5 (fixes in 7b18393b). Approving.
Verified fixed:
- Record on a shared computer is refused on both sides: the server returns 403 before minting a connection (
botRecordRefusal), and the computer refuses Record unless it was started for one id (computerOwnerinperson-access.ts). - The Bot acting before the recording settled:
control.tscounts the endings before it re-syncs admission inreleaseandcancel, and a Bot action meanwhile getsRECORDING_ENDING. - Release, cancel, stop and reset have the longer request timeout. A failed release or cancel calls
wheelMayHaveReturned. viewForno longer shows another person's counts. The copy and raw-error nits are fixed, and Stop has its own 90 s call.
One condition for when this and #871 meet: keep Record limited to a computer started for one Bot, i.e. isolated Bots, which is what the computer already enforces. On a person's computer under #871, every other Bot they use, a public Bot someone else wrote included, can read the Recorder token in that Bot's profile. So "refused only on Helm shared" isn't enough there. As written, botRecordRefusal would refuse every #871 provider, since per-bot no longer exists, so whichever PR merges second needs to line it up this way.
Not blocking: the once-a-second read per viewer, and the three extra features kept in this PR.
Brings in #879: a Bot's browser now also closes when the computer's memory runs high and its idle background tabs close, sparing any browser whose wheel a person holds, so a recording is never cut off by either.
… a person's own computer A Bot's browser (#868) is now a seat's, like everything else on a computer: each route resolves the asker's seat for the Bot and reaches that browser, on their own computer or on an isolated Bot's computer of its own, with the person named to the computer where people share one. Recorder connections are still filed under the Bot, but a seat on a person's computer reads, hands back and clears only that person's: before, Ben opening his own Bot screen ended Ada's live recording on her computer as left behind. The Bot's network policy is pushed to the seat's computer. Record is refused only on the computer every person shares. A person's computer is now told whose it is (COMPUTER_PERSON_ID, from the supervisor and the Sandbox template), so it records in its Bots' browsers there and still refuses a person's own browser.
Stacked on #858 (the Recorder and the OpenBot Browser), into its branch.
What this changes
A Bot's screen, opened larger from a conversation, becomes a browser a person can work in, and recording there replaces "Teach a browser workflow". A skill can now be recorded inside OpenBot, in the Bot's own browser, with no extension to install and no Connect step.
app/src/components/browser/browser-bar.tsx).--disable-extensions-exceptand--load-extension, as Playwright documents and as the hosted person's browser already loads it), kept current, driven over the browser's DevTools pipe, and idle until someone records: it injects nothing into a page and holds no connection until then.heldBy), and ends a recording the moment the wheel leaves them (hand back, a request ending, a reset or restart), waiting for the recording's last events to reach Intelligence before the Recorder disconnects (bounded at twice the gateway's own timeout). A person's own take is now recorded assource: "manual", so its request text is never shown back as the Bot's reason.viabot-browser) for the person who pressed it and hands it to the Recorder over the DevTools launcher; Stop revokes it, and one the computer ended on its own is revoked on the next read. Settings, Browsers lists it as "'s browser" while it lives and says "recording now" for any browser that is recording (it now refreshes while open)./api/demonstrations, their store and scheduling, and the computer's gesture capture on the live screen). Skills already saved from it stay as ordinary skills; thedemonstrationstable stays, unread, with no migration.Contract:
shared/bot-browser.ts.Where it runs
heldBy) lives in the computer's existing control store, besideholder, on the computer that runs the Bot's browser.recorder_connections(viabot-browser, install id derived from the Bot), so any replica can find and revoke it.GET /api/computers/:botId/browseronce a second while the screen is open; the live screen is the existing stream./api/computers/:botId/browser/*behind the same ingress, each with a per-request timeout covering its own computer waits, so none ends in an empty reply.Boundary and audit
recorder.connectedandrecorder.revokedwith itsviaand reason (stopped,ended,not-started), in the same transaction as the change.Changelog
CHANGELOG.mdunderUnreleased: "Record a task in a Bot's own browser, with nothing to install".Proof
Run against the real product: a single-user OpenBot, its managed computer (bundled Chromium, headless on macOS, shown through the live screen), real Google Chrome driving the app, and a real local Intelligence.
test:ci10,498 pass and 0 fail, CI's real-browser computer list 115 pass and 0 fail, every Docker image built and booted. The one failure is the desktop Rust test that needs port 3001, which fails the same way on main on that machine.recording.startedandoutcome; typed passwords and emails arrive masked; Stop answers in about 300 ms and the tail is never cut.fromincluded.