Skip to content

feat(cli): restore cursor overlay through page.evaluate - #2869

Open
shrey150 wants to merge 1 commit into
agent/browse-v4-4-runtimefrom
agent/browse-v4-cli-cursor-overlay
Open

feat(cli): restore cursor overlay through page.evaluate#2869
shrey150 wants to merge 1 commit into
agent/browse-v4-4-runtimefrom
agent/browse-v4-cli-cursor-overlay

Conversation

@shrey150

@shrey150 shrey150 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Restore Browse's visible cursor as a CLI-owned DOM overlay, without adding a cursor API to core Stagehand V4.

  • Keep the overlay implementation in one dedicated cursor-overlay.ts file.
  • Install it idempotently for the current document through page.evaluate(CURSOR_OVERLAY_SCRIPT) and for future navigations through page.addInitScript(...).
  • Keep injection in the top frame and update the marker from coordinate input, including when input lands inside a child frame.
  • Track mouse movement using a fixed, pointer-events-disabled element.

Stack (#2872)

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime and commands to Stagehand V4 #2834 — Stagehand V4 runtime and standard command parity
  3. feat(cli): restore cursor overlay through page.evaluate #2869 — CLI-owned cursor overlay
  4. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; V3 network parity
  5. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; supported V3 parity/release checkpoint
  6. test(evals): exercise the workspace V4 CLI #2838 — eval and packaging integration
  7. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  8. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Why this is separate

The cursor is a self-contained optional visual feature with different review concerns from the combined V4 runtime/command migration: injected DOM/CSS, idempotency, event handling, and screenshot behavior. Keeping it additive on #2834 lets this feature be reviewed or reverted without disturbing browser lifecycle or commands.

E2E Test Matrix

Fresh post-flatten verification used the actual built CLI at final head 8428e338e. Every daemon command used an isolated owner-only runtime directory.

Command / flow Observed output Confidence / sufficiency
pnpm install --frozen-lockfile Lockfile passed supply-chain policy, was already up to date, and installation completed Proves the flattened child resolves exactly from its committed lockfile
pnpm exec turbo run build --filter=browse Protocol, extension, Stagehand SDK, and Browse CLI built successfully (4/4 tasks) Prevents stale workspace SDK/extension artifacts from producing a false cursor result
pnpm --filter browse test 25 files / 387 tests passed Full Browse unit/integration suite on the exact final head
Built CLI: browse open https://example.com --remote; browse cursor; navigate the same page to the public IANA Example Domains page; browse mouse hover 220 140; inspect #__browse_cursor_overlay__ One marker existed after navigation at left:220px, top:140px, with pointer-events:none; the init-script mouse listener was installed Proves CLI-owned cursor injection, navigation persistence, and coordinate synchronization in a real Browserbase browser
browse screenshot --path <artifact>.png Fresh 189,489-byte PNG; manual inspection shows the black/white cursor at the asserted post-navigation position Visual proof from the exact final head; the PR-body API cannot upload the fresh local artifact directly
browse stop Completed successfully Covers live Browserbase session/daemon cleanup

The already-uploaded screenshot below remains representative visual proof of the same overlay behavior. A fresh exact-head artifact was also generated and manually inspected during the final verification.

Browse V4 CLI cursor overlay surviving navigation and pointing inside an iframe

The live targets were public documentation domains; no LLM path or customer data was involved.

@changeset-bot

changeset-bot Bot commented Aug 31, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 8428e33

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files

Architecture diagram
sequenceDiagram
    participant User as Browse CLI User
    participant CLI as Browse CLI
    participant Runtime as Runtime Command Handlers
    participant Manager as Page Manager
    participant Page as Active Browser Page
    participant DOM as Page DOM Overlay
    participant Browser as Browser Input and Renderer

    Note over CLI,DOM: CLI-owned cursor visualization (no Stagehand core cursor API)

    User->>CLI: browse cursor
    CLI->>Runtime: Dispatch cursor command
    Runtime->>Manager: Resolve activePage()
    Manager-->>Runtime: Active page
    Runtime->>Page: evaluate(CURSOR_OVERLAY_SCRIPT)
    Page->>DOM: Ensure fixed overlay exists

    alt Overlay already installed
        DOM-->>Page: Reuse __browse_cursor_overlay__
    else Overlay not installed
        DOM->>DOM: Create fixed SVG cursor element
        DOM->>DOM: Set aria-hidden, z-index, pointer-events none
        DOM->>DOM: Append overlay to document root
        DOM->>DOM: Install capture-phase mousemove listener
    end

    Page-->>Runtime: Evaluation complete
    Runtime-->>CLI: Return enabled=true
    CLI-->>User: Cursor overlay enabled

    User->>CLI: browse mouse hover x y
    CLI->>Browser: Move mouse to viewport coordinates
    Browser->>DOM: Dispatch mousemove event
    DOM->>DOM: Clamp clientX and clientY to non-negative values
    DOM->>DOM: Update fixed overlay left and top

    User->>CLI: browse screenshot --path cursor.png
    CLI->>Page: Capture current page
    Page->>Browser: Render DOM and overlay
    Browser-->>CLI: Screenshot including visible cursor
    CLI-->>User: Save cursor.png
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/cursor-overlay.ts
Comment thread packages/cli/src/lib/driver/commands/runtime.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread packages/cli/src/lib/driver/commands/runtime.ts
Comment thread packages/cli/src/lib/driver/commands/mouse.ts Outdated
@shrey150
shrey150 force-pushed the agent/browse-v4-cli-cursor-overlay branch from dc60e5f to 8428e33 Compare September 11, 2026 18:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant