Skip to content

improvement(computer-use): fold empty layout groups and compact the accessibility tree shown to the model - #9997

Merged
waleedlatif1 merged 3 commits into
stagingfrom
feat/computer-use-compact-observation
Nov 13, 2026
Merged

improvement(computer-use): fold empty layout groups and compact the accessibility tree shown to the model#9997
waleedlatif1 merged 3 commits into
stagingfrom
feat/computer-use-compact-observation

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

  • The computer-use accessibility tree shown to the model folds empty layout wrappers (AXGroup, AXSplitGroup, AXLayoutArea/Item, AXUnknown with no text, state or action); their children point at the nearest shown ancestor, so the observation budget goes to controls and content
  • Rows drop the selected window's id (window= only for other windows) and fractional screen coordinates; action names stay JSON-encoded so custom actions remain exact; a one-line treeFormat legend explains the row shape
  • omittedNodeCount and truncated now count only shown nodes, so folded wrappers are not reported as omitted

Type of Change

  • Improvement

Testing

  • New projection test: 300 nested empty groups no longer crowd out 150 buttons, and the buttons point at the window; fails on the previous projection
  • computer-tool-result.test.ts (13)
  • Benchmark (68 macOS tasks, Opus 5.5, production worker loop, paired against the current build in parallel): identical pass/fail on every task (p=1.0); cost $1.17→$1.01/episode, calls 13.5→13.0, wall 55→52.5 s, failed-call rate 3.3%→2.5%; median observation 21.7K→18.5K chars, observations omitting nodes 44.5%→36.0%, p90 omitted nodes 254→85

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@waleedlatif1
improvement(computer-use): fold empty layout groups and compact rows …
2bd3203 
…in the model's accessibility tree

Wrapper groups with no text, state or action no longer spend the observation budget;
their children point at the nearest shown ancestor. Rows drop the selected window id,
JSON array syntax and fractional coordinates, and a one-line legend explains the format.
@vercel

vercel Bot commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 13, 2026 1:22pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge.

Summary

Compacts the accessibility tree shown to the model.

  • The computer-use tree folds empty layout groups and keeps controls linked to visible parents.
  • Computer-use rows leave out repeated window IDs and use rounded screen coordinates.

Reviews (3) · Last reviewed commit: "fix(computer-use): keep JSON-encoded act..." · Reviewed by Greptile

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 13, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/mothership/tools/client/computer-tool-result.ts Resolved
@waleedlatif1

waleedlatif1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

waleedlatif1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 13, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/lib/mothership/tools/client/computer-tool-result.ts Resolved
@waleedlatif1

waleedlatif1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

waleedlatif1 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Nov 13, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit afb6653 into staging Nov 13, 2026
81 checks passed
@waleedlatif1
waleedlatif1 deleted the feat/computer-use-compact-observation branch November 13, 2026 17:48
This was referenced Nov 13, 2026
Merged
Merged

This branch was previously deployed

1 inactive deployment
Preview — d3aa47e3 Deployed Nov 13, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@waleedlatif1