fix(droid-control): preserve evidence across automation workflows - #49
Merged
Merged
Conversation
Make desktop-control usable without a separately installed personal Cua skill. Bundle exact upstream documentation and its MIT license, with an offline importer/checker and plugin-only packaging tests. Route explicit cua-only requests before browser defaults, keep shared desktop interaction under one controller, and distinguish verified task completion from input delivery and partial capture artifacts. Pin the reference to the contribution in trycua/cua#3719 and bump the plugin to 1.1.0 without claiming upstream merge or runtime certification. Closes CLI-2242 Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Maintain desktop guidance with the plugin instead of enforcing immutable upstream copies. Remove the importer, hash manifest, second skill entrypoint, discovery exception, and tests that only exercised that machinery. Keep one documentation-only relocation/link check. Route platform recovery through the shared workflow, use invoke_menu for Finder navigation, and remove competing skill-install and legacy browser mutation-enable recipes. Preserve the original Cua license and attribution. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Keep the desktop workflow in a 77-line skill instead of shipping upstream manuals. Retain routine setup, exact targets, permission boundaries, recovery, recording ownership, and evidence handoff; remove SDK/browser tutorials, release history, and the documentation-packaging suite. Restore the original CI workflow and retain the Cua MIT license unchanged. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
factory-ain3sh
self-requested a review
September 11, 2026 05:49
Make commands follow the shared desktop ownership policy and distinguish blocked observations from application failures. Preflight persistent Cua recording connections and verify decoded output against actual commitments. Isolate each render's staged inputs, normalize fidelity and formats, and apply speed once to every clip. Use the longest clip and a shared padded timeline so comparisons retain their proof intervals. Encode BT.709 yuv420p. Cover helper isolation, cancellation, fidelity, timing, invalid inputs, and real encoded output with behavioral tests and path-scoped CI. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Use desktop-use, terminal-use, and browser-use as the capability entrypoints. Route terminal work through terminal-use while preserving the existing tuistory and true-input backends, CLI commands, and runtime behavior. Update skill discovery and every maintained skill/path reference together. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
factory-ain3sh
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What
Give
droid-controlone capability-based path from an automation request to verified evidence.terminal-useselects tuistory or true-input,browser-useoperates agent-browser, anddesktop-useoperates Cua while honoring native-input constraints. The desktop skill is 80 lines, without copied manuals or a separate Cua skill prerequisite. The media path has independently passing coverage from 13 helper tests, including two real renders, and seven duration tests; this is not new live-GUI certification.Composition of
origin/master...2b8c15c, computed withdiff-composition.py(additions plus deletions):Core breakdown: the render helper (221 changed lines) owns staging, input validation, fidelity, and duration probing.
Showcase.tsx(270) andduration.ts(43) share playback timing and transition padding..skillsrc(6), plugin metadata (2), andRoot.tsx(1) connect discovery, versioning, and resolved defaults.Why
Correct individual driver instructions are insufficient when commands can demand competing desktop workers, two
recording.mp4inputs overwrite one another, or rendering truncates the longer proof. A blocked observation also cannot honestly become an application PASS or FAIL.How
The orchestrator selects a capability and owns delegation; executable names remain implementation details. Shared desktop input stays with one controller, while independent captures and renders can run separately. Each render owns isolated inputs, applies speed once, retains the longest clip, and holds shorter panels. Verification follows the resolved output contract and requires decoding plus observed proof frames.
Change Shape
Repro Recipe
With existing Remotion dependencies, browser, and ffmpeg installed, run from this branch:
Related Issue
Closes CLI-2242.
Reviewer Guide
Review depth: Standard, with attention to media timing and input ownership.
terminal-use,browser-use,desktop-use. Check native-input precedence, backend selection, and parent-owned desktop interaction.scripts/render-showcase.sh→remotion/src/lib/duration.ts→remotion/src/compositions/Showcase.tsx. Check isolated paths, all-clip duration, exactly-once speed, and transition padding.Plugin paths are relative to
plugins/droid-control/; discovery and CI configuration are repository-root files.Risk & Impact
desktop-control→desktop-use,tuistory→terminal-use, andagent-browser→browser-use. External skill-name references must update; executable commands and backend identifiers do not change. There are no old-name aliases.speedNoteremains supported.Verification
Media verified @
95af904, unchanged through2b8c15c: independent 13/13 helper tests with zero skips, including two real renders; 7/7 duration tests and TypeScript checking on Node 22.19.0. Outputs decoded successfully as limited-range BT.709yuv420p; an eight-playback-frame case rendered without transition failure and displayed its expected final state.Naming/routing verified @
2b8c15c: canonical directories/frontmatter, local links and anchors, discovery exclusions, terminal backend routing, and absence of stale skill references. Runtime files and executable command fences remain unchanged;desktop-useis 80 lines.Regression coverage: same-basename and concurrent staging, scoped cleanup, fidelity selection, unsupported PNG input, finite speeds, longest-clip timing, cast pause preservation, and encoded-output checks. Negative controls reproduced the old failures.
Standard validators: scoped React/structural scans, Node 22 typecheck and tests, skill discovery, rename/link checks, shell syntax, and whitespace checks.
Not tested: a new live Cua recording or OS-permission flow. Synthetic media is not native-GUI certification. Render artifacts were inspected locally but are not attached.