Skip to content

fix(droid-control): preserve evidence across automation workflows - #49

Merged
factory-ain3sh merged 5 commits into
masterfrom
ainesh/cli-2242-desktop-control
Sep 14, 2026
Merged

factory-ain3sh merged 5 commits into
masterfrom
ainesh/cli-2242-desktop-control

Conversation

@factory-droid

@factory-droid factory-droid Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Description

What

Give droid-control one capability-based path from an automation request to verified evidence. terminal-use selects tuistory or true-input, browser-use operates agent-browser, and desktop-use operates Cua while honoring native-input constraints. The desktop skill is 80 lines, without copied manuals or a separate Cua skill prerequisite. The media path has independently passing coverage from 13 helper tests, including two real renders, and seven duration tests; this is not new live-GUI certification.

Composition of origin/master...2b8c15c, computed with diff-composition.py (additions plus deletions):

Share Lines Files Scope Reviewer action
43.1% 816 22 Capability/workflow guidance, license, and CI Check routing and evidence ownership; prose here controls agent behavior.
28.7% 543 6 Media implementation, discovery, and metadata Read input isolation and the shared playback timeline.
26.0% 493 2 Helper integration and duration tests Check concurrent staging and actual encoded-output assertions.
2.2% 41 4 Diagram labels, test command, and staging ignore Skim for consistency with the runtime contract.

Core breakdown: the render helper (221 changed lines) owns staging, input validation, fidelity, and duration probing. Showcase.tsx (270) and duration.ts (43) share playback timing and transition padding. .skillsrc (6), plugin metadata (2), and Root.tsx (1) connect discovery, versioning, and resolved defaults.

Why

Correct individual driver instructions are insufficient when commands can demand competing desktop workers, two recording.mp4 inputs overwrite one another, or rendering truncates the longer proof. A blocked observation also cannot honestly become an application PASS or FAIL.

How

The orchestrator selects a capability and owns delegation; executable names remain implementation details. Shared desktop input stays with one controller, while independent captures and renders can run separately. Each render owns isolated inputs, applies speed once, retains the longest clip, and holds shorter panels. Verification follows the resolved output contract and requires decoding plus observed proof frames.

Change Shape

Request + method constraints
  → terminal-use | browser-use | desktop-use
  → driver-owned interaction and capture
  → isolated render inputs + shared playback timeline, when video is requested
  → observed PASS / FAIL / BLOCKED evidence

Repro Recipe

With existing Remotion dependencies, browser, and ffmpeg installed, run from this branch:

REPO=$(git rev-parse --show-toplevel)
RUN=$(mktemp -d)
mkdir -p "$RUN/before" "$RUN/after"
ffmpeg -v error -f lavfi -i color=red:s=320x240:r=30:d=2 -c:v libx264 "$RUN/before/recording.mp4"
ffmpeg -v error -f lavfi -i color=blue:s=320x240:r=30:d=5 -c:v libx264 "$RUN/after/recording.mp4"
"$REPO/plugins/droid-control/scripts/render-showcase.sh" \
  --props-inline '{"layout":"side-by-side","labels":["BEFORE","AFTER"],"speed":2,"title":"Input isolation","subtitle":"Distinct same-named clips","preset":"minimal","keys":[],"effects":[]}' \
  --fidelity compact --output "$RUN/proof.mp4" \
  "$RUN/before/recording.mp4" "$RUN/after/recording.mp4"
ffprobe -v error -select_streams v:0 -show_entries stream=nb_frames,pix_fmt,color_space "$RUN/proof.mp4"
# Expect 300 frames, yuv420p, bt709. Open proof.mp4: red and blue remain distinct.

Related Issue

Closes CLI-2242.

Reviewer Guide

Review depth: Standard, with attention to media timing and input ownership.

  1. Capability boundary: orchestrator → terminal-use, browser-use, desktop-use. Check native-input precedence, backend selection, and parent-owned desktop interaction.
  2. Media boundary: scripts/render-showcase.shremotion/src/lib/duration.tsremotion/src/compositions/Showcase.tsx. Check isolated paths, all-clip duration, exactly-once speed, and transition padding.
  3. Evidence boundary: capture/compose/verify skills and command templates → helper/duration tests and scoped CI. Check that blocked work stays unobserved and encoded output meets the declared contract.

Plugin paths are relative to plugins/droid-control/; discovery and CI configuration are repository-root files.

Risk & Impact

  • Skill names intentionally change: desktop-controldesktop-use, tuistoryterminal-use, and agent-browserbrowser-use. External skill-name references must update; executable commands and backend identifiers do not change. There are no old-name aliases.
  • Cast conversion no longer silently compresses idle time. Trim source recordings explicitly before calculating output-timeline overlay timestamps; free-text speedNote remains supported.
  • Total duration includes the four-second title, frame-rounded playback, and 3.5-second outro. Shorter clips hold their final frame; held frames appear during the outro transition.
  • Cancellation is process-group scoped. A signal sent only to the helper PID takes effect after its child exits.
  • CI provisions ffmpeg, checksum-verified agg, and the Remotion browser. No runtime dependency was added; driver capabilities and user/host permissions still require live inspection.

Verification

Media verified @ 95af904, unchanged through 2b8c15c: independent 13/13 helper tests with zero skips, including two real renders; 7/7 duration tests and TypeScript checking on Node 22.19.0. Outputs decoded successfully as limited-range BT.709 yuv420p; an eight-playback-frame case rendered without transition failure and displayed its expected final state.

Naming/routing verified @ 2b8c15c: canonical directories/frontmatter, local links and anchors, discovery exclusions, terminal backend routing, and absence of stale skill references. Runtime files and executable command fences remain unchanged; desktop-use is 80 lines.

Regression coverage: same-basename and concurrent staging, scoped cleanup, fidelity selection, unsupported PNG input, finite speeds, longest-clip timing, cast pause preservation, and encoded-output checks. Negative controls reproduced the old failures.

Standard validators: scoped React/structural scans, Node 22 typecheck and tests, skill discovery, rename/link checks, shell syntax, and whitespace checks.

Not tested: a new live Cua recording or OS-permission flow. Synthetic media is not native-GUI certification. Render artifacts were inspected locally but are not attached.

Make desktop-control usable without a separately installed personal Cua
skill. Bundle exact upstream documentation and its MIT license, with an
offline importer/checker and plugin-only packaging tests.

Route explicit cua-only requests before browser defaults, keep shared
desktop interaction under one controller, and distinguish verified task
completion from input delivery and partial capture artifacts.

Pin the reference to the contribution in trycua/cua#3719 and bump the
plugin to 1.1.0 without claiming upstream merge or runtime certification.

Closes CLI-2242

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Maintain desktop guidance with the plugin instead of enforcing immutable
upstream copies. Remove the importer, hash manifest, second skill entrypoint,
discovery exception, and tests that only exercised that machinery.

Keep one documentation-only relocation/link check. Route platform recovery
through the shared workflow, use invoke_menu for Finder navigation, and
remove competing skill-install and legacy browser mutation-enable recipes.
Preserve the original Cua license and attribution.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Keep the desktop workflow in a 77-line skill instead of shipping upstream
manuals. Retain routine setup, exact targets, permission boundaries, recovery,
recording ownership, and evidence handoff; remove SDK/browser tutorials,
release history, and the documentation-packaging suite.

Restore the original CI workflow and retain the Cua MIT license unchanged.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@factory-ain3sh
factory-ain3sh self-requested a review September 11, 2026 05:49
Make commands follow the shared desktop ownership policy and distinguish
blocked observations from application failures. Preflight persistent Cua
recording connections and verify decoded output against actual commitments.

Isolate each render's staged inputs, normalize fidelity and formats, and
apply speed once to every clip. Use the longest clip and a shared padded
timeline so comparisons retain their proof intervals. Encode BT.709 yuv420p.

Cover helper isolation, cancellation, fidelity, timing, invalid inputs, and
real encoded output with behavioral tests and path-scoped CI.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@factory-droid factory-droid Bot changed the title feat(droid-control): make desktop workflows self-contained fix(droid-control): preserve desktop evidence end to end Sep 14, 2026
Use desktop-use, terminal-use, and browser-use as the capability entrypoints.
Route terminal work through terminal-use while preserving the existing
tuistory and true-input backends, CLI commands, and runtime behavior.

Update skill discovery and every maintained skill/path reference together.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@factory-droid factory-droid Bot changed the title fix(droid-control): preserve desktop evidence end to end fix(droid-control): preserve evidence across automation workflows Sep 14, 2026
@factory-ain3sh
factory-ain3sh merged commit ee87c3a into master Sep 14, 2026
3 checks passed
@factory-ain3sh
factory-ain3sh deleted the ainesh/cli-2242-desktop-control branch September 14, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant