Live screen streams of leased devices #593

Request: none

PoC: #576 (feasibility and cost, iOS only; its handoff comment covers the Android work on feature/vendor-provenance).

Problem

An operator cannot see what is on a leased device. An agent drives a simulator or an emulator on a worker, maybe a machine in another room, behind a gateway. When the agent hangs, loops or fails a test, the only way to see the screen is to log in to that machine and find the right window. Nothing in Simlock shows a picture.

Who it is for

• An operator watching agents' devices in the console, on any lease in the fleet.
• An agent or a tool that holds a lease and wants a live picture of its own device over the HTTP API, for example to record a failing run.

Outcome

• In the console, a lease's page has a Screen panel. Pressing Show screen shows the device's screen live. Pressing Stop, leaving the page or closing the tab stops it.
• Over HTTP, one GET on a lease's stream URL returns the live picture for as long as the connection stays open.
• This works for leases on iOS simulators and on Android emulators, on a single worker and through a gateway, with the same URL and the same bytes.
• Nothing is captured while nobody looks. A capture never outlives its viewer or its lease.

How it works for the user

sequenceDiagram
  participant U as Operator (console)
  participant G as Gateway
  participant W as Worker
  participant D as Device
  U->>G: Show screen (GET /v1/stream/leases/L1)
  G->>G: lease L1 known, caller may see it?
  alt unknown, or another agent's lease
    G-->>U: 404 UNKNOWN_LEASE
  end
  G->>W: relay to the worker that holds L1
  W->>W: lease still held by that requester?
  W->>D: start capture
  alt device cannot be filmed (physical device)
    W-->>U: 415 STREAM_NOT_SUPPORTED
  else capture tool missing on the worker
    W-->>U: 503 STREAM_UNAVAILABLE
  end
  W-->>U: 200, picture headers, then frames
  U->>G: Stop / leave page (request closed)
  G->>W: relay cancelled
  W->>D: capture stopped
  Note over W,D: the lease ends while watched: capture stopped, the stream ends

Before (any daemon today):

$ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" \
    http://gw:7777/v1/stream/leases/L1
404

After (stream.enabled: true on the gateway and the worker):

$ curl -sD - -o screen.bin -H "Authorization: Bearer $TOKEN" \
    http://gw:7777/v1/stream/leases/L1
HTTP/1.1 200 OK
content-type: application/vnd.simlock.stream+h264
simlock-stream-codec: h264
simlock-stream-width: 330
simlock-stream-height: 720
simlock-stream-fps: 30
x-simlock-lease: L1
^C        # closing the request stops the capture on the worker

$ curl -s -H "Authorization: Bearer $TOKEN" http://gw:7777/v1/stream/leases/L9
{"error":{"code":"STREAM_NOT_SUPPORTED","message":"This lease is on a physical device, and a live screen is captured from a simulator or an emulator only"}}

Examples

What could go wrong

• A picture freezes and looks live. Likely only if a capture stalls without ending; a stream that goes 30 s without a frame ends instead.
• A capture keeps running after its lease ends, on a device someone else now holds. Unlikely: the lease end stops it, and the device's reclaim sweeps what is left.
• An agent sees another agent's screen. Unlikely: ownership is checked on the gateway and again on the worker.
• An Android stream is slow or choppy (about 21 fps while scrolling, measured under heavy host load). Likely on busy machines; the encoder measurement below says how much.
• A worker's CPU rises while a screen is watched: one capture per viewer.

Words used

• stream: one open GET /v1/stream/leases/{leaseId} and the picture it carries, e.g. the console's Screen panel while it is on.
• capture: the process (or device-side server) that films one device for one stream, e.g. one simlock-screen-capture per viewer of a simulator.
• envelope: the body's framing: 4-byte big-endian length (counting the tag), 1-byte tag, payload. Tag 0x05 JSON meta first, 0x01 codec description (avcC), 0x02 key frame, 0x03 delta frame.
• key frame: a frame a decoder can start from; a delta frame needs the frames before it.
• relay target: a name a gateway sends a relayed HTTP exchange to on a worker, e.g. agent-device, and now stream.
• capability: an optional module a worker reports with a state (available, unavailable with a reason, off), e.g. stream: unavailable (the screen capture program is not installed).
• scrcpy server: the small Android program from the scrcpy project that encodes an Android screen on the device; Simlock ships it unchanged.
• smart socket: adb's own way to reach a socket inside a device through the adb server (host:transport:<serial>, then localabstract:<name>), with no host port opened.

Non-goals

• Input. No touches, keys or text through the stream; v1 is view-only. Named follow-up.
• Physical devices, iOS and Android. Refused in v1.
• Audio.
• Recording or saving streams on the worker, and any history of who watched.
• A CLI command. The console and the HTTP API only.
• Choosing a device by whether it can be streamed (requires), and showing "being watched" anywhere.
• A GPU against software encoder measurement for Android emulators. Named follow-up; it decides whether a faster Android picture is worth a config change.
• Any codec but H.264.

Completion conditions

• With stream.enabled: true, an operator presses Show screen on a leased iOS simulator in the console, on a single worker, and sees the screen move as an agent drives it.
• The same on a leased Android emulator.
• The same through a gateway, for both platforms.
• GET /v1/stream/leases/{leaseId} answers each row of Examples with the status and code shown.
• Closing the request ends the capture on the worker: no capture process, no app_process for scrcpy in the emulator, and no scrcpy jar left in /data/local/tmp.
• Releasing a lease while it is watched ends the stream, and leaves the same nothing behind.
• With stream.enabled: false, no stream route exists and the worker reports the stream capability off.
• The npm package carries the iOS capture tool and the scrcpy server; nothing is downloaded to stream.
• The capture tool and the scrcpy server sit under vendor/ with their license texts and a NOTICE.md each, and the check that enforces it passes.

Open questions

• Does the shipped iOS capture binary need code signing or notarization beyond the linker's ad-hoc signature, and should it be universal (arm64 and x86_64) or arm64 only?
• When a device rotates, does each platform send a new description frame with the new size, and does the console's decoder follow it? Not measured in feat(stream): a lease's screen, streamed to whoever is looking at it (PoC) #576.
• How does a checkout get the iOS capture tool: built by pnpm build on macOS, or by a separate script? Until it is built, iOS streams answer STREAM_UNAVAILABLE.
• How long may an Android key frame be apart? scrcpy's default interval is 10 s, so a viewer that falls behind may wait up to that long for the picture to resume.
• ADR 0030 (Optional modules share one contract and load only when turned on #591, docs(adr): 0030 optional modules share one contract and load only when turned on #592) is not merged. This feature lands on its module contract; if 0030 changes, ADR 0031 §1 follows it.

Decisions

• ADR 0031 — A screen stream is an optional module, and a driver films the device: the route, the envelope, the close-is-stop rule, the driver capability, both platforms, the relay, no business events, vendored code. Proposed, PR docs(adr): 0031 a screen stream is an optional module, and a driver films the device #594.
• ADR 0030 — Optional modules share one contract and load only when turned on (docs(adr): 0030 optional modules share one contract and load only when turned on #592): stream is a module on it; one name for capability, relay target and config key.
• ADR 0024 — Leasing calls lease-end participants before reclaim: how a lease end stops captures.
• ADR 0026, ADR 0028 — the relay over the uplink and its byte channels; ADR 0031 amends 0028 §4: the 330 s deadline covers the channel and the response head, then a 330 s idle timer bounds every relayed body.

Technical spec

Facts

Modules touched

flowchart LR
  MAIN["module host (ADR 0030)"] --> ST["stream/ (new) *"]
  ST --> LP["leasing: LeaseEndParticipant"]
  ST --> SCR["core/screen.ts: Driver.screenCapture *"]
  IOS["drivers/ios *"] --> SCR
  AND["drivers/android *"] --> SCR
  IOS --> VS["vendor/screen-capture (binary in package) *"]
  AND --> VJ["vendor/scrcpy (jar) *"]
  GW["gateway/relay.ts: head deadline, body idle timer *"]
  ST -. gateway ctx.relay .-> GW
  UI["ui/src/screen *"] -. HTTP .-> ST

(* = new or changed.)

sequenceDiagram
  participant B as Browser
  participant G as Gateway stream
  participant R as Relay + channel
  participant W as Worker stream
  participant D as Driver screen
  B->>G: GET /v1/stream/leases/L1
  G->>G: lease index: L1 on W, caller owns it; W reports stream available
  G->>R: relay(W, GET, namespaced requester)
  R->>W: relay target "stream"
  W->>W: registry: requester matches; register capture
  W->>D: screenCapture.open(device, settings)
  D-->>W: envelope bytes (meta first)
  W-->>R: 200 + picture headers
  R-->>G: head (deadline ends, 330 s idle timer starts)
  G->>G: check header values
  G-->>B: 200, then bytes unparsed
  alt worker refuses
    W-->>G: 4xx/5xx JSON, read up to 16 KiB, code shape-checked
    G-->>B: same status and code
  end

Surface and extension points

• src/stream/index.ts: the module's definition for ADR 0030's registry, and nothing else the host needs.
• src/core/screen.ts: ScreenCapture, ScreenFeed, ScreenNotSupportedError (415) and ScreenUnavailableError (503); Driver.screenCapture? (ADR 0031 §4). The second implementation exists (Android), so the one shape is fixed now. The envelope's tag values and meta shape sit beside it.

Contract and event changes

• No new contract operation; no protocol bump from this feature.
• New HTTP route GET /v1/stream/leases/{leaseId}; codes STREAM_NOT_SUPPORTED (415) and STREAM_UNAVAILABLE (503) join the HTTP error table.
• Config: stream.enabled, stream.fps, stream.maxDimension, stream.bitrateKbps. No stream.command; drivers.ios.screenCaptureCommand overrides the packaged iOS capture program.
• Events: none. Both EVENTS.md files gain a paragraph saying a stream emits no event and why (ADR 0031 §8).

Other code on the same state

• Leasing's lease end and reclaim (F1): the lease-end hook closes every capture of that lease and waits for the closes (bounded by each driver's close and leasing's signal); the reclaim then sweeps scrcpy jars on that emulator. Lease end wins.
• The Android driver's private adb server: the scrcpy client's adb push and shell, and its smart-socket connection, go through it, scoped to the emulator's serial. The driver's dispose stops captures before it stops the adb server (ADR 0030 beforeDriversDispose).
• The iOS driver's shutdown and erase of a simulator: a capture of a device being shut down fails, and its stream errors rather than freezing.
• agent-device's relay target (F6): a second name in the same table; neither reads the other's exchanges.
• The link's channels (F12): a stream holds one channel for its life; at 64 open channels a new relay waits.

Rules in play

• architecture.md 1–3, 9, 11, 12, 15, 18–21: the core names no platform; ports for spawn, timers and sockets; every wait bounded; every capture ends in one state.
• safety.md 4, 9, 10: no download; Simlock's own adb server for emulators; worker answers and headers are claims.
• events.md: no events, and the catalogue says so.
• vendor-and-third-party-code.md (lands with this feature from feature/vendor-provenance).
• documentation.md: docs/HTTP-API.md, docs/CONFIGURATION.md, docs/CONSOLE.md, and the src/stream/AGENTS.md manual.

Tests

Seam: the HTTP app over a worker daemon with fake drivers, and the gateway–worker in-memory pair (src/daemon/relay-gateway.e2e.test.ts); drivers' screen code against a fake process runner and socket port (src/drivers/android/*.test.ts).

• each row of Examples answers the status and code shown, on a worker and through a gateway.
• a lease that ends while its capture is starting stops that capture once it starts.
• a lease that ends while watched ends the stream and closes the driver's screen.
• closing the request closes the driver's screen, before and after the first frame.
• a capture silent for 30 s ends its stream with an error, on a fake clock.
• a relayed body that keeps sending outlives the 330 s deadline; one silent for the idle time ends WORKER_UNREACHABLE with relay.cancel; a head that does not arrive in time fails WORKER_UNREACHABLE (any relay target).
• a worker refusal body over 16 KiB, or with a code that does not match the shape, reaches the caller as STREAM_UNAVAILABLE.
• a worker 200 with a missing or out-of-range picture header reaches the caller as STREAM_UNAVAILABLE.
• the gateway forwards the worker's body bytes unchanged.
• an operator sees any lease; an agent with another agent's lease gets the unknown-lease answer.
• the module turned off mounts no route and reports off.
• iOS: the tool is spawned with that device's set and the picture options; a physical device is refused not-supported; a missing binary reports unavailable.
• Android: the jar is pushed to a per-capture path, the server started with the fixed options, the device socket reached through host:transport:<serial> and localabstract:scrcpy_<scid> with no adb forward; Annex-B becomes AVCC with an avcC description; deltas are dropped only until the next key frame; a physical device is refused not-supported.
• Android reclaim removes Simlock's scrcpy jars for that serial.
• a lease that ends after a capture's first frame but before its 200 closes the capture and answers the unknown-lease refusal.
• slow lane: a leased iOS simulator and a leased Android emulator each stream through a gateway, and after Stop nothing is left on the worker or in the emulator.

Written by an agent.