Drive simlock devices with agent-device through a gateway #511

Request: #510

Problem

An agent can drive a simlock device with the agent-device CLI only by
pointing the CLI at the worker that owns the device (#466). In a fleet,
agents know one address: the gateway. On a gateway, every
/v1/agent-device URL answers 404, so an agent in a fleet cannot use
agent-device through simlock at all, or must know which worker to call and
hold a token minted on that worker.

Who it is for

• Agents in a fleet that reach simlock only through a gateway, and want to
  drive a device with agent-device the way they do through a worker.
• The operator of that fleet, who needs to see which workers can serve
  agent-device and why a worker cannot.

Outcome

• With agentDevice.enabled on the gateway, an agent points the
  agent-device CLI at http://<gateway>:<port>/v1/agent-device with its
  gateway token. It then opens, drives, uploads to, downloads artifacts
  from and closes a device the same way it does through a worker.
• open through a gateway lands only on a worker whose agent-device daemon
  is running. When no connected worker has one, open is refused at once,
  with the reason.
• Uploads, artifact downloads and diagnostics go to the worker that holds
  the agent-device lease they name, so one token can hold devices on
  several workers.
• When the worker holding a device drops off the gateway, the agent gets
  the gateway's usual WORKER_UNREACHABLE answer. The simlock lease keeps
  running down on the worker, and work goes on if the worker comes back in
  time. A gateway restart does not end the agent's simlock lease.
• simlock worker list, simlock status and the console show, for each
  worker, whether agent-device is available, off, or unavailable and why.

How it works for the user

sequenceDiagram
  participant CLI as agent-device CLI
  participant G as simlock gateway
  participant W as simlock worker (agent-device daemon running)
  CLI->>G: open (gateway token)
  alt no connected worker has agent-device running
    G-->>CLI: refused, with the reason
  else every matching device is taken
    G-->>CLI: waits in simlock's queue, up to 330 s
  else
    G->>W: simlock lease, then agent-device lease, on one worker
    G-->>CLI: device opened
  end
  CLI->>G: snapshot, install ./App.app, artifact download
  G->>W: sent to the worker holding that agent-device lease
  alt the worker is not connected to the gateway
    G-->>CLI: 503 WORKER_UNREACHABLE (simlock lease still runs on the worker)
  end
  CLI->>G: close
  G->>W: agent-device lease ends, then the simlock lease ends

Before, on a gateway:

$ export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<gateway token>
$ agent-device connect proxy --daemon-base-url http://gw:4700/v1/agent-device
$ agent-device open MyApp --platform ios --device "iPhone 17 Pro"
404 Not Found

After, with agentDevice.enabled on the gateway:

$ agent-device open MyApp --platform ios --device "iPhone 17 Pro"
(opens MyApp on an iPhone 17 Pro leased from one of the gateway's workers)
$ agent-device install ./MyApp.app
$ agent-device snapshot
$ agent-device close

No worker can serve agent-device:

$ agent-device open MyApp --platform ios
agent-device is not available on this simlock: no worker connected to this gateway can serve agent-device

An agent-device CLI too old to send the agent-device lease id on uploads:

$ agent-device install ./MyApp.app
refused: this agent-device is too old to use through a gateway; update it

The operator's view (the format may differ; each worker shows one of the
three states):

$ simlock worker list
ID       LABEL     STATE      AGENT-DEVICE
w-1a2b   mac-mini  connected  available
w-3c4d   ci-07     connected  off
w-5e6f   laptop    connected  unavailable: found 0.20.1, need 0.21.25 or later

Examples

What a gateway answers on /v1/agent-device:

The agentDevice keys on a gateway:

What could go wrong

• A big upload shares the worker's one connection to the gateway with
  simlock's own messages. It can slow other agents on that worker, and an
  upload that takes over agent-device's 5 minutes fails. Likely on slow
  links.
• An agent whose worker drops off sees WORKER_UNREACHABLE and may give up
  on a device it still holds. The device stays held until its simlock lease
  runs out.
• The gateway's view of a worker's agent-device state lags behind the
  worker. An open can land on a worker whose agent-device daemon just
  stopped, and then fail with the reason. Rare.

Words used

• gateway: a simlock daemon that owns no devices; workers connect to it, and
  agents reach the whole fleet through it.
• worker: a simlock daemon that owns devices on one machine.
• simlock lease: simlock's grant of one device to one agent, with a TTL,
  e.g. what simlock list shows.
• agent-device lease: agent-device's own lease on the device; it shares its
  id and lifetime with the simlock lease.
• capable worker: a connected worker whose agent-device daemon is running.
• agent-device state of a worker: available (its agent-device daemon is
  running), off (agentDevice.enabled is not set on it), or unavailable
  with the reason the gateway would give an agent that tried to use it
  (usually what simlock doctor says on that worker).
• upload: an install or reinstall of an app file on the agent's machine,
  which the CLI sends to agent-device before installing it.

Non-goals

• Any promise on upload size or speed through a gateway.
• Binary frames or flow control on the worker's connection to the gateway.
• Routing uploads, artifacts or diagnostics by token, or any fallback for an
  agent-device CLI that does not send the agent-device lease id.
• Any change to how the gateway places ordinary simlock leases.
• Any change to connecting the CLI straight to a worker, including a worker
  that has joined a gateway.

Completion conditions

• On one machine running a gateway and one worker, with agent-device built
  from the simlock-integration branch that carries the lease-id change, a
  slow-lane test passes on iOS and on Android: through the gateway, open,
  snapshot, install of an uploaded app, an artifact download and close
  work, and after close the simlock lease is gone from
  simlock list on the gateway.
• With agentDevice.enabled unset on the gateway, every
  /v1/agent-device URL on it answers 404.
• With no connected worker running agent-device, open through the gateway
  is refused at once, with the reason.
• With two workers, one with agent-device running and one with it off,
  every open through the gateway lands on the first.
• A command for a device whose worker has disconnected answers
  503 WORKER_UNREACHABLE. When the worker reconnects before the simlock
  lease runs out, the next command works.
• After a gateway restart and the worker's reconnect, the agent's next
  heartbeat and command work on the same device.
• Through a gateway, an upload, artifact download or diagnostics request
  with no agent-device lease id is refused, and the refusal says to update
  agent-device.
• simlock worker list, simlock status and the console show each worker's
  agent-device state, with the reason when it is unavailable.
• agentDevice.leaseTtlCapMs set on the gateway caps the simlock lease TTL
  of an agent connecting through it.

Open questions

Tasks

flowchart LR
  T1[#519 capabilities] --> T2[#513 requires]
  T2 --> T3[#514 relay]
  T3 --> T4[#515 front on gateway]
  T4 --> T5[#516 devices + lease header]
  T5 --> T6[#517 unreachable + restart]
  T5 --> T7[#518 slow lane]

• feat(fleet): workers advertise their capabilities in status and worker list #519 Workers advertise their capabilities — operators see each worker's
  agent-device state (available / off / unavailable and why) in
  simlock worker list, simlock status and the console.
  Risk: a worker shown as able to serve agent-device when it cannot.
• feat(fleet): a lease request may require a capability #513 A lease request may require a capability — simlock places a lease
  only on a worker that has what it needs, and refuses at once when none
  does. Risk: a request waits for a worker that will never take it.
• feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514 The gateway relays an HTTP exchange to a worker module — nothing
  visible yet; requests and their bodies can cross the uplink with a
  deadline and cancel. Risk: a large transfer slows other calls to that
  worker.
• feat(agent-device): open, commands, heartbeats and close through a gateway #515 open, commands, heartbeats and close through a gateway — an agent
  drives a device through the gateway. Risk: commands reach the wrong
  worker, or a lease outlives its agent.
• feat(agent-device): devices, uploads, artifacts and diagnostics through a gateway #516 devices, uploads, artifacts and diagnostics through a gateway — the
  rest of agent-device works through the gateway. Risk: an upload lands on
  the wrong worker, or devices misses a device.
• feat(agent-device): a disconnected worker is not an ended lease, and a gateway restart keeps the device #517 A disconnected worker is not an ended lease, and a gateway restart
  keeps the device — agents keep their device through a worker's blip or a
  gateway restart. Risk: an agent gives up on a device it still holds.
• test(e2e): agent-device through a gateway on real devices #518 agent-device through a gateway on real devices — the slow lane
  proves it on iOS and Android. Risk: the lane passes while real devices
  fail.

Decisions

• ADR 0023 — agent-device is served by an optional edge module: the front
  runs on the gateway and reaches the worker's host through a relay adapter
  of the host link (ADR 0026 narrows its "neither part changes").
• ADR 0005 — gateway and worker modes: workers dial out; the gateway never
  reaches in, so every agent-device request reaches the worker over the
  worker's own connection.
• ADR 0026 — agent-device through a gateway (accepted, docs(adr): 0026 agent-device through a gateway #512): workers advertise capabilities and a lease request may require one; the host link relays over the uplink; uploads, artifacts and diagnostics carry the agent-device lease id.

Written by an agent.