feat(agent-device): open, commands, heartbeats and close through a gateway #515

Part of #511.

Scope

After this PR, an agent can point the agent-device CLI at a gateway with
agentDevice.enabled and drive a device: open, commands, heartbeats and
close work as they do on a worker. open lands only on a worker whose
agent-device is available. When none is, it is refused at once with the
reason.

devices, uploads, artifact downloads and diagnostics are #516. The task is
just over the size limit and stays whole: allocate, heartbeat, command and
release share the front's lease handling and the relay host link, and none
of them works through a gateway without the others. What the
front answers when the device's worker is disconnected, and after a gateway
restart, is #517. Until #516 merges, a request that names no lease (other
than /health) is refused through a gateway.

In short

The agent-device front runs on the gateway. Each request it hands to a host
names the agent-device lease it is for. On a gateway, the host link finds
that lease's worker and relays the request there (#514). The worker hands
it to its agent-device host, which registers as the relay target
agent-device. A relayed request's agent-device tenant on the worker is
gw:<gateway instance id>:<principal>, so an agent on the gateway never
shares agent-device sessions with an agent on the worker's own tokens.

sequenceDiagram
  participant CLI as agent-device CLI
  participant F as front (gateway, new)
  participant GL as relay host link (new)
  participant W as worker: relay target agent-device (new)
  CLI->>F: allocate
  F->>F: lease.request requires agent-device (#513)
  F->>GL: forward(allocate, lease L)
  GL->>GL: lease index: L is on W (remembered)
  GL->>W: relay (#514), tenant gw:<id>:<principal>
  W-->>CLI: agent-device's reply
  CLI->>F: command / heartbeat / release (lease L)
  F->>GL: forward(…, lease L)
  GL->>W: relay
  alt no worker has agent-device available
    F-->>CLI: agent-device is not available on this simlock: no worker connected to this gateway can serve agent-device
  end

# before, on a gateway
$ agent-device open MyApp --platform ios
Remote daemon is unavailable

# after, with agentDevice.enabled on the gateway
$ agent-device open MyApp --platform ios
(opened on an iPhone leased from a capable worker)
$ agent-device snapshot
$ agent-device close

A capable worker here: connected, not drained, its view lists
agent-device as available (#519), and the gateway has read its catalog
and its lease.maxTtlMs. A drained worker keeps serving the leases it
holds, but takes no new one.

If this goes wrong, an agent through a gateway gets a device it cannot
drive, commands reach the wrong worker, or a lease outlives its agent.

Technical spec

Modules touched

flowchart LR
  M["daemon: main.ts gateway + worker composition *"] --> IDX["agent-device: index.ts *"]
  M --> CFG["core: config.ts *"]
  IDX --> F["front: front, context, allocate, heartbeat, release *"]
  F --> HL["front: host-link.ts *"]
  RL["front: relay host link (new)"] -.->|implements| HL
  RL --> P["fleet port (built in main.ts): relay, lease lookup, capable workers"]
  P --> GR["gateway: GatewayRelay (#514), lease index, worker registry"]
  M --> RT["worker: relayTargets option (#514)"]
  RT --> H["host: AgentDeviceHost.forward"]

(* = changed, new = new file.)

• Config (src/core/config.ts). Gateway mode reads agentDevice
  (add it to GATEWAY_CONFIG_KEYS). agentDevice.command stays worker-only:
  set on a gateway, it is warned about and ignored, the way
  WORKER_ONLY_GATEWAY_KEYS handles gateway.url. leaseTtlCapMs keeps its
  existing check in both modes.
• Host-link port (src/agent-device/front/host-link.ts).
  • status(leaseId?): with a lease id, the status of that lease's host;
    without, the status for a new lease.
  • forward(request, identity, leaseId?).
  • InProcessHostLink ignores leaseId in both, so worker mode behaves
    as today.
• Front (front.ts, context.ts, allocate.ts, heartbeat.ts,
  release.ts).
  • handleRpc reads status() with the call's lease id for a heartbeat,
    a release and a command that names one, and with none otherwise.
  • Every forward and exchange passes the lease id the front already
    read: the allocate's chosen id (plan.leaseId), the heartbeat's, the
    release's, and the command's (commandLeaseId).
  • allocate sends the host link's leaseRequires on its
    lease.request: ["agent-device"] from the relay host link, nothing
    from InProcessHostLink, so a direct worker request is unchanged. A
    CAPABILITY_MISSING answer takes today's path for a refused
    lease.request: refused by simlock: <message>.
  • After the grant, allocate reads status(plan.leaseId) and forwards
    with that leaseMaxTtlMs. If that status is unavailable (the worker's
    agent-device went down after the grant), it releases the simlock lease
    and answers the notAvailable reply with that reason; no agent-device
    lease is made.
  • /health reads status() with no lease id and, when running, forwards
    with no lease id. Any /health answer the front makes itself (not
    available, or a relay that failed) has status 200, as today's
    unavailable answer does, because the CLI hides the reason of a /health
    at 500 or above.
• Relay host link (new, src/agent-device/front/relay-host-link.ts).
  It reads a narrow port, built in main.ts from the gateway's parts, so
  the edge imports nothing from src/gateway:
  • leaseOf(leaseId): the lease index's worker id and owner, or none.
  • capableWorkers(): each capable worker (above), with its platforms
    (from its catalog) and its lease.maxTtlMs.
  • relay(workerId, request, tenant, leaseId?) (feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514).
  • The gateway's instance id, for the tenant prefix.
  • It remembers, per lease id, the worker and owner it last read from
    leaseOf, and keeps routing to that worker after the index drops the
    lease (as simlock release --all does), so the worker's agent-device
    daemon still answers for it. The fleet port tells the link when the index
    removes a lease; the link stamps the entry then, and drops it once the
    gateway's own lease.maxTtlMs has passed since that stamp, checked on
    every forward and by a sweep every 60 s.
  • The port's running status carries leaseMaxTtlMs only with a lease
    id; the front reads the ceiling from status(leaseId) alone.
  • status() with no lease id: running when capableWorkers() is
    non-empty, with the union of their platforms; otherwise unavailable
    with reason no worker connected to this gateway can serve agent-device.
  • status(leaseId) for a lease it can place (index or memory) and whose
    owner is the identity's principal: running with that worker's platforms
    and ceiling (agentDeviceLeaseMaxTtlMs of its lease.maxTtlMs) when the
    worker is connected and lists agent-device as available (drained or
    not), else unavailable with reason the worker holding this lease cannot serve agent-device. For any other lease, the same as status(),
    so nothing about another token's lease shows, and the front goes on to
    forward, which refuses it.
  • forward with a lease id it can place, owned by the identity's
    principal: relays to that worker with tenant gw:<gateway instance id>:<principal>. A principal is a token id simlock mints (tok_ and a
    generated id), so the tenant stays inside feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514's 300-character bound; a
    tenant over it is refused refused: this token cannot be used through a gateway before anything is relayed. A lease it cannot place, or one another principal
    owns, answers refused: lease <id> is not one of yours on this gateway
    (status 403) before anything is relayed. A relay that fails
    (WORKER_UNREACHABLE, RELAY_TARGET_MISSING) answers the
    notAvailable reply naming that error's message; feat(agent-device): a disconnected worker is not an ended lease, and a gateway restart keeps the device #517 refines the
    disconnected case.
  • forward with no lease id: /health goes to any capable worker; any
    other request is refused with refused: this agent-device is too old to use through a gateway; update it.
• Gateway composition (src/daemon/main.ts, startGatewayDaemon):
  with agentDevice.enabled, builds the front over the relay host link and
  passes it as the HTTP mounts, with dispatch the gateway's own,
  maxRunMs the gateway's lease.maxTtlMs and the gateway's
  leaseTtlCapMs. On stop, it cancels waiting allocates and disposes the
  renewals (cancelWaitingAllocates, disposeAgentDeviceEdge), as the
  worker composition does.
• Worker composition: with agentDevice.enabled, registers the relay
  target agent-device (feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514's relayTargets), which calls
  AgentDeviceHost.forward with identity { requesterId: tenant, role: "agent" }. When the host's own status is unavailable (the gateway's
  view lagged), the target answers the front's notAvailable reply with
  that reason instead of throwing, so the agent gets the reason. The host's
  release-scope recording and lease-end participant then apply to relayed
  leases as to direct ones.

Contract and event changes

None of its own: it uses #513's requires and #514's relay. No events.

Other code on the same state

The state is the simlock leases the front makes, its front map, and the
agent-device tenants on the worker.

• Gateway lease coordinator (fleet-coordinator.ts): grants, renews,
  releases and cancels the front's leases through the gateway's dispatcher,
  as for an HTTP client. The front's requesterId is the lease id, so the
  worker sees gw:<gateway>:<lease id>, as for every gateway lease.
• Gateway lease index (lease-index.ts): the relay host link reads it
  and never writes it. When the index drops a lease, the link's memory still
  routes it: the heartbeat reaches agent-device, which answers the lease
  gone (the worker's lease-end participant released it), and the front's
  existing path (leaseGoneReason, then releaseOurs, whose simlock release
  finds the lease unknown) answers LEASE_ENDED_BY_HOST.
• FleetLeaseCoordinator.dispose: rejects every waiting request with
  DAEMON_STOPPING when the gateway stops. The edge's
  cancelWaitingAllocates runs first, so the front's own waiting requests
  end cancelled.
• Worker host lease-end participant (host/lease-end.ts): ends the
  agent-device lease before simlock wipes the device, for relayed leases
  too, since it reads the host's own record.
• The worker's own front: still serves agents that connect to the
  worker directly, under their own principal as tenant. A relayed tenant
  always starts gw:, which no worker token's principal does, so the two
  never meet in agent-device.

Rules in play

• architecture.md: the edge imports nothing from the gateway; it gets a
  narrow port. The front reads lease ids in one place. Every exit of
  allocate leaves one named state.
• safety.md: a lease id from the wire routes only after its owner is
  checked against the token's principal.
• testing.md: every table row is a test.

Tests

Seam: real gateway and worker daemons in one process over a real uplink,
HTTP into the gateway's /v1/agent-device, FakeAgentDevice as each
worker's agent-device daemon; modelled on src/agent-device/edge.test.ts
and src/daemon/gateway-fleet.e2e.test.ts.

• with agentDevice.enabled unset on the gateway, /v1/agent-device/health answers 404, as today
• with agentDevice.command set on a gateway, the gateway logs that it ignores it and starts; with agentDevice.leaseTtlCapMs above lease.maxTtlMs on a gateway, the gateway fails to start, as a worker does
• with agentDevice.enabled unset on the gateway, /rpc, upload/preflight, artifacts/<id> and a diagnostics URL each answer 404
• /health answers the proxy's health when a worker is capable, and the notAvailable reply naming no worker connected to this gateway can serve agent-device, with status 200, when none is; when the relay to the capable worker fails, it answers the notAvailable reply with status 200
• with the only capable worker drained, allocate answers the notAvailable reply, while a command on a lease it already holds still works
• a worker whose view says available but whose catalog was not read yet does not count as capable
• allocate through the gateway makes a simlock lease on the capable worker, owned by the token's principal, whose id and gateway requester are the CLI's lease id, and its lease.request carries requires: ["agent-device"]
• allocate on a worker, directly, sends no requires, as today
• with two workers, one capable, every allocate over 5 tries lands on the capable one
• the allocate the worker's agent-device daemon receives names the granted device, carries ttlMs equal to that worker's ceiling, and has tenant gw:<gateway instance id>:<principal>
• allocate with no capable worker answers the notAvailable reply, and nothing is leased
• allocate --platform ios when the capable workers have Android only is refused refused: ios is not available on this simlock
• allocate when only an incapable worker knows the model answers refused by simlock: No worker connected to this gateway can serve agent-device for this request, and nothing is leased
• allocate whose lease's worker stops being capable between the grant and the forward releases the simlock lease and answers the notAvailable reply naming the worker holding this lease cannot serve agent-device
• allocate relayed to a worker whose own agent-device went down while the gateway's view still said available answers the notAvailable reply with the worker's reason, and the simlock lease is released
• with every matching device leased, allocate waits and is granted once one is released
• a CLI that hangs up while its allocate waits leaves the gateway's queue, and nothing is granted later
• a command for the lease reaches that worker's agent-device daemon with the prefixed tenant, and its streamed reply reaches the CLI unchanged
• a heartbeat reaches the worker's agent-device daemon first, then renews the simlock lease to now plus its ttlMs
• a heartbeat for a lease whose worker stopped being capable answers the notAvailable reply naming the worker holding this lease cannot serve agent-device
• while a command runs longer than the TTL, the simlock lease is renewed every TTL/3 through the gateway and does not expire
• release goes to the worker's agent-device daemon first, then releases the simlock lease, and the device is reclaimed
• a release the agent-device daemon fails keeps the simlock lease
• after simlock release --all on the gateway, the worker's agent-device lease is released before the wipe, and the next heartbeat reaches that worker and answers LEASE_ENDED_BY_HOST
• a command with no device selector naming a lease id another token holds, and one the gateway does not know, answer refused: lease <id> is not one of yours on this gateway with status 403, and no worker receives it; with --udid, the same command answers refused: simlock picks the device, as on a worker
• after simlock release --all, once the fake clock has moved the gateway's lease.maxTtlMs plus 1 s past the index's removal of the lease, the link's memory no longer holds it, and a heartbeat for it answers refused: lease <id> is not one of yours on this gateway; one second before that, the heartbeat is still relayed
• a request with no lease id other than /health is refused with refused: this agent-device is too old to use through a gateway; update it
• a relay that fails with RELAY_TARGET_MISSING answers the notAvailable reply with that message
• an agent on a worker token with the same principal name as a gateway agent, connected to the worker directly, does not see the gateway agent's agent-device session, and the gateway agent's commands still work
• agentDevice.leaseTtlCapMs on the gateway caps the TTL of a lease made through it
• stopping the gateway while an allocate waits ends its lease request cancelled, not DAEMON_STOPPING, and a running command's renewals stop

Assumptions

• Assumption: the tenant a gateway sends agent-device is gw.<gateway instance id>.<principal> (dots, not colons), because agent-device accepts only 1-128 chars of letters, digits, dot, underscore, hyphen; a character outside letters, digits and hyphen is written _<hex>_ and a name over 128 characters is cut and ends in a hash, so every input gives a valid, distinct tenant; simlock's own lease requester stays gw:<id>:<requester> (decided in delivery, pending the maintainer).

Done when

• Every Tests line passes.
• docs/HTTP-API.md (/v1/agent-device/*) says a gateway with
  agentDevice.enabled serves it, and docs/CONFIGURATION.md marks
  agentDevice.enabled and agentDevice.leaseTtlCapMs as read on both
  sides and agentDevice.command as worker side.

Out of scope

• devices, and uploads, artifacts and diagnostics by lease header (feat(agent-device): devices, uploads, artifacts and diagnostics through a gateway #516).
• A disconnected worker, and a gateway restart (feat(agent-device): a disconnected worker is not an ended lease, and a gateway restart keeps the device #517).

Depends on

• feat(fleet): a lease request may require a capability #513
• feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514

Approval

• Approved for delivery

Written by an agent.