feat(agent-device): a disconnected worker is not an ended lease, and a gateway restart keeps the device #517

Part of #511.

Scope

After this PR, through a gateway:

• when the worker holding an agent's device is disconnected, or has just
  reconnected and the gateway has not read it yet, every request for that
  device answers one clear reply, "not connected; try again", without being
  sent anywhere. The simlock lease is untouched. Once the worker is back and
  read, before the simlock lease ran out, the next request works;
• after a gateway restart, once the worker reconnects, the agent's next
  request of any kind works on the same device. In the first 60 s after the
  gateway starts, a request for a lease the gateway does not know yet gets
  the same "try again" reply instead of a refusal.

On a worker nothing changes: a simlock restart still ends the agent's
agent-device lease (its agent-device daemon restarts with it), and the next
heartbeat answers LEASE_ENDED_BY_HOST, as today.

In short

A gateway keeps a disconnected worker's view, leases included, so the front
still sees the lease. Today (#515) a request for it is relayed, the relay
fails, and the agent gets agent-device is not available on this simlock: <the relay's error>, while a renewal fails and is tried again at the next
tick. This task names that case: the host link says a lease is
unreachable, and the front answers before relaying.

After a gateway restart the front's map is empty and the lease index fills
again as workers reconnect. The front then adopts the agent's leases:
it rebuilds their entries from what the gateway knows.

New terms:

• unreachable lease: a lease the gateway issued whose worker is not
  connected, or is connected but not yet read since it reconnected. It may
  still exist; only the worker can say. Example: a laptop worker that lost
  Wi-Fi for 20 s.
• adopted lease: a lease the front rebuilds an entry for after a
  gateway restart. A lease this gateway process asked for itself is never
  adopted. Example: lease 3f2a…, owned by tok_ci1, requester 3f2a…, on
  worker w-1a2b, made before the restart.

sequenceDiagram
  participant CLI as agent-device CLI
  participant F as front (gateway)
  participant W as worker
  W--xF: uplink drops
  CLI->>F: snapshot (lease L)
  F->>F: status(L) = unreachable (new)
  F-->>CLI: 503 "not connected; try again" (new)
  W->>F: uplink back, status read
  CLI->>F: snapshot (lease L)
  F->>W: relayed, as before
  Note over F: gateway restarts: map empty
  CLI->>F: heartbeat (lease L)
  F->>F: adopt this token's leases this process did not ask for (new)
  F->>W: heartbeat, then renew

# worker w-5e6f lost its uplink; before (#515)
$ agent-device snapshot
agent-device is not available on this simlock: Worker w-5e6f is not reachable

# after
$ agent-device snapshot
agent-device is not available on this simlock: the worker holding this device is not connected to the gateway; try again
$ agent-device snapshot        # the worker is back
(the snapshot)

If this goes wrong, an agent gives up on a device it still holds, keeps
using a device simlock already took back, or loses its device on every
gateway restart.

Technical spec

Modules touched

flowchart LR
  F["front: front.ts, heartbeat, release, devices-filter *"] --> HL["host-link port *"]
  F --> RR["front: request-rules.ts *"]
  F --> FM["front: front-map.ts *"]
  RL["front: relay host link (#515) *"] -.->|implements| HL
  RL --> P["fleet port (main.ts) *"]
  P --> LI["gateway: lease index, worker registry"]
  SH["shared: replies.ts *"]
  T["test rig: uplink listener close hook *"]

(* = changed.)

• Reply. src/agent-device/shared/replies.ts gains workerUnreachable():
  the notAvailable reply with reason the worker holding this device is not connected to the gateway; try again, status 503, details.reason
WORKER_UNREACHABLE. An RPC answer carries details; a non-RPC answer is
  frontReply's plain body with the same message and status, and no
  details, as every non-RPC refusal today.
• Host-link port.
  • status(leaseId) can also answer { state: "unreachable" }. The relay
    host link answers it, for a lease the token owns that it can place
    (index or memory, feat(agent-device): open, commands, heartbeats and close through a gateway #515), when the lease's worker is not connected, or
    is connected but its view has not been refreshed since it connected. In
    the first 60 s after the gateway started, it also answers it for a lease
    it cannot place at all. InProcessHostLink never answers it.
  • The relay host link's forward, when the relay fails with
    WORKER_UNREACHABLE, answers workerUnreachable() instead of feat(agent-device): open, commands, heartbeats and close through a gateway #515's
    notAvailable reply. The link is the one place that decides it.
  • adoptable(identity): leases the front may adopt for this principal,
    each with its facts. InProcessHostLink answers none. The relay host
    link answers every lease in the index that is owned by the principal,
    whose requester is the lease id itself (the mark every lease the front
    makes carries), whose id is not one the front asked for in this process
    (below), and whose worker is connected, refreshed since, and lists the lease and
    its device. Facts: the device address, platform and model (from the
    worker view's device record), the target from the model's class in that
    worker's catalog (phone → mobile, tv → tv, desktop →
    desktop, the reverse of CLASS_OF_TARGET in allocate.ts; mobile
    when the catalog has no class for it), and the lease's ttlMs and
    ttlDeadline. The record has no device name, so an adopted entry has
    none.
• Front.
  • At the start of every request on /v1/agent-device with an agent token
    (RPC and non-RPC alike, before checkRequest), the front adds an entry
    for each adoptable lease its map lacks.
  • The front records the lease id of every lease.request its allocate
    sends, in memory, for the life of the process (an entry is dropped once
    the index no longer holds that lease). Those ids are never adoptable, so
    an allocate that ended without keeping its lease (the CLI hung up, or
    agent-device refused) is never adopted later. The rule reads no clock,
    so a worker whose clock differs from the gateway's changes nothing.
  • An adopted entry has no device key. The first call naming a deviceKey
    for it sets it (FrontMap sets it only when unset, so of two calls at
    once the first to reach the map wins, and the other is refused
    refused: simlock picks the device by wrongDeviceKey when its key
    differs). Until then, wrongDeviceKey accepts any key for an adopted
    entry.
  • An adopted entry has no name: --device <name> falls back to today's
    model check, and a shard list names it by address.
  • handleRpc, for a heartbeat, release or command whose status(leaseId)
    is unreachable, answers workerUnreachable(); nothing is forwarded,
    nothing is marked, and the simlock lease is untouched.
  • The non-RPC routes read status(leaseId) before forwarding (feat(agent-device): devices, uploads, artifacts and diagnostics through a gateway #516), so
    a lease header naming an unreachable lease answers workerUnreachable();
    devices answers it when any of the token's leases is unreachable.
  • A renewal tick for an unreachable lease fails as today and is tried
    again at the next tick; the entry and its timer stay. lease.list keeps
    listing a disconnected worker's leases, so tokenDevices and the
    allocate's map refresh never mark one ended.
• Fleet port (main.ts): adds each lease index entry's worker
  connection state and whether the view was refreshed since connecting, the
  gateway's start time (for the 60 s window), and the worker view's lease,
  device and catalog records, read-only.
• Test rig. The gateway's uplink listener (src/ports/uplink-websocket.ts)
  gains a test-only way to close one worker's socket from the gateway side;
  the worker's GatewayUplink then redials on its own backoff, as after a
  network drop.

Contract and event changes

None.

Other code on the same state

The state is the front map and the gateway's lease index.

• Lease index (src/gateway/lease-index.ts): rebuilt from each
  reconnecting worker's leases. A lease that expired while its worker was
  disconnected has no fact to remove it: the index drops it once it is
  missing from two snapshots in a row. Until then status(leaseId) places
  it on a connected worker, the relay reaches agent-device, which answers
  the lease gone, and the front's existing heartbeat path answers
  LEASE_ENDED_BY_HOST.
• Worker registry connected() (worker-registry.ts): clears
  catalogReadAt on reconnect; the relay host link reads "refreshed since
  connecting" from the view, so a lease is unreachable, not "cannot serve",
  in that moment.
• Worker lease TTL: keeps running on the worker while the uplink is
  down; the front's renewals fail meanwhile.
• lease.list on the gateway (dispatcher.ts #ownedFleetLeases):
  lists a disconnected worker's leases from its kept view. A fresh view with
  no leases yet leaves them out; adoptable offers only leases the view
  lists, and the front adopts before tokenDevices reads, so the two agree.
• stillWaiting in allocate.ts: reads lease.list only to wait out
  daemon start; unchanged.
• token-devices.ts: unchanged.
• Today's heartbeat path for a lease the map lacks (heartbeat.ts):
  still runs in worker mode, and on a gateway for a lease adoptable does
  not offer.

Rules in play

• architecture.md: every exit leaves one named state (live, ended,
  unreachable); one place decides it (the host link).
• safety.md: adopting checks ownership against the token, never the
  request.

Tests

Seam: real gateway and worker daemons in one process, HTTP into the
gateway's /v1/agent-device, FakeAgentDevice per worker; the worker's
uplink is cut from the gateway side through the new test hook, and the
gateway is restarted as src/daemon/gateway-fleet.e2e.test.ts does.

• with the device's worker disconnected, a command, a heartbeat and a release each answer 503 with details.reason WORKER_UNREACHABLE and the workerUnreachable message, and no agent-device daemon receives them
• with the device's worker disconnected, an upload with the lease header answers 503 with that message and no details, and devices answers the RPC reply with details
• while the worker is disconnected, the simlock lease is not released, and the front's entry is not marked ended
• the uplink dropping mid-command answers the command with workerUnreachable()
• right after the worker reconnects and before its view is refreshed, a command answers workerUnreachable(); once it is refreshed, within the TTL, the next command reaches it, and the next heartbeat renews the lease
• a worker that stays disconnected past the TTL: once it reconnects, the next heartbeat answers LEASE_ENDED_BY_HOST
• a lease known only from the link's memory (after simlock release --all) whose worker then disconnects answers workerUnreachable()
• a running command's renewal tick while the worker is disconnected fails, and the entry and its timer stay; after the reconnect the next tick renews
• an allocate by the same token while its other lease's worker is disconnected does not mark that lease ended
• after a gateway restart and the worker's reconnect, the next heartbeat for the lease reaches the worker's agent-device daemon and renews the simlock lease, and a following command works
• after a gateway restart, a command, a release, devices and an upload with the lease header, each as the token's first request, work on the adopted lease
• after a gateway restart, an adopted lease on a TV model reads target tv, and test --target tv sharding uses it
• after a gateway restart, a command naming a deviceKey on the adopted lease is accepted and sets it; a later command naming another deviceKey is refused refused: simlock picks the device
• within 60 s of a gateway restart, before the worker reconnects, a heartbeat, a command, a release and an upload with the lease header for the lease each answer workerUnreachable(); after 60 s, with another capable worker connected, a heartbeat for it answers refused: lease <id> is not one of yours on this gateway
• after a gateway restart, a lease of another token is not adopted, and a heartbeat naming it is refused
• after a gateway restart, a lease of this token whose requester is not its own id is not adopted
• a lease this gateway process asked for whose allocate failed after the grant (the uplink dropped mid-allocate) is never adopted, and stays out of devices, also when the worker's clock is an hour behind or ahead of the gateway's
• after a gateway restart, a lease made before it is adopted when the worker's clock is an hour behind or ahead of the gateway's
• on a worker, after a simlock restart, a command for the old lease is not adopted and the next heartbeat answers LEASE_ENDED_BY_HOST, as today

Done when

• Every Tests line passes.
• docs/HTTP-API.md (/v1/agent-device/*) describes the unreachable
  answer and that a gateway restart keeps the agent's device.

Out of scope

• Telling the agent which worker it is on, or when it is back.
• A gateway-start window other than 60 s, or a config key for it.
• A device name on an adopted lease.

Depends on

• feat(agent-device): devices, uploads, artifacts and diagnostics through a gateway #516

Approval

• Approved for delivery

Written by an agent.