feat(fleet): a gateway relays an HTTP exchange to a worker module over the uplink #514

Part of #511.

Scope

After this PR, a gateway can hand one HTTP exchange to a module on a worker
and get the response back, body included, over the worker's uplink. A worker
module registers itself as a relay target under its capability name.
The gateway relays to it with a deadline, cancels it when its caller goes
away, and fails it with WORKER_UNREACHABLE when the uplink drops.

Nobody uses it yet: no target is registered in a real daemon. #515
registers agent-device. Tests use a fake target.

The task is at the size limit (about 30 Tests lines) and stays whole: the
worker's and the gateway's halves of the relay speak one protocol, and
neither half can be tested end to end without the other.

In short

New terms:

• relay target: a function on a worker, registered under a capability
  name, that takes a Request, a tenant name and a lease id, and answers a
  Response. Example: the target agent-device (feat(agent-device): open, commands, heartbeats and close through a gateway #515) hands the request
  to the agent-device host.
• relay call: one exchange through the uplink: a relay.request call,
  its request body as relay.body calls, and its response as pushes.

sequenceDiagram
  participant G as gateway: GatewayRelay (new)
  participant S as worker: DaemonServer *
  participant T as relay target (registered)
  G->>S: relay.request {relayId, target, method, url, headers, tenant, leaseId, hasBody}
  S->>T: Request (body is a stream fed by relay.body)
  G->>S: relay.body {relayId, seq, chunk} ×n (at most 4 unanswered)
  G->>S: relay.body {relayId, end: true}
  T-->>S: Response
  S-->>G: push relay-head {status, headers}
  S-->>G: push relay-chunk ×n
  S-->>G: relay.request result {} (body ended)
  alt caller gone / deadline
    G->>S: relay.cancel {relayId}
    S->>T: abort the Request's signal
    S-->>G: relay.request result {cancelled: true}
  else uplink drops
    G-->>G: fail the call: WORKER_UNREACHABLE
    S->>T: abort every relay of that connection
  end

If this goes wrong, a relayed upload arrives cut short or in the wrong
order, a relayed request runs on after its caller is gone, or a large
transfer holds the uplink and other calls on it time out.

Technical spec

Modules touched

flowchart LR
  OP["contract: operations, pushes, errors *"]
  GR["gateway: relay.ts GatewayRelay (new)"] --> FP["gateway: fleet-ports liveClient, worker error translation *"]
  FC["gateway: fleet-coordinator *"] --> FP
  CR["gateway: component-relay *"] --> FP
  GR --> AC["simlock-client: admin client *, wire *"]
  AC -->|uplink| SV["daemon: server.ts *"]
  SV --> RR["daemon: relay.ts relay targets (new)"]
  M["daemon: main.ts StartDaemonOptions, gateway composition *"] --> RR
  M --> GR

(* = changed, new = new file.)

Contract (src/contract/operations.ts, pushes.ts, errors.ts). Every
field is bounded by schema, since each arrives over the wire:

Headers are filtered both ways before the bounds apply. The gateway drops
the caller's authorization and x-agent-device-token, so a worker never
holds a gateway credential (#515 puts them back into an upload ticket on the
gateway side). Both sides drop the framing and hop-by-hop headers, which the
relay re-creates: connection, keep-alive, transfer-encoding,
content-length, upgrade, te, trailer, proxy-authorization and
proxy-authenticate.

• relay.request, role admin: input above. Output {} once the response
  body has ended, or { cancelled: true } once a cancel took effect. A
  worker holds at most 16 active relays per connection; a 17th fails
  BAD_REQUEST.
• relay.body, role admin: relayId, then either seq and chunk, or
  end: true. Output {}.
• relay.cancel, role admin: relayId. Output {}.
• Pushes relay-head (requestId, status, headers) and relay-chunk
  (requestId, chunk), keyed by the relay.request call's frame id like
  the four request-scoped pushes today (progress, output, started,
  component-progress).
• New error code RELAY_TARGET_MISSING: kind domain, HTTP 503, CLI
  exit 1, details { target }.

Worker (src/daemon/server.ts, new src/daemon/relay.ts):

• All three operations are refused FORBIDDEN on any session that is not
  the gateway's uplink, like lease.request's owner.
• Relay operations wait behind the worker's startup gate like every other
  operation. A relay.request registers its relayId when its frame is
  read, before any later frame on the connection is handled, so its
  relay.body frames always find it.
• relay.request naming a relayId already active on that connection fails
  BAD_REQUEST. One naming a target no module registered fails
  RELAY_TARGET_MISSING before anything is called.
• It builds a Request for url with the method and headers. With
  hasBody, the body is a stream that relay.body calls feed in seq
  order. A relay.body is answered once the stream has handed that chunk to
  the target, so the gateway's window holds the sender back. A seq out of
  order or repeated, a chunk after end, or any relay.body for a relay
  opened with hasBody: false, fails that call with BAD_REQUEST and
  aborts the relay. A relay.body for a relay id the worker does not hold
  (never opened, failed, ended or cancelled) answers {} and is dropped:
  the gateway learns the outcome from relay.request.
• It calls the target with the Request, tenant and leaseId, sends
  relay-head, then the response body as relay-chunk pushes of at most
  256 KiB each, then answers the call. It reads the next part of the
  target's body only once the previous push has been written to the socket,
  as device.exec output does, so the worker never queues a whole response.
  For a status of 204, 205 or 304, or a HEAD, it sends no chunk.
• A target Response whose headers break the bounds (after filtering) fails
  the call with INTERNAL, and the worker logs which bound.
• A target that throws before its Response fails the call with INTERNAL,
  and the worker logs the error's message. A response body that errors
  after relay-head fails the call with INTERNAL after the chunks sent so
  far.
• When the call settles for any reason, every relay.body still unanswered
  is answered {}, so the gateway's sender never waits on a target that
  stopped reading.
• relay.cancel, and the close of the connection the relay came on, abort
  the Request's signal. After a cancel, the worker sends no more pushes for
  that relay and answers relay.request { cancelled: true }. relay.cancel
  for a relay id the worker does not hold answers {}.
• Relay targets come from a new relayTargets option of startDaemon
  (capability name → target). Nothing passes one in this PR outside tests.

Client (src/simlock-client/wire.ts, client.ts). CallHooks gain
onRelayHead and onRelayChunk, routed by frame id like onOutput. The
client class gains relayRequest, relayBody and relayCancel, declared
on a new interface GatewayWorkerClient in src/gateway/ (the
SimlockAdminClient methods the gateway uses, plus these three). The
gateway's worker links hold a GatewayWorkerClient; SimlockAdminClient,
which simlock/admin exports, does not change.

Gateway (new src/gateway/relay.ts, exported from the gateway's
index.ts). A class GatewayRelay, built in startGatewayDaemon from the
worker directory, with a deadlineMs constructor option (default 330 000:
agent-device's proxy allows 300 s per exchange, plus a margin; no config
key). Tests reach it through a gatewayRelay callback option of
StartDaemonOptions, which the gateway composition calls with it, test-only
like agentDeviceUpstream. One method:
relay(workerId, { target, request, tenant, leaseId? }) → Promise<Response>.

• The worker's live client comes from liveClient (fleet-ports.ts); none
  → WORKER_UNREACHABLE, before anything is sent.
• It sends relay.request, then reads the request body in chunks of at
  most 256 KiB and sends each as relay.body, with at most 4 sent and not
  yet answered, then end. Once the call settles it stops reading the
  caller's body and cancels it.
• It resolves with a Response once relay-head arrives: a null body for
  204, 205, 304 or HEAD, else a stream fed by relay-chunk pushes and
  closed when the call answers {}.
• The client's wire routes relay-head and relay-chunk by requestId
  before checking the rest of the push, so a push that fails its schema
  reaches its relay (as onRelayInvalid) and is not dropped. The relay then
  fails with INTERNAL (message worker <id> sent an invalid relay response) and sends relay.cancel.
• relay.request answering {} before any relay-head, or
  { cancelled: true } when this side sent no cancel, fails the relay with
  INTERNAL (message worker <id> ended the relay without a response).
• A worker's INTERNAL after the head errors the caller's body stream with
  that message.
• At most 8 relays run at once per worker; the next waits for one to end,
  and the wait counts against its deadline.
• It buffers at most 64 MiB of relay-chunk data the caller has not read
  yet. Past that it sends relay.cancel and errors the body stream with a
  message saying the reader fell behind.
• The request's signal aborting, or the caller's request body erroring,
  sends relay.cancel. Before the head, the call rejects with the abort
  reason or the body's error; after it, the response body stream errors with
  it.
• The deadline runs from the relay() call and covers the whole exchange,
  both bodies included. Passing it sends relay.cancel and fails the call
  with WORKER_UNREACHABLE, message worker <id> did not finish the relayed request within <n> ms, <n> the deadline in milliseconds, or errors the
  body stream with that message if the head already came.
• The uplink dropping fails the call, or errors the body stream, with
  WORKER_UNREACHABLE. Which client errors mean "the uplink is lost" is
  decided in one function in fleet-ports.ts: an error of kind
  transport. FleetLeaseCoordinator#classifyRelayedError, the relay and
  the component relay's fromError all call it (the component relay keeps
  answering such a loss as an install result of unknown).
• relayRequest, relayBody and relayCancel are only on
  GatewayWorkerClient; simlock/admin does not export them, and
  docs/CLIENT.md does not list them.

Contract and event changes

• New operations relay.request, relay.body, relay.cancel; new pushes
  relay-head, relay-chunk; new error code RELAY_TARGET_MISSING.
• This PR bumps the protocol by one.
• No events: a relay is transport, not a business fact.

Other code on the same state

The state is the uplink connection and the calls in flight on it. Every
call shares the one connection in order, so a large relay delays the
others; #511 accepts this (no flow control).

• Worker link refresh (worker-link.ts): status.get and the other
  refresh calls have a 10 s timeout; two timeouts in a row close the link,
  and the closed link wins: the relay fails as an uplink drop.
• Forwarded lease.request (gateway.leaseRequestTimeoutMs) and
  device.exec (gateway.execTimeoutMs) in fleet-coordinator.ts: a
  delayed answer can time out as WORKER_UNREACHABLE while the worker
  granted the lease, which the gateway already handles as "Dispatched, then
  the uplink drops".
• Worker events (events.subscribe pushes) that drive view refreshes,
  and the component relay's worker.install-component: delayed in the
  same way; neither has a deadline the relay can break.
• Request-scoped pushes (server.ts): each routes by frame id, so relay
  pushes never mix with another call's.
• Connection close (server.ts #closeConnection): today it silences
  pushes; it now also aborts that connection's relays. A command run by
  device.exec still keeps running (pinned today by src/daemon/server.test.ts
  "stops pushing when the connection dies, and lets the command finish
  anyway").
• The component relay (component-relay.ts): now asks the shared
  function whether the uplink is lost, so DAEMON_STOPPING and the other
  transport errors read as unknown there too, not only
  DAEMON_CONNECTION_LOST.
• Forwarded lease.renew, lease.release and lease.cancel
  (#forwardToWorker): have no timeout. A renewal delayed behind a large
  relay can arrive after the lease's deadline, and the lease then expires on
  the worker. Drive simlock devices with agent-device through a gateway #511 accepts this with "no flow control"; a worker link that
  misses two status.get reads in a row closes, which fails the relay and
  every call on it with WORKER_UNREACHABLE.

Rules in play

• architecture.md: the core names no relay target; targets register
  through an option. Every cross-process wait is bounded (deadline, window,
  buffer). One translation of a lost uplink.
• safety.md: every relay field is wire input, bounded before use, in both
  directions.
• testing.md: order, cancel and drop each fail a named assertion.

Tests

Seam: real gateway and worker daemons in one process over a real uplink,
modelled on src/daemon/gateway-fleet.e2e.test.ts. The worker gets a fake
relay target through relayTargets; the test gets the gateway's
GatewayRelay through gatewayRelay; schema bounds through
src/contract/operations.test.ts.

• a GET relayed to the fake target arrives with its URL (origin, path and query), method, headers, tenant and lease id, and its status, headers and body come back unchanged
• a 20 MiB PUT body arrives at the target byte for byte, and a 20 MiB response body comes back byte for byte in relay-chunk pushes of at most 256 KiB
• while a 20 MiB relay runs, a lease.renew forwarded to the same worker answers
• the gateway never has more than 4 relay.body calls unanswered, checked with a target that reads its body slowly
• a target that answers 401 without reading its body: the caller gets the 401, every relay.body is answered, and the gateway stops reading the caller's body
• a response the target streams in parts reaches the gateway caller part by part, before the target's body ends
• 204, 205 and 304 answers and a HEAD answer reach the caller with a null body, and the worker sends no relay-chunk for them
• a caller that reads nothing while the target streams 80 MiB slowly gets its body stream errored, with a message saying the reader fell behind, once 64 MiB is buffered, and the target's request signal aborts before it has sent all 80 MiB
• aborting the caller's request signal before the head comes fails the call; after the head, it errors the caller's body stream; either way the target's request signal aborts and relay.request answers { cancelled: true } with no push after the cancel
• a caller's request body that errors mid-upload aborts the target's request signal
• a target that never answers is cancelled after a 200 ms deadline, and the caller gets WORKER_UNREACHABLE with message worker <id> did not finish the relayed request within 200 ms; one that stalls after its head gets its body stream errored with that message
• closing the uplink mid-relay fails the caller with WORKER_UNREACHABLE, or errors its body stream after the head, and aborts the target's request signal
• a relay to a worker with no uplink fails with WORKER_UNREACHABLE and the target sees nothing
• a relay naming a target nobody registered fails with RELAY_TARGET_MISSING
• a target that throws fails the call with INTERNAL and the worker logs its message; a response body that errors after the head fails the call with INTERNAL after the chunks already sent
• a relay and a device.exec with output running at once on one worker each get only their own pushes
• relay.request, relay.body and relay.cancel from an admin session that is not the gateway's uplink answer FORBIDDEN
• a relay.request reusing an active relayId fails BAD_REQUEST; a relay.body for an unknown relay id answers {}; relay.cancel for an unknown relay id answers {}
• a relay.body with a seq out of order, a repeated seq, a chunk after end, or any chunk for a relay opened with hasBody: false fails BAD_REQUEST and aborts the target's request
• each bound in the field table is refused BAD_REQUEST one past its limit (a 65-character relay id, an upper-case target, an unknown method, a relative or file: URL, a 4097-character URL, 101 headers, a header name with a space, a value with a newline, an 8193-character value, a 301-character tenant, a bad lease id, GET with hasBody), and passes at its limit
• the caller's authorization, x-agent-device-token, connection and transfer-encoding headers never reach the target, and a target's content-length and connection headers never reach the caller
• a target answering 101 headers fails the call with INTERNAL, and the worker logs it
• a 17th relay on one connection fails BAD_REQUEST; the gateway's 9th relay to one worker starts only after one of 8 ends
• a relay.body frame sent right behind its relay.request frame is never dropped (checked with seq 0 plus end sent back to back, 50 times)
• a chunk that is not standard base64, and one that decodes to 262 145 bytes, fail BAD_REQUEST

Gateway-side protocol violations are driven by writing raw frames into the
client's wire (src/simlock-client/wire.ts) over an in-memory connection,
modelled on the wire's existing tests, with the gateway's GatewayRelay on
top:

• a relay-head with status 101, and a relay-chunk that is not base64, fail the relay with INTERNAL naming the worker, and the gateway sends relay.cancel
• relay.request answering {} with no head, and { cancelled: true } with no cancel sent, fail the relay with INTERNAL saying the worker ended it without a response
• a worker INTERNAL after the head errors the caller's body stream with that message
• DAEMON_STOPPING from a worker reads as a lost uplink in the relay, the coordinator and the component relay
• SimlockAdminClient as simlock/admin exports it has no relay methods
• RELAY_TARGET_MISSING maps to HTTP 503 and CLI exit 1

Done when

• Every Tests line passes.
• docs/internal/ARCHITECTURE.md's gateway section names the relay
  operations, and docs/internal/COMPONENTS.md lists GatewayRelay and the
  worker's relay targets.

Out of scope

• Registering agent-device as a target, and any HTTP route that uses the
  relay (feat(agent-device): open, commands, heartbeats and close through a gateway #515).
• Binary frames and flow control for the uplink as a whole.

Depends on

• feat(fleet): a lease request may require a capability #513

Approval

• Approved for delivery

Written by an agent.