feat(fleet): a lease request may require a capability #513

Part of #511.

Scope

After this PR, a lease request can say which capabilities its worker must
have (#519), and simlock places it only on such a worker:

• through a gateway, workers that lack one are skipped;
• when the fleet could serve the request but no connected worker that could
  serve it has every required capability available, the request fails
  with CAPABILITY_MISSING: at once when it arrives, and on the next
  dispatch pass when it is already waiting;
• a worker answering directly refuses the same way when it lacks one.

Nothing in simlock sends requires yet; the agent-device front starts to in
#515, through a gateway only. No CLI flag, MCP parameter or HTTP body field
is added; the programmatic client (simlock/client) passes the field
through, since its requestLease takes the contract's input.

In short

lease.request gains an optional requires: a list of capability names.
Example: requires: ["agent-device"] places the lease only on a worker
whose status lists agent-device as available.

flowchart LR
  R[lease.request requires X] --> A{fast-fail table on all workers}
  A -->|rejects| T[that rejection, as today]
  A -->|route or wait| B{same table on workers with X available, new}
  B -->|rejects| F[CAPABILITY_MISSING, new]
  B -->|route or wait| S[routing stages; new filter: has X]
  S --> W[worker checks X again, new]
  W -->|lacks X: view was stale| N[gateway tries the next worker]
  W -->|has X| G[granted]

If this goes wrong, an agent-device lease lands on a worker that cannot run
agent-device and open fails after the grant, or a request waits for a
worker that will never take it.

Technical spec

Modules touched

flowchart LR
  OP["contract: operations lease.request *, errors *"]
  MCP["mcp: contracts (omit requires) *"] --> OP
  GD["gateway: dispatcher #leaseRequest *"] --> FC["gateway: fleet-coordinator *"]
  FC --> AS["gateway: routing/serviceability assess *"]
  FC --> RT["gateway: routing.ts, routing/index.ts, pipeline.ts *"]
  RT --> ST["gateway: routing/stages/has-capabilities (new)"]
  FC -->|requestLease| WD["daemon: dispatcher #leaseRequest *"]
  WD --> CP["capability provider (#519)"]

(* = changed, new = new file.)

• Contract.
  • leaseRequestBaseSchema gains requires: up to 8 names, each
    capabilityNameSchema (feat(fleet): workers advertise their capabilities in status and worker list #519), no name twice.
  • New error code CAPABILITY_MISSING: kind domain, HTTP 422, CLI exit
    12, details { capabilities: string[] }.
  • requires together with idempotencyKey fails BAD_REQUEST (message
    requires cannot be combined with idempotencyKey). Nothing sends both,
    and a replay would otherwise have to be checked against a capability
    state that changed since; refusing keeps the request books unchanged.
  • leaseSimulatorInputSchema in src/mcp/contracts.ts omits requires,
    as it omits fleetRequestId, so the MCP tool and tools/list are
    unchanged.
• Where requires travels. On the lease request's options
  (LeaseRequestOptions and the gateway waiter's options), not on
  DeviceRequest, so the requestSpec that lease.requested and
  lease.rejected carry is unchanged. RoutableRequest
  (routing/pipeline.ts) and routable() in fleet-coordinator.ts gain
  it. The gateway sends it on every requestLease (#attempt).
• Worker check. #leaseRequest in src/daemon/dispatcher.ts checks
  requires after its existing TTL cap, owner and fleetRequestId
  checks and before it calls leasing. It reads feat(fleet): workers advertise their capabilities in status and worker list #519's capability sources
  for the required names, each with feat(fleet): workers advertise their capabilities in status and worker list #519's 1 s budget (past it, the name
  counts as not available). Any name not available fails the request with
  CAPABILITY_MISSING, details.capabilities the missing names in the
  order requested, and message This worker cannot serve <names> (names
  joined by , ). It stores nothing and emits no event, for a direct
  request and a gateway probe alike, as the checks before it do today
  (none of them records lease.declined). The check runs once: a direct
  request that then waits in the worker's queue is granted even if the
  capability turns unavailable meanwhile.
• Fast-fail table. assess in src/gateway/routing/serviceability.ts,
  when requires is non-empty, runs its table on every view first, as
  today. A rejection stands as it is. When that answers route-or-wait, it
  runs the table again on the views that list every required name as
  available. Any rejection from that second run (the no-worker one
  included) becomes CAPABILITY_MISSING, reason unresolvable-spec,
  details.capabilities the required names in the order requested, message
  No worker connected to this gateway can serve <names> for this request.
  assess runs on every dispatch pass for every waiting request, so a
  request already waiting fails this way too once no capable worker is
  left. A worker whose view is starting has no capability list (feat(fleet): workers advertise their capabilities in status and worker list #519), so
  it counts as lacking every capability: a request waiting only on the
  capable worker fails when that worker restarts, and one that arrives
  before the gateway has read any capable worker's status fails at once.
• Routing stage. A new filter stage has-capabilities
  (src/gateway/routing/stages/has-capabilities.ts, exported from
  routing/index.ts), appliesToPhysical: true, keeps a worker only when
  its view lists every required name as available. It runs second in
  warm-then-free, right after takes-requests. With no requires it keeps
  every worker.
• Stale view. CAPABILITY_MISSING joins CANNOT_SERVE_CODES in
  fleet-coordinator.ts: a worker that answers it is left out for this
  request and the walk goes on. When no worker is left, the caller gets
  the worker's CAPABILITY_MISSING, as for the other cannot-serve codes
  today.
• ADR 0026 narrows ADR 0009 for the new stage and the new code.

Contract and event changes

• lease.request input: requires (new, optional).
• New error code CAPABILITY_MISSING (422, exit 12).
• This PR bumps the protocol by one.
• Events: none new. A gateway rejection is the existing lease.rejected
  with reason unresolvable-spec. docs/EVENTS.md and
  docs/internal/EVENTS.md add CAPABILITY_MISSING where they list the
  codes it carries.

Other code on the same state

The state is a waiting fleet request and the worker views it is routed on.

• #rememberCannotServe / #admit (fleet-coordinator.ts): a worker
  that answered CAPABILITY_MISSING is excluded for this request only, as
  for the other cannot-serve codes.
• Worker view refresh (worker-registry.ts, worker-link.ts): a
  capability that turns available makes a waiting request routable at the
  next pass; one that turns unavailable on the only capable worker fails
  every waiting request that requires it at the next pass.
• #rejectUnservable: answers CAPABILITY_MISSING with no change other
  than the new code; it already records lease.rejected.
• Lease request books (worker and gateway): unchanged; a request with
  requires never carries an idempotencyKey.
• The dispatcher's other checks (src/daemon/dispatcher.ts:366-391):
  run first; a session that is not the uplink sending fleetRequestId is
  refused by them before the capability is read.

Rules in play

• architecture.md: the fast-fail table stays in assess alone; the stage
  only filters.
• safety.md: requires is wire input, bounded by schema.
• events.md: no new event; the catalogs name the new code.

Tests

Seam: real gateway and worker daemons in one process over a real uplink,
modelled on src/daemon/gateway-fleet.e2e.test.ts, driving
lease.request through each daemon's dispatch, with each worker's
capability provider set by the test; the stage through a unit test modelled
on src/gateway/routing/stages/idle-queue.test.ts; the error table through
src/contract/errors.test.ts.

• has-capabilities keeps a worker listing every required name as available, drops one listing it unavailable, off or not at all, keeps every worker when requires is empty, and applies to physical requests
• through the gateway, a request with requires: ["agent-device"] is granted on the worker that has it available, never on the one without, over 10 requests
• through the gateway, when the only capable worker's matching devices are all leased, the request waits and is granted there once one is released
• through the gateway, with no worker listing it available, the request fails at once with CAPABILITY_MISSING, message No worker connected to this gateway can serve agent-device for this request, details.capabilities ["agent-device"], and lease.rejected carries reason unresolvable-spec
• through the gateway, when only a worker without the capability knows the model, the request fails at once with CAPABILITY_MISSING
• through the gateway, a request for an unknown model when no worker has the capability fails with UNKNOWN_MODEL, as today
• through the gateway, with no connected worker at all, a request with requires fails with NO_CAPACITY, as today
• through the gateway, a request waiting for the only capable worker fails with CAPABILITY_MISSING at the next pass after that worker's view says unavailable
• through the gateway, a request waiting for a busy capable worker is granted on a second worker once that worker's capability turns available and it has a free device
• through the gateway, a worker whose view says available but whose own check answers CAPABILITY_MISSING is skipped, and the request is granted on the next capable worker; with no next worker, the caller gets that CAPABILITY_MISSING
• through the gateway, a request waiting only on the capable worker fails with CAPABILITY_MISSING at the next pass after that worker restarts into starting
• through the gateway, requires: ["agent-device", "x-other"] when no worker lists x-other fails with details.capabilities ["agent-device", "x-other"] and a message naming agent-device, x-other
• a physical request with requires is routed only to a capable worker
• directly on a worker without the capability, the request fails with CAPABILITY_MISSING, message This worker cannot serve agent-device, nothing is stored in its lease requests, and no event is emitted
• directly on a worker with requires: ["x-other", "agent-device"] and only agent-device available, the refusal names only x-other
• directly on a worker whose capability source takes 2 s, the request fails with CAPABILITY_MISSING within 1.5 s
• directly on a worker, a request with requires that waits in the queue is granted once a device frees, even though the capability turned unavailable while it waited
• directly on a worker with the capability, the request is granted as today
• requires together with idempotencyKey fails BAD_REQUEST and nothing is stored
• a request with requires absent or [] is routed exactly as today, and its lease.requested requestSpec has no requires
• requires with an upper-case name, a duplicate name, or 9 names fails with BAD_REQUEST
• CAPABILITY_MISSING maps to HTTP 422 and CLI exit 12
• the MCP lease tool's input schema has no requires
• simlock/client's requestLease with requires against a worker without the capability rejects with CAPABILITY_MISSING

Done when

• Every Tests line passes.
• docs/HTTP-API.md (fast-fail table, error table), docs/CLI.md (exit
  codes, "A request no worker can serve") and both EVENTS.md files name
  CAPABILITY_MISSING, and docs/CLIENT.md describes requires.

Out of scope

• Any sender of requires (feat(agent-device): open, commands, heartbeats and close through a gateway #515).
• A CLI flag, MCP parameter or HTTP body field for requires.

Depends on

• feat(fleet): workers advertise their capabilities in status and worker list #519

Approval

• Approved for delivery

Written by an agent.