Skip to content

Add an MCP server to pace-node for inspecting and operating the cluster - #1744

Merged
christianparpart merged 79 commits into
masterfrom
feature/pace-node-mcp
Nov 4, 2026
Merged

christianparpart merged 79 commits into
masterfrom
feature/pace-node-mcp

Conversation

@christianparpart

Copy link
Copy Markdown
Member

An operator working in a Claude Code session can now ask a running pace-node about its cluster and act on it through the Model Context Protocol (revision 2026-07-28, with a 2025-11-25 fallback). Questions it answers: who is compiling what right now, what the other nodes are doing, which toolchains the fleet runs and whether an upgrade is advisable, why two nodes report different fingerprints for the same toolchain, how CPU load and compile counts developed over hours, days or months, and who triggered how many compiles. Actions it covers: seeing pending joiners, approving or rejecting them, opening the auto-admit window, cordoning, renaming nodes, and changing a node's memory or disk cache budget live and persistently. The dashboard remains the regular admin's tool; MCP shares its derivations rather than computing a second answer.

The surface is never network-bound: --mcp-listen on a non-loopback host is a startup refusal and every request is re-checked as loopback before its body is parsed. It adds no credential and no delegation construct — the node forwards to the leader or a named member over proven 0xFC as itself. Acting tools need operator standing (a voter); a learner is refused by name. cluster_forget is deliberately not offered, and secrets in the effective configuration are redacted.

Changes

  • Core and protocol: a strict, bounded JSON parser/writer (Core/Json); the MCP envelope and tool dispatch (Protocol/Mcp/); admin HTTP routes may accept a bounded POST body.
  • The surface: --mcp-listen, loopback only, restart-only; forwarding through IMcpUpstream (leader / named node / self, the self-call presenting the node's own proof).
  • Read tools: fleet overview, running compiles, conditions, node status (by id or display name), history, toolchains with a conservative upgrade report, toolchain_diff, enrollment status, admission explanation.
  • Action tools: enrollment approve/reject/auto-admit, cordon, set_display_name, set_cache_budget, clear_cache_budget.
  • Wire (0xFC v17, introduced on this branch): FLEET-TEXT carries the fleet JSON and is readable by a member that proved its roster key; new FINGERPRINT-INPUTS (0x25), MEMBER-EDIT (0x26), MEMBER-REPORT (0x27); AskOrigin on ADMIT/LEASE.
  • Replicated state: member display names (unique under ASCII case folding, deciding nothing) and per-node cache budget overrides. CommandVersion 7 / StateVersion 11, so a store written by an earlier build is refused as UnsupportedFormatVersion.
  • Budgets: a cache tier can be resized live (disk eviction in slices on a driver thread). An override records the node's configured budget as its baseline; a later local edit to the config wins, raises the latched cache-budget-override-dropped condition and asks the leader to drop the override — decided by the recorded baseline, never by clocks. Overrides survive restarts; application runs off the Raft apply path and the reactors. --cache-memory/--cache-disk become reloadable.
  • History: CPU busy per mille in node history; history filed under the machine's proven key, top 256 machines by recent activity (--history-requesters); compiles per requester and per toolchain over time with coverage per bucket; upgrade advice weighs observed toolchain demand.
  • Fingerprints: each node captures the inputs of a toolchain fingerprint once, so toolchain_diff can say which input differs between two nodes.
  • pacectl: --cluster-name, --cluster-budget, --cluster-budget-clear; the node report shows the MCP port, budget sources and overrides.
  • Docs and rulebook: docs/operations/mcp.md; new rule sections (MCP loopback-only and one derivation with the dashboard, fingerprint inputs captured not re-gathered, budget overrides by baseline, display names decide nothing) with their AGENT.md tripwires.

Draft: the full local gate (gcc-release + clang-debug with the pinned clang-tidy), flake-rate checks and a final whole-branch review are still in progress, and the branch will be rebased onto master before it is marked ready.

@christianparpart christianparpart added the type/feature New capability, or a user-visible extension of one label Nov 4, 2026
@github-actions github-actions Bot added area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon area/distributed Distributed/, apps/fastcache-compile-node/ - dispatch, workers, scheduler area/cluster Consensus/, Cluster/ - Raft, discovery, membership area/protocol Protocol/, Auth/ - memcached, RESP, autodetect, the auth gate area/net Net/, Async/, Server/ - sockets, reactors, connections area/storage Cache/, CowTree/ - engine, LRU, layered, sharded, compression area/platform Platform/, Config/, Cli/ - service registration, config lookup, the CLI table area/observability Metrics/, logging, stats and scrape surfaces area/packaging packaging/, CPack, deb/rpm/pkg/msi, installers area/core Core/ - the shared foundation; high blast radius labels Nov 4, 2026
Pace::Json parses RFC 8259 strictly (no duplicate keys, lone surrogates,
invalid UTF-8, leading zeros or trailing data) under named ParseLimits,
and writes compact output. Integers outside int64 become doubles instead
of wrapping. AppendString is the reusable string escaper.

Signed-off-by: Christian Parpart <christian@parpart.family>
Json_test.cpp was left out of the previous commit. Tagged [json]; picked
up by the src/Pace/*_test.cpp glob in src/tests/CMakeLists.txt.

Signed-off-by: Christian Parpart <christian@parpart.family>
AdminRoute gains a method and a body ceiling; the declared Content-Length is
judged against the matched route before any body byte is buffered (413), a POST
without a length is 411, Transfer-Encoding is 501, duplicate framing headers are
400, and the body is read within a total BodyTimeout. 405's Allow is derived from
the routes registered on the path. AdminRequest carries the method, the body and
an owned peerHost; a throwing handler costs a 500.

Signed-off-by: Christian Parpart <christian@parpart.family>
…11-25 fallback

McpServer turns one HTTP request into one HTTP response over a fixed tool
table: header/body agreement (-32020, base64 sentinel decoded), required
_meta fields (-32602), unsupported versions (-32022 with data.supported),
server/discover, tools/list with annotations and schemas, tools/call with a
required-key and primitive-type argument check, Origin validation (403),
notifications (202) and a throwing handler answered 500/-32603 and logged.
A legacy client is served from the same table, selected only by
`initialize` or a 2025-11-25 MCP-Protocol-Version header; no session is
minted. Method dispatch is a table of {method, era mask, handler} rows.

Signed-off-by: Christian Parpart <christian@parpart.family>
A tool handler that throws is now answered 200 with a tool result carrying
isError:true and "internal error in <tool>; see the node log", rather than a
500 with -32603: an HTTP MCP client reads a non-2xx as the transport failing,
so the model never learned which tool broke (ruling R8). The cause stays in
the node log.

A tool that declares an outputSchema and answers success with no structured
content is refused the same way and logged (ruling R9): a client enforcing
output schemas rejects it, and one reading only the text is handed a wrong
answer that looks right.

Also pins the legacy ping result as {} and the Mutating annotations as
readOnlyHint:false, destructiveHint:false.

Signed-off-by: Christian Parpart <christian@parpart.family>
…d built-ins

AdminHttpServer takes an AdminServerProfile: the name its accept loop reports
under (to AcceptLoopHealth and the log), how many requests it serves at once,
and whether it answers /metrics and /healthz. All three were constants of the
one admin surface; the MCP surface runs the same hardened HTTP code and must
report as itself, carry its own cap, and not grow a scrape endpoint by reuse.
The default profile is the admin surface's, so every existing caller is
unchanged.

Also pins what peerHost is off a real socket: "127.0.0.1" over IPv4 and "::1"
over IPv6, unbracketed and portless.

Signed-off-by: Christian Parpart <christian@parpart.family>
A new NodeSurface row, Mcp, behind --mcp-listen=[host:]port (yaml mcp_listen):
off unless set, a bare port binds 127.0.0.1, and any address that is not a
loopback literal -- the wildcard, another host, or a name, localhost included
-- is a StartupPolicyRejection row (McpListenNotLoopbackRefusal), so the
start, --print-surfaces and --install-service refuse it alike. The service
registration carries the flag only when it was typed (emitIfExplicit).

McpEndpoint serves one route, POST /mcp, on a thread of its own through the
same AdminHttpServer code the admin surface runs, profiled as surface "mcp"
with MaxConcurrentRequests = 4 and no /metrics or /healthz. Before the
envelope is parsed, every request asks McpAdmits whether the connection's
kernel-reported peer is on this machine; one that is not is answered 403 and
moves pace_mcp_requests_refused_not_loopback_total, a new counter on a new
NodeMcp metrics surface (absent on a node without --mcp-listen). The tool
table is empty for now; tools/list answers [].

main chains the MCP start onto the admin surface's, so a refusal of either
ends the start through one branch. node-mcp-e2e drives a real node over curl
in both eras and checks a network address is refused at startup.

Signed-off-by: Christian Parpart <christian@parpart.family>
…en 0xFC

NodeIoUpstream puts one 0xFC request frame to the Raft leader or to a
named member from the blocking MCP thread: the call is posted onto the
node's I/O loop as a spawned flow, dialled through the existing
IUpstreamDialer (ProvenSchedulerDialer in production), and awaited on a
condition variable bounded by McpUpstream::CallDeadline (10 s). A wait
that runs out posts the cancellation onto the loop, which closes the
connection the flow is parked on, so the frame unwinds and is freed
there. NotLeader is followed for leader calls only, at most
MaxLeaderRedirects times, judged by Cc::RedirectTarget; a target that is
this node is answered in-process. AppliedUpstreamDirectory maps member
ids to their applied schedulerEndpoint, matched exactly.

The dialer now says which half of opening failed (UpstreamOpeningFault),
so a refused proof is reported apart from an unreachable node, and takes
its reply ceiling as a parameter for callers that forward reports.
ProvenLeader moves into the shared admission fixtures.

Signed-off-by: Christian Parpart <christian@parpart.family>
ProvenSchedulerDialer filed every unproved handshake as ProofRefused, so
MCP told a person their node's identity was rejected when the leader was
only busy past the handshake deadline, or restarting and not yet able to
judge the key. NodeProofAttempt now records whether its last exchange
was answered (ProofExchange), and UpstreamOpeningFault has one row per
remedy: HandshakeIncomplete, ProofDeferred, ProofNotOffered,
PeerUntrusted and ProofRefused. McpUpstream maps them through
OpeningFaultTable to Unreachable, NotReady, NoProofOffered,
PeerUntrusted and ProofRefused. The admission relay reads none of this
and fails open exactly as before.

Signed-off-by: Christian Parpart <christian@parpart.family>
FLEET-TEXT gains a trailing format field (FleetTextFormat: Text = 0,
Json = 1). With Json the reply is RenderFleetJson over the same history
view /fleet.json uses, so MCP and the dashboard share one derivation;
a section is refused for Json rather than silently dropped. Leader-only
semantics are unchanged (a follower answers NotLeader naming the leader).

WireSurface gains Mcp = 0x04, mapped from NodeSurface::Mcp in NODE-STATUS
and rendered by `pacectl node` as mcp-port.

CurrentVersion and MinSupportedVersion move 16 -> 17.

Signed-off-by: Christian Parpart <christian@parpart.family>
Nothing earlier on the Fleet family's path asks IsSupported, so a version-16
three-field request was answered MalformedFrame and moved the malformed
counter. FleetTextResponder now judges the version before decoding and answers
UnsupportedVersion naming the range (uncounted, with a rationale). The wire
comments now say who refuses.

Signed-off-by: Christian Parpart <christian@parpart.family>
The leader's fleet document now carries every compile in progress -- one
row per admission ticket, queued, waiting on an identical storable compile,
or running -- each running one joined to its worker lease by the ticket id
the lease now carries, never by cache key. A ticket granted on its own
caller reports worker `local` and no expiry; anything the join cannot
establish is null. Totals are counted over every ticket before the list is
cut at FleetJsonCompileCeiling.

/fleet.json also lists every outstanding lease up to FleetJsonLeaseCeiling
(the page and /fleet.txt keep the oldest 50) with
leases-outstanding-truncated, so FleetJsonSchema moves to 3. The page gains
a Compiles section driven by the same column table.

The fleet's JSON escaper is folded into Json::AppendString, the tree's one
JSON escaper (\b and \f are now written short), and /fleet.json answers
through AnswerFleetText as /fleet.txt and the fleet-text verb do.

Signed-off-by: Christian Parpart <christian@parpart.family>
A fleet document published the queue's depth and the compile totals from
two separate acquisitions of the admission lock, so a ticket arriving
between them made `depth` and `compile-totals.queued` disagree in one
document. `SchedulerService::AdmissionReportNow()` now returns the queue
view and the joined tickets under ONE acquisition, and is the only door to
either half: `AdmissionDetailNow()` and `CompileTickets()` are gone, so a
two-read report cannot be written.

A compile row whose request named no fingerprint now renders it absent
(`null`), as the toolchain column already did, never as an empty string.

Signed-off-by: Christian Parpart <christian@parpart.family>
…EET-TEXT

A member's MCP fleet tools reach the leader over the proven, sealed 0xFC
connection, and a node holds no dashboard credential. The fleet read gate
now takes what the connection proved (FleetReader): a live identity key the
roster holds (RestsOnProvenIdentity, read off the same fold RefusePeer asked)
stands in for the credential. The fleet subscription stays by address.

Signed-off-by: Christian Parpart <christian@parpart.family>
fleet_overview, running_compiles, conditions, history and enrollment_status,
each put to the leader over IMcpUpstream (FLEET-TEXT Json / series section,
ENROLL-CONTROL List) and projected mechanically: snake_case keys, every
Millis column in seconds as <key>_s, null kept null. main now builds the
upstream (AppliedUpstreamDirectory told by the consensus observers, the
proven dialer, this node's own router for a call to itself) behind a gate
that refuses at once when the 0xFC loop is not turning.

Signed-off-by: Christian Parpart <christian@parpart.family>
…le says who reads FLEET-TEXT

The directory read the scheduling leader alone, which is EMPTY while this node
leads, so every fleet tool on the leader answered no-leader-known. The role
observer now tells it whether this node leads (ObservedRole), and the leader
endpoint is then its own. The e2e drives both tools through main's wiring on a
one-node scheduler, with and without --dashboard-token-file.

distributed-compilation.md and LiveStatsResponder's comment now state the
policy c67ede0 introduced: FLEET-TEXT admits a caller proving a live roster
key without the dashboard token; the live-stats fleet stream stays by address.
An absent compile total renders `-` in the running_compiles text, never 0.

Signed-off-by: Christian Parpart <christian@parpart.family>
`node_status {node?}` gives one node's full status. It asks this node in-process
by default, or a member named by its exact id over the proven 0xFC connection.
It decodes the NODE-STATUS reply `pacectl node` reads, and both render from the
shared `NodeStatusView`.

NODE-STATUS runtime fields 26-29 are added within 0xFC v17:
- slot ceilings: `SlotCeilingsFor` over the load the last heartbeat sent, with
  the binding ceiling named
- cache tiers, carried positionally by StorageTier, so an absent tier stays
  absent
- the configuration in force after reloads, through a mandatory disclosure
  table; --requirepass is redacted at the sender and never becomes text
- budget overrides, declared and still unfilled (Task 19 fills them)

`ResolveNodeName` matches exact ids through a table of match rows, so the
display-name match is one more row. There is no prefix matching. An unknown or
ambiguous name is an error that lists the candidates.

`pacectl node` shows the new cells. The MCP tool helpers move into
`McpToolSupport.hpp`, shared by the fleet and node tools.

Signed-off-by: Christian Parpart <christian@parpart.family>
…t figures, marked bytes

- The redacted settings are exactly the option rows whose operand is
  `SecretOperand` (`=<secret>`), and a test holds that equality.
- A setting value that is not UTF-8 reads `<non-utf8: ...>`, with each
  invalid byte spelled `\xHH`, and MCP flags it `non_utf8`. It is never
  silently replaced.
- `Pace::Json` carries uint64 values exactly (`AsUint`, `Json::Unsigned`),
  every u64 in `node_status` goes through `Json::Unsigned`, and `HitRatio`
  cannot wrap.
- `tiers` and `effectiveConfig` are count-prefixed like `budgetOverrides`, so
  an engaged empty list is no longer read back as absent.
- Slot ceilings go through `NodeStatusView::CeilingsOf` for both pacectl and
  MCP.
- NODE-STATUS withholds `effectiveConfig` from a caller admitted by
  `--fleet-open` alone (`RestsOnIdentifiedCaller`).
- The `StartupPolicyRejection` whitespace is restored. The anonymous
  `LogLevelName` becomes the public `NodeLogLevelName`, and the
  `NodeLogLevelSpelling` wrapper is gone.

Signed-off-by: Christian Parpart <christian@parpart.family>
…ts reach; review round 1

- `BudgetOverrideApplier::Stop()`: idempotent, joins the thread, and every later `OnState`/`ReloadAccepted` is a
  no-op, with a pass abandoned between tiers. `WorkerBody` calls it as soon as its serve loop ends, and a
  `StopBeforeTeardown` guard declared after the `0xFC` surface stops it on every early return too -- on the leader a
  report is answered in-process through that surface's router, which was destroyed before the applier.
- A failed re-report of an override already in force keeps it in force (no grow-then-shrink).
- `cache-budget-override-dropped` survives a reform: the latch is a `BudgetDropLatch` `main` owns across bodies,
  and each applier raises the row from it at construction.
- `AttributedMemberOf`: a candidate key no member holds names nobody, so a loopback caller that proved such a key
  falls through to the leader's own id.
- MCP `node_status` output schema documents each `budget_overrides` row field.

Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
Signed-off-by: Christian Parpart <christian@parpart.family>
node-mcp-hostile-e2e starts a real pace-node with --mcp-listen beside
--admin-listen and its cache tier, and sends the MCP port a 1 MiB body
(413), a document 33 containers deep (400/-32700, with a 32-deep control
answered 200), bytes that are not UTF-8 (400/-32700), a body that stops
at 10 of 100 bytes (408 and closed after BodyTimeout, timed on the
client's monotonic clock) and twenty idle connects. After every case,
and during the two that hold a connection, /healthz answers inside one
second and a pace-cc compile through the node's tier is a HIT; once the
idle client lets go, MCP answers again within a derived bound, and the
node stops cleanly.

It does not assert that MCP answers WHILE the twenty are held: it does
not (one blocking thread, FirstByteTimeout 30 s per silent peer, 614 s
measured for twenty), and that is reported for a ruling rather than
written down as expected behaviour.

A sibling of node-mcp-e2e rather than more cases in it: that fixture
already runs near its 300 s budget, and this one names pace-cc too.

Signed-off-by: Christian Parpart <christian@parpart.family>
node-mcp-enrollment-e2e starts a one-node consensus leader and two
joiners with --fleet-seed at it, on this host's primary IPv4 address (a
fleet on loopback offers itself to no joiner; a host with none skips).
The leader's enrollment_status lists both, pending, under the keys they
printed; enrollment_approve answers decision approved and the joiner
counts itself a learner; enrollment_reject answers decision rejected.
From the learner's own MCP surface, approve, reject and auto-approve are
refused naming its missing operator standing with the remedy, and its
enrollment_status is refused as a control verb; the leader's list does
not move.

Carried from Task 10's review (finding 6): the decision field was
unproven against a real leader until now.

Signed-off-by: Christian Parpart <christian@parpart.family>
clang-format 22.1.8 (.clang-format-version), as `scripts/local-gate.sh`
runs it, re-wraps a TEST_CASE name line in ToolchainProbe_test.cpp and
re-aligns StartMcpSurfaceOrExplain's parameters in McpEndpoint.cpp. No
change but whitespace.

Signed-off-by: Christian Parpart <christian@parpart.family>
…builds

The gcc-release leg (g++ 16.2, -O3, -Wnull-dereference as an error) had
never built this branch (ruling R6), and refused McpFleetTools.cpp and
McpToolchainTools.cpp. Every site was the same shape: a member checked
non-null in one expression (`totals->AsObject() == nullptr`, a
`count_if(...) == 1`) and dereferenced again in another
(`*totals->AsObject()`, `findIfOrNull(...)->id`), which the optimiser
cannot connect.

Each pointer is now taken once, checked, and the checked local is read:
the compile totals object, a condition row's members, the keyed
history's `keys` (through a new `ElementsAt`, empty when absent, beside
`ElementsOf`), the demand section's `keys`, and `NamingWhere`'s member,
found first and then counted. No answer changes: every path the checks
refused before is refused the same way.

Signed-off-by: Christian Parpart <christian@parpart.family>
…e tests

The same gcc-release leg refused eleven test translation units under
-Wnull-dereference. `REQUIRE(p != nullptr)` followed by `*p` is not
enough for the optimiser: a failed REQUIRE returns through Catch2's
handler as far as it can see. So `Testing::Unwrap` gains an overload for
a pointer -- the optional overload's contract, a REQUIRE first and a
shared default never observed -- and the flagged `*x.AsString()`,
`*x->AsArray()`, `x.AsObject()->size()` sites read through it.

Also from that leg: two `for (auto const [a, b]: { ... })` loops copied
each pair (-Wrange-loop-construct) and take a reference now, and one
CompileCacheWire case's lambda replaced `parts.back()` of a split that
could be empty -- a payload with no fields now answers "not refused",
which fails the case rather than reading past the end.

Signed-off-by: Christian Parpart <christian@parpart.family>
… one in 2 s

The MCP surface ran AdminHttpServer's accept loop over a blocking
listener, which serves each connection to its end before the next
accept: MaxConcurrentRequests = 4 was never reached, and a peer that
sent nothing held the whole surface for the admin FirstByteTimeout
(30 s). Twenty silent connects shut MCP off for 614 s, measured.

AdminServerProfile gains two columns. `serving`
(AdminConnectionServing): OnAcceptThread, the admin surface's model and
the default, or OnOwnThreads, where the server owns one thread per slot
and each connection hops onto one before it reads, so a held
connection holds a slot and the connection past the cap is answered
503 by the accept loop. `deadlines` (AdminDeadlines): the first-byte,
per-read, head, body and linger bounds as a row, the default row being
the admin surface's existing constants, which now read from it.

McpEndpoint states its own row: first byte 2 s (the 30 s exists for a
browser's preconnect, which no MCP client opens), head 2 s in total,
request and body 2 s as before, and a 400 ms linger so a 503 written past
the cap does not stall the accept thread. The loopback re-check per
request is unchanged.

Measured, twenty silent connects held by the client: an MCP request made
at once is answered 503 in 1.4 s, MCP answers 200 within a second of
polling while the twenty are still held, and the server has ended all
twenty itself (16 answered 503, 4 closed silently after 2 s).

Tests: a request is answered while an idle connection holds a slot, the
fifth concurrent connection is answered 503 within the first-byte
bound, an idle connection is closed after MCP's own bound, and the
hostile e2e asserts the twenty-connect case above. Neutered: serving on
the accept thread fails the first two cases and the e2e; dropping MCP's
deadlines row fails the third and the e2e (a 503 after 7.9 s).

The docs' troubleshooting row and the rulebook's MCP-threads bullet said
one request at a time; both now say what is true.

Signed-off-by: Christian Parpart <christian@parpart.family>
`std::vector::insert_range` and `std::set::insert_range` are C++23 members the libstdc++ and libc++ on
CI's Linux gcc and clang legs do not ship, so FingerprintDiff.cpp, CompilerVersion.cpp and
McpToolchainTools.cpp did not compile there. Each is now the iterator-pair `insert`, which every
standard library has. No other C++23 container range member or library feature the branch adds is
missing from master's own usage (`std::ranges::contains` is used on master already).

MSVC's /W4 /WX refused AdmissionQueue.cpp(195), C4457: a loop local named `ticket` hid the
`Settle(Ticket ticket, ...)` parameter. Renamed `waiterTicket`. A `-Wshadow-all` syntax-only pass
over the 187 TUs the branch touches, filtered to branch-added lines (with a positive control: the
unfixed AdmissionQueue line is reported), found two more of the same shape in McpFleetTools.cpp --
a structured binding and a lambda parameter named `state` beside a local `state` -- renamed
`ticketState`. The other hits are same-name init-captures (`x = std::move(x)`), which master
already uses widely under MSVC /W4.

Signed-off-by: Christian Parpart <christian@parpart.family>
…ing sum

`NamingOfFleetRow` compared the row's `name_source` to the literal "replicated", the word the
leader's renderer spells through `Distributed::NameSourceWord`; a rename of that word would have
silently nulled every display name `fleet_overview` and `conditions` report. It now asks the table.

`FleetKeyedHistory.cpp` carried a private `SaturatingAdd` identical to `Core/SaturatingAdd.hpp`'s;
the copy is gone and the shared one is used.

Signed-off-by: Christian Parpart <christian@parpart.family>
… loop cannot pass vacuously

The `Unwrap(T const*)` doc block had been pasted inside `UnwrapMutable`'s, so the two read as one
and `UnwrapMutable` had none. Each now carries its own, and the pointer overload names the
default-initializable `optional` overload whose contract it shares.

The fleet `conditions` case iterated `Unwrap(documentRows[index].AsObject())` with nothing
required first: a row that was not an object iterated nothing and passed, and a document shorter
than the tool's list subscripted past its end. It now requires the document to hold at least as
many rows, each row to be an object, and each to be filed under the machine being compared -- the
document lists the other machine's row after these two. The other `Unwrap(...AsObject/AsArray())`
sites the branch adds are already preceded by a REQUIRE or compared against a non-empty value.

Signed-off-by: Christian Parpart <christian@parpart.family>
…r state goes

A stop that landed during a forwarded MCP call was not bounded the way the surface said, and part
of it was undefined:

- `AdminHttpServer` declared `_inFlight` after `_connections`, so a request still running when
  `Shutdown`'s drain gave up decremented a destroyed atomic while the pool joined. The pool is now
  declared LAST, after everything a connection borrows.
- `Shutdown` drained with `DrainBound`'s five seconds whatever the profile; a forwarded call runs
  up to `McpUpstream::CallDeadline` (10 s), so every stop during one logged a request abandoned.
  The ceiling is a column of `AdminDeadlines` (`drain`, the old five seconds by default); MCP's row
  states `CallDeadline + 5 s`, and `static_assert`s derive it from the call, the response write and
  the linger, and from a request still being read. `McpEndpoint::WorstCaseStop` (accept poll +
  drain, 15.5 s) is asserted within a quarter of the units' `TimeoutStopSec=120`.
- Nothing stopped a tool asking more questions once a stop began. `McpStopFlag`, owned by
  `McpForwarding` and raised by `~McpEndpoint` before it asks its accept loop to stop, is folded
  into `LoopGatedUpstream`, which refuses every later call `Stopping` at once -- so the drain waits
  for the one call in flight.
- `toolchain_diff` asks up to 66 questions, each bounded but their sum not: a peer answering just
  inside each deadline held the call for minutes. `McpToolchainDiff::TotalBudget` (30 s) is
  checked before every question by a per-call decorator, so the call answers within
  `LongestCall` (40 s) or is a tool error saying the budget ran out -- never a partial diff.

Also from the final review (M4): `McpUpstream::MaxReplyBytes` is `static_assert`ed above a whole
FINGERPRINT-INPUTS page and a refusal and below `MaxReportReply`, with the reason; and a reply the
seal refused for its size is reported `TooLarge` (`UpstreamOpening::sealFault`), never as a peer
that went away before it answered. Stale one-thread wording in `AdminHttpServer.hpp` and
`McpEndpoint.hpp` now describes the serving profile, and every `503` as lingering on the accept
thread.

Tests: the endpoint stopped while a pool thread is inside a slow forwarded call stops within
`WorstCaseStop`, refuses the tool's second call `Stopping` without forwarding it, and logs no
abandoned request; the gate refuses `Stopping` ahead of the loop; the budget refuses past 30 s and
a faster fleet compares whole; an oversized reply is `TooLarge` beside a plain close and a non-size
seal fault, both `Unreachable`. Each was shown red with its fix neutered. The member-order change
cannot be: the server's memory is freed only after every member destructor, so ASan sees no use
of freed memory either way, and with the drain sized right the request ends before the join.

Signed-off-by: Christian Parpart <christian@parpart.family>
`docs/operations/mcp.md` said a request is answered within its forwarded call's 10-second
deadline, which `toolchain_diff` -- up to 66 calls -- never promised, and that idle connects make
the surface refuse "for about that long, not longer". It now says each refusal lingers on the
accept thread for up to 400 ms, that a local process that keeps holding or reopening connections
keeps MCP refusing while it does, that `toolchain_diff` starts no call after 30 s and answers
within 40, and what a tool sees while the node stops. `docs/tools/pace-node.md` no longer reads as
a single serving thread, and the hostile e2e's `TruncatedAtMost` derivation counts MCP's 400 ms
linger rather than the dashboard's two seconds.

Signed-off-by: Christian Parpart <christian@parpart.family>
`fleet_overview` and `conditions` handed `std::vector<std::string>` to `std::format` (McpFleetTools.cpp,
three sites). C++23 range formatting (P2286) is absent from libstdc++ 14, which CI's Linux gcc
legs and its clang-on-libstdc++ legs build against, so the TU did not compile there; the local
gate's newer toolchain formats ranges and could not see it. The lists are now joined explicitly,
through `McpSupport::Joined`, moved out of McpToolchainTools.cpp so both tool files share one --
which also drops the brackets and quotes a sentence never wanted.

Proof against CI's own library: every TU the branch touches, or that includes a header it touches
(388), syntax-checked with Ubuntu 24.04's g++-14 / libstdc++ 14. Before this change the probe
reports McpFleetTools.cpp:382 and :384 (the per-TU output is cut at five lines, which hid :641);
after it, no TU reports an error.

Signed-off-by: Christian Parpart <christian@parpart.family>
`McpEndpoint::WorstCaseStop` and `docs/operations/mcp.md` stated the stop bound unconditionally. It
holds while every client reads its answer: a loopback peer that reads a response just fast enough
to keep each blocking write moving holds that write, and the serving thread the join waits for,
past the drain. Nor did the formula count a `503` the accept thread is lingering on when the stop
arrives (up to 400 ms). Both now say so; the behaviour is unchanged and tracked separately.

`toolchain_diff`'s budget refusal said it "starts no question 30000 ms after" it began, and was
prefixed as though the side's node had failed. It now says "more than", names the question that
was not asked, and says the time spent does not tell which node was slow.

An orphaned `-- GCC's` line in `Unwrap.hpp`'s pointer-overload doc is joined back into its sentence.

Signed-off-by: Christian Parpart <christian@parpart.family>
MSVC translates a universal-character-name even inside a raw string literal, so JSON text that
spelled `\ud83d` there was refused (C3850 for a lone surrogate) and mis-tokenized after it, and
`é` would silently have become UTF-8 instead of the escape the parser is meant to see.
Json_test.cpp now spells every `\u` escape either across two adjacent raw literals (`\` ends one,
`u` starts the next) or in an ordinary literal with the backslash doubled; the bytes handed to the
parser are unchanged. Sweep of every C++ file the branch touches (871 raw literals): no `\u`/`\U`
followed by four hex digits remains inside a raw literal.

C4702: `FAIL(...)` followed by `return` is unreachable code to MSVC. The three such helpers the
branch adds -- `BudgetOverrideApplier_test`'s `Memory()`, `SchedulerAdmission_test`'s
`CompileJsonOf` and `FingerprintInputsResponder_test`'s `AllPages` -- now find, then REQUIRE, with
nothing after it. The sweep for `FAIL(`/`SKIP(` followed by a statement and for `std::abort();`
followed by code in branch-added lines found only these three (the two `SKIP`s are inside an `if`).

Signed-off-by: Christian Parpart <christian@parpart.family>
…er than refusing to start

The display names moved StateVersion 10 -> 11 and CommandVersion 6 -> 7, and the decoder
read one version each. v0.6.0 is the first release to carry this code, so a node that ran
it even once holds v6 entries in its own log; the upgraded node refused them (#1542) and
exited 78 at every start, which is why the MSI upgrade from the latest release rolled back.

ReadableStateLayouts and ReadableCommandLayouts now name every layout the decoder reads.
v10 and v6 are strict prefixes of this build's layouts under unchanged verbs, so the fields
they lack read as absent; each is still held to its own arity, a verb v6 never had is
refused by name, and every other version is refused as before.

Signed-off-by: Christian Parpart <christian@parpart.family>
…ache it warmed itself

The held toolchain_diff in 2g holds only on a WARM start. When the fingerprint cache
misses, the registering survey walks the tree itself and keeps what it walked, so the
toolchain is Recorded before the held capture runs, and the diff correctly answers
complete. The fixture read the caller's shared cache (~/.local/state/pace-cc), so
whether it passed depended on whether some earlier run or test had fingerprinted
toolchains[0]. That is why it failed on Linux-gcc-release-arm64 and then on x86-64,
while it passed on developer machines.

The acting node now reads a cache this run owns. A first node fills that cache with
its capture held and never released, and the fixture checks that the cache is
non-empty. The held diff is then preceded by a check that the acting node's
registering survey logged no `hashing N toolchain file(s)` line, so a cold start is
refused by name instead of being misreported as a wrong answer.

Signed-off-by: Christian Parpart <christian@parpart.family>
…ent, which bash 3.2 does not brace-expand

macOS-clang-release failed node-mcp-e2e with "fleet_overview content
(no-token): the body is not JSON: {"jsonrpc":"2.0","error":{"code":-32700,
"message":"Parse error"}}" -- the server answering exactly what it was sent.

The fixture runs under /bin/bash, which on macOS is 3.2.57. bash 3.2
brace-expands a command ARGUMENT whose double-quoted `$(...)` holds an
escaped `\"` and a `{...,...}`: `json_check ... "$(body_of "$(curl ...
--data "{\"jsonrpc\":\"2.0\",...}")")"` split into words and curl sent
`"jsonrpc":"2.0"` as the body. An assignment's right-hand side is not
brace-expanded, which is why the identical requests made through
`response="$(curl ...)"` one line earlier answered 200.

Five sites had the shape; each now fetches into a variable first. The one
remaining argument-form call (`explain_admission`) carries a one-key object,
which has no comma to expand.

Reproduced on Linux with a locally built bash 3.2.57: the pre-fix script
fails with the CI's exact line; the fixed one passes end to end. Under
bash 5.3 node-mcp-e2e passes as before.

Signed-off-by: Christian Parpart <christian@parpart.family>
`/fleet.json refuses an unknown range in JSON and still ignores a section` never ran on the
Windows legs: Catch2 accepts `/`-led options there, so the name ctest handed it failed as
`Unrecognised token: /fleet.json`, while every POSIX leg ran and passed it. Renamed to begin with
`fleet.json`.

`test-name-hygiene` already refused a name beginning with `-`; it now refuses `/` too, and its
remedy says why (an option on Windows alone, so this scan is the only place a POSIX run can see
it). The self-test gains a `slash` case; with the scan's pattern put back to `^-` that case alone
goes red ("slash: expected refuse, got accept"), and the real tree with the old name planted back
is refused by name. `testing.md` and AGENT.md's tripwire bullet name `/` beside `-`, within the
same lines.

Signed-off-by: Christian Parpart <christian@parpart.family>
`A path that is not UTF-8 travels as its bytes` threw on the Windows legs ("No mapping for the
Unicode character exists in the target multi-byte code page"). The throw was the TEST's: it built
the path from narrow bytes that are not UTF-8, which `std::filesystem::path`'s narrow constructor
refuses there before anything else runs. But the effective configuration rendered every path
setting with `path.string()`, which on Windows converts UTF-16 to narrow text and throws for a
sequence that does not convert -- a lone surrogate, which a name read through a wide API can hold
-- turning a NODE-STATUS request into a 500.

`PathBytes` now renders a path without converting through the narrow code page: on POSIX its
native bytes, on Windows its code units as WTF-8 (UTF-8 for every valid sequence, a lone surrogate
as the three bytes UTF-8 would give it), so a name that is not text travels as bytes that are not
UTF-8 and the reader marks them, as it does a POSIX path's stray bytes. The case builds the path
from each platform's own form -- `\xff` bytes on POSIX, a lone `\xD800` code unit on Windows,
expecting `\xED\xA0\x80` -- and asserts the bytes travel. The Windows encoder was checked off
Windows against a lone surrogate, a pair and a reversed pair.

Signed-off-by: Christian Parpart <christian@parpart.family>
`The node series records this machine's CPU from its own sampler` failed once on the clang-tsan
leg with no TSan report: the sampler thread takes its own first sample at construction, at a
moment the case did not choose. When it landed AFTER the case's first clock advance, the case's
own `SampleOnce` took the scripted absent reading and the thread's late sample put 420 into the
same minute, so no bucket was left with the CPU slot absent.

The case now waits (`Testing::WaitUntil`, bounded) until the construction sample has landed in the
node series before it advances, so the readings are consumed in one order: absent into the first
minute, 420 into each later one. What it asserts is unchanged. Renamed without its comma, which
made the name unselectable by hand.

Proof, by a 300 ms delay injected into the scripted sampler's first call (not committed): without
the wait 6 of 10 runs failed at exactly the CI's CHECK; with it 10 of 10 passed. The committed
case passed 200 of 200 runs beside 16 busy loops. No sibling case constructs a FleetSampler over
`ScriptedLoadSampler`.

Signed-off-by: Christian Parpart <christian@parpart.family>
…d command

Signed-off-by: Christian Parpart <christian@parpart.family>
… is damage, not another build's

A version 6 command naming a verb v0.6.0 never had is bytes no build wrote, so it
is refused MalformedFrame -- a held one now reads as a damaged store -- while a
verb byte past CommandKind::Last stays UnknownMessageType. The intact-store
refusal no longer denies that any conversion exists beside a context naming the
one this build has, and both readable-layout tables are static_asserted to name
each version once, oldest first.

Signed-off-by: Christian Parpart <christian@parpart.family>
…renders its state directory through it

The lenient UTF-16 to WTF-8 encoder moves out of EffectiveConfig's anonymous
namespace into Platform/NarrowText as Wtf8FromUtf16 (portable, tested on every
platform over char16_t) beside PathBytes, which EffectiveConfig, NODE-STATUS's
state directory and the received-history log line now share instead of
path.string(), which throws on Windows for a path that does not convert.

Signed-off-by: Christian Parpart <christian@parpart.family>
@christianparpart
christianparpart marked this pull request as ready for review November 4, 2026 08:39
@christianparpart
christianparpart merged commit 7aecefd into master Nov 4, 2026
26 of 27 checks passed
@christianparpart
christianparpart deleted the feature/pace-node-mcp branch November 4, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cluster Consensus/, Cluster/ - Raft, discovery, membership area/compile-cache CompileCache/, the 0xFC wire, the executor, PathCanon area/core Core/ - the shared foundation; high blast radius area/distributed Distributed/, apps/fastcache-compile-node/ - dispatch, workers, scheduler area/launcher apps/fastcache-cc/ - the compiler launcher and its cache key area/net Net/, Async/, Server/ - sockets, reactors, connections area/observability Metrics/, logging, stats and scrape surfaces area/packaging packaging/, CPack, deb/rpm/pkg/msi, installers area/platform Platform/, Config/, Cli/ - service registration, config lookup, the CLI table area/protocol Protocol/, Auth/ - memcached, RESP, autodetect, the auth gate area/storage Cache/, CowTree/ - engine, LRU, layered, sharded, compression type/feature New capability, or a user-visible extension of one

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant