scripts: small relay networks have a repeatable production lifecycle test

README.md

@@ -46,7 +46,7 @@ Site hosting and mirroring are on by default; Marmot and GRASP are off until ena
 
 **For scripts and agents**
 
-- [Scripts and agents](docs/13-scripts-and-agents.md): a relay end to end with a key and curl, every management method including Git storage inventory, and the relay as a configuration file.
+- [Scripts and agents](docs/13-scripts-and-agents.md): a relay end to end with a key and curl, every management method including Git storage inventory, the relay as a configuration file, and a finite production network exercise.
 - [HTTP reference](docs/14-http-reference.md): every path, method, auth and answer.
 
 **For developers**

docs/12-develop-extend.md

@@ -74,7 +74,7 @@ scripts/
            build-templates.mjs                fold relay-templates/ into src/gen/templates.ts
   check/   check-console.mjs check-celld.mjs check-config.mjs   run by npm run typecheck; check-config also checks any file
   dev/     dev-signer.mjs seed.mjs stage.mjs junk.mjs shot.mjs zaptest.mjs   npm run dev:signer, dev:seed, dev:stage, dev:junk, dev:shot, dev:zaptest
-  ops/     margin.mjs relay.mjs               npm run margin; npm run relay check|plan|push|pull
+  ops/     margin.mjs network.mjs relay.mjs    npm run margin; npm run test:network; npm run relay check|plan|push|pull
 
 Every script has an npm name (package.json), and the docs use those names: a tutorial never says node and a path.
 wrangler.jsonc        the Worker on Cloudflare
@@ -87,6 +87,7 @@ wrangler.celld.jsonc  the same Worker on celld (docs/16)
 npm test                  # unit and Durable Object tests
 npm run typecheck
 npm run test:conformance  # against RELAY_URL, default ws://127.0.0.1:7447
+npm run test:network -- plan  # print the finite production topology; run and cleanup are manual
 ```
 
 The conformance suite needs a claimed relay. Against a dev server:
@@ -97,6 +98,13 @@ CLAIM=1 RELAY_URL=ws://dev.localhost:8787 npm run test:conformance
 
 CI runs typecheck and the object tests on every push; the conformance suite against the Worker on `celld dev` is a separate workflow run on demand ([Hosting without Cloudflare](16-hosting-without-cloudflare.md)). Work lands through pull requests in small commits that each typecheck on their own; the branch checks and review carry the change to `main`.
 
+The production network exercise in [Scripts and agents](13-scripts-and-agents.md)
+is an operator-run check of relay relationships. It creates five small,
+run-owned relays, uses finite one-shot jobs, samples only public events from
+the two named external relays and tears down its own relays. It is not a
+scheduled test, load test or capacity benchmark. Its private manifest supports
+manual cleanup after an interrupted run.
+
 ## Add a management method
 
 1. Add an entry to `METHODS` in `src/manage.ts`: the action it needs (`roles.ts`), `reads: true` if it changes nothing, and `run`. The handler takes what it uses from the call: `str(i)` and `num(i)` for parameters, `s` for settings, `reply({ result })` or `reply({ error }, 400)` to answer. `supportedmethods`, the permission check and the moderation log read the same entry.

docs/13-scripts-and-agents.md

@@ -18,6 +18,66 @@ A relay for a script, an agent or a service, from nothing to handover, without a
 
 The cheap way to learn a community is its views: `GET /view/profiles` is every member's name and picture in one signed record, `GET /view/relays` is where those members also publish, and `GET /view/zaps` says what the place values. Each is one request and no websocket; the relay's information document lists which views it keeps.
 
+## Exercise a relay network in production
+
+`npm run test:network` runs a finite, manually invoked production exercise. It
+creates five small relays, gives run-owned identities, records the topology,
+exercises the relationships, extracts a report and deletes the relays it
+created. It has no timer, workflow or standing job outside the run.
+
+The topology is deliberately small and represents different data-flow shapes:
+
+| Role | Relationship | Activity and size assumption |
+|---|---|---|
+| `peer-a`, `peer-b` | equal peers, each pulls from the other | short bursts, idle catch-up and repeat sync; both are small |
+| `satellite` | pulls selected public profiles from `relay.damus.io` and `nos.lol` | a small node samples two much larger external sources |
+| `personal` | exchanges selected notes with the peers, sends public tasks to `hub` and pulls results back | low-volume personal activity |
+| `hub` | receives tasks and publishes results for `personal`; pulls selected profile context from `satellite` | a small agent-facing relay |
+
+The task and result path is a public relay workflow. It does not represent
+confidential agent computation. The encrypted kind-4 fixture stays on the
+personal relay, and the exercise checks that it is not copied to the hub.
+External sampling is read-only and bounded to three sampled profile authors per source and one retained kind-0
+profile per author. Each sync job has a 120-second wait budget; the run has a
+20-minute scenario budget, with cleanup allowed to finish afterward. A public
+source can be empty, unavailable or rate limited, so its result is reported as
+observed or inconclusive rather than treated as a capacity claim. The two
+external relays are treated as larger than this network; their size is not
+measured.
+
+Start with an offline plan, then run the exercise. The default output directory
+is private state under the user's local application data directory. A supplied
+directory must be new and is created with mode `700`.
+
+```
+npm run test:network -- plan
+npm run test:network -- run [new-output-directory] [domain]
+npm run test:network -- cleanup <output-directory>
+```
+
+The run writes a private `manifest.json` and `report.json` in that directory;
+the manifest contains the run keys and is mode `600`, so it must not be
+committed or shared. The manifest is written before the first claim and after
+each state change. The `finally` cleanup extracts final `stats` and storage
+inventory, then verifies each relay's owner before calling `deleterelay`. If a
+process loses a claim reply or stops before cleanup, run `cleanup` against the
+same directory after checking that the manifest is the intended run. Cleanup
+only follows the generated names and matching owners, and reports any relay it
+could not delete. Baseline fuel snapshots follow provisioning; final snapshots precede deletion.
+Their deltas cover the exercise and observation calls, excluding provisioning
+and teardown. The full final counters also contain provisioning usage. These
+are tenant meters, not a provider invoice. The report includes signed synthetic
+fixtures, job results, per-source coverage and operation-lock retry counts.
+A members-only source refuses an unauthenticated pull; membership alone does
+not supply a job with credentials.
+
+The checks cover bidirectional peer convergence, deduplication, idle catch-up,
+filtered task and result flow, private-event isolation, unauthorized writes and
+management, a refused members-only pull, and separation from external profile
+traffic. One-shot server jobs are removed after each check. The exercise is a
+bounded correctness and relationship test, not a load test or evidence of
+large-relay capacity.
+
 ## Signing a request (NIP-98)
 
 Every management call and every door that needs a key takes an `Authorization: Nostr <base64 event>` header. The event is:

package.json

@@ -20,7 +20,8 @@
     "dev:stage": "node scripts/dev/stage.mjs",
     "dev:junk": "node scripts/dev/junk.mjs",
     "dev:shot": "node scripts/dev/shot.mjs",
-    "dev:zaptest": "node scripts/dev/zaptest.mjs"
+    "dev:zaptest": "node scripts/dev/zaptest.mjs",
+    "test:network": "node scripts/ops/network.mjs"
   },
   "dependencies": {
     "@noble/hashes": "^2.4.0",

scripts/ops/network-client.mjs

@@ -0,0 +1,167 @@
+// The manually invoked network test client: signed management calls and a
+// bounded, read-only WebSocket sample. It keeps retries narrow because a
+// mutation whose result is unclear must remain visible to the harness.
+import WebSocket from "ws";
+import { finalizeEvent, verifyEvent } from "nostr-tools/pure";
+import { getToken } from "nostr-tools/nip98";
+
+const REQUEST_TIMEOUT = 20_000;
+const SAMPLE_TIMEOUT = 15_000;
+const SAMPLE_BYTES = 256 * 1024;
+const RETRIES = 30;
+const RETRY_DELAY = 500;
+const RETRY_MAX_WAIT = 2_000;
+const RETRY_WINDOW = 60_000;
+
+export const clientMetrics = { requests: 0, operationRetries: 0, operationWaitMs: 0 };
+
+const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
+
+export class NetworkError extends Error {
+  constructor(message, status, body) {
+    super(message);
+    this.name = "NetworkError";
+    this.status = status;
+    this.body = body;
+  }
+}
+
+const parseBody = async (response) => {
+  const text = await response.text();
+  if (!text) return null;
+  try {
+    return JSON.parse(text);
+  } catch {
+    return text;
+  }
+};
+
+const retryable = (status, body) => {
+  if (status !== 429) return false;
+  const reason = body && typeof body === "object"
+    ? `${body.error ?? ""} ${body.message ?? ""}`
+    : String(body ?? "");
+  return /operation in progress/i.test(reason);
+};
+
+const errorText = (body) => {
+  if (body && typeof body === "object") return String(body.error ?? body.message ?? "request failed").slice(0, 300);
+  return String(body ?? "request failed").slice(0, 300);
+};
+
+// request sends one signed JSON POST and returns its parsed response. Only a
+// relay's explicit transient operation refusal is retried; timeout, network
+// and other HTTP failures remain errors because their mutation outcome is
+// unknown.
+export async function request(node, sk, body, rpc = false) {
+  const url = typeof node === "string" ? node : node.url;
+  const payload = body ?? {};
+  const retryUntil = Date.now() + RETRY_WINDOW;
+  for (let attempt = 0; attempt <= RETRIES; attempt++) {
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT);
+    try {
+      clientMetrics.requests++;
+      const authorization = await getToken(url, "POST", (event) => finalizeEvent(event, sk), true, payload);
+      const response = await fetch(url, {
+        method: "POST",
+        redirect: "error",
+        signal: controller.signal,
+        headers: { "content-type": rpc ? "application/nostr+json+rpc" : "application/json", authorization },
+        body: JSON.stringify(payload),
+      });
+      const result = await parseBody(response);
+      if (response.ok) return result;
+      if (retryable(response.status, result) && attempt < RETRIES && Date.now() < retryUntil) {
+        const delay = Math.min(RETRY_MAX_WAIT, RETRY_DELAY * (attempt + 1), retryUntil - Date.now());
+        if (delay <= 0) throw new NetworkError(errorText(result), response.status, result);
+        clientMetrics.operationRetries++;
+        clientMetrics.operationWaitMs += delay;
+        await sleep(delay);
+        continue;
+      }
+      throw new NetworkError(errorText(result), response.status, result);
+    } catch (error) {
+      if (error instanceof NetworkError) throw error;
+      throw new NetworkError(error instanceof Error ? error.message : String(error), 0, null);
+    } finally {
+      clearTimeout(timer);
+    }
+  }
+  throw new NetworkError("request retries exhausted", 429, null);
+}
+
+// rpc calls one NIP-86 method and returns its result. A relay error remains a
+// NetworkError so the harness can stop before issuing the next mutation.
+export async function rpc(node, sk, method, ...params) {
+  const response = await request(node, sk, { method, params }, true);
+  if (response && typeof response === "object" && "error" in response) {
+    throw new NetworkError(errorText(response), 200, response);
+  }
+  return response && typeof response === "object" && "result" in response ? response.result : response;
+}
+
+const validSample = (event, filter) => {
+  if (!event || typeof event !== "object" || !verifyEvent(event)) return false;
+  if (filter.kinds?.length && !filter.kinds.includes(event.kind)) return false;
+  if (filter.since !== undefined && event.created_at < filter.since) return false;
+  if (filter.until !== undefined && event.created_at > filter.until) return false;
+  if (filter.authors?.length && !filter.authors.includes(event.pubkey)) return false;
+  return true;
+};
+
+// sample reads at most limit valid signed events from a relay. It never
+// answers AUTH or sends EVENT. An authenticated source is allowed to complete
+// its read or refuse it. Incoming data and time are bounded for endpoints.
+export function sample(url, filter = {}, limit = 3) {
+  const requested = { ...filter, kinds: filter.kinds?.length ? [...filter.kinds] : [1], limit: Math.min(Math.max(limit, 1), 3) };
+  const wsURL = url.replace(/^https:/, "wss:").replace(/^http:/, "ws:");
+  return new Promise((resolve) => {
+    const events = [];
+    let settled = false;
+    let timer;
+    let socket;
+    const finish = (status, error = "") => {
+      if (settled) return;
+      settled = true;
+      clearTimeout(timer);
+      if (socket) {
+        socket.removeAllListeners("open");
+        socket.removeAllListeners("message");
+        try {
+          if (socket.readyState === WebSocket.OPEN || socket.readyState === WebSocket.CLOSING) socket.close();
+          else socket.terminate();
+        } catch {
+          try { socket.terminate(); } catch { /* already closed */ }
+        }
+      }
+      resolve({ events, status, error });
+    };
+    try {
+      socket = new WebSocket(wsURL, { maxPayload: SAMPLE_BYTES, followRedirects: false });
+      socket.on("open", () => socket.send(JSON.stringify(["REQ", "network-sample", requested])));
+      socket.on("message", (data) => {
+        if (settled) return;
+        const raw = Buffer.isBuffer(data) ? data : Buffer.from(String(data));
+        if (raw.length > SAMPLE_BYTES) return finish("failed", "sample response exceeded 256 KiB");
+        let message;
+        try { message = JSON.parse(raw.toString("utf8")); } catch { return; }
+        if (!Array.isArray(message)) return;
+        if (message[0] === "AUTH") return;
+        if (message[0] === "CLOSED") return finish("refused", String(message[2] ?? "source refused query").slice(0, 300));
+        if (message[0] === "EOSE" && message[1] === "network-sample") return finish("complete");
+        if (message[0] !== "EVENT" || message[1] !== "network-sample") return;
+        let valid = false;
+        try { valid = validSample(message[2], requested); } catch { return; }
+        if (!valid) return;
+        if (!events.some((event) => event.id === message[2].id)) events.push(message[2]);
+        if (events.length >= requested.limit) finish("limit");
+      });
+      socket.on("error", (error) => finish("failed", error.message.slice(0, 300)));
+      socket.on("close", () => finish(events.length ? "partial" : "failed", events.length ? "source closed before EOSE" : "connection closed"));
+      timer = setTimeout(() => finish("timeout", "source did not answer within 15 seconds"), SAMPLE_TIMEOUT);
+    } catch (error) {
+      finish("failed", error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300));
+    }
+  });
+}