diff --git a/README.md b/README.md index e672af7..b354ec0 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ Site hosting and mirroring are on by default; Marmot and GRASP are off until ena **For scripts and agents** -- [Scripts and agents](docs/13-scripts-and-agents.md): a relay end to end with a key and curl, every management method including Git storage inventory, and the relay as a configuration file. +- [Scripts and agents](docs/13-scripts-and-agents.md): a relay end to end with a key and curl, every management method including Git storage inventory, the relay as a configuration file, and a finite production network exercise. - [HTTP reference](docs/14-http-reference.md): every path, method, auth and answer. **For developers** diff --git a/docs/12-develop-extend.md b/docs/12-develop-extend.md index a5d2160..d1a9964 100644 --- a/docs/12-develop-extend.md +++ b/docs/12-develop-extend.md @@ -74,7 +74,7 @@ scripts/ build-templates.mjs fold relay-templates/ into src/gen/templates.ts check/ check-console.mjs check-celld.mjs check-config.mjs run by npm run typecheck; check-config also checks any file dev/ dev-signer.mjs seed.mjs stage.mjs junk.mjs shot.mjs zaptest.mjs npm run dev:signer, dev:seed, dev:stage, dev:junk, dev:shot, dev:zaptest - ops/ margin.mjs relay.mjs npm run margin; npm run relay check|plan|push|pull + ops/ margin.mjs network.mjs relay.mjs npm run margin; npm run test:network; npm run relay check|plan|push|pull Every script has an npm name (package.json), and the docs use those names: a tutorial never says node and a path. wrangler.jsonc the Worker on Cloudflare @@ -87,6 +87,7 @@ wrangler.celld.jsonc the same Worker on celld (docs/16) npm test # unit and Durable Object tests npm run typecheck npm run test:conformance # against RELAY_URL, default ws://127.0.0.1:7447 +npm run test:network -- plan # print the finite production topology; run and cleanup are manual ``` The conformance suite needs a claimed relay. Against a dev server: @@ -97,6 +98,13 @@ CLAIM=1 RELAY_URL=ws://dev.localhost:8787 npm run test:conformance CI runs typecheck and the object tests on every push; the conformance suite against the Worker on `celld dev` is a separate workflow run on demand ([Hosting without Cloudflare](16-hosting-without-cloudflare.md)). Work lands through pull requests in small commits that each typecheck on their own; the branch checks and review carry the change to `main`. +The production network exercise in [Scripts and agents](13-scripts-and-agents.md) +is an operator-run check of relay relationships. It creates five small, +run-owned relays, uses finite one-shot jobs, samples only public events from +the two named external relays and tears down its own relays. It is not a +scheduled test, load test or capacity benchmark. Its private manifest supports +manual cleanup after an interrupted run. + ## Add a management method 1. Add an entry to `METHODS` in `src/manage.ts`: the action it needs (`roles.ts`), `reads: true` if it changes nothing, and `run`. The handler takes what it uses from the call: `str(i)` and `num(i)` for parameters, `s` for settings, `reply({ result })` or `reply({ error }, 400)` to answer. `supportedmethods`, the permission check and the moderation log read the same entry. diff --git a/docs/13-scripts-and-agents.md b/docs/13-scripts-and-agents.md index 826b3ae..878e16c 100644 --- a/docs/13-scripts-and-agents.md +++ b/docs/13-scripts-and-agents.md @@ -18,6 +18,66 @@ A relay for a script, an agent or a service, from nothing to handover, without a The cheap way to learn a community is its views: `GET /view/profiles` is every member's name and picture in one signed record, `GET /view/relays` is where those members also publish, and `GET /view/zaps` says what the place values. Each is one request and no websocket; the relay's information document lists which views it keeps. +## Exercise a relay network in production + +`npm run test:network` runs a finite, manually invoked production exercise. It +creates five small relays, gives run-owned identities, records the topology, +exercises the relationships, extracts a report and deletes the relays it +created. It has no timer, workflow or standing job outside the run. + +The topology is deliberately small and represents different data-flow shapes: + +| Role | Relationship | Activity and size assumption | +|---|---|---| +| `peer-a`, `peer-b` | equal peers, each pulls from the other | short bursts, idle catch-up and repeat sync; both are small | +| `satellite` | pulls selected public profiles from `relay.damus.io` and `nos.lol` | a small node samples two much larger external sources | +| `personal` | exchanges selected notes with the peers, sends public tasks to `hub` and pulls results back | low-volume personal activity | +| `hub` | receives tasks and publishes results for `personal`; pulls selected profile context from `satellite` | a small agent-facing relay | + +The task and result path is a public relay workflow. It does not represent +confidential agent computation. The encrypted kind-4 fixture stays on the +personal relay, and the exercise checks that it is not copied to the hub. +External sampling is read-only and bounded to three sampled profile authors per source and one retained kind-0 +profile per author. Each sync job has a 120-second wait budget; the run has a +20-minute scenario budget, with cleanup allowed to finish afterward. A public +source can be empty, unavailable or rate limited, so its result is reported as +observed or inconclusive rather than treated as a capacity claim. The two +external relays are treated as larger than this network; their size is not +measured. + +Start with an offline plan, then run the exercise. The default output directory +is private state under the user's local application data directory. A supplied +directory must be new and is created with mode `700`. + +``` +npm run test:network -- plan +npm run test:network -- run [new-output-directory] [domain] +npm run test:network -- cleanup +``` + +The run writes a private `manifest.json` and `report.json` in that directory; +the manifest contains the run keys and is mode `600`, so it must not be +committed or shared. The manifest is written before the first claim and after +each state change. The `finally` cleanup extracts final `stats` and storage +inventory, then verifies each relay's owner before calling `deleterelay`. If a +process loses a claim reply or stops before cleanup, run `cleanup` against the +same directory after checking that the manifest is the intended run. Cleanup +only follows the generated names and matching owners, and reports any relay it +could not delete. Baseline fuel snapshots follow provisioning; final snapshots precede deletion. +Their deltas cover the exercise and observation calls, excluding provisioning +and teardown. The full final counters also contain provisioning usage. These +are tenant meters, not a provider invoice. The report includes signed synthetic +fixtures, job results, per-source coverage and operation-lock retry counts. +A members-only source refuses an unauthenticated pull; membership alone does +not supply a job with credentials. + +The checks cover bidirectional peer convergence, deduplication, idle catch-up, +filtered task and result flow, private-event isolation, unauthorized writes and +management, a refused members-only pull, and separation from external profile +traffic. One-shot server jobs are removed after each check. The exercise is a +bounded correctness and relationship test, not a load test or evidence of +large-relay capacity. + ## Signing a request (NIP-98) Every management call and every door that needs a key takes an `Authorization: Nostr ` header. The event is: diff --git a/package.json b/package.json index 11366a5..beffd46 100644 --- a/package.json +++ b/package.json @@ -20,7 +20,8 @@ "dev:stage": "node scripts/dev/stage.mjs", "dev:junk": "node scripts/dev/junk.mjs", "dev:shot": "node scripts/dev/shot.mjs", - "dev:zaptest": "node scripts/dev/zaptest.mjs" + "dev:zaptest": "node scripts/dev/zaptest.mjs", + "test:network": "node scripts/ops/network.mjs" }, "dependencies": { "@noble/hashes": "^2.4.0", diff --git a/scripts/ops/network-client.mjs b/scripts/ops/network-client.mjs new file mode 100644 index 0000000..f8d0a93 --- /dev/null +++ b/scripts/ops/network-client.mjs @@ -0,0 +1,167 @@ +// The manually invoked network test client: signed management calls and a +// bounded, read-only WebSocket sample. It keeps retries narrow because a +// mutation whose result is unclear must remain visible to the harness. +import WebSocket from "ws"; +import { finalizeEvent, verifyEvent } from "nostr-tools/pure"; +import { getToken } from "nostr-tools/nip98"; + +const REQUEST_TIMEOUT = 20_000; +const SAMPLE_TIMEOUT = 15_000; +const SAMPLE_BYTES = 256 * 1024; +const RETRIES = 30; +const RETRY_DELAY = 500; +const RETRY_MAX_WAIT = 2_000; +const RETRY_WINDOW = 60_000; + +export const clientMetrics = { requests: 0, operationRetries: 0, operationWaitMs: 0 }; + +const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); + +export class NetworkError extends Error { + constructor(message, status, body) { + super(message); + this.name = "NetworkError"; + this.status = status; + this.body = body; + } +} + +const parseBody = async (response) => { + const text = await response.text(); + if (!text) return null; + try { + return JSON.parse(text); + } catch { + return text; + } +}; + +const retryable = (status, body) => { + if (status !== 429) return false; + const reason = body && typeof body === "object" + ? `${body.error ?? ""} ${body.message ?? ""}` + : String(body ?? ""); + return /operation in progress/i.test(reason); +}; + +const errorText = (body) => { + if (body && typeof body === "object") return String(body.error ?? body.message ?? "request failed").slice(0, 300); + return String(body ?? "request failed").slice(0, 300); +}; + +// request sends one signed JSON POST and returns its parsed response. Only a +// relay's explicit transient operation refusal is retried; timeout, network +// and other HTTP failures remain errors because their mutation outcome is +// unknown. +export async function request(node, sk, body, rpc = false) { + const url = typeof node === "string" ? node : node.url; + const payload = body ?? {}; + const retryUntil = Date.now() + RETRY_WINDOW; + for (let attempt = 0; attempt <= RETRIES; attempt++) { + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT); + try { + clientMetrics.requests++; + const authorization = await getToken(url, "POST", (event) => finalizeEvent(event, sk), true, payload); + const response = await fetch(url, { + method: "POST", + redirect: "error", + signal: controller.signal, + headers: { "content-type": rpc ? "application/nostr+json+rpc" : "application/json", authorization }, + body: JSON.stringify(payload), + }); + const result = await parseBody(response); + if (response.ok) return result; + if (retryable(response.status, result) && attempt < RETRIES && Date.now() < retryUntil) { + const delay = Math.min(RETRY_MAX_WAIT, RETRY_DELAY * (attempt + 1), retryUntil - Date.now()); + if (delay <= 0) throw new NetworkError(errorText(result), response.status, result); + clientMetrics.operationRetries++; + clientMetrics.operationWaitMs += delay; + await sleep(delay); + continue; + } + throw new NetworkError(errorText(result), response.status, result); + } catch (error) { + if (error instanceof NetworkError) throw error; + throw new NetworkError(error instanceof Error ? error.message : String(error), 0, null); + } finally { + clearTimeout(timer); + } + } + throw new NetworkError("request retries exhausted", 429, null); +} + +// rpc calls one NIP-86 method and returns its result. A relay error remains a +// NetworkError so the harness can stop before issuing the next mutation. +export async function rpc(node, sk, method, ...params) { + const response = await request(node, sk, { method, params }, true); + if (response && typeof response === "object" && "error" in response) { + throw new NetworkError(errorText(response), 200, response); + } + return response && typeof response === "object" && "result" in response ? response.result : response; +} + +const validSample = (event, filter) => { + if (!event || typeof event !== "object" || !verifyEvent(event)) return false; + if (filter.kinds?.length && !filter.kinds.includes(event.kind)) return false; + if (filter.since !== undefined && event.created_at < filter.since) return false; + if (filter.until !== undefined && event.created_at > filter.until) return false; + if (filter.authors?.length && !filter.authors.includes(event.pubkey)) return false; + return true; +}; + +// sample reads at most limit valid signed events from a relay. It never +// answers AUTH or sends EVENT. An authenticated source is allowed to complete +// its read or refuse it. Incoming data and time are bounded for endpoints. +export function sample(url, filter = {}, limit = 3) { + const requested = { ...filter, kinds: filter.kinds?.length ? [...filter.kinds] : [1], limit: Math.min(Math.max(limit, 1), 3) }; + const wsURL = url.replace(/^https:/, "wss:").replace(/^http:/, "ws:"); + return new Promise((resolve) => { + const events = []; + let settled = false; + let timer; + let socket; + const finish = (status, error = "") => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (socket) { + socket.removeAllListeners("open"); + socket.removeAllListeners("message"); + try { + if (socket.readyState === WebSocket.OPEN || socket.readyState === WebSocket.CLOSING) socket.close(); + else socket.terminate(); + } catch { + try { socket.terminate(); } catch { /* already closed */ } + } + } + resolve({ events, status, error }); + }; + try { + socket = new WebSocket(wsURL, { maxPayload: SAMPLE_BYTES, followRedirects: false }); + socket.on("open", () => socket.send(JSON.stringify(["REQ", "network-sample", requested]))); + socket.on("message", (data) => { + if (settled) return; + const raw = Buffer.isBuffer(data) ? data : Buffer.from(String(data)); + if (raw.length > SAMPLE_BYTES) return finish("failed", "sample response exceeded 256 KiB"); + let message; + try { message = JSON.parse(raw.toString("utf8")); } catch { return; } + if (!Array.isArray(message)) return; + if (message[0] === "AUTH") return; + if (message[0] === "CLOSED") return finish("refused", String(message[2] ?? "source refused query").slice(0, 300)); + if (message[0] === "EOSE" && message[1] === "network-sample") return finish("complete"); + if (message[0] !== "EVENT" || message[1] !== "network-sample") return; + let valid = false; + try { valid = validSample(message[2], requested); } catch { return; } + if (!valid) return; + if (!events.some((event) => event.id === message[2].id)) events.push(message[2]); + if (events.length >= requested.limit) finish("limit"); + }); + socket.on("error", (error) => finish("failed", error.message.slice(0, 300))); + socket.on("close", () => finish(events.length ? "partial" : "failed", events.length ? "source closed before EOSE" : "connection closed")); + timer = setTimeout(() => finish("timeout", "source did not answer within 15 seconds"), SAMPLE_TIMEOUT); + } catch (error) { + finish("failed", error instanceof Error ? error.message.slice(0, 300) : String(error).slice(0, 300)); + } + }); +} diff --git a/scripts/ops/network.mjs b/scripts/ops/network.mjs new file mode 100644 index 0000000..077b1c5 --- /dev/null +++ b/scripts/ops/network.mjs @@ -0,0 +1,335 @@ +// The finite relay-network exercise, invoked with npm run test:network. +// A private manifest precedes every claim so cleanup also covers lost replies. +// Only run-created relays receive writes; external sources only receive reads. +import { mkdir, readFile, writeFile, rename } from "node:fs/promises"; +import { homedir } from "node:os"; +import { join, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { randomBytes } from "node:crypto"; +import { finalizeEvent, generateSecretKey, getPublicKey } from "nostr-tools/pure"; +import { encrypt } from "nostr-tools/nip04"; +import { rpc, request, sample, clientMetrics } from "./network-client.mjs"; + +export const ROLES = ["peer-a", "peer-b", "satellite", "personal", "hub"]; +export const EXTERNAL = ["wss://relay.damus.io", "wss://nos.lol"]; +const now = () => Math.floor(Date.now() / 1000); +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +const secret = (actor) => Uint8Array.from(Buffer.from(actor.secret, "hex")); +const key = (m, node) => secret(m.actors[node.owner]); +const log = (s) => console.log(new Date().toISOString() + " " + s); + +export function manifest(domain = "bind.ws") { + if (!/^[a-z0-9]+(?:[.-][a-z0-9]+)*\.[a-z]{2,}$/.test(domain)) throw new Error("invalid: domain must be a DNS hostname"); + const id = randomBytes(6).toString("hex"); + const actors = Object.fromEntries(["alice", "bob", "carole", "agent", "larry"].map((name) => { + const sk = generateSecretKey(); + return [name, { secret: Buffer.from(sk).toString("hex"), pubkey: getPublicKey(sk) }]; + })); + return { version: 1, id, domain, createdAt: new Date().toISOString(), actors, + nodes: ROLES.map((role, i) => ({ role, owner: ["alice", "bob", "carole", "alice", "carole"][i], + slug: `net-${id}-${role}`, url: `https://net-${id}-${role}.${domain}/`, state: "planned" })) }; +} + +// validateManifest prevents recovery from following arbitrary addresses in a file. +export function validateManifest(m) { + if (m.version !== 1 || !/^[a-f0-9]{12}$/.test(m.id) || !/^[a-z0-9]+(?:[.-][a-z0-9]+)*\.[a-z]{2,}$/.test(m.domain)) throw new Error("invalid: network manifest identity"); + if (!Array.isArray(m.nodes) || m.nodes.length !== ROLES.length) throw new Error("invalid: network manifest nodes"); + for (const [i, node] of m.nodes.entries()) { + const slug = `net-${m.id}-${ROLES[i]}`; + if (node.role !== ROLES[i] || node.slug !== slug || node.url !== `https://${slug}.${m.domain}/`) throw new Error("invalid: network manifest target"); + const actor = m.actors[node.owner]; + if (!actor || !/^[a-f0-9]{64}$/.test(actor.secret) || getPublicKey(secret(actor)) !== actor.pubkey) throw new Error("invalid: network manifest owner"); + if (!["planned", "claiming", "claimed", "deleted"].includes(node.state)) throw new Error("invalid: network manifest state"); + } + return m; +} + +export async function save(path, value) { + await writeFile(path + ".tmp", JSON.stringify(value, null, 2) + "\n", { mode: 0o600 }); + await rename(path + ".tmp", path); +} + +// cleanup verifies ownership through an authenticated uncached call before deletion. +// An unclaimed response also resolves a claim or delete whose reply was lost. +export async function cleanup(m, persist, call = rpc) { + validateManifest(m); + const results = []; + for (const node of [...m.nodes].reverse()) { + if (node.state === "planned" || node.state === "deleted") continue; + try { + let p; + let unclaimed = false; + try { p = await call(node, key(m, node), "getpolicy"); } + catch (e) { + if (e.status !== 403 || e.message !== "restricted: this relay is unclaimed") throw e; + unclaimed = true; + } + if (!unclaimed) { + if (!p || p.owner !== m.actors[node.owner].pubkey) throw new Error("restricted: cleanup owner mismatch"); + const r = await call(node, key(m, node), "deleterelay", node.slug); + if (r?.deleted !== true || r.name !== node.slug) throw new Error("error: deletion was not acknowledged"); + } + node.state = "deleted"; + delete node.cleanupError; + results.push({ role: node.role, deleted: true }); + } catch (e) { + node.cleanupError = e.message; + results.push({ role: node.role, deleted: false, error: e.message }); + } + await persist(); + } + return results; +} + +export function usageDelta(before, after) { + const delta = {}; + for (const k of Object.keys(after ?? {})) { + if (typeof after[k] === "number" && typeof before?.[k] === "number") delta[k] = after[k] - before[k]; + else if (after[k] && typeof after[k] === "object" && !Array.isArray(after[k])) delta[k] = usageDelta(before?.[k], after[k]); + } + return delta; +} + +async function run(dir, domain) { + await mkdir(dir, { recursive: false, mode: 0o700 }); + const m = manifest(domain); + const manifestPath = join(dir, "manifest.json"); + const reportPath = join(dir, "report.json"); + const persist = () => save(manifestPath, m); + const report = { version: 1, id: m.id, startedAt: m.createdAt, domain, nodes: [], checks: [], jobs: [], external: [], fixtures: [], actors: Object.fromEntries(Object.entries(m.actors).map(([name, actor]) => [name, actor.pubkey])), + limits: { relays: 5, externalSources: 2, externalAuthorsPerSource: 3, jobTimeoutSeconds: 120, runTimeoutSeconds: 1200 }, + assumptions: { size: "five small relays; external sources treated as much larger, their size is not measured", activity: "short bursts, idle gaps, catch-up and repeat sync; no load or capacity claim", privacy: "public tasks and results, encrypted kind-4 message held only at personal relay" } }; + const checkpoint = async () => { await persist(); await save(reportPath, report); }; + await checkpoint(); + log(`Manifest: ${manifestPath}`); + let interrupted = false; + const stop = () => { interrupted = true; }; + process.on("SIGINT", stop); + process.on("SIGTERM", stop); + const deadline = Date.now() + 1200000; + const active = () => { if (interrupted || Date.now() > deadline) throw new Error("error: network exercise interrupted or timed out"); }; + const node = (role) => m.nodes.find((n) => n.role === role); + const call = (n, method, ...args) => rpc(n, key(m, n), method, ...args); + const ws = (n) => n.url.replace("https:", "wss:").replace(/\/$/, ""); + const authors = [m.actors.alice.pubkey, m.actors.bob.pubkey]; + const event = (actor, text, kind = 1, tags = []) => { + const e = finalizeEvent({ kind, content: text, created_at: now(), tags: [["t", `bindws-network-${m.id}`], ["expiration", String(now() + 3600)], ...tags] }, secret(m.actors[actor])); + report.fixtures.push(e); + return e; + }; + const publish = async (n, e, actor) => { + active(); + const r = await request(n.url + "events", secret(m.actors[actor]), e); + if (!r.accepted) throw new Error(r.message ?? "error: event refused"); + return e; + }; + const query = (n, ids, actor = n.owner) => request(n.url + "query", secret(m.actors[actor]), [{ ids, limit: 100 }]); + const check = async (name, fn) => { + active(); + log(name); + const start = Date.now(); + try { const detail = await fn(); report.checks.push({ name, passed: true, milliseconds: Date.now() - start, detail }); } + catch (e) { report.checks.push({ name, passed: false, milliseconds: Date.now() - start, error: e.message }); log(`FAILED: ${e.message}`); } + await checkpoint(); + }; + const expectIds = async (n, ids, actor) => { + const found = (await query(n, ids, actor)).map((e) => e.id); + if (found.length !== ids.length || ids.some((id) => !found.includes(id))) throw new Error(`error: ${n.role} has ${found.length}/${ids.length} expected events`); + return { expected: ids, found }; + }; + const absent = async (n, ids, actor) => { + const found = await query(n, ids, actor); + if (found.length) throw new Error(`error: ${n.role} exposed ${found.length} excluded events`); + return { excluded: ids }; + }; + const job = async (n, kind, urls, filter, label, strict = true) => { + active(); + const start = Date.now(); + const j = await call(n, "addjob", { kind, relays: urls, filter, every: 0 }); + const entry = { role: n.role, label, id: j.id, kind, relays: urls, filter }; + report.jobs.push(entry); + await checkpoint(); + try { + while (Date.now() - start < 120000) { + active(); + const state = (await call(n, "listjobs")).find((x) => x.id === j.id); + if (!state) throw new Error("error: job disappeared"); + if (!state.running && state.nextRun === 0 && state.last) { + entry.result = state.last; + entry.targetStatus = state.targetStatus; + entry.milliseconds = Date.now() - start; + if (strict && (state.last.error || state.last.refused || state.last.sources?.some((s) => s.status === "failed" || s.status === "refused" || s.partial))) throw new Error(`error: ${label}: ${state.last.error || "partial or refused sync"}`); + return entry; + } + await sleep(1500); + } + throw new Error(`error: ${label} exceeded 120 seconds`); + } catch (e) { entry.error = e.message; throw e; } + finally { await call(n, "removejob", j.id); await checkpoint(); } + }; + try { + for (const n of m.nodes) { + active(); + n.state = "claiming"; + await persist(); + const r = await call(n, "claim"); + if (!r.claimed || r.owner !== m.actors[n.owner].pubkey) throw new Error("restricted: generated relay was not claimed by this run"); + n.state = "claimed"; + await persist(); + // Quiet disables unsolicited delivery, notifications and unrelated features. + await call(n, "applypreset", "quiet"); + await call(n, "setpolicy", { writes: "allowlist", reads: "open", directoryPublic: false, guestReplies: false, openKinds: [], features: { sync: true, count: true } }); + for (const actor of ({ "peer-a": ["bob"], "peer-b": ["alice"], satellite: [], personal: ["agent"], hub: ["alice", "agent"] })[n.role]) if (actor !== n.owner) await call(n, "setmember", m.actors[actor].pubkey, { name: actor }); + const baseline = await call(n, "stats"); + report.nodes.push({ role: n.role, url: n.url, owner: m.actors[n.owner].pubkey, baseline, policy: await call(n, "getpolicy") }); + log(`Created ${n.role}: ${n.url}`); + await checkpoint(); + } + const a = node("peer-a"), b = node("peer-b"), personal = node("personal"), hub = node("hub"), satellite = node("satellite"); + const filter = { authors, kinds: [1], since: now() - 60 }; + const peerEvents = []; + await check("equal peers converge in both directions", async () => { + for (let i = 0; i < 4; i++) { + peerEvents.push(await publish(a, event("alice", `peer a note ${i}`), "alice")); + peerEvents.push(await publish(b, event("bob", `peer b note ${i}`), "bob")); + } + await job(a, "pull", [ws(b)], filter, "b to a"); + await job(b, "pull", [ws(a)], filter, "a to b"); + const ids = peerEvents.map((e) => e.id); + return [await expectIds(a, ids), await expectIds(b, ids)]; + }); + await check("repeat sync has no duplicate events", async () => { + await job(a, "pull", [ws(b)], filter, "repeat b to a"); + const ids = peerEvents.map((e) => e.id); + await publish(a, peerEvents[0], "alice"); + return expectIds(a, ids); + }); + await check("an idle peer catches up with the next burst", async () => { + await sleep(2500); + const events = []; + for (let i = 0; i < 3; i++) events.push(await publish(a, event("alice", `catch-up ${i}`), "alice")); + await absent(b, events.map((e) => e.id)); + await job(b, "pull", [ws(a)], filter, "catch-up a to b"); + return expectIds(b, [...peerEvents, ...events].map((e) => e.id)); + }); + await check("the personal relay exchanges selected notes with the peer network", async () => { + const note = await publish(personal, event("alice", "public personal note for the small peer network"), "alice"); + await job(personal, "push", [ws(a)], { authors: [m.actors.alice.pubkey], kinds: [1] }, "personal to peer a"); + await job(b, "pull", [ws(a)], filter, "personal note through peer a to b"); + await expectIds(b, [note.id]); + await job(personal, "pull", [ws(b)], { authors: [m.actors.bob.pubkey], kinds: [1] }, "bob from peer network to personal"); + return expectIds(personal, peerEvents.filter((e) => e.pubkey === m.actors.bob.pubkey).map((e) => e.id)); + }); + let privateEvent; + await check("personal tasks reach the hub and agent results return", async () => { + const task = await publish(personal, event("alice", "public task: count the words in this synthetic note"), "alice"); + privateEvent = await publish(personal, event("alice", await encrypt(secret(m.actors.alice), m.actors.bob.pubkey, "synthetic private note"), 4, [["p", m.actors.bob.pubkey]]), "alice"); + await job(personal, "push", [ws(hub)], { authors: [m.actors.alice.pubkey], kinds: [1] }, "personal tasks to hub"); + await expectIds(hub, [task.id], "agent"); + const result = await publish(hub, event("agent", "public result: synthetic task complete", 1, [["e", task.id], ["p", m.actors.alice.pubkey]]), "agent"); + await job(personal, "pull", [ws(hub)], { authors: [m.actors.agent.pubkey], kinds: [1] }, "agent results to personal"); + return { task: task.id, result: result.id, received: await expectIds(personal, [result.id]), privateExcluded: await absent(hub, [privateEvent.id], "bob") }; + }); + await check("private messages stay visible only to their parties", async () => { + if (!privateEvent) throw new Error("error: private fixture was not published"); + return { recipient: await expectIds(personal, [privateEvent.id], "bob"), stranger: await absent(personal, [privateEvent.id], "larry") }; + }); + await check("non-member writes and management are refused", async () => { + for (const n of [a, personal, hub]) { + try { await publish(n, event("larry", "unauthorized synthetic write"), "larry"); throw new Error("error: unauthorized write accepted"); } + catch (e) { if (e.status !== 400 || !/restricted:|blocked:|auth-required:/.test(e.message)) throw e; } + try { await rpc(n, secret(m.actors.larry), "setpolicy", { writes: "open" }); throw new Error("error: unauthorized management accepted"); } + catch (e) { if (e.status !== 403 || !e.message.startsWith("restricted:")) throw e; } + } + }); + await check("author and kind filters exclude unrelated hub data", async () => { + const unrelated = await publish(hub, event("carole", "hub local note"), "carole"); + const metadata = await publish(hub, event("agent", JSON.stringify({ name: "synthetic agent" }), 0), "agent"); + await job(personal, "pull", [ws(hub)], { authors: [m.actors.agent.pubkey], kinds: [1] }, "filtered hub repeat"); + return absent(personal, [unrelated.id, metadata.id]); + }); + await check("members-only sources report a refused unauthenticated pull", async () => { + await call(hub, "setpolicy", { reads: "members" }); + try { + const r = await job(personal, "pull", [ws(hub)], { authors: [m.actors.agent.pubkey], kinds: [1] }, "restricted source", false); + if (!r.result.sources?.some((s) => s.status === "refused")) throw new Error("error: restricted source was not classified as refused"); + return r.result; + } finally { await call(hub, "setpolicy", { reads: "open" }); } + }); + for (const source of EXTERNAL) { + active(); + log(`Read-only external sample: ${source}`); + const selection = await sample(source, { kinds: [0], since: now() - 86400, limit: 3 }, 3); + const entry = { source, sampling: { status: selection.status, error: selection.error }, sampledIds: selection.events.map((e) => e.id), authors: [...new Set(selection.events.map((e) => e.pubkey))] }; + report.external.push(entry); + if (!entry.authors.length) { entry.outcome = "inconclusive: no public sample"; await checkpoint(); continue; } + try { + const j = await job(satellite, "pull", [source], { authors: entry.authors, kinds: [0], since: now() - 86400 }, `external ${source}`, false); + const received = await request(satellite.url + "query", key(m, satellite), [{ authors: entry.authors, kinds: [0], limit: 10 }]); + entry.receivedIds = received.map((e) => e.id); + entry.job = j.result; + entry.outcome = received.length ? "observed public profile import" : "inconclusive: no matching profiles imported"; + } catch (e) { entry.outcome = "inconclusive: " + e.message; } + if (entry.receivedIds?.length) { + await check("selected satellite profiles reach the agent hub", async () => { + const context = await job(hub, "pull", [ws(satellite)], { authors: entry.authors, kinds: [0] }, "satellite profile context to hub"); + entry.hubContext = { job: context.result, received: await expectIds(hub, entry.receivedIds) }; + return entry.hubContext; + }); + } + await checkpoint(); + } + await check("the external satellite stays separate from synthetic peer traffic", async () => absent(satellite, peerEvents.map((e) => e.id))); + } catch (e) { + report.error = e.message; + log(e.message); + } finally { + log("Extracting final usage and tearing down run-created relays"); + for (const n of m.nodes.filter((n) => n.state === "claimed")) { + const entry = report.nodes.find((x) => x.role === n.role); + try { + const stats = await call(n, "stats"); + if (entry) { entry.storage = await call(n, "storagestats"); entry.final = stats; entry.usageDelta = usageDelta(entry.baseline.fuel, stats.fuel); } + } catch (e) { if (entry) entry.extractionError = e.message; } + } + report.cleanup = await cleanup(m, persist); + report.client = { ...clientMetrics }; + report.finishedAt = new Date().toISOString(); + report.passed = !report.error && report.checks.length > 0 && report.checks.every((c) => c.passed) && report.cleanup.every((c) => c.deleted) && !report.nodes.some((n) => n.extractionError); + await checkpoint(); + process.off("SIGINT", stop); + process.off("SIGTERM", stop); + } + log(`${report.passed ? "PASS" : "FAIL"}: ${report.checks.filter((c) => c.passed).length}/${report.checks.length} checks; report ${reportPath}`); + if (!report.passed) process.exitCode = 1; +} + +export async function main(args) { + const [command, target, domain, ...extra] = args; + if (extra.length || !["run", "cleanup", "plan"].includes(command)) { + console.log("npm run test:network -- plan\nnpm run test:network -- run [new-output-directory] [domain]\nnpm run test:network -- cleanup "); + if (command && command !== "--help") process.exitCode = 1; + return; + } + if (command === "plan") { + console.log(JSON.stringify({ roles: ROLES, externalReadOnly: EXTERNAL, run: "finite one-shot jobs, no schedule", cleanup: "finally and recoverable manifest", assertions: "peer convergence, dedup, catch-up, task/result filters, privacy and access refusal" }, null, 2)); + return; + } + if (command === "cleanup") { + if (!target || domain) throw new Error("invalid: cleanup needs one output directory"); + const path = join(resolve(target), "manifest.json"); + const m = validateManifest(JSON.parse(await readFile(path, "utf8"))); + const results = await cleanup(m, () => save(path, m)); + await save(join(resolve(target), "cleanup.json"), results); + console.log(JSON.stringify(results, null, 2)); + if (results.some((r) => !r.deleted)) process.exitCode = 1; + return; + } + const root = join(homedir(), ".local", "share", "bindws", "network-tests"); + if (!target) await mkdir(root, { recursive: true, mode: 0o700 }); + await run(target ? resolve(target) : join(root, new Date().toISOString().replace(/[:.]/g, "-") + "-" + randomBytes(3).toString("hex")), domain ?? "bind.ws"); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + main(process.argv.slice(2)).catch((e) => { console.error(e.message); process.exitCode = 1; }); +} diff --git a/test/unit/network-client.test.ts b/test/unit/network-client.test.ts new file mode 100644 index 0000000..8295e35 --- /dev/null +++ b/test/unit/network-client.test.ts @@ -0,0 +1,75 @@ +import { createServer } from "node:http"; +import { describe, expect, it } from "vitest"; +import { WebSocketServer } from "ws"; +import { finalizeEvent, generateSecretKey } from "nostr-tools/pure"; +// @ts-expect-error the production-only helper is JavaScript by design. +import { request, sample } from "../../scripts/ops/network-client.mjs"; + +const key = generateSecretKey(); + +describe("network test client", () => { + it("does not retry an ambiguous mutation response", async () => { + let calls = 0; + const server = createServer((_req, res) => { + calls++; + res.writeHead(503, { "content-type": "application/json" }); + res.end(JSON.stringify({ error: "error: upstream unavailable" })); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", () => resolve())); + const address = server.address(); + const url = `http://127.0.0.1:${typeof address === "object" && address ? address.port : 0}/`; + await expect(request({ url }, key, { mutation: true })).rejects.toMatchObject({ status: 503, message: "error: upstream unavailable" }); + expect(calls).toBe(1); + await new Promise((resolve, reject) => server.close((error) => error ? reject(error) : resolve())); + }); + + it("retries an explicit operation-in-progress refusal and returns the response", async () => { + let calls = 0; + const server = createServer((_req, res) => { + calls++; + res.writeHead(calls === 1 ? 429 : 200, { "content-type": "application/json" }); + res.end(JSON.stringify(calls === 1 ? { error: "restricted: relay operation in progress; retry" } : { result: { ok: true } })); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", () => resolve())); + const address = server.address(); + const url = `http://127.0.0.1:${typeof address === "object" && address ? address.port : 0}/`; + await expect(request({ url }, key, { method: "stats", params: [] }, true)).resolves.toEqual({ result: { ok: true } }); + expect(calls).toBe(2); + await new Promise((resolve, reject) => server.close((error) => error ? reject(error) : resolve())); + }); + + it("samples signed events after an unsolicited auth challenge without publishing", async () => { + const server = new WebSocketServer({ port: 0 }); + let clientMessages: string[] = []; + const event = finalizeEvent({ kind: 0, content: JSON.stringify({ name: "sample" }), tags: [], created_at: Math.floor(Date.now() / 1000) }, key); + server.on("connection", (socket) => { + socket.on("message", (data) => { + clientMessages.push(data.toString()); + socket.send(JSON.stringify(["AUTH", "challenge"])); + socket.send(JSON.stringify(["EVENT", "network-sample", { id: "bad" }])); + socket.send(JSON.stringify(["EVENT", "network-sample", event])); + socket.send(JSON.stringify(["EOSE", "network-sample"])); + }); + }); + await new Promise((resolve) => server.once("listening", () => resolve())); + const address = server.address(); + const result = await sample(`ws://127.0.0.1:${typeof address === "object" && address ? address.port : 0}`, { kinds: [0] }); + expect(result.status).toBe("complete"); + expect(result.events.map((x: { id: string }) => x.id)).toEqual([event.id]); + expect(clientMessages).toHaveLength(1); + expect(clientMessages[0]).not.toContain("AUTH"); + expect(clientMessages[0]).not.toContain("EVENT"); + server.close(); + }); + + it("reports a source that closes before completing its sample", async () => { + const server = new WebSocketServer({ port: 0 }); + server.on("connection", (socket) => socket.close()); + await new Promise((resolve) => server.once("listening", () => resolve())); + const address = server.address(); + const result = await sample(`ws://127.0.0.1:${typeof address === "object" && address ? address.port : 0}`); + expect(result.status).toBe("failed"); + expect(result.error).toContain("closed"); + server.close(); + }); +}); diff --git a/test/unit/network.test.ts b/test/unit/network.test.ts new file mode 100644 index 0000000..a6f9a02 --- /dev/null +++ b/test/unit/network.test.ts @@ -0,0 +1,85 @@ +import { describe, it, expect, vi } from "vitest"; +const { manifest, validateManifest, cleanup, usageDelta } = await import(new URL("../../scripts/ops/network.mjs", import.meta.url).href); + +describe("network cleanup", () => { + it("rejects a recovery manifest that points outside its run before making a request", async () => { + const m = manifest(); + m.nodes[0].state = "claimed"; + m.nodes[0].url = "https://someone-else.bind.ws/"; + const call = vi.fn(); + await expect(cleanup(m, vi.fn(), call)).rejects.toThrow("manifest target"); + expect(call).not.toHaveBeenCalled(); + }); + + it("recovers a lost claim reply and persists each deletion independently", async () => { + const m = manifest(); + m.nodes[0].state = "claiming"; + m.nodes[1].state = "claimed"; + const saved: string[][] = []; + const call = vi.fn(async (n: { owner: string; slug: string }, _key: unknown, method: string, slug?: string) => { + if (method === "getpolicy") return { owner: m.actors[n.owner].pubkey }; + expect(method).toBe("deleterelay"); + expect(slug).toBe(n.slug); + return { deleted: true, name: slug }; + }); + const results = await cleanup(m, async () => saved.push(m.nodes.map((n: { state: string }) => n.state)), call); + expect(results).toHaveLength(2); + expect(results.every((r: { deleted: boolean }) => r.deleted)).toBe(true); + expect(saved).toHaveLength(2); + expect(saved[0][0]).toBe("claiming"); + expect(saved[1][0]).toBe("deleted"); + expect(call).toHaveBeenCalledTimes(4); + }); + + it("continues after a failed deletion and never deletes a mismatched owner", async () => { + const m = manifest(); + m.nodes[0].state = "claimed"; + m.nodes[1].state = "claimed"; + const call = vi.fn(async (n: { role: string; owner: string; slug: string }, _key: unknown, method: string) => { + if (method === "getpolicy") return { owner: n.role === "peer-b" ? "ff".repeat(32) : m.actors[n.owner].pubkey }; + return { deleted: true, name: n.slug }; + }); + const r = await cleanup(m, vi.fn(), call); + expect(r[0].deleted).toBe(false); + expect(r[1].deleted).toBe(true); + expect(call.mock.calls.filter((c) => c[2] === "deleterelay")).toHaveLength(1); + expect(m.nodes[1].state).toBe("claimed"); + }); + + it("treats only the authenticated unclaimed response as an already completed deletion", async () => { + const m = manifest(); + m.nodes[0].state = "claiming"; + const call = vi.fn(async () => { throw Object.assign(new Error("restricted: this relay is unclaimed"), { status: 403 }); }); + expect((await cleanup(m, vi.fn(), call))[0].deleted).toBe(true); + expect(call).toHaveBeenCalledTimes(1); + m.nodes[0].state = "claiming"; + call.mockImplementation(async () => { throw Object.assign(new Error("restricted: not the relay owner"), { status: 403 }); }); + expect((await cleanup(m, vi.fn(), call))[0].deleted).toBe(false); + expect(m.nodes[0].state).toBe("claiming"); + }); + + it("keeps malformed successful ownership responses pending for recovery", async () => { + for (const policy of [null, undefined, false, {}, { owner: "" }]) { + const m = manifest(); + m.nodes[0].state = "claimed"; + const call = vi.fn(async () => policy); + const results = await cleanup(m, vi.fn(), call); + expect(results[0].deleted).toBe(false); + expect(m.nodes[0].state).toBe("claimed"); + expect(call).toHaveBeenCalledTimes(1); + } + }); + + it("rejects duplicate nodes and a secret that does not match the recorded owner", () => { + const m = manifest(); + m.nodes[1] = { ...m.nodes[0] }; + expect(() => validateManifest(m)).toThrow("manifest target"); + const second = manifest(); + second.actors.alice.pubkey = second.actors.bob.pubkey; + expect(() => validateManifest(second)).toThrow("manifest owner"); + }); +}); + +it("reports counter deltas without confusing flags or absent samples with zero usage", () => { + expect(usageDelta({ activeMs: 2, outOfFuel: false }, { activeMs: 9, outOfFuel: true, rowsRead: 10 })).toEqual({ activeMs: 7 }); +});