From ec9bc93296a8364ceddf1ac20012722c906cb184 Mon Sep 17 00:00:00 2001 From: Lann Martin Date: Sun, 23 Aug 2026 15:08:59 -0400 Subject: [PATCH] Every relay dial carries a deadline; a stalling relay cannot pin the endpoint A relay (or middlebox) that accepted TCP and then stalled the handshake pinned all three dial sites forever: bind hung its caller, ensure-relay held the relay-opening claim so the URL stayed permanently "opening", and the home-relay redial slot pinned until process restart (#93). The dials shipped unbounded because dropping an in-flight import future in-task wedged the polyengine host (polymorph-components/polyengine#239); that is fixed as of polyengine 0.5.1 (A23: a guest's cancel of a host import is a prompt discard), so the timeout returns as the plain select it always wanted to be. - DIAL_TIMEOUT (10s) matches upstream's relay connect timeout (iroh-1.0.3 src/socket/transports/relay/actor.rs CONNECT_TIMEOUT) and sits under ensure-relay's 30s claim-wait. dial_relay races RelayConn::connect against the clock and drops the loser; all three sites use it. The deadline surfaces as the WIT `timed-out` variant; other dial failures stay `connect-failed`. - The "do not add a timeout here" rulings on pump/redial_home (and the module doc's never-dropped-mid-flight claim) are repealed; the teardown resolve-before-return discipline stays as hygiene. - polyengine floor 0.5.1 in host-polyengine (caret floor + both locks + the README pin note): under 0.5.0 the first timed-out dial would wedge the consumer's store, so the floor is correctness, not housekeeping. - Exam scenario 8 is the gate: a stub that accepts and never writes; bind and two foreign-relay connects must fail timed-out inside the deadline window (the second connect proves the claim released), and the home-relay redial must keep redialing through the stall and recover once a real relay returns, with no guest trap across the dropped dials. Known residue: on the polyengine host a timed-out dial's host-side connect keeps running until it settles on its own (A23 discard is delivery-only). The fix is marking polymorph:websocket's connect abortable() (polyengine A24), a sibling PR. Falsified against the unfixed endpoint: scenario 8 fails on both attempts with `timed out after 30000 ms: leg A bind`. Gates: just check, just build, just exam-polyengine (9/9, leg C exercised with an owned relay), just matrix (13/13). Fixes #93 --- endpoint/src/endpoint_impl.rs | 97 ++++-- experiments/iroh-relay-ws/host/deno.json | 12 +- experiments/iroh-relay-ws/host/deno.lock | 43 +-- host-polyengine/README.md | 10 +- host-polyengine/deno.json | 6 +- host-polyengine/deno.lock | 41 +-- host-polyengine/src/run-endpoint.ts | 394 ++++++++++++++++++++++- 7 files changed, 520 insertions(+), 83 deletions(-) diff --git a/endpoint/src/endpoint_impl.rs b/endpoint/src/endpoint_impl.rs index 796233b..bc5d749 100644 --- a/endpoint/src/endpoint_impl.rs +++ b/endpoint/src/endpoint_impl.rs @@ -17,10 +17,11 @@ //! timeout, signaling deadline) is observed, and how stale a missed //! wake edge can go. //! -//! An in-flight import is a component-model subtask and is always -//! awaited to completion, never dropped mid-flight (the teardown -//! discipline; the kick future is guest-local, so re-creating it each -//! select turn cancels nothing). All of it runs on the component-model +//! An in-flight import is a component-model subtask; dropping one is a +//! `subtask.cancel`, which the minimum hosts (polyengine 0.5.1 / A23, +//! wasmtime) settle promptly, so a select race may drop its loser (the +//! kick future is guest-local, so re-creating it each select turn +//! cancels nothing). All of it runs on the component-model //! async ABI's single cooperative thread: the `RefCell` borrows never //! cross an await, and a fired waker resumes its task through the //! scheduler, never synchronously. @@ -235,6 +236,15 @@ const REDIAL_MAX_DELAY: Duration = Duration::from_secs(5); /// restarts the backoff, a younger one consumes the next delay. const REDIAL_ESTABLISHED: Duration = Duration::from_secs(10); +/// Deadline on every relay dial (`bind`, `ensure-relay`, the home +/// redial): a relay that accepts the socket and then stalls the +/// handshake fails the dial instead of pinning it (issue #93). The +/// value matches upstream's relay connect timeout (iroh-1.0.3 +/// src/socket/transports/relay/actor.rs, CONNECT_TIMEOUT) and sits +/// well under `ensure_relay`'s 30s claim-wait, so a stalled dial +/// resolves its waiters before they give up. +const DIAL_TIMEOUT: Duration = Duration::from_secs(10); + struct ChannelEntry { wire: Rc, /// The relay-authenticated peer the channel was signaled with; @@ -783,19 +793,20 @@ fn on_event( /// The endpoint's I/O task: relayed datagrams in and out, noq's timers, /// the flush after every kick, and the home relay's redial. The two /// long-lived import futures stay pinned across iterations and are -/// resolved before the task returns (an in-flight import is a -/// component-model subtask; jco traps on cancelling one — the teardown -/// discipline). Channel receives live in a persistent set with the same -/// discipline: closing every channel resolves them before the task -/// returns. The redial slot is the exception: an in-flight relay dial is -/// dropped at teardown rather than awaited. +/// resolved before the task returns; channel receives live in a +/// persistent set with the same discipline: closing every channel +/// resolves them before the task returns. The redial slot is the +/// exception: an in-flight relay dial is dropped at teardown rather +/// than awaited. /// -/// The two kinds of drop are not the same hazard. Dropping an in-flight -/// import to settle a select race INSIDE a live task wedges the host, -/// which sees an activation that never parks, finishes, or traps (see -/// [`redial_home`]). Dropping one as the whole task ends is the -/// host-driven cancellation `ensure_relay` already takes when its call -/// is cancelled, and is sound. +/// Dropping an in-flight import future is a component-model +/// `subtask.cancel`, and is safe both mid-task (settling a select +/// race, as [`dial_relay`]'s deadline does) and at task end. The +/// minimum host is part of that claim: polyengine settles the cancel +/// as a prompt discard only from 0.5.1 (A23; earlier versions wedge +/// the store, polyengine#239). The resolve-before-return discipline +/// above is retained as teardown hygiene, not a correctness +/// requirement. /// /// A failed home relay retires its connection from the pool and arms a /// redial; the pump ends only on close, never on a wire failure. @@ -1172,17 +1183,36 @@ fn next_redial_delay(attempt: &mut u32, home_since: &mut Option) -> Dur type RedialFuture = futures::future::LocalBoxFuture<'static, Result>; +/// A failed [`dial_relay`]: the deadline, or the dial's own failure. +enum DialError { + /// The dial did not resolve within [`DIAL_TIMEOUT`]. + TimedOut, + /// The dial resolved with a failure. + Failed(String), +} + +/// One relay dial bounded by [`DIAL_TIMEOUT`]: `RelayConn::connect` +/// raced against the clock, the loser's in-flight future dropped. +/// The drop is a component-model `subtask.cancel`; the minimum hosts +/// (polyengine 0.5.1 / A23, wasmtime) settle it promptly — a discard +/// or a real cancellation. On a discard the host-side connect may +/// still run to its natural end; only delivery is renounced. +async fn dial_relay(url: &str, identity: &Identity) -> Result { + let mut dial = pin!(RelayConn::connect(url, identity).fuse()); + let mut timer = pin!(monotonic_clock::wait_for(DIAL_TIMEOUT.as_nanos() as u64).fuse()); + select_biased! { + opened = dial => opened.map_err(DialError::Failed), + _ = timer => Err(DialError::TimedOut), + } +} + /// One home-relay redial: wait out the backoff, then connect and /// authenticate. /// -/// The dial carries no timeout of its own, and must not: a timeout is a -/// second future racing the dial, so every attempt drops whichever of -/// the two loses — the in-task select-race drop [`pump`]'s doc rules -/// out. Dropping the timer that way wedges the polyengine host, which -/// reports it as a resumed activation whose claim is never released. A -/// relay that accepts the socket and then stalls the handshake -/// therefore holds this slot until the socket resolves; nothing else in -/// the endpoint waits on it. +/// The dial is bounded by [`DIAL_TIMEOUT`] (via [`dial_relay`]): a +/// relay that accepts the socket and then stalls the handshake fails +/// the attempt, and the pump's error arm rearms the next redial with +/// backoff (issue #93). async fn redial_home( url: String, identity: Rc, @@ -1191,7 +1221,10 @@ async fn redial_home( if !delay.is_zero() { monotonic_clock::wait_for(delay.as_nanos() as u64).await; } - RelayConn::connect(&url, &identity).await + dial_relay(&url, &identity).await.map_err(|e| match e { + DialError::TimedOut => "relay dial timed out".to_string(), + DialError::Failed(e) => e, + }) } type ChannelRecvFuture = @@ -1545,7 +1578,7 @@ impl EndpointRes { normalized: normalized.clone(), armed: true, }; - let opened = RelayConn::connect(url, &self.identity).await; + let opened = dial_relay(url, &self.identity).await; claim.armed = false; let mut st = self.shared.borrow_mut(); st.relay_opening.remove(&normalized); @@ -1553,7 +1586,10 @@ impl EndpointRes { st.wake_waiters(); return match opened { Ok(conn) => Ok(st.register_relay(url, Rc::new(conn))), - Err(e) => Err(Error::ConnectFailed(format!("relay {url}: {e}"))), + Err(DialError::TimedOut) => { + Err(Error::TimedOut(format!("relay {url}: dial timed out"))) + } + Err(DialError::Failed(e)) => Err(Error::ConnectFailed(format!("relay {url}: {e}"))), }; } let started = Instant::now(); @@ -1645,9 +1681,12 @@ impl GuestEndpoint for EndpointRes { )); } - let relay = RelayConn::connect(&relay_url, &identity) + let relay = dial_relay(&relay_url, &identity) .await - .map_err(Error::ConnectFailed)?; + .map_err(|e| match e { + DialError::TimedOut => Error::TimedOut("relay dial timed out".into()), + DialError::Failed(e) => Error::ConnectFailed(e), + })?; let mut reset_key = [0u8; 32]; getrandom::fill(&mut reset_key).map_err(other)?; diff --git a/experiments/iroh-relay-ws/host/deno.json b/experiments/iroh-relay-ws/host/deno.json index 2fa9b44..6b3fe66 100644 --- a/experiments/iroh-relay-ws/host/deno.json +++ b/experiments/iroh-relay-ws/host/deno.json @@ -9,12 +9,12 @@ "lib": ["dom", "dom.iterable", "dom.asynciterable", "deno.ns"] }, "imports": { - "@polyengine/runtime/embedder": "jsr:@polyengine/runtime@0.5.0/embedder", - "@polyengine/runtime/shim": "jsr:@polyengine/runtime@0.5.0/shim", - "@polyengine/protocol": "jsr:@polyengine/protocol@0.2.2", - "@polyengine/translator": "jsr:@polyengine/translator@0.5.0", - "@polyengine/wasi": "jsr:@polyengine/wasi@0.5.0", - "@polyengine/wasi/io": "jsr:@polyengine/wasi@0.5.0/io", + "@polyengine/runtime/embedder": "jsr:@polyengine/runtime@0.5.1/embedder", + "@polyengine/runtime/shim": "jsr:@polyengine/runtime@0.5.1/shim", + "@polyengine/protocol": "jsr:@polyengine/protocol@0.2.3", + "@polyengine/translator": "jsr:@polyengine/translator@0.5.1", + "@polyengine/wasi": "jsr:@polyengine/wasi@0.5.1", + "@polyengine/wasi/io": "jsr:@polyengine/wasi@0.5.1/io", "@polymorph/webrtc-datachannels": "jsr:@polymorph/webrtc-datachannels@0.5.0", "@polymorph/websocket": "jsr:@polymorph/websocket@0.5.0" }, diff --git a/experiments/iroh-relay-ws/host/deno.lock b/experiments/iroh-relay-ws/host/deno.lock index 96cef24..8c1fff4 100644 --- a/experiments/iroh-relay-ws/host/deno.lock +++ b/experiments/iroh-relay-ws/host/deno.lock @@ -1,12 +1,13 @@ { "version": "5", "specifiers": { - "jsr:@polyengine/protocol@0.2.2": "0.2.2", - "jsr:@polyengine/protocol@~0.2.2": "0.2.2", - "jsr:@polyengine/runtime@0.5": "0.5.0", - "jsr:@polyengine/runtime@0.5.0": "0.5.0", - "jsr:@polyengine/translator@0.5.0": "0.5.0", - "jsr:@polyengine/wasi@0.5.0": "0.5.0", + "jsr:@polyengine/protocol@0.2.3": "0.2.3", + "jsr:@polyengine/protocol@~0.2.2": "0.2.3", + "jsr:@polyengine/protocol@~0.2.3": "0.2.3", + "jsr:@polyengine/runtime@0.5.1": "0.5.1", + "jsr:@polyengine/runtime@~0.5.1": "0.5.1", + "jsr:@polyengine/translator@0.5.1": "0.5.1", + "jsr:@polyengine/wasi@0.5.1": "0.5.1", "jsr:@polymorph/webrtc-datachannels@0.5.0": "0.5.0", "jsr:@polymorph/websocket@0.5.0": "0.5.0", "npm:node-datachannel@0.32.3": "0.32.3", @@ -14,25 +15,25 @@ "npm:werift@0.22.2": "0.22.2" }, "jsr": { - "@polyengine/protocol@0.2.2": { - "integrity": "ff470d7183924119ae181cc513093318f1587f7ad5c4bc02db7344e90c9ff52b" + "@polyengine/protocol@0.2.3": { + "integrity": "e3f0ac4165eb691d7231164ce28cf75e275c99496aa538e032297febdc52b371" }, - "@polyengine/runtime@0.5.0": { - "integrity": "36ea9b74dd38c0f5fb941695cfa3223190fda54bc18f7328600ece082dbbb794", + "@polyengine/runtime@0.5.1": { + "integrity": "6337c94d623bc2013fdeb324a5206860a916b8897b39cf4bec65b1bf87c287d0", "dependencies": [ - "jsr:@polyengine/protocol@~0.2.2" + "jsr:@polyengine/protocol@~0.2.3" ] }, - "@polyengine/translator@0.5.0": { - "integrity": "2f9770f468bd0c649732e6107d69715dea715c71564fe5b0ba0997b5d70e81c3", + "@polyengine/translator@0.5.1": { + "integrity": "16d7ac88997f89f10145cc109737b749bfebd37c9ea1aefe0a41b44c01b75586", "dependencies": [ - "jsr:@polyengine/runtime@0.5" + "jsr:@polyengine/runtime@~0.5.1" ] }, - "@polyengine/wasi@0.5.0": { - "integrity": "61275e66895084807393bf93876b74be9cc1ca37641d56a07cf8c7b9a78ce091", + "@polyengine/wasi@0.5.1": { + "integrity": "1cb71ff91c24415d85a3b15bbb35462757586fa1478febba605cab4c1b2dbe72", "dependencies": [ - "jsr:@polyengine/protocol@~0.2.2" + "jsr:@polyengine/protocol@~0.2.3" ] }, "@polymorph/webrtc-datachannels@0.5.0": { @@ -624,10 +625,10 @@ }, "workspace": { "dependencies": [ - "jsr:@polyengine/protocol@0.2.2", - "jsr:@polyengine/runtime@0.5.0", - "jsr:@polyengine/translator@0.5.0", - "jsr:@polyengine/wasi@0.5.0", + "jsr:@polyengine/protocol@0.2.3", + "jsr:@polyengine/runtime@0.5.1", + "jsr:@polyengine/translator@0.5.1", + "jsr:@polyengine/wasi@0.5.1", "jsr:@polymorph/webrtc-datachannels@0.5.0", "jsr:@polymorph/websocket@0.5.0" ], diff --git a/host-polyengine/README.md b/host-polyengine/README.md index 6a6f241..e8f64f8 100644 --- a/host-polyengine/README.md +++ b/host-polyengine/README.md @@ -33,7 +33,8 @@ pinned module graph + the `node-datachannel` addon `src/run-endpoint.ts` — the endpoint lifecycle (bind + identity through idempotent teardown), the relay and WebRTC wires, the issue #10 concurrency rows, and the liveness/recovery rows (idle survival, relay -outage). Each scenario names its assertions where it lives; the exam's +outage, stalling-relay dial deadlines). Each scenario names its +assertions where it lives; the exam's summary line is the inventory. The exam retries the handshake-shaped scenarios a bounded number of @@ -45,8 +46,11 @@ guest's and is latent on every host. ## The pin polyengine and the sibling host modules arrive from JSR under caret -constraints on one minor line: the `@polyengine/{runtime,translator,wasi}@^0.5.0` -lockstep family, plus `@polyengine/protocol@^0.2.2` (versioned independently +constraints on one minor line: the `@polyengine/{runtime,translator,wasi}@^0.5.1` +lockstep family (0.5.1 is a floor, not a convenience: the endpoint's +dial timeouts drop in-flight import futures, which polyengine handles as +a prompt discard only from 0.5.1 — A23; under 0.5.0 the drop wedges the +store, polyengine#239), plus `@polyengine/protocol@^0.2.2` (versioned independently of the lockstep family — the A22 host-ABI vocabulary line) and `jsr:@polymorph/*@^0.5.0`. `deno.lock` pins the resolved versions and carries integrity, enforced with `--frozen`. `@polyengine/translator` ships diff --git a/host-polyengine/deno.json b/host-polyengine/deno.json index fe797e7..1443fb3 100644 --- a/host-polyengine/deno.json +++ b/host-polyengine/deno.json @@ -10,10 +10,10 @@ "exclude": ["jsr:@polyengine/*", "jsr:@polymorph/*"] }, "imports": { - "@polyengine/runtime/embedder": "jsr:@polyengine/runtime@^0.5.0/embedder", + "@polyengine/runtime/embedder": "jsr:@polyengine/runtime@^0.5.1/embedder", "@polyengine/protocol": "jsr:@polyengine/protocol@^0.2.2", - "@polyengine/translator": "jsr:@polyengine/translator@^0.5.0", - "@polyengine/wasi": "jsr:@polyengine/wasi@^0.5.0", + "@polyengine/translator": "jsr:@polyengine/translator@^0.5.1", + "@polyengine/wasi": "jsr:@polyengine/wasi@^0.5.1", "@polymorph/webcrypto": "jsr:@polymorph/webcrypto@^0.5.0", "@polymorph/websocket": "jsr:@polymorph/websocket@^0.5.0", "@polymorph/webrtc-datachannels": "jsr:@polymorph/webrtc-datachannels@^0.5.0" diff --git a/host-polyengine/deno.lock b/host-polyengine/deno.lock index 04328da..1a11ad6 100644 --- a/host-polyengine/deno.lock +++ b/host-polyengine/deno.lock @@ -1,10 +1,11 @@ { "version": "5", "specifiers": { - "jsr:@polyengine/protocol@~0.2.2": "0.2.2", - "jsr:@polyengine/runtime@0.5": "0.5.0", - "jsr:@polyengine/translator@0.5": "0.5.0", - "jsr:@polyengine/wasi@0.5": "0.5.0", + "jsr:@polyengine/protocol@~0.2.2": "0.2.3", + "jsr:@polyengine/protocol@~0.2.3": "0.2.3", + "jsr:@polyengine/runtime@~0.5.1": "0.5.1", + "jsr:@polyengine/translator@~0.5.1": "0.5.1", + "jsr:@polyengine/wasi@~0.5.1": "0.5.1", "jsr:@polymorph/webcrypto@0.5": "0.5.0", "jsr:@polymorph/webrtc-datachannels@0.5": "0.5.0", "jsr:@polymorph/websocket@0.5": "0.5.0", @@ -12,37 +13,37 @@ "npm:werift@0.22.2": "0.22.2" }, "jsr": { - "@polyengine/protocol@0.2.2": { - "integrity": "ff470d7183924119ae181cc513093318f1587f7ad5c4bc02db7344e90c9ff52b" + "@polyengine/protocol@0.2.3": { + "integrity": "e3f0ac4165eb691d7231164ce28cf75e275c99496aa538e032297febdc52b371" }, - "@polyengine/runtime@0.5.0": { - "integrity": "36ea9b74dd38c0f5fb941695cfa3223190fda54bc18f7328600ece082dbbb794", + "@polyengine/runtime@0.5.1": { + "integrity": "6337c94d623bc2013fdeb324a5206860a916b8897b39cf4bec65b1bf87c287d0", "dependencies": [ - "jsr:@polyengine/protocol" + "jsr:@polyengine/protocol@~0.2.3" ] }, - "@polyengine/translator@0.5.0": { - "integrity": "2f9770f468bd0c649732e6107d69715dea715c71564fe5b0ba0997b5d70e81c3", + "@polyengine/translator@0.5.1": { + "integrity": "16d7ac88997f89f10145cc109737b749bfebd37c9ea1aefe0a41b44c01b75586", "dependencies": [ "jsr:@polyengine/runtime" ] }, - "@polyengine/wasi@0.5.0": { - "integrity": "61275e66895084807393bf93876b74be9cc1ca37641d56a07cf8c7b9a78ce091", + "@polyengine/wasi@0.5.1": { + "integrity": "1cb71ff91c24415d85a3b15bbb35462757586fa1478febba605cab4c1b2dbe72", "dependencies": [ - "jsr:@polyengine/protocol" + "jsr:@polyengine/protocol@~0.2.3" ] }, "@polymorph/webcrypto@0.5.0": { "integrity": "b9b03e2855a83f8817bb9d6f6a0cb5ec261a4dd3c09c957b741eeb7eba639ef9", "dependencies": [ - "jsr:@polyengine/protocol" + "jsr:@polyengine/protocol@~0.2.2" ] }, "@polymorph/webrtc-datachannels@0.5.0": { "integrity": "fef8803ef87826ce8c5cc6545460ce2e41ef63d5c31d10f63e9fb0666a752734", "dependencies": [ - "jsr:@polyengine/protocol", + "jsr:@polyengine/protocol@~0.2.2", "npm:node-datachannel", "npm:werift" ] @@ -50,7 +51,7 @@ "@polymorph/websocket@0.5.0": { "integrity": "84a7862e9bdeaedca92f7b57b12f5ee1348878734906feebb4d37634ac6659ae", "dependencies": [ - "jsr:@polyengine/protocol" + "jsr:@polyengine/protocol@~0.2.2" ] } }, @@ -610,9 +611,9 @@ "workspace": { "dependencies": [ "jsr:@polyengine/protocol@~0.2.2", - "jsr:@polyengine/runtime@0.5", - "jsr:@polyengine/translator@0.5", - "jsr:@polyengine/wasi@0.5", + "jsr:@polyengine/runtime@~0.5.1", + "jsr:@polyengine/translator@~0.5.1", + "jsr:@polyengine/wasi@~0.5.1", "jsr:@polymorph/webcrypto@0.5", "jsr:@polymorph/webrtc-datachannels@0.5", "jsr:@polymorph/websocket@0.5" diff --git a/host-polyengine/src/run-endpoint.ts b/host-polyengine/src/run-endpoint.ts index 5ac3c20..fb88869 100644 --- a/host-polyengine/src/run-endpoint.ts +++ b/host-polyengine/src/run-endpoint.ts @@ -132,6 +132,19 @@ const OUTAGE_RECOVERY_MS = 20_000; // so teardown happens with a relay dial pending. const OUTAGE_TEARDOWN_MS = 2_000; +// The guest's relay dial deadline (endpoint_impl.rs DIAL_TIMEOUT). +// The stall assertions bound BOTH sides: a dial that fails much +// earlier did not time out (it errored), much later did not have a +// working deadline. +const DIAL_TIMEOUT_MS = 10_000; +const DIAL_TIMEOUT_SLACK_LOW_MS = 500; +const DIAL_TIMEOUT_SLACK_HIGH_MS = 10_000; + +// How long the home-relay redial probe (issue #93) waits for the pump to +// arm a second dial against the stalling stub, proving the first timed +// out and the redial slot survived it. +const STALL_REDIAL_WAIT_MS = 35_000; + /** * Bounded retries around the RefCell borrow hazard (see the header). The * budget is per-shape because the shapes lose the race at very different @@ -496,6 +509,8 @@ interface RelayControl { url(): string; stop(): Promise; start(): Promise; + /** Whether this run owns the relay process (can actually stop it). */ + owned(): boolean; } interface OutageReport { @@ -633,6 +648,288 @@ async function outageProbeBody( }; } +// --- the stalling-relay probe (issue #93) ------------------------------------ + +/** A relay stub that accepts TCP and never completes the relay handshake: + * the guest's dial sits reading a socket that never writes back. */ +interface StallStub { + readonly url: string; + accepts(): number; + close(): Promise; +} + +/** + * Start a stall stub on `port` (0 for ephemeral). Every accepted + * connection is drained and held open, never written to — the shape + * that pinned `bind`/`ensure_relay`/the home-relay redial before issue + * #93's dial deadline. + */ +async function startStallStub(port: number): Promise { + const listener = Deno.listen({ hostname: "127.0.0.1", port }); + const addr = listener.addr as Deno.NetAddr; + let accepted = 0; + let closing = false; + const conns = new Set(); + + const acceptLoop = (async () => { + try { + for await (const conn of listener) { + accepted++; + conns.add(conn); + drain(conn); + } + } catch { + // The listener closed out from under the accept loop; expected on + // shutdown. + } + })(); + + function drain(conn: Deno.Conn): void { + (async () => { + const buf = new Uint8Array(4096); + try { + for (;;) { + const n = await conn.read(buf); + if (n === null) break; + } + } catch { + // The socket was closed by the peer or by our own shutdown; + // this stub never writes, so nothing else can go wrong here. + } finally { + conns.delete(conn); + } + })(); + } + + return { + url: `http://127.0.0.1:${addr.port}`, + accepts: () => accepted, + close: async () => { + if (closing) return; + closing = true; + try { + listener.close(); + } catch { /* already closed */ } + for (const conn of conns) { + try { + conn.close(); + } catch { /* already closed */ } + } + await acceptLoop; + }, + }; +} + +/** The outcome of one relay dial: `kind` is the WIT error variant (or + * "resolved" if the dial succeeded), `elapsedMs` how long it took. Never + * throws itself — a real dial hang is caught by the caller's `deadline`. */ +interface DialOutcome { + readonly kind: string; + readonly elapsedMs: number; +} + +async function awaitDialOutcome(p: Promise): Promise { + const t0 = performance.now(); + let kind: string; + try { + await p; + kind = "resolved"; + } catch (err) { + if (err instanceof ComponentException) { + const payload = err.payload as { kind?: string } | undefined; + kind = payload?.kind ?? "unknown"; + } else { + kind = describeError(err); + } + } + return { kind, elapsedMs: performance.now() - t0 }; +} + +/** Whether an elapsed dial time falls inside the deadline's slack window: + * much earlier means the dial errored rather than timing out; much later + * means the deadline did not actually bound it. */ +function inDialTimeoutWindow(ms: number): boolean { + return ms >= DIAL_TIMEOUT_MS - DIAL_TIMEOUT_SLACK_LOW_MS && + ms <= DIAL_TIMEOUT_MS + DIAL_TIMEOUT_SLACK_HIGH_MS; +} + +interface StallLegAReport { + readonly kind: string; + readonly elapsedMs: number; + readonly accepts: number; +} + +interface StallLegBReport { + readonly firstKind: string; + readonly firstMs: number; + readonly secondKind: string; + readonly secondMs: number; + readonly accepts: number; +} + +interface StallLegCReport { + readonly wentDown: boolean; + readonly accepts: number; + readonly echoed: string; +} + +interface StallReport { + readonly legA: StallLegAReport; + readonly legB: StallLegBReport; + /** `undefined` when the relay was adopted rather than owned (scenario 7's + * shape): this run cannot stop the real relay, so the redial leg cannot + * run. */ + readonly legC: StallLegCReport | undefined; +} + +/** + * One stalling-relay probe (issue #93): a relay that accepts TCP and never + * completes the handshake must fail every dial that reaches it at the + * guest's dial deadline, never hang it, and never pin the redial slot it + * occupies. Gathers data only — every assertion runs once, after the + * retry loop, against the returned report (scenario 7's shape: a `check()` + * inside a retried body would poison the verdict on a transient first + * attempt). + */ +async function stallProbeOnce(control: RelayControl): Promise { + // Leg A: bind against a stall stub directly. + const legA = await (async (): Promise => { + const stub = await startStallStub(0); + try { + const inst = await newEndpointInstance({ label: "stall-bind" }); + const outcome = await deadline( + awaitDialOutcome( + bindEndpoint(inst, { alpns: [ALPN], relayUrl: stub.url, webrtc: false }), + ), + 30_000, + "leg A bind", + ); + return { kind: outcome.kind, elapsedMs: outcome.elapsedMs, accepts: stub.accepts() }; + } finally { + await stub.close(); + } + })(); + + // Leg B: a foreign-relay open through connect, and its claim-release. + // Any throw here still owns two bound endpoints, whose pumps would + // otherwise redial through every later scenario. + const legB = await (async (): Promise => { + const stub = await startStallStub(0); + const server = await newEndpointInstance({ label: "stall-b-server" }); + const client = await newEndpointInstance({ label: "stall-b-client" }); + let sep: Endpoint | undefined; + let cep: Endpoint | undefined; + try { + const bindOptions = { alpns: [ALPN], relayUrl: control.url(), webrtc: false }; + sep = await deadline(bindEndpoint(server, bindOptions), 30_000, "server bind"); + cep = await deadline(bindEndpoint(client, bindOptions), 30_000, "client bind"); + const serverId = await sep.id(); + const addrs: TransportAddr[] = [{ kind: "relay", value: stub.url }]; + + const first = await deadline( + awaitDialOutcome(cep.connect({ endpointId: serverId, addrs }, ALPN)), + 30_000, + "leg B first connect", + ); + // The claim-release assertion: before issue #93's fix, a second + // dialer against the same foreign relay waited out a 30s claim + // instead of dialing, because the first dial never released it. + const second = await deadline( + awaitDialOutcome(cep.connect({ endpointId: serverId, addrs }, ALPN)), + 30_000, + "leg B second connect", + ); + return { + firstKind: first.kind, + firstMs: first.elapsedMs, + secondKind: second.kind, + secondMs: second.elapsedMs, + accepts: stub.accepts(), + }; + } finally { + if (cep) await closeQuietly(cep, "client close after leg B"); + if (sep) await closeQuietly(sep, "server close after leg B"); + await stub.close(); + } + })(); + + // Leg C: the home-relay redial keeps redialing through a stall, and the + // endpoint recovers once a working relay returns. + if (!control.owned()) { + return { legA, legB, legC: undefined }; + } + + const server = await newEndpointInstance({ label: "stall-c-server" }); + let sep: Endpoint | undefined; + try { + const legC = await (async (): Promise => { + sep = await deadline( + bindEndpoint(server, { alpns: [ALPN], relayUrl: control.url(), webrtc: false }), + 30_000, + "server bind", + ); + const serverId = await sep.id(); + + await control.stop(); + let wentDown = false; + for (let i = 0; i < 100; i++) { + if (!await portListening(RELAY_PORT)) { + wentDown = true; + break; + } + await settle(100); + } + + const stub = await startStallStub(RELAY_PORT); + let accepts = 0; + try { + const started = performance.now(); + while (performance.now() - started < STALL_REDIAL_WAIT_MS) { + accepts = stub.accepts(); + if (accepts >= 2) break; + await settle(200); + } + accepts = stub.accepts(); + } finally { + await stub.close(); + } + + await control.start(); + + const client = await newEndpointInstance({ label: "stall-c-client" }); + let cep: Endpoint | undefined; + try { + cep = await deadline( + bindEndpoint(client, { alpns: [ALPN], relayUrl: control.url(), webrtc: false }), + 30_000, + "client bind", + ); + const addrs: TransportAddr[] = [{ kind: "relay", value: control.url() }]; + const conn = await deadline( + cep.connect({ endpointId: serverId, addrs }, ALPN), + 60_000, + "recovery connect", + ); + const sconn = await deadline(sep.accept(), 60_000, "recovery accept"); + const echoed = await echoRoundtrip(conn, sconn, "post-stall recovery echo"); + await conn.close(CLOSE_CODE, CLOSE_REASON); + await deadline(conn.waitClosed(), 30_000, "client wait-closed"); + return { wentDown, accepts, echoed }; + } finally { + if (cep) await closeQuietly(cep, "client close after leg C"); + } + })(); + if (sep) await closeQuietly(sep, "server close after leg C"); + return { legA, legB, legC }; + } catch (err) { + // Any throw in leg C must still leave the real relay running for the + // teardown scenario, mirroring scenario 7's restore-on-failure care. + if (!await portListening(RELAY_PORT)) await control.start(); + if (sep) await closeQuietly(sep, "server close after a failed leg C"); + throw err; + } +} + async function main(): Promise { installPanicWatchdog(); console.log("iroh endpoint exam (polyengine / stock Deno)"); @@ -646,6 +943,7 @@ async function main(): Promise { start: async () => { relay = await startRelay(); }, + owned: () => relay.owned, }; try { // -- 1 ------------------------------------------------------------------- @@ -1018,9 +1316,103 @@ async function main(): Promise { ); // -- 8 ------------------------------------------------------------------- + await scenario( + 8, + "stalling relay: dials fail at the deadline instead of pinning (issue #93)", + async (v) => { + let r: StallReport | undefined; + let lastError = ""; + // Two attempts only, as scenario 7: each costs the full stall + // budget in wall time, and legs B/C run handshakes that rarely + // lose the RefCell race (see the header). + const attempts = 2; + for (let attempt = 1; attempt <= attempts && !r; attempt++) { + takeGuestPanics(); + try { + r = await stallProbeOnce(relayControl); + } catch (err) { + lastError = describeError(err); + console.log(` attempt ${attempt}/${attempts} failed: ${lastError}`); + if (relay.owned && !await portListening(RELAY_PORT)) await relayControl.start(); + await settle(100); + } + } + if (!r) throw new Error(`no attempt completed; last: ${lastError}`); + await settle(); + const panics = takeGuestPanics(); + + // Every assertion runs exactly once, against the attempt that + // completed — never inside the retried body (a check-false in a + // discarded attempt must not poison a later clean one). + check( + v, + r.legA.kind === "timed-out", + `leg A: bind against a stalling relay rejected ${r.legA.kind} ` + + `after ${r.legA.elapsedMs.toFixed(0)} ms`, + ); + check( + v, + inDialTimeoutWindow(r.legA.elapsedMs), + `leg A: elapsed ${r.legA.elapsedMs.toFixed(0)} ms is within ` + + `[${DIAL_TIMEOUT_MS - DIAL_TIMEOUT_SLACK_LOW_MS}, ` + + `${DIAL_TIMEOUT_MS + DIAL_TIMEOUT_SLACK_HIGH_MS}] ms of DIAL_TIMEOUT_MS`, + ); + check(v, r.legA.accepts >= 1, `leg A: the stub accepted ${r.legA.accepts} dial(s)`); + + check( + v, + r.legB.firstKind === "timed-out", + `leg B: the first connect through a stalling foreign relay rejected ` + + `${r.legB.firstKind} after ${r.legB.firstMs.toFixed(0)} ms`, + ); + check(v, inDialTimeoutWindow(r.legB.firstMs), `leg B: first elapsed within the window`); + check( + v, + r.legB.secondKind === "timed-out", + `leg B: a second connect through the same stalling relay also rejected ` + + `${r.legB.secondKind} after ${r.legB.secondMs.toFixed(0)} ms — the first dial ` + + `released its claim on the relay instead of pinning it`, + ); + check(v, inDialTimeoutWindow(r.legB.secondMs), `leg B: second elapsed within the window`); + check( + v, + r.legB.accepts >= 2, + `leg B: the stub accepted ${r.legB.accepts} dial(s) (both connects reached it)`, + ); + + if (r.legC) { + check(v, r.legC.wentDown, `leg C: the real relay stopped accepting on ${RELAY_PORT}`); + check( + v, + r.legC.accepts >= 2, + `leg C: the home-relay redial armed a second dial against the stalling stub ` + + `within ${STALL_REDIAL_WAIT_MS} ms (accepts: ${r.legC.accepts})`, + ); + check( + v, + r.legC.echoed === MESSAGE.toUpperCase(), + "leg C: a fresh connect + echo succeeded after the real relay returned " + + "(the redial slot survived the stalled dials)", + ); + } else { + v.notes.push("leg C skipped: the relay was pre-existing and adopted, ran only A+B"); + } + + check( + v, + panics.length === 0, + `no guest trap across the stalled-dial drops (${panics.join("; ")})`, + ); + + v.detail = `bind timed out at ${r.legA.elapsedMs.toFixed(0)} ms; leg B claim released; ` + + (r.legC ? `redial re-armed and recovery echo OK` : `leg C skipped (adopted relay)`); + }, + ); + + // -- 9 ------------------------------------------------------------------- // Last by necessity: this scenario stops the relay every later // scenario would need. - await scenario(8, "teardown: close + wait-closed, relay reaped", async (v) => { + await scenario(9, "teardown: close + wait-closed, relay reaped", async (v) => { const inst = await newEndpointInstance({ label: "teardown" }); const ep = await deadline( bindEndpoint(inst, { alpns: [ALPN], relayUrl: relay.url, webrtc: false }),