Skip to content

A silent relay wire is detected: quiet wires are pinged, unanswered pings retire the connection - #97

Merged
lannbot merged 1 commit into
mainfrom
relay-liveness
Aug 23, 2026
Merged

A silent relay wire is detected: quiet wires are pinged, unanswered pings retire the connection#97
lannbot merged 1 commit into
mainfrom
relay-liveness

Conversation

@lannbot

@lannbot lannbot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

A relay that completed the handshake and then went silent-but-open was undetectable (#96): the pump retires and redials only on a websocket error, and a stalled wire errors nothing. The client answered server pings (core/src/relay.rs) but never initiated one and tracked no deadline. Consequences: silent home-death for the home relay (no redial ever arms; new dials black-hole; connections die at the QUIC idle timeout), and stale foreign pool entries whose transmits black-hole. A middlebox that holds TCP open while dropping payload produces exactly this shape.

This is the post-handshake sibling of #93 (fixed by #95, which bounded the dial itself).

The fix — upstream's mechanism

Upstream iroh runs a client-initiated ping loop in its relay actor: PING_INTERVAL = 15s, reset on any inbound relay message (so pings fire only on a quiet wire; the constant's own doc picks 15s as half QUIC's default 30s max-idle-timeout), with the pong bounded by ping_tracker's PING_TIMEOUT = 5s; a ping timeout is a connection error feeding the reconnect path. Same shape here:

  • Mechanism in core, policy in the pump. RelayConn stamps last_inbound on every inbound frame and offers probe_step (one policy step: clear an answered probe / request a ping on a quiet wire / declare death past the deadline) and send_ping. The pump's tick arm sweeps the pool: Dead on the home relay takes exactly the receive-error arm's path (retire + redial, immediate here since the connection lived ≥ 10s); Dead on a foreign relay retires it and its URL mapping, so a later dial reconnects.
  • Divergences, recorded at the definition sites: any inbound frame counts as the probe's answer (upstream correlates pong payloads — for detecting a silent wire, any-frame has equal power, so the ping payload is an uncorrelated counter); the pong bound is the fixed 5s cap (upstream shrinks it by measured RTT; this client tracks none).
  • The probe can never become the pin it hunts: ping sends are detached (spawn_local), so a backpressured wire cannot stall the pump — a ping that never leaves still reaches the pong deadline, which is the right verdict for that wire either way.

The regression guard

Exam scenario 9, owned-relay-gated like the outage row: a mute relay stub — accepts the websocket upgrade, selects the offered subprotocol, sends the one-byte immediate confirms-auth, then never replies while staying OPEN. No socket error is ever available; the probe alone must drive recovery:

  • the stub observes ≥ 1 client PING frame (measured: first at 15200 ms);
  • the endpoint retires the dead wire and redials — a second accept on the still-open stub (measured: 4867 ms after the ping);
  • once the real relay returns, a fresh dial + echo completes; no guest trap throughout.

Falsified against the unfixed endpoint (guest changes stashed, component rebuilt): scenario 9 fails with the client's liveness probe reached the mute wire: 0 PING(s) and the exam reports FAIL.

Not in this PR

The send side: the pump awaits datagram sends inline, and whether a host's websocket send can pend indefinitely against a backpressured wire (pinning the pump) is unmeasured on the wasmtime host — #96's open item 2, now tracked separately in #98.

Gates

  • just check (fmt, clippy, validate-wit, test): pass
  • just build: pass
  • just exam-polyengine: 10/10 with an owned relay — scenario 9: ping at 15200 ms, redial at 4867 ms after it, recovery echo OK
  • just matrix: all 13 pairings pass

Fixes #96

…ings retire the connection

A relay that completed the handshake and then went silent-but-open was
undetectable (#96): the pump retired and redialed only on a websocket
error, and a stalled wire errors nothing — silent home-death for the
home relay, black-holing pool entries for foreign ones. The client
answered server pings but never initiated its own.

The probe, upstream's mechanism (iroh-1.0.3 relay actor):

- RelayConn stamps every inbound frame (last-inbound) and offers
  probe_step/send_ping — mechanism in core, policy in the pump.
- The pump's tick sweeps the pool: a wire quiet for
  RELAY_PING_INTERVAL (15s, upstream PING_INTERVAL, reset-on-inbound
  semantics) is pinged; a ping unanswered for RELAY_PING_TIMEOUT (5s,
  upstream ping_tracker PING_TIMEOUT) declares the wire dead — the
  home relay retires into the redial, a foreign relay leaves the pool.
- Divergences recorded at the definition sites: any inbound frame
  counts as the answer (upstream correlates pong payloads; equal power
  against a silent wire), and the pong bound is fixed (upstream
  shrinks it by measured RTT; we track none).
- Ping sends are detached (spawn_local): a backpressured wire cannot
  pin the pump, and a ping that never leaves still reaches the pong
  deadline — the correct verdict for that wire either way.

The gate is exam scenario 9: a mute-relay stub completes the relay
handshake (immediate confirms-auth) and then never replies while
staying OPEN — no socket error is ever available. The endpoint must
ping it (observed on the stub), retire and redial the dead wire (a
second accept on the still-open stub), and recover once the real relay
returns. Measured: first ping at 15.2s, redial 4.9s after it.

Falsified against the unfixed endpoint: scenario 9 fails with
"the client's liveness probe reached the mute wire: 0 PING(s)".

Gates: just check, just build, just exam-polyengine (10/10),
just matrix (13/13).

Fixes #96
@lannbot
lannbot enabled auto-merge August 23, 2026 19:49
@lannbot
lannbot merged commit 6a5e8f4 into main Aug 23, 2026
1 check passed
@lannbot
lannbot deleted the relay-liveness branch August 23, 2026 19:54
@lannbot lannbot mentioned this pull request Aug 23, 2026
lannbot pushed a commit that referenced this pull request Aug 23, 2026
Patch release carrying the relay-robustness line landed since 0.5.0:
home-relay failure is a redial rather than endpoint death (#91), every
relay dial carries a deadline (#95), and quiet relay wires are pinged
with unanswered pings retiring the connection (#97). No WIT or JS
surface change; the published @polyengine/runtime floor is already
^0.5.1 (A23 prompt-discard of dropped import futures, polyengine#239).

Gates: build-components + deno publish --dry-run clean; full CI on the
PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

endpoint: a relay that stalls after the handshake is undetectable — no client-initiated liveness probe on the relay wire

2 participants