Skip to content

feat(reachability): add the setup command with IP certificates - #205

Closed
Rinse12 wants to merge 10 commits into
masterfrom
feat/reachability-setup
Closed

Rinse12 wants to merge 10 commits into
masterfrom
feat/reachability-setup

Conversation

@Rinse12

@Rinse12 Rinse12 commented Nov 2, 2026

Copy link
Copy Markdown
Member

Part of #198 (step 5). Based on #204 and #201, so their commits show here until those merge.

Summary

reachability setup
  repo missing? init it the way the daemon does (not while a daemon holds the lock)
  public IPs = interfaces + kubo /id addresses + router external IP when no public IPv4
  behind NAT? ask router for TCP 443, TCP 80, UDP 443
  no public IP          -> explicit statement of what would change that
  TCP 443 busy          -> names the holder; Path B arrives in step 6
  otherwise             -> Caddy issues (up to 90 s); keep only the names that succeeded
    bind denied on Linux   -> offer sudo setcap when a TTY can answer, else print and exit 2
    bind denied in Docker  -> print cap_add or a port mapping, never sudo
  apply kubo config; restart kubo only if it changed
  verify kubo's real listen addresses

Gap in the issue, handled here. Linux setcap on Caddy does not help kubo, which also has to bind UDP 443. kubo starts normally, logs nothing and just doesn't listen, so "WebTransport, always" would report success with nothing listening. Setup now checks swarm/addrs/listen after the restart and prints the fix: the ip_unprivileged_port_start sysctl, which covers both binaries, or setcap on kubo, which is lost whenever kubo updates. Verified by running kubo as an unprivileged user.

Not done from the issue:

  • NAT-PMP. pmpNat needs the router's address, and only UPnP discovery is implemented. libp2p's own upnp-nat is also UPnP only.
  • The 20-minute lease refresh and release on shutdown belong to the daemon (step 7). Until then a mapping lasts its one-hour TTL.

Guard: setup refuses the production Let's Encrypt directory when CI is set.

Evidence

  • Before: bitsocial reachability setup did not exist.
    After: npm run test:cli passes.
daemon + Pebble + 127.0.0.1 as the "public" address:
  setup                -> exit 0, announces /ip4/127.0.0.1/tcp/<port>/tls/ws
  kubo config          -> AutoTLS off, loopback filter gone, AppendAnnounce set
  kubo after restart   -> swarm/addrs/listen includes the local ws port
  status --json        -> mode ip, certificate covers 127.0.0.1
  setup again          -> kubo config unchanged, no re-issue
  CI=true, no Pebble   -> refuses production Let's Encrypt

No real Let's Encrypt issuance has run yet. The manual staging test from #198 is still needed before anyone uses this for real.

Merge Danger

Door: two-way

disable (step 8) reverses the kubo edits, but until it lands reverting means editing the kubo config by hand.

Blast Radius: opt-in

Nothing changes unless an operator runs setup. A run writes the kubo config and restarts kubo, and until step 7 the daemon does not spawn Caddy, so this PR should not ship without step 7.

loadKuboConfigFile looked in .ipfs-bitsocial-cli while kubo runs in
.bitsocial-cli.ipfs or $IPFS_PATH, so the read always came back empty and the
daemon fell back to env vars and defaults. Both now use resolveKuboRepoPath.

That makes the multiaddr parsers reachable for the first time, so they now
handle IPv6 hosts, list-valued addresses and unix sockets.
computeKuboConfig(kuboConfig, reachabilityConfig, enabled) is the one place that
decides the kubo edits. planKuboConfig also returns a record of what changed,
which tells our entries from identical operator ones and lets disable restore
the touched fields byte for byte.
An explicit KUBO_RPC_URL or IPFS_GATEWAY_URL can name a kubo other than the one in
this repo, so it still wins over the kubo config. Connectivity checks now strip the
brackets a URL keeps around an IPv6 host.
reachability/config.json holds the operator's choices and refuses to load a
newer version rather than drop its fields. The daemon's check results go in a
separate file so they cannot overwrite a setup in progress.
…e status

Caddy keeps certificates for old addresses and other CAs until they expire, so
status now filters to the configured CA and current names. A damaged runtime
status file reads as no checks yet, and the text output prints domain and Caddy
checks.
…le tests

Caddy is downloaded on first use and checked against a sha256 pinned in source.
The generated Caddyfile keeps Caddy off UDP 443 and port 80. The integration test
issues a certificate for 127.0.0.1 from Pebble and completes a js-libp2p handshake
with kubo through Caddy.
Discovers public addresses from interfaces, kubo and the router, obtains an IP
certificate, applies the kubo config and restarts kubo. It checks what kubo really
listens on, because kubo stays silent when it cannot bind UDP 443.
@coderabbitai

coderabbitai Bot commented Nov 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d274d289-58b1-4dec-82f8-b894e19d79e1
Commits

Reviewing files that changed from the base of the PR and between 6121b54 and 654557c.

Files ignored due to path filters: 3
  • package-lock.json is excluded by !**/package-lock.json
  • test/fixtures/pebble/listener-cert.pem is excluded by !**/*.pem
  • test/fixtures/pebble/listener-key.pem is excluded by !**/*.pem
Files selected for processing: 26
  • README.md
  • package.json
  • src/cli/commands/daemon.ts
  • src/cli/commands/reachability/setup.ts
  • src/cli/commands/reachability/status.ts
  • src/ipfs/startIpfs.ts
  • src/reachability/caddy.ts
  • src/reachability/discovery.ts
  • src/reachability/guidance.ts
  • src/reachability/kubo-config.ts
  • src/reachability/libp2p-dial.ts
  • src/reachability/nat.ts
  • src/reachability/setup.ts
  • src/reachability/state.ts
  • src/util.ts
  • test/cli/daemon-honours-kubo-config-addresses.test.ts
  • test/cli/reachability-setup.integration.test.ts
  • test/cli/reachability-status.test.ts
  • test/helpers/pebble.ts
  • test/kubo/kubo-repo-path.test.ts
  • test/reachability/caddy-pebble.integration.test.ts
  • test/reachability/caddy.test.ts
  • test/reachability/discovery.test.ts
  • test/reachability/kubo-config.test.ts
  • test/reachability/setup.test.ts
  • test/reachability/state.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T13:22:34.063255Z 654557c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 654557cb81

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/reachability/setup.ts
Comment on lines +191 to +193
if (!exitedEarly) {
await stopCaddy(spec.adminPort).catch(() => caddy.kill());
await exited;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep Caddy running after certificate issuance

On the normal first-run path, setup starts Caddy, obtains the certificate, and then unconditionally stops Caddy before returning. No daemon code imports or starts Caddy, so the newly announced /tls/ws address has no TLS proxy listening as soon as bitsocial reachability setup exits, making browser WSS dials fail despite setup reporting success.

Useful? React with 👍 / 👎.

Comment thread src/reachability/nat.ts
Comment on lines +16 to +18
export async function discoverNatRouter(timeoutMs = 5000): Promise<NatRouter | undefined> {
const { upnpNat } = await import("@achingbrain/nat-port-mapper");
const client = upnpNat({ description: "bitsocial reachability", autoRefresh: false, ttl: MAPPING_TTL_MS });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Refresh router mappings before their lease expires

For nodes behind a UPnP router, these mappings are created with a one-hour TTL and automatic refresh disabled. The only mapPorts call in src occurs during setup, so after an hour the advertised TCP 443 and UDP 443 endpoints become unreachable while the persisted configuration still reports reachability as enabled.

Useful? React with 👍 / 👎.

Comment on lines +44 to +50
export default class Setup extends Command {
static override description =
"Make this node dialable from browsers: discover public addresses, obtain a certificate, configure kubo and verify. Safe to rerun";

static override flags = {
"pkcOptions.dataPath": Flags.directory({ description: "Data path of the node", required: false }),
staging: Flags.boolean({ description: "Use the Let's Encrypt staging CA. Its certificates are not trusted by browsers", default: false }),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Regenerate README documentation for the new commands

The commit adds reachability setup and reachability status, but the generated README contains no entries for either command; its only change is the daemon's Kubo path. This leaves the new user-facing commands and flags absent from the repository's command reference, contrary to the requirement to update documentation when commands or flags change.

AGENTS.md reference: AGENTS.md:L70-L81

Useful? React with 👍 / 👎.

Comment thread src/reachability/setup.ts
Comment on lines +136 to +139
const fromInterfaces = deps.publicIpsFromInterfaces();
const identity = await kuboIdentity(options.kuboRpcUrl);
const fromKubo = identity ? publicIpsFromMultiaddrs(identity.addresses) : [];
const ips = new Set([...fromInterfaces, ...fromKubo]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: a rerun after the public IP changes keeps announcing the old IP.

fromKubo comes from kubo's /id Addresses, which include the AppendAnnounce entries a previous setup wrote (e.g. /ip4/OLD_IP/tcp/443/tls/ws). Discovery never excludes them, so the old IP A is treated as a current one. waitForCertificates (src/reachability/caddy.ts, ~L205-209) counts any unexpired cert on disk as issued, and short-lived IP certs last about 6 days.

Scenario: setup runs on IP A, the ISP moves the host to B, and the operator reruns setup within ~6 days. issuedIps becomes [A, B], and both land in the Caddyfile, publicIps and kubo's AppendAnnounce, so the dead A stays announced. After A's cert expires, every rerun sends a doomed ACME validation for A and waits the full 90 s.

Suggested fix: drop hosts from previous.announce / previous.kuboEdits.addedAppendAnnounce out of fromKubo, or use only observed addresses. Alternatively, require certs issued after setup started.

Code

Comment thread src/reachability/setup.ts
Comment on lines +187 to +190
const wait = await Promise.race([
waitForCertificates(options.dataPath, spec.acmeDirectory, names, timeoutMs),
exited.then(() => undefined)
]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: the CLI hangs for up to ~90 s when Caddy exits early.

When the exited promise wins the Promise.race, the losing waitForCertificates call is abandoned but keeps polling on 500 ms setTimeouts until its 90 s deadline. It takes no AbortSignal, and the timers are not unref'd. The success path in src/cli/commands/reachability/setup.ts does not call process.exit.

Scenario: Caddy dies at startup (config error, or a non-root bind denial the user declines to fix), and setup continues to the "unreachable" warning and returns. The process then sits idle for most of the 90 s before Node exits. In the setcap retry path, the first attempt's poller also keeps running next to the second.

Suggested fix: pass an AbortSignal to waitForCertificates and abort it when Caddy exits, or unref the poll timer.

Code

@Rinse12

Rinse12 commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Consolidated into #208.

@Rinse12 Rinse12 closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant