Skip to content

feat(reachability): keep the node reachable from the daemon - #207

Closed
Rinse12 wants to merge 12 commits into
masterfrom
feat/reachability-daemon
Closed

Rinse12 wants to merge 12 commits into
masterfrom
feat/reachability-daemon

Conversation

@Rinse12

@Rinse12 Rinse12 commented Nov 2, 2026

Copy link
Copy Markdown
Member

Part of #198 (step 7). Based on #206, so earlier commits show here until those merge.

Summary

flowchart TD
    start[daemon start] --> cfg{reachability/config.json}
    cfg -- missing --> line[one line naming the setup command]
    cfg -- disabled --> quiet[nothing]
    cfg -- enabled --> run[log addresses and expiry, keep Caddy up]
    run -- every 5 s --> watch[start, adopt, reload or stop Caddy to match the stored state]
    run -- every 5 min --> checks[re-derive IPs, verify names, check expiry]
    checks -- IP changed on 2 readings --> move[Caddyfile, reload, wait for certificate, AppendAnnounce, restart kubo]
    checks -- name failing --> warn[warn hourly; drop at 24 h; restore on recovery]
    run -- every 20 min --> lease[refresh router mappings]
Loading

Choices the issue left open, each with a test:

  • Two consecutive readings before acting on an IP change. A kubo restart is too costly to spend on one odd peer report.
  • No result for a family is not a change. A router that stops answering must not drop the address.
  • The new address is announced only once its certificate exists. Otherwise an IP change would announce a dead address for up to 90 s.
  • --forced and --announced names only have to resolve. A proxy answers for them by design, so they would otherwise be wrongly dropped after 24 hours. This adds domainForced to the config.
  • A dropped name is kept and restored when it recovers. This adds droppedAnnounce to the config.
  • The failure clock lives in runtime-status.json, so a daemon restart doesn't reset the 24 hours.
  • A leftover Caddy on the admin port is adopted, e.g. after the daemon was killed.

A failure anywhere in this runtime logs a warning and never stops the daemon.

Found by the end-to-end test: the daemon sends stderr only to its log file, so console.warn would never have reached an operator's terminal. These warnings now go to stdout.

Evidence

  • Before: setup left a config that nothing served.
    After: npm run test:cli: 62 files, 715 passed, 1 skipped.
daemon, then setup while it runs    -> daemon starts Caddy, js-libp2p handshake reaches kubo
stop Caddy through its admin API    -> daemon brings it back, handshake works again
stop the daemon                     -> Caddy's admin port goes away
unconfigured daemon                 -> exactly one "Browsers cannot reach this node" line on stdout
fake clock: name fails              -> warn at 0 h, silent at 30 min, warn at 61 min, dropped after 24 h, restored on recovery
second runtime after a restart      -> the 24 hours keep counting

The IP-change path is tested with fakes for the router and Caddy. No real router or address change has exercised it, and the UPnP refresh is untested against real hardware.

Merge Danger

Door: two-way

Blast Radius: daemon-startup

Every daemon now loads this module at start and may print one extra line. It stays off for a daemon using an external kubo through --pkcOptions.kuboRpcClientsOptions. An enabled node gains a Caddy child process, and on an IP change the daemon restarts kubo.

loadKuboConfigFile looked in .ipfs-bitsocial-cli while kubo runs in
.bitsocial-cli.ipfs or $IPFS_PATH, so the read always came back empty and the
daemon fell back to env vars and defaults. Both now use resolveKuboRepoPath.

That makes the multiaddr parsers reachable for the first time, so they now
handle IPv6 hosts, list-valued addresses and unix sockets.
computeKuboConfig(kuboConfig, reachabilityConfig, enabled) is the one place that
decides the kubo edits. planKuboConfig also returns a record of what changed,
which tells our entries from identical operator ones and lets disable restore
the touched fields byte for byte.
An explicit KUBO_RPC_URL or IPFS_GATEWAY_URL can name a kubo other than the one in
this repo, so it still wins over the kubo config. Connectivity checks now strip the
brackets a URL keeps around an IPv6 host.
reachability/config.json holds the operator's choices and refuses to load a
newer version rather than drop its fields. The daemon's check results go in a
separate file so they cannot overwrite a setup in progress.
…e status

Caddy keeps certificates for old addresses and other CAs until they expire, so
status now filters to the configured CA and current names. A damaged runtime
status file reads as no checks yet, and the text output prints domain and Caddy
checks.
…le tests

Caddy is downloaded on first use and checked against a sha256 pinned in source.
The generated Caddyfile keeps Caddy off UDP 443 and port 80. The integration test
issues a certificate for 127.0.0.1 from Pebble and completes a js-libp2p handshake
with kubo through Caddy.
Discovers public addresses from interfaces, kubo and the router, obtains an IP
certificate, applies the kubo config and restarts kubo. It checks what kubo really
listens on, because kubo stays silent when it cannot bind UDP 443.
--domain gets a normal certificate when 443 is free, or prints a proxy snippet for
the detected server when it is busy. --announce is added only after a libp2p
handshake through it succeeds. A name that does not resolve here is refused unless
--force is passed.
The daemon spawns Caddy and keeps it up, re-derives public IPs and verifies announced
names every 5 minutes, and moves the certificate and announcement on an IP change. A
stale name warns hourly and is dropped after 24 hours. Without a configuration it
prints one line naming the setup command.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T13:37:10.459020Z 140081f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Nov 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 2479259e-b6f5-494b-8c6b-3697721f296f
Commits

Reviewing files that changed from the base of the PR and between 6121b54 and 140081f.

Files ignored due to path filters: 3
  • package-lock.json is excluded by !**/package-lock.json
  • test/fixtures/pebble/listener-cert.pem is excluded by !**/*.pem
  • test/fixtures/pebble/listener-key.pem is excluded by !**/*.pem
Files selected for processing: 29
  • README.md
  • package.json
  • src/cli/commands/daemon.ts
  • src/cli/commands/reachability/setup.ts
  • src/cli/commands/reachability/status.ts
  • src/ipfs/startIpfs.ts
  • src/reachability/caddy.ts
  • src/reachability/discovery.ts
  • src/reachability/guidance.ts
  • src/reachability/kubo-config.ts
  • src/reachability/libp2p-dial.ts
  • src/reachability/nat.ts
  • src/reachability/runtime.ts
  • src/reachability/setup.ts
  • src/reachability/state.ts
  • src/util.ts
  • test/cli/daemon-honours-kubo-config-addresses.test.ts
  • test/cli/reachability-daemon.integration.test.ts
  • test/cli/reachability-setup.integration.test.ts
  • test/cli/reachability-status.test.ts
  • test/helpers/pebble.ts
  • test/kubo/kubo-repo-path.test.ts
  • test/reachability/caddy-pebble.integration.test.ts
  • test/reachability/caddy.test.ts
  • test/reachability/discovery.test.ts
  • test/reachability/kubo-config.test.ts
  • test/reachability/runtime.test.ts
  • test/reachability/setup.test.ts
  • test/reachability/state.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@Rinse12

Rinse12 commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Consolidated into #208.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 140081fbc0

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +372 to +374
// Only addresses that hold a certificate move. A proxy-only node has nothing here to change.
if (config.publicIps.length > 0) {
const nextIps = planIpChange(config.publicIps, derived);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Retry IP certificate issuance after a failed change

When an IP change is detected but certificate issuance temporarily fails for every new address, handleIpChange persists publicIps: []. Every subsequent check then skips planIpChange because of this guard, so the daemon never retries issuance even after the CA or network recovers, and WSS remains disabled until the operator reruns setup. The same guard also prevents a configuration with no initially issued IP from gaining one later.

Useful? React with 👍 / 👎.

Comment on lines +249 to +252
if (!fromInterfaces.some((ip) => net.isIPv4(ip))) {
this.router ??= await discoverNatRouter();
const external = await this.router?.externalIp().catch(() => undefined);
if (external) for (const ip of publicIpsFromMultiaddrs([`/ip4/${external}`])) ips.add(ip);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Map ports immediately after discovering the router

After a host/router reboot, the saved reachability configuration can exist while its UPnP mappings no longer do. The first check discovers the router and reads its external IP but does not call mapPorts; that only happens on the independent 20-minute lease-refresh timer, leaving TCP/UDP 443 and TCP 80 unreachable for up to 20 minutes after daemon startup. Create the mappings when the router is first discovered rather than waiting for the refresh interval.

Useful? React with 👍 / 👎.

Comment on lines +289 to +302
const issued = this.deps.waitForIssuance
? await this.deps.waitForIssuance(config, nextIps)
: (await waitForCertificates(this.deps.dataPath, config.acmeDirectory, nextIps)).issued;
if (issued.length !== nextIps.length) {
this.deps.warn(`No certificate could be issued for ${nextIps.filter((ip) => !issued.includes(ip)).join(", ")} after the IP change.`);
await fs.writeFile(caddyfilePath(this.deps.dataPath), generateCaddyfile({ ...spec, ips: issued }));
await this.watchCaddy();
}
const port = config.httpsPort ?? 443;
const ipAddress = (ip: string) => `/${net.isIPv6(ip) ? "ip6" : "ip4"}/${ip}/tcp/${port}/tls/ws`;
const oldIpAddresses = config.publicIps.map(ipAddress);
const announce = [...issued.map(ipAddress), ...config.announce.filter((address) => !oldIpAddresses.includes(address))];
this.deps.log(`Public address changed from ${config.publicIps.join(", ") || "none"} to ${issued.join(", ") || "none"}. Announcing: ${announce.join(", ") || "nothing"}`);
return this.saveAndApply({ ...config, publicIps: issued, announce });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: one failed certificate wait after an IP change disables IP reachability permanently.

If waitForCertificates times out (ACME rate limit/outage, router not yet forwarding 80/443), issued is [], so the config is saved with publicIps: [], the IP announce addresses are removed, and watchCaddy stops Caddy. runChecksOnce only plans IP changes when config.publicIps.length > 0 (line 373), so it never retries, even after the problem clears. An IP-only node stays unreachable over WSS until the operator manually reruns bitsocial reachability setup.

Suggested fix: on total failure keep the old publicIps/announce (or record the target IPs) so the next check retries.

const issued = this.deps.waitForIssuance
? await this.deps.waitForIssuance(config, nextIps)
: (await waitForCertificates(this.deps.dataPath, config.acmeDirectory, nextIps)).issued;
if (issued.length !== nextIps.length) {
this.deps.warn(`No certificate could be issued for ${nextIps.filter((ip) => !issued.includes(ip)).join(", ")} after the IP change.`);
await fs.writeFile(caddyfilePath(this.deps.dataPath), generateCaddyfile({ ...spec, ips: issued }));
await this.watchCaddy();
}
const port = config.httpsPort ?? 443;
const ipAddress = (ip: string) => `/${net.isIPv6(ip) ? "ip6" : "ip4"}/${ip}/tcp/${port}/tls/ws`;
const oldIpAddresses = config.publicIps.map(ipAddress);
const announce = [...issued.map(ipAddress), ...config.announce.filter((address) => !oldIpAddresses.includes(address))];
this.deps.log(`Public address changed from ${config.publicIps.join(", ") || "none"} to ${issued.join(", ") || "none"}. Announcing: ${announce.join(", ") || "nothing"}`);
return this.saveAndApply({ ...config, publicIps: issued, announce });
}

Comment on lines +224 to +226
if (await this.caddy.isUp()) {
this.caddyRunning = true;
this.loadedCaddyfile = undefined;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: an adopted Caddy is never watched, so it is never restarted if it dies.

When isUp() is true the code sets caddyRunning = true but keeps no child and registers no exit hook; only the daemon-started path (line ~229) resets the flag. Scenario: the operator reruns bitsocial reachability setup while the daemon's Caddy is failing; the daemon adopts setup's temporary Caddy within 5 s, setup then calls stopCaddy, and the daemon still believes Caddy is running. WSS stays down until the daemon restarts (same for a leftover Caddy from a killed daemon that later dies).

Suggested fix: while Caddy is adopted (no managed child), re-check isUp() each tick and clear caddyRunning when it stops answering.

// Left by a daemon that was killed, or by a setup still in progress: adopt it, don't fight it.
if (await this.caddy.isUp()) {
this.caddyRunning = true;
this.loadedCaddyfile = undefined;
} else {

Comment on lines +328 to +331
if (verdict.restore) {
const stillDropped = (next.droppedAnnounce ?? []).filter((address) => address !== entry.address);
next = { ...next, announce: [...next.announce, entry.address], ...(stillDropped.length > 0 ? { droppedAnnounce: stillDropped } : {}) };
if (stillDropped.length === 0) delete next.droppedAnnounce;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: restoring a name can append a duplicate announce address.

judgeName decides restore from status.domains[name].droppedFromAnnounce (runtime-status.json), not from config.droppedAnnounce. After a name is dropped, the warning tells the operator to fix DNS and rerun bitsocial reachability setup; setup rebuilds announce including the domain and neither clears droppedAnnounce nor the status flag. On the next check the name is healthy, restore is true, and announce: [...next.announce, entry.address] adds the address a second time (persisted, shown in status, and a misleading "announced again" log).

Suggested fix: dedupe on restore (e.g. [...new Set([...next.announce, entry.address])]) or decide restore from config.droppedAnnounce.includes(address).

}
if (verdict.restore) {
const stillDropped = (next.droppedAnnounce ?? []).filter((address) => address !== entry.address);
next = { ...next, announce: [...next.announce, entry.address], ...(stillDropped.length > 0 ? { droppedAnnounce: stillDropped } : {}) };
if (stillDropped.length === 0) delete next.droppedAnnounce;
droppedFromAnnounce = false;

private async handleIpChange(config: ReachabilityConfig, nextIps: string[]): Promise<ReachabilityConfig> {
// Moves only IP-certificate addresses. A domain's or proxy's address does not contain our IP.
const spec = {
domains: config.domain && config.announce.some((address) => address.includes(`/${config.domain}/`)) ? [config.domain] : [],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: a dropped domain loses its Caddy site on IP change and is not re-added when restored.

spec.domains only includes the domain if it is currently in config.announce. If the domain was dropped (it's in droppedAnnounce) and the IP then changes, the regenerated Caddyfile has no site for it. When DNS recovers, checkNames puts the address back into announce (line ~330) but nothing regenerates the Caddyfile, so the node announces /dns4/<domain>/tcp/443/tls/ws while Caddy has no site/certificate for it and browser TLS handshakes fail (and renewals stop).

Suggested fix: also count config.droppedAnnounce when building domains, or regenerate the Caddyfile on restore.

const spec = {
domains: config.domain && config.announce.some((address) => address.includes(`/${config.domain}/`)) ? [config.domain] : [],
wsPort: config.wsPort,

Comment on lines +139 to +142
await this.watchCaddy();
this.timers.push(setInterval(() => void this.watchCaddy().catch((error) => this.deps.warn(String(error))), CADDY_WATCH_INTERVAL_MS));
this.timers.push(setInterval(() => void this.runChecks().catch((error) => this.deps.warn(String(error))), CHECK_INTERVAL_MS));
this.timers.push(setInterval(() => void this.refreshLeases().catch((error) => this.deps.warn(String(error))), LEASE_REFRESH_INTERVAL_MS));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: a single failure of the first watchCaddy() leaves the runtime dead for the daemon's lifetime.

start() awaits watchCaddy() before creating any timer. watchCaddy can throw (ensureCaddyBinary download fails when offline at boot, or reloadCaddy fails on an adopted Caddy). Then start() rejects, daemon.ts only logs "Reachability failed to start", and the Caddy watch, IP/name checks and lease refresh never run until the daemon is restarted.

Suggested fix: wrap the first call like the interval calls (await this.watchCaddy().catch(warn)) or create the timers first.

}
await this.watchCaddy();
this.timers.push(setInterval(() => void this.watchCaddy().catch((error) => this.deps.warn(String(error))), CADDY_WATCH_INTERVAL_MS));
this.timers.push(setInterval(() => void this.runChecks().catch((error) => this.deps.warn(String(error))), CHECK_INTERVAL_MS));
this.timers.push(setInterval(() => void this.refreshLeases().catch((error) => this.deps.warn(String(error))), LEASE_REFRESH_INTERVAL_MS));
for (const timer of this.timers) timer.unref();

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant