Skip to content

vtun: wait for the network before starting, and pace the restart loop - #2406

Draft
openipc-ai wants to merge 1 commit into
masterfrom
vtun-dns-wait
Draft

vtun: wait for the network before starting, and pace the restart loop#2406
openipc-ai wants to merge 1 commit into
masterfrom
vtun-dns-wait

Conversation

@openipc-ai

Copy link
Copy Markdown
Collaborator

Problem

Reported by @usa- in #2319: after a power cut, a camera comes back with vtund filling the whole 64 KB syslog ring with the same two lines, so nothing else from that boot survives to be read.

Aug 29 09:05:21 gk7205v300-imx335 daemon.err vtund[21576]: syntax error line 5
Aug 29 09:05:21 gk7205v300-imx335 daemon.err vtund[21576]: No hosts defined

It is not a name-resolution failure, despite where the report started. The session name is the MAC of the default-route interface:

identity_src=$(ip r | awk '/default/ {print $5}' | head -n 1)
identity_mac=$(cat /sys/class/net/"$identity_src"/address | tr 'a-z' 'A-Z')

On a boot that reaches tunnel before the network is up, ip r matches nothing, identity_src is empty, cat /sys/class/net//address fails, and config() writes an empty session name — which is line 5 of the file it generates. vtund names the line it is looking at and exits.

The loop had no delay at all, so it did that as fast as the SoC allowed:

(while true; do identity; interface; config; vtund ... ; done) &

Two syslog lines per pass, no pause between passes, 64 KB gone in well under a minute.

What changes

One file, general/package/vtund-openipc/files/tunnel.

  • identity() reports that it has no identity instead of returning empty strings the caller cannot tell from a MAC, and the loop waits for one rather than writing a config it knows is broken.
  • The name half of the report is real too — vtund resolves the server itself and exits when it cannot — so the wait covers that as well, reusing the literal-address test from /usr/sbin/wireguard so a numeric server never waits on a resolver it does not need.
  • The failures that remain get a floor under their restart rate. vtund's own keepalive is 10:5, so it reconnects a live session by itself; this delay only paces the restarts it could not.
  • Waiting happens inside the loop that tunnel already backgrounds, so boot is not blocked. S98vtun is unchanged.

At most two syslog lines per outage episode: one on entering the wait, one on leaving it.

Hardware tested on

None — I have no camera running vtund, which is why this is a draft. The Evidence below is a stubbed busybox-ash run and the repository's own checks, not a board. Someone with the reproducer from #2319 needs to confirm it on hardware before this is ready.

@usa- — you have both the fleet and the repro; would you try this one?

Evidence

Not from a camera. vtund, ip, nslookup and logger are shell stubs; the shell is the busybox ash this tree builds. The scenario is "no default route yet", the #2319 boot.

Before:

old: vtund launched 17 times in 2s with no default route
     line 5 of the config it fed vtund: | {|

After:

new: vtund launched 0 times in 2s with no default route
     config: NOT WRITTEN
     SYSLOG waiting for a default route and for vtun.localhost to resolve
     SLEEP 10
     SLEEP 10

The generated config, with the identity empty, showing what line 5 actually is:

     4	}
     5	 {
     6		password ;

The other three cases:

B: default route up, hostname does not resolve  -> waits, config NOT WRITTEN
C: literal IP server (10.8.0.1)                 -> zero nslookup calls, starts at once
D: everything ready                             -> line5=|AABBCCDDEEFF {|

Repository checks:

STRICT=1 test_shell_parse.sh          -> 144 scripts, all parsed clean under busybox ash
STRICT=1 test_strip_shell_comments.sh -> all checks passed
ci-matrix.py --self-test              -> ok (99 boards, 136 packages, 56 cases)
ci-matrix.py --stdin                  -> 98/99 boards, narrowed to the boards that ship vtund

Scope

  • No kernel patches under general/package/all-patches/linux/ (those go to OpenIPC/linux)
  • No files specific to a single retail camera model (those go to OpenIPC/builder)
  • No probing or bring-up tooling (that goes to OpenIPC/ipctool)
  • Nothing under general/overlay/ or in a shared load_<vendor> script hardcodes a value specific to my board
  • Package sources come from an OpenIPC repository, and any version bump keeps at least the specificity of the pin it replaces (a new package should pin a full 40-character SHA)
  • No LD_PRELOAD, and no binaries that cannot be rebuilt from source
  • New code is selected by a defconfig, so CI actually builds it

The unticked box is deliberate: this changes no *_SITE and no *_VERSION, and vtund's existing SITE is sourceforge rather than an OpenIPC repository, so ticking it would claim something untrue.

Reported in #2319: a camera that loses power comes back with vtund filling
the whole 64KB syslog ring with

    vtund[21576]: syntax error line 5
    vtund[21576]: No hosts defined

and nothing else from that boot survives.

It is not a name-resolution failure, despite where the report started. The
session name is the MAC of the default-route interface, so on a boot that
reaches here before the network is up, `ip r` matches nothing, identity_src
is empty, `cat /sys/class/net//address` fails, and config() writes an empty
session name -- which is line 5 of the file it generates. vtund says exactly
what it is looking at and exits. The loop had no delay at all, so it did that
as fast as the SoC allowed: 17 launches in two seconds in a stubbed run, two
syslog lines each, which fills 64KB in under a minute.

So identity() now reports that it has no identity instead of returning empty
strings the caller cannot tell from a MAC, and the loop waits for one. The
name half of the original report is real too -- vtund resolves the server
itself and exits when it cannot -- so the wait covers that as well, reusing
the literal-address test from /usr/sbin/wireguard so a numeric server never
waits on a resolver it does not need.

The remaining failures get a floor under their restart rate. vtund's own
keepalive is 10:5, so it reconnects a live session by itself; this delay only
paces the restarts it could not.
@usa-

usa- commented Sep 13, 2026

Copy link
Copy Markdown
Contributor

Unfortunately, all of my cameras are currently deployed in production and are remote, so I cannot reliably reproduce the conditions where the camera boots before the network is available or DNS is temporarily unavailable. I was able to test similar WireGuard behaviour when I had a test camera locally, but that camera is no longer available to me. The camera I used to test other issues today is also located in another country.

One possible improvement, which may also make this and similar scripts easier to maintain, would be to extract the run_cmd helper recently added to WireGuard into a common script, for example /usr/sbin/common, and use it for syslog error logging from /usr/sbin/wireguard, /usr/sbin/tunnel, and potentially the other scripts in /usr/sbin as well as elsewhere in the firmware.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants