fix(worker): log the reconnect after a stable connection drops at INFO

[bharadwaj-pendyala]

Closes #6108.

Problem

Worker._connection_task resets retry_count to 0 at worker.py:1118, immediately after a successful ws_connect. Every failure after that computes retry_delay = min(0 * 2, 10), so the first retry following any control-plane websocket drop logs:

WARNING failed to connect to livekit, retrying in 0s

and then reconnects successfully. The reporter sees these a few times a day across worker replicas, every one self-healing. They trip warning-level alerting with nothing behind them, so operators learn to filter the whole message, which then hides the backoff events that do matter.

Solution

Log that retry at INFO instead, under two conditions:

• the connection had been up for at least STABLE_CONNECTION_INTERVAL (30s) before it dropped, and
• the failure was the connection going away: APIStatusError from the unexpected-close path in _run_ws, aiohttp.ClientError, or OSError.

Everything else stays at WARNING: a worker that never connected, one dropping faster than the interval, any retry that had to back off, and any fault that is not a lost connection.

Both conditions are load-bearing. Demoting on retry_delay == 0 alone silences real outages, which I verified against this base commit before settling on the current shape:

Scenario	main	plain retry_delay == 0 demotion	this PR
Upgrade accepted, register handshake never answered (3670 retries in 2s)	WARNING	INFO	WARNING
Registers, then drops straight back (3450 retries in 2s)	WARNING	INFO	WARNING
Stable connection drops, retry succeeds	WARNING	INFO	INFO

Neither hot loop backs off or reaches max_retry, because retry_count is reset by the accepted upgrade, so demoting them would have made a permanently broken worker silent.

Scope

One behaviour change, in one function. I deliberately left the retry_count reset at line 1118 alone: moving it is what would actually stop those two hot loops, but it changes backoff and max_retry semantics well beyond this issue, so it belongs in its own change with a maintainer's call on the semantics.

STABLE_CONNECTION_INTERVAL is a policy choice and I would rather you pick it than me. Any duration threshold moves the blind spot instead of removing it: a proxy that kills the connection every 31s still logs INFO forever. If you prefer a rate limit on the demotion, escalating when the quiet reconnect fires more than N times in a window, I will switch it. Raised the same question on the issue.

Validation

Base commit 060f107, all commands run locally.

• uv run pytest tests/test_worker_connection.py --unit → 7 passed. Six are new, one per row of the table above plus the backoff and non-transport-fault cases. Each was confirmed red before the fix and green after.
• uv run pytest tests/ --unit → 1498 passed, 5 skipped. Baseline on a clean tree is 1492 passed, so the 6 new tests are the only difference.
• uv run pytest tests/test_worker_connection.py --unit --concurrent → 7 passed. The module is marked no_concurrent because the virtual_time clock is process-global, matching test_connection_pool.py and the other virtual-time modules.
• uv run ruff check and uv run ruff format --check on both changed files → clean.
• uv run python scripts/check_types.py → 2 errors, both pre-existing missing cv2 and loguru stubs in livekit-plugins-bithuman. Identical on a clean tree; nothing in the changed files.

The 3670 and 3450 figures come from driving _connection_task with a stubbed ws_connect for 2 seconds and counting the retry records.

Note

I opened this without waiting for a reply on the issue, since the failing behaviour is unambiguous and the change is small. The threshold question above is the one assumption I made on your behalf, and I am happy to rework it.

This change was written with AI assistance. Every claim above comes from a command I ran on the base commit.

[CLAassistant]

All committers have signed the CLA.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.