Parent epic: https://github.com/DIG-Network/dig_ecosystem/issues/3180
Found by: the DIG-Network/dig-updater#81 lane while fixing
DIG-Network/dig-updater#77
The defect
dign updater check-now reports IO_ERROR when it probes inside the restart window.
It is the same class as dig-updater#77 — which was "the beacon reports a failed service restart when
the restart succeeded" — but on the dig-node CLI side, where #77's fix does not reach. That fix
landed in dig-updater's own judge_restart; this surface probes independently and draws its own
conclusion.
Why it is worth a ticket rather than tolerating
The operator sees a failure for something that succeeded. install.rs renames the running image
aside, so there is a real window in which the new bytes are on disk, the service is mid-restart, and a
probe legitimately cannot reach it. Reporting that transient state as IO_ERROR is indistinguishable,
to the person reading it, from an update that genuinely broke.
That matters more under the epic's decided policy — silent staged install, restart when convenient
(https://github.com/DIG-Network/dig_ecosystem/issues/3180). Under a policy where nothing asks permission
and nothing blocks, the status surface is the only thing an operator has. A surface that cries wolf
during the normal restart window trains them to ignore it, which is exactly how the one real failure gets
ignored too.
The shape of the fix, from what #77 established
dig-updater#77's root cause was an asymmetry: an already-stopped service was tolerated for Stop, but
there was no already-running tolerance for Start, so sc.exe 1056 read as a fault. The general
fix was to judge from the service's observed run state rather than from an exit code — a three-valued
ServiceRunState with a bounded settle, not a boolean.
The same principle applies here: a probe that cannot reach the service during a restart has not
measured a failure, it has failed to measure. Those are different answers and must not share an
encoding. unknown/restart in progress is the honest result; IO_ERROR claims knowledge the probe
does not have.
Do not simply widen the timeout. That converts a wrong answer into a slower wrong answer, and it
interacts badly with the ~9s settle dig-updater now performs per service-backed component.
Related surface worth checking in the same pass
dig-updater#76 (in PR #81) separated installed{version, activation} from available, with activation
being active | pending_restart | unknown and defaulting to unknown. If dign renders update
status anywhere else, it should consume that distinction rather than re-deriving one — two independent
derivations of "is it active?" will disagree, and the one shown to the operator is the one they act on.
Acceptance
dign updater check-now, run during a real restart window, reports that a restart is in progress —
never IO_ERROR. A genuine I/O failure still reports IO_ERROR, proven by a test that distinguishes the
two rather than asserting only the happy path.
Parent epic: https://github.com/DIG-Network/dig_ecosystem/issues/3180
Found by: the DIG-Network/dig-updater#81 lane while fixing
DIG-Network/dig-updater#77
The defect
dign updater check-nowreportsIO_ERRORwhen it probes inside the restart window.It is the same class as dig-updater#77 — which was "the beacon reports a failed service restart when
the restart succeeded" — but on the dig-node CLI side, where #77's fix does not reach. That fix
landed in
dig-updater's ownjudge_restart; this surface probes independently and draws its ownconclusion.
Why it is worth a ticket rather than tolerating
The operator sees a failure for something that succeeded.
install.rsrenames the running imageaside, so there is a real window in which the new bytes are on disk, the service is mid-restart, and a
probe legitimately cannot reach it. Reporting that transient state as
IO_ERRORis indistinguishable,to the person reading it, from an update that genuinely broke.
That matters more under the epic's decided policy — silent staged install, restart when convenient
(https://github.com/DIG-Network/dig_ecosystem/issues/3180). Under a policy where nothing asks permission
and nothing blocks, the status surface is the only thing an operator has. A surface that cries wolf
during the normal restart window trains them to ignore it, which is exactly how the one real failure gets
ignored too.
The shape of the fix, from what #77 established
dig-updater#77's root cause was an asymmetry: an already-stopped service was tolerated for
Stop, butthere was no already-running tolerance for
Start, sosc.exe1056 read as a fault. The generalfix was to judge from the service's observed run state rather than from an exit code — a three-valued
ServiceRunStatewith a bounded settle, not a boolean.The same principle applies here: a probe that cannot reach the service during a restart has not
measured a failure, it has failed to measure. Those are different answers and must not share an
encoding.
unknown/restart in progressis the honest result;IO_ERRORclaims knowledge the probedoes not have.
Do not simply widen the timeout. That converts a wrong answer into a slower wrong answer, and it
interacts badly with the ~9s settle dig-updater now performs per service-backed component.
Related surface worth checking in the same pass
dig-updater#76 (in PR #81) separated
installed{version, activation}fromavailable, withactivationbeing
active | pending_restart | unknownand defaulting tounknown. Ifdignrenders updatestatus anywhere else, it should consume that distinction rather than re-deriving one — two independent
derivations of "is it active?" will disagree, and the one shown to the operator is the one they act on.
Acceptance
dign updater check-now, run during a real restart window, reports that a restart is in progress —never
IO_ERROR. A genuine I/O failure still reportsIO_ERROR, proven by a test that distinguishes thetwo rather than asserting only the happy path.