rtu-usb: claim the HID interface, send full-length reports, allow disabling reset

[d01]

Updated 2026-08-16: the description of the reset (now section 5) was revised —
continued instrumented testing disproved my "fatal" characterization, and the
reset handling has since been reworked into commit 48797e4. Earlier update:
the original version of this PR removed the device reset outright, which was
wrong of me — see networkupstools/nut#2609.

Four commits for the rtu_usb backend: two bugs in the basic I/O path, two
more found in day-long soak testing, and a rework of how reset-on-open is
scoped. Found getting NUT's apc_modbus working with an APC Smart-UPS X1500
(051d:0003, firmware "UPS 16.0").

1. The HID interface is never claimed for I/O

_usb_get_hid_descriptor() claims the interface just long enough to read the
report descriptor and then releases it. On a match, _modbus_rtu_usb_connect()
stores the handle and breaks without claiming anything, so every subsequent
interrupt transfer runs against an interface still owned by the kernel HID
driver: the OUT transfer is not delivered, and the device's input reports go to
usbhid rather than to us.

Now claimed for the life of the connection and released in
_modbus_rtu_usb_close().

2. Short interrupt OUT transfers are ignored

The report descriptor declares output report 0x90 as 63 bytes, but the send
path transfers payload_chunk_len + 1 — 7 bytes for a 6-byte request. This
firmware accepts the short transfer at the host-controller level and then
ignores it: no reply arrives, and the Modbus engine is left unresponsive until
the USB cable is reseated. Sending the full declared length, zero padded, makes
the identical request work.

The OUT transfer also passed timeout 0 (wait forever), so an unresponsive
endpoint blocked the caller indefinitely rather than returning an error — which
is how this bug presented, and made it much harder to find. Now finite.

3. Flush until a full inter-frame interval of silence (63583ff)

The pre-send flush drained for a fixed 10 ms, which lets a straggling reply
from an earlier exchange survive into the next one and get mistaken for its
reply. APC's AN176 documents a 35 ms inter-frame minimum for these devices;
the flush now drains until it has seen a full 45 ms of continuous silence.

4. One receive deadline across skipped reports (a832970)

The receive path restarted the full response timeout for every non-Modbus HID
report it skipped. Against a device that broadcasts periodic notification
reports (this one emits several per second when degraded), the configured
timeout could effectively never fire — a "2 s timeout" only fired on total
device muteness. One deadline now spans the whole receive.

5. Reset-on-open: only the matched device, after the callback (48797e4), plus modbus_rtu_usb_set_reset_on_open()

Two problems with the reset added in #11, one of scoping and one I
mischaracterized:

Scoping: the reset ran inside enumeration, on every USB device on the bus
the code could open, before the device-selection callback was consulted —
so unrelated devices got reset while probing, and a callback had no way to
veto the reset for the device actually selected. 48797e4 moves it to fire
only on the matched device, after the callback has run. The default stays
enabled, so existing behavior for matched devices is preserved; the new
modbus_rtu_usb_set_reset_on_open() lets a caller that knows its hardware
opt out.

The mischaracterization: an earlier version of this PR called the reset
"fatal" on the X1500 and disabled it by default. Longer instrumented testing
showed that was wrong, and I want the record straight since it maligned the
mechanism beyond the data. A single reset is survivable on this model — it
typically goes deaf for up to 2.5 minutes and recovers fully, sometimes
resuming within seconds; it has absorbed more than a dozen isolated resets.
What actually wedged it (until a physical cable reseat) was resets in close
succession: my "fatal" case was a hidden restart loop — every driver start
issued this reset before the callback could veto it, the post-reset deaf
window beat the first-read timeout, systemd restarted the driver, and the
next start reset it again.

One data point on the #11 rationale (recovering a desynced device serving
stale replies), for whoever weighs the default: on this model the reset
usually does not shed the queued stale reply — across a dozen resets issued
against a known standing desync, two shed it, and the pattern is consistent
with the reset discarding only a reply already staged in the endpoint buffer
while a reply the firmware still withholds internally survives. Other
hardware may well behave differently; whether the default should change, or
this should be a device quirk or desync-triggered, is a judgement call for
someone with visibility across more hardware than I have — happy to rework
it whichever way you prefer.

Testing

All against NUT 2.8.4's apc_modbus, statically linked against this branch,
on the one X1500 (so review by anyone with other rtu_usb hardware would be
valuable):

• Fixes 1 and 2: with both, the driver starts, reads the inventory block and
  polls cleanly; with either missing, the 516:636 inventory read fails
  immediately and the driver exits with "Can't read inventory information
  from the UPS".
• Fixes 3 and 4 have run continuously for several days; a 24-hour soak came
  back with zero faults (no reconnects, no staleness, no read failure
  surviving a single retry).
• 48797e4 validated live since: seven deliberately armed resets each fired
  on the matched device only, after the callback; a full host reboot with a
  vetoing callback produced zero resets (confirmed in the kernel log).

Note on authorship

Developed with assistance from Claude (Anthropic); the commit carries a
Co-Authored-By trailer. All findings were verified on real hardware.

[d01]

Pushed three more commits from continued testing against an APC Smart-UPS (051d:0003):

• 63583ff — flush drains until a full 45 ms inter-frame interval of silence (AN176 documents a 35 ms interframe minimum; the previous 10 ms window let stragglers from an earlier exchange survive into the next one and get mistaken for its reply).
• a832970 — the receive path enforces one deadline across skipped non-Modbus reports. Previously the full timeout restarted for every non-Modbus HID report, so against a device emitting periodic notification reports the configured response timeout could effectively never fire.
• 48797e4 — reset-on-open resets only the matched device, and only after the selection callback has run. Before, enumeration reset every USB device on the bus it could open, before the callback could veto via modbus_rtu_usb_set_reset_on_open(). The default stays enabled, so existing behavior for matched devices is preserved.

All three have been running against the APC unit for several days through NUT's apc_modbus; the latest 24 h soak came back with zero faults.

[EchterAgo]

I'll test this on my units

[d01]

@EchterAgo Thanks! One heads-up before you start, so you're not testing against a stale description: I've just revised the PR body in light of further instrumented testing. The original "the reset is fatal" characterization was wrong — what wedged my unit was a restart loop clustering resets (every driver start issued a hidden reset before the callback could veto it); isolated resets are survivable here. The body now covers all four commits, including the 48797e4 ordering rework and its live validation, plus measurements on how often the reset actually sheds a queued stale reply on this model. Full correction with the data posted on networkupstools/nut#2609.

No code has changed since your comment — only the description.