Skip to content

srt: limit retransmissions to avoid a NAK-driven retransmit storm - #2199

Open
RomanHerbstmann wants to merge 1 commit into
pedroSG94:masterfrom
RomanHerbstmann:fix/srt-limit-retransmissions
Open

RomanHerbstmann wants to merge 1 commit into
pedroSG94:masterfrom
RomanHerbstmann:fix/srt-limit-retransmissions

Conversation

@RomanHerbstmann

Copy link
Copy Markdown
Contributor

SRT: limit retransmissions

Motivation

Under a network bottleneck the SRT client can multiply its own traffic many times over. Receivers that use periodic NAK reports (e.g. gosrt, used by MediaMTX) report every missing packet again on each NAK interval (20 ms minimum) until the packet is received or its latency expires. CommandsManager.reSendPackets resends every packet in each reported range unconditionally, so a single lost packet can be sent up to latency / 20 ms times (100 times with a 2 s latency). On a link that is already too slow, the retransmits are lost too and the loss rate climbs.

In our measurement (Android emulator → MediaMTX 1.18.2, ingress policed to 250 kbit/s, ~300–450 kbit/s media, latency 2000 ms), about 600,000 packets were offered to the limiter within 150 s, a few percent of them original media packets, and 95 % were dropped.

In addition, packetsLost increases on every NAK report, including repeated reports of the same packet, so it overstates the real loss by a large factor and cannot be used as a loss-rate signal.

libsrt avoids this by limiting the retransmission overhead relative to the input bandwidth (SRTO_OHEADBW, default 25 %).

Change

  • Retransmit budget: CommandsManager keeps a token bucket for retransmissions. The rate is a percentage of the measured media rate (default 25 %, minimum 8 kB/s). The bucket allows an immediate burst of up to 0.5 s of media, so short loss events on a healthy link are recovered at once. The average stays at the configured percentage.
  • Oldest first: packets are resent in sequence order. Once the budget is exhausted, no further packets are resent for that NAK; the next NAK report retries them.
  • No pointless resends: a packet is not resent if it cannot arrive before its latency expires, or if it was already sent within max(rtt + 4 * rttVariance, 20 ms) (capped at latency / 4). RTT and RTT variance come from the ACK packets.
  • Configuration: SrtClient.setRetransmitOverhead(percent), exposed as SrtStreamClient.setRetransmitOverhead(percent). A value <= 0 restores the previous unlimited behavior.
  • Unique loss counter: new packetsLostUnique / SrtStreamClient.getPacketsLostUnique() counts each lost sequence number once. packetsLost is unchanged.
  • bytesSend still counts only original packets, so bitrate and stall detection based on it are unaffected.

Testing

  • Unit tests: new CommandsManagerTest (time gate, budget and refill, strict oldest-first, packets about to expire, burst on a healthy link, unlimited mode, sequence wrap, unique count, reset).
  • End-to-end: Android emulator publishing to MediaMTX 1.18.2 over SRT (latency 2000 ms). A hard ingress policer (tc police, small burst) on the MediaMTX UDP port for 150 s. Fresh stream per run, same app build except for this change.
During the 150 s bottleneck 250 kbit/s before 250 kbit/s after 400 kbit/s before 400 kbit/s after
Packets offered to the limiter 602,802 27,351 (−95 %) 43,207 20,194 (−53 %)
Dropped by the limiter 95 % 32 % 29 % 5 %
packetsLost / packetsLostUnique at the end 263,749 / 5,914 11,358 / 652
  • What it does not fix: at 250 kbit/s the media rate is above the link capacity, so keyframes still cannot complete and the picture still stalls. Only a lower bitrate helps there. The change stops the self-amplification, so once the bitrate is lowered below the link capacity, the stream can recover instead of being kept down by retransmits.
  • No regression: at 400 kbit/s and on the unthrottled link, segment durations and live-edge delay stayed the same. There were no reconnects, and bytesSend only counts original packets, so stall detection is unchanged.

If you prefer the limit to be opt-in (default 0), that is a one-line change.

Receivers with periodic NAK reports (e.g. gosrt) report every missing
packet again on each NAK interval (20 ms minimum). reSendPackets resent
all reported packets unconditionally, so under a bottleneck one lost
packet could be sent up to latency / 20 ms times and loss amplified itself.

- Token bucket for retransmissions: 25 % of the measured media rate by
  default (libsrt SRTO_OHEADBW), burst of up to 0.5 s of media.
- Resend oldest first; stop resending once the budget is exhausted.
- Skip packets that cannot arrive before their latency expires and
  already retransmitted packets reported again within
  max(rtt + 4 * rttVariance, 20 ms).
- setRetransmitOverhead(percent), <= 0 restores the unlimited behavior.
- packetsLostUnique counts each lost sequence number once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant