Conversation
The launch() statement has no timeout parameter, and async_script_launch()
registers the returned fd with reactor_add_reader() - i.e. with a timeout of
zero. In io_wait, a zero timeout never expires
(io_wait.h: "if (timeout) timeout += get_ticks()",
io_wait_loop.h: "e->timeout!=0 && e->timeout<=curr_time"), so when the reply
of a launch operation never arrives, the fd stays in the reactor for the
lifetime of the process: the descriptor and, for a module using a socket per
command, its local ephemeral port are leaked for good.
This is what happens with any module opening a fresh socket per operation
(e.g. rtpengine, which builds one UDP socket per async command): every lost
reply leaks one port, and over time the ephemeral port range
(net.ipv4.ip_local_port_range, shared between TCP and UDP) is exhausted -
after which every new connect() fails with EAGAIN and the platform appears
to be unable to reach any of its media servers.
The script async() path does not suffer from this, because the core passes
the script timeout to the module and registers the fd with
reactor_add_reader_with_timeout(); on expiry it calls the module's
timeout_f() (tm/async.c: "was_timeout ? ctx->async.timeout_f : ..."), which
lets the module release what it allocated.
Give launch() the same treatment:
* arm the timeout when the module provided both timeout_f and timeout_s.
launch has no script timeout, so the value comes from the module itself
via ctx->async.timeout_s - which async.h already documents as the
module's output parameter ("on output: an updated timeout, if any, as
processed by the module"). Modules which declare nothing keep today's
behaviour (no timeout);
* pass the reactor's was_timeout flag down to async_launch_resume() and
run timeout_f() instead of resume_f() on expiry, mirroring the script
async path. All five handle_io() dispatchers already have the flag at
hand (the F_SCRIPT_ASYNC case right above them forwards it);
* keep the timeout armed when a resume function switches the fd via
ASYNC_CHANGE_FD.
The launch() statement has no timeout parameter, so every launch()ed rtpengine command (a fresh UDP socket per command) stays in the reactor forever if its reply is lost - the fd and its local ephemeral port leak permanently. A production platform running launch(rtpengine_delete()) on call teardown exhausted net.ipv4.ip_local_port_range this way, after which every new UDP connect() failed with EAGAIN and all media servers started failing at once. Declare the module's own timeout to the core (async.h documents ctx->async.timeout_s as a module output parameter) so the per-command socket is reclaimed. The value is generous on purpose - it only ever fires on a lost reply - and a script-provided async() timeout still takes precedence when it is the smaller one. Configurable via rtpengine_async_timeout (default 5 seconds).
Author
|
Closing in favor of a follow-up PR containing only the clear-cut bug fixes (crash/heap-corruption class), split out from the behavioral changes discussed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
launch()operations have no timeout in the core, whileasync()ones do. For any module that opens a socket per operation, a reply which never arrives leaks the descriptor and its local ephemeral port for the lifetime of the process — because a zero timeout inio_waitnever expires.This fixes that, and is the root cause of a production outage we hit on a ~10-node rtpengine deployment: after weeks of
launch(rtpengine_delete())on call teardown,net.ipv4.ip_local_port_range(shared between TCP and UDP) was exhausted, every new UDPconnect()started failing withEAGAIN(11), and all rtpengine nodes appeared unreachable at once.Why it leaks
async_script_launch()registers the module's fd withreactor_add_reader():which expands to
io_watch_add(..., prio, 0, ...)— timeout = 0. And zero means "never expires":io_wait.h:478—if (timeout) timeout += get_ticks();io_wait_loop.h:114—e->timeout!=0 && e->timeout<=curr_timeThe script
async()path instead usesreactor_add_reader_with_timeout(...)and calls the module'stimeout_f()on expiry (tm/async.c:133), so its fds are always reclaimed. Neither is true forlaunch(), which also never invokestimeout_fat all.Changes
Core (commit 1)
timeout_fandtimeout_s. Sincelaunch()takes no timeout parameter, the value comes from the module itself throughctx->async.timeout_s— whichasync.halready documents as a module output parameter: "on output: an updated timeout, if any, as processed by the module". Modules declaring nothing keep today's behaviour (no timeout), so this cannot break existinglaunch()users.was_timeoutflag intoasync_launch_resume()and runtimeout_f()instead ofresume_f()on expiry, mirroring the script async path. All fivehandle_io()dispatchers already hold that flag — theF_SCRIPT_ASYNCcase directly above eachF_LAUNCH_ASYNCcase forwards it.ASYNC_CHANGE_FD.rtpengine (commit 2)
Declare a (modparam-configurable, default 5s) timeout so its per-command socket is reclaimed. The value only ever fires on a lost reply, and a script
async()timeout still takes precedence when it is smaller — so there is no behaviour change for the async path.Testing
The core change is additive: when no module declares a timeout, the registration call is the same one as before. The rtpengine part was built and run on a production 3.6.x deployment (as a local overlay) — see the commit message for the incident it resolves.
Note
For 3.6.x the same patch applies with only cosmetic differences (no
profilingcalls, noASYNC_SET_RESUME_F); happy to submit a backport to the stable branch if wanted.