Skip to content

core: add timeout support to launch() async operations - #4260

Closed
zbyslb wants to merge 2 commits into
OpenSIPS:masterfrom
zbyslb:core-launch-timeout
Closed

zbyslb wants to merge 2 commits into
OpenSIPS:masterfrom
zbyslb:core-launch-timeout

Conversation

@zbyslb

@zbyslb zbyslb commented Sep 14, 2026

Copy link
Copy Markdown

Summary

launch() operations have no timeout in the core, while async() ones do. For any module that opens a socket per operation, a reply which never arrives leaks the descriptor and its local ephemeral port for the lifetime of the process — because a zero timeout in io_wait never expires.

This fixes that, and is the root cause of a production outage we hit on a ~10-node rtpengine deployment: after weeks of launch(rtpengine_delete()) on call teardown, net.ipv4.ip_local_port_range (shared between TCP and UDP) was exhausted, every new UDP connect() started failing with EAGAIN(11), and all rtpengine nodes appeared unreachable at once.

Why it leaks

async_script_launch() registers the module's fd with reactor_add_reader():

if (reactor_add_reader(fd,F_LAUNCH_ASYNC,RCT_PRIO_ASYNC,(void*)ctx)<0){

which expands to io_watch_add(..., prio, 0, ...) — timeout = 0. And zero means "never expires":

  • io_wait.h:478if (timeout) timeout += get_ticks();
  • io_wait_loop.h:114e->timeout!=0 && e->timeout<=curr_time

The script async() path instead uses reactor_add_reader_with_timeout(...) and calls the module's timeout_f() on expiry (tm/async.c:133), so its fds are always reclaimed. Neither is true for launch(), which also never invokes timeout_f at all.

Changes

Core (commit 1)

  • Arm the timeout when the module declared both timeout_f and timeout_s. Since launch() takes no timeout parameter, the value comes from the module itself through ctx->async.timeout_s — which async.h already documents as a module output parameter: "on output: an updated timeout, if any, as processed by the module". Modules declaring nothing keep today's behaviour (no timeout), so this cannot break existing launch() users.
  • Thread the reactor's was_timeout flag into async_launch_resume() and run timeout_f() instead of resume_f() on expiry, mirroring the script async path. All five handle_io() dispatchers already hold that flag — the F_SCRIPT_ASYNC case directly above each F_LAUNCH_ASYNC case forwards it.
  • Keep the timeout armed when a resume function swaps the fd through ASYNC_CHANGE_FD.

rtpengine (commit 2)

Declare a (modparam-configurable, default 5s) timeout so its per-command socket is reclaimed. The value only ever fires on a lost reply, and a script async() timeout still takes precedence when it is smaller — so there is no behaviour change for the async path.

Testing

The core change is additive: when no module declares a timeout, the registration call is the same one as before. The rtpengine part was built and run on a production 3.6.x deployment (as a local overlay) — see the commit message for the incident it resolves.

Note

For 3.6.x the same patch applies with only cosmetic differences (no profiling calls, no ASYNC_SET_RESUME_F); happy to submit a backport to the stable branch if wanted.

The launch() statement has no timeout parameter, and async_script_launch()
registers the returned fd with reactor_add_reader() - i.e. with a timeout of
zero. In io_wait, a zero timeout never expires
(io_wait.h: "if (timeout) timeout += get_ticks()",
io_wait_loop.h: "e->timeout!=0 && e->timeout<=curr_time"), so when the reply
of a launch operation never arrives, the fd stays in the reactor for the
lifetime of the process: the descriptor and, for a module using a socket per
command, its local ephemeral port are leaked for good.

This is what happens with any module opening a fresh socket per operation
(e.g. rtpengine, which builds one UDP socket per async command): every lost
reply leaks one port, and over time the ephemeral port range
(net.ipv4.ip_local_port_range, shared between TCP and UDP) is exhausted -
after which every new connect() fails with EAGAIN and the platform appears
to be unable to reach any of its media servers.

The script async() path does not suffer from this, because the core passes
the script timeout to the module and registers the fd with
reactor_add_reader_with_timeout(); on expiry it calls the module's
timeout_f() (tm/async.c: "was_timeout ? ctx->async.timeout_f : ..."), which
lets the module release what it allocated.

Give launch() the same treatment:

  * arm the timeout when the module provided both timeout_f and timeout_s.
    launch has no script timeout, so the value comes from the module itself
    via ctx->async.timeout_s - which async.h already documents as the
    module's output parameter ("on output: an updated timeout, if any, as
    processed by the module"). Modules which declare nothing keep today's
    behaviour (no timeout);
  * pass the reactor's was_timeout flag down to async_launch_resume() and
    run timeout_f() instead of resume_f() on expiry, mirroring the script
    async path. All five handle_io() dispatchers already have the flag at
    hand (the F_SCRIPT_ASYNC case right above them forwards it);
  * keep the timeout armed when a resume function switches the fd via
    ASYNC_CHANGE_FD.
The launch() statement has no timeout parameter, so every launch()ed
rtpengine command (a fresh UDP socket per command) stays in the reactor
forever if its reply is lost - the fd and its local ephemeral port leak
permanently. A production platform running launch(rtpengine_delete()) on
call teardown exhausted net.ipv4.ip_local_port_range this way, after which
every new UDP connect() failed with EAGAIN and all media servers started
failing at once.

Declare the module's own timeout to the core (async.h documents
ctx->async.timeout_s as a module output parameter) so the per-command
socket is reclaimed. The value is generous on purpose - it only ever fires
on a lost reply - and a script-provided async() timeout still takes
precedence when it is the smaller one.

Configurable via rtpengine_async_timeout (default 5 seconds).
@zbyslb

zbyslb commented Sep 15, 2026

Copy link
Copy Markdown
Author

Closing in favor of a follow-up PR containing only the clear-cut bug fixes (crash/heap-corruption class), split out from the behavioral changes discussed here.

@zbyslb zbyslb closed this Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants