Skip to content

Refuse new connections as soon as a server stops. - #377

Open
jakeonfire wants to merge 1 commit into
socketry:mainfrom
jakeonfire:stop-listening-on-stop
Open

jakeonfire wants to merge 1 commit into
socketry:mainfrom
jakeonfire:stop-listening-on-stop

Conversation

@jakeonfire

Copy link
Copy Markdown

Fixes #369.

When a server stops, its workers finish the requests they have, but the listening socket stays open until the last worker exits. Connections that arrive in the meantime are queued by the kernel and then reset. With this change they are refused at once, and requests in flight still finish.

Problem

Falcon::Service::Server#stop closes the listener in the parent. In a forked or hybrid container each worker process holds its own descriptor for the same socket, so that close doesn't stop it listening. Each worker stops accepting when it's interrupted, and the kernel keeps completing handshakes for a socket that no one will accept from again.

Fix

When the workers are in other processes, #stop now calls shutdown(SHUT_RD) on the listening sockets before it closes them (Falcon::Listener#shutdown). For a TCP socket on Linux this takes the socket out of the listening state for every process that shares it. Accepted connections are separate sockets, so they aren't affected.

A worker whose accept loop is waiting on the socket sees the shutdown as EINVAL from accept. The bound endpoint is now a small subclass, Falcon::Listener::Endpoint, whose accept loop waits to be stopped once its socket is no longer listening, exactly as it waits while no connections arrive. An error from a socket that is still listening is raised as before.

The stop runs in the parent and only when the whole service stops. A restart (SIGHUP) creates a new container that shares the listener and never calls service.stop, so it isn't affected: 30 of 30 requests across a restart succeeded.

The shutdown is deliberately narrow:

  • TCP only. A UNIX socket that is shut down refuses connections but stays readable, and accept returns EAGAIN, so an accept loop on it spins. UNIX listeners are left alone.
  • Multi-process containers only. In a threaded container, closing the only descriptor already stops the socket listening. Shutting it down first wakes every accept loop just before the parent closes the socket they share, and on Ruby 3.3 that close then blocked in most runs of a test I wrote for it.
  • Linux only. Other platforms don't stop a listening socket when it's shut down.

Why not close the listener in each worker

That's what I suggested in the issue. It turned out to have three problems:

  • It doesn't refuse until every worker has acted on its interrupt. A worker that is busy holds its copy, and the socket keeps listening. With one of two workers stuck in a 3 second CPU-bound request, 9 of 10 late connections were still reset. With this change all 10 were refused.
  • It can make a draining worker spin. With io-event's epoll selector, a cancelled accept leaves its registration in place until the next event. If the worker then closes its descriptor while another process still holds the socket, the kernel can't drop that registration, and it fires for as long as a connection is queued. In the same scenario a worker that was asleep in a request used 2.25 seconds of CPU, against 0.01 seconds without the workaround.
  • Threads share one descriptor. In a hybrid container the threads of a fork share the fork's copy. A thread can't close it without affecting the others, and it can't tell a graceful stop from a restart of that one thread without inspecting the cause of its cancellation.

The hang also only happens when no worker will accept again, which is a whole-service stop. If a single worker stops, the others keep accepting from the same queue. So the parent is the one place that knows when to stop listening, and it's where the existing close already tried to.

This is separate from #368 and socketry/async#478. Those are about telling a running response that a stop has begun, which needs a signal inside the worker. Here nothing in the worker needs to be told.

Reproduction, before and after

The reproduction from the issue: two slow requests in flight, SIGTERM, then a new connection every 250 ms. Linux arm64, Ruby 4.0.7, async 2.46.0, async-container 0.38.0, async-http 0.105.0, io-endpoint 0.19.0, io-event 1.22.3.

Container Before (0.57.0) After
forked (falcon-host) 17 reset after up to 4.3 s, 3 refused 20 refused at once
hybrid (2 forks, 2 threads) 17 reset after up to 4.3 s, 3 refused 20 refused at once
threaded 20 refused at once unchanged

Both in-flight requests completed in every run, and the workers logged no task failures in the forked and hybrid runs.

Forked, before:

in-flight /slow requests started before SIGTERM:
  200 OK                 after 5.01s
  200 OK                 after 5.01s
new requests sent after SIGTERM (t = seconds after SIGTERM):
  t=0.25s  ECONNRESET             after 4.26s
  t=0.51s  ECONNRESET             after 4.00s
  ...
  t=4.27s  ECONNRESET             after 0.24s
  t=4.52s  ECONNREFUSED           after 0.00s

Forked, after:

in-flight /slow requests started before SIGTERM:
  200 OK                 after 5.01s
  200 OK                 after 5.01s
new requests sent after SIGTERM (t = seconds after SIGTERM):
  t=0.25s  ECONNREFUSED           after 0.00s
  t=0.51s  ECONNREFUSED           after 0.00s
  ...
  t=5.06s  ECONNREFUSED           after 0.00s

Limits

  • TCP on Linux only, as described above. macOS has the same underlying behavior as Linux (connections queue while any process holds the socket), so it is unchanged by this PR, not fixed. The tests that need the refusal are skipped off Linux.
  • UNIX listeners are unchanged. That includes the IPC sockets that Falcon::Environment::Application binds.
  • The backlog at the moment of the stop is reset. Connections that completed their handshake but weren't accepted yet are reset when the socket shuts down. Before, a worker might still have accepted one in the instant before its interrupt arrived. That window is a few milliseconds, where the hang used to last the whole drain.
  • Falcon::Service::Cluster is not covered. Each cluster worker binds its own socket and closes it when it exits, so a connection to a worker that is draining still waits. I left it out to keep this change small.
  • A socket shared with a process outside the service stops listening there too. That's the intent for forked workers, but it would also apply to a descriptor that was handed in from outside and is expected to outlive the service.

Not verified

  • I ran the suite on Linux only: all of it on Ruby 4.0.7, and the changed test files repeatedly on Ruby 3.3 and 3.4. I haven't run it on macOS.
  • I couldn't run bake decode:index:coverage locally (the decode gem fails on this Ruby's Prism), so documentation coverage is unchecked.

Types of Changes

  • Bug fix.

Contribution

AI disclaimer: I worked on this with Claude.

🤖 Generated with Claude Code

A forked or hybrid worker holds its own descriptor for the listening
socket, so closing the parent's copy in `Service::Server#stop` left the
socket listening until the last worker exited. Connections that arrived
while the workers drained were queued by the kernel and then reset.

When the workers are in other processes, `#stop` now shuts the socket
down before closing it. For a TCP socket on Linux that applies to every
process that shares it, so new connections are refused at once and
requests in flight still finish. UNIX sockets, and other platforms,
behave as before. So do threaded containers, where closing the only
descriptor already stops the socket listening.

A worker's accept loop sees the shutdown as `EINVAL`, so the bound
endpoint now waits to be stopped instead of failing the task.

Fixes socketry#369.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workers keep their inherited listening socket open during a graceful stop, so new connections hang and then reset

1 participant