Repository navigation
Refuse new connections as soon as a server stops. - #377
Open
jakeonfire wants to merge 1 commit into
Open
jakeonfire wants to merge 1 commit into
jakeonfire wants to merge 1 commit into
Conversation
A forked or hybrid worker holds its own descriptor for the listening socket, so closing the parent's copy in `Service::Server#stop` left the socket listening until the last worker exited. Connections that arrived while the workers drained were queued by the kernel and then reset. When the workers are in other processes, `#stop` now shuts the socket down before closing it. For a TCP socket on Linux that applies to every process that shares it, so new connections are refused at once and requests in flight still finish. UNIX sockets, and other platforms, behave as before. So do threaded containers, where closing the only descriptor already stops the socket listening. A worker's accept loop sees the shutdown as `EINVAL`, so the bound endpoint now waits to be stopped instead of failing the task. Fixes socketry#369. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #369.
When a server stops, its workers finish the requests they have, but the listening socket stays open until the last worker exits. Connections that arrive in the meantime are queued by the kernel and then reset. With this change they are refused at once, and requests in flight still finish.
Problem
Falcon::Service::Server#stopcloses the listener in the parent. In a forked or hybrid container each worker process holds its own descriptor for the same socket, so that close doesn't stop it listening. Each worker stops accepting when it's interrupted, and the kernel keeps completing handshakes for a socket that no one will accept from again.Fix
When the workers are in other processes,
#stopnow callsshutdown(SHUT_RD)on the listening sockets before it closes them (Falcon::Listener#shutdown). For a TCP socket on Linux this takes the socket out of the listening state for every process that shares it. Accepted connections are separate sockets, so they aren't affected.A worker whose accept loop is waiting on the socket sees the shutdown as
EINVALfromaccept. The bound endpoint is now a small subclass,Falcon::Listener::Endpoint, whose accept loop waits to be stopped once its socket is no longer listening, exactly as it waits while no connections arrive. An error from a socket that is still listening is raised as before.The stop runs in the parent and only when the whole service stops. A restart (
SIGHUP) creates a new container that shares the listener and never callsservice.stop, so it isn't affected: 30 of 30 requests across a restart succeeded.The shutdown is deliberately narrow:
acceptreturnsEAGAIN, so an accept loop on it spins. UNIX listeners are left alone.Why not close the listener in each worker
That's what I suggested in the issue. It turned out to have three problems:
The hang also only happens when no worker will accept again, which is a whole-service stop. If a single worker stops, the others keep accepting from the same queue. So the parent is the one place that knows when to stop listening, and it's where the existing
closealready tried to.This is separate from #368 and socketry/async#478. Those are about telling a running response that a stop has begun, which needs a signal inside the worker. Here nothing in the worker needs to be told.
Reproduction, before and after
The reproduction from the issue: two slow requests in flight,
SIGTERM, then a new connection every 250 ms. Linux arm64, Ruby 4.0.7, async 2.46.0, async-container 0.38.0, async-http 0.105.0, io-endpoint 0.19.0, io-event 1.22.3.falcon-host)Both in-flight requests completed in every run, and the workers logged no task failures in the forked and hybrid runs.
Forked, before:
Forked, after:
Limits
Falcon::Environment::Applicationbinds.Falcon::Service::Clusteris not covered. Each cluster worker binds its own socket and closes it when it exits, so a connection to a worker that is draining still waits. I left it out to keep this change small.Not verified
bake decode:index:coveragelocally (thedecodegem fails on this Ruby's Prism), so documentation coverage is unchecked.Types of Changes
Contribution
AI disclaimer: I worked on this with Claude.
🤖 Generated with Claude Code