Skip to content

Auto-cycling of WebSocket connections - with new pre-disconnect hook - #247

Merged
onelapahead merged 3 commits into
mainfrom
webhook-cycle
Sep 11, 2026
Merged

Auto-cycling of WebSocket connections - with new pre-disconnect hook#247
onelapahead merged 3 commits into
mainfrom
webhook-cycle

Conversation

@peterbroadhurst

@peterbroadhurst peterbroadhurst commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

WebSockets are a long-lived connection to a single server instance, often routed through multiple complex network devices/proxies.

Leaving a single connection active indefinitely has some real-world challenges:

  1. Network devices (including k8s ingress servers) commonly have a max lifetime on connections in minutes
  2. If there are a pool of servers, no rebalancing ever happens

This PR proposes a new feature usable by all users of the wsclient client, which provides managed cycling of websockets on a regular interval.

The rotation cycles is as follows:

  1. The new connection is established
    • No cleanup activities start until we are have a new connection open/dialed
  2. A pre-disconnect handler (new hook) is called on the old connection
    • This allows cleanup of subscriptions etc. on the old connection
  3. The existing post-connect handler is called on the new connection
    • Moving subscriptions etc. to the new connection
  4. New send activity is routed to the new connection
    • Note that receives are processed from both connections in this window
  5. The old connection is left open for a configurable quiesce window
    • This guards against failure of in-flight request/response (JSON/RPC etc.) calls during the switchover
  6. The old connection is closed
    • The cycle timer is set only at the end of this process (note theres's never more than two connections)

Note while the change count seems large, the majority of the change is the moving of logic out of wsclient.go into wsconnection.go, rather than a fundamental change to the loops and heartbeat processing.

@peterbroadhurst
peterbroadhurst requested a review from a team as a code owner August 31, 2026 13:19

@onelapahead onelapahead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really enjoyed this one, hard to see a 🐛 from here, just had small code nits and tried my best to read it all.

The wsConnection abstraction is great and very useful in a future world of client-side load balancing/high availability.

Think like ffresty we need a EnableClientMetrics and some counters for all these potential failures / state transitions and timers on any sends. But I'll contribute that separately.

Test server seems correct in how to test such a complex mode of operation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this warrant a separate test file ?

Comment thread pkg/wsclient/wsclient.go

// New creates a new outbound client that can be connected to a remote server.
// ** Recommend using NewWithConfig directly **
func New(ctx context.Context, config *WSConfig, beforeConnect WSPreConnectHandler, afterConnect WSPostConnectHandler) (WSClient, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: not important for this PR, but using the example to hightlight that variadic functional options here would have allowed the addition of a WSPreDisconnectHandler to New without needing a separate func: https://github.com/uber-go/guide/blob/master/style.md#functional-options.

So unlike Java, in a lot of our libraries I'm trying to make sure we avoid so many constructors/funcs when one can do with lots of options.

Comment thread pkg/wsclient/wsclient.go
Comment on lines +260 to +261
close(c.promoted) // sole connection - immediately the consumer of the shared send channel
w.current = c

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is likely fine - but mutex ?

isDemoted = true
case <-timeoutContext.Done():
if err := c.heartbeatCheck(); err != nil {
l.Errorf("WS %s closing: %s", w.url, err)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warn I believe

Comment thread pkg/wsclient/wsclient.go
// Start reading, and atomically switch all new sends over to the new connection
newC.startReader()
if !w.promoteConnection(old, newC) {
// The client was closed while we were cycling - clean up the orphaned new connection

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be good to have a WARN log here ? I think that will be one of many if the client was closed while we did this, but just so we know it closed in the middle of a cycle/promotion

Base automatically changed from fix-webhook-close to main September 1, 2026 09:19
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
Signed-off-by: Peter Broadhurst <peter.broadhurst@kaleido.io>
@onelapahead
onelapahead merged commit 11b0d0e into main Sep 11, 2026
3 checks passed
@onelapahead
onelapahead deleted the webhook-cycle branch September 11, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants