Skip to content

ZOOKEEPER-5120: (C client) Forced close leaks request queues and corrupts session state - #2495

Open
apurtell wants to merge 3 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5120
Open

apurtell wants to merge 3 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5120

Conversation

@apurtell

@apurtell apurtell commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Calling threads close the client socket directly during server reconfigurations or failed async writes, bypassing proper cleanup and producing invalid internal states. This triggers several race conditions: healthy connections are dropped when the wake-up pipe backs up, pending requests are routed to new servers prematurely, watchers are exposed to inconsistent states, and authentication failures are masked.

The fix is to manage the lifecycle strictly in the I/O driver. Calling threads no longer close sockets or mutate client state themselves. Instead they set a disconnect request flag and signal the I/O thread. The driver handles the teardown deterministically, notifying watchers of a reconnecting state, cleanly failing in-flight requests with connection loss, and hardening the wake up path against pipe congestion and signal interruptions.

Add Zookeeper_operations tests testServerListChangeFailsPendingRequests, testSendFailureDefersTeardown, testPingSendFailureTearsDown, testServerListChangeKeepsAuthFailedState (ST),
testServerListChangeDuringSessionEstablishment, testFullWakeupPipeKeepsConnection and testServerListChangeFailsBlockedSyncCall (MT).

Release note: When zoo_set_servers() moves the client to another server, it asks the I/O driver to drop the connection. Until the driver acts, zoo_state() reports the current state. If the session was connected, the watcher then receives ZOO_CONNECTING_STATE. Outstanding requests complete with ZCONNECTIONLOSS.

@apurtell apurtell changed the title ZOOKEEPER-5120: (C client) Forced closes leaks request queues and corrupts session state ZOOKEEPER-5120: (C client) Forced close leaks request queues and corrupts session state Nov 6, 2026
@anmolnar

Copy link
Copy Markdown
Contributor

AI-assisted review on behalf of @anmolnar

Thanks @apurtell. I read the diff and the surrounding driver code and traced the lock order and the request/teardown interleavings; I did not build or run it locally. The change looks correct to me: the reconfig lock is never nested, no wakeup can be lost, and a stale disconnect request cannot hit a later connection because only zookeeper_interest() connects and it clears the flag first. +1 once CI is green, with a few comments:

  1. zookeeper_interest(): the new teardown runs before if (rc != ZOK) return. If update_addrs() fails on the same call, the socket is closed and we return without setting *fd/*interest/*tv; do_io() ignores the return code. Master has the same hole and ZOOKEEPER-5121: (C client) DNS lookup failures can disrupt the client #2496 fixes it, so either merge ZOOKEEPER-5121: (C client) DNS lookup failures can disrupt the client #2496 first or set the outputs before this return.

  2. Please add a line to the zoo_set_servers() doc and the release note: zoo_state() now stays CONNECTED until the driver runs, watchers get CONNECTING, and outstanding requests complete with ZCONNECTIONLOSS.

  3. In the THREADED build send_ping() can now only fail on allocation/serialization, but the failure is logged through handle_socket_error_msg() as a socket error with an unrelated errno. A plain LOG_ERROR plus handle_error() would be clearer.

  4. Tests:

    • testFullWakeupPipeKeepsConnection holds the reconfig lock across zoo_aget(). At DEBUG log level zoo_aget() calls zoo_get_current_server(), which takes the same non-recursive mutex, so the test deadlocks. Releasing the lock after the wakeup_io_thread() assertion avoids that.
    • The millisleep(200) can let that test pass without reaching the EAGAIN path.
    • Could we have an MT test where a blocked zoo_get() returns ZCONNECTIONLOSS after zoo_set_servers() drops the server?

Nits: update_addrs() wakes its own thread when called from zookeeper_interest(); wakeup_io_thread() is declared in both zk_adaptor.h and mt_adaptor.c; the commit subject still says "Forced closes leaks".

Andrew Purtell and others added 3 commits November 19, 2026 16:18
…rupts session state

Calling threads close the client socket directly during server
reconfigurations or failed async writes, bypassing proper cleanup and
producing invalid internal states. This triggers several race conditions:
healthy connections are dropped when the wake-up pipe backs up, pending
requests are routed to new servers prematurely, watchers are exposed to
inconsistent states, and authentication failures are masked.

The fix is to manage the lifecycle strictly in the I/O driver. Calling
threads no longer close sockets or mutate client state themselves. Instead
they set a disconnect request flag and signal the I/O thread. The driver
handles the teardown deterministically, notifying watchers of a
reconnecting state, cleanly failing in-flight requests with connection
loss, and hardening the wake up path against pipe congestion and signal
interruptions.

Add Zookeeper_operations tests testServerListChangeFailsPendingRequests,
testSendFailureDefersTeardown, testPingSendFailureTearsDown,
testServerListChangeKeepsAuthFailedState (ST),
testServerListChangeDuringSessionEstablishment and
testFullWakeupPipeKeepsConnection (MT).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both changes add StatusCompletion and portOf() to TestOperations.cc.
Put one identical copy after AsyncVoidOperationCompletion in both
changes, so that they merge in either order without a conflict or a
duplicate definition. Move the other new includes and helpers away from
the lines that ZOOKEEPER-5119 changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o_set_servers

- Set safe outputs at the start of zookeeper_interest(): fd -1, no
  interest, and a timeout of recv_timeout/60. An early error return then
  does not leave them uninitialized, and it does not cause a busy loop.
- Document in zoo_set_servers() that, when the call must drop the
  connection, it asks the driver to do it. Until the driver acts,
  zoo_state() returns the current state. If the session was connected,
  the watcher then receives ZOO_CONNECTING_STATE. Outstanding requests
  complete with ZCONNECTIONLOSS.
- Keep the "failed to send PING request" log line on a failed ping
  send, then do the teardown with handle_error().
- When a disconnect is pending, wake the IO thread from
  zoo_set_servers(), not from update_addrs(). The IO thread also calls
  update_addrs(), and it does not need to wake itself.
- Remove the duplicate wakeup_io_thread() declaration from mt_adaptor.c.
- Update the close_zsock() comment, because only the driver closes the
  session socket now.
- testFullWakeupPipeKeepsConnection parks the IO thread in a poll() mock.
  It no longer holds the reconfig lock and sleeps. At DEBUG log level,
  zoo_aget() takes the reconfig lock, so the old test could deadlock.
- Add testServerListChangeFailsBlockedSyncCall (MT): a synchronous
  zoo_get() that waits when zoo_set_servers() drops the server returns
  ZCONNECTIONLOSS. A poll() mock with a 60 second idle timeout makes the
  test fail if zoo_set_servers() does not wake the IO thread. The mock
  does not report the session socket as writable, so the close at the
  end of the test does not send a close request.
- Declare SessionEvents before CloseFinally, so that the close sends its
  events to a live object.
- Close the handle before the join in the MT test, so that a failed
  assertion cannot hang the test. TestJob::join() is now idempotent.
  This change and ZOOKEEPER-5119 add the same lines, so they merge in
  either order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@anmolnar thanks for the review. I rebased the branch onto current master and force-pushed it. Your points are addressed in addendum commit 1c0aa08.

1. Early return in zookeeper_interest(). The function now sets safe outputs (no descriptor, no interest, a short timeout) before any early return. The hunk is byte-identical to the one in #2496, so the two PRs merge in either order.

2. Docs. The zoo_set_servers() doc now says that when the call must drop the connection, it asks the driver to do it. Until the driver acts, zoo_state() reports the current state. If the session was connected, the watcher then gets ZOO_CONNECTING_STATE. Outstanding requests complete with ZCONNECTIONLOSS. The description has the matching release-note line.

3. Ping send failure. The client logs "failed to send PING request" and then calls handle_error(). It no longer reports a socket error with an unrelated errno.

4. Tests.

  • testFullWakeupPipeKeepsConnection no longer holds the reconfig lock across zoo_aget() or sleeps. It parks the IO thread in a poll() mock, so the test always reaches the full-pipe path. It also no longer deadlocks at DEBUG.
  • New MT test testServerListChangeFailsBlockedSyncCall: a blocked zoo_get() returns ZCONNECTIONLOSS when zoo_set_servers() drops the server. Its poll() mock idles for 60 s, so the test fails if zoo_set_servers() does not wake the driver. I checked this by removing the wakeup.
  • The new tests now declare SessionEvents before CloseFinally, so the close cannot use a freed watcher context. The new MT test closes the handle before the join, so a failed assertion cannot hang the binary.

Nits. zoo_set_servers() now wakes the driver when it drops the connection, and update_addrs() no longer does, so the driver does not wake itself. The duplicate wakeup_io_thread() declaration is gone. A squash merge takes its subject from the PR title, which is already correct.

Tests run. C suites through the Maven full-build profile on Linux x86_64 (RHEL 9, JDK 17, OpenSSL 3.5). ST OK (59). MT 105 run, 3 failures: testSSL, testSSLServerOnly and testReadOnlyWithSSL (ZSSLCONNECTIONERROR). They are not related to this PR, and they fail the same way on master. With OpenSSL 3.2 and later, the test certificates do not verify. ZOOKEEPER-5135 (#2510) fixes the certificates, and it can merge first. With #2510 applied, every test passes, both on master (ST OK (55), MT OK (102)) and with all three PRs merged (ST OK (75), MT OK (106)).

@apurtell

Copy link
Copy Markdown
Contributor Author

I had Claude help more directly so any of the the PRs in the set #2494, #2495, and #2496, each currently under review, can be applied in any order and in any order each of these PRs will land without conflicts and with good test results.

@anmolnar anmolnar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review on behalf of @anmolnar

Thanks for the update, @apurtell. I re-read the branch at 1c0aa08 (not built or run locally). All points from my first pass are addressed:

  • zookeeper_interest() now sets safe outputs before any early return.
  • The zoo_set_servers() doc describes the deferred drop, the CONNECTING event and ZCONNECTIONLOSS for outstanding requests.
  • The ping failure path keeps the plain log line and tears down with handle_error().
  • The wakeup moved from update_addrs() to zoo_set_servers(), so the driver no longer wakes itself. update_addrs() has no other caller that needs one.
  • testFullWakeupPipeKeepsConnection parks the IO thread in the poll mock, which removes both the DEBUG-level deadlock and the sleep.
  • testServerListChangeFailsBlockedSyncCall covers the blocked sync call in the MT build.

It also merges cleanly with #2496 and #2494.

One optional follow-up, not introduced here: the receive-timeout branch of zookeeper_interest() still returns the closed descriptor in *fd after handle_socket_error_msg(). Setting *fd = -1 there, as the ping branch now does, would make the two consistent.

+1 from my side once CI is green for the ST and MT builds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants