Repository navigation
Conversation
|
AI-assisted review on behalf of @anmolnar Thanks @apurtell. I read the diff and the surrounding driver code and traced the lock order and the request/teardown interleavings; I did not build or run it locally. The change looks correct to me: the reconfig lock is never nested, no wakeup can be lost, and a stale disconnect request cannot hit a later connection because only zookeeper_interest() connects and it clears the flag first. +1 once CI is green, with a few comments:
Nits: update_addrs() wakes its own thread when called from zookeeper_interest(); wakeup_io_thread() is declared in both zk_adaptor.h and mt_adaptor.c; the commit subject still says "Forced closes leaks". |
…rupts session state Calling threads close the client socket directly during server reconfigurations or failed async writes, bypassing proper cleanup and producing invalid internal states. This triggers several race conditions: healthy connections are dropped when the wake-up pipe backs up, pending requests are routed to new servers prematurely, watchers are exposed to inconsistent states, and authentication failures are masked. The fix is to manage the lifecycle strictly in the I/O driver. Calling threads no longer close sockets or mutate client state themselves. Instead they set a disconnect request flag and signal the I/O thread. The driver handles the teardown deterministically, notifying watchers of a reconnecting state, cleanly failing in-flight requests with connection loss, and hardening the wake up path against pipe congestion and signal interruptions. Add Zookeeper_operations tests testServerListChangeFailsPendingRequests, testSendFailureDefersTeardown, testPingSendFailureTearsDown, testServerListChangeKeepsAuthFailedState (ST), testServerListChangeDuringSessionEstablishment and testFullWakeupPipeKeepsConnection (MT). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both changes add StatusCompletion and portOf() to TestOperations.cc. Put one identical copy after AsyncVoidOperationCompletion in both changes, so that they merge in either order without a conflict or a duplicate definition. Move the other new includes and helpers away from the lines that ZOOKEEPER-5119 changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o_set_servers - Set safe outputs at the start of zookeeper_interest(): fd -1, no interest, and a timeout of recv_timeout/60. An early error return then does not leave them uninitialized, and it does not cause a busy loop. - Document in zoo_set_servers() that, when the call must drop the connection, it asks the driver to do it. Until the driver acts, zoo_state() returns the current state. If the session was connected, the watcher then receives ZOO_CONNECTING_STATE. Outstanding requests complete with ZCONNECTIONLOSS. - Keep the "failed to send PING request" log line on a failed ping send, then do the teardown with handle_error(). - When a disconnect is pending, wake the IO thread from zoo_set_servers(), not from update_addrs(). The IO thread also calls update_addrs(), and it does not need to wake itself. - Remove the duplicate wakeup_io_thread() declaration from mt_adaptor.c. - Update the close_zsock() comment, because only the driver closes the session socket now. - testFullWakeupPipeKeepsConnection parks the IO thread in a poll() mock. It no longer holds the reconfig lock and sleeps. At DEBUG log level, zoo_aget() takes the reconfig lock, so the old test could deadlock. - Add testServerListChangeFailsBlockedSyncCall (MT): a synchronous zoo_get() that waits when zoo_set_servers() drops the server returns ZCONNECTIONLOSS. A poll() mock with a 60 second idle timeout makes the test fail if zoo_set_servers() does not wake the IO thread. The mock does not report the session socket as writable, so the close at the end of the test does not send a close request. - Declare SessionEvents before CloseFinally, so that the close sends its events to a live object. - Close the handle before the join in the MT test, so that a failed assertion cannot hang the test. TestJob::join() is now idempotent. This change and ZOOKEEPER-5119 add the same lines, so they merge in either order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1a139dd to
1c0aa08
Compare
|
Reply from Claude Code (an AI assistant), on behalf of @apurtell. @anmolnar thanks for the review. I rebased the branch onto current master and force-pushed it. Your points are addressed in addendum commit 1c0aa08. 1. Early return in 2. Docs. The 3. Ping send failure. The client logs "failed to send PING request" and then calls 4. Tests.
Nits. Tests run. C suites through the Maven |
anmolnar
left a comment
There was a problem hiding this comment.
AI-assisted review on behalf of @anmolnar
Thanks for the update, @apurtell. I re-read the branch at 1c0aa08 (not built or run locally). All points from my first pass are addressed:
- zookeeper_interest() now sets safe outputs before any early return.
- The zoo_set_servers() doc describes the deferred drop, the CONNECTING event and ZCONNECTIONLOSS for outstanding requests.
- The ping failure path keeps the plain log line and tears down with handle_error().
- The wakeup moved from update_addrs() to zoo_set_servers(), so the driver no longer wakes itself. update_addrs() has no other caller that needs one.
- testFullWakeupPipeKeepsConnection parks the IO thread in the poll mock, which removes both the DEBUG-level deadlock and the sleep.
- testServerListChangeFailsBlockedSyncCall covers the blocked sync call in the MT build.
It also merges cleanly with #2496 and #2494.
One optional follow-up, not introduced here: the receive-timeout branch of zookeeper_interest() still returns the closed descriptor in *fd after handle_socket_error_msg(). Setting *fd = -1 there, as the ping branch now does, would make the two consistent.
+1 from my side once CI is green for the ST and MT builds.
Calling threads close the client socket directly during server reconfigurations or failed async writes, bypassing proper cleanup and producing invalid internal states. This triggers several race conditions: healthy connections are dropped when the wake-up pipe backs up, pending requests are routed to new servers prematurely, watchers are exposed to inconsistent states, and authentication failures are masked.
The fix is to manage the lifecycle strictly in the I/O driver. Calling threads no longer close sockets or mutate client state themselves. Instead they set a disconnect request flag and signal the I/O thread. The driver handles the teardown deterministically, notifying watchers of a reconnecting state, cleanly failing in-flight requests with connection loss, and hardening the wake up path against pipe congestion and signal interruptions.
Add Zookeeper_operations tests testServerListChangeFailsPendingRequests, testSendFailureDefersTeardown, testPingSendFailureTearsDown, testServerListChangeKeepsAuthFailedState (ST),
testServerListChangeDuringSessionEstablishment, testFullWakeupPipeKeepsConnection and testServerListChangeFailsBlockedSyncCall (MT).
Release note: When
zoo_set_servers()moves the client to another server, it asks the I/O driver to drop the connection. Until the driver acts,zoo_state()reports the current state. If the session was connected, the watcher then receivesZOO_CONNECTING_STATE. Outstanding requests complete withZCONNECTIONLOSS.