Skip to content

ZOOKEEPER-5119: (C client) Switching from RO to RW servers corrupts state - #2494

Open
apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5119
Open

apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5119

Conversation

@apurtell

@apurtell apurtell commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

When a read-only C client detects an available read-write server, it drops its existing read-only connection to upgrade the session. The C client's implementation performs unsafe request teardown, returning a closed descriptor to the event loop and sharing global state across client instances.

The fix is to treat the RO to RW upgrade as a reconnect, handling it consistently as such, isolating all state to the individual affected handle. This change maintains full API and ABI compatibility. Applications with requests in flight during a server upgrade will observe expected connection-loss behavior after the fix, instead of crashes or silent data corruption.

Release note: requests that are pending when a read-only session switches to a read/write server now complete with ZCONNECTIONLOSS.

@anmolnar

Copy link
Copy Markdown
Contributor

AI-assisted review on behalf of @anmolnar

Thanks for this. The core fix looks correct to me, and the problems on master are real:

  • cleanup(zh, ZOK) completes every pending request with a header-only reply and err = 0. Sync callers wake with ZOK and no result, and a pending multi goes to deserialize_multi with a zeroed MultiHeader, which runs off the op list into assert(entry).
  • After the switch, zookeeper_interest fell through and returned the closed fd with ZOOKEEPER_READ.
  • The file-static addr_rw_server could be consumed by any other handle in the process, dangled after its owner was closed, and was never cleared on a server list change.

Failing the requests with ZCONNECTIONLOSS, returning fd = -1 / no interest / zero timeout, and moving the hint into the handle all look right. This was reviewed by reading the code and tracing the new test by hand against the mocks; I have not built or run it.

A few things I'd like to see addressed:

1. The ping-failure hunk has no effect and overlaps with #2495 (zookeeper.c, *fd = zh->fd->sock after send_ping fails)

The comment says a failed send can close the socket, but nothing on the send_ping → adaptor_send_queue → flush_send_queue → send_buffer path closes it on this branch, so *fd is assigned the value it already holds. The actual problem on this path (error returned with the socket open and a zero timeout) is what #2495 fixes, by rewriting these same lines with a full teardown. I'd suggest dropping this hunk and the matching sentence in the commit message, and leaving the path to #2495.

2. Merge order with #2495

The two PRs conflict in three places: the hunk above, the comment block in update_addrs next to has_addr_rw_server = 0, and the CPPUNIT_TEST list in TestOperations.cc. The last two are keep-both. Whichever lands second will need a rebase.

3. ping_rw_server leaks the probe socket when connect fails (pre-existing)

rc = zookeeper_connect(...); if (rc < 0) return 0; returns without closing fd.sock. A read-only session usually means the other servers are unreachable, so failed probes are the common case and each one leaks a descriptor. Not introduced here, but since this PR is about this path and already edits the function, would you be willing to fix it here? Note that a plain goto out is not enough in the OpenSSL build: fd.ssl_sock and fd.ssl_ctx are initialised only after the connect, and close_zsock reads them.

4. The hint is not covered by the reconfig lock (minor)

update_addrs clears has_addr_rw_server under the reconfig lock, possibly on an application thread via zoo_set_servers. ping_rw_server and the connect block touch the flag and addr_rw_server on the IO thread without it. If the list changes between the probe's addrvec_peek and its has_addr_rw_server = rc, the clear is lost and the next connect goes to a server from the old list. The unlocked addrvec_peek(&zh->addrs, ...) against addrvec_free(&zh->addrs) is the older half of the same gap. Taking the reconfig lock around the peek and flag update, and around the consume in zookeeper_interest, would close both.

5. Tests

  • CPPUNIT_ASSERT_EQUAL(portOf(rwServer), portOf(zh->addr_cur)) cannot fail as written: the probe only peeks at the cursor, so without the hint zoo_cycle_next_server returns the same entry. Advancing the cursor before the reconnect would make it meaningful. The has_addr_rw_server == 0 check and the second-handle check (port 2181) are the ones doing the work, and the latter is a good test of the isolation fix.
  • It would be good to also cover: the hint being dropped by a server list change, fd == -1 after a receive timeout, and a failed probe (no switch, cursor advances). A THREADED case where a blocked sync call returns ZCONNECTIONLOSS would be a nice addition.

Nits

  • Requests pending during an RO to RW upgrade now complete with ZCONNECTIONLOSS. That is correct and consistent with other disconnects, but worth a line in the release note.
  • The description mentions leaking event-loop descriptors; what the diff fixes is returning a closed descriptor to the event loop.
  • zh->has_addr_rw_server = rc depends on rc being 0/1 at that point; rc != 0 would be more robust.

Andrew Purtell and others added 2 commits November 19, 2026 16:18
…tate

When a read-only session finds a read/write server, zookeeper_interest()
now handles the switch as a normal reconnect:

- Fail pending requests with ZCONNECTIONLOSS instead of a faked ZOK reply.
- Return fd -1, no interest, and a zero timeout after the switch, so the
  caller does not poll the closed socket and reconnects at once. Also
  return fd -1 after a receive timeout, and the current socket after a
  failed ping send.
- Keep the read/write server hint in the handle (has_addr_rw_server)
  instead of a file-static pointer that any handle could consume. Clear
  the hint when the server list changes.

Add test Zookeeper_operations::testReadOnlyToReadWriteTransition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ocket

- Drop the ping-failure change that returned zh->fd->sock. The socket
  is still open at that point, so the change had no effect. The original
  commit message is wrong on this point.
- Close the probe socket when the probe connect fails. Set the SSL
  fields before the first exit, because close_zsock() reads them.
- Read and update the server list cursor and the read/write server hint
  under the reconfig lock. The probe itself runs without the lock,
  because it can block. A version counter on the server list makes the
  probe drop its result when zoo_set_servers() changes the list during
  the probe.
- Store a 0/1 value in has_addr_rw_server.
- Add tests: a hint dropped by a server list change, a probe made stale
  by a server list change, a probe that finds another read-only server,
  a failed probe connect, a receive timeout, and a synchronous call
  during the switch (MT).
- The read-only test helper ends the reconfig pass, as a real connect
  does. Thus a reconnect without the hint follows the cursor, and the
  transition test checks that only the hint selects the read/write
  server.
- Close the handle before the join in the MT test, so that a failed
  assertion cannot hang the test. TestJob::join() is now idempotent.
- Put the shared test helpers in one block after
  AsyncVoidOperationCompletion, so that this change and ZOOKEEPER-5120
  merge in either order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@anmolnar thanks for the review. I rebased the branch onto current master and force-pushed it. Your points are addressed in addendum commit 265a10a.

1. Ping-failure hunk. Dropped. It had no effect, and #2495 owns that path. The sentence about it in the original commit message is wrong. The squash message should leave it out.

2. Merge order with #2495. I moved the hunks in both PRs apart, and both PRs now add one byte-identical block of shared test helpers. This PR, #2495 and #2496 merge cleanly in all six orders, and every order gives the same tree.

3. Probe socket leak. Fixed. The SSL fields are set right after socket(), and every failure path in ping_rw_server goes through close_zsock().

4. Reconfig lock. The cursor peek, the hint publish and the hint consume now run under the lock. The probe itself does not, because it can block for seconds and would stall zoo_set_servers(). Instead, update_addrs bumps a version counter when it replaces the list, and a probe that overlaps a list change drops its result.

5. Tests. The portOf assertion could already fail, but for a different reason than the test comment gave. forceConnected() left the handle in the reconfig pass of zookeeper_init, so a reconnect without the hint took the first new address. The test helper now ends the reconfig pass, as a real connect does. The cursor step then means what it says, and only the hint can select the read/write server. New tests cover the cases you listed: a hint dropped by a list change, a probe made stale by a list change, a probe that finds another read-only server, a failed probe connect (which also checks the leak), fd == -1 after a receive timeout, and an MT blocked zoo_get() that returns ZCONNECTIONLOSS.

Nits. has_addr_rw_server is set from rc != 0. The description now says "returning a closed descriptor to the event loop" and has the release-note line.

Left for follow-ups. These are outside this PR's scope:

  • handle_error still moves the cursor without the lock. That is the same kind of race as in point 4, on a different path.
  • After a hinted connect, the cursor still points at the read/write server. With ZOOKEEPER-5121: (C client) DNS lookup failures can disrupt the client #2496 merged, a failed connect to that server tries it once more before it moves on.
  • The first probe of a new session goes to the current server, because the first connect does not move the cursor. This costs one probe interval.

Tests run. C suites through the Maven full-build profile on Linux x86_64 (RHEL 9, JDK 17, OpenSSL 3.5). ST OK (61). MT 103 run, 3 failures: testSSL, testSSLServerOnly and testReadOnlyWithSSL (ZSSLCONNECTIONERROR). They are not related to this PR, and they fail the same way on master. With OpenSSL 3.2 and later, the test certificates do not verify. ZOOKEEPER-5135 (#2510) fixes the certificates, and it can merge first. With #2510 applied, every test passes, both on master (ST OK (55), MT OK (102)) and with all three PRs merged (ST OK (75), MT OK (106)).

@apurtell

Copy link
Copy Markdown
Contributor Author

I had Claude help more directly so any of the the PRs in the set #2494, #2495, and #2496, each currently under review, can be applied in any order and in any order each of these PRs will land without conflicts and with good test results.

@anmolnar anmolnar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review on behalf of @anmolnar

Thanks for the quick turnaround. I went through the new commit (265a10a) and all the points from the first round are addressed:

  • The ping-failure hunk is gone and the path is left to #2495. A trial merge of this branch with the current #2495 head is clean, and the merged update_addrs and ping-failure code read correctly together.
  • ping_rw_server now closes the probe socket on every exit, with the SSL fields set before the first one.
  • The hint and the cursor are handled under the reconfig lock, and the addrs_version check drops a probe result when the list changes during the probe. I checked the lock sites for nested acquisition and found none (zoo_cycle_next_server is called after the unlock), and I could not construct an interleaving that leaves a hint for a server outside the current list.
  • The transition test now moves the cursor past the probed server, so the port assertion can fail without the hint. The new tests each fail on the pre-fix code for the reason they name.

As before, this is from reading the code and tracing the tests against the mocks; I have not built or run it. LGTM once CI is green for both the ST and MT builds. testReadOnlyToReadWriteFailsSyncCall depends on thread timing, so it would be good to see it pass more than once.

Non-blocking, for merge time:

  • The first commit message still says "and the current socket after a failed ping send", which the second commit corrects. Please drop that sentence from the squashed message.
  • Requests pending during the RO to RW switch now complete with ZCONNECTIONLOSS. Worth a line in the JIRA release note if it is not there already.

Optional:

  • A probe made stale by a server list change returns 0, so the caller still advances the cursor, now on the new list. Harmless, just noting it.
  • The MT test documents that the first probe of a new session goes to the server it is already connected to, because a connect in reconfig mode does not advance the zh->addrs cursor. That predates this PR; it might deserve a follow-up JIRA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants