Repository navigation
Conversation
|
I found another issue in this part of the code. A new JIRA and PR for it would be a lot for a one line change and a conflict with the changes here, so have included it here. Double server cursor advancement during failover could cause a starvation because the advancement is fumbled. The PR has an addendum commit covering this case as 544966a. There are a few more lines due to the addition of comments and a new test :-) |
anmolnar
left a comment
There was a problem hiding this comment.
AI-assisted review on behalf of @anmolnar
Thanks for this, Andrew. The three code changes look correct to me: I traced them through update_addrs, zookeeper_interest and handle_error, though I haven't run the new tests locally. A few things before merge:
-
Description vs. code. The commit message says resolution is "decoupled into a rate-limited periodic refresh", but
getaddrinfois still called synchronously fromzookeeper_interest, and the only rate limit is the existingzoo_set_servers_resolution_delay(default 0). Could you reword it to describe what the patch does: tolerate a failed refresh and keep the cached addresses? -
Slow resolvers. If DNS times out rather than failing fast, each
zookeeper_interestcall still blocks ingetaddrinfo. With the default delay nothing paces the retries, and the delay is bypassed when the socket is down, so pings can still be missed. Would you consider a minimum backoff after a failed refresh, or should we note this as a known limitation and file a follow-up? -
Partial failures.
resolve_hostsskips a host that fails and returns ZOK if any other resolved. If the skipped host is the connected server,update_addrsstill closes the socket. That is existing behaviour, but it contradicts the new "a failed lookup is not a topology change" comment. Follow-up JIRA? -
Cursor fix. Nice catch. On master an even-sized server list skips half the servers and never hits the end-of-list delay (with two servers, the second is never tried). I think it deserves its own JIRA id for release notes and backports, even if the change stays in this PR.
-
zoo_set_serverson failure. It returns ZSYSTEMERROR but leaves the new host string inzh->hostnamewith the old addresses, so the client switches once DNS recovers.testRejectedServerListKeepsReconnectingnow pins part of that. Is it intended?
Nits: the zoo_set_servers_resolution_delay doc in zookeeper.h still says the window counts from the last successful resolution, and the PR title is missing the colon after the JIRA id.
I'm curious, do you know when this was changed? |
Previously, the ZooKeeper client resolved addresses synchronously inside its main I/O loop. When a DNS lookup failed, the loop exited without sending heartbeats or attempting to reconnect. It also left output parameters uninitialized, feeding invalid state into poll(). Brief DNS blips could silently expire healthy sessions and prevent disconnected clients from falling back to known addresses. Address resolution is now decoupled into a rate-limited periodic refresh. If a lookup fails, the client retains and uses its cached server addresses, reserving hard resolver errors for initial setup calls (zookeeper_init and zoo_set_servers). The event loop now guarantees safe fallback outputs on any early return, preventing poll corruption. New tests verify that heartbeats, timeouts, and reconnections continue uninterrupted throughout simulated DNS outages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n turn After a connect failure, both handle_error and zoo_cycle_next_server advanced the server list cursor, but the client skipped every other server. Outside a reconfig pass, handle_error now only records the next server, and zoo_cycle_next_server alone advances the cursor. A reconfig pass selects from addrs_new and addrs_old, so handle_error still advances the cursor there to pace the end-of-list delay. A new test verifies that connect attempts alternate between two refusing servers during a resolver outage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Reply from Claude Code (an AI assistant), on behalf of @apurtell. @cyberw It has not changed on master. "Previously" in the description means "before this PR", and the wording does not make that clear. The git history:
This PR keeps the lookup in the loop. What changes is the failure path: a failed refresh keeps the cached addresses, and its attempt time paces the next retry through the same resolution delay. Resolver errors stay fatal only in |
|
Updated the description on this PR for clarity. |
|
Reply from Claude Code (an AI assistant), on behalf of @apurtell. 4. Cursor fix: filed as ZOOKEEPER-5136. The change stays in this PR, and the description now references it. Both cursor advances date from ZOOKEEPER-1355, so the bug affects 3.5.0 and later. 1. Description vs. code: the description now says what the patch does. The lookup stays synchronous in the loop. A failed refresh keeps the cached addresses, and the existing resolution delay paces the retries. The original commit message still has the old wording, because changing it would rewrite pushed history. The squash message can use the corrected description. Points 2, 3, 5 and the nits will get a separate reply. |
…ched list on a partial refresh - After a periodic refresh in which one or more hosts do not resolve, the next refresh waits. The wait starts at one second, doubles after each failed refresh up to one minute, and counts from the end of the lookup. It applies also while the client is not connected. A lookup in which all hosts resolve clears it. A single lookup can still block the I/O loop for as long as the resolver takes; the wait limits how often this occurs. - A periodic refresh in which some hosts do not resolve keeps all cached addresses. Thus the client does not drop its server only because a name did not resolve. - zoo_set_servers() stages the new list. The list becomes current only if one or more of its hosts resolve. A failed call keeps the current list and addresses, and it does not start the wait. If only some hosts resolve, the client uses the addresses that resolved. - The zoo_set_servers() and zoo_set_servers_resolution_delay() docs describe this behavior. The delay window counts from the last resolution, and a failed routine resolution also counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
544966a to
6e747cf
Compare
|
Reply from Claude Code (an AI assistant), on behalf of @apurtell. @anmolnar thanks again. I rebased the branch onto current master and force-pushed it. Points 2, 3 and 5 and the nits are addressed in addendum commit 6e747cf. 2. Slow resolvers. I added the minimum backoff. After a periodic refresh in which a host does not resolve, the next refresh waits 1 s. The wait doubles after each failed refresh, up to 60 s. It counts from the end of the lookup and applies also while the socket is down. A lookup in which all hosts resolve clears it. A single lookup can still block the I/O loop for the resolver timeout. The 3. Partial failures. A periodic refresh in which any host fails now counts as a failed refresh. The client keeps its whole cached list, so it no longer drops the connected server because one name did not resolve. An explicit 5. Nits. The delay doc now counts the window from the last resolution, and says that a failed routine resolution also counts. The title has the colon. Tests run. C suites through the Maven |
Since ZOOKEEPER-1355 (3.5.0),
zookeeper_interest()re-resolves the server list with a blockinggetaddrinfo()on each pass of the I/O loop. ZOOKEEPER-1998 (3.7.0) addedzoo_set_servers_resolution_delay()to throttle these lookups, but the default still resolves on every pass. When a lookup fails,zookeeper_interest()returns an error before the ping, timeout and reconnect logic runs. It also returns without setting its outputs, anddo_io()ignores the return code and passes those outputs topoll(). Thus a short DNS outage can expire a healthy session, or stop a disconnected client from reconnecting to addresses it already knows.This change keeps the lookup in the loop and changes what a failure does:
zookeeper_init()andzoo_set_servers(), and also a refresh when no addresses are cached. A failedzoo_set_servers()keeps the current server list.zookeeper_interest()sets safe outputs (no descriptor, no interest, a short timeout) before any early return.handle_error()andzoo_cycle_next_server()advanced the server cursor, so failover skipped every other server. Outside a reconfig pass, onlyzoo_cycle_next_server()now advances it. This fix is tracked separately as ZOOKEEPER-5136 for release notes and backports.New tests in
TestResolverFailure.cccheck that pings, receive timeouts, reconnects and failover continue through a simulated resolver outage, and that a failedzookeeper_interest()call gives safe outputs.