Skip to content

ZOOKEEPER-5121: (C client) DNS lookup failures can disrupt the client - #2496

Open
apurtell wants to merge 3 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5121
Open

apurtell wants to merge 3 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5121

Conversation

@apurtell

@apurtell apurtell commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Since ZOOKEEPER-1355 (3.5.0), zookeeper_interest() re-resolves the server list with a blocking getaddrinfo() on each pass of the I/O loop. ZOOKEEPER-1998 (3.7.0) added zoo_set_servers_resolution_delay() to throttle these lookups, but the default still resolves on every pass. When a lookup fails, zookeeper_interest() returns an error before the ping, timeout and reconnect logic runs. It also returns without setting its outputs, and do_io() ignores the return code and passes those outputs to poll(). Thus a short DNS outage can expire a healthy session, or stop a disconnected client from reconnecting to addresses it already knows.

This change keeps the lookup in the loop and changes what a failure does:

  • A refresh from the I/O loop in which a host does not resolve keeps all cached addresses and does not return an error. The next refresh then waits, starting at 1 s and doubling up to 60 s, also while the socket is down. A resolver error still fails zookeeper_init() and zoo_set_servers(), and also a refresh when no addresses are cached. A failed zoo_set_servers() keeps the current server list.
  • zookeeper_interest() sets safe outputs (no descriptor, no interest, a short timeout) before any early return.
  • After a connect failure, both handle_error() and zoo_cycle_next_server() advanced the server cursor, so failover skipped every other server. Outside a reconfig pass, only zoo_cycle_next_server() now advances it. This fix is tracked separately as ZOOKEEPER-5136 for release notes and backports.

New tests in TestResolverFailure.cc check that pings, receive timeouts, reconnects and failover continue through a simulated resolver outage, and that a failed zookeeper_interest() call gives safe outputs.

@apurtell

apurtell commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor Author

I found another issue in this part of the code. A new JIRA and PR for it would be a lot for a one line change and a conflict with the changes here, so have included it here. Double server cursor advancement during failover could cause a starvation because the advancement is fumbled. The PR has an addendum commit covering this case as 544966a. There are a few more lines due to the addition of comments and a new test :-)

@anmolnar anmolnar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review on behalf of @anmolnar

Thanks for this, Andrew. The three code changes look correct to me: I traced them through update_addrs, zookeeper_interest and handle_error, though I haven't run the new tests locally. A few things before merge:

  1. Description vs. code. The commit message says resolution is "decoupled into a rate-limited periodic refresh", but getaddrinfo is still called synchronously from zookeeper_interest, and the only rate limit is the existing zoo_set_servers_resolution_delay (default 0). Could you reword it to describe what the patch does: tolerate a failed refresh and keep the cached addresses?

  2. Slow resolvers. If DNS times out rather than failing fast, each zookeeper_interest call still blocks in getaddrinfo. With the default delay nothing paces the retries, and the delay is bypassed when the socket is down, so pings can still be missed. Would you consider a minimum backoff after a failed refresh, or should we note this as a known limitation and file a follow-up?

  3. Partial failures. resolve_hosts skips a host that fails and returns ZOK if any other resolved. If the skipped host is the connected server, update_addrs still closes the socket. That is existing behaviour, but it contradicts the new "a failed lookup is not a topology change" comment. Follow-up JIRA?

  4. Cursor fix. Nice catch. On master an even-sized server list skips half the servers and never hits the end-of-list delay (with two servers, the second is never tried). I think it deserves its own JIRA id for release notes and backports, even if the change stays in this PR.

  5. zoo_set_servers on failure. It returns ZSYSTEMERROR but leaves the new host string in zh->hostname with the old addresses, so the client switches once DNS recovers. testRejectedServerListKeepsReconnecting now pins part of that. Is it intended?

Nits: the zoo_set_servers_resolution_delay doc in zookeeper.h still says the window counts from the last successful resolution, and the PR title is missing the colon after the JIRA id.

@cyberw

cyberw commented Nov 16, 2026

Copy link
Copy Markdown

Previously, the ZooKeeper client resolved addresses synchronously inside its main I/O loop

I'm curious, do you know when this was changed?

Andrew Purtell and others added 2 commits November 19, 2026 16:18
Previously, the ZooKeeper client resolved addresses synchronously
inside its main I/O loop. When a DNS lookup failed, the loop exited
without sending heartbeats or attempting to reconnect. It also left
output parameters uninitialized, feeding invalid state into poll().
Brief DNS blips could silently expire healthy sessions and prevent
disconnected clients from falling back to known addresses.

Address resolution is now decoupled into a rate-limited periodic
refresh. If a lookup fails, the client retains and uses its cached
server addresses, reserving hard resolver errors for initial setup
calls (zookeeper_init and zoo_set_servers).

The event loop now guarantees safe fallback outputs on any early
return, preventing poll corruption.

New tests verify that heartbeats, timeouts, and reconnections
continue uninterrupted throughout simulated DNS outages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n turn

After a connect failure, both handle_error and zoo_cycle_next_server
advanced the server list cursor, but the client skipped every other
server.

Outside a reconfig pass, handle_error now only records the next
server, and zoo_cycle_next_server alone advances the cursor. A reconfig
pass selects from addrs_new and addrs_old, so handle_error still
advances the cursor there to pace the end-of-list delay.

A new test verifies that connect attempts alternate between two
refusing servers during a resolver outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@cyberw It has not changed on master. "Previously" in the description means "before this PR", and the wording does not make that clear.

The git history:

  • ZOOKEEPER-1355 (ed62259, in 3.5.0) added update_addrs() and called it from zookeeper_interest(). Since then, the driver re-resolves the host list with a blocking getaddrinfo() inside the I/O loop. A lookup error returns before the ping, timeout and reconnect logic runs.
  • ZOOKEEPER-1998 (a908001, in 3.7.0) added zoo_set_servers_resolution_delay() to throttle these lookups. The default delay is 0, so by default a connected client still resolves on every loop pass.

This PR keeps the lookup in the loop. What changes is the failure path: a failed refresh keeps the cached addresses, and its attempt time paces the next retry through the same resolution delay. Resolver errors stay fatal only in zookeeper_init() and zoo_set_servers().

@apurtell

Copy link
Copy Markdown
Contributor Author

Updated the description on this PR for clarity.

@apurtell

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@anmolnar

4. Cursor fix: filed as ZOOKEEPER-5136. The change stays in this PR, and the description now references it. Both cursor advances date from ZOOKEEPER-1355, so the bug affects 3.5.0 and later.

1. Description vs. code: the description now says what the patch does. The lookup stays synchronous in the loop. A failed refresh keeps the cached addresses, and the existing resolution delay paces the retries. The original commit message still has the old wording, because changing it would rewrite pushed history. The squash message can use the corrected description.

Points 2, 3, 5 and the nits will get a separate reply.

…ched list on a partial refresh

- After a periodic refresh in which one or more hosts do not resolve,
  the next refresh waits. The wait starts at one second, doubles after
  each failed refresh up to one minute, and counts from the end of the
  lookup. It applies also while the client is not connected. A lookup
  in which all hosts resolve clears it. A single lookup can still block
  the I/O loop for as long as the resolver takes; the wait limits how
  often this occurs.
- A periodic refresh in which some hosts do not resolve keeps all cached
  addresses. Thus the client does not drop its server only because a
  name did not resolve.
- zoo_set_servers() stages the new list. The list becomes current only
  if one or more of its hosts resolve. A failed call keeps the current
  list and addresses, and it does not start the wait. If only some hosts
  resolve, the client uses the addresses that resolved.
- The zoo_set_servers() and zoo_set_servers_resolution_delay() docs
  describe this behavior. The delay window counts from the last
  resolution, and a failed routine resolution also counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell apurtell changed the title ZOOKEEPER-5121 (C client) DNS lookup failures can disrupt the client ZOOKEEPER-5121: (C client) DNS lookup failures can disrupt the client Nov 20, 2026
@apurtell

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@anmolnar thanks again. I rebased the branch onto current master and force-pushed it. Points 2, 3 and 5 and the nits are addressed in addendum commit 6e747cf.

2. Slow resolvers. I added the minimum backoff. After a periodic refresh in which a host does not resolve, the next refresh waits 1 s. The wait doubles after each failed refresh, up to 60 s. It counts from the end of the lookup and applies also while the socket is down. A lookup in which all hosts resolve clears it. A single lookup can still block the I/O loop for the resolver timeout. The zoo_set_servers_resolution_delay() doc says so. Moving the lookup off the I/O thread is a larger change, and I can file a follow-up for it.

3. Partial failures. A periodic refresh in which any host fails now counts as a failed refresh. The client keeps its whole cached list, so it no longer drops the connected server because one name did not resolve. An explicit zoo_set_servers() call is different: there the caller chose a new list, so the client uses the addresses that resolved, and the doc says so.

5. zoo_set_servers() on failure. That was not intended. The call now stages the new list, and the list becomes current only if it resolves. A failed call keeps the current list and addresses, so the rejected list does not come back when DNS recovers. testRejectedServerListKeepsReconnecting now checks for this.

Nits. The delay doc now counts the window from the last resolution, and says that a failed routine resolution also counts. The title has the colon.

Tests run. C suites through the Maven full-build profile on Linux x86_64 (RHEL 9, JDK 17, OpenSSL 3.5). ST OK (65). MT 102 run, 3 failures: testSSL, testSSLServerOnly and testReadOnlyWithSSL (ZSSLCONNECTIONERROR). They are not related to this PR, and they fail the same way on master. With OpenSSL 3.2 and later, the test certificates do not verify. ZOOKEEPER-5135 (#2510) fixes the certificates, and it can merge first. With #2510 applied, every test passes, both on master (ST OK (55), MT OK (102)) and with all three PRs merged (ST OK (75), MT OK (106)).

@apurtell

Copy link
Copy Markdown
Contributor Author

I had Claude help more directly so any of the the PRs in the set #2494, #2495, and #2496, each currently under review, can be applied in any order and in any order each of these PRs will land without conflicts and with good test results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants