Skip to content

ZOOKEEPER-5123: (C client) Broadcast completion of auth can return ZOK for rejected credentials - #2498

Open
apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5123
Open

apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5123

Conversation

@apurtell

@apurtell apurtell commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

When the C client received a reply to a SetAuth request, auth_completion_func ran every pending callback with the status of that one reply without checking which credential the reply belongs to. The client also put SetAuth frames at the front of the send queue, so frames from back-to-back calls went out in reverse order.

There were additional related problems on the same path:

  • Credentials that the client sends again during a reconnect could complete newly issued zoo_add_auth calls before the server checked them.
  • An auth reply on a handle with no credentials dereferenced a NULL list and crashed the I/O thread.
  • Concurrent zoo_add_auth calls could both send the last credential in the list.
  • If a frame could not be queued zoo_add_auth would return an error but leave the callback registered.

Each handle now keeps a FIFO of pending SetAuth frames in the order of frames on the wire. Each reply completes only the entry at the head.

Related fixes:

  • An auth reply with no pending frame is logged and ignored. Before, it dereferenced a NULL credential list.
  • zoo_add_auth holds the auth lock while it queues the frame of its own credential. Concurrent calls no longer send the tail credential twice. The frame goes to the end of the send queue.
  • zoo_add_auth checks its allocations. If the frame cannot be queued, the credential is not added and its completion never runs.
  • zoo_add_auth no longer sends a frame in ZOO_ASSOCIATING_STATE. Session establishment sends the credential with its completion.

Adds the Zookeeper_authCompletion suite in tests/TestAuthCompletion.cc.

@anmolnar anmolnar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI-assisted review on behalf of @anmolnar

Thanks for this. I traced the change against zookeeper.c and the server's processPacket, and it looks correct to me: the server answers each SetAuth frame once and in order, the pending FIFO and the send queue are updated together under the auth lock at the same end, front insertion at session establishment cannot land ahead of a partly written buffer, and after a rejection the remaining pending completions are finished with ZAUTHFAILED through handle_error. I also walked the two zoo_add_auth races with the I/O thread (around finalize_session_establishment and around cleanup) and both end with one frame per credential. I did not build or run the tests.

A few points, none blocking:

  1. The reconnect path is not covered. testResentCredentialDoesNotCompleteNewRequest never disconnects, so the loop in send_auth_info that hands pending completions back, remove_auth_frames, and front insertion with more than one stored credential are all unexercised. Could you add (a) two credentials stored before connect(), one accepted and one rejected, asserting each completion gets its own result, and (b) a disconnect and reconnect with replies held, then a new zoo_add_auth, asserting the resend's reply does not complete it?
  2. testConcurrentAddAuthSendsEachCredential passes NULL completions and ignores the return of zoo_add_auth. A completion per call, asserted to run once with ZOK, would also check matching under concurrency.
  3. zookeeper.h says the completion receives ZOK or ZAUTHFAILED. A request that is still pending when an earlier credential is rejected now gets ZAUTHFAILED, and on a disconnect it gets the connection-loss code while the credential is kept and resent without a completion. The second part is not new, but a sentence in the header would make the contract clear.
  4. Small: zoo_add_auth still does not call adaptor_send_queue when connected, so the frame waits for the I/O thread's next wake-up (same on master); a ZSYSTEMERROR from send_info_packet is returned as ZMARSHALLINGERROR; and auth_info.state is now write-only.

Andrew Purtell and others added 2 commits November 19, 2026 16:19
…K for rejected credentials

When the C client received a reply to a SetAuth request,
auth_completion_func ran every pending callback with the status of that
one reply without checking which credential the reply belongs to. The
client also put SetAuth frames at the front of the send queue, so frames
from back-to-back calls went out in reverse order.

There were additional related problems on the same path:

- Credentials that the client sends again during a reconnect could
  complete newly issued zoo_add_auth calls before the server checked
  them.
- An auth reply on a handle with no credentials dereferenced a NULL
  list and crashed the I/O thread.
- Concurrent zoo_add_auth calls could both send the last credential in
  the list.
- If a frame could not be queued zoo_add_auth would return an error but
  leave the callback registered.

Each handle now keeps a FIFO of pending SetAuth frames in the order of
frames on the wire. Each reply completes only the entry at the head.

Related fixes:
- An auth reply with no pending frame is logged and ignored. Before, it
  dereferenced a NULL credential list.
- zoo_add_auth holds the auth lock while it queues the frame of its own
  credential. Concurrent calls no longer send the tail credential twice.
  The frame goes to the end of the send queue.
- zoo_add_auth checks its allocations. If the frame cannot be queued,
  the credential is not added and its completion never runs.
- zoo_add_auth no longer sends a frame in ZOO_ASSOCIATING_STATE. Session
  establishment sends the credential with its completion.

Adds the Zookeeper_authCompletion suite in tests/TestAuthCompletion.cc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… contract

Address review feedback:

- Add tests for two stored credentials with separate results, for a
  reconnect resend that does not complete a new request, and for
  zoo_add_auth on the I/O thread during session establishment. The last
  test drives the code that takes back pending completions and drops
  queued SetAuth frames.
- Give each call in the concurrency test a completion, and check that it
  runs once with ZOK and that zoo_add_auth returns ZOK.
- Document in zookeeper.h which result a pending completion receives
  after an earlier rejection or a disconnect.
- Send the frame at once when connected, as the other async calls do.
- Return ZSYSTEMERROR from zoo_add_auth as itself.
- Remove the write-only auth_info.state field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell

apurtell commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

Thanks @anmolnar. Your review points are addressed in 86f63df. I also rebased the branch onto current master.

  1. Added testStoredCredentialsCompleteSeparately (two stored credentials, one accepted and one rejected, each completion gets its own result) and testReconnectResendDoesNotCompleteNewRequest (disconnect with replies held, reconnect, then a new zoo_add_auth that the resend replies do not complete). One note: on a disconnect, cleanup already drains the pending FIFO and frees to_send. So the reconnect test covers front insertion with several stored credentials, but it does not reach the hand-back loop or remove_auth_frames. Those run only if a frame is queued after the state becomes connected and before send_auth_info runs. The new MT test testAddAuthDuringSessionEstablishment hits that window every time: a log callback calls zoo_add_auth on the I/O thread at the "session establishment complete" message. The test checks that the credential goes out once and that its completion runs once with ZOK.
  2. The concurrency test now gives each call a completion. It checks that each completion runs once with ZOK, and it checks every return value.
  3. Made an update to the zoo_add_auth doc in zookeeper.h.
  4. When connected, zoo_add_auth now calls nonblocking_send after it queues the frame, like the other async calls. If that send fails, the socket closes without cleanup, and the next session establishment takes back the pending entry and its frame. ZSYSTEMERROR is now returned as itself, and auth_info.state is removed.

The ST and MT suites pass. I applied #2510 locally for the OpenSSL certificate failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants