Skip to content

ZOOKEEPER-5125 (C client) zoo_multi error handling crashes, misreads results, and hangs - #2500

Open
apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5125
Open

apurtell wants to merge 2 commits into
apache:masterfrom
apurtell:ZOOKEEPER-5125

Conversation

@apurtell

@apurtell apurtell commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

The C client's multi() path has several related defects.

When a synchronous zoo_multi() call fails, the err fields of the zoo_op_result_t array stay uninitialized, so an application might mistake a failed sub-operation as a success. NULL ops or results arrays will crash the client instead of returning an error. On an expired or unrecoverable handle, the client still queues requests, so a synchronous call can block forever, for example a zero count zoo_multi().

Inside a multi op handling the server answers a container create with a Create2Response, which holds a path and a stat. However the client decodes it as a CreateResponse which doesn't know about the stat, so every result after it will be decoded in a misaligned manner, essentially reading garbage.

The fix fills every result slot with the failure code on all failure paths, and returns ZBADARGUMENTS for NULL arrays. A multi that fails a call-time check returns that code, for example ZBADARGUMENTS for an invalid path or ZINVALIDSTATE for an expired handle, instead of ZMARSHALLINGERROR. New requests on a closing or unrecoverable handle are refused at once. Each sub-result is decoded correctly. A reply with more or fewer results than operations returns ZMARSHALLINGERROR instead of crashing the application.

Behaviour change outside multi: the refusal is in do_add_completion, so it applies to every request. A synchronous call that races with session expiry no longer hangs, and an asynchronous call in that window fails at once instead of being queued.

New mock server tests Zookeeper_multiFailure and real server test Zookeeper_multi::testCreateContainer cover these cases.

@anmolnar

Copy link
Copy Markdown
Contributor

AI-assisted review on behalf of @anmolnar

Thanks for this. I traced each fix against zookeeper.c and checked the container-create wire format against FinalRequestProcessor and MultiResponse; all five look correct to me. I also checked that fail_multi_results cannot run for a multi that was queued, and that the new check in do_add_completion is ordered correctly against free_completions on expiry and auth failure. I did not build or run the tests.

A few points, none blocking:

  1. Error code on call-time failures. zoo_amulti passes every call-time failure through nonblocking_send, which turns any negative code into ZMARSHALLINGERROR. So a multi on an expired handle or with an invalid path returns ZMARSHALLINGERROR (as on master), and with this change that code is also written into every results[i].err. The new tests assert only rc != ZOK, so they do not show which code is intended. Could zoo_amulti return the real code (ZINVALIDSTATE, ZBADARGUMENTS) when rc < 0, with the tests asserting it? If the mapping is kept on purpose, a note in the description would help.

  2. Scope of the do_add_completion change. It applies to every request, not only multi: it closes the same hang for any synchronous call that races with session expiry, and async calls in that window now fail instead of queuing. I think that is right, but it is a behaviour change outside multi and worth a line in the description and the JIRA.

  3. Test coverage. The two count-mismatch branches in deserialize_multi (more or fewer results than operations) have no test. The mock responses in TestMultiFailure.cc would make both easy to add. An async zoo_amulti case for the container create would also be useful.

  4. Small.

    • The comment on deserialize_multi says each sub-result is decoded by its wire type, but only the ZOO_CREATE2_OP case is.
    • The zoo_multi / zoo_amulti docs in zookeeper.h could mention that every result slot is filled on failure and list the new ZBADARGUMENTS and ZMARSHALLINGERROR returns.

Andrew Purtell and others added 2 commits November 19, 2026 16:19
…zed, desync on container creates, abort on header-only errors, and hang on dead handles

Fix a group of related defects in the C client multi-op path:

- Synchronous zoo_multi() failures left the err fields of zoo_op_result_t
  uninitialized. Fill every result slot with the failure code when the
  multi fails at call time (validation, send failure, closed or
  unrecoverable handle) and when a pending multi is drained on
  connection loss or session expiry.
- A container create inside a multi gets a Create2Response (path and
  stat). The client decoded it as a CreateResponse, which left the stat
  in the stream and misaligned all following sub-results. Decode the
  record by its wire type.
- A multi reply with an error in its header (for example ZNOTREADONLY)
  has no body. The synchronous path tried to decode it and hit
  assert(entry). Fail every sub-operation with the header error instead.
  Treat replies with more or fewer sub-results than operations as
  ZMARSHALLINGERROR instead of crashing.
- do_add_completion() queued requests on an unrecoverable handle, where
  no teardown drains them. A synchronous caller, for example a
  zero-count zoo_multi() on an expired session, waited forever. Refuse
  the request instead.
- zoo_amulti() dereferenced NULL ops or results arrays. Return
  ZBADARGUMENTS.

Add Zookeeper_multiFailure mock-server tests and a real-server
Zookeeper_multi::testCreateContainer test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r reply mismatches

Address review feedback:

- Return ZBADARGUMENTS, ZINVALIDSTATE, and ZSYSTEMERROR from zoo_amulti
  as themselves. A serializer error still gives ZMARSHALLINGERROR. The
  tests now assert the exact code.
- Add mock tests for a reply with more results than operations, for a
  reply with fewer results, and for an async container create.
- Correct the deserialize_multi comment about record formats.
- Document in zookeeper.h the return codes of zoo_amulti, and that every
  result slot is filled on failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apurtell

apurtell commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Reply from Claude Code (an AI assistant), on behalf of @apurtell.

@anmolnar Thanks for the review. Addressed in 3606a01:

  1. zoo_amulti now returns call-time codes as they are (ZBADARGUMENTS, ZINVALIDSTATE, ZSYSTEMERROR); only serializer errors map to ZMARSHALLINGERROR. The tests assert the exact code in the return value and in every result slot.
  2. Added a line to the description on the do_add_completion scope. I'll add it to the JIRA too.
  3. Added mock tests for more and fewer results than operations, and an async zoo_amulti container create.
  4. Fixed the deserialize_multi comment, and documented the return codes and the result slot filling in zookeeper.h.

With this, the multi test in #2499 can assert ZBADARGUMENTS. Whichever of the two lands second will tighten it during its rebase.

All C client tests pass on Linux with OpenSSL 3.2+, when #2510 is applied for the test certificates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants