Repository navigation
ZOOKEEPER-5117: Don't leave a quota half created, and repair one on retry - #2488
Open
SpookyJumpyBeans wants to merge 2 commits into
Open
SpookyJumpyBeans wants to merge 2 commits into
SpookyJumpyBeans wants to merge 2 commits into
Conversation
SpookyJumpyBeans
added a commit
to SpookyJumpyBeans/boyidai
that referenced
this pull request
Nov 3, 2026
Contributor
|
I see that the failing CI checks were run on Ubuntu 26.04. We already fixed the CI checks on master to use Ubuntu 24 since then. |
…etry setquota created the limit and stat nodes with two separate calls, so a failure in between left only the limit node. Retrying then failed with NoNode on the stat node, and the server doesn't enforce a quota without its stat node. delquota also deleted them one at a time, which could leave a stat node behind that made setquota fail with NodeExists. Create both nodes in one multi, replacing a stat node left without its limit node, and create a missing stat node when the limit node exists. The server computes the usage when a stat node is created. delquota now deletes the limit and stat nodes in one multi and no longer deletes other children of the quota node. That failed with NotEmpty when a child quota node had children; empty leftovers are now left in place.
SpookyJumpyBeans
force-pushed
the
ZOOKEEPER-5117
branch
from
November 13, 2026 03:24
77ee5aa to
77830fe
Compare
Author
|
Rebased onto master, thanks! |
Thanks @PDavid for reviewing, we have rebased it. |
PDavid
approved these changes
Nov 18, 2026
PDavid
left a comment
Contributor
There was a problem hiding this comment.
Many thanks for fixing this, this looks good to me. 👍
Comment on lines
+647
to
+652
| try { | ||
| zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); | ||
| fail("should not create a node when it exceeds the hard count quota"); | ||
| } catch (QuotaExceededException e) { | ||
| //expected | ||
| } |
Contributor
There was a problem hiding this comment.
Super minor:
This may be shortened by using assertThrows:
Suggested change
| try { | |
| zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT); | |
| fail("should not create a node when it exceeds the hard count quota"); | |
| } catch (QuotaExceededException e) { | |
| //expected | |
| } | |
| // should not create a node when it exceeds the hard count quota | |
| assertThrows(QuotaExceededException.class, | |
| () -> zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT)); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/ZOOKEEPER-5117
SetQuotaCommand.createQuotacreated a quota'szookeeper_limitsandzookeeper_statsnodes with two separate creates. If it failed in between, only the limit node was left:setquotatook the "quota exists" branch, read the stat node first, and failed withNoNodeevery time.ZooKeeperServer.checkQuotaandDataTree.updateQuotaStatreturn when the stat node is missing, so a hard quota wasn't enforced.delquotadeleted the two nodes one at a time too, so it could leave a stat node without a limit node. The nextsetquotathen failed withNodeExists, and a retry kept the old stat node, which isn't updated while there is no limit node.Changes
A new quota's limit and stat nodes are created in one
multi. A stat node left without its limit node is deleted in the samemulti:When the limit node exists but the stat node doesn't,
setquotacreates the stat node, together with a check that the limit node still exists, before updating the limit.In both cases the server computes the real usage when the stat node is created (
DataTree.updateQuotaForPath), so the 0/0 the client writes is replaced. Creating the quota already relied on this.delquotadeletes the limit and stat nodes in onemulti. It no longer deletes the quota node's other children. That failed withNotEmptywhen a leftover child quota node had children, which made the quota impossible to delete. Empty leftovers are now left in place.multierrors are rethrown with the failed operation's path, so the CLI still says which node it was.Not changed
setquotastop leaving that state and repair it, rather than changing the server.delquotathensetquotafixes it.delquotaon a path with only a stat node still says the quota doesn't exist.setquotareplaces that stat node.setquota/delquotaon the same path still aren't serialized. Adelquotaracing the repair of a limit-only quota can still leave a stat node behind.Tests
New tests in
ZooKeeperQuotaTest:testSetQuotaWhenStatNodeIsMissing: a limit node without a stat node. It shows that a create over the hard limit succeeds, runssetquota, then checks the usage and that the limit is enforced, also after a server restart, which replays the txn log.testSetQuotaWhenLimitNodeIsMissing: a real quota whose limit node is deleted, so its stat node goes stale. It runssetquotaand makes the same checks.testDeleteQuotaWithLeftoverChildQuotaNode:delquotaon a quota whose node also holds a leftover child quota node.testSetAndDeleteQuotaDontWriteQuotaNodesOneByOne: a client that fails after one quota node was written alone. It fails if either command goes back to separate creates or deletes.All four fail on master (
NoNode,NodeExists,NotEmpty,ConnectionLoss) and pass with the change.ZooKeeperQuotaTest,EnforceQuotaTestandQuorumQuotaTestpass, and so does checkstyle.