Skip to content

ZOOKEEPER-5117: Don't leave a quota half created, and repair one on retry - #2488

Open
SpookyJumpyBeans wants to merge 2 commits into
apache:masterfrom
SpookyJumpyBeans:ZOOKEEPER-5117
Open

SpookyJumpyBeans wants to merge 2 commits into
apache:masterfrom
SpookyJumpyBeans:ZOOKEEPER-5117

Conversation

@SpookyJumpyBeans

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ZOOKEEPER-5117

SetQuotaCommand.createQuota created a quota's zookeeper_limits and zookeeper_stats nodes with two separate creates. If it failed in between, only the limit node was left:

  • Retrying setquota took the "quota exists" branch, read the stat node first, and failed with NoNode every time.
  • The limit node registers the quota with the server, but ZooKeeperServer.checkQuota and DataTree.updateQuotaStat return when the stat node is missing, so a hard quota wasn't enforced.

delquota deleted the two nodes one at a time too, so it could leave a stat node without a limit node. The next setquota then failed with NodeExists, and a retry kept the old stat node, which isn't updated while there is no limit node.

Changes

  • A new quota's limit and stat nodes are created in one multi. A stat node left without its limit node is deleted in the same multi:

    List<Op> ops = new ArrayList<>();
    if (zk.exists(statPath, false) != null) {
        ops.add(Op.delete(statPath, -1));
    }
    ops.add(Op.create(quotaPath, quota.getStatsBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT));
    ops.add(createStatOp(statPath));
    multi(zk, ops);
  • When the limit node exists but the stat node doesn't, setquota creates the stat node, together with a check that the limit node still exists, before updating the limit.

  • In both cases the server computes the real usage when the stat node is created (DataTree.updateQuotaForPath), so the 0/0 the client writes is replaced. Creating the quota already relied on this.

  • delquota deletes the limit and stat nodes in one multi. It no longer deletes the quota node's other children. That failed with NotEmpty when a leftover child quota node had children, which made the quota impossible to delete. Empty leftovers are now left in place.

  • multi errors are rethrown with the failed operation's path, so the CLI still says which node it was.

Not changed

  • The server still doesn't enforce a quota whose stat node is missing. This change makes setquota stop leaving that state and repair it, rather than changing the server.
  • A quota that already has both nodes but an outdated stat node keeps it. delquota then setquota fixes it.
  • delquota on a path with only a stat node still says the quota doesn't exist. setquota replaces that stat node.
  • Concurrent setquota/delquota on the same path still aren't serialized. A delquota racing the repair of a limit-only quota can still leave a stat node behind.

Tests

New tests in ZooKeeperQuotaTest:

  • testSetQuotaWhenStatNodeIsMissing: a limit node without a stat node. It shows that a create over the hard limit succeeds, runs setquota, then checks the usage and that the limit is enforced, also after a server restart, which replays the txn log.
  • testSetQuotaWhenLimitNodeIsMissing: a real quota whose limit node is deleted, so its stat node goes stale. It runs setquota and makes the same checks.
  • testDeleteQuotaWithLeftoverChildQuotaNode: delquota on a quota whose node also holds a leftover child quota node.
  • testSetAndDeleteQuotaDontWriteQuotaNodesOneByOne: a client that fails after one quota node was written alone. It fails if either command goes back to separate creates or deletes.

All four fail on master (NoNode, NodeExists, NotEmpty, ConnectionLoss) and pass with the change. ZooKeeperQuotaTest, EnforceQuotaTest and QuorumQuotaTest pass, and so does checkstyle.

SpookyJumpyBeans added a commit to SpookyJumpyBeans/boyidai that referenced this pull request Nov 3, 2026
@JHSUYU

JHSUYU commented Nov 6, 2026 •

Copy link
Copy Markdown

@kezhuw, @anmolnar, @maoling @PDavid

@PDavid

PDavid commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

I see that the failing CI checks were run on Ubuntu 26.04. We already fixed the CI checks on master to use Ubuntu 24 since then.
Can you please rebase your PR to have the fix so that the CI checks hopefully run properly?

…etry

setquota created the limit and stat nodes with two separate calls, so a
failure in between left only the limit node. Retrying then failed with
NoNode on the stat node, and the server doesn't enforce a quota without
its stat node. delquota also deleted them one at a time, which could
leave a stat node behind that made setquota fail with NodeExists.

Create both nodes in one multi, replacing a stat node left without its
limit node, and create a missing stat node when the limit node exists.
The server computes the usage when a stat node is created. delquota now
deletes the limit and stat nodes in one multi and no longer deletes other
children of the quota node. That failed with NotEmpty when a child quota
node had children; empty leftovers are now left in place.
@SpookyJumpyBeans

Copy link
Copy Markdown
Author

Rebased onto master, thanks!

@JHSUYU

JHSUYU commented Nov 16, 2026

Copy link
Copy Markdown

I see that the failing CI checks were run on Ubuntu 26.04. We already fixed the CI checks on master to use Ubuntu 24 since then. Can you please rebase your PR to have the fix so that the CI checks hopefully run properly?

Thanks @PDavid for reviewing, we have rebased it.

@JHSUYU

JHSUYU commented Nov 17, 2026

Copy link
Copy Markdown

@PDavid By the way, if you have time, could you kindly review two other PRs? Sorry if they’re not in your area. We’re happy to revise them based on any feedback.
#2506
#2504

@PDavid PDavid left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many thanks for fixing this, this looks good to me. 👍

Comment on lines +647 to +652
try {
zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
fail("should not create a node when it exceeds the hard count quota");
} catch (QuotaExceededException e) {
//expected
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super minor:

This may be shortened by using assertThrows:

Suggested change
try {
zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
fail("should not create a node when it exceeds the hard count quota");
} catch (QuotaExceededException e) {
//expected
}
// should not create a node when it exceeds the hard count quota
assertThrows(QuotaExceededException.class,
() -> zk.create(path + "/over", "data".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT));

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants