Skip to content

[CALCITE-7794] Reduce the work HepPlanner does per rule application attempt - #5272

Open
michaelbraun wants to merge 1 commit into
apache:mainfrom
michaelbraun:CALCITE-7794
Open

michaelbraun wants to merge 1 commit into
apache:mainfrom
michaelbraun:CALCITE-7794

Conversation

@michaelbraun

@michaelbraun michaelbraun commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

HepPlanner applies a rule collection by trying every rule against every vertex, so rule application attempts can outnumber matches. Three pieces of avoidable work sat on that path.

  • applyRule tested graph.vertexSet().contains(vertex) on every call. Both call sites try every rule against the same vertex object in a loop that only exits early on a match, so membership cannot change within it: 50 rules meant 50 set lookups where one suffices. The test moves to the two callers, and applyRules returns early for an empty rule collection so the hoisted test is not paid when there is no rule to apply.
  • applyRule allocated the bindings list and the nodeChildren map before calling matchOperands, which rejects on its first statement, so for the large majority of attempts both were allocated and discarded.
  • The ConverterRule and CommonRelSubExprRule branches ran before the operand was tested, traversing a vertex's parents for attempts that could never match.

applyRule now tests the operand first, and matchOperands is split into a thin entry point making that test plus matchOperandsWithOperandMatched assuming it, so it is not repeated for attempts that pass. The matching logic is unchanged and recursive descent still goes through the checking entry point.

Measurements

HepQueryVarietyBenchmark, added here, plans queries converted from SQL and measures setRoot plus findBestExp. collections is how many addRuleCollection calls the program makes, rulesPerColl how many rules each holds, firing% the share of the rule set able to rewrite these queries. Means over four rounds and five query shapes, percent faster than unmodified HepPlanner:

collections rulesPerColl firing% stock µs patched µs improved
1 1 0 23.7 23.4 1.0%
1 1 50 23.9 23.5 1.6%
1 1 100 27.6 27.3 0.8%
1 5 0 26.1 24.6 7.5%
1 5 50 31.7 29.9 6.2%
1 5 100 34.7 32.8 5.2%
1 50 0 47.2 31.3 35.5%
1 50 50 335.4 305.5 20.7%
1 50 100 356.8 319.8 18.0%
10 1 0 37.5 35.6 6.2%
10 1 50 47.4 45.1 5.0%
10 1 100 52.5 49.9 4.2%
10 5 0 59.6 44.6 27.6%
10 5 50 319.2 297.3 14.2%
10 5 100 328.2 307.9 12.6%
10 50 0 265.8 111.1 59.0%
10 50 50 612.5 468.4 30.6%
10 50 100 716.2 571.8 25.5%

A collection of one rule gains least, which follows from which change applies: hoisting the membership test saves nothing there, since one rule means one lookup either way. Above one rule per collection that test starts to pay, which is most of what this changes. Gains fall as more of the rule set can fire, since transformation work then grows while the rejection cost does not.

The non-firing rules are real ones for INTERSECT, MINUS, MATCH_RECOGNIZE, TABLESAMPLE and exchange, which none of these queries contain, so the attempts they add reject at the operand depths real rules reject at rather than all at the top level. Plans are identical whatever the non-firing share, confirming those rules never fire.

Test plan

  • HepPlannerTest, RelOptRulesTest, MaterializedViewRelOptRulesTest, CombineRelOptRulesTest — 1064 tests, 0 failures
  • checkstyleMain, checkstyleJmh, autostyleCheck clean
  • full ./gradlew build

🤖 Generated with Claude Code

* Matches the children of {@code rel} against those of {@code operand}, for a {@code rel}
* that {@code operand} is already known to match.
*/
private static boolean matchOperandsWithOperandMatched(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like the name of this method, could use help here

boolean forceConversions, int nMatches) {
while (iter.hasNext()) {
HepRelVertex vertex = iter.next();
// Once per vertex: a match ends the loop below, so membership cannot change in it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would also appreciate help phrasing this better

…ttempt

Rule application attempts greatly outnumber matches, so applyRule now tests
the rule's operand before doing any other work, skipping the ConverterRule
and CommonRelSubExprRule checks and the bindings and node-children
allocations for an attempt that cannot match. The graph-membership check
moves to the two callers, which try every rule against the same vertex and
so need it only once, and applyRules returns early when there is no rule at
all.

Adds a benchmark that plans a range of query shapes through rules that fire,
over varying numbers of rule collections and rules per collection, so
planning changes can be measured against realistic queries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@michaelbraun
michaelbraun marked this pull request as ready for review September 18, 2026 01:43
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant