Skip to content

Benchmark the FFE scale evaluation suite - #13485

Draft
leoromanovsky wants to merge 1 commit into
leo.romanovsky/ffe-json-frozen-valuesfrom
leo.romanovsky/ffe-scale-suite-benchmark
Draft

leoromanovsky wants to merge 1 commit into
leo.romanovsky/ffe-json-frozen-valuesfrom
leo.romanovsky/ffe-scale-suite-benchmark

Conversation

@leoromanovsky

Copy link
Copy Markdown
Contributor

What Does This Do

Adds ScaleSuiteBenchmark, a JMH benchmark for the opt-in scale evaluation suite of the shared FFE fixtures. It evaluates each case of the suite through the provider and reports the time and allocation of one evaluation for each case file.

This PR is stacked on #13484. It pins ffe-system-test-data to the head of DataDog/ffe-system-test-data#78, which adds the suite.

Motivation

The canonical fixtures use small lists, short attribute values, and small JSON values. The scale suite uses sizes from production configurations:

  • ONE_OF and NOT_ONE_OF lists with 10,000 entries;
  • regex conditions on 2 KB inputs;
  • 300 conditions that share 10 patterns, and 20 allocations of 5 conditions;
  • a 200-attribute context;
  • JSON values with hundreds of nodes.

At these sizes, the problems are mainly time and memory, so a benchmark is the right place for the suite. CanonicalFixtureTest reads only evaluation-cases/, and the suite is opt-in because of its size, so the unit tests do not run it.

Changes

  • ScaleSuiteBenchmark (dd-openfeature/src/jmh):
    • @Param cases: json-values, one-of, regex, rules, wide-context (the case files of the suite).
    • The setup parses the suite configuration with UniversalFlagConfigParser.parse, and evaluates every case once. It throws if a value or reason differs from the expected one, so the benchmark does not measure a wrong result.
    • Each operation evaluates the next case of the selected file through Provider.
  • ffe-system-test-data submodule: 6df74e6 → 0081d8f (the head of Add an opt-in scale evaluation suite ffe-system-test-data#78). It contains the current pin.

Run: ./gradlew :dd-openfeature:jmh -Pjmh.includes=ScaleSuiteBenchmark -Pjmh.profilers=gc

Results

On top of #13484, macOS arm64, JDK 17, one fork, 3 warm-up and 5 measurement iterations of 2 s. The host was loaded, so the times are only an indication. The allocation values are exact.

Cases Time per evaluation Allocation per evaluation
json-values 115 ns 368 B
wide-context 146 ns 368 B
rules 758 ns 590 B
one-of 49.9 µs 173 KB
regex 1.40 ms 14 KB

The one-of and regex results show the next things to fix: ONE_OF scans and copies each list on each evaluation, and the backtracking regex engine is slow on long inputs that do not match.

Decisions

  • A benchmark, not a unit test. The suite exists to show size-dependent cost. A unit test cannot measure that, and it would add a slow case to every build. The setup check keeps the benchmark honest about correctness.
  • Check the results in the setup. A setup failure stops the benchmark for that case file. JMH reports the error, but the Gradle task still exits 0, so check the output for Unexpected result.
  • Pin to the fixture PR head. Add an opt-in scale evaluation suite ffe-system-test-data#78 is still open. Repin to its merge commit before this PR merges, so the pin is on the default branch.
  • No new dependency. The benchmark reads the case files with jackson-core, which the module already uses.

Contributor Checklist

Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: no release notes Changes to exclude from release notes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 82.58% (+0.00%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5a15d5c | Docs | Give us feedback!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant