Repository navigation
Benchmark the FFE scale evaluation suite - #13485
Draft
leoromanovsky wants to merge 1 commit into
Draft
leoromanovsky wants to merge 1 commit into
leoromanovsky wants to merge 1 commit into
Conversation
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Adds
ScaleSuiteBenchmark, a JMH benchmark for the opt-inscaleevaluation suite of the shared FFE fixtures. It evaluates each case of the suite through the provider and reports the time and allocation of one evaluation for each case file.This PR is stacked on #13484. It pins
ffe-system-test-datato the head of DataDog/ffe-system-test-data#78, which adds the suite.Motivation
The canonical fixtures use small lists, short attribute values, and small JSON values. The
scalesuite uses sizes from production configurations:ONE_OFandNOT_ONE_OFlists with 10,000 entries;At these sizes, the problems are mainly time and memory, so a benchmark is the right place for the suite.
CanonicalFixtureTestreads onlyevaluation-cases/, and the suite is opt-in because of its size, so the unit tests do not run it.Changes
ScaleSuiteBenchmark(dd-openfeature/src/jmh):@Param cases:json-values,one-of,regex,rules,wide-context(the case files of the suite).UniversalFlagConfigParser.parse, and evaluates every case once. It throws if a value or reason differs from the expected one, so the benchmark does not measure a wrong result.Provider.ffe-system-test-datasubmodule:6df74e6→0081d8f(the head of Add an opt-in scale evaluation suite ffe-system-test-data#78). It contains the current pin.Run:
./gradlew :dd-openfeature:jmh -Pjmh.includes=ScaleSuiteBenchmark -Pjmh.profilers=gcResults
On top of #13484, macOS arm64, JDK 17, one fork, 3 warm-up and 5 measurement iterations of 2 s. The host was loaded, so the times are only an indication. The allocation values are exact.
json-valueswide-contextrulesone-ofregexThe
one-ofandregexresults show the next things to fix:ONE_OFscans and copies each list on each evaluation, and the backtracking regex engine is slow on long inputs that do not match.Decisions
Unexpected result.jackson-core, which the module already uses.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issue