Repository navigation
Add an opt-in scale evaluation suite - #78
Draft
leoromanovsky wants to merge 3 commits into
Draft
leoromanovsky wants to merge 3 commits into
leoromanovsky wants to merge 3 commits into
Conversation
The suite uses only root-configuration syntax at sizes that production configurations reach: 10,000-entry ONE_OF and NOT_ONE_OF lists with string and number attributes, regex conditions on 2 KB inputs, 300 conditions that share 10 patterns, 20 allocations of 5 conditions, a 200-attribute context, and JSON values with 719 nodes, 2,000 keys and 16 levels of nesting. It is a separate suite because its configuration is about 1.3 MB and some regex cases are slow for backtracking engines. The cases check behavior only; they define no time or memory limits. No flag uses shards, and all values are synthetic.
- Add ci/generate-scale-suite.py with --write and --check modes, and run --check in the fixture validation workflow. The suite files are no longer hand-maintained. - Give each of the 300 duplicate-pattern allocations its own variation, so a result shows which allocation won. Add cases for the first and last pattern groups. - Move the condition that tells the 20 rules apart to the end of each rule, so an evaluator runs all 5 conditions of each rule. - README: say that suites also isolate fixtures that are too large or slow for every core run, drop the production-size claim, state that JSON integers must stay integers, and document the generator. - CONTRIBUTING: describe when a standard-syntax suite is the right place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The current fixtures use small lists, short attribute values, and small JSON values. Configurations can be much larger than these fixtures:
ONE_OFlists with thousands of entries;At these sizes, SDKs show bugs and performance problems that the small fixtures do not show. For example, a backtracking regex engine takes quadratic time for
.*@(a|b)$on a long input that does not match. An SDK that converts JSON integers to doubles returns them with a different type, such as984287.0for984287.This suite gives every SDK the same large inputs, with expected results. SDK test suites can check behavior at scale, and benchmarks can use the same files as input.
Changes
ci/generate-scale-suite.py: generates the suite.--writeregenerates the files;--checkfails when they differ. The fixture validation workflow runs--check.evaluation-suites/scale/ufc-config.json: 11 flags. They use only syntax from the root configuration.scale-one-of-strings,scale-not-one-of-strings: 10,000-entry string lists, sorted in byte order.scale-one-of-numbers: a 10,000-entry list of numeric strings, checked with integer and string attributes.scale-regex-wildcard:.*@(alpha|bravo)\.(example|test)$on 2 KB inputs.scale-regex-charclass:^[a-zA-Z0-9._%+-]+@scale\.test$on 2 KB inputs.scale-regex-duplicates: 300 allocations that share 10 patterns, each with its own variation, then a final pattern.scale-deep-rules: 20 allocations of 5 conditions (GTE,LT,ONE_OF,MATCHES,ONE_OF). The condition that tells the allocations apart comes last, so every condition runs.scale-wide-context: a rule on the last of 200 attributes.scale-json-large(719 nodes) andscale-json-wide(2,000 keys) use strings with non-ASCII characters, integers, exact binary floats, booleans, and nulls.scale-json-deep(16 levels of nesting) uses integers and booleans.evaluation-suites/scale/cases/: 33 cases in 5 files (one-of.json,regex.json,rules.json,wide-context.json,json-values.json). They check first, middle, last, and absent list entries, near misses, which duplicate-pattern allocation wins, matches and misses on long inputs, and full JSON value equality.README.md: the suite in the directory tree, in the suite table, and a "Scale suite" section. The section states that JSON integers must stay integers, and how to regenerate the files. The suite introduction now says that suites also isolate fixtures that are too large or slow for every core run.CONTRIBUTING.md: when a standard-syntax suite is the right place, and how to generate large fixtures.Decisions
.test,.example,.invalid).test-json-number-precision.jsonthat Add JSON number precision fixture #77 adds. The suite still requires an integer to stay an integer: a result of5.0for5fails.Verification
python3 ci/generate-scale-suite.py --check: the files match the generator. A hand edit to one expected value makes it fail.python3 ci/validate-fixtures.py: validated 4 suites, 486 cases in 65 files.python3 -m unittest ci/test_validate_fixtures.py: 30 tests pass.scale.Checklist
ufc-config.jsoncases/