Skip to content

Add an opt-in scale evaluation suite - #78

Draft
leoromanovsky wants to merge 3 commits into
mainfrom
leo.romanovsky/scale-suite
Draft

leoromanovsky wants to merge 3 commits into
mainfrom
leo.romanovsky/scale-suite

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The current fixtures use small lists, short attribute values, and small JSON values. Configurations can be much larger than these fixtures:

  • ONE_OF lists with thousands of entries;
  • regex conditions on long attribute values, such as user agents;
  • many conditions that share the same patterns;
  • many allocations for each flag;
  • JSON variant values with hundreds of nodes.

At these sizes, SDKs show bugs and performance problems that the small fixtures do not show. For example, a backtracking regex engine takes quadratic time for .*@(a|b)$ on a long input that does not match. An SDK that converts JSON integers to doubles returns them with a different type, such as 984287.0 for 984287.

This suite gives every SDK the same large inputs, with expected results. SDK test suites can check behavior at scale, and benchmarks can use the same files as input.

Changes

  • ci/generate-scale-suite.py: generates the suite. --write regenerates the files; --check fails when they differ. The fixture validation workflow runs --check.
  • evaluation-suites/scale/ufc-config.json: 11 flags. They use only syntax from the root configuration.
    • scale-one-of-strings, scale-not-one-of-strings: 10,000-entry string lists, sorted in byte order.
    • scale-one-of-numbers: a 10,000-entry list of numeric strings, checked with integer and string attributes.
    • scale-regex-wildcard: .*@(alpha|bravo)\.(example|test)$ on 2 KB inputs.
    • scale-regex-charclass: ^[a-zA-Z0-9._%+-]+@scale\.test$ on 2 KB inputs.
    • scale-regex-duplicates: 300 allocations that share 10 patterns, each with its own variation, then a final pattern.
    • scale-deep-rules: 20 allocations of 5 conditions (GTE, LT, ONE_OF, MATCHES, ONE_OF). The condition that tells the allocations apart comes last, so every condition runs.
    • scale-wide-context: a rule on the last of 200 attributes.
    • scale-json-large (719 nodes) and scale-json-wide (2,000 keys) use strings with non-ASCII characters, integers, exact binary floats, booleans, and nulls. scale-json-deep (16 levels of nesting) uses integers and booleans.
  • evaluation-suites/scale/cases/: 33 cases in 5 files (one-of.json, regex.json, rules.json, wide-context.json, json-values.json). They check first, middle, last, and absent list entries, near misses, which duplicate-pattern allocation wins, matches and misses on long inputs, and full JSON value equality.
  • README.md: the suite in the directory tree, in the suite table, and a "Scale suite" section. The section states that JSON integers must stay integers, and how to regenerate the files. The suite introduction now says that suites also isolate fixtures that are too large or slow for every core run.
  • CONTRIBUTING.md: when a standard-syntax suite is the right place, and how to generate large fixtures.

Decisions

  • A separate, opt-in suite. CONTRIBUTING used suites only for syntax that not every SDK supports. This suite uses standard syntax, but its configuration is about 1.3 MB, and some regex cases are slow for backtracking engines. As a suite, it is parsed only by consumers that enable it, so the core runs stay fast.
  • Behavior only, no time or memory limits. This repository is a behavioral contract. Timing depends on the machine, so a limit here would fail at random in downstream CI. Benchmarks can measure time on the same inputs.
  • No shards. Each expected result follows from which allocation matches, not from a hash. A new SDK can check the results without a reference implementation.
  • Generated, with a CI check. The files come from a seeded generator, so a maintainer changes the generator instead of 40,000 lines of JSON.
  • Synthetic data only. Email-like values use reserved domains (.test, .example, .invalid).
  • JSON integers stay within ±2^53. A platform with only double-precision numbers can still represent every value exactly. Integers above 2^53 are a separate contract, in the core case test-json-number-precision.json that Add JSON number precision fixture #77 adds. The suite still requires an integer to stay an integer: a result of 5.0 for 5 fails.

Verification

  • python3 ci/generate-scale-suite.py --check: the files match the generator. A hand edit to one expected value makes it fail.
  • python3 ci/validate-fixtures.py: validated 4 suites, 486 cases in 65 files.
  • python3 -m unittest ci/test_validate_fixtures.py: 30 tests pass.
  • A dd-openfeature (Java) runner over the suite:
  • The downstream compatibility jobs are inconclusive at both base and head, and they do not run feature suites, so they give no signal for scale.

Checklist

  • JSON files are valid (no syntax errors)
  • New test cases reference existing flags in the suite's ufc-config.json
  • New flags have corresponding test cases in the suite's cases/
  • Expected results match the flag configuration logic

The suite uses only root-configuration syntax at sizes that production
configurations reach: 10,000-entry ONE_OF and NOT_ONE_OF lists with
string and number attributes, regex conditions on 2 KB inputs, 300
conditions that share 10 patterns, 20 allocations of 5 conditions, a
200-attribute context, and JSON values with 719 nodes, 2,000 keys and
16 levels of nesting.

It is a separate suite because its configuration is about 1.3 MB and
some regex cases are slow for backtracking engines. The cases check
behavior only; they define no time or memory limits. No flag uses
shards, and all values are synthetic.
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
- Add ci/generate-scale-suite.py with --write and --check modes, and run
  --check in the fixture validation workflow. The suite files are no
  longer hand-maintained.
- Give each of the 300 duplicate-pattern allocations its own variation,
  so a result shows which allocation won. Add cases for the first and
  last pattern groups.
- Move the condition that tells the 20 rules apart to the end of each
  rule, so an evaluator runs all 5 conditions of each rule.
- README: say that suites also isolate fixtures that are too large or
  slow for every core run, drop the production-size claim, state that
  JSON integers must stay integers, and document the generator.
- CONTRIBUTING: describe when a standard-syntax suite is the right place.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant