Skip to content

Resolve feature flag attributes once per evaluation and cache split feature metadata - #13489

Draft
leoromanovsky wants to merge 3 commits into
leo.romanovsky/ffe-scale-suite-benchmarkfrom
leo.romanovsky/ffe-allocation-scan
Draft

leoromanovsky wants to merge 3 commits into
leo.romanovsky/ffe-scale-suite-benchmarkfrom
leo.romanovsky/ffe-allocation-scan

Conversation

@leoromanovsky

Copy link
Copy Markdown
Contributor

What Does This Do

This PR is stacked on #13485, which adds the scale-suite benchmarks and the baseline from master. Review only the commits after it. It is a sibling of #13484, #13486, and #13487.

Evaluating a flag with many allocations costs less time and memory, with the same results:

  • Each evaluation reads a context attribute once, however many allocations and conditions use it. Before, every condition read the attribute from the context again and converted its value, and the conversion boxed numbers.
  • A split's feature metadata (the __dd_feature.*, __dd_exposure_feature.* and __dd_evaluation_feature.* entries) is built once, the first time the split is served, and cached on the split. Before, each evaluation that served the split concatenated a new key for each feature and destination.

The served allocation, variant, reason, error codes, and metadata do not change.

Motivation

We need to win a performance-sensitive customer, and the scale benchmarks of #13485 are the basis for this work. In those benchmarks, a flag with 1,000 allocations (scale/variations) costs 53–61 µs and 28 KB per evaluation, and its worst case costs 113 µs and 83 KB. A profile of the worst cases shows two costs:

  • In scale/rules-sweep (up to 2,500 conditions over a few attributes), most of the time is in Structure.convertValue, because each condition converts the same numeric attribute again.
  • In scale/variations, the matched split carries 200 features for 3 destinations. Each evaluation builds 600 metadata keys by string concatenation, which is about 48 KB of the 83 KB.
flowchart LR
  subgraph Before
    A1[allocation 1..N] --> C1[each condition: context.getValue + convertValue]
    S1[matched split] --> F1[each evaluation: prefix + key for every feature and destination]
  end
  subgraph After
    A2[allocation 1..N] --> E2[EvaluationScope: resolve once, then reuse]
    S2[matched split] --> F2[SplitFeatures, built on first serve and cached on the split]
  end
Loading

Additional Notes

Scale suite results

Same machine and settings as the #13485 baseline: AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. The master (#13485) and branch jars ran back to back. Other benchmarks and builds shared the machine, with a load average of 10–12, so differences of about 10% or less are within noise. Rows marked "all" are the mean over the cases of the file.

Case master (#13485) This PR Change
scale/variations, all 61.0 µs (±15.3), 28,067 B 37.1 µs (±4.3), 9,627 B 1.6× faster, 66% less allocation
scale/variations, matches allocation 999 (200 features) 113 µs, 83,288 B 73 µs, 27,920 B 1.5× faster, 66% less allocation
scale/variations, matches the fallback after 1,000 allocations 50.7 µs, 392 B 49.9 µs, 432 B within noise
scale/rules-sweep, all 38.4 µs (±17.6), 392 B 32.6 µs (±4.7), 893 B within noise
scale/rules-sweep, 50x10x5 match (2,500 conditions) 96.6 µs, 392 B 85.0 µs, 528 B 12% faster
scale/rules 3.53 µs, 1,904 B 3.06 µs, 2,016 B 13% faster
scale/time-windows 847 ns, 368 B 1,066 ns, 408 B within noise: separate runs per case gave 795 / 981 ns before and 819 / 1,001 ns after
scale/wide-context 724 ns, 368 B 640 ns, 408 B within noise
scale/flags 264 ns (±107), 330 B 232 ns, 360 B within noise

An earlier back-to-back pair on the same machine gave the same picture. For example, the rules-sweep 50x10x5 match went from 102 µs to 78 µs there.

The remaining 28 KB on the 200-feature case is the HashMap behind ImmutableMetadata: about 600 entries and its resizes. The OpenFeature builder takes entries one at a time and cannot be presized. The remaining time of the 1,000-allocation scan is mostly the ONE_OF list scan, which #13486 replaces with an index.

ScaleSuiteLoadBenchmark is not affected: the configuration parse does no new work.

Decisions

  • One per-evaluation object. EvaluationScope holds the context and the attributes resolved so far. It is created only when the first allocation with rules is reached, and evaluateRules and evaluateCondition take it in place of the context. The shard-hashing and SemVer sibling PRs also keep state for one evaluation, and they can add their state to this object.
  • Small cost for flags with a single condition. The first attribute is kept in fields. Arrays are allocated only for a second attribute, and a HashMap only past 8 attributes, so lookups do not grow quadratically on very wide flags. A flag with one condition pays one 40-byte object, which accounts for the +30–40 B above. A flag without rules pays nothing.
  • Same attribute semantics. The id attribute still falls back to the targeting key only when the context has no id key. An explicit id that maps to null stays null. An attribute that is read once and then changes in the context during the same evaluation is not read again.
  • Feature metadata cached on the split, not in the parser. The configuration model (ufc) does not depend on the evaluator. Split caches what the evaluator derives from its features. The entries have the same keys, the same value types (numbers become doubles, as before), and the same rule that the first feature of each key wins. The warning for an unknown destination still fires when a split is served, once per destination name.
  • Out of scope. ONE_OF lookups (Index ONE_OF and NOT_ONE_OF lists when the configuration loads #13486), regex (Evaluate feature flag regex conditions in linear time #13487), shard hashing, and SemVer parsing belong to sibling PRs, and this PR does not change those code paths.

Testing

  • EvaluationScopeTest: each attribute is read from the context once, below and past 8 attributes, a missing attribute is cached as missing, and the id fallback is checked, including an explicit null id.
  • SplitFeaturesTest: the entries for each destination, first-key-wins deduplication, incomplete and unsupported features, numbers widened to doubles, the evaluation-features marker, one shared empty preparation, the warning for unknown destinations, and that the split computes its entries once.
  • CanonicalFixtureTest still checks the exact HOOK, EXPOSURE, and EVALUATION feature maps, serialId, doLog, and the allocation key for every fixture case. The relevant files include test-case-split-features, test-case-serial-id-metadata, test-case-experimentation-opt-out-do-log, test-case-start-and-end-date-flag, test-case-microsecond-date-flag, and test-case-time-bounded-holdout-flag. The FlagEvaluatorTest feature and warning tests pass unchanged.

Contributor Checklist

🤖 Generated with Claude Code

leoromanovsky and others added 3 commits November 17, 2026 02:40
… at load

A flag with many allocations re-read the same context attributes in every
allocation: a context lookup and a value conversion each time, and the
conversion boxes numbers. Each evaluation now keeps the attributes it has
resolved, so a flag with 1,000 allocations reads its attribute once.

A matched split's features were turned into metadata entries on every
evaluation, concatenating a prefixed key for each feature and destination.
The split now resolves its entries once, when it is built, and an
evaluation only copies them. Unknown destinations are now reported when
the configuration loads, still once per name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The split caches what the evaluator derives from its features instead of
building it in its constructor, so the configuration model does not
depend on the evaluator and an unknown destination is still reported
when a split is served, as before. The allocation and rule scans keep
their for-each loops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Most flags read one attribute, so the evaluation scope keeps it in fields
and allocates its arrays only for a second attribute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@datadog-official

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 82.62%

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 462d85c | Docs | Give us feedback!

@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.04 s 14.04 s [-0.8%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent 12.87 s 13.08 s [-2.4%; -0.8%] (maybe better)
startup:petclinic:appsec:Agent 17.69 s 17.55 s [-0.1%; +1.7%] (no difference)
startup:petclinic:iast:Agent 16.85 s 17.65 s [-8.7%; -0.4%] (maybe better)
startup:petclinic:profiling:Agent 17.20 s 17.28 s [-1.5%; +0.6%] (no difference)
startup:petclinic:sca:Agent 17.74 s 17.58 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:tracing:Agent 16.71 s 16.34 s [-2.0%; +6.6%] (no difference)

Commit: 462d85cc · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant