Skip to content

test(ffe): cover the SPLIT reason for time-bounded sharded allocations - #73

Merged
danyal002 merged 2 commits into
mainfrom
danyal.khan/time-bounded-split-reason-fixture
Nov 11, 2026
Merged

danyal002 merged 2 commits into
mainfrom
danyal.khan/time-bounded-split-reason-fixture

Conversation

@danyal002

@danyal002 danyal002 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Time-bounded holdouts must report SPLIT for both sharded groups. Reporting DEFAULT hides the assignment reason, while incorrect allocation selection can return the wrong serial ID or logging decision. The existing bounded, single-unsharded fixture covers DEFAULT; this adds the holdout shape and consolidates the complementary coverage from #74.

Changes

Add eight cases for time-bounded-holdout-flag: four inside the holdout, two outside it, and two matching a targeting override. Every case asserts the value, reason, selected serial ID, and doLog decision. The original six subjects, salts, holdout activation metadata, and unbounded rollout are preserved.

flowchart LR
  Subject[Subject] --> Target{tier is internal?}
  Target -->|yes| Override[TARGETING_MATCH / targeted / no log]
  Target -->|no| Member{Inside holdout traffic?}
  Member -->|yes| Group[SPLIT / status-quo or shipped / log]
  Member -->|no| Rollout[STATIC / shipped / no log]
Loading

Document the subject buckets and regenerate the committed protobuf, offline wire, and manifest from the updated UFC with the existing converter lock.

Decisions

For eligible allocations, matching targeting rules take precedence when classifying the reason. Without targeting rules, a bounded allocation with one unsharded split reports DEFAULT, a sharded selection reports SPLIT, and the unbounded unsharded fallback reports STATIC. The existing date fixture supplies the bounded DEFAULT control, while this fixture keeps an unbounded STATIC fallback.

The active 2022–2050 window avoids depending on an imminent clock boundary. These cases verify evaluator output, including the logging decision; they do not prove exposure delivery or exact start/end-boundary behavior. Generated protobuf reasons remain unspecified where absent in authored UFC; generation does not imply protobuf evaluator parity.

Validation of the combined fixture:

  • dd-trace-go@69789cc762d04a203b80b54aebef7d839f5fdeca: 447 fixture evaluations pass, including all eight holdout cases and existing opt-out cases.
  • dd-trace-java@8061195730434bc7b58a8a5b38f8dad80c658168 (#13352): 448 canonical-runner tests pass, including all eight holdout cases and existing opt-out cases.
  • Java before the fix (98c28c6c01208f18053a026b471b951f2282d16b): exactly the four holdout SPLIT cases fail with actual DEFAULT; targeting, fallback, and all other tests pass.
  • Static fixture validation and all 72 Python tooling tests pass. Generation, repeat-generation, artifact integrity, catalog schema, and all nine Node tooling tests pass. The eight new cases also match the pinned JavaScript evaluator on authored JSON; their binary/wire diagnostics differ only in the documented UNKNOWN reason.

Checklist

  • JSON files are valid (no syntax errors)
  • New test cases reference existing flags in config/ufc-config.json
  • New flags have corresponding test cases in evaluation-cases/
  • Expected results match the flag configuration logic

The template's config path refers to the canonical ufc-config.json at the repository root.

Add a holdout-shaped flag: a time-bounded allocation with sharded splits
returns SPLIT, and subjects outside the holdout traffic fall through to an
unbounded rollout that returns STATIC.
@danyal002
danyal002 marked this pull request as ready for review November 10, 2026 21:21
@danyal002
danyal002 requested a review from a team as a code owner November 10, 2026 21:22
@leoromanovsky leoromanovsky added crucible: reviewing A Crucible review has started; this PR has reviewer attention. and removed crucible: reviewing A Crucible review has started; this PR has reviewer attention. labels Nov 11, 2026

@leoromanovsky leoromanovsky left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The combined fixture preserves the original holdout coverage, adds targeting precedence and selected-metadata assertions, and includes regenerated artifacts. Local Go and Java validation passes; pre-fix Java fails exactly the four expected SPLIT cases. Fixture CI passes; remaining downstream advisory jobs are still running.

@danyal002
danyal002 merged commit de191d2 into main Nov 11, 2026
10 checks passed
@danyal002
danyal002 deleted the danyal.khan/time-bounded-split-reason-fixture branch November 11, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants