Skip to content

Cover time-bounded holdout evaluation in shared SDK fixtures - #74

Closed
leoromanovsky wants to merge 1 commit into
mainfrom
leoromanovsky/holdout-reason-fixtures
Closed

leoromanovsky wants to merge 1 commit into
mainfrom
leoromanovsky/holdout-reason-fixtures

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Superseded by #73, which now combines the holdout cases with targeting precedence, serial-ID and logging assertions, and regenerated rules artifacts.

Motivation

Holdout evaluations in Java reported DEFAULT instead of SPLIT because a time window took precedence over sharding. The shared corpus covered time-bounded unsharded allocations, so it could not detect the regression fixed by dd-trace-java#13352.

Changes

Add five canonical evaluations covering both holdout groups, a subject outside holdout membership, and targeting overrides for subjects in each group. They assert the returned value, reason, selected serial ID, and doLog.

flowchart LR
    C[Shared holdout UFC and cases] --> R[Native SDK fixture loop]
    R --> H[Both holdout groups: SPLIT]
    R --> F[Unsharded timed fallback: DEFAULT]
    R --> T[Targeted override: TARGETING_MATCH]
Loading

Decisions

Use the existing core fixture format and real membership/group hashing, with fixed subjects documented in the README. The active window spans 2020–2099. These cases cover evaluator behavior and logging eligibility; exposure delivery and exact clock boundaries remain outside this fixture's scope.

The unchanged Java canonical runner provides the regression proof against the proposed corpus:

  • Before the fix, 98c28c6: 443 passed; exactly the two holdout cases failed with expected SPLIT, actual DEFAULT.
  • With the fix, 8061195: all 445 canonical tests passed, including all five new cases and their serial-ID/logging assertions.

Static fixture validation and all 72 validator tests pass. Other SDKs were not run locally. Downstream consumers must update their fixture pins after merge.

Checklist

  • JSON files are valid (no syntax errors)
  • New test cases reference existing flags in ufc-config.json (repository root)
  • New flags have corresponding test cases in evaluation-cases/
  • Expected results match the flag configuration logic

@leoromanovsky
leoromanovsky marked this pull request as ready for review November 10, 2026 22:59
@leoromanovsky
leoromanovsky requested a review from a team as a code owner November 10, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant