Repository navigation
Cover time-bounded holdout evaluation in shared SDK fixtures - #74
Closed
leoromanovsky wants to merge 1 commit into
Closed
leoromanovsky wants to merge 1 commit into
leoromanovsky wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #73, which now combines the holdout cases with targeting precedence, serial-ID and logging assertions, and regenerated rules artifacts.
Motivation
Holdout evaluations in Java reported
DEFAULTinstead ofSPLITbecause a time window took precedence over sharding. The shared corpus covered time-bounded unsharded allocations, so it could not detect the regression fixed by dd-trace-java#13352.Changes
Add five canonical evaluations covering both holdout groups, a subject outside holdout membership, and targeting overrides for subjects in each group. They assert the returned value, reason, selected serial ID, and
doLog.flowchart LR C[Shared holdout UFC and cases] --> R[Native SDK fixture loop] R --> H[Both holdout groups: SPLIT] R --> F[Unsharded timed fallback: DEFAULT] R --> T[Targeted override: TARGETING_MATCH]Decisions
Use the existing core fixture format and real membership/group hashing, with fixed subjects documented in the README. The active window spans 2020–2099. These cases cover evaluator behavior and logging eligibility; exposure delivery and exact clock boundaries remain outside this fixture's scope.
The unchanged Java canonical runner provides the regression proof against the proposed corpus:
98c28c6: 443 passed; exactly the two holdout cases failed with expectedSPLIT, actualDEFAULT.8061195: all 445 canonical tests passed, including all five new cases and their serial-ID/logging assertions.Static fixture validation and all 72 validator tests pass. Other SDKs were not run locally. Downstream consumers must update their fixture pins after merge.
Checklist
ufc-config.json(repository root)evaluation-cases/