Skip to content

Add explicit evaluation reasons to UFC fixtures - #76

Draft
typotter wants to merge 2 commits into
mainfrom
typo/ufc-explicit-reasons
Draft

typotter wants to merge 2 commits into
mainfrom
typo/ufc-explicit-reasons

Conversation

@typotter

Copy link
Copy Markdown
Contributor

Motivation

The protobuf evaluator uses the reason supplied on each split. UFC fixtures omit that metadata, so successful evaluations return UNKNOWN. Add explicit reasons to the fixture inputs so protobuf evaluation can report why a value was selected.

Changes

  • Add 229 split reasons across the core, flag-selection and dependent-flags UFC configurations.
  • Regenerate the core protobuf, ConfigurationWire and artifact manifest.
  • Preserve all evaluation cases and expectations, flag values, targeting rules, shard ranges and malformed-input structures.

Decisions

Use SPLIT when non-trivial sharding determines the assignment, ahead of TARGETING_MATCH. Treat full-range shards as no splitting when choosing the reason, while preserving their original structure. Use STATIC for unconditional first/only allocations without dates, and DEFAULT for later catch-alls or time-window-only allocations.

Leave two intentionally malformed split objects unannotated. The converter continues to preserve source data without synthesising reasons or normalising fixtures.

Validation: 72 Python tests, nine Node tests, fixture/schema/integrity checks and repeat generation pass. Original JSON evaluation remains unchanged. Offline Chromium lifecycle passes. The core browser corpus reports 309 passed, 46 failed and 20 conversion omissions. The failures comprise 37 reason-expectation mismatches, six existing missing-PARSE_ERROR cases and three unavailable feature-destination observations. Expectations remain unchanged so these differences stay visible. Named suites received static validation; this run does not claim their browser execution.

Checklist

  • JSON files are valid (no syntax errors)
  • New test cases reference existing flags in config/ufc-config.json — not applicable; no new cases
  • New flags have corresponding test cases in evaluation-cases/ — not applicable; no new flags
  • Expected results match the flag configuration logic — reason differences remain visible; expectations are intentionally unchanged

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The full-range shard in ufc-config.json incorrectly uses SPLIT instead of TARGETING_MATCH, causing the wrong protobuf evaluation reason and requiring correction plus regeneration of the core artifacts and manifest.

1 open issue
What changed in this PR

Adds explicit split reasons to UFC fixtures so protobuf evaluation can report why a value was selected instead of returning UNKNOWN.

Changes:

  • Adds 229 reasons across the core, flag-selection, and dependent-flags configurations.
  • Regenerates core protobuf and ConfigurationWire artifacts and updates manifest hashes.
  • Preserves evaluation expectations, targeting logic, shard ranges, and malformed-input structures.
File Description
ufc-config.json Adds reasons to core fixture splits.
rules-resources/​generated/​manifest.json Updates source and generated artifact hashes.
rules-resources/​generated/​canonical-evaluator/​configuration-wire.json Refreshes the encoded configuration with split reasons.
evaluation-suites/​flag-selection/​ufc-config.json Adds static reasons to flag-selection fixtures.
evaluation-suites/​dependent-flags/​ufc-config.json Adds static, targeting, and default reasons to dependency fixtures.

Review effort: Balanced

Comment thread ufc-config.json Outdated
}
]
],
"reason": "SPLIT"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants