Skip to content

FIX recognize Azure SelfHarm in scorer evaluation - #2616

Open
Imran Ahamed (immu4989) wants to merge 1 commit into
microsoft:mainfrom
immu4989:fix/scorer-evaluation-category-mapping
Open

FIX recognize Azure SelfHarm in scorer evaluation#2616
Imran Ahamed (immu4989) wants to merge 1 commit into
microsoft:mainfrom
immu4989:fix/scorer-evaluation-category-mapping

Conversation

@immu4989

Copy link
Copy Markdown
Contributor

Description

Azure Content Safety returns SelfHarm, which is not recognized by HarmCategory.parse_many. Scorer evaluation therefore rejects it against the registered self_harm dataset category, both when it is the only score and when selecting from all four Azure scores.

Add the explicit selfharm alias for SELF_HARM. This completes the remaining Azure case after #2575 while preserving strict rejection of unrelated categories.

Related to #2570; reproduction and proposed scope: #2570 (comment).

Tests and Documentation

  • Parameterized regression coverage for all four registered Azure categories, with single-score and multi-score results. Both SelfHarm cases fail before the fix and pass afterward.
  • Evaluator, Azure Content Safety scorer, and harm-category unit tests: 118 passed.
  • All applicable pre-commit checks passed, including Ruff and type checking.
  • No API calls or dataset/metrics changes; no documentation changes needed for this alias correction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant