[azure-ai-evaluation] DocumentRetrievalEvaluator fidelity sorts labels as strings; BLEU/GLEU/METEOR tokenize response and ground truth differently #49830
Description
- Package: azure-ai-evaluation 1.18.10 (code unchanged on main at 8371069)
- Python: 3.9, Linux
Two metric-correctness bugs in built-in evaluators. Both give wrong scores silently, without errors or warnings.
1. DocumentRetrievalEvaluator: Fidelity is wrong when ground_truth_label_max >= 10
_document_retrieval.py, _compute_fidelity → calculate_weighted_sum_by_rating:
label_counts = {str(i): 0 for i in range(s, self.ground_truth_label_max + 1)}
...
sorted_label_counts = [x[1] for x in sorted(label_counts.items(), key=lambda x: x[0])]
weights = [(math.pow(2, i + 1) - 1) for i in range(s, self.ground_truth_label_max + 1)]The counts are keyed by str(label) and sorted as strings ("1", "10", "2", …), but the weights are in numeric order. When the scale goes up to 10 or more, counts are multiplied by the wrong weights. The default 0–4 scale is not affected.
Repro:
from azure.ai.evaluation import DocumentRetrievalEvaluator
e = DocumentRetrievalEvaluator(ground_truth_label_min=0, ground_truth_label_max=10)
gt = [{"document_id": "a", "query_relevance_label": 10},
{"document_id": "b", "query_relevance_label": 2}]
ret = [{"document_id": "b", "relevance_score": 1.0}]
print(e(retrieval_ground_truth=gt, retrieved_documents=ret)["document_retrieval_properties"]["fidelity"])
# 0.6818..., and fidelity_result is "pass"Only the low-relevance document was retrieved, so the expected Fidelity is (2³−1) / ((2¹¹−1) + (2³−1)) ≈ 0.0034. The evaluator reports 0.68 and marks it as passing the 0.5 threshold.
Fix: key the counts by int (or index a list by label - s) so counts and weights stay in the same order.
2. BLEU / GLEU / METEOR: the tokenizer is chosen separately for the response and the ground truth
_common/utils.py, nltk_tokenize:
if not text.isascii():
tokens = NISTTokenizer().international_tokenize(text)
else:
tokens = nltk.word_tokenize(text)_bleu.py, _gleu.py and _meteor.py call this once for the ground truth and once for the response. If only one side contains a single non-ASCII character (a curly quote, an em dash, "café"), the two sides are tokenized differently, and ordinary words stop matching:
from azure.ai.evaluation import BleuScoreEvaluator
from azure.ai.evaluation._common.utils import nltk_tokenize
ref = "The U.S. economy didn't grow; prices rose 5.00 percent in March."
b = BleuScoreEvaluator()
b(response=ref, ground_truth=ref)["bleu_score"] # 1.0
b(response=ref + " —", ground_truth=ref)["bleu_score"] # 0.446
b(response=ref + " -", ground_truth=ref)["bleu_score"] # 0.925 (ASCII hyphen, for comparison)
nltk_tokenize(ref) # ['The', 'U.S.', 'economy', 'did', "n't", ...]
nltk_tokenize(ref + " —") # ['The', 'U', '.', 'S', '.', 'economy', 'didn', "'", 't', ...]LLM responses often contain typographic quotes and dashes while references are plain ASCII, so this can move scores a lot on real data.
Fix: pick one tokenizer per (response, ground truth) pair, for example the international tokenizer if either text is non-ASCII, and use it for both sides. Add a unit test where only one side has a non-ASCII character.
I'm happy to send a PR with fixes and tests for both if that's welcome.
Activity
github-actions commentedon Nov 4, 2026
Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.
//cc: Hanchi Wang (@luigiw) Neehar Duvvuri (@needuv) Ankit Singhal (@singankit)
github-actions commentedon Nov 4, 2026
🎯 Agentic Issue Triage
Summary: Two silent metric-correctness bugs in
azure-ai-evaluationbuilt-in evaluators:DocumentRetrievalEvaluatorfidelity miscomputes whenground_truth_label_max >= 10due to lexicographic (string) sorting of integer label keys, and the BLEU/GLEU/METEOR tokenizer in_common/utils.pycan pick different tokenization strategies for the response vs. ground truth when only one side contains a non-ASCII character.📋 Issue Details
Package:
azure-ai-evaluation1.18.10 (confirmed unchanged on main at commit 8371069)Affected API:
DocumentRetrievalEvaluator._compute_fidelity/calculate_weighted_sum_by_rating(_document_retrieval.py);nltk_tokenize(_common/utils.py) used byBleuScoreEvaluator,GleuScoreEvaluator,MeteorScoreEvaluatorScenarios:
ground_truth_label_max >= 10scale produces a wrong weighted sum because label counts are keyed bystr(label)and sorted as strings ("1", "10", "2", ...) while weights are generated in numeric order, misaligning counts to weights.nltk_tokenizechooses the NIST international tokenizer vs.nltk.word_tokenizeindependently per string, producing incompatible token boundaries for otherwise-identical ASCII text.Root ask: Fix both correctness bugs (keying counts by
int/index instead of string-sorted keys; selecting a single shared tokenizer per response/ground-truth pair based on whether either side is non-ASCII) and add regression tests. Author has offered to submit a PR.🔎 Debugging / Reproduction Notes
Both repros in the issue are self-contained and appear logically sound based on reading the referenced source snippets (not independently executed in this triage pass):
0.68reported vs. an expected ≈0.0034, withfidelity_resultincorrectly marked "pass" against the 0.5 threshold — a false negative on a real quality signal.bleu_scoredropping from1.0to0.446when an em dash is appended to only one side, versus0.925for an ASCII hyphen, demonstrating the asymmetry is specific to the non-ASCII branch selection.No related open issues were found for either specific bug via search.
Suggested investigation steps:
mainto confirm the described behavior before merging a fix.label - sor usingintkeys) against scales both below and at/above 10 to avoid regressing the default 0–4 scale.🏷️ Label Confidence
Client— azure-ai-evaluation is a data-plane client package (notazure-mgmt-*).Evaluation— issue explicitly references theazure-ai-evaluationpackage and files under/sdk/evaluation/.👥 Owner Routing
# ServiceLabel: %Evaluation(line 257) — exact match scanning bottom-to-top for the predicted service label "Evaluation".@luigiw,@needuv,@singankit# ServiceLabel:entries for "Evaluation" were encountered; this was the first and only match found scanning upward from the end of CODEOWNERS.14H034160212 commentedon Nov 6, 2026
Duplicate of #49829 (opened twice by mistake).