Repository navigation
[azure-ai-evaluation] DocumentRetrievalEvaluator fidelity is wrong when ground_truth_label_min < 0 (label 0 treated as unlabeled) #49877
Description
Package: azure-ai-evaluation 1.18.11. The code is unchanged on main at ef67a44.
I noticed this while working on #49829 and #49850. Those fix a different fidelity bug, for label scales of 10 or more. This one is separate.
Describe the bug
DocumentRetrievalEvaluator accepts any integer ground_truth_label_min, including negative ones. Some judgement sets use them. The TREC Web track, for example, labels junk pages -2. The fidelity code assumes that the configured minimum means "irrelevant", so on a scale of -1 to 2, label 0 is the lowest relevant label.
The evaluator also uses 0 as a placeholder for retrieved documents that have no ground-truth label (holes), and treats "every retrieved label is 0" as "no retrieved document is labeled". When ground_truth_label_min < 0:
- If every retrieved document is labeled 0, all metrics are set to 0, fidelity included, even though relevant documents were retrieved. Retrieving one more irrelevant document makes fidelity go from 0 to 1.
- A hole gets label 0 and counts as a relevant document in fidelity. An irrelevant document plus a hole gives fidelity 1.
Scales that start at 0 or above are not affected.
To reproduce
from azure.ai.evaluation import DocumentRetrievalEvaluator
ev = DocumentRetrievalEvaluator(ground_truth_label_min=-1, ground_truth_label_max=2)
gt = [
{"document_id": "a", "query_relevance_label": 0}, # lowest relevant label
{"document_id": "b", "query_relevance_label": -1}, # irrelevant
]
def fidelity(ids):
docs = [{"document_id": d, "relevance_score": float(len(ids) - i)} for i, d in enumerate(ids)]
p = ev(retrieval_ground_truth=gt, retrieved_documents=docs)["document_retrieval_properties"]
return p["fidelity"], p["top3_max_relevance"]
print(fidelity(["a"])) # (0, 0) expected (1.0, 0)
print(fidelity(["a", "b"])) # (1.0, 0)
print(fidelity(["b", "x"])) # (1.0, 0) expected (0, -1): "x" has no labelExpected behavior
- Retrieving only
a, the only relevant document, gives fidelity 1.0, the same as retrievingaandb. - A document without a ground-truth label does not count as relevant, so
["b", "x"]gives fidelity 0.
Root cause
In azure/ai/evaluation/_evaluators/_document_retrieval/_document_retrieval.py on main:
- Line 422 gives holes the label
0:
qrels_lookup[doc_id] if doc_id in qrels_lookup else 0 - Line 431,
if not any(result_docs_groundtruth_labels):, is commented "if none of the retrieved docs are labeled". It is true whenever every label is 0, so a real label 0 is handled like a hole. calculate_weighted_sum_by_rating(line 199) counts labels fromground_truth_label_min + 1up. With a minimum of -1 that includes 0, so holes count as relevant.
When the minimum is 0 or above, a real label 0 is either impossible or is the "irrelevant" minimum, so both shortcuts give the right result there.
Suggested fix
- Give holes the label
min(0, ground_truth_label_min). That is 0 as before on scales that start at 0 or above, and the irrelevant minimum on scales that start below 0. - Replace
not any(labels)withall(label == hole_label for label in labels). In the shortcut, reporttop1_relevanceandtop3_max_relevanceashole_label.
Both changes leave the results unchanged for scales that start at 0 or above. I have a PR with regression tests ready and can open it.
Related, not covered by the fix above
NDCG@3 and XDCG@3 use the absolute label value, so they are not well defined for negative labels. The gain 2**label - 1 is negative below 0, and label 0 has zero gain even when it is relevant. For example, retrieving only b (label -1) on main gives ndcg@3 = 1.58. The fix above sends that case through the shortcut, so it reports 0, but other orderings that mix label -1 and label 0 can still give NDCG above 1. Fidelity weights have the same problem: with ground_truth_label_min=-2, label -1 is treated as relevant but gets weight 2**0 - 1 = 0.
Could the maintainers say whether negative label scales are meant to be supported? If they are, the gains could be computed relative to ground_truth_label_min, though that would change NDCG and XDCG for scales that start above 0. If they are not, __init__ could reject ground_truth_label_min < 0.
Activity
github-actions commentedon Nov 7, 2026
Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.
//cc: Hanchi Wang (@luigiw) Neehar Duvvuri (@needuv) Ankit Singhal (@singankit)
github-actions commentedon Nov 7, 2026
🎯 Agentic Issue Triage
Summary:
DocumentRetrievalEvaluatormis-computes fidelity and related metrics whenground_truth_label_min < 0, because retrieved documents with the real label0are indistinguishable from unlabeled "holes" that are also assigned label0.📋 Issue Details
Package:
azure-ai-evaluation1.18.11 (reproduces onmainatef67a44134)Affected API:
DocumentRetrievalEvaluatorinazure/ai/evaluation/_evaluators/_document_retrieval/_document_retrieval.py(hole-labeling at line ~422, shortcut check at line ~431,calculate_weighted_sum_by_ratingat line ~199)Scenarios:
ground_truth_label_minis negative (e.g., -1 or -2, as used by TREC Web track judgment sets), a real ground-truth label of0is treated the same as an unlabeled document hole.Root ask: Fix fidelity (and confirm intended behavior for NDCG@3/XDCG@3) so negative label scales are handled correctly, or document/reject negative
ground_truth_label_minif unsupported. Author states they already have a PR with regression tests ready to open.🔎 Debugging / Reproduction Notes
The reporter provides a clear, minimal, runnable repro and a precise root-cause analysis pointing to three specific lines/behaviors in
_document_retrieval.py:0regardless ofground_truth_label_min.not any(result_docs_groundtruth_labels)is used as a "no labeled docs" shortcut, which is also true whenever every present label is a genuine0.calculate_weighted_sum_by_ratingcounts fromground_truth_label_min + 1, which includes0as "relevant" when the minimum is negative, causing holes to count as relevant.The reporter also flags a related, separate concern (not part of the main fix): NDCG@3/XDCG@3 gain formula
2**label - 1is not well-defined for negative labels and can exceed 1 for certain negative-label orderings; this needs a maintainer decision on whether negative scales are supported at all.A quick search of open issues did not surface an existing duplicate for this specific negative-label fidelity bug.
Suggested investigation steps:
hole_label = min(0, ground_truth_label_min)andall(label == hole_label for label in labels)) against the repro script and confirm it leaves scales starting at 0+ unchanged.ground_truth_label_minscales are officially supported; if yes, adjust NDCG@3/XDCG@3 gain calculation relative to the minimum, or if no, add validation in__init__to reject negative values.🏷️ Label Confidence
Client—azure-ai-evaluationis a data-plane SDK package (notazure-mgmt-*).Evaluation— issue explicitly names the package and file path underazure/ai/evaluation/_evaluators/_document_retrieval/, matching thesdk/evaluationdirectory.Client+Evaluationpairing.👥 Owner Routing
# ServiceLabel: %Evaluation(around line 257) — directly matches the predicted service label "Evaluation".@luigiw,@needuv,@singankit%Event Grid(with AzureSdkOwners) appeared just below but does not match the "Evaluation" label; the%Evaluationentry was the first (and only) match for the predicted service label.