[v0.5.0] benchmark scoring — correctness scorer with manual-adjudication hooks (CodeRabbit follow-up tests)

[ayhammouda]

Refs #63.
Refs #88 (already closed by #100 — this is a follow-up, not a re-open).

Summary

Follow-up to #100 (merged @ 89921f3). CodeRabbit's re-triggered review on
#100 completed roughly one minute after the merge, with 2 Trivial
nitpicks on tests/benchmarks/test_scoring.py — both valid. The fixes were
pushed to the original agent/88-correctness-scorer branch (f37987c), but
since #100 had already merged, that commit could not land through it. This
PR cherry-picks that single commit onto current main.

Purely additive changes to the test file introduced by #100 itself — no
merged-at-the-time tests, no source files, and no other paths touched.

CodeRabbit triage (findings from the #100 review)

#	Finding	Severity	Triage	Resolution
1	No test asserts per-tool-model rollup values (all tests used a single competitor)	Trivial / quick win	Valid — fixed	Added test_score_run_produces_per_tool_model_rollups: two competitors, one carrying an explicit id:provider/model tool_model_key; asserts per-entry total_cells/scored_cells/pending_manual_scoring_cells and the persisted scoring-rollups.json. Adapted from CodeRabbit's suggested diff — the real rollup key is by_tool_model_key, not by_tool_model (the suggestion as written would KeyError).
2	is_error: True tool calls untested for grounding-evidence inference	Trivial / low value	Valid — fixed	Added test_score_run_does_not_count_error_tool_calls_as_grounding_evidence: an errored tool call leaves grounding_evidence_present False in both the scoring record and the adjudication-queue entry, matching _has_grounding_evidence's successful-calls-only behavior.

Detailed reply also posted on the #100 thread.

CodeRabbit status on this PR: rate limited again (both bots), no
findings produced; the CodeRabbit check itself reports pass. Substantively
moot — this diff consists of the two tests CodeRabbit itself suggested in
its #100 review, adapted only where its template referenced a nonexistent
key (by_tool_model vs the real by_tool_model_key).

Why this approach

#100 merged before the review landed, so the review fixes need their own PR
— agents never commit to main directly (AGENT-EXECUTION-PIPELINE.md §1).
Cherry-picking the already-validated commit keeps the change identical to
what was verified on the original branch.

Supervisor-review check

No §7 trigger fires: diff is +69 lines in one test file, additive only; no
forbidden-territory path, no dependency change, no pyproject.toml/uv.lock
change, no network or async introduced.

Validation gate (full 8-command run on this branch, on top of current main)

$ uv run ruff check src/ tests/
[]

$ uv run ruff check benchmarks/
[]

$ uv run pyright src/
0 errors, 0 warnings, 0 informations

$ uv run pyright benchmarks/
0 errors, 0 warnings, 0 informations

$ uv run pytest --tb=short -q
426 passed

$ uv run pytest tests/benchmarks -q
119 passed

$ uv run pytest tests/benchmarks/test_scoring.py -q
21 passed

$ uv run python-docs-mcp-server doctor
All checks passed.

[coderabbitai]

Warning

Review limit reached

@ayhammouda, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 22c53519-5272-451e-98aa-c126c8148d04

📥 Commits

Reviewing files that changed from the base of the PR and between 89921f3 and 01ca107.

📒 Files selected for processing (1)

• tests/benchmarks/test_scoring.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/88-coderabbit-followup

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbiteu]

Warning

Review limit reached

@ayhammouda, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 36 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c004799e-9079-4826-9bd4-ef50ca5589b3

📥 Commits

Reviewing files that changed from the base of the PR and between 89921f3 and 01ca107.

📒 Files selected for processing (1)

• tests/benchmarks/test_scoring.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/88-coderabbit-followup

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}