Skip to content

calibration: record a 'confirmed' HumanOverrideEvent when an owner closes an aiReviewLowConfidenceHold PR #8123

Description

@JSONbored

Context

Implements the decision in #8106: reuse the existing "held for manual review" mechanism as the confirmation signal. src/settings/agent-actions.ts's AgentActionPlanInput has three hold fields sharing one shape — migrationCollisionHold, unlinkedIssueMatchHold, aiReviewLowConfidenceHold (each { reason: string; comment: string } | undefined) — every one routing a would-be-close PR into a manual-review hold (the manualReview label) instead of an automatic close. src/rules/advisory.ts's resolveAiReviewLowConfidenceHold produces the aiReviewLowConfidenceHold value specifically for a sub-floor ai_consensus_defect/ai_review_split finding. None of the three currently records anything when the maintainer actually acts on a held PR.

src/review/outcomes-wire.ts's recordReversalSignals already distinguishes the repo owner from a contributor via ownerLogin === senderLogin (comparing repoFullName's owner segment against payload.sender.login) — this issue reuses that exact check.

Requirements

  • Wherever a PR's closed webhook event is currently handled (the sibling of recordReversalSignals's reopened handling in src/review/outcomes-wire.ts — read the full file to find the existing closed-action branch, or add one following the same shape if none exists for this purpose today), when the closing actor is the repo owner (same ownerLogin === senderLogin check recordReversalSignals already uses) AND the PR was held via aiReviewLowConfidenceHold specifically (the only one of the three holds with a directly attributable ruleId today — ai_consensus_defect/ai_review_split, via AI_JUDGMENT_BLOCKER_CODES):
  • Scope: aiReviewLowConfidenceHold only in this issue. migrationCollisionHold and unlinkedIssueMatchHold do not currently carry a ruleId-equivalent identifier the calibration module could key on — extending confirmation-recording to them is a separate follow-up once/if they're given one, not part of this issue.
  • Do not touch resolveAiReviewLowConfidenceHold or any of the three hold fields' own production logic — this issue only reads the closing webhook event, it doesn't change what causes a hold or what a hold does.

Deliverables

  • The owner-closes-a-held-PR detection + recordHumanOverride("confirmed") call, scoped to aiReviewLowConfidenceHold/AI_JUDGMENT_BLOCKER_CODES as described.
  • Tests covering: an owner closing a PR that was held via aiReviewLowConfidenceHold and had a matching fired event records "confirmed" for the correct ruleId; a CONTRIBUTOR closing the same PR records nothing (owner-only, mirrors recordReversalSignals's own contributor exclusion for the reversed side); an owner closing a PR that was NOT held via aiReviewLowConfidenceHold records nothing; an owner closing a held PR with no matching fired event (e.g. queryRuleHistory returns empty) records nothing; the write degrades silently on a SignalStore failure.

Test Coverage Requirements

99%+ patch coverage (branch-counted), this repo's standard gate. Cover the owner-vs-contributor branch, the held-vs-not-held branch, the matching-vs-no-matching-fired-event branch, and the fail-open .catch branch, each as an explicit, separate test case.

Expected Outcome

ai_consensus_defect/ai_review_split's precision numbers (via #8085's scoreBacktest) stop being reversal-only floors and start reflecting real positive confirmations too — the first rule(s) in this whole system with a genuine, non-inferred "this automated call was correct" signal.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions