Context
Implements the decision in #8106: reuse the existing "held for manual review" mechanism as the confirmation signal. src/settings/agent-actions.ts's AgentActionPlanInput has three hold fields sharing one shape — migrationCollisionHold, unlinkedIssueMatchHold, aiReviewLowConfidenceHold (each { reason: string; comment: string } | undefined) — every one routing a would-be-close PR into a manual-review hold (the manualReview label) instead of an automatic close. src/rules/advisory.ts's resolveAiReviewLowConfidenceHold produces the aiReviewLowConfidenceHold value specifically for a sub-floor ai_consensus_defect/ai_review_split finding. None of the three currently records anything when the maintainer actually acts on a held PR.
src/review/outcomes-wire.ts's recordReversalSignals already distinguishes the repo owner from a contributor via ownerLogin === senderLogin (comparing repoFullName's owner segment against payload.sender.login) — this issue reuses that exact check.
Requirements
- Wherever a PR's
closed webhook event is currently handled (the sibling of recordReversalSignals's reopened handling in src/review/outcomes-wire.ts — read the full file to find the existing closed-action branch, or add one following the same shape if none exists for this purpose today), when the closing actor is the repo owner (same ownerLogin === senderLogin check recordReversalSignals already uses) AND the PR was held via aiReviewLowConfidenceHold specifically (the only one of the three holds with a directly attributable ruleId today — ai_consensus_defect/ai_review_split, via AI_JUDGMENT_BLOCKER_CODES):
- Scope:
aiReviewLowConfidenceHold only in this issue. migrationCollisionHold and unlinkedIssueMatchHold do not currently carry a ruleId-equivalent identifier the calibration module could key on — extending confirmation-recording to them is a separate follow-up once/if they're given one, not part of this issue.
- Do not touch
resolveAiReviewLowConfidenceHold or any of the three hold fields' own production logic — this issue only reads the closing webhook event, it doesn't change what causes a hold or what a hold does.
Deliverables
Test Coverage Requirements
99%+ patch coverage (branch-counted), this repo's standard gate. Cover the owner-vs-contributor branch, the held-vs-not-held branch, the matching-vs-no-matching-fired-event branch, and the fail-open .catch branch, each as an explicit, separate test case.
Expected Outcome
ai_consensus_defect/ai_review_split's precision numbers (via #8085's scoreBacktest) stop being reversal-only floors and start reflecting real positive confirmations too — the first rule(s) in this whole system with a genuine, non-inferred "this automated call was correct" signal.
Links & Resources
Context
Implements the decision in #8106: reuse the existing "held for manual review" mechanism as the confirmation signal.
src/settings/agent-actions.ts'sAgentActionPlanInputhas three hold fields sharing one shape —migrationCollisionHold,unlinkedIssueMatchHold,aiReviewLowConfidenceHold(each{ reason: string; comment: string } | undefined) — every one routing a would-be-close PR into a manual-review hold (themanualReviewlabel) instead of an automatic close.src/rules/advisory.ts'sresolveAiReviewLowConfidenceHoldproduces theaiReviewLowConfidenceHoldvalue specifically for a sub-floorai_consensus_defect/ai_review_splitfinding. None of the three currently records anything when the maintainer actually acts on a held PR.src/review/outcomes-wire.ts'srecordReversalSignalsalready distinguishes the repo owner from a contributor viaownerLogin === senderLogin(comparingrepoFullName's owner segment againstpayload.sender.login) — this issue reuses that exact check.Requirements
closedwebhook event is currently handled (the sibling ofrecordReversalSignals'sreopenedhandling insrc/review/outcomes-wire.ts— read the full file to find the existingclosed-action branch, or add one following the same shape if none exists for this purpose today), when the closing actor is the repo owner (sameownerLogin === senderLogincheckrecordReversalSignalsalready uses) AND the PR was held viaaiReviewLowConfidenceHoldspecifically (the only one of the three holds with a directly attributableruleIdtoday —ai_consensus_defect/ai_review_split, viaAI_JUDGMENT_BLOCKER_CODES):createSignalStore(env).queryRuleHistory(code, sinceMs)for each code inAI_JUDGMENT_BLOCKER_CODES(30-day lookback, matching calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104's own lookback constant) and checking for a matchingtargetKey.recordHumanOverride({ ruleId: code, targetKey: targetId, verdict: "confirmed", occurredAt: nowIso() })on the samecreateSignalStore(env)..catch(() => undefined)on the write, matching every other write site added by this epic. Must never affect the actual PR-close handling.aiReviewLowConfidenceHoldonly in this issue.migrationCollisionHoldandunlinkedIssueMatchHolddo not currently carry aruleId-equivalent identifier the calibration module could key on — extending confirmation-recording to them is a separate follow-up once/if they're given one, not part of this issue.resolveAiReviewLowConfidenceHoldor any of the three hold fields' own production logic — this issue only reads the closing webhook event, it doesn't change what causes a hold or what a hold does.Deliverables
recordHumanOverride("confirmed")call, scoped toaiReviewLowConfidenceHold/AI_JUDGMENT_BLOCKER_CODESas described.aiReviewLowConfidenceHoldand had a matching fired event records"confirmed"for the correct ruleId; a CONTRIBUTOR closing the same PR records nothing (owner-only, mirrorsrecordReversalSignals's own contributor exclusion for the reversed side); an owner closing a PR that was NOT held viaaiReviewLowConfidenceHoldrecords nothing; an owner closing a held PR with no matching fired event (e.g.queryRuleHistoryreturns empty) records nothing; the write degrades silently on aSignalStorefailure.Test Coverage Requirements
99%+ patch coverage (branch-counted), this repo's standard gate. Cover the owner-vs-contributor branch, the held-vs-not-held branch, the matching-vs-no-matching-fired-event branch, and the fail-open
.catchbranch, each as an explicit, separate test case.Expected Outcome
ai_consensus_defect/ai_review_split's precision numbers (via #8085'sscoreBacktest) stop being reversal-only floors and start reflecting real positive confirmations too — the first rule(s) in this whole system with a genuine, non-inferred "this automated call was correct" signal.Links & Resources
src/settings/agent-actions.ts(AgentActionPlanInput.aiReviewLowConfidenceHoldand its siblings)src/rules/advisory.ts(resolveAiReviewLowConfidenceHold,AI_JUDGMENT_BLOCKER_CODES)src/review/outcomes-wire.ts(recordReversalSignals, the owner-vs-contributor check to reuse — read its currentclosed-event handling, if any, before starting)src/review/signal-tracking-wire.ts(createSignalStore)