Design + wire an advisory-only backtest check for PRs touching ORB/AMS deterministic rules

[JSONbored]

Problem

#8083-#8088 (once merged) give Loopover a pure, tested set of primitives — build a corpus, export a snapshot, score a candidate classifier, compare against a baseline, split held-out, render a report — but nothing runs them against a real PR yet. Per the parent epic's (#8082) Boundaries, no rule change should ever be auto-blocked by this without an explicit maintainer design decision first — this repo's auto-tune.ts precedent is human-in-the-loop and tightening-only, and the same posture applies here even more so, since a backtest score is a NEW kind of signal with no live track record yet.

Scope note (expanded): this issue covers wiring the backtest check for BOTH deterministic gate rules and AI-judgment surfaces. The scoring primitives (#8083-#8086) are generic over any ruleId and any classify function — they don't distinguish a deterministic regex from an LLM-backed judgment call. #8101 + #8104 together wire every isConfiguredGateBlocker finding code — deterministic (missing_linked_issue, duplicate_pr_risk, secret_leak, and others) and AI-judgment-backed (linked_issue_scope_mismatch, ai_consensus_defect, ai_review_split) alike — into the same shared calibration module, generically, at one call site.

Corrected scope (important — this issue previously described "candidate = the PR branch's live behavior" imprecisely): BacktestCase (#8083) only stores outcome + metadata — never the original diff/issue text a rule evaluated. That means a classify function can only correctly re-simulate a decision using what's actually IN the corpus. Without raw context, this check can only honestly backtest CONFIG/THRESHOLD changes (e.g. LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR, DEFAULT_AI_REVIEW_CLOSE_CONFIDENCE) — a classify function comparing a stored metadata.confidence against a new threshold value needs no raw content. Backtesting a LOGIC/regex/detection change honestly requires re-running the new logic against the original diff/issue content, which #8129 and #8130 add to the corpus (excluding secret_leak, deliberately, for security reasons — see #8130). This issue is blocked on both landing first, so "candidate = the PR branch's live behavior" is actually achievable rather than a comparison that looks real but isn't measuring what it claims to.

Area

CI wiring + a maintainer-run/maintainer-triggered workflow. Likely .github/workflows/** plus a new script under scripts/ that invokes the calibration primitives from #8083-#8088. Touches how PRs that change deterministic gate rules (src/rules/**, src/review/content-lane/**, src/settings/agent-actions.ts) OR AI-judgment surfaces (src/services/linked-issue-satisfaction.ts, src/services/ai-review.ts) are reviewed. AMS's own policy-check modules remain out of scope (see Boundaries; tracked separately as #8107, blocked on this epic).

Proposal

• Audit .github/workflows/ first for an existing scoped-CI-check pattern to mirror — .github/workflows/selfhost.yml is the closest existing precedent in this repo: pull_request trigger with types: [opened, synchronize, reopened, ready_for_review] (the explicit list matters — the default omits ready_for_review, so marking a draft PR ready wouldn't otherwise trigger a run) plus a paths: filter, draft PRs skipped via if: github.event.pull_request.draft != true, and a concurrency group keyed on github.ref so superseded pushes cancel in-progress runs. Mirror this shape rather than inventing a new one.
• Phase 1: advisory only. For each ruleId whose corpus (calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101/calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104/calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129/calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130) has real data AND whose detection logic the PR's diff actually touches, run the backtest — baseline = current main's exported snapshot (scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084), candidate = the PR branch's code re-run (via a classify function built from the PR's changed logic) against each stored case's raw context (calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129/calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130) or, for a pure threshold/config change, against the stored metadata value alone. Post renderBacktestComparison's (calibration: render a backtest score/comparison report as Markdown #8088) output as a PR comment or check-run summary. Never block merge on the result in this phase.
• The question of whether/when a future phase should gate merge on a "regressed" verdict is tracked separately as Decide + implement whether a REGRESSED backtest verdict should ever gate a merge (Phase 2) #8105, deliberately not decided in this issue.
• Decide where the "baseline" snapshot lives (checked into the repo via scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084's CLI and refreshed on a cadence, vs. computed fresh on every run) and who/what refreshes it. A thin/empty corpus for a newly-wired ruleId is an expected, not-blocking state — the backtest comparison degrades to null precision/recall per calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085's own null-when-no-data discipline; it doesn't error.

Deliverables

• A written decision (in this issue's comments) on the phase-1 trigger scope described above.
• The advisory CI wiring itself, phase 1 only (comment/report, never a merge block), covering every isConfiguredGateBlocker code calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101/calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104 wire, with real logic-change backtesting for every code except secret_leak once calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129/calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130 land.

Resources

• Epic: verifiable pre-merge backtest for ORB/AMS deterministic rules #8082, calibration: pure BacktestCase corpus builder from RuleFiredEvent/HumanOverrideEvent pairs #8083, scripts: export the rule-precision backtest corpus as a versioned, checksummed JSON snapshot #8084, calibration: pure confusion-matrix scorer for a candidate rule classifier against the backtest corpus #8085, calibration: Pareto-floor comparator between two BacktestScoreReports #8086, calibration: deterministic seeded held-out/visible split of the backtest corpus #8087, calibration: render a backtest score/comparison report as Markdown #8088, calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101, calibration: wire recordRuleFired/recordHumanOverride for the remaining isConfiguredGateBlocker codes (excludes linked_issue_scope_mismatch, covered by #8101) #8104, Decide + implement whether a REGRESSED backtest verdict should ever gate a merge (Phase 2) #8105, calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129, calibration: capture bounded raw context for the remaining isConfiguredGateBlocker codes, excluding secret_leak #8130
• src/review/auto-tune.ts (the tightening-only, human-in-the-loop precedent this should follow)
• .github/workflows/selfhost.yml (the concrete CI trigger/concurrency/draft-skip pattern to mirror)
• src/services/linked-issue-satisfaction.ts, src/rules/advisory.ts (AI_JUDGMENT_BLOCKER_CODES) — the concrete AI-judgment backtest targets

Boundaries

No merge-blocking in this issue (see #8105 for that decision). No change to any live gate rule's or AI-judgment surface's actual behavior. AMS-side wiring is out of scope here (ORB only) — tracked separately as #8107, explicitly blocked until this epic is complete. secret_leak never gets logic-change backtesting, by permanent design (see #8130) — this check degrades to threshold-only for that one code, which is acceptable since secret_leak has no tunable threshold to backtest in the first place.

maintainer-only — CI wiring plus a trust/safety-adjacent policy decision, not a build task for contributors.

[linear-code]

JSO-5587