Skip to content

calibration: capture bounded raw context (issueText/prTitle/prBody/diff) in linked_issue_scope_mismatch's fired-event metadata #8129

Description

@JSONbored

Context

src/services/linked-issue-satisfaction.ts's LinkedIssueSatisfactionInput already defines exactly the raw context this rule evaluates: issueText (bounded to MAX_ISSUE_TEXT_CHARS = 6000), prTitle, prBody (bounded to MAX_BODY_CHARS = 2000), diff (bounded to MAX_DIFF_CHARS = 60000). #8101 (closed, merged) recorded a RuleFiredEvent for this finding but only captured { confidence } in its metadata — not the raw input the assessment was actually based on. Without that raw input, #8082's backtest primitives can only validate CONFIG/THRESHOLD changes (e.g. LINKED_ISSUE_SATISFACTION_CONFIDENCE_FLOOR) against this corpus, never a change to the assessment prompt or logic itself — there'd be nothing to re-run new logic against.

Requirements

  • At the same recordRuleFired call site calibration: wire linked_issue_scope_mismatch into the shared signal-tracking module (fired + reversal) #8101 added (src/queue/processors.ts, inside the linkedIssueSatisfactionGateMode === "block" && status === "unaddressed" block), extend the metadata argument to also include the same bounded fields LinkedIssueSatisfactionInput already uses:
    {
      confidence: result.result.confidence,
      issueText: <bounded to MAX_ISSUE_TEXT_CHARS>,
      prTitle: args.pr.title,
      prBody: <bounded to MAX_BODY_CHARS>,
      diff: <bounded to MAX_DIFF_CHARS>,
    }
    Import and reuse MAX_ISSUE_TEXT_CHARS/MAX_BODY_CHARS/MAX_DIFF_CHARS directly from linked-issue-satisfaction.ts — do not redefine new bound constants.
  • If any of these values aren't already in scope at the exact recordRuleFired call site, thread them from wherever they were already computed for the original LinkedIssueSatisfactionInput construction earlier in the same function — do not re-fetch or re-derive them a second time.

Deliverables

  • metadata at the recordRuleFired call site extended with the four bounded raw-context fields above.
  • A test confirming the recorded event's metadata contains all four fields, and that at least one field is correctly truncated when the raw value exceeds its bound.

Test Coverage Requirements

99%+ patch coverage (branch-counted), including the truncation branch for at least one over-limit field.

Expected Outcome

linked_issue_scope_mismatch's corpus becomes rich enough to backtest a change to the assessment prompt or logic, not just its confidence floor — a future classify function can re-run new logic against the stored issueText/prTitle/prBody/diff and compare the result to the recorded label.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.help wantedExtra attention is needed

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions