Skip to content

dataset(code-review): add reaction gold answers from BCApps PR 12254-neg-pr12254-20261020 - #979

Draft
Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-12254-neg-pr12254-20261020-synthetic-eval
Draft

Wenjie Fan (gggdttt) wants to merge 1 commit into
mainfrom
self-improvement/bcapps-12254-neg-pr12254-20261020-synthetic-eval

Conversation

@gggdttt

@gggdttt Wenjie Fan (gggdttt) commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds code-review gold answers from explicit BCApps PR feedback.

Negative guards use exact source-file patches at immutable reviewed revisions, not invented synthetic code. Their targets are omitted from both expected and ignored comments; new negative-only entries have no expected findings.
Existing eligible gold answers are reused without changing their payloads.

Accepted findings

Rejected findings

Knowledge dependency

Validation

See the gate-owned offline evaluation section below.

Generated by the BC-ALAgentsInternal self-improvement workflow.

Offline evaluation: execution_error

Evaluation arm did not complete successfully.

Preparation attempt 1: validated.

Selection: new. New: synthetic__telemetry-scope-customercontent-01. Reused (complete payloads): ``.

  • synthetic__telemetry-scope-customercontent-01 / false_positive_guard / Migrate more Http Web Request Mgt. callers to native HttpClient BCApps#12254 (comment): Reuses the exact pipeline-validated negative snapshot patch/repo/baseCommit for this comment and records expected_comments=[] to reflect the developer's thumbs-down rejection of the TelemetryScope::All suggestion. Limitation: the developer's rationale (DataClassification=CustomerContent prevents visibility) contradicts the cited article's explicit guidance that TelemetryScope must be chosen independently of DataClassification, so this is not a demonstrated clean false positive -- only the literal rejection outcome is captured, and the guard's validity should be re-checked by a human reviewer rather than assumed.
    Coverage references and outcome shapes are validated mechanically. Semantic equivalence, severity calibration and recommendation quality are NOT proved by these checks or a matching F1.

Common engine code: ecf8e31759d6ddd6d78e3a0b7836b40134368009; dataset SHA256: 83D409BCA6DEA8996780E80ABBF5A311A1F8393AF93382B2BA7131C400E0C1D7.
Model: gpt-5.6-luna; judge: gpt-5.3-codex. Exact D: synthetic__telemetry-scope-customercontent-01.
Dataset base: 7c98b750c200d88ed0a81daaea5e71184a3af01b; candidate: 89ed7e70135695844a21c31ec04e43852c17c5d9.

Arm Evaluation commit Engine pin Knowledge pin
baseline 9867f68cfe99c39ad7717d05cd3fcf5cbe2e474b edd46ef34e9b38d7cc2fc3e1c1c806ff8e476ab6 f820aa1e8515a6483b828d9a5b1c55112461b4d9
candidate 6b10f44b849e602fb61ffedb80284d16ec390f49 f14f14d354c63c4ceb0a8c534bbccbdd3839c446 5066626e7c09b29bdae1129c61d3a8609fb85c1b

Baseline

Run: https://github.com/microsoft/BC-Bench/actions/runs/39427115557; conclusion: failure; wall clock: 1.7 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.
Metrics are not accepted: the result contract or execution failed. Inspect the retained raw artifacts.

Candidate

Run: https://github.com/microsoft/BC-Bench/actions/runs/39427432333; conclusion: success; wall clock: 7.2 minutes.
Raw findings and runtime pin proofs are retained in the self-improvement-feedback artifact and the linked evaluation run.
Metrics are not accepted: the result contract or execution failed. Inspect the retained raw artifacts.

Missing telemetry is unavailable, not zero. Evaluation-only metrics exclude candidate generation and are not the full-cycle cost.
Human review must verify source-patch fidelity, gold correctness, and target attribution. No automatic merge or branch-protection claim is made.

Cycle usage and elapsed time (running, as of 2026-10-20T13:12:36.8742662+00:00)

Orchestrator work: 1299.474 seconds. This is not the completed GitHub workflow duration.

Stage / attempt Status / outcome Elapsed seconds
preflight / 1 completed / 8.556
quality_preparation / 1 completed / 359.678
gold_preparation / 1 completed / selected_new 325.433
evaluation_preparation / 1 completed / 25.126
baseline / 1 failed / failure 135.6
candidate / 1 completed / success 444.376
gate_verdict / 1 completed / execution_error 0.007
publication / 1 running / 0.015
receipts / 1 not_started / unavailable
quality_validation / 1 completed / 4.31
gold_attempt / 1 completed / validated 311.026
Measure Known subtotal Whole-cycle total Known / required components
ai_credit: aiCredits 185.91039 unavailable 2 / 6
premium_request: premiumRequests 82 unavailable 2 / 6
token: inputTokens 9939882 unavailable 3 / 6
token: outputTokens 104826 unavailable 3 / 6
token: totalTokens 10044708 unavailable 3 / 6

Subtotals are CLI/Bench-reported usage, not invoices. Missing values are not zero. Emitted-span coverage does not prove complete child/background billing; judge usage is not exported. Premium requests are not AI credits or currency.
Nested stage, agent and API durations are not added into cycle wall clock. Root queue/setup, telemetry export, artifact upload and later dashboard publication are excluded.
PR evidence is an as-of snapshot before publication completes. Final-for-orchestrator metrics are emitted to self-improvement-cycle.json in the root self-improvement-feedback artifact and its Actions summary when persistence succeeds; a running checkpoint is not final evidence.
Root run: https://github.com/microsoft/BC-ALAgentsInternal/actions/runs/39425353290/attempts/1
Telemetry limitations: arm_usage_identity_or_schema_unavailable.

@github-code-quality

Copy link
Copy Markdown
Contributor

Code coverage overview

Languages: Python

Python / code-coverage/pytest

main 7c98b75 self-improvement/bca... 89ed7e7 +/-
88% 88% 0%

ℹ️ Calculations represent line coverage. Results may change as calculations are updated.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant