Surface Stryker mutation testing results and run it daily with a Mutation Test Improver workflow - #11110
Open
Amaury Levé (Evangelink) wants to merge 12 commits into
Open
Conversation
The mutation-testing workflow ran successfully every week but its score only appeared in raw console logs and a downloadable artifact zip, so results went unnoticed. Tee the Stryker output to a log file and add a step that scrapes the mutation score and killed/survived/timeout counts into \. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- mutation-testing.yml: change the Stryker schedule from weekly to daily. - New mutation-test-improver.md (+ compiled .lock.yml): triggers on workflow_run completion of "Mutation testing", parses the report, maintains a rolling [mutation-test-improver] Monthly Report issue with the score trend and survived-mutant hotspots, and opens small draft PRs adding regression tests for survived mutants it verifies (by re-running Stryker locally) are genuine test gaps. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 08:12
View session
Amaury Levé (Evangelink)
added
the
state/needs-review
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Expert Review — PR #11110: Mutation Test Improver Agentic Workflow
Summary
This PR adds a new gh-aw agentic workflow (mutation-test-improver) that reacts to the daily Stryker.NET mutation-testing run, maintains a monthly report issue, and optionally opens draft PRs to kill high-value survived mutants. It also changes the mutation-testing schedule from weekly to daily and adds a job summary step.
Overall this is a well-structured workflow with good safety constraints (draft PRs, max limits, pre-activation gating, protected files, title prefixes). A few items need attention.
Verdict Table
| # | Dimension | Verdict | Notes |
|---|---|---|---|
| 1 | Algorithmic Correctness | docs/dev-guide.md still says "weekly" after cron changed to daily |
|
| 2 | Threading & Concurrency | N/A | No shared mutable state |
| 3 | Resource Management | ✅ Clean | Timeout set to 60 min, max AI credits 1500 |
| 4 | Performance | Weekly→daily is 7× more CI cost — inline comment | |
| 5 | Error Handling | ✅ Clean | Pre-activation handles gh pr list failures; failed runs handled in Step 2 |
| 6 | Public API Surface | N/A | No public API changes |
| 7 | Binary/Source Compatibility | N/A | Workflow-only change |
| 8 | Naming & Conventions | ✅ Clean | Title prefixes, label choices consistent |
| 9 | Security | update-issue.target: "*" is broader than needed — inline comment |
|
| 10 | Localization | N/A | No resource strings |
| 11 | Cross-TFM Correctness | N/A | |
| 12 | Test Coverage | N/A | Workflow itself; agent prompt includes verification steps |
| 13 | Documentation | "weekly" → "daily" inconsistency in docs/dev-guide.md |
|
| 14 | Code Style & Formatting | ✅ Clean | |
| 15 | Diagnostics & Logging | ✅ Clean | Console log captured, ANSI stripped, step summary generated |
| 16 | IPC & Wire Format | N/A | |
| 17 | Configuration & Build | ✅ Clean | strict: true in lock metadata, SHA-pinned actions, digest-pinned containers |
| 18 | Edge Cases | ✅ Clean | Handles failed/cancelled runs, duplicate run detection, equivalent mutants |
| 19 | Dependency Management | ✅ Clean | All action pins look correct (SHA + version comment) |
| 20 | Agentic Workflow Compliance | ✅ Clean | No strict: false, lock.yml is compiler-generated (v0.86.2), no banned type/bug/type/feature/type/task labels, no TODOs without issue refs |
| 21 | Prompt Injection Resilience | ✅ Clean | Threat-detection configured with detection model; Stryker reports treated as untrusted per prompt; safe-outputs enforce title prefixes and max limits |
| 22 | Scope Discipline | Agent body says (label \testing`)but safe-outputs appliestype/automation, type/test-gap` — mismatch |
Agentic Workflow Specific Checks
- ✅
strict: trueconfirmed in lock metadata - ✅ Lock file generated by compiler v0.86.2, not hand-edited
- ✅ All
uses:lines are SHA-pinned with version comments - ✅ No
type/bug,type/feature, ortype/tasklabels used - ✅ No TODO comments without issue references
- ✅
workflow_runtrigger guarded against forks (repository.idand!forkchecks) - ✅ Safe-outputs: PRs are draft, title-prefixed, with max limits and protected files
⚠️ update-issue.target: "*"could be tightened with arequired-title-prefix
No blocking issues found. Four minor items flagged inline.
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The reporting pipeline can fail, and the agent workflow has issue lifecycle and configuration defects.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (6)
| Severity | Finding |
|---|---|
 |
.github/workflows/mutation-testing.yml — tee cannot create its parent directory. On a fresh runner this directory does not exist until… |
 |
.github/workflows/mutation-test-improver.md — When the PR cap is reached, this check exits 1 and fails the entire pre_activation job instead of… |
|
.github/workflows/mutation-test-improver.md — This workflow creates tracking issues without setting the repository's required native Issue Type.… |
|
.github/workflows/mutation-test-improver.md — Stopping here bypasses Step 5's issue creation and month-rollover logic. If the first processed run… |
 |
.github/workflows/mutation-testing.yml — The message says the raw log appears “below,” but the summary never appends it. Point users to the… |
|
docs/dev-guide.md — This still documents a weekly cadence, while the workflow cron in this PR now runs every day.… |
What changed in this PR
Adds daily, visible Stryker mutation testing and an automated workflow for reporting and addressing test gaps.
Changes:
- Runs mutation testing daily and publishes summary metrics.
- Adds a mutation-test improvement agent with monthly reporting and draft PR creation.
- Documents mutation-testing outputs and artifacts.
| File | Description |
|---|---|
docs/dev-guide.md |
Documents workflow reporting and artifacts. |
.github/workflows/mutation-testing.yml |
Schedules daily runs and generates summaries. |
.github/workflows/mutation-test-improver.md |
Defines the mutation-improvement agent. |
.github/workflows/mutation-test-improver.lock.yml |
Compiled agentic workflow. |
Suppressed comments (1)
.github/workflows/mutation-testing.yml:54
- These pipelines return status 1 when a value is absent, causing the Actions bash step to exit before reaching the intended
?/warning fallbacks. This is especially likely for the incomplete-output failure case that thisalways()step is meant to summarize; tolerate a missing match for each optional field.
killed=$(echo "$clean_log" | grep -oE 'Killed: *[0-9]+' | tail -n1 | grep -oE '[0-9]+')
survived=$(echo "$clean_log" | grep -oE 'Survived: *[0-9]+' | tail -n1 | grep -oE '[0-9]+')
timeout=$(echo "$clean_log" | grep -oE 'Timeout: *[0-9]+' | tail -n1 | grep -oE '[0-9]+')
score=$(echo "$clean_log" | grep -oE 'final mutation score is [0-9.]+ *%' | tail -n1 | grep -oE '[0-9.]+')
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Amaury Levé (Evangelink)
enabled auto-merge (squash)
September 8, 2026 09:03
Fix review comments on the Stryker mutation-testing workflow and the new Mutation Test Improver agentic workflow: - document the daily schedule and its cost tradeoff - create the Stryker output directory before tee writes the console log - correct the job-summary fallback text - make the PR-cap pre-activation check a clean activation gate - scope report issue updates to the workflow labels - expose Issue Type for report creation and instruct new reports to use Task - keep failed-run handling flowing through monthly report creation/rollover Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 09:29
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Authentication, timeout, reporting-link, and failure-recovery issues can prevent reliable mutation reporting and improvement runs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (4)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — Exit code 4 from gh means authentication is required, not “no matches”; an empty PR list returns… |
|
.github/workflows/mutation-test-improver.md — The 60-minute agent budget is insufficient for the workflow's maximum path of two sequential full… |
|
.github/workflows/mutation-test-improver.md — A manual invocation parses the most recent Mutation testing run, but this instruction links Run… |
|
.github/workflows/mutation-testing.yml — These optional scrapes run under the step's fail-fast shell, so any missing line makes grep… |
Issues resolved since last review (6)
| Severity | Finding |
|---|---|
|
docs/dev-guide.md — This still documents a weekly cadence, while the workflow cron in this PR now runs every day.… View resolved comment |
|
.github/workflows/mutation-testing.yml — The message says the raw log appears “below,” but the summary never appends it. Point users to the… View resolved comment |
|
.github/workflows/mutation-test-improver.md — Stopping here bypasses Step 5's issue creation and month-rollover logic. If the first processed run… View resolved comment |
|
.github/workflows/mutation-test-improver.md — This workflow creates tracking issues without setting the repository's required native Issue Type.… View resolved comment |
|
.github/workflows/mutation-test-improver.md — When the PR cap is reached, this check exits 1 and fails the entire pre_activation job instead of… View resolved comment |
|
.github/workflows/mutation-testing.yml — tee cannot create its parent directory. On a fresh runner this directory does not exist until… View resolved comment |
Suppressed comments (1)
.github/workflows/mutation-test-improver.md:165
- Memory and safe outputs are persisted by independent downstream jobs, so the run ID can be recorded even when the monthly issue update fails. A rerun then takes this unconditional
nooppath and can never repair the missing report entry. Only treat a run as complete when both memory and the current issue's Run History contain its ID.
- If memory shows this run id was already processed, call `noop` with an explanation and stop.
Fix follow-up review comments on the mutation testing workflows: - authenticate the PR-cap pre-activation gh call and treat gh failures as errors - raise the Mutation Test Improver timeout for Stryker verification runs - preserve the upstream Mutation testing run id for artifact and report links - tolerate missing Stryker summary lines when scraping partial console output Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Ensure the Mutation Test Improver only noops duplicate upstream Stryker runs after confirming the current monthly report already contains the upstream run id. This lets reruns repair a missing report entry when memory was updated but the safe-output issue update failed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the suppressed review-body note as well: duplicate upstream Stryker runs now only |
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 09:45
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The agent may process the wrong source revision, skip reporting under its PR cap, and use an unprovisioned .NET toolchain.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — This gate skips the entire workflow once three PRs are open, so it also skips the daily… |
|
.github/workflows/mutation-test-improver.md — The upstream SHA is only preserved as metadata; it is never checked out. For a workflow_run, the… |
Issues resolved since last review (4)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-testing.yml — These optional scrapes run under the step's fail-fast shell, so any missing line makes grep… View resolved comment |
|
.github/workflows/mutation-test-improver.md — A manual invocation parses the most recent Mutation testing run, but this instruction links Run… View resolved comment |
|
.github/workflows/mutation-test-improver.md — The 60-minute agent budget is insufficient for the workflow's maximum path of two sequential full… View resolved comment |
|
.github/workflows/mutation-test-improver.md — Exit code 4 from gh means authentication is required, not “no matches”; an empty PR list returns… View resolved comment |
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/mutation-test-improver.md:191
- These commands bypass the repository's required toolchain setup. The generated agent job does not install .NET, while
.github/copilot-instructions.md:24-31requires the build scripts to provision the pinned SDK andglobal.json:27-32explicitly errors when it is unavailable. Make the verification steps run the repository build script first and invoke tool restore/Stryker through$GITHUB_WORKSPACE/.dotnet/dotnet, then regenerate the lock file.
Fix follow-up review feedback on the Mutation Test Improver workflow: - remove the pre-activation PR cap that skipped daily report updates - make the PR cap suppress only auto-fix PR creation in Step 4 - fetch and check out the upstream Mutation testing SHA before analysis - require repo build-script toolchain setup and repo-local dotnet for tests/Stryker Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the suppressed review-body note about toolchain setup as well: Step 4 now requires running the repository build script first to provision the pinned |
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 10:00
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The test-only output boundary is unenforced, and the timeout does not cover the documented maximum verification path.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The safe-output policy does not enforce the workflow's stated tests-only boundary:… |
|
.github/workflows/mutation-test-improver.md — The 120-minute budget still does not cover the documented worst case. Two candidates can each take… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The upstream SHA is only preserved as metadata; it is never checked out. For a workflow_run, the… View resolved comment |
|
.github/workflows/mutation-test-improver.md — This gate skips the entire workflow once three PRs are open, so it also skips the daily… View resolved comment |
Address follow-up review feedback on the Mutation Test Improver workflow: - restrict generated PR changes to the ServerMode.Client.Sources unit test project - cap each workflow run to two total Stryker verification runs so the timeout budget is bounded Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 10:12
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Stryker failures can be masked, artifact download lacks agent authentication, and PR-update permissions bypass the tests-only boundary.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 3
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The tests-only restriction applies only to create-pull-request. The compiled configuration for… |
|
.github/workflows/mutation-test-improver.md — The agent sandbox does not retain an authenticated gh session: the compiled workflow cleans Git… |
|
.github/workflows/mutation-testing.yml — This pipeline reports tee's exit status, so a nonzero Stryker exit is masked whenever tee… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The 120-minute budget still does not cover the documented worst case. Two candidates can each take… View resolved comment |
|
.github/workflows/mutation-test-improver.md — The safe-output policy does not enforce the workflow's stated tests-only boundary:… View resolved comment |
Address review feedback on the mutation workflows: - enforce the same test-directory allowlist for push-to-pull-request-branch - require authenticated GitHub Actions artifact tools instead of shell gh download - enable pipefail so Stryker failures propagate through tee Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 10:24
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The agent cannot download the report through the current firewall, and it incorrectly treats timeout mutants as test gaps.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The artifact tool only returns a signed download URL, and the current compiled firewall allowlist… |
|
.github/workflows/mutation-test-improver.md — Stryker treats Timeout as detected, counting it with Killed in the mutation-score numerator; it… |
Issues resolved since last review (3)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-testing.yml — This pipeline reports tee's exit status, so a nonzero Stryker exit is masked whenever tee… View resolved comment |
|
.github/workflows/mutation-test-improver.md — The agent sandbox does not retain an authenticated gh session: the compiled workflow cleans Git… View resolved comment |
|
.github/workflows/mutation-test-improver.md — The tests-only restriction applies only to create-pull-request. The compiled configuration for… View resolved comment |
Address review feedback on Mutation Test Improver: - allow signed Actions artifact download hosts through the workflow firewall - document Stryker's detected-mutant score formula - rank Survived and NoCoverage mutants as actionable gaps instead of Timeout mutants Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 10:37
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Candidate selection, score calculation, issue scoping, and multi-PR workspace isolation contain correctness risks.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 2
New issues introduced by this change (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — required-labels is not a workflow-specific write boundary: these same two labels are used by… |
|
.github/workflows/mutation-test-improver.md — The stated denominator is incorrect when the report contains CompileError or RuntimeError… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — Stryker treats Timeout as detected, counting it with Killed in the mutation-score numerator; it… View resolved comment |
|
.github/workflows/mutation-test-improver.md — The artifact tool only returns a signed download URL, and the current compiled firewall allowlist… View resolved comment |
Suppressed comments (4)
.github/workflows/mutation-test-improver.md:160
- This only filters failed attempts. A successfully killed mutant remains
Survivedin each daily report until its draft PR merges, so the workflow can open duplicate PRs for it on subsequent days despite memory recording the existing PR. Exclude every already-attempted mutant while its outcome is still active.
4. Filter out mutants already recorded as equivalent or already attempted-and-failed in memory.
.github/workflows/mutation-test-improver.md:164
- Step 3 includes
NoCoverageas an actionable undetected status, but this loop narrows candidates back toSurvived, so uncovered mutants may never be selected for a regression test. Keep both statuses in the bounded fix loop.
For **at most 2** of the remaining highest-value survived mutants (favor ones in behaviorally meaningful code — public API surfaces, error handling, boundary conditions — over pure boilerplate or generated code), and with **at most 2 total Stryker verification runs across the whole workflow run**:
.github/workflows/mutation-test-improver.md:176
- Multiple
create-pull-requestcalls snapshot the current workspace but do not automatically clean it. Without an explicit reset, a second PR includes the first candidate's test changes, and an abandoned candidate's edits can leak into a later PR. Restore the test worktree after each snapshot or abandoned attempt before moving to another mutant.
- If it did not flip, don't force it — try at most one more angle only when verification budget remains; otherwise abandon this mutant, record the attempt outcome in memory, and move to the next candidate.
9. For each mutant you successfully kill, create a small draft PR from a fresh branch (`mutation-test-improver/<short-desc>`) with:
.github/workflows/mutation-test-improver.md:210
- This report template contradicts Step 3 by treating
Timeoutmutants as hotspots and omittingNoCoverage. Timeouts are already detected and should not be presented as test gaps; reportSurvived/NoCoverageinstead.
{File-ranked list of the highest-value remaining survived/timeout mutants, with links, from this run}
Address review feedback on Mutation Test Improver: - require the workflow title prefix as well as labels for issue updates - use Stryker's detected-mutant score denominator excluding invalid mutants - treat Survived and NoCoverage as actionable test-gap statuses - avoid retrying active already-attempted mutants and clean the test worktree between candidates Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the suppressed review-summary items as well: the workflow now filters out any already-attempted active mutant (including ones with an open draft PR), carries |
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 10:52
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Mutant identity persistence and branch advancement can cause incorrect suppression or unverified draft PRs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — Stryker assigns mutant IDs sequentially, so adding/removing an earlier mutation can renumber every… |
Issues resolved since last review (2)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — The stated denominator is incorrect when the report contains CompileError or RuntimeError… View resolved comment |
|
.github/workflows/mutation-test-improver.md — required-labels is not a workflow-specific write boundary: these same two labels are used by… View resolved comment |
Suppressed comments (1)
.github/workflows/mutation-test-improver.md:177
- The test is verified against the detached
upstream_head_sha, but the generated safe-output job later checks out the then-current default branch (mutation-test-improver.lock.yml:1923-1928) before applying the patch. Ifmainadvances during this long run, the PR is created on a revision that was never built or mutation-tested, and the verified mutant may have changed or disappeared. Recheck the branch SHA immediately before emitting the PR (or rebase and repeat verification).
9. For each mutant you successfully kill, create a small draft PR from a fresh branch (`mutation-test-improver/<short-desc>`) with:
Address review feedback on Mutation Test Improver: - key equivalent and attempted mutants by a revalidated stable fingerprint - avoid suppressing current mutants using sequential Stryker ids alone - recheck the default branch SHA before emitting a verified draft PR Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the suppressed review-summary item about safe-output PR creation racing with the default branch. Step 4 now requires re-reading the default branch SHA immediately before emitting a PR; if it changed from the verified |
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 11:07
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Same-run reporting and memory can claim a queued PR was opened before safe-output processing assigns or confirms it.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review tier: Balanced
Findings: 1
New issues introduced by this change (1)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — create_pull_request is only queued here and is applied after the agent finishes; the generated… |
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — Stryker assigns mutant IDs sequentially, so adding/removing an earlier mutation can renumber every… View resolved comment |
Make Mutation Test Improver treat safe-output PR creation as asynchronous: - record verified fixes as pending PR attempts during the same run - reconcile actual GitHub PRs before marking attempts opened or adding links - avoid suppressing retries based on PRs that may not have materialized Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Addressed the latest review-summary item about queued safe-output PRs. Mutation Test Improver now records same-run verified fixes as |
Copilot started reviewing on behalf of
Amaury Levé (Evangelink)
September 8, 2026 11:21
View session
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The autonomous write-capable workflow and large generated lock file warrant final human validation despite no specific defect found.
Review tier: Balanced
Findings: None
Issues resolved since last review (1)
| Severity | Finding |
|---|---|
|
.github/workflows/mutation-test-improver.md — create_pull_request is only queued here and is applied after the agent finishes; the generated… View resolved comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Stryker.NET mutation testing was already configured (tool pin,
stryker-config.jsonscoped to theMicrosoft.Testing.Platform.ServerMode.Client.Sourcesclient, and a weekly.github/workflows/mutation-testing.ymljob) and ran successfully every week, but the results were invisible: the mutation score only appeared in raw console logs and inside a downloadable artifact zip that nobody routinely opens.Changes
.github/workflows/mutation-testing.yml): tee Stryker's output, scrape the score and killed/survived/timeout counts into$GITHUB_STEP_SUMMARY. Updateddocs/dev-guide.md.0 5 * * 1to0 5 * * *. Recent runs took ~10-30 minutes, well within the 90-minute timeout.mutation-test-improver.md(+ compiled.lock.yml), modeled ontest-improver.mdconventions: triggers onworkflow_runcompletion of "Mutation testing", parses the report, maintains a rolling[mutation-test-improver] Monthly Report {YYYY}-{MM}issue (score trend, survived-mutant hotspots, known equivalents, suggested actions), and opens small draft PRs (capped at 2) adding regression tests for survived mutants it verifies (by re-running Stryker locally) are genuine test gaps. It never widens mutation-testing scope itself.Verification
gh aw compile mutation-test-improver --strictsucceeds.python .github/scripts/check_action_pins.pypasses (no unintended action pin rewrites).gh run list/gh run viewthat the existing scheduled Stryker runs succeed and produce a parseablemutation-report.json.Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com