TestPilot.NET

Making automated tests smarter.

TestPilot.NET is an open-source test intelligence platform for .NET that sits on top of your existing test framework (NUnit, xUnit, or MSTest) and tells you things your test runner won't: which tests are flaky, which are slow relative to the rest of your suite, and where your test source itself has quality problems — empty tests, Thread.Sleep-based waits, undisposed bUnit contexts, hard-coded Playwright delays.

It doesn't run your tests. It reads the results your framework already produces, optionally looks at your test project's source via Roslyn, and reports on the health of your suite.

---

Status: V1 available

The core pipeline works end-to-end today: parse test results → analyze test health → report, across NUnit, xUnit, and MSTest, plus real Roslyn-based source analysis (including bUnit- and Playwright-aware diagnostics) and basic git integration. See What's in V1 below for the exact feature list, and Roadmap for what's next.

---

Quick start

Prerequisites: .NET 8 SDK (build and runtime).

git clone https://github.com/sgordon5691/TestPilot.NET.git
cd TestPilot.NET
dotnet build TestPilot.slnx

The repo includes sample projects under Test Projects/ — one per framework/library TestPilot understands — each with the same deliberate mix of a slow test, an empty test, and a flaky test, so you can see TestPilot do something interesting immediately. Here's the NUnit one:

# 1. Run the sample project's tests and produce an NUnit3 XML result file.
#    --results-directory pins down where the logger's relative path resolves, regardless
#    of your current directory - a known quirk of the NUnit test logger, not TestPilot.
dotnet test "Test Projects/SampleApp.Tests" --results-directory "Test Projects/SampleApp.Tests" --logger "nunit;LogFilePath=result.xml"

# 2. Analyze it.
dotnet run --project src/TestPilot.Cli -- analyze "Test Projects/SampleApp.Tests/result.xml" --project "Test Projects/SampleApp.Tests/SampleApp.Tests.csproj"

You should see output like:

TestPilot.NET Analysis Report
Suite: SampleApp.Tests.dll

Health Metrics:
  Flakiness Score:   0.0 %
  Average Duration:  00:00:00.2177069
  Runs Analyzed:     1

Diagnostics (3):
  [INFO] SLOW001: Test averages 208ms, 4.8x the suite average of 44ms. (TestCase: SampleApp.Tests.CalculatorTests.SlowOperation_CompletesEventually)
  [WARNING] TP0001: Test method 'NotImplementedYet' has an empty body and does not verify anything
  [WARNING] TP0002: Avoid calling Thread.Sleep in test code; it slows down the suite and can cause timing-dependent flakiness

Run steps 1–2 a few more times in a row and you'll also see FLAKY001 appear once the sample's intentionally non-deterministic test has landed on both outcomes across your accumulated run history.

Point it at your own NUnit project the same way — swap the paths for your own test project's .csproj and result file.

Other frameworks

The other samples work the same way, just with --framework set and a TRX result (xUnit/MSTest emit TRX via dotnet test --logger trx, not NUnit3 XML):

# xUnit
dotnet test "Test Projects/SampleApp.XUnit.Tests" --results-directory "Test Projects/SampleApp.XUnit.Tests" --logger "trx;LogFileName=result.trx"
dotnet run --project src/TestPilot.Cli -- analyze "Test Projects/SampleApp.XUnit.Tests/result.trx" --framework xunit --project "Test Projects/SampleApp.XUnit.Tests/SampleApp.XUnit.Tests.csproj"

# MSTest
dotnet test "Test Projects/SampleApp.MSTest.Tests" --results-directory "Test Projects/SampleApp.MSTest.Tests" --logger "trx;LogFileName=result.trx"
dotnet run --project src/TestPilot.Cli -- analyze "Test Projects/SampleApp.MSTest.Tests/result.trx" --framework mstest --project "Test Projects/SampleApp.MSTest.Tests/SampleApp.MSTest.Tests.csproj"

# bUnit (NUnit-based) - also demonstrates the BUNIT0001 analyzer
dotnet test "Test Projects/SampleApp.BUnit.Tests" --results-directory "Test Projects/SampleApp.BUnit.Tests" --logger "nunit;LogFilePath=result.xml"
dotnet run --project src/TestPilot.Cli -- analyze "Test Projects/SampleApp.BUnit.Tests/result.xml" --project "Test Projects/SampleApp.BUnit.Tests/SampleApp.BUnit.Tests.csproj"

# Playwright (NUnit-based) - build first, then a one-time browser install; also demonstrates PW0001
dotnet build "Test Projects/SampleApp.Playwright.Tests"
powershell "Test Projects/SampleApp.Playwright.Tests/bin/Debug/net8.0/playwright.ps1" install chromium
dotnet test "Test Projects/SampleApp.Playwright.Tests" --results-directory "Test Projects/SampleApp.Playwright.Tests" --logger "nunit;LogFilePath=result.xml"
dotnet run --project src/TestPilot.Cli -- analyze "Test Projects/SampleApp.Playwright.Tests/result.xml" --project "Test Projects/SampleApp.Playwright.Tests/SampleApp.Playwright.Tests.csproj"

---

Usage

testpilot analyze

testpilot analyze <result-file> [--format console|json|markdown|sarif|html] [--output <path>] [--history-dir <dir>] [--project <csproj>] [--framework nunit|xunit|mstest]

Option	Description
<result-file>	Path to a result file: NUnit3 XML for --framework nunit (default), TRX for xunit/mstest. Required.
--format	console (default), json, markdown, sarif, or html. sarif emits SARIF 2.1.0, the format GitHub Code Scanning (upload-sarif) and most other CI security/quality dashboards consume, so diagnostics show up as inline PR annotations instead of only in a standalone report. html is a single self-contained page (light/dark aware) for sharing a report without a CI dashboard.
--output	Write the report to a file instead of stdout.
--history-dir	Directory for TestPilot's local run-history store. Default: .testpilot.
--project	Path to the test project's .csproj. Enables real Roslyn-based test discovery and TestPilot's built-in Roslyn analyzers. Without it, TestPilot still runs the history-based analyzers (FLAKY001, SLOW001), just without source-level diagnostics.
--framework	nunit (default), xunit, or mstest — which adapter parses <result-file> and discovers tests in --project.

testpilot changed-files

testpilot changed-files --since <ref> [--repo <path>]

Lists files changed since a given git ref (commit SHA, branch, tag), including uncommitted changes — the building block for future test-impact analysis (not implemented yet).

Configuration file

Drop a testpilot.json in your working directory to set defaults instead of repeating flags. Every field is optional — set only the ones you want to stop repeating on the command line, and any field omitted here falls back to its built-in default or to whatever you pass on the CLI.

CLI flags always override testpilot.json, which overrides the built-in defaults.

Just pin the project and framework (the two flags you'd otherwise type on every run):

{
  "project": "MyProject.Tests/MyProject.Tests.csproj",
  "framework": "xunit"
}

# --project and --framework now come from testpilot.json
dotnet run --project src/TestPilot.Cli -- analyze result.trx

CI pipeline defaulting to a Markdown report, with history kept outside the repo checkout (still pass --output on the command line — it's not a config field):

{
  "format": "markdown",
  "historyDirectory": "/var/cache/testpilot-history"
}

Minimal — just redirect run history (e.g. a monorepo where each package needs its own history store), leaving --project/--framework/--format as CLI flags since they differ per invocation:

{
  "historyDirectory": ".testpilot/my-package"
}

Field	Equivalent flag	Default if omitted
format	--format	console
historyDirectory	--history-dir	.testpilot
project	--project	none (source-level diagnostics skipped)
framework	--framework	nunit

--output has no testpilot.json equivalent — it's always passed on the command line.

Diagnostics reference

All of TestPilot's own Roslyn analyzers run together against whatever --project you point at — each only reports when its target library is actually used, so there's no cost for projects that don't reference bUnit or Playwright.

Rule ID	Source	What it flags
FLAKY001	Application analyzer	A test with both passing and failing outcomes across recorded history.
SLOW001	Application analyzer	A test averaging more than 3x the suite's own average duration.
TP0001	Roslyn analyzer	A test method ([Test]/[Fact]/[TestMethod]) with an empty body.
TP0002	Roslyn analyzer	A call to Thread.Sleep in test code.
BUNIT0001	Roslyn analyzer (bUnit)	A local bUnit context (BunitContext/TestContext) created without using.
PW0001	Roslyn analyzer (Playwright)	A hard-coded Task.Delay inside a method with an IPage parameter, instead of Playwright's auto-waiting.

---

What's in V1

• Three framework adapters — NUnit (streaming NUnit3 XML), xUnit and MSTest (shared streaming TRX reader) — all with Roslyn-based test discovery ([Test]/[Fact]/[TestMethod]).
• Analysis pipeline — pluggable IAnalyzer pipeline; ships FLAKY001 and SLOW001.
• Roslyn analyzers — TP0001 (empty test, multi-framework) and TP0002 (Thread.Sleep usage), plus opt-in-flavored BUNIT0001 (bUnit) and PW0001 (Playwright) — all runnable both by TestPilot's own CLI and as normal analyzer packages in an IDE/build.
• Reporting — Console, JSON, and Markdown report generators.
• Run history — local, append-only JSON-Lines store (.testpilot/history/) so flaky/slow detection improves as you accumulate runs.
• Real source discovery — --project loads your test project via MSBuildWorkspace for genuine Roslyn-based analysis, not just result-file parsing.
• Git integration (basic) — changed-files lists what changed since a ref, via LibGit2Sharp.
• Configuration — testpilot.json for repo-committed defaults, including a --framework default.

Not yet built

• Test-impact analysis (mapping changed files to affected tests)
• AI-assisted insights
• Additional Roslyn analyzers (poor naming, shared mutable state, duplicate tests, improper async usage, etc.)
• A plugin system for selectively loading analyzer packages — right now all built-in analyzers (including bUnit/Playwright) run together regardless of which libraries a project uses; harmless since each only fires when its target symbols are present, but not yet configurable

---

Architecture

Clean Architecture, dependency direction enforced by an automated test suite (TestPilot.ArchitectureTests):

Presentation (Cli)
      ↓
Application (analysis pipeline, built-in analyzers)
      ↓
Domain (models, zero external dependencies)
      ↑
Infrastructure / framework adapters / Reporting / Roslyn analyzers

Domain and Application never reference NUnit, xUnit, MSTest, Roslyn, Git, MSBuild, or any AI provider directly — those live in Infrastructure and the adapter/reporting/analyzer projects, isolated behind interfaces defined in Domain/Abstractions.

src/
  TestPilot.Domain               Core models: TestSuite, TestCase, TestRun, Diagnostic, AnalysisResult, ...
  TestPilot.Application           AnalysisPipeline, FlakyTestAnalyzer, SlowTestAnalyzer
  TestPilot.Abstractions          Plugin contracts: ITestFrameworkAdapter, IAnalyzer, IReportGenerator, IHistoryStore
  TestPilot.Infrastructure         History store, MSBuildWorkspace loader, git integration, Roslyn diagnostic mapping/running
  TestPilot.Analyzers               Roslyn DiagnosticAnalyzers: TP0001, TP0002 (framework-agnostic)
  TestPilot.Analyzers.BUnit          Roslyn DiagnosticAnalyzer: BUNIT0001
  TestPilot.Analyzers.Playwright     Roslyn DiagnosticAnalyzer: PW0001
  TestPilot.NUnit                   NUnit3 XML result parsing + test discovery
  TestPilot.XUnit                   xUnit test discovery ([Fact]) + TRX result parsing
  TestPilot.MSTest                  MSTest test discovery ([TestMethod]) + TRX result parsing
  TestPilot.Trx                      Shared streaming TRX reader used by XUnit and MSTest adapters
  TestPilot.Reporting                Console/, Json/, Markdown/, Sarif/, and Html/ report generators
  TestPilot.Cli                       Composition root / CLI entry point

tests/
  One *.Tests project per src/ project above, plus:
  TestPilot.ArchitectureTests      Enforces the dependency directions shown above
  TestPilot.Cli.IntegrationTests   Spawns the real built Cli against a committed sample project

Building the solution (dotnet build TestPilot.slnx) and running the full test suite (dotnet test TestPilot.slnx) requires only the .NET 8 SDK.

---

Roadmap

Done (V1):

• Core architecture, NUnit/xUnit/MSTest adapters, analysis pipeline, Roslyn analyzers (including bUnit- and Playwright-aware ones), all three reporters, run history, real source discovery, basic git integration, configuration file.

Done (since v1.0.0):

• SARIF reporter, for GitHub Code Scanning and other SARIF-consuming CI dashboards
• HTML reporter, for sharing a report without a CI dashboard

Next:

• Test-impact analysis built on the existing changed-files primitive
• Additional Roslyn analyzers (naming, shared state, duplicate tests, async usage)
• A plugin system for selectively loading analyzer packages

Later:

• Optional AI-assisted insights (never required — everything above works without one)
• IDE extensions (VS Code, Visual Studio, Rider)
• Web dashboard

---

Contributing

Contributions, suggestions, feature requests, and bug reports are welcome. Formal contribution guidelines will be published as the project matures; for now, issues and PRs are the best way to get involved.

---

License

MIT — see LICENSE.