Run SOS test harness on Helix by steveisok · Pull Request #6018 · dotnet/diagnostics

steveisok · 2026-09-09T17:25:41Z

Summary

add deterministic capture-family sharding with one shared SOS correlation payload per submission
add Windows and Unix launchers, including the musl-safe dotnet SOS.Tests.dll invocation
integrate SOS Helix execution into the pipeline with 8 dump + 2 live work items generally and 32 dump + 2 live work items on macOS
serialize ARM64 and Alpine dump work, and restrict Alpine dump coverage to Heap and Full

Supersedes #6002, whose native stacked-PR metadata could not be repaired after its parent was squash-merged.

Validation

SOS.Tests Debug build
45 focused shard, validity, capability, and Unix payload tests
reporter-enabled no-submit work-item graphs for Windows x64, Linux x64/ARM64, linux-musl x64, and macOS x64/ARM64
XML, YAML, and JSON parsing; Bash syntax, LF/executable mode, diff whitespace, local-skip scope, and HELIX-to-SOSHARNESS boundary checks

Add deterministic capture-family sharding, a shared SOS Helix payload, cross-platform launchers, and pipeline scheduling while preserving the lower harness layers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Validate the project-owned shared payload independently from SDK-added correlation payloads, and retain the Azure Pipelines reporter in CI submissions.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71

Avoid deterministic native LLDB and dotnet-dump host crashes when dump tests run concurrently on Linux ARM64, while preserving existing shard counts and concurrency elsewhere.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71

Avoid intermittent .NET 8 Mini dump capture failures in the unprivileged Alpine Helix container while preserving Heap, Full, and live coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71

Limit unprivileged Alpine Helix dump work items to one MTP thread so concurrent snapshot families cannot race createdump's /proc authorization. Keep all eight shards, Heap/Full coverage, and live parallelism. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 48106f1a-69fc-430d-917a-89bfe78a5f71

Copilot

Copilot review overview

🟡 Changes recommended

The pipeline template uses condition: succeeded() incorrectly nested under template parameters, which is likely to break YAML/template evaluation (plus Helix work item commands should quote correlation payload paths).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review tier: Lite
Findings: 1 · 1

New issues introduced by this change (2)

Severity	Finding
	`eng/pipelines/build.yml` — `condition: succeeded()` is indented under the `parameters:` block, so it will be passed as a…
	`eng/helix/SOS.Tests.Helix.proj` — The Helix work item `Command` strings don't quote `$HELIX_CORRELATION_PAYLOAD` /…

What changed in this PR

This PR integrates the SOS.Tests harness into Helix execution, adding deterministic matrix sharding (capture-family based) and Helix launchers/payload wiring so SOS scenarios can run reliably across Windows, Linux (including musl), and macOS.

Changes:

Add deterministic capture-family sharding support and allow shard/allow-list runs to skip empty theories instead of failing discovery.
Add Helix launchers and payload routing (upload-root handling, writable overlays for executables/native assets, LLDB/DbgEng configuration).
Wire SOS.Tests Helix submission into the Azure Pipelines templates and official pipeline graph (including Helix SDK enablement).

File	Description
src/tests/SOS.Tests/WindowsTheoryAttribute.cs	Allow empty-theory skipping when sharding/allow-lists are active.
src/tests/SOS.Tests/UnixPayloadTests.cs	New tests validating upload-root routing and Unix writable overlay behavior.
src/tests/SOS.Tests/TestMatrices.cs	Apply shard filtering consistently while keeping unsharded enumeration available.
src/tests/SOS.Tests/TestConfigValidityTests.cs	Add coverage for musl dump-kind allow-list behavior.
src/tests/SOS.Tests/TestConfigShardTests.cs	New unit tests for sharding controls, stable hash, and capture-family semantics.
src/tests/SOS.Tests/SosTheoryAttribute.cs	Allow empty-theory skipping when sharding/allow-lists are active.
src/tests/SOS.Tests/SosReplayAttribute.cs	Route replay outputs to a Helix upload root when configured.
src/tests/SOS.Tests/SOS.Tests.csproj	Replace OfficialBuild skip with harness-controlled local-skip knob.
src/tests/SOS.Tests/README.md	Document sharding, new environment controls, and Helix execution model.
src/tests/SOS.TestHarness/ToolPaths.cs	Add DbgEng override and use overridable .NET root for host runtime discovery.
src/tests/SOS.TestHarness/TestConfig.cs	Implement shard selection, stable hashing, and reusable allow-list helper.
src/tests/SOS.TestHarness/SnapshotStore.cs	Add writable executable overlay support and enforce prebuilt-only execution mode.
src/tests/SOS.TestHarness/RepoLayout.cs	Add repo/dotnet/native/scratch override roots and directory resolution helper.
src/tests/SOS.TestHarness/HostDiagnostics.cs	Route crash-dump diagnostics to upload root when configured.
global.json	Add Microsoft.DotNet.Helix.Sdk to msbuild SDK list.
eng/pipelines/build.yml	Add Helix submission knobs, helix-only mode, and additional-architecture build flow.
eng/helix/sos/run-sos-tests.sh	New Unix Helix launcher handling overlays, LLDB setup, and sharded execution.
eng/helix/sos/run-sos-tests.cmd	New Windows Helix launcher handling signature checks and sharded execution.
eng/helix/sos/debuggee-entitlements.plist	New macOS debuggee entitlements for Helix execution.
eng/helix/SOS.Tests.Helix.proj	New Helix submission project defining payload + sharded work items.
diagnostics.yml	Update pipeline stage/legs to submit SOS.Tests on Helix across platforms/architectures.

Suppressed comments (1)

eng/pipelines/build.yml:340

Same indentation issue as above: condition: succeeded() is nested inside parameters: for the Helix template step, which will be treated as a template parameter rather than a step condition.

            ${{ if eq(variables['System.TeamProject'], 'public') }}:
              Creator: dotnet-bot
            ${{ else }}:
              HelixAccessToken: $(HelixApiAccessToken)
            condition: succeeded()

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

+            ${{ if eq(variables['System.TeamProject'], 'public') }}:
+              Creator: dotnet-bot
+            ${{ else }}:
+              HelixAccessToken: $(HelixApiAccessToken)
+            condition: succeeded()


+        <Command Condition="'$(TargetOS)' == 'Windows_NT'">%25HELIX_CORRELATION_PAYLOAD%25\eng\helix\sos\run-sos-tests.cmd $(Configuration) $(TargetRid) %(_SOSShard.ShardIndex) %(_SOSShard.ShardCount) %(_SOSShard.Liveness) $(NetCoreAppTestTargetFramework)</Command>
+        <Command Condition="'$(TargetOS)' != 'Windows_NT' and '%(_SOSShard.MaxParallelThreads)' == ''">$(SOSHelixPlatformTestEnvironment)$(SOSHelixTestEnvironment)bash $HELIX_CORRELATION_PAYLOAD/eng/helix/sos/run-sos-tests.sh $(Configuration) $(TargetRid) %(_SOSShard.ShardIndex) %(_SOSShard.ShardCount) %(_SOSShard.Liveness) $(NetCoreAppTestTargetFramework)</Command>
+        <Command Condition="'$(TargetOS)' != 'Windows_NT' and '%(_SOSShard.MaxParallelThreads)' != ''">$(SOSHelixPlatformTestEnvironment)$(SOSHelixTestEnvironment)bash $HELIX_CORRELATION_PAYLOAD/eng/helix/sos/run-sos-tests.sh $(Configuration) $(TargetRid) %(_SOSShard.ShardIndex) %(_SOSShard.ShardCount) %(_SOSShard.Liveness) $(NetCoreAppTestTargetFramework) %(_SOSShard.MaxParallelThreads)</Command>


Publish executable overlays only after a uniquely staged file is complete and executable. Preserve a valid winner across concurrent harness processes and cover concurrent preparation with a runnable Unix payload test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d99a69bd-fec9-4447-ba95-64bc8a854d16

steveisok · 2026-09-10T01:19:52Z

Alpine x64 release failures are #6011. Known issue may need expanding.

steveisok · 2026-09-10T20:54:31Z

Closing in favor of #6019. Thanks @max-charlamb !

## Summary - move SOS.Tests payload staging into the test project and use a generic Helix dispatcher - run one work item per OS, RID, and configuration instead of sharding the test matrix - centralize repository, payload, scratch, executable, and upload paths in RepoLayout - move SOS-specific submission into dedicated pipeline templates with centralized queue definitions - preserve platform preparation for Windows signatures, Unix executable overlays, macOS codesigning, and Alpine restrictions This is based on and supersedes the implementation approach in #6018. ## Validation - built SOS.Tests for win-x64 Debug - ran 16 focused configuration and payload tests locally - staged and validated the Windows Helix payload - ran the focused tests from the staged payload using standard Helix environment variables - parsed the modified pipeline YAML and validated the generic Helix project configuration --------- Co-authored-by: Max Charlamb <maxcharlamb@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 90d32376-768b-4f8c-a982-54247c372cf3

steveisok and others added 5 commits September 9, 2026 13:21

Run the SOS test harness on Helix

cca9c81

Add deterministic capture-family sharding, a shared SOS Helix payload, cross-platform launchers, and pipeline scheduling while preserving the lower harness layers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI lite review requested due to automatic review settings September 9, 2026 17:25

steveisok requested a review from a team as a code owner September 9, 2026 17:25

Copilot started reviewing on behalf of steveisok September 9, 2026 17:26 View session

steveisok mentioned this pull request Sep 9, 2026

Run the SOS test harness on Helix #6002

Closed

Copilot AI reviewed Sep 9, 2026

View reviewed changes

max-charlamb mentioned this pull request Sep 9, 2026

Refactor SOS test harness Helix integration #6019

Merged

build-analysis Bot mentioned this pull request Sep 10, 2026

dotnet-dump crashes enumerating sync blocks for .NET 10 Heap dumps on Alpine #6011

Open

steveisok closed this Sep 10, 2026

Navigation Menu

Uh oh!

Run SOS test harness on Helix - #6018

Run SOS test harness on Helix#6018
steveisok wants to merge 6 commits into
mainfrom
steveisok-sos-helix-integration-standalone

steveisok commented Sep 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

steveisok commented Sep 10, 2026

Uh oh!

steveisok commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

steveisok commented Sep 9, 2026

Summary

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

steveisok commented Sep 10, 2026

Uh oh!

steveisok commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants