[release/10.0] Backport Helix job monitor

[akoeplinger]

Description

Backport the Helix job monitor integration from main/release/11.0, including the groundwork from #129690 and #132150 and the re-enablement in #131969.

Includes the subsequent fixes for empty stages (#132019), conditional monitor inclusion (#132882, #132884), parameter forwarding (#133002), performance monitoring (#132807, #133480), and internal credentials (#133633, #133688, #133885).

• Use Microsoft.DotNet.Helix.JobMonitor version 10.0.0-beta.26461.103, matching release/10.0's existing Arcade/VMR build.
• Preserve the existing SDK, shared Arcade templates, queues, and release/10.0 job layouts.
• Follow upstream enablement, except scheduled libraries outerloop runs retain release/10.0's existing warning-only reporting policy.
• Leave SuperPMI's post-Helix processing unchanged. The separate perf-slow enablement in [PERF] Use standalone Helix monitor for slow performance jobs #133726 is not included.

Customer Impact

CI infrastructure only; no shipped runtime changes. Moves Helix waiting and test-result reporting into the standalone monitor for the enabled pipelines.

Regression

Not a product regression fix; backports existing CI infrastructure and its follow-up fixes.

Testing

• Validated YAML/JSON/XML configuration and preservation of unrelated settings.
• Checked 43 entry pipelines, 149 forwarding sites, and 516 public/internal, PR/scheduled/manual, and normal/staging combinations, plus disabled-mode behavior.
• Exercised MSBuild child-property forwarding and the pinned SDK's waiting/reporter properties, including environment-based opt-in.
• Restored the monitor and verified its CLI compatibility under .NET 10.

No product build or live Azure DevOps pipeline execution was performed locally.

Risk

Changes CI scheduling and result reporting, not product behavior. Uses the monitor package matching the branch's existing toolset and retains the release/10.0-specific scheduled outerloop failure policy. End-to-end Azure DevOps validation is still required.

Note

This PR description and backport were prepared with GitHub Copilot.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[Copilot]

🟡 Changes recommended

Unresolved critical credential-scope findings can allow false-green monitoring, with additional policy and forwarding issues.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Backports standalone Helix job monitoring to release/10.0 CI pipelines, including tool pinning, configuration forwarding, and pipeline enablement.

Changes:

• Pins and registers the Helix Job Monitor.
• Propagates monitoring settings through MSBuild and pipeline templates.
• Enables monitoring across runtime, CoreCLR, libraries, WASM, installer, performance, and extra-platform pipelines.

File summaries

File	Change
src/libraries/sendtohelix.proj	Forwards monitor settings to child projects.
eng/Version.Details.xml	Pins the monitor dependency.
eng/Version.Details.props	Defines monitor package properties.
eng/pipelines/runtimelab.yml	Enables runtimelab monitoring.
eng/pipelines/runtime.yml	Adds runtime monitoring and credentials.
eng/pipelines/runtime-wasm.yml	Enables WASM monitoring.
eng/pipelines/runtime-wasm-optional.yml	Enables optional WASM monitoring.
eng/pipelines/runtime-wasm-non-libtests.yml	Enables non-library WASM monitoring.
eng/pipelines/runtime-wasm-libtests.yml	Enables WASM library monitoring.
eng/pipelines/runtime-wasm-dbgtests.yml	Enables WASM debug-test monitoring.
eng/pipelines/runtime-sanitized.yml	Enables sanitizer monitoring.
eng/pipelines/runtime-maccatalyst.yml	Enables Mac Catalyst monitoring.
eng/pipelines/runtime-llvm.yml	Enables LLVM monitoring.
eng/pipelines/runtime-ioslikesimulator.yml	Enables simulator monitoring.
eng/pipelines/runtime-ioslike.yml	Enables iOS-like monitoring.
eng/pipelines/runtime-extra-platforms.yml	Enables extra-platform monitoring.
eng/pipelines/runtime-cet.yml	Enables CET monitoring.
eng/pipelines/runtime-androidemulator.yml	Enables Android emulator monitoring.
eng/pipelines/runtime-android.yml	Enables Android monitoring.
eng/pipelines/performance/perf.yml	Adds performance monitoring.
eng/pipelines/libraries/run-test-job.yml	Forwards the monitor option.
eng/pipelines/libraries/outerloop.yml	Adds outerloop monitoring.
eng/pipelines/libraries/outerloop-mono.yml	Enables Mono outerloop monitoring.
eng/pipelines/libraries/helix.yml	Passes monitor settings to submissions.
eng/pipelines/installer/helix.yml	Forwards monitor settings to installer submissions.
eng/pipelines/extra-platforms/runtime-extra-platforms-wasm.yml	Propagates monitoring to WASM jobs.
eng/pipelines/extra-platforms/runtime-extra-platforms-other.yml	Propagates monitoring to other platforms.
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml	Propagates monitoring to Mac Catalyst jobs.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml	Propagates monitoring to simulator jobs.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml	Propagates monitoring to iOS-like jobs.
eng/pipelines/extra-platforms/runtime-extra-platforms-androidemulator.yml	Propagates monitoring to emulator jobs.
eng/pipelines/extra-platforms/runtime-extra-platforms-android.yml	Propagates monitoring to Android jobs.
eng/pipelines/coreclr/tieringtest.yml	Enables tiering-test monitoring.
eng/pipelines/coreclr/templates/run-paltests-step.yml	Forwards monitoring to PAL tests.
eng/pipelines/coreclr/runtime-nativeaot-outerloop.yml	Enables NativeAOT monitoring.
eng/pipelines/coreclr/runincontext.yml	Enables RunInContext monitoring.
eng/pipelines/coreclr/release-tests.yml	Enables release-test monitoring.
eng/pipelines/coreclr/r2r.yml	Enables R2R monitoring.
eng/pipelines/coreclr/r2r-extra.yml	Enables extra R2R monitoring.
eng/pipelines/coreclr/nativeaot-post-build-steps.yml	Propagates monitoring to NativeAOT tests.
eng/pipelines/coreclr/libraries-pgo.yml	Enables PGO monitoring.
eng/pipelines/coreclr/libraries-jitstressregs.yml	Enables JIT stress-reg monitoring.
eng/pipelines/coreclr/libraries-jitstress2-jitstressregs.yml	Enables combined JIT stress monitoring.
eng/pipelines/coreclr/libraries-jitstress.yml	Enables JIT stress monitoring.
eng/pipelines/coreclr/libraries-jitstress-random.yml	Enables randomized JIT stress monitoring.
eng/pipelines/coreclr/libraries-gcstress0x3-gcstress0xc.yml	Enables library GC stress monitoring.
eng/pipelines/coreclr/libraries-gcstress-extra.yml	Enables extra library GC stress monitoring.
eng/pipelines/coreclr/interpreter.yml	Enables interpreter monitoring.
eng/pipelines/coreclr/gcstress0x3-gcstress0xc.yml	Enables GC stress monitoring.
eng/pipelines/coreclr/gcstress-extra.yml	Enables extra GC stress monitoring.
eng/pipelines/coreclr/gc-standalone.yml	Enables standalone GC monitoring.
eng/pipelines/coreclr/gc-simulator.yml	Enables GC simulator monitoring.
eng/pipelines/coreclr/gc-longrunning.yml	Enables long-running GC monitoring.
eng/pipelines/coreclr/crossgen2.yml	Enables Crossgen2 monitoring.
eng/pipelines/coreclr/crossgen2-outerloop.yml	Enables Crossgen2 outerloop monitoring.
eng/pipelines/coreclr/crossgen2-gcstress.yml	Enables Crossgen2 GC stress monitoring.
eng/pipelines/coreclr/crossgen2-composite.yml	Enables composite Crossgen2 monitoring.
eng/pipelines/coreclr/ci.yml	Enables CoreCLR CI monitoring.
eng/pipelines/common/variables.yml	Defines the monitor toggle.
eng/pipelines/common/templates/wasm-runtime-tests.yml	Propagates monitoring to runtime tests.
eng/pipelines/common/templates/wasm-library-tests.yml	Propagates monitoring to library tests.
eng/pipelines/common/templates/wasm-library-aot-tests.yml	Propagates monitoring to AOT tests.
eng/pipelines/common/templates/simple-wasm-build-tests.yml	Propagates monitoring to simple WASM tests.
eng/pipelines/common/templates/runtimes/send-to-helix-step.yml	Adds the monitor MSBuild property.
eng/pipelines/common/templates/runtimes/run-test-job.yml	Forwards monitoring to runtime submissions.
eng/pipelines/common/templates/runtimes/build-runtime-tests-and-send-to-helix.yml	Forwards monitoring through runtime builds.
eng/pipelines/common/templates/browser-wasm-build-tests.yml	Propagates monitoring to browser tests.
.config/dotnet-tools.json	Registers the pinned monitor tool.

Review details

Suppressed comments (7)

eng/pipelines/extra-platforms/runtime-extra-platforms-wasm.yml:39

• This enables the standalone monitor for the WASI AOT block below, but that block sets shouldContinueOnError: true (line 47), so its Helix failures are intentionally non-gating. The monitor job still defaults to continueOnError: false and failWorkItemsWithFailedTests: true, which will make those failures fail the stage even though the submitter step succeeds. Keep the monitor disabled for this block or explicitly propagate the non-gating policy.
  eng/pipelines/extra-platforms/runtime-extra-platforms-wasm.yml:128
• This enables the standalone monitor for the WASI AOT block below, but that block sets shouldContinueOnError: true (line 135), so its Helix failures are intentionally non-gating. The monitor job still defaults to continueOnError: false and failWorkItemsWithFailedTests: true, which will make those failures fail the stage even though the submitter step succeeds. Keep the monitor disabled for this block or explicitly propagate the non-gating policy.
  eng/pipelines/extra-platforms/runtime-extra-platforms-wasm.yml:158
• This enables the standalone monitor for the smoke-test block below, but shouldContinueOnError is true for runtime-extra-platforms (not(parameters.isWasmOnlyBuild) at line 167), so those Helix failures are intentionally non-gating there. The monitor job still defaults to failing on failed work items and will turn this into a failed stage. Make this forwarding conditional on parameters.isWasmOnlyBuild, or otherwise carry the non-gating policy into the monitor.
  eng/pipelines/performance/perf.yml:65
• The public perf pipeline also omits DotNet-HelixApi-Access at stage scope, but this unconditionally forwards $(HelixApiAccessToken). That leaves the monitor with an unresolved non-empty token on public runs, so authenticated discovery can fail and allowNoHelixJobs: true can hide the failure. Use the internal token only for the internal project and '' for public runs.
  eng/pipelines/performance/perf.yml:63
• Adding the monitor job here does not enable out-of-band submission for the performance workloads: neither runtime-wasm-perf-jobs.yml@performance nor runtime-perf-jobs.yml@performance receives a useHelixMonitor/jobParameters value from this file. The referenced performance run-performance-job.yml currently invokes its send-to-helix-step.yml without /p:EnableHelixJobMonitor, so these submitter jobs will continue waiting synchronously while the new monitor independently processes the same jobs. Forward the flag through the performance-template parameter chain and into its send step.
  eng/pipelines/runtime.yml:76
• On public runs the stage-level DotNet-HelixApi-Access group is intentionally omitted above, so HelixApiAccessToken is undefined here. Azure then passes the unresolved $(HelixApiAccessToken) to the monitor instead of an anonymous empty token; discovery can fail while allowNoHelixJobs: true turns the run green. Select the token only for the internal project and pass '' otherwise.
  eng/pipelines/runtime.yml:54
• This enables the monitor for staging PR runs as well, but eng/pipelines/common/xplat-setup.yml:19 turns every submitter job in a *staging PR into shouldContinueOnError. That setting is forwarded to the Helix send step, while this standalone monitor still has continueOnError: false and treats failed work items as fatal, so it can make intentionally non-gating staging failures fail the stage. Exclude staging PRs from monitor mode or propagate the same non-gating policy to the monitor.

• Files reviewed: 68/68 changed files
• Comments generated: 41
• Review effort level: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.