Use helix job monitor by dibarbet · Pull Request #84810 · dotnet/roslyn

dibarbet · 2026-08-07T21:02:02Z

Implements the helix job monitor from arcade.

Since the monitor runs per stage, I combined some of the stages so that we only have two monitor jobs (corresponding to each build artifacts type). This should not negatively impact retries since the job monitor retries individual work items internally.

Note - we do lose some of the custom monitoring:

Queue backup warning - the monitor job itself regularly logs waiting work item counts, so this doesn't seem too impactful.
No synthetic timeout test results - we could consider contributing this feature to the arcade monitor.

Microsoft Reviewers: Open in CodeFlow

azure-pipelines · 2026-08-07T21:02:22Z

Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

Let cancellation propagate, retain submitted Helix job IDs in logs, and remove documentation changes from this feature. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

Resolve the Arcade dependency conflict by aligning the Helix job monitor with Arcade 11.0.0-beta.26407.6. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

azure-pipelines · 2026-08-10T20:39:36Z

Azure Pipelines: Successfully started running 2 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

This PR updates Roslyn’s Helix test submission and Azure Pipelines configuration to rely on the Arcade Helix Job Monitor for external monitoring/reporting, replacing the in-tool Helix polling/cancellation logic previously implemented in RunTests.

Changes:

Update RunTests’ Helix submission project to enable Helix Job Monitor and simplify Helix submission flow.
Wire Azure Pipelines to add Helix job monitor jobs and add the Helix job monitor tool/dependency versions.
Remove the in-repo HelixApi polling implementation used for Helix job monitoring.

Show a summary per file

File	Description
src/Tools/RunTests/Options.cs	Updates Helix option help text to reflect external job monitor usage.
src/Tools/RunTests/HelixTestRunner.cs	Removes Helix API polling/cancellation logic; enables Helix job monitor in generated Helix project; simplifies submission execution path.
src/Tools/RunTests/HelixApi.cs	Removes the custom Helix REST API helper and related response models.
eng/Version.Details.xml	Adds `Microsoft.DotNet.Helix.JobMonitor` toolset dependency entry.
eng/Version.Details.props	Adds properties for Helix Job Monitor package/version.
eng/pipelines/test-windows-job.yml	Removes explicit `SYSTEM_ACCESSTOKEN` export from the test submission step.
eng/pipelines/test-unix-job.yml	Removes explicit `SYSTEM_ACCESSTOKEN` export from the test submission step.
eng/common/dotnet.sh	Adds a dotnet bootstrapper used by the job monitor template to restore/run tools.
dotnet-tools.json	Adds a dotnet tool entry for the Helix Job Monitor.
azure-pipelines.yml	Adds pool provider variables template, renames/reshapes Windows test stages, and adds Helix Job Monitor jobs to stages.

Review details

Suppressed comments (2)

azure-pipelines.yml:347

Same issue here: without dependsOn/condition, the Helix job monitor can start before Helix submissions complete or get skipped if a test job fails. It should depend on the Helix test jobs in this stage and run under succeededOrFailed() so results are still reported when tests fail.

  - template: /eng/common/core-templates/job/helix-job-monitor.yml
    parameters:
      helixAccessToken: $(HelixApiAccessToken)

azure-pipelines.yml:395

The Helix job monitor job should depend on the Helix submission job(s) in this stage; otherwise it can start before any Helix jobs exist and fail early. Also set condition: succeededOrFailed() so the monitor still publishes results when the test job fails.

    - template: /eng/common/core-templates/job/helix-job-monitor.yml
      parameters:
        helixAccessToken: $(HelixApiAccessToken)

Files reviewed: 9/10 changed files
Comments generated: 5
Review effort level: Lite

RunTests still uses SYSTEM_ACCESSTOKEN to query Azure DevOps test history before submitting Helix work. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

…x-job-monitor

Copilot

Review details

Suppressed comments (1)

dotnet-tools.json:16

The new local tool entry doesn’t specify rollForward: false like the existing powershell tool entry. If the intent is to keep tool execution deterministic across agents (especially for CI), this tool should follow the same setting.

    "microsoft.dotnet.helix.jobmonitor": {
      "version": "11.0.0-beta.26407.8",
      "commands": [
        "dotnet-helix-job-monitor"
      ]
    }

Files reviewed: 9/10 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

Review details

Suppressed comments (2)

src/Tools/RunTests/HelixTestRunner.cs:85

Process returned by StartHelixJob is no longer disposed. This can leak OS handles/pipes for stdout/stderr redirection during long CI runs. Wrap the process in a using declaration (or dispose it in the finally).

        var helixProjectFilePath = await CreateHelixArtifactsAsync(options, assemblies, cts.Token).ConfigureAwait(false);
        var process = StartHelixJob(options, helixProjectFilePath);
        try
        {
            await process.WaitForExitAsync(cts.Token);
            return process.ExitCode;
        }

eng/Version.Details.xml:159

The new Microsoft.DotNet.Helix.JobMonitor dependency uses a different Arcade SHA than the other Arcade toolset dependencies at the same version (11.0.0-beta.26407.8). This inconsistency can break dependency flow/update tooling because the SHA is expected to correspond to the commit that produced that version.

    <Dependency Name="Microsoft.DotNet.Helix.JobMonitor" Version="11.0.0-beta.26407.8">
      <Uri>https://github.com/dotnet/arcade</Uri>
      <Sha>41232a67a0c1a77d425a6bef59ded8a81a2fdf3c</Sha>
    </Dependency>

Files reviewed: 9/10 changed files
Comments generated: 0 new
Review effort level: Lite

jjonescz · 2026-08-11T08:11:16Z

      helixApiAccessToken: $(HelixApiAccessToken)
      poolParameters: ${{ parameters.windowsPool }}

+  - template: /eng/common/core-templates/job/helix-job-monitor.yml


So we must have one monitor leg per one stage?

This will continue to offer us the ability to initiate a retry earlier for shorter stages. Retries are performed by rerunning the monitor job which then retries only the failed work items. David did combine some small stages where it made sense.

Makes sense but I think the main motivation for the monitor job is to save CI resources, so one job would do that better. I guess it's a tradeoff. (Also the monitor job only retries the actually failed work item so it will already be so much faster to retry failed jobs than what we have now.)

Yeah - and the arcade implementation requires a monitor per stage. So I merged a few stages where it made sense. We now have just two windows stages (tied to the build debug or release)

jjonescz · 2026-08-11T08:14:23Z

cc @jaredpar

dibarbet · 2026-08-11T17:54:31Z

-
-                    if (workItems.Waiting > 0 && elapsed > TimeSpan.FromMinutes(20))
-                    {
-                        ConsoleUtil.Warning($"Helix job {helixJobId} has {details.WorkItems.Waiting} queued work items after {elapsed:hh\\:mm}. This indicates a queue backup");


The job monitor itself does log info regularly. Example snippet:

2026-08-10T21:53:47.2130985Z info: ℹ️ Status: 1 processed / 2 completed / 3 running / 0 waiting jobs 2026-08-10T21:53:47.2131389Z 32 processed / 42 completed / 33 running / 0 waiting work items 2026-08-10T21:53:47.9756396Z info: 0 test results for job 'Test_Windows_CoreClr_IOperation_Debug - windows.10.amd64.open (7838176b-9a1d-43c9-a043-94294c12bcd9)' processed. 2026-08-10T21:55:19.3488874Z info: Job Test_Windows_CoreClr_RuntimeAsync_Debug - windows.10.amd64.open (29ff0112-b7a9-4b46-9408-e33c602df27f) completed. Processing test results... 2026-08-10T21:55:19.3489484Z https://helix.dot.net/api/2019-06-17/jobs/29ff0112-b7a9-4b46-9408-e33c602df27f/details 2026-08-10T21:55:19.3497562Z info: ✅ Job 'Test_Windows_CoreClr_RuntimeAsync_Debug - windows.10.amd64.open (29ff0112-b7a9-4b46-9408-e33c602df27f)' succeeded (5 passed, 0 failed) 2026-08-10T21:55:19.3498296Z https://helix.dot.net/api/2019-06-17/jobs/29ff0112-b7a9-4b46-9408-e33c602df27f/details 2026-08-10T21:55:19.8863112Z info: ℹ️ Status: 2 processed / 3 completed / 2 running / 0 waiting jobs 2026-08-10T21:55:19.8863638Z 42 processed / 47 completed / 28 running / 0 waiting work items

However we lose the synthetic test results - we could consider augmenting the helix monitor with that capability

Print a green message after successful submission directing users to the stage monitor for status and automatic retries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

Copilot

Review details

Suppressed comments (1)

src/Tools/RunTests/HelixTestRunner.cs:80

Process implements IDisposable and should be disposed to avoid leaking OS handles in longer-running invocations. Since this method now only waits for submission to exit, using a using var is a simple way to ensure cleanup.

        var helixProjectFilePath = await CreateHelixArtifactsAsync(options, assemblies, cts.Token).ConfigureAwait(false);
        var process = StartHelixJob(options, helixProjectFilePath);
        try

Files reviewed: 9/10 changed files
Comments generated: 0 new
Review effort level: Lite

Copilot

Review details

Files reviewed: 7/8 changed files
Comments generated: 0 new
Review effort level: Lite

…x-job-monitor

Copilot

Review details

Files reviewed: 7/8 changed files
Comments generated: 0 new
Review effort level: Lite

## Summary - backport the Roslyn `RunTests` changes needed for the standalone Helix Job Monitor - consolidate Windows Helix submissions and add monitor jobs for Windows, Linux, and macOS - pin the monitor tool to the Arcade `release/10.0` build used by #85150 This PR is stacked on #85150 so the diff contains only the test runner and CI infrastructure changes. After #85150 merges, this PR can be retargeted to `release/dev18.0`. Backports the relevant behavior from #84810, #84872, #84873, and #85070. ###### Microsoft Reviewers: [Open in CodeFlow](https://microsoft.github.io/open-pr/?codeflow=https://github.com/dotnet/roslyn/pull/85153) --------- Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4f9174b1-67dc-4def-b0c1-3b8e1746e681

Use helix job monitor

6ac3357

dibarbet added 4 commits August 7, 2026 14:58

fix

1211117

Make Arcade dotnet script executable

a2e5f09

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 250706b2-8fc3-4688-aabf-7de5304b4b5c

dibarbet marked this pull request as ready for review August 10, 2026 20:39

dibarbet requested a review from a team as a code owner August 10, 2026 20:39

Copilot AI lite review requested due to automatic review settings August 10, 2026 20:39

Copilot started reviewing on behalf of dibarbet August 10, 2026 20:39 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Comment thread src/Tools/RunTests/HelixTestRunner.cs

Comment thread azure-pipelines.yml

Comment thread eng/pipelines/test-windows-job.yml

Comment thread eng/pipelines/test-unix-job.yml

Comment thread dotnet-tools.json

Copilot AI review requested due to automatic review settings August 10, 2026 20:53

Copilot started reviewing on behalf of dibarbet August 10, 2026 20:53 View session

Merge remote-tracking branch 'upstream/main' into dibarbet-adopt-heli…

19d201c

…x-job-monitor

Copilot AI reviewed Aug 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 10, 2026 20:59

Copilot started reviewing on behalf of dibarbet August 10, 2026 20:59 View session

Copilot AI reviewed Aug 10, 2026

View reviewed changes

JoeRobich approved these changes Aug 11, 2026

View reviewed changes

jjonescz approved these changes Aug 11, 2026

View reviewed changes

jjonescz added the Area-Infrastructure label Aug 11, 2026

dibarbet commented Aug 11, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 11, 2026 18:00

Copilot started reviewing on behalf of dibarbet August 11, 2026 18:01 View session

Copilot AI reviewed Aug 11, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings August 11, 2026 18:07

Copilot started reviewing on behalf of dibarbet August 11, 2026 18:08 View session

revert unnecessary changes

e6f7116

dibarbet force-pushed the dibarbet-adopt-helix-job-monitor branch from 2573521 to e6f7116 Compare August 11, 2026 18:08

Copilot AI reviewed Aug 11, 2026

View reviewed changes

akhera99 approved these changes Aug 11, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into dibarbet-adopt-heli…

db47dc0

…x-job-monitor

Copilot AI review requested due to automatic review settings August 12, 2026 18:02

Copilot started reviewing on behalf of dibarbet August 12, 2026 18:03 View session

Copilot AI reviewed Aug 12, 2026

View reviewed changes

JoeRobich approved these changes Aug 12, 2026

View reviewed changes

dibarbet merged commit 802bbca into dotnet:main Aug 12, 2026
30 checks passed

dibarbet deleted the dibarbet-adopt-helix-job-monitor branch August 12, 2026 19:59

dotnet-maestro Bot mentioned this pull request Aug 13, 2026

[main] Source code updates from dotnet/roslyn dotnet/dotnet#8265

Merged

dotnet-bot mentioned this pull request Aug 15, 2026

[Automated] PRs inserted in VS build main-12114.117 #84898

Closed

davidwengier mentioned this pull request Aug 24, 2026

Update roslyn to 5.11.0-1.26424.9 dotnet/vscode-csharp#9701

Merged

jjonescz added this to the 18.11 milestone Aug 25, 2026

JoeRobich mentioned this pull request Sep 3, 2026

[release/dev18.0] Enable Helix Job Monitor in CI #85153

Merged

Uh oh!

Conversation

dibarbet commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Microsoft Reviewers: Open in CodeFlow

Uh oh!

azure-pipelines Bot commented Aug 7, 2026

Uh oh!

azure-pipelines Bot commented Aug 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Uh oh!

jjonescz Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

JoeRobich Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

jjonescz Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

dibarbet Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

jjonescz commented Aug 11, 2026

Uh oh!

dibarbet Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

dibarbet commented Aug 7, 2026 •

edited

Loading

dibarbet Aug 11, 2026 •

edited

Loading