Skip to content

Use helix job monitor #84810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dibarbet merged 10 commits into from
Aug 12, 2026
+82 −486
Merged

Use helix job monitor #84810

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6ac3357
Use helix job monitor
dibarbet Aug 7, 2026
1211117
fix
dibarbet Aug 7, 2026
a2e5f09
Make Arcade dotnet script executable
dibarbet Aug 7, 2026
b1959af
Address Helix monitor review feedback
dibarbet Aug 10, 2026
e032e62
Merge upstream/main into Helix job monitor branch
dibarbet Aug 10, 2026
cb06955
Preserve test history authentication
dibarbet Aug 10, 2026
19d201c
Merge remote-tracking branch 'upstream/main' into dibarbet-adopt-heli…
dibarbet Aug 10, 2026
86d3b6a
Point submitters to Helix job monitor
dibarbet Aug 11, 2026
e6f7116
revert unnecessary changes
dibarbet Aug 11, 2026
db47dc0
Merge remote-tracking branch 'upstream/main' into dibarbet-adopt-heli…
dibarbet Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 51 additions & 49 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@ pr:
- '*.md'

variables:
- template: /eng/common/templates/variables/pool-providers.yml

# This value is conditionally overriden by the variable group DotNet-HelixApi-Access.
# ADO only lets us conditionally include a variable group on individual stages.
- name: HelixApiAccessToken
Expand Down Expand Up @@ -199,7 +201,7 @@ stages:
name: 'Managed'
container: 'mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream9'

- stage: Windows_Debug_Desktop
- stage: Windows_Debug_Tests
dependsOn: Windows_Debug_Build
variables:
- ${{ if ne(variables['System.TeamProject'], 'public') }}:
Expand All @@ -216,53 +218,6 @@ stages:
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- stage: Windows_Release_Desktop
dependsOn: Windows_Release_Build
variables:
- ${{ if ne(variables['System.TeamProject'], 'public') }}:
- group: DotNet-HelixApi-Access
jobs:
- ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Release 32'
jobName: Test_Windows_Desktop_Release_32
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x86
helixQueueName: $(HelixWindowsVsQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Release 64'
jobName: Test_Windows_Desktop_Release_64
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x64
helixQueueName: $(HelixWindowsVsQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Spanish Release 64'
jobName: Test_Windows_Desktop_Spanish_Release_64
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x64
helixQueueName: $(HelixWindowsSpanishQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- stage: Windows_Debug_CoreClr
dependsOn: Windows_Debug_Build
variables:
- ${{ if ne(variables['System.TeamProject'], 'public') }}:
- group: DotNet-HelixApi-Access
jobs:
- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows CoreClr Debug'
Expand Down Expand Up @@ -331,12 +286,51 @@ stages:
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- stage: Windows_Release_CoreClr
- template: /eng/common/core-templates/job/helix-job-monitor.yml
parameters:
helixAccessToken: $(HelixApiAccessToken)

Comment thread
dibarbet marked this conversation as resolved.
Resolved
- stage: Windows_Release_Tests
dependsOn: Windows_Release_Build
variables:
- ${{ if ne(variables['System.TeamProject'], 'public') }}:
- group: DotNet-HelixApi-Access
jobs:
- ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Release 32'
jobName: Test_Windows_Desktop_Release_32
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x86
helixQueueName: $(HelixWindowsVsQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Release 64'
jobName: Test_Windows_Desktop_Release_64
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x64
helixQueueName: $(HelixWindowsVsQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows Desktop Spanish Release 64'
jobName: Test_Windows_Desktop_Spanish_Release_64
testArtifactName: Transport_Artifacts_Windows_Release
configuration: Release
testArguments: -testDesktop -testArch x64
helixQueueName: $(HelixWindowsSpanishQueueName)
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- template: eng/pipelines/test-windows-job.yml
parameters:
testRunName: 'Test Windows CoreClr Release'
Expand All @@ -348,6 +342,10 @@ stages:
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.windowsPool }}

- template: /eng/common/core-templates/job/helix-job-monitor.yml

@jjonescz jjonescz Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we must have one monitor leg per one stage?

@JoeRobich JoeRobich Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will continue to offer us the ability to initiate a retry earlier for shorter stages. Retries are performed by rerunning the monitor job which then retries only the failed work items. David did combine some small stages where it made sense.

@jjonescz jjonescz Aug 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense but I think the main motivation for the monitor job is to save CI resources, so one job would do that better. I guess it's a tradeoff. (Also the monitor job only retries the actually failed work item so it will already be so much faster to retry failed jobs than what we have now.)

@dibarbet dibarbet Aug 11, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah - and the arcade implementation requires a monitor per stage. So I merged a few stages where it made sense. We now have just two windows stages (tied to the build debug or release)

parameters:
helixAccessToken: $(HelixApiAccessToken)

- stage: Unix_Debug_CoreClr
dependsOn: Unix_Build
variables:
Expand Down Expand Up @@ -392,6 +390,10 @@ stages:
helixApiAccessToken: $(HelixApiAccessToken)
poolParameters: ${{ parameters.ubuntuPool }}

- template: /eng/common/core-templates/job/helix-job-monitor.yml
parameters:
helixAccessToken: $(HelixApiAccessToken)

- stage: Correctness
dependsOn: []
jobs:
Expand Down
6 changes: 6 additions & 0 deletions dotnet-tools.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,12 @@
"pwsh"
],
"rollForward": false
},
"microsoft.dotnet.helix.jobmonitor": {
"version": "11.0.0-beta.26407.8",
"commands": [
"dotnet-helix-job-monitor"
]
}
}
Comment thread
dibarbet marked this conversation as resolved.
Resolved
}
2 changes: 2 additions & 0 deletions eng/Version.Details.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ This file should be imported by eng/Versions.props
<PropertyGroup>
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetArcadeSdkPackageVersion>11.0.0-beta.26407.8</MicrosoftDotNetArcadeSdkPackageVersion>
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26407.8</MicrosoftDotNetHelixJobMonitorPackageVersion>
<MicrosoftDotNetHelixSdkPackageVersion>11.0.0-beta.26407.8</MicrosoftDotNetHelixSdkPackageVersion>
<MicrosoftDotNetXliffTasksPackageVersion>11.0.0-beta.26407.8</MicrosoftDotNetXliffTasksPackageVersion>
<!-- dotnet-dotnet dependencies -->
Expand Down Expand Up @@ -50,6 +51,7 @@ This file should be imported by eng/Versions.props
<PropertyGroup>
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetArcadeSdkVersion>$(MicrosoftDotNetArcadeSdkPackageVersion)</MicrosoftDotNetArcadeSdkVersion>
<MicrosoftDotNetHelixJobMonitorVersion>$(MicrosoftDotNetHelixJobMonitorPackageVersion)</MicrosoftDotNetHelixJobMonitorVersion>
<MicrosoftDotNetHelixSdkVersion>$(MicrosoftDotNetHelixSdkPackageVersion)</MicrosoftDotNetHelixSdkVersion>
<MicrosoftDotNetXliffTasksVersion>$(MicrosoftDotNetXliffTasksPackageVersion)</MicrosoftDotNetXliffTasksVersion>
<!-- dotnet-dotnet dependencies -->
Expand Down
4 changes: 4 additions & 0 deletions eng/Version.Details.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,5 +153,9 @@
<Uri>https://github.com/dotnet/roslyn</Uri>
<Sha>3d32d464e2949f054086fbb5346e4beea0c6df56</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Helix.JobMonitor" Version="11.0.0-beta.26407.8">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>41232a67a0c1a77d425a6bef59ded8a81a2fdf3c</Sha>
</Dependency>
</ToolsetDependencies>
</Dependencies>
Empty file modified eng/common/dotnet.sh
100644 → 100755
View file
Open in desktop
Empty file.
3 changes: 1 addition & 2 deletions eng/pipelines/test-unix-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,9 @@ jobs:
args: --ci --helix --configuration ${{ parameters.configuration }} --helixQueueName ${{ parameters.helixQueueName }} --helixApiAccessToken ${{ parameters.helixApiAccessToken }} ${{ parameters.testArguments }}
${{ else }}:
args: --ci --helix --configuration ${{ parameters.configuration }} --helixQueueName ${{ parameters.helixQueueName }} ${{ parameters.testArguments }}
displayName: Test
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)

displayName: Test
Comment thread
dibarbet marked this conversation as resolved.
Resolved
- template: publish-logs.yml
Comment thread
dibarbet marked this conversation as resolved.
Resolved
parameters:
configuration: ${{ parameters.configuration }}
Expand Down
1 change: 0 additions & 1 deletion eng/pipelines/test-windows-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@ jobs:
arguments: -ci -helix -configuration ${{ parameters.configuration }} -helixQueueName ${{ parameters.helixQueueName }} ${{ parameters.testArguments }} -collectDumps
Comment thread
dibarbet marked this conversation as resolved.
Resolved
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)

Comment thread
dibarbet marked this conversation as resolved.
Resolved
- template: publish-logs.yml
parameters:
configuration: ${{ parameters.configuration }}
Expand Down
Loading