-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Create a SuperPMI collection pipeline #42053
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
kunalspathak
merged 138 commits into
dotnet:master
from
BruceForstall:AddSuperpmiPipeline
Oct 1, 2020
Merged
Changes from all commits
Commits
Show all changes
138 commits
Select commit
Hold shift + click to select a range
1e17728
Create a SuperPMI collection pipeline
BruceForstall 15c4dc2
Only build Windows_NT_x64, disable test builds and runs
kunalspathak 48ff794
TEMPORARY: disable normal PR triggered jobs
BruceForstall 4b827f3
some template files for superpmi
kunalspathak 2351e74
Little trial error# 1
kunalspathak 668ee17
fix the superpmi-setup.ps1
kunalspathak 6a5ff58
Disable CoreClrTestBuildHost, fix the project path in superpmi-send-t…
kunalspathak 37e8734
Skip build, download, create core_root
kunalspathak 659a42d
Just copy superpmi scripts, import pipeline-logging-functions
kunalspathak 0a9fb92
add a dummy step
kunalspathak 6cde75b
add missing space before message
kunalspathak c704c8e
Replace message with script
kunalspathak 24a4127
Temporarily disable dependsOn
kunalspathak a255934
disable another dependsOn
kunalspathak 29745be
Pass superpmi.proj file
kunalspathak 9f310a3
logging around send-to-helix
kunalspathak e147f7d
fix superpmi.proj to include ItemGroup
kunalspathak 566ac67
have placeholder.txt in WorkItem folder
kunalspathak 1b329e1
fix workitem folder creation, script name
kunalspathak 1df47fb
Re-enable build coreclr, libraries, download, core_root
kunalspathak 944d0e5
fix superpmi.py path
kunalspathak 893a288
clone jitutils, build and copy pmi.dll
kunalspathak c0be11a
Pass Core_Root as payload
kunalspathak c0efc11
Use hardcoded checked
kunalspathak 4a34890
Use .dotnet from runtime repo
kunalspathak 63b4847
Fix the pmi.dll location and add -output_mch_path
kunalspathak 96e2f50
fix the output path
kunalspathak e824683
Try publishing SourcesDirectory
kunalspathak 3a60c31
Upload to HELIX_WORKITEM_UPLOAD_ROOT
kunalspathak ae09edd
Include test in collection and partition core_root and test
kunalspathak ce5069b
Enable CoreClrTestBuildHost
kunalspathak b66ded2
Add dependsOn P1 test
kunalspathak a695623
Add DownloadFilesFromResults
kunalspathak 23b875f
Properly copy the Tests
kunalspathak d5742dd
Workaround for https://github.com/dotnet/arcade/pull/6179
kunalspathak 0df199e
Disable test run, SuperPMI just system.web.dll
kunalspathak 1b602b1
Fix the mch path
kunalspathak 0732905
Comment partition for test
kunalspathak e17335d
Remove commented code
kunalspathak a71b4fc
Increase timeout to 12 hours
kunalspathak dfde053
fast pass
kunalspathak b4ab9dc
Update the machine pool
kunalspathak 178fe35
Add a dummy step
kunalspathak d6f8da8
Use SuperPMI_Min.zip that contains all the required files
kunalspathak da8a370
Another way of Extracting cab
kunalspathak 42e0910
download in powershell, extract files separately
kunalspathak f9bf039
fix the zip path
kunalspathak 74ab28b
Another logic of download zip
kunalspathak 3c08321
fix path
kunalspathak 10bf908
fix newdir path
kunalspathak 7af08bc
more fixes
kunalspathak f6c74b4
fix extract step
kunalspathak 42aa6de
add test python script and modify creator
kunalspathak ade6523
Map secret to env
kunalspathak 3d7a26a
decrease timeout, fix superpmi path in proj, print unzipped contents
kunalspathak 8e35c22
Add a step to download and dir artifacts
kunalspathak f09b5fe
Attempt to call super_pmi upload
kunalspathak e999395
list all contents
kunalspathak 5bbec4b
Also dir SourcesDir\artifacts
kunalspathak 05ee4bc
Use DownloadPipelineArtifact
kunalspathak 65c4277
Use the right key
kunalspathak 1b4b47e
Upload from helixresults
kunalspathak 71e6960
cleanup run-superpmi-job.yml
kunalspathak 265dd0d
Re-enable superpmi for CORE_ROOT
kunalspathak d776d42
fix the dependsOn
kunalspathak a97404c
Replace checked with $(BuildConfig)
kunalspathak 66ef8e1
Replace checked with $(BuildConfig)
kunalspathak a73331f
change helix queue for windows and add for arm64
kunalspathak 126c56c
Use FileSeparateChar in superpmi.proj
kunalspathak d90fb99
Add Linux x64
kunalspathak b552cb6
Fix the helix queue, property erroro
kunalspathak 1c777ef
logging in shell
kunalspathak ebc84ff
Fix PayloadDir and CorrelationPayloadDir
kunalspathak a5164dc
Make timeout to 5 hours
kunalspathak 7d2a5b7
Print workitem contents
kunalspathak 882ffd4
fix WorkItemDirectory
kunalspathak 42a4d65
Just copy pmi.dll
kunalspathak ef0f597
fix the pmi_assemblies path
kunalspathak 3a9ebdf
disable reporter
kunalspathak 37b8be4
pop put JitUtilsDirectory before deleting it
kunalspathak f21c60d
chmod for superpmi-setup.sh
kunalspathak 712aeed
just copy pmi.dll for unix
kunalspathak 5520771
fix the parameter to unix
kunalspathak fa7a00b
Add logging to understand -pmi_assemblies issue
kunalspathak ef8f32e
Add "binaries" folder where payload is present
kunalspathak bb18d27
remove precommand
kunalspathak fc0b1f1
Pass Core_Root as CorrelationPayload
kunalspathak dc882a4
Add back the workaround for https://github.com/dotnet/arcade/pull/6179
kunalspathak 0f1a184
fix checked for Linux
kunalspathak b21a032
mkdir for linux
kunalspathak 6cd3c45
fix path in HelixResultsDestinationDir
kunalspathak 676edb4
Enable Windows x86/arm64
kunalspathak 2032e44
fix the python path
kunalspathak 12ecebc
First version of superpmi-setup.py
kunalspathak 7833e9f
Attempt to use superpmi-setup.py
kunalspathak ebaf62a
Fix spelling of LibrariesArtifacts, set python path
kunalspathak 7151b9b
TestArtifacts commented
kunalspathak c9a0d4a
superpmi-setup script for linux
kunalspathak d54858c
Enable Linux_x64
kunalspathak 62bd023
Added pydocs
kunalspathak 774ffda
Add option for exclude_directories
kunalspathak 3370619
Moved all the logic to superpmi-setup.py
kunalspathak 8e40713
Replace superpmi-setup.ps1 and superpmi-setup.sh with superpmi-setup.py
kunalspathak 3b502f1
Fix up in superpmi-setup.py
kunalspathak a7277eb
fix display name
kunalspathak e4aa69b
add missing spacE
kunalspathak 65c0e57
Use differnet conditional
kunalspathak 03fa06d
ignore PermissionError while cleaning up temp folder
kunalspathak b92b571
fix workitem copy for linux
kunalspathak 5b7f263
fix copy for unix
kunalspathak f148f2f
Publish logs
kunalspathak b7a7128
Print LibrariesArtifacts
kunalspathak 14aa98c
remove space in front of publish logs desc
kunalspathak f2a3d7d
Add missing quotes for LibrariesArtifacts
kunalspathak 4c6f804
Download x64 coreclr to make upload work
kunalspathak 9728ff3
Enable Windows x86/arm64, Linux arm/arm64
kunalspathak fd8db24
Delete superpmi-setup.ps1, superpmi-setup.sh
kunalspathak 00e4f1e
Fix space around LibrariesArtifacts
kunalspathak 0e565c2
Download real coreclr x64
kunalspathak 4415d08
ignore native dlls
kunalspathak 33a8764
Make superpmi.py Python3.6 friendly
kunalspathak e71deee
Minor cleanup
kunalspathak 1cb260c
Cleanup superpmi-send-to-helix to use helix-inner-step
kunalspathak fe7e8fc
Remove redundant steps
kunalspathak 38aff91
Merge mch files
kunalspathak b9bca85
Pass osGroup
kunalspathak 26bee59
Make superpmi-setup script takes tests_directory
kunalspathak e981022
Enable SPMI collection for P1
kunalspathak cca1003
General cleanup
kunalspathak f27d3cc
fix merge_mch pattern for OS
kunalspathak fd63f34
Log the SPMI collection output and put it in upload_root folder
kunalspathak 1c64bad
Replace robocopy/rsync with shutil.copy2
kunalspathak 2f4abfc
fix path of pmi.dll to be copied
kunalspathak f6a3e47
Include linux .so and file_exe files in copy_directories
kunalspathak a5312c0
Try publishing just .mch file
kunalspathak ece1619
Revert "Try publishing just .mch file"
kunalspathak 22a2ead
Disable tests SPMI collection on 42053
kunalspathak 04d3e40
Review feedback
kunalspathak File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| <Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test"> | ||
|
|
||
| <PropertyGroup Condition="'$(AGENT_OS)' == 'Windows_NT'"> | ||
| <FileSeparatorChar>\</FileSeparatorChar> | ||
| </PropertyGroup> | ||
| <PropertyGroup Condition="'$(AGENT_OS)' != 'Windows_NT'"> | ||
| <FileSeparatorChar>/</FileSeparatorChar> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- The directory structure of pmiAssembliesPayload is | ||
| source\workitem\pmiAssembliesDirectory\<machine_specific_load_0>\binaries | ||
| source\workitem\pmiAssembliesDirectory\<machine_specific_load_1>\binaries | ||
| ... | ||
| source\workitem\pmiAssembliesDirectory\<machine_specific_load_N>\binaries | ||
|
|
||
| PmiAssembliesPayload - Path that will be sent to helix machine to run collection on | ||
| PmiAssembliesDirectory - Path on helix machine itself where superpmi.py will discover the sent assemblies. | ||
| --> | ||
| <PropertyGroup Condition="'$(AGENT_OS)' == 'Windows_NT'"> | ||
| <Python>%HELIX_PYTHONPATH%</Python> | ||
| <PmiAssembliesPayload>$(WorkItemDirectory)\pmiAssembliesDirectory</PmiAssembliesPayload> | ||
| <PmiAssembliesDirectory>%HELIX_WORKITEM_PAYLOAD%\binaries</PmiAssembliesDirectory> | ||
| <SuperPMIDirectory>%HELIX_CORRELATION_PAYLOAD%\superpmi</SuperPMIDirectory> | ||
| <OutputMchPath>%HELIX_WORKITEM_UPLOAD_ROOT%</OutputMchPath> | ||
| <!-- Workaround until https://github.com/dotnet/arcade/pull/6179 is not available --> | ||
| <HelixResultsDestinationDir>$(BUILD_SOURCESDIRECTORY)\artifacts\helixresults</HelixResultsDestinationDir> | ||
| <WorkItemCommand>$(SuperPMIDirectory)\superpmi.py collect --pmi -pmi_location $(SuperPMIDirectory)\pmi.dll </WorkItemCommand> | ||
| </PropertyGroup> | ||
| <PropertyGroup Condition="'$(AGENT_OS)' != 'Windows_NT'"> | ||
| <Python>$HELIX_PYTHONPATH</Python> | ||
| <PmiAssembliesPayload>$(WorkItemDirectory)/pmiAssembliesDirectory</PmiAssembliesPayload> | ||
| <PmiAssembliesDirectory>$HELIX_WORKITEM_PAYLOAD/binaries</PmiAssembliesDirectory> | ||
| <SuperPMIDirectory>$HELIX_CORRELATION_PAYLOAD/superpmi</SuperPMIDirectory> | ||
| <OutputMchPath>$HELIX_WORKITEM_UPLOAD_ROOT</OutputMchPath> | ||
| <!-- Workaround until https://github.com/dotnet/arcade/pull/6179 is not available --> | ||
| <HelixResultsDestinationDir>$(BUILD_SOURCESDIRECTORY)/artifacts/helixresults</HelixResultsDestinationDir> | ||
| <WorkItemCommand>$(SuperPMIDirectory)/superpmi.py collect --pmi -pmi_location $(SuperPMIDirectory)/pmi.dll </WorkItemCommand> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup Condition="'$(WorkItemCommand)' != ''"> | ||
| <WorkItemCommand>$(Python) $(WorkItemCommand) -pmi_assemblies $(PmiAssembliesDirectory) -arch $(Architecture) -build_type $(BuildConfig) -core_root $(SuperPMIDirectory)</WorkItemCommand> | ||
| </PropertyGroup> | ||
|
|
||
| <PropertyGroup> | ||
| <EnableAzurePipelinesReporter>false</EnableAzurePipelinesReporter> | ||
| <EnableXUnitReporter>false</EnableXUnitReporter> | ||
| <WorkItemTimeout>5:00</WorkItemTimeout> | ||
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)"> | ||
| <PayloadDirectory>%(Identity)</PayloadDirectory> | ||
| </HelixCorrelationPayload> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <!-- libraries payload --> | ||
| <LibraryPartitions Include="$([System.IO.Directory]::GetDirectories($(LibrariesArtifacts)))"/> | ||
| <Partition Include="@(LibraryPartitions -> '%(Filename)')" PmiAssemblies="Core_Root$(FileSeparatorChar)%(Filename)" OutputMchName="libraries.pmi" PartitionId="%(Filename)" /> | ||
|
|
||
| <!-- tests payload --> | ||
| <!-- TODO: Disable SPMI for P1 tests --> | ||
| <!-- <TestPartitions Include="$([System.IO.Directory]::GetDirectories($(TestsArtifacts)))"/> | ||
| <Partition Include="@(TestPartitions -> '%(Filename)')" PmiAssemblies="Tests$(FileSeparatorChar)%(Filename)" OutputMchName="tests.pmi" PartitionId="%(Filename)" /> --> | ||
| </ItemGroup> | ||
|
|
||
| <ItemGroup> | ||
| <HelixWorkItem Include="@(Partition)"> | ||
| <OutputFileName>%(HelixWorkItem.OutputMchName).$(MchFileTag).%(HelixWorkItem.PartitionId)</OutputFileName> | ||
| <PayloadDirectory>$(PmiAssembliesPayload)$(FileSeparatorChar)%(HelixWorkItem.PmiAssemblies)</PayloadDirectory> | ||
| <Command>$(WorkItemCommand) -output_mch_path $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).mch -log_file $(OutputMchPath)$(FileSeparatorChar)%(OutputFileName).log</Command> | ||
| <Timeout>$(WorkItemTimeout)</Timeout> | ||
| <DownloadFilesFromResults>%(OutputFileName).mch;%(OutputFileName).mch.mct;%(OutputFileName).log</DownloadFilesFromResults> | ||
| </HelixWorkItem> | ||
| </ItemGroup> | ||
|
|
||
| <!-- | ||
| This is useful for local testing to print the produced helix items | ||
| To use this when you are changing how items are produced, uncomment the target | ||
| and replace the Project item at the top of the file with this: | ||
| <Project DefaultTargets="printItems"> | ||
|
|
||
| Once you've done that you can run this to see the results: | ||
| dotnet msbuild .\scenarios.proj /v:n | ||
| --> | ||
| <!-- | ||
| <PropertyGroup> | ||
| <HelixTargetQueues>Some_Queue</HelixTargetQueues> | ||
| <LibrariesArtifacts>E:\temp\power\test1</LibrariesArtifacts> | ||
| <TestsArtifacts>E:\temp\power\test1</TestsArtifacts> | ||
| </PropertyGroup> | ||
| <Target Name="printItems"> | ||
| <Message Text="@(HelixWorkItem -> 'name: %(HelixWorkItem.Identity) | ||
| dir: %(HelixWorkItem.PayloadDirectory) | ||
| pre: %(HelixWorkItem.PreCommands) | ||
| command: %(HelixWorkItem.Command) | ||
| post: %(HelixWorkItem.PostCommands) | ||
| timeout: %(HelixWorkItem.Timeout) '"/> | ||
| </Target> | ||
| --> | ||
|
|
||
| </Project> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # Please remember to update the documentation if you make changes to these parameters! | ||
| parameters: | ||
| HelixSource: 'pr/default' # required -- sources must start with pr/, official/, prodcon/, or agent/ | ||
| HelixType: 'tests/default/' # required -- Helix telemetry which identifies what type of data this is; should include "test" for clarity and must end in '/' | ||
| HelixBuild: $(Build.BuildNumber) # required -- the build number Helix will use to identify this -- automatically set to the AzDO build number | ||
| HelixTargetQueues: '' # required -- semicolon delimited list of Helix queues to test on; see https://helix.dot.net/ for a list of queues | ||
| HelixAccessToken: '' # required -- access token to make Helix API requests; should be provided by the appropriate variable group | ||
| HelixPreCommands: '' # optional -- commands to run before Helix work item execution | ||
| HelixPostCommands: '' # optional -- commands to run after Helix work item execution | ||
| WorkItemDirectory: '' # optional -- a payload directory to zip up and send to Helix; requires WorkItemCommand; incompatible with XUnitProjects | ||
| CorrelationPayloadDirectory: '' # optional -- a directory to zip up and send to Helix as a correlation payload | ||
| IncludeDotNetCli: false # optional -- true will download a version of the .NET CLI onto the Helix machine as a correlation payload; requires DotNetCliPackageType and DotNetCliVersion | ||
| DotNetCliPackageType: '' # optional -- either 'sdk' or 'runtime'; determines whether the sdk or runtime will be sent to Helix; see https://raw.githubusercontent.com/dotnet/core/master/release-notes/releases.json | ||
| DotNetCliVersion: '' # optional -- version of the CLI to send to Helix; based on this: https://raw.githubusercontent.com/dotnet/core/master/release-notes/releases.json | ||
| EnableXUnitReporter: false # optional -- true enables XUnit result reporting to Mission Control | ||
| WaitForWorkItemCompletion: true # optional -- true will make the task wait until work items have been completed and fail the build if work items fail. False is "fire and forget." | ||
| Creator: '' # optional -- if the build is external, use this to specify who is sending the job | ||
| DisplayNamePrefix: 'Send job to Helix' # optional -- rename the beginning of the displayName of the steps in AzDO | ||
| condition: succeeded() # optional -- condition for step to execute; defaults to succeeded() | ||
| continueOnError: false # optional -- determines whether to continue the build if the step errors; defaults to false | ||
| BuildConfig: 'checked' # optional -- Mostly, superpmi will be run on checked builds | ||
| LibrariesArtifacts: '' | ||
| TestsArtifacts: '' | ||
|
|
||
| steps: | ||
| - template: /eng/pipelines/common/templates/runtimes/send-to-helix-inner-step.yml | ||
| parameters: | ||
| osGroup: ${{ parameters.osGroup }} | ||
| sendParams: $(Build.SourcesDirectory)/eng/common/superpmi/superpmi.proj /restore /t:Test /bl:$(Build.SourcesDirectory)/artifacts/log/$(BuildConfig)/SendToHelix.binlog | ||
| displayName: ${{ parameters.DisplayNamePrefix }} | ||
| condition: ${{ parameters.condition }} | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
| environment: | ||
| MchFileTag: $(MchFileTag) | ||
| BuildConfig: ${{ parameters.BuildConfig }} | ||
| LibrariesArtifacts: ${{ parameters.LibrariesArtifacts }} | ||
| TestsArtifacts: ${{ parameters.TestsArtifacts }} | ||
| HelixSource: ${{ parameters.HelixSource }} | ||
| HelixType: ${{ parameters.HelixType }} | ||
| HelixBuild: ${{ parameters.HelixBuild }} | ||
| HelixTargetQueues: ${{ parameters.HelixTargetQueues }} | ||
| HelixAccessToken: ${{ parameters.HelixAccessToken }} | ||
| HelixPreCommands: ${{ parameters.HelixPreCommands }} | ||
| HelixPostCommands: ${{ parameters.HelixPostCommands }} | ||
| WorkItemDirectory: ${{ parameters.WorkItemDirectory }} | ||
| CorrelationPayloadDirectory: ${{ parameters.CorrelationPayloadDirectory }} | ||
| IncludeDotNetCli: ${{ parameters.IncludeDotNetCli }} | ||
| DotNetCliPackageType: ${{ parameters.DotNetCliPackageType }} | ||
| DotNetCliVersion: ${{ parameters.DotNetCliVersion }} | ||
| EnableXUnitReporter: ${{ parameters.EnableXUnitReporter }} | ||
| WaitForWorkItemCompletion: ${{ parameters.WaitForWorkItemCompletion }} | ||
| Creator: ${{ parameters.Creator }} | ||
| SYSTEM_ACCESSTOKEN: $(System.AccessToken) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| trigger: none | ||
|
|
||
| pr: none | ||
|
|
||
| # schedules: | ||
kunalspathak marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # - cron: "0 4 * * *" | ||
| # displayName: Daily at 8:00 PM (UTC-8:00) | ||
| # branches: | ||
| # include: | ||
| # - master | ||
| # always: true | ||
|
|
||
| jobs: | ||
| # | ||
| # Checkout repository | ||
| # | ||
| - template: /eng/pipelines/common/checkout-job.yml | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/common/build-coreclr-and-libraries-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| # Linux tests are built on the OSX machines. | ||
| # - OSX_x64 | ||
| - Linux_arm | ||
| - Linux_arm64 | ||
| - Linux_x64 | ||
| - Windows_NT_x64 | ||
| - Windows_NT_x86 | ||
| - Windows_NT_arm64 | ||
| - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64 | ||
| jobParameters: | ||
| testGroup: outerloop | ||
|
|
||
| # TODO: Disable SPMI for P1 tests | ||
kunalspathak marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # - template: /eng/pipelines/common/platform-matrix.yml | ||
| # parameters: | ||
| # jobTemplate: /eng/pipelines/common/templates/runtimes/build-test-job.yml | ||
| # buildConfig: checked | ||
| # platforms: | ||
| # - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64 | ||
| # jobParameters: | ||
| # testGroup: outerloop | ||
| # liveLibrariesBuildConfig: Release | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/coreclr/templates/superpmi-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| # Linux tests are built on the OSX machines. | ||
| # - OSX_x64 | ||
| - Linux_arm | ||
| - Linux_arm64 | ||
| - Linux_x64 | ||
| - Windows_NT_x64 | ||
| - Windows_NT_x86 | ||
| - Windows_NT_arm64 | ||
| - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64 | ||
| helixQueueGroup: ci | ||
| helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml | ||
| jobParameters: | ||
| testGroup: outerloop | ||
| liveLibrariesBuildConfig: Release | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| parameters: | ||
| steps: [] # optional -- any additional steps that need to happen before pulling down the jitutils repo and sending the jitutils to helix (ie building your repo) | ||
| variables: [] # optional -- list of additional variables to send to the template | ||
| jobName: '' # required -- job name | ||
| displayName: '' # optional -- display name for the job. Will use jobName if not passed | ||
| pool: '' # required -- name of the Build pool | ||
| container: '' # required -- name of the container | ||
| buildConfig: '' # required -- build configuration | ||
| archType: '' # required -- targeting CPU architecture | ||
| osGroup: '' # required -- operating system for the job | ||
| osSubgroup: '' # optional -- operating system subgroup | ||
| extraSetupParameters: '' # optional -- extra arguments to pass to the setup script | ||
| frameworks: ['netcoreapp3.0'] # optional -- list of frameworks to run against | ||
| continueOnError: 'false' # optional -- determines whether to continue the build if the step errors | ||
| dependsOn: '' # optional -- dependencies of the job | ||
| timeoutInMinutes: 320 # optional -- timeout for the job | ||
| enableTelemetry: false # optional -- enable for telemetry | ||
| liveLibrariesBuildConfig: '' # optional -- live-live libraries configuration to use for the run | ||
| runtimeType: 'coreclr' # optional -- Sets the runtime as coreclr or mono | ||
| codeGenType: 'JIT' # optional -- Decides on the codegen technology if running on mono | ||
| projectFile: 'superpmi.proj' # optional -- project file to build helix workitems | ||
| runKind: '' # required -- test category | ||
|
|
||
| jobs: | ||
| - template: xplat-pipeline-job.yml | ||
| parameters: | ||
| dependsOn: ${{ parameters.dependsOn }} | ||
| buildConfig: ${{ parameters.buildConfig }} | ||
| archType: ${{ parameters.archType }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| osSubgroup: ${{ parameters.osSubgroup }} | ||
| liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }} | ||
| enableTelemetry: ${{ parameters.enableTelemetry }} | ||
| enablePublishBuildArtifacts: true | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
|
|
||
| ${{ if ne(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.displayName }}' | ||
| ${{ if eq(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.jobName }}' | ||
|
|
||
| timeoutInMinutes: ${{ parameters.timeoutInMinutes }} | ||
|
|
||
| variables: | ||
|
|
||
| - ${{ each variable in parameters.variables }}: | ||
| - ${{ if ne(variable.name, '') }}: | ||
| - name: ${{ variable.name }} | ||
| value: ${{ variable.value }} | ||
| - ${{ if ne(variable.group, '') }}: | ||
| - group: ${{ variable.group }} | ||
|
|
||
| - HelixApiAccessToken: '' | ||
| - HelixPreCommand: '' | ||
| - MchFileTag: '${{ parameters.osGroup }}.${{ parameters.archType }}.${{ parameters.buildConfig }}' | ||
|
|
||
| - ${{ if eq(parameters.osGroup, 'Windows_NT') }}: | ||
| - name: PythonScript | ||
| value: 'py -3' | ||
| - name: PipScript | ||
| value: 'py -3 -m pip' | ||
| - name: Core_Root_Dir | ||
| value: '$(Build.SourcesDirectory)\artifacts\tests\coreclr\${{ parameters.osGroup }}.${{ parameters.archType }}.${{ parameters.buildConfig }}\Tests\Core_Root' | ||
| - name: MchFilesLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\helixresults\' | ||
| - ${{ if ne(parameters.osGroup, 'Windows_NT') }}: | ||
| - name: PythonScript | ||
| value: 'python3' | ||
| - name: PipScript | ||
| value: 'pip3' | ||
| - name: Core_Root_Dir | ||
| value: '$(Build.SourcesDirectory)/artifacts/tests/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.$(buildConfigUpper)/Tests/Core_Root' | ||
| - name: MchFilesLocation | ||
| value: '$(Build.SourcesDirectory)/artifacts/helixresults/' | ||
| workspace: | ||
| clean: all | ||
| pool: | ||
| ${{ parameters.pool }} | ||
| container: ${{ parameters.container }} | ||
| strategy: | ||
| matrix: | ||
| ${{ each framework in parameters.frameworks }}: | ||
| ${{ framework }}: | ||
| _Framework: ${{ framework }} | ||
| steps: | ||
| - ${{ parameters.steps }} | ||
|
|
||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi-setup.py -source_directory $(Build.SourcesDirectory) -core_root_directory $(Core_Root_Dir) -arch $(archType) -mch_file_tag $(MchFileTag) -libraries_directory $(Core_Root_Dir) -tests_directory $(managedTestArtifactRootFolderPath) -max_size 50 # size in MB | ||
| displayName: ${{ format('SuperPMI setup ({0})', parameters.osGroup) }} | ||
|
|
||
| # Run superpmi collection in helix | ||
| - template: /eng/common/templates/steps/superpmi-send-to-helix.yml | ||
| parameters: | ||
| HelixSource: '$(HelixSourcePrefix)/$(Build.Repository.Name)/$(Build.SourceBranch)' # sources must start with pr/, official/, prodcon/, or agent/ | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I beliueve this is no longer needed, this is automatically set by the Helix SDK nowadays. |
||
| HelixType: 'test/superpmi/$(Kind)/$(_Framework)/$(Architecture)' | ||
| HelixAccessToken: $(HelixApiAccessToken) | ||
| HelixTargetQueues: $(Queue) | ||
| HelixPreCommands: $(HelixPreCommand) | ||
| Creator: $(Creator) | ||
| WorkItemTimeout: 4:00 # 4 hours | ||
| WorkItemDirectory: '$(WorkItemDirectory)' | ||
| CorrelationPayloadDirectory: '$(CorrelationPayloadDirectory)' | ||
| ProjectFile: ${{ parameters.projectFile }} | ||
| BuildConfig: ${{ parameters.buildConfig }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| LibrariesArtifacts: '$(LibrariesArtifacts)' | ||
| TestsArtifacts: '$(TestsArtifacts)' | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
| displayName: Publish SuperPMI collection | ||
| inputs: | ||
| targetPath: $(Build.SourcesDirectory)/artifacts/helixresults | ||
| artifactName: 'SuperPMI_Result_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)_${{ parameters.runtimeType }}_${{ parameters.codeGenType }}' | ||
| continueOnError: true | ||
| condition: always() | ||
|
|
||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi.py merge-mch -pattern $(MchFilesLocation)libraries.pmi*.mch -output_mch_path $(MchFilesLocation)libraries.pmi.$(MchFileTag).mch | ||
| displayName: Merge libraries SuperPMI collections | ||
|
|
||
| # TODO: Disable SPMI for P1 tests | ||
| # - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi.py merge-mch -pattern $(MchFilesLocation)tests.pmi*.mch -output_mch_path $(MchFilesLocation)tests.pmi.$(MchFileTag).mch | ||
| # displayName: Merge tests SuperPMI collections | ||
|
|
||
| # For now, we won't upload merged collection as an artifact. | ||
|
|
||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi.py upload -arch $(archType) -build_type $(buildConfig) -mch_files $(MchFilesLocation)libraries.pmi.$(MchFileTag).mch -core_root $(Build.SourcesDirectory)/artifacts/bin/coreclr/$(osGroup).x64.$(buildConfigUpper) | ||
kunalspathak marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| displayName: Upload SuperPMI libraries collection to Azure Storage | ||
| env: | ||
| CLRJIT_AZ_KEY: $(clrjit_key1) # secret key stored as variable in pipeline | ||
kunalspathak marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| # TODO: Disable SPMI for P1 tests | ||
| # - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/superpmi.py upload -arch $(archType) -build_type $(buildConfig) $(MchFilesLocation)tests.pmi.$(MchFileTag).mch -core_root $(Build.SourcesDirectory)/artifacts/bin/coreclr/$(osGroup).x64.$(buildConfigUpper) | ||
| # displayName: Upload SuperPMI tests collection to Azure Storage | ||
| # env: | ||
| # CLRJIT_AZ_KEY: $(clrjit_key1) # secret key stored as variable in pipeline | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
| displayName: Publish Logs | ||
| inputs: | ||
| targetPath: $(Build.SourcesDirectory)/artifacts/log | ||
| artifactName: 'SuperPMI_Logs_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)_${{ parameters.runtimeType }}_${{ parameters.codeGenType }}_${{ parameters.runKind }}' | ||
| continueOnError: true | ||
| condition: always() | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You don't need this. You can use
MSBuild::NormalizeDirectoryorMSBuild::NormalizePathapis. We have a bunch of usages and examples in the runtime repo.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that this APIs need to be used with full paths, it doesn't work with relative paths.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you do need it... because you're using it to calculate the commands you're going to run on helix.