Create a SuperPMI collection pipeline #42053

BruceForstall · 2020-09-10T02:12:59Z

Update from Kunal:

Added runtime superpmi pipeline that does the following:

Builds CoreClr/Libraries/Core_Root
Runs SuperPMI setup script that:
- Clones and builds jitutils to produce pmi.dll.
- Prepare correlation payload directory. This is the folder that contains Core_Root, pmi.dll, superpmi scripts that is needed to run SPMI collection. This is a common directory that is sent to all the helix machines that will do the collection.
- Scans the folder on which SPMI need to be done (in this case, it is just Core_Root, but in future will be TestArtifacts) and partition the .dlls/.exe in chunks of 50MB. These partition constitutes “payload” directory that is sent to individual helix machines to do the collection in parallel. For e.g. Core_Root folder is 150 MB (just managed .dll and .exe) and is divided in 3 partitions so we can do SPMI collection on 3 machines in parallel, each doing it on payload of 50 MB.
Sends payload/correlation directories to helix machine and performs the SPMI collection.
Once helix machines are done with collection, there will be multiple .mch files produced from each partition (libraries.0.mch, libraries.1.mch, libraries.2.mch in this case). We bring them back on azure machine and merge and de-duplicate all the .mch files into a single libraries.mch file.
Upload the merged libraries.mch file to azure blob storage.

I also had to update superpmi.py to make it Python3.5 compliant. I didn't remove the test run part but commented the portion of tests collection with TODO intentionally so when I have to enable it, I will know what to enable.

eng/common/templates/steps/superpmi-send-to-helix.yml

Start with a clone of jitstress.yml, and go from there.

…o-helix

Note that shutil.copy2 is very slow specially on Linux as compared to rsync. But I think we are ok to have slowness in return of simpler code.

This reverts commit a5312c0.

- Tests run times out on Windows - Some of the partition's collection failed for tests because reply was not clean - Here are the details when libraries/tests were run: https://dev.azure.com/dnceng/internal/_build/results?buildId=832814&view=results - Hence, disable tests run and just have libraries run for now. - I didn't remove the test run part but commented the portion of tests collection with TODO intentionally so when I have to enable it, I will know what to enable.

kunalspathak · 2020-09-28T18:58:49Z

@dotnet/jit-contrib

kunalspathak · 2020-09-28T19:41:33Z

@safern - Thanks a lot for helping me through this. If you get a chance, please take a look at the overall pipeline structure.

BruceForstall

Looks good.

(I can't approve this because it's technically "my" PR)

eng/pipelines/runtime.yml

eng/pipelines/coreclr/superpmi.yml

eng/pipelines/coreclr/templates/superpmi-job.yml

eng/pipelines/coreclr/templates/run-superpmi-job.yml

src/coreclr/scripts/superpmi-setup.py

kunalspathak · 2020-09-30T00:27:11Z

Looks good.

(I can't approve this because it's technically "my" PR)

Thanks Bruce for reviewing. I will incorporate the feedback and approve it! :)

kunalspathak

LGTM

safern · 2020-10-01T18:20:59Z

eng/common/superpmi/superpmi.proj

+    <FileSeparatorChar>\</FileSeparatorChar>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(AGENT_OS)' != 'Windows_NT'">
+    <FileSeparatorChar>/</FileSeparatorChar>


You don't need this. You can use MSBuild::NormalizeDirectory or MSBuild::NormalizePath apis. We have a bunch of usages and examples in the runtime repo.

Note that this APIs need to be used with full paths, it doesn't work with relative paths.

Maybe you do need it... because you're using it to calculate the commands you're going to run on helix.

safern · 2020-10-01T18:26:44Z

eng/pipelines/coreclr/templates/run-superpmi-job.yml

+      # Run superpmi collection in helix
+    - template: /eng/common/templates/steps/superpmi-send-to-helix.yml
+      parameters:
+        HelixSource: '$(HelixSourcePrefix)/$(Build.Repository.Name)/$(Build.SourceBranch)' # sources must start with pr/, official/, prodcon/, or agent/


I beliueve this is no longer needed, this is automatically set by the Helix SDK nowadays.

eng/pipelines/coreclr/templates/run-superpmi-job.yml

In dotnet#42053, I forgot to push a change to enable the superpmi pipeline. This change will trigger it every Sunday at 9:00 AM PST.

* enable superpmi collection pipeline In #42053, I forgot to push a change to enable the superpmi pipeline. This change will trigger it every Sunday at 9:00 AM PST. * fix space changes

Dotnet-GitSync-Bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Sep 10, 2020

kunalspathak force-pushed the AddSuperpmiPipeline branch from 76a0c54 to 278c532 Compare September 16, 2020 00:01

MattGal reviewed Sep 17, 2020

View reviewed changes

eng/common/templates/steps/superpmi-send-to-helix.yml Outdated Show resolved Hide resolved

kunalspathak force-pushed the AddSuperpmiPipeline branch from 1ad9f75 to 0b5407e Compare September 23, 2020 00:31

runfoapp bot mentioned this pull request Sep 23, 2020

OSX deprovision jaredpar/runfo#41

Closed

kunalspathak force-pushed the AddSuperpmiPipeline branch from 1f58683 to cdb6b20 Compare September 24, 2020 06:13

kunalspathak mentioned this pull request Sep 24, 2020

superpmi.py script improvements #42668

Merged

kunalspathak force-pushed the AddSuperpmiPipeline branch 2 times, most recently from bb642b7 to 383f0c5 Compare September 25, 2020 00:36

BruceForstall and others added 21 commits September 25, 2020 12:17

Create a SuperPMI collection pipeline

1e17728

Start with a clone of jitstress.yml, and go from there.

Only build Windows_NT_x64, disable test builds and runs

15c4dc2

TEMPORARY: disable normal PR triggered jobs

48ff794

some template files for superpmi

4b827f3

Little trial error# 1

2351e74

fix the superpmi-setup.ps1

668ee17

Disable CoreClrTestBuildHost, fix the project path in superpmi-send-t…

6a5ff58

…o-helix

Skip build, download, create core_root

37e8734

Just copy superpmi scripts, import pipeline-logging-functions

659a42d

add a dummy step

0a9fb92

add missing space before message

6cde75b

Replace message with script

c704c8e

Temporarily disable dependsOn

24a4127

disable another dependsOn

a255934

Pass superpmi.proj file

29745be

logging around send-to-helix

9f310a3

fix superpmi.proj to include ItemGroup

e147f7d

have placeholder.txt in WorkItem folder

566ac67

fix workitem folder creation, script name

1b329e1

Re-enable build coreclr, libraries, download, core_root

1df47fb

fix superpmi.py path

944d0e5

kunalspathak added 4 commits September 25, 2020 12:18

Make superpmi-setup script takes tests_directory

26bee59

Enable SPMI collection for P1

e981022

General cleanup

cca1003

fix merge_mch pattern for OS

f27d3cc

kunalspathak force-pushed the AddSuperpmiPipeline branch from 8f10370 to f27d3cc Compare September 25, 2020 21:27

kunalspathak added 7 commits September 25, 2020 22:43

Log the SPMI collection output and put it in upload_root folder

fd63f34

Replace robocopy/rsync with shutil.copy2

1c64bad

Note that shutil.copy2 is very slow specially on Linux as compared to rsync. But I think we are ok to have slowness in return of simpler code.

fix path of pmi.dll to be copied

2f4abfc

Include linux .so and file_exe files in copy_directories

f6a3e47

Try publishing just .mch file

a5312c0

Revert "Try publishing just .mch file"

ece1619

This reverts commit a5312c0.

kunalspathak marked this pull request as ready for review September 28, 2020 18:58

BruceForstall commented Sep 30, 2020

View reviewed changes

runfoapp bot mentioned this pull request Sep 30, 2020

Mono System.Text.Json.Tests on Windows timing out #42677

Open

Review feedback

04d3e40

kunalspathak approved these changes Oct 1, 2020

View reviewed changes

kunalspathak merged commit 6b33e41 into dotnet:master Oct 1, 2020

safern reviewed Oct 1, 2020

View reviewed changes

eng/pipelines/coreclr/templates/run-superpmi-job.yml Show resolved Hide resolved

kunalspathak added a commit to kunalspathak/runtime that referenced this pull request Oct 5, 2020

enable superpmi collection pipeline

511ac33

In dotnet#42053, I forgot to push a change to enable the superpmi pipeline. This change will trigger it every Sunday at 9:00 AM PST.

kunalspathak mentioned this pull request Oct 5, 2020

enable superpmi collection pipeline #43040

Merged

kunalspathak mentioned this pull request Oct 12, 2020

Add back superpmi pipeline files #43323

Merged

ghost locked as resolved and limited conversation to collaborators Dec 7, 2020

BruceForstall deleted the AddSuperpmiPipeline branch February 28, 2021 03:08

Create a SuperPMI collection pipeline #42053

Create a SuperPMI collection pipeline #42053

Uh oh!

Conversation

BruceForstall commented Sep 10, 2020 • edited by kunalspathak Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kunalspathak commented Sep 28, 2020

Uh oh!

kunalspathak commented Sep 28, 2020

Uh oh!

BruceForstall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kunalspathak commented Sep 30, 2020

Uh oh!

kunalspathak left a comment

Choose a reason for hiding this comment

Uh oh!

safern Oct 1, 2020

Choose a reason for hiding this comment

Uh oh!

safern Oct 1, 2020

Choose a reason for hiding this comment

Uh oh!

safern Oct 1, 2020

Choose a reason for hiding this comment

Uh oh!

safern Oct 1, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BruceForstall commented Sep 10, 2020 •

edited by kunalspathak

Loading