Pipeline to run Exploratory fuzzing tool to catch codegen issues #59489
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| # This job definition automates the exploratory tool. | ||
|
kunalspathak marked this conversation as resolved.
|
||
|
|
||
| trigger: none | ||
|
|
||
| schedules: | ||
| - cron: "0 6 * * *" | ||
| displayName: Daily at 11:30 PM (UTC-8:00) | ||
|
Contributor
There was a problem hiding this comment. It seems like daily is too often, and weekly might make more sense. But I'm ok with this; we can change our mind about scheduling later.
Contributor
Author
There was a problem hiding this comment. Ok, I will schedule it for weekly for now. |
||
| branches: | ||
| include: | ||
| - main | ||
|
kunalspathak marked this conversation as resolved.
|
||
| always: true | ||
|
|
||
| jobs: | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/common/build-coreclr-and-libraries-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| # Linux tests are built on the OSX machines. | ||
| # - OSX_x64 | ||
| - Linux_arm | ||
| - Linux_arm64 | ||
| - Linux_x64 | ||
| - windows_x64 | ||
| - windows_x86 | ||
| - windows_arm64 | ||
| - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64 | ||
| jobParameters: | ||
| testGroup: outerloop | ||
|
|
||
| - template: /eng/pipelines/common/platform-matrix.yml | ||
| parameters: | ||
| jobTemplate: /eng/pipelines/coreclr/templates/jit-exploratory-job.yml | ||
| buildConfig: checked | ||
| platforms: | ||
| # Linux tests are built on the OSX machines. | ||
| # - OSX_x64 | ||
| - Linux_arm | ||
| - Linux_arm64 | ||
| - Linux_x64 | ||
| - windows_x64 | ||
| - windows_x86 | ||
| - windows_arm64 | ||
| helixQueueGroup: ci | ||
| helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml | ||
| jobParameters: | ||
| testGroup: outerloop | ||
| liveLibrariesBuildConfig: Release | ||
| collectionType: pmi | ||
| collectionName: libraries | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| parameters: | ||
| buildConfig: '' | ||
| archType: '' | ||
| osGroup: '' | ||
| osSubgroup: '' | ||
| container: '' | ||
| runtimeVariant: '' | ||
| testGroup: '' | ||
| framework: net6.0 # Specify the appropriate framework when running release branches (ie netcoreapp3.0 for release/3.0) | ||
| liveLibrariesBuildConfig: '' | ||
| variables: {} | ||
| runtimeType: 'coreclr' | ||
| pool: '' | ||
| codeGenType: 'JIT' | ||
| projetFile: '' | ||
| runKind: '' | ||
| runJobTemplate: '/eng/pipelines/coreclr/templates/jit-run-exploratory-job.yml' | ||
| additionalSetupParameters: '' | ||
| collectionType: '' | ||
| collectionName: '' | ||
|
|
||
| ### Exploratory job | ||
|
|
||
| ### Each exploratory job depends on a corresponding build job with the same | ||
| ### buildConfig and archType. | ||
|
|
||
| jobs: | ||
| - template: ${{ parameters.runJobTemplate }} | ||
| parameters: | ||
| # Compute job name from template parameters | ||
| jobName: ${{ format('exploratory_{0}{1}_{2}_{3}_{4}_{5}_{6}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind) }} | ||
| displayName: ${{ format('Exploratory {0}{1} {2} {3} {4} {5} {6}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind) }} | ||
| pool: ${{ parameters.pool }} | ||
| buildConfig: ${{ parameters.buildConfig }} | ||
| archType: ${{ parameters.archType }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| osSubgroup: ${{ parameters.osSubgroup }} | ||
| runtimeVariant: ${{ parameters.runtimeVariant }} | ||
| liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }} | ||
| runtimeType: ${{ parameters.runtimeType }} | ||
| codeGenType: ${{ parameters.codeGenType }} | ||
| runKind: ${{ parameters.runKind }} | ||
| testGroup: ${{ parameters.testGroup }} | ||
| collectionType: ${{ parameters.collectionType }} | ||
| collectionName: ${{ parameters.collectionName }} | ||
| additionalSetupParameters: ${{ parameters.additionalSetupParameters }} | ||
| # Test job depends on the corresponding build job | ||
| dependsOn: | ||
| - ${{ format('coreclr_{0}_product_build_{1}{2}_{3}_{4}', parameters.runtimeVariant, parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }} | ||
| - ${{ format('libraries_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }} | ||
|
|
||
| variables: ${{ parameters.variables }} | ||
|
|
||
| frameworks: | ||
| - ${{ parameters.framework }} | ||
| steps: | ||
| # Extra steps that will be passed to the exploratory template and run before sending the job to helix (all of which is done in the template) | ||
|
|
||
| # Optionally download live-built libraries | ||
| - template: /eng/pipelines/common/download-artifact-step.yml | ||
| parameters: | ||
| unpackFolder: $(librariesDownloadDir) | ||
| cleanUnpackFolder: false | ||
| artifactFileName: '$(librariesBuildArtifactName)$(archiveExtension)' | ||
| artifactName: '$(librariesBuildArtifactName)' | ||
| displayName: 'live-built libraries' | ||
|
|
||
| # Download coreclr | ||
| - template: /eng/pipelines/common/download-artifact-step.yml | ||
| parameters: | ||
| unpackFolder: $(buildProductRootFolderPath) | ||
| artifactFileName: '$(buildProductArtifactName)$(archiveExtension)' | ||
| artifactName: '$(buildProductArtifactName)' | ||
| displayName: 'Coreclr product build' | ||
|
|
||
| # Create Core_Root | ||
| - script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) $(buildConfig) $(archType) generatelayoutonly $(librariesOverrideArg) | ||
| displayName: Create Core_Root | ||
| condition: succeeded() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| parameters: | ||
| steps: [] # optional -- any additional steps that need to happen before pulling down the jitutils repo and sending the jitutils to helix (ie building your repo) | ||
| variables: [] # optional -- list of additional variables to send to the template | ||
| jobName: '' # required -- job name | ||
| displayName: '' # optional -- display name for the job. Will use jobName if not passed | ||
| pool: '' # required -- name of the Build pool | ||
| container: '' # required -- name of the container | ||
| buildConfig: '' # required -- build configuration | ||
| archType: '' # required -- targeting CPU architecture | ||
| osGroup: '' # required -- operating system for the job | ||
| osSubgroup: '' # optional -- operating system subgroup | ||
| extraSetupParameters: '' # optional -- extra arguments to pass to the setup script | ||
| frameworks: ['net6.0'] # optional -- list of frameworks to run against | ||
| continueOnError: 'false' # optional -- determines whether to continue the build if the step errors | ||
| dependsOn: '' # optional -- dependencies of the job | ||
| timeoutInMinutes: 320 # optional -- timeout for the job | ||
| enableTelemetry: false # optional -- enable for telemetry | ||
| liveLibrariesBuildConfig: '' # optional -- live-live libraries configuration to use for the run | ||
| runtimeType: 'coreclr' # optional -- Sets the runtime as coreclr or mono | ||
| codeGenType: 'JIT' # optional -- Decides on the codegen technology if running on mono | ||
| runKind: '' # required -- test category | ||
| collectionType: '' | ||
| collectionName: '' | ||
| dependOnEvaluatePaths: false | ||
|
|
||
| jobs: | ||
| - template: xplat-pipeline-job.yml | ||
| parameters: | ||
| dependsOn: ${{ parameters.dependsOn }} | ||
| buildConfig: ${{ parameters.buildConfig }} | ||
| archType: ${{ parameters.archType }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| osSubgroup: ${{ parameters.osSubgroup }} | ||
| liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }} | ||
| enableTelemetry: ${{ parameters.enableTelemetry }} | ||
| enablePublishBuildArtifacts: true | ||
| continueOnError: ${{ parameters.continueOnError }} | ||
| dependOnEvaluatePaths: ${{ parameters.dependOnEvaluatePaths }} | ||
| timeoutInMinutes: ${{ parameters.timeoutInMinutes }} | ||
|
|
||
| ${{ if ne(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.displayName }}' | ||
| ${{ if eq(parameters.displayName, '') }}: | ||
| displayName: '${{ parameters.jobName }}' | ||
|
|
||
| variables: | ||
|
|
||
| - ${{ each variable in parameters.variables }}: | ||
| - ${{ if ne(variable.name, '') }}: | ||
| - name: ${{ variable.name }} | ||
| value: ${{ variable.value }} | ||
| - ${{ if ne(variable.group, '') }}: | ||
| - group: ${{ variable.group }} | ||
|
|
||
| - HelixApiAccessToken: '' | ||
| - HelixPreCommand: '' | ||
|
|
||
| - ${{ if eq(parameters.osGroup, 'windows') }}: | ||
| - name: PythonScript | ||
| value: 'py -3' | ||
| - name: PipScript | ||
| value: 'py -3 -m pip' | ||
| - name: Core_Root_Dir | ||
| value: '$(Build.SourcesDirectory)\artifacts\tests\coreclr\${{ parameters.osGroup }}.${{ parameters.archType }}.${{ parameters.buildConfig }}\Tests\Core_Root' | ||
| - name: IssuesLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\helixresults\' | ||
| - name: IssuesSummaryLocation | ||
| value: '$(Build.SourcesDirectory)\artifacts\issues_summary\' | ||
|
|
||
| - ${{ if ne(parameters.osGroup, 'windows') }}: | ||
| - name: PythonScript | ||
| value: 'python3' | ||
| - name: PipScript | ||
| value: 'pip3' | ||
| - name: Core_Root_Dir | ||
| value: '$(Build.SourcesDirectory)/artifacts/tests/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.$(buildConfigUpper)/Tests/Core_Root' | ||
| - name: IssuesLocation | ||
| value: '$(Build.SourcesDirectory)/artifacts/helixresults/' | ||
| - name: IssuesSummaryLocation | ||
| value: '$(Build.SourcesDirectory)/artifacts/issues_summary/' | ||
| workspace: | ||
| clean: all | ||
| pool: | ||
| ${{ parameters.pool }} | ||
| container: ${{ parameters.container }} | ||
| strategy: | ||
| matrix: | ||
| ${{ each framework in parameters.frameworks }}: | ||
| ${{ framework }}: | ||
| _Framework: ${{ framework }} | ||
| steps: | ||
| - ${{ parameters.steps }} | ||
|
|
||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/antigen_setup.py -source_directory $(Build.SourcesDirectory) -core_root_directory $(Core_Root_Dir) -arch $(archType) -platform $(osGroup) | ||
| displayName: ${{ format('Antigen setup ({0}-{1})', parameters.osGroup, parameters.archType) }} | ||
|
|
||
| # Run exploratory tool in helix | ||
| - template: /eng/pipelines/coreclr/templates/superpmi-send-to-helix.yml | ||
| parameters: | ||
| HelixSource: '$(HelixSourcePrefix)/$(Build.Repository.Name)/$(Build.SourceBranch)' # sources must start with pr/, official/, prodcon/, or agent/ | ||
| HelixType: 'test/superpmi/$(CollectionName)/$(CollectionType)/$(_Framework)/$(Architecture)' | ||
| HelixAccessToken: $(HelixApiAccessToken) | ||
| HelixTargetQueues: $(Queue) | ||
| HelixPreCommands: $(HelixPreCommand) | ||
| Creator: $(Creator) | ||
| WorkItemTimeout: 4:00 # 4 hours | ||
| WorkItemDirectory: '$(WorkItemDirectory)' | ||
| CorrelationPayloadDirectory: '$(CorrelationPayloadDirectory)' | ||
| ProjectFile: 'exploratory.proj' | ||
| BuildConfig: ${{ parameters.buildConfig }} | ||
| osGroup: ${{ parameters.osGroup }} | ||
| RunConfiguration: '$(RunConfiguration)' | ||
| ToolName: '$(ToolName)' | ||
| continueOnError: true # Run the future step i.e. merge-mch step even if this step fails. | ||
|
|
||
| - template: /eng/pipelines/common/upload-artifact-step.yml | ||
| parameters: | ||
| rootFolder: $(IssuesLocation) | ||
| includeRootFolder: false | ||
| archiveType: $(archiveType) | ||
| tarCompression: $(tarCompression) | ||
| archiveExtension: $(archiveExtension) | ||
| artifactName: 'Antigen_Issues_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)' | ||
| displayName: ${{ format('Upload artifacts Antigen issues for {0}-{1}', parameters.osGroup, parameters.archType ) }} | ||
|
|
||
| # Always upload the available issues-summary.txt files | ||
| - task: CopyFiles@2 | ||
| displayName: Copying issues-summary.txt of all partitions | ||
| inputs: | ||
| sourceFolder: '$(IssuesLocation)' | ||
| contents: '**/issues-summary-*.txt' | ||
| targetFolder: '$(IssuesSummaryLocation)' | ||
|
|
||
| - task: PublishPipelineArtifact@1 | ||
| displayName: Publish issues-summary.txt files | ||
| inputs: | ||
| targetPath: $(IssuesSummaryLocation) | ||
| artifactName: 'Issues_Summary_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)' | ||
|
|
||
| - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/antigen_unique_issues.py -issues_directory $(IssuesSummaryLocation) | ||
| displayName: ${{ format('Print unique issues ({0})', parameters.osGroup) }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Documentation of Exploratory tool Antigen | ||
|
kunalspathak marked this conversation as resolved.
|
||
|
|
||
| Antigen is a an exploratory fuzzing tool to test JIT compiler. Currently the source code is present at https://github.com/kunalspathak/antigen and will eventually become a separate repository under "dotnet" organization or part of [dotnet/jitutils](https://github.com/dotnet/jitutils) repository. | ||
|
|
||
| ## Overview | ||
|
|
||
| Antigen generates random test cases, compile them using Roslyn APIs and then execute it against `CoreRun.exe` in baseline and test mode. Baseline mode is without tiered JIT with minimum optimizations enabled while test mode is run by setting various combination of `COMPlus_*` environment variables. The output of both modes are compared to ensure they are same. If the output is different or an assert is hit in test mode, the tool saves the produced C# file in a `UniqueIssue` folder. Similar issues are placed in `UniqueIssueN` folder. | ||
|
|
||
| Antigen also comes with `Trimmer` that can be used to reduce the generated file and still reproduce the issue. Some more work needs to be done in this tool to make it efficient. | ||
|
|
||
| ## Pipeline | ||
|
|
||
| Antigen tool is ran every night using CI pipeline. It can also be triggered on PRs. The pipeline would run Antigen tool for 3 hours on x86/x64 platforms and 2 hours for arm/arm64 platforms to generate test cases and verify if it contains issues. Once the run duration is complete, for each OS/arch, the pipeline will upload the issues that Antigen has found has an artifact that can be downloaded. The issues will be `.cs` files that will contain the program's output, environment variables that are needed to reproduce the issue. Since there can be several issues, the pipeline will just upload at most 5 issues from each `UniqueIssueN` folder. | ||
|
|
||
| ### Pipeline details | ||
|
|
||
| 1. `eng/pipeline/coreclr/jit-exploratory.yml` : This is the main pipeline which will perform Coreclr/libraries build and then further trigger `jit-exploratory-job.yml` pipeline. | ||
| 1. `eng/pipeline/coreclr/templates/jit-exploratory-job.yml` : This pipeline will download all the Coreclr/libraries artifacts and create `CORE_ROOT` folder. It further triggers `jit-run-exploratory-job.yml` pipeline. | ||
| 1. `eng/pipeline/coreclr/templates/jit-run-exploratory-job.yml` : This pipeline will perform the actual run in 3 steps: | ||
| * `src/coreclr/scripts/antigen_setup.py`: This script will clone the Antigen repo, build and prepare the payloads to be sent for testing. | ||
| * `src/coreclr/scripts/antigen_run.py`: This script will execute Antigen tool and upload back the issues. | ||
| * `src/coreclr/scripts/antigen_unique_issues.py`: In addition to uploading the issues, this script will also print the output of unique issues it has found so the developer can quickly take a look at them and decide which platform's artifacts to download. If there is any issue found by Antigen, this script will make the pipeline as "FAIL". | ||
| 1. `src/coreclr/scripts/exploratory.proj`: This proj file is the one that creates the helix jobs. Currently, this file configures to run Antigen on 4 partitions. Thus, if Antigen can generate and test 1000 test cases on 1 machine, with current setup, the pipeline will be able to test 4000 test cases. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this file only used for testing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is used to skip building coreclr/libraries and download these from previous runs. This helps in fast testing during pipeline development.