Pipeline to run Exploratory fuzzing tool to catch codegen issues

eng/pipelines/common/download-specific-artifact-step.yml

@@ -12,10 +12,10 @@ steps:
     inputs:
       buildType: specific
       project:  'internal' # 'internal' or 'public'
-      pipeline: 'superpmi-replay'
+      pipeline: 'Antigen'
       buildVersionToDownload: specific
-      branchName: 'your/branch/having/artifacts'
-      buildId: '1274841'
+      branchName: 'kpathak/antigen-ci'
+      buildId: '1385069'
       downloadType: single
       downloadPath: '$(Build.SourcesDirectory)/__download__'
       artifactName: '${{ parameters.artifactName }}'

eng/pipelines/coreclr/exploratory.yml

@@ -0,0 +1,51 @@
+# This job definition automates the exploratory tool.
+
+trigger: none
+
+schedules:
+- cron: "0 14 * * 0"
+  displayName: Sun at 6:00 AM (UTC-8:00)
+  branches:
+    include:
+    - main
+  always: true
+
+jobs:
+
+- template: /eng/pipelines/common/platform-matrix.yml
+  parameters:
+    jobTemplate: /eng/pipelines/common/build-coreclr-and-libraries-job.yml
+    buildConfig: checked
+    platforms:
+    # Linux tests are built on the OSX machines.
+    # - OSX_x64
+    - Linux_arm
+    - Linux_arm64
+    - Linux_x64
+    - windows_x64
+    - windows_x86
+    - windows_arm64
+    - CoreClrTestBuildHost # Either OSX_x64 or Linux_x64
+    jobParameters:
+      testGroup: outerloop
+
+- template: /eng/pipelines/common/platform-matrix.yml
+  parameters:
+    jobTemplate: /eng/pipelines/coreclr/templates/jit-exploratory-job.yml
+    buildConfig: checked
+    platforms:
+    # Linux tests are built on the OSX machines.
+    # - OSX_x64
+    - Linux_arm
+    - Linux_arm64
+    - Linux_x64
+    - windows_x64
+    - windows_x86
+    - windows_arm64
+    helixQueueGroup: ci
+    helixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.yml
+    jobParameters:
+      testGroup: outerloop
+      liveLibrariesBuildConfig: Release
+      collectionType: pmi
+      collectionName: libraries

eng/pipelines/coreclr/templates/jit-exploratory-job.yml

@@ -0,0 +1,79 @@
+parameters:
+  buildConfig: ''
+  archType: ''
+  osGroup: ''
+  osSubgroup: ''
+  container: ''
+  runtimeVariant: ''
+  testGroup: ''
+  framework: net6.0 # Specify the appropriate framework when running release branches (ie netcoreapp3.0 for release/3.0)
+  liveLibrariesBuildConfig: ''
+  variables: {}
+  runtimeType: 'coreclr'
+  pool: ''
+  codeGenType: 'JIT'
+  projetFile: ''
+  runKind: ''
+  runJobTemplate: '/eng/pipelines/coreclr/templates/jit-run-exploratory-job.yml'
+  additionalSetupParameters: ''
+  collectionType: ''
+  collectionName: ''
+
+### Exploratory job
+
+### Each exploratory job depends on a corresponding build job with the same
+### buildConfig and archType.
+
+jobs:
+- template: ${{ parameters.runJobTemplate }}
+  parameters:
+    # Compute job name from template parameters
+    jobName: ${{ format('exploratory_{0}{1}_{2}_{3}_{4}_{5}_{6}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind) }}
+    displayName: ${{ format('Exploratory {0}{1} {2} {3} {4} {5} {6}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig, parameters.runtimeType, parameters.codeGenType, parameters.runKind) }}
+    pool: ${{ parameters.pool }}
+    buildConfig: ${{ parameters.buildConfig }}
+    archType: ${{ parameters.archType }}
+    osGroup: ${{ parameters.osGroup }}
+    osSubgroup: ${{ parameters.osSubgroup }}
+    runtimeVariant: ${{ parameters.runtimeVariant }}
+    liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }}
+    runtimeType: ${{ parameters.runtimeType }}
+    codeGenType: ${{ parameters.codeGenType }}
+    runKind: ${{ parameters.runKind }}
+    testGroup: ${{ parameters.testGroup }}
+    collectionType: ${{ parameters.collectionType }}
+    collectionName: ${{ parameters.collectionName }}
+    additionalSetupParameters: ${{ parameters.additionalSetupParameters }}
+    # Test job depends on the corresponding build job
+    dependsOn:
+    - ${{ format('coreclr_{0}_product_build_{1}{2}_{3}_{4}', parameters.runtimeVariant, parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.buildConfig) }}
+    - ${{ format('libraries_build_{0}{1}_{2}_{3}', parameters.osGroup, parameters.osSubgroup, parameters.archType, parameters.liveLibrariesBuildConfig) }}
+
+    variables: ${{ parameters.variables }}
+
+    frameworks:
+      - ${{ parameters.framework }}
+    steps:
+    # Extra steps that will be passed to the exploratory template and run before sending the job to helix (all of which is done in the template)
+
+    # Optionally download live-built libraries
+    - template: /eng/pipelines/common/download-artifact-step.yml
+      parameters:
+        unpackFolder: $(librariesDownloadDir)
+        cleanUnpackFolder: false
+        artifactFileName: '$(librariesBuildArtifactName)$(archiveExtension)'
+        artifactName: '$(librariesBuildArtifactName)'
+        displayName: 'live-built libraries'
+
+    # Download coreclr
+    - template: /eng/pipelines/common/download-artifact-step.yml
+      parameters:
+        unpackFolder: $(buildProductRootFolderPath)
+        artifactFileName: '$(buildProductArtifactName)$(archiveExtension)'
+        artifactName: '$(buildProductArtifactName)'
+        displayName: 'Coreclr product build'
+
+    # Create Core_Root
+    - script: $(Build.SourcesDirectory)/src/tests/build$(scriptExt) $(buildConfig) $(archType) generatelayoutonly $(librariesOverrideArg)
+      displayName: Create Core_Root
+      condition: succeeded()

eng/pipelines/coreclr/templates/jit-run-exploratory-job.yml

@@ -0,0 +1,141 @@
+parameters:
+  steps: []                       # optional -- any additional steps that need to happen before pulling down the jitutils repo and sending the jitutils to helix (ie building your repo)
+  variables: []                   # optional -- list of additional variables to send to the template
+  jobName: ''                     # required -- job name
+  displayName: ''                 # optional -- display name for the job. Will use jobName if not passed
+  pool: ''                        # required -- name of the Build pool
+  container: ''                   # required -- name of the container
+  buildConfig: ''                 # required -- build configuration
+  archType: ''                    # required -- targeting CPU architecture
+  osGroup: ''                     # required -- operating system for the job
+  osSubgroup: ''                  # optional -- operating system subgroup
+  extraSetupParameters: ''        # optional -- extra arguments to pass to the setup script
+  frameworks: ['net6.0']          # optional -- list of frameworks to run against
+  continueOnError: 'false'        # optional -- determines whether to continue the build if the step errors
+  dependsOn: ''                   # optional -- dependencies of the job
+  timeoutInMinutes: 320           # optional -- timeout for the job
+  enableTelemetry: false          # optional -- enable for telemetry
+  liveLibrariesBuildConfig: ''    # optional -- live-live libraries configuration to use for the run
+  runtimeType: 'coreclr'          # optional -- Sets the runtime as coreclr or mono
+  codeGenType: 'JIT'              # optional -- Decides on the codegen technology if running on mono
+  runKind: ''                     # required -- test category
+  collectionType: ''
+  collectionName: ''
+  dependOnEvaluatePaths: false
+
+jobs:
+- template: xplat-pipeline-job.yml
+  parameters:
+    dependsOn: ${{ parameters.dependsOn }}
+    buildConfig: ${{ parameters.buildConfig }}
+    archType: ${{ parameters.archType }}
+    osGroup: ${{ parameters.osGroup }}
+    osSubgroup: ${{ parameters.osSubgroup }}
+    liveLibrariesBuildConfig: ${{ parameters.liveLibrariesBuildConfig }}
+    enableTelemetry: ${{ parameters.enableTelemetry }}
+    enablePublishBuildArtifacts: true
+    continueOnError: ${{ parameters.continueOnError }}
+    dependOnEvaluatePaths: ${{ parameters.dependOnEvaluatePaths }}
+    timeoutInMinutes: ${{ parameters.timeoutInMinutes }}
+
+    ${{ if ne(parameters.displayName, '') }}:
+      displayName: '${{ parameters.displayName }}'
+    ${{ if eq(parameters.displayName, '') }}:
+      displayName: '${{ parameters.jobName }}'
+
+    variables:
+
+    - ${{ each variable in parameters.variables }}:
+      - ${{ if ne(variable.name, '') }}:
+        - name: ${{ variable.name }}
+          value: ${{ variable.value }}
+      - ${{ if ne(variable.group, '') }}:
+        - group: ${{ variable.group }}
+
+    - HelixApiAccessToken: ''
+    - HelixPreCommand: ''
+
+    - ${{ if eq(parameters.osGroup, 'windows') }}:
+      - name: PythonScript
+        value: 'py -3'
+      - name: PipScript
+        value: 'py -3 -m pip'
+      - name: Core_Root_Dir
+        value: '$(Build.SourcesDirectory)\artifacts\tests\coreclr\${{ parameters.osGroup }}.${{ parameters.archType }}.${{ parameters.buildConfig }}\Tests\Core_Root'
+      - name: IssuesLocation
+        value: '$(Build.SourcesDirectory)\artifacts\helixresults\'
+      - name: IssuesSummaryLocation
+        value: '$(Build.SourcesDirectory)\artifacts\issues_summary\'
+
+    - ${{ if ne(parameters.osGroup, 'windows') }}:
+      - name: PythonScript
+        value: 'python3'
+      - name: PipScript
+        value: 'pip3'
+      - name: Core_Root_Dir
+        value: '$(Build.SourcesDirectory)/artifacts/tests/coreclr/${{ parameters.osGroup }}.${{ parameters.archType }}.$(buildConfigUpper)/Tests/Core_Root'
+      - name: IssuesLocation
+        value: '$(Build.SourcesDirectory)/artifacts/helixresults/'
+      - name: IssuesSummaryLocation
+        value: '$(Build.SourcesDirectory)/artifacts/issues_summary/'
+    workspace:
+      clean: all
+    pool:
+      ${{ parameters.pool }}
+    container: ${{ parameters.container }}
+    strategy:
+      matrix:
+        ${{ each framework in parameters.frameworks }}:
+          ${{ framework }}:
+            _Framework: ${{ framework }}
+    steps:
+    - ${{ parameters.steps }}
+
+    - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/antigen_setup.py -source_directory $(Build.SourcesDirectory) -core_root_directory $(Core_Root_Dir) -arch $(archType) -platform $(osGroup)
+      displayName: ${{ format('Antigen setup ({0}-{1})', parameters.osGroup, parameters.archType) }}
+
+      # Run exploratory tool in helix
+    - template: /eng/pipelines/coreclr/templates/superpmi-send-to-helix.yml
+      parameters:
+        HelixSource: '$(HelixSourcePrefix)/$(Build.Repository.Name)/$(Build.SourceBranch)' # sources must start with pr/, official/, prodcon/, or agent/
+        HelixType: 'test/superpmi/$(CollectionName)/$(CollectionType)/$(_Framework)/$(Architecture)'
+        HelixAccessToken: $(HelixApiAccessToken)
+        HelixTargetQueues: $(Queue)
+        HelixPreCommands: $(HelixPreCommand)
+        Creator: $(Creator)
+        WorkItemTimeout: 2:30 # 2.5 hours
+        WorkItemDirectory: '$(WorkItemDirectory)'
+        CorrelationPayloadDirectory: '$(CorrelationPayloadDirectory)'
+        ProjectFile: 'exploratory.proj'
+        BuildConfig: ${{ parameters.buildConfig }}
+        osGroup: ${{ parameters.osGroup }}
+        RunConfiguration: '$(RunConfiguration)'
+        ToolName: '$(ToolName)'
+        continueOnError: true # Run the future step i.e. merge-mch step even if this step fails.
+
+    - template: /eng/pipelines/common/upload-artifact-step.yml
+      parameters:
+        rootFolder: $(IssuesLocation)
+        includeRootFolder: false
+        archiveType: $(archiveType)
+        tarCompression: $(tarCompression)
+        archiveExtension: $(archiveExtension)
+        artifactName: 'Antigen_Issues_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)'
+        displayName: ${{ format('Upload artifacts Antigen issues for {0}-{1}', parameters.osGroup, parameters.archType ) }}
+
+    # Always upload the available issues-summary.txt files
+    - task: CopyFiles@2
+      displayName: Copying issues-summary.txt of all partitions
+      inputs:
+        sourceFolder: '$(IssuesLocation)'
+        contents: '**/issues-summary-*.txt'
+        targetFolder: '$(IssuesSummaryLocation)'
+
+    - task: PublishPipelineArtifact@1
+      displayName: Publish issues-summary.txt files
+      inputs:
+        targetPath: $(IssuesSummaryLocation)
+        artifactName: 'Issues_Summary_$(osGroup)$(osSubgroup)_$(archType)_$(buildConfig)'
+
+    - script: $(PythonScript) $(Build.SourcesDirectory)/src/coreclr/scripts/antigen_unique_issues.py -issues_directory $(IssuesSummaryLocation)
+      displayName: ${{ format('Print unique issues ({0})', parameters.osGroup) }}

src/coreclr/scripts/antigen.md

@@ -0,0 +1,23 @@
+# Documentation of Exploratory tool Antigen
+
+Antigen is a an exploratory fuzzing tool to test JIT compiler. Currently the source code is present at https://github.com/kunalspathak/antigen and will eventually become a separate repository under "dotnet" organization or part of [dotnet/jitutils](https://github.com/dotnet/jitutils) repository.
+
+## Overview
+
+Antigen generates random test cases, compile them using Roslyn APIs and then execute it against `CoreRun.exe` in baseline and test mode. Baseline mode is without tiered JIT with minimum optimizations enabled while test mode is run by setting various combination of `COMPlus_*` environment variables. The output of both modes are compared to ensure they are same. If the output is different or an assert is hit in test mode, the tool saves the produced C# file in a `UniqueIssue` folder. Similar issues are placed in `UniqueIssueN` folder.
+
+Antigen also comes with `Trimmer` that can be used to reduce the generated file and still reproduce the issue. Some more work needs to be done in this tool to make it efficient.
+
+## Pipeline
+
+Antigen tool is ran every night using CI pipeline. It can also be triggered on PRs. The pipeline would run Antigen tool for 3 hours on x86/x64 platforms and 2 hours for arm/arm64 platforms to generate test cases and verify if it contains issues. Once the run duration is complete, for each OS/arch, the pipeline will upload the issues that Antigen has found has an artifact that can be downloaded. The issues will be `.cs` files that will contain the program's output, environment variables that are needed to reproduce the issue. Since there can be several issues, the pipeline will just upload at most 5 issues from each `UniqueIssueN` folder.
+
+### Pipeline details
+
+1. `eng/pipeline/coreclr/jit-exploratory.yml` : This is the main pipeline which will perform Coreclr/libraries build and then further trigger `jit-exploratory-job.yml` pipeline.
+1. `eng/pipeline/coreclr/templates/jit-exploratory-job.yml` : This pipeline will download all the Coreclr/libraries artifacts and create `CORE_ROOT` folder. It further triggers `jit-run-exploratory-job.yml`  pipeline.
+1. `eng/pipeline/coreclr/templates/jit-run-exploratory-job.yml` : This pipeline will perform the actual run in 3 steps:
+   * `src/coreclr/scripts/antigen_setup.py`: This script will clone the Antigen repo, build and prepare the payloads to be sent for testing.
+   * `src/coreclr/scripts/antigen_run.py`: This script will execute Antigen tool and upload back the issues.
+   * `src/coreclr/scripts/antigen_unique_issues.py`: In addition to uploading the issues, this script will also print the output of unique issues it has found so the developer can quickly take a look at them and decide which platform's artifacts to download. If there is any issue found by Antigen, this script will make the pipeline as "FAIL".
+1. `src/coreclr/scripts/exploratory.proj`: This proj file is the one that creates the helix jobs. Currently, this file configures to run Antigen on 4 partitions. Thus, if Antigen can generate and test 1000 test cases on 1 machine, with current setup, the pipeline will be able to test 4000 test cases.