diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000000..7d256687ced --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,185 @@ +# MSBuild - Microsoft Build Engine + +This repo contains the code for the MSBuild build engine, including its public C# API, its internal implementation of the MSBuild programming language, and core targets and tasks used for builds for .NET and Visual Studio. + +Performance is very important--minimize allocations, avoid LINQ, and use the most efficient algorithms possible. The code should be easy to read and understand, but performance is the top priority. + +The code is written in C# and should follow the .NET coding conventions. Use the latest C# features where appropriate, including C# 13 features and especially collection expressions--prefer `[]` to `new Type[]`. + +You should generally match the style of surrounding code when making edits, but if making a substantial change, you can modernize more aggressively. +New files should use nullable types but don't refactor aggressively existing code. + +Generate tests for new codepaths, and add tests for any bugs you fix. Use the existing test framework, which is xUnit with Shouldly assertions. Use Shouldly assertions for all assertions in modified code, even if the file is predominantly using xUnit assertions. + +Always reference these instructions first and fallback to search or bash commands only when you encounter unexpected information that does not match the info here. + +## Working Effectively + +#### Bootstrap and Build the Repository +NEVER build the repository with just `dotnet build MSBuild.sln` or `dotnet build src/.../Project.csproj`. +Run these commands in sequence to set up a complete development environment: + +**Windows:** +```cmd +# Full build with restore - NEVER CANCEL: Takes ~2-3 minutes. Set timeout to 300+ seconds. +build.cmd --build + +# Set up bootstrap environment for using built MSBuild +artifacts\msbuild-build-env.bat +``` + +**macOS/Linux:** +```bash +# Full build with restore - NEVER CANCEL: Takes ~2-3 minutes. Set timeout to 300+ seconds. +./build.sh --build + +# Set up bootstrap environment for using built MSBuild +source artifacts/sdk-build-env.sh +``` + +### Test the Repository +**Windows:** +```cmd +# Run all tests - NEVER CANCEL: Takes ~9 minutes but some tests may fail (this is expected). Set timeout to 900+ seconds. +build.cmd --test + +# Run individual test project (recommended for validation): +artifacts\msbuild-build-env.bat +dotnet test src/Framework.UnitTests/Microsoft.Build.Framework.UnitTests.csproj +``` + +**macOS/Linux:** +```bash +# Run all tests - NEVER CANCEL: Takes ~9 minutes but some tests may fail (this is expected). Set timeout to 900+ seconds. +./build.sh --test + +# Run individual test project (recommended for validation): +source artifacts/sdk-build-env.sh +dotnet test src/Framework.UnitTests/Microsoft.Build.Framework.UnitTests.csproj +``` + +**CRITICAL**: Some unit tests fail in the full test suite due to environment/CI dependencies. This is EXPECTED and normal in development environments. Individual test projects typically work correctly. + +### Using the Built MSBuild + +After building, use the bootstrap environment to work with the locally built MSBuild: + +**Windows:** +```cmd +# Set up environment (run after every new shell session) +artifacts\msbuild-build-env.bat + +# Verify environment is working +dotnet --version +# Should show something like: 10.0.100-preview.7.25372.107 + +# Build a project using the built MSBuild +dotnet build src/Samples/Dependency/Dependency.csproj + +# Run MSBuild directly +dotnet artifacts/bin/bootstrap/core/MSBuild.dll --help +``` + +**macOS/Linux:** +```bash +# Set up environment (run after every new shell session) +source artifacts/sdk-build-env.sh + +# Verify environment is working +dotnet --version +# Should show something like: 10.0.100-preview.7.25372.107 + +# Build a project using the built MSBuild +dotnet build src/Samples/Dependency/Dependency.csproj + +# Run MSBuild directly +dotnet artifacts/bin/bootstrap/core/MSBuild.dll --help +``` + +## Validation + +### Always Test These Scenarios After Making Changes: +1. **Full build validation**: + - Windows: `build.cmd --build` must complete successfully + - macOS/Linux: `./build.sh --build` must complete successfully +2. **Bootstrap environment**: + - Windows: `artifacts\msbuild-build-env.bat && dotnet --version` shows correct preview version + - macOS/Linux: `source artifacts/sdk-build-env.sh && dotnet --version` shows correct preview version +3. **Sample build**: `dotnet build src/Samples/Dependency/Dependency.csproj` succeeds +4. **Individual tests**: Choose a relevant test project and run `dotnet test [project.csproj]` +5. **MSBuild help**: `dotnet artifacts/bin/bootstrap/core/MSBuild.dll --help` shows usage + +### Manual Testing Requirements: +- ALWAYS test the full build after code changes +- Verify the bootstrap environment works correctly with `dotnet --version` +- Test MSBuild executable can display help and basic functionality +- Build at least one sample project to verify core functionality + +## Common Tasks + +### Build Commands and Timing +**Windows:** +- **`build.cmd --build`** - Full build: ~2-3 minutes. NEVER CANCEL. Use 300+ second timeout. +- **`build.cmd --test`** - Run all tests: ~9 minutes with some failures expected. NEVER CANCEL. Use 900+ second timeout. +- **`build.cmd --clean`** - Clean build artifacts: ~30 seconds. + +**macOS/Linux:** +- **`./build.sh --build`** - Full build: ~2-3 minutes. NEVER CANCEL. Use 300+ second timeout. +- **`./build.sh --test`** - Run all tests: ~9 minutes with some failures expected. NEVER CANCEL. Use 900+ second timeout. +- **`./build.sh --clean`** - Clean build artifacts: ~30 seconds. + +### Development Workflow +1. Make your changes to source code +2. Run the full build to compile (WAIT for completion - takes 2-3 minutes): + - Windows: `build.cmd --build` + - macOS/Linux: `./build.sh --build` +3. Set up environment: + - Windows: `artifacts\msbuild-build-env.bat` + - macOS/Linux: `source artifacts/sdk-build-env.sh` +4. Test your changes: `dotnet build src/Samples/Dependency/Dependency.csproj` +5. Run relevant individual tests, not the full test suite +6. Commit your changes + +### Key Project Structure +``` +src/ +├── Build/ # Core MSBuild engine (Microsoft.Build) +├── MSBuild/ # MSBuild command-line tool +├── Framework/ # MSBuild Framework (Microsoft.Build.Framework) +├── Tasks/ # Built-in MSBuild tasks (Microsoft.Build.Tasks) +├── Utilities/ # MSBuild utilities (Microsoft.Build.Utilities) +├── Samples/ # Sample projects and extensions +└── [Component].UnitTests/ # Unit tests for each component + +artifacts/ +├── bin/ # Built binaries and tools +├── sdk-build-env.sh # Bootstrap environment script (Linux/macOS) +├── msbuild-build-env.bat # Bootstrap environment script (Windows) +└── packages/ # Built NuGet packages + +documentation/ +├── wiki/ # Developer documentation +├── specs/ # Technical specifications +└── *.md # Various documentation files +``` + +## Troubleshooting + +### Common Issues and Solutions + +**Build fails with "Could not resolve SDK":** +- Ensure you run `source artifacts/sdk-build-env.sh` after building +- Verify `dotnet --version` shows the preview/RC/internal version (e.g. 10.0.100-preview.7.25372.107) + +**Tests fail:** +- Run individual test projects instead: `dotnet test src/Framework.UnitTests/Microsoft.Build.Framework.UnitTests.csproj` + +**Certificate errors with external projects:** +- Use the repository's sample projects for testing instead of creating new external projects +- The bootstrap environment is designed for building the MSBuild repository itself + +### Files You Should NOT Modify +- `global.json` - Controls .NET SDK version +- `NuGet.config` - Package source configuration +- `artifacts/` directory contents - Generated during build +- `.dotnet/` directory contents - Local SDK location used to build diff --git a/.vsts-dotnet.yml b/.vsts-dotnet.yml index 5fdf27e94a8..dab0ef31e9f 100644 --- a/.vsts-dotnet.yml +++ b/.vsts-dotnet.yml @@ -1,8 +1,6 @@ trigger: - main - vs* -- exp/* -- perf/* # If defined here, these values are not overrideable # Once they exist, we should define these as "runtime parameters" @@ -20,8 +18,8 @@ parameters: displayName: Enable OptProf data collection for this build type: boolean default: true -- name: requireDefaultChannelsEnabled - displayName: Require Default Channels +- name: enableSigningValidation + displayName: Enable Signing Validation type: boolean default: true @@ -40,23 +38,17 @@ variables: # if OptProfDropName is set as a parameter, set OptProfDrop to the parameter and unset SourceBranch - ${{ if ne(parameters.OptProfDropName, 'default') }}: - name: OptProfDrop - value: ${{parameters.OptProfDropName}} + value: ${{ parameters.OptProfDropName }} - name: SourceBranch value: '' # Override SkipApplyOptimizationData to true when disabling OptProf data collection - ${{ if eq(parameters.EnableOptProf, false) }}: - name: SkipApplyOptimizationData value: true - - ${{ if and(not(startsWith(variables['Build.SourceBranch'], 'refs/heads/exp/')), not(startsWith(variables['Build.SourceBranch'], 'refs/heads/perf/'))) }}: - - name: requireDefaultChannels - value: ${{ parameters.requireDefaultChannelsEnabled }} - name: EnableReleaseOneLocBuild value: true # Enable loc for vs17.14 - name: Codeql.Enabled value: true - # ensures we don't build and push experimental versions to official feeds as release versions - - name: IsExperimental - value: ${{ startsWith(variables['Build.SourceBranch'], 'refs/heads/exp/') }} - group: DotNet-MSBuild-SDLValidation-Params - group: AzureDevOps-Artifact-Feeds-Pats - name: cfsNugetWarnLevel @@ -65,6 +57,7 @@ variables: value: none - name: NugetSecurityAnalysisWarningLevel value: none + resources: repositories: - repository: 1ESPipelineTemplates @@ -96,266 +89,46 @@ extends: suppressionsFile: $(Build.SourcesDirectory)/eng/CredScanSuppressions.json stages: + - ${{ if or(startsWith(variables['Build.SourceBranch'], 'refs/heads/vs'), eq(variables['Build.SourceBranch'], 'refs/heads/main')) }}: + - stage: localization + displayName: Localization + jobs: + # The localization setup for release/ branches. Note difference in LclPackageId. main branch is handled separately below. + # Used for vs17.2, vs17.4, vs17.6 etc. branches only. + # When the branch is setup for localization (the localization ticket needs to be created - https://aka.ms/ceChangeLocConfig, requesting change from one release branch to another), + # set 'EnableReleaseOneLocBuild' to true. + - ${{ if startsWith(variables['Build.SourceBranch'], 'refs/heads/vs') }}: + - template: /eng/common/templates-official/job/onelocbuild.yml@self + parameters: + MirrorRepo: 'msbuild' + LclSource: lclFilesfromPackage + LclPackageId: 'LCL-JUNO-PROD-MSBUILDREL' + MirrorBranch: ${{ replace(variables['Build.SourceBranch'], 'refs/heads/', '') }} + JobNameSuffix: '_release' + condition: ${{ variables.EnableReleaseOneLocBuild }} + # The localization setup for main branch. Note difference in package ID. Should not be used with release/ branches. + - ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}: + - template: /eng/common/templates-official/job/onelocbuild.yml@self + parameters: + MirrorRepo: 'msbuild' + LclSource: lclFilesfromPackage + LclPackageId: 'LCL-JUNO-PROD-MSBUILD' + MirrorBranch: 'main' + JobNameSuffix: '_main' + condition: eq(variables['Build.SourceBranch'], 'refs/heads/main') + - stage: build displayName: Build - jobs: - # The localization setup for release/ branches. Note difference in LclPackageId. main branch is handled separately below. - # Used for vs17.2, vs17.4, vs17.6 etc. branches only. - # When the branch is setup for localization (the localization ticket needs to be created - https://aka.ms/ceChangeLocConfig, requesting change from one release branch to another), - # set 'EnableReleaseOneLocBuild' to true. - - ${{ if startsWith(variables['Build.SourceBranch'], 'refs/heads/vs') }}: - - template: /eng/common/templates-official/job/onelocbuild.yml@self - parameters: - MirrorRepo: 'msbuild' - LclSource: lclFilesfromPackage - LclPackageId: 'LCL-JUNO-PROD-MSBUILDREL' - MirrorBranch: ${{ replace(variables['Build.SourceBranch'], 'refs/heads/', '') }} - JobNameSuffix: '_release' - condition: ${{ variables.EnableReleaseOneLocBuild }} - # The localization setup for main branch. Note difference in package ID. Should not be used with release/ branches. - - ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}: - - template: /eng/common/templates-official/job/onelocbuild.yml@self - parameters: - MirrorRepo: 'msbuild' - LclSource: lclFilesfromPackage - LclPackageId: 'LCL-JUNO-PROD-MSBUILD' - MirrorBranch: 'main' - JobNameSuffix: '_main' - condition: eq(variables['Build.SourceBranch'], 'refs/heads/main') - - - job: Windows_NT - pool: - name: VSEngSS-MicroBuild2022-1ES - demands: - - agent.os -equals Windows_NT - - timeoutInMinutes: 180 - - variables: - - group: Publish-Build-Assets - - name: TeamName - value: MSBuild - - name: VisualStudio.MajorVersion - value: 18 - - name: VisualStudio.ChannelName - value: 'int.main' - - name: VisualStudio.DropName - value: Products/$(System.TeamProject)/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber) - - steps: - - task: NuGetToolInstaller@1 - displayName: 'Install NuGet.exe' - - pwsh: Get-MpComputerStatus - - - pwsh: Set-MpPreference -DisableRealtimeMonitoring $true - - - task: PowerShell@2 - displayName: Setup Private Feeds Credentials - inputs: - filePath: $(Build.SourcesDirectory)/eng/common/SetupNugetSources.ps1 - arguments: -ConfigFile $(Build.SourcesDirectory)/NuGet.config -Password $Env:Token - env: - Token: $(dn-bot-dnceng-artifact-feeds-rw) - - - task: NuGetCommand@2 - displayName: Restore internal tools - inputs: - command: restore - feedsToUse: config - restoreSolution: 'eng\common\internal\Tools.csproj' - nugetConfigPath: 'eng\common\internal\NuGet.config' - restoreDirectory: '$(Build.SourcesDirectory)\.packages' - - - task: MicroBuildSigningPlugin@4 - displayName: Install MicroBuild plugin - inputs: - signType: $(SignType) - zipSources: false - feedSource: https://devdiv.pkgs.visualstudio.com/_packaging/MicroBuildToolset/nuget/v3/index.json - ${{ if eq(variables['System.TeamProject'], 'DevDiv') }}: - ConnectedPMEServiceName: 6cc74545-d7b9-4050-9dfa-ebefcc8961ea - ${{ else }}: - ConnectedPMEServiceName: 248d384a-b39b-46e3-8ad5-c2c210d5e7ca - env: - SYSTEM_ACCESSTOKEN: $(System.AccessToken) - condition: and(succeeded(), in(variables['SignType'], 'test', 'real')) - - - task: MicroBuildOptProfPlugin@6 - inputs: - ProfilingInputsDropName: '$(VisualStudio.DropName)' - ShouldSkipOptimize: true - AccessToken: '$(System.AccessToken)' - feedSource: 'https://devdiv.pkgs.visualstudio.com/DefaultCollection/_packaging/MicroBuildToolset/nuget/v3/index.json' - displayName: 'Install OptProf Plugin' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - # Required by MicroBuildBuildVSBootstrapper - - task: MicroBuildSwixPlugin@4 - inputs: - dropName: $(VisualStudio.DropName) - - - script: eng/CIBuild.cmd - -configuration $(BuildConfiguration) - -officialBuildId $(Build.BuildNumber) - -officialSkipApplyOptimizationData $(SkipApplyOptimizationData) - /p:RepositoryName=$(Build.Repository.Name) - /p:VisualStudioIbcSourceBranchName=$(SourceBranch) - /p:VisualStudioDropAccessToken=$(System.AccessToken) - /p:VisualStudioDropName=$(VisualStudio.DropName) - /p:DotNetSignType=$(SignType) - /p:TeamName=MSBuild - /p:DotNetPublishUsingPipelines=true - /p:VisualStudioIbcDrop=$(OptProfDrop) - /p:GenerateSbom=true - /p:SuppressFinalPackageVersion=$(IsExperimental) - displayName: Build - condition: succeeded() - - # Required by Microsoft policy - - template: eng\common\templates-official\steps\generate-sbom.yml@self - - # Publish OptProf configuration files - - task: 1ES.PublishArtifactsDrop@1 - inputs: - dropServiceURI: 'https://devdiv.artifacts.visualstudio.com' - buildNumber: 'ProfilingInputs/DevDiv/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber)' - sourcePath: '$(Build.SourcesDirectory)\artifacts\OptProf\$(BuildConfiguration)\Data' - toLowerCase: false - usePat: true - displayName: 'OptProf - Publish to Artifact Services - ProfilingInputs' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - # Build VS bootstrapper - # Generates $(Build.StagingDirectory)\MicroBuild\Output\BootstrapperInfo.json - - task: MicroBuildBuildVSBootstrapper@3 - inputs: - vsMajorVersion: $(VisualStudio.MajorVersion) - channelName: $(VisualStudio.ChannelName) - manifests: $(VisualStudio.SetupManifestList) - outputFolder: '$(Build.SourcesDirectory)\artifacts\VSSetup\$(BuildConfiguration)\Insertion' - displayName: 'OptProf - Build VS bootstrapper' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - # Publish run settings - - task: PowerShell@2 - inputs: - filePath: eng\common\sdk-task.ps1 - arguments: -configuration $(BuildConfiguration) - -task VisualStudio.BuildIbcTrainingSettings - /p:VisualStudioDropName=$(VisualStudio.DropName) - /p:BootstrapperInfoPath=$(Build.StagingDirectory)\MicroBuild\Output\BootstrapperInfo.json - /p:VisualStudioIbcTrainingSettingsPath=$(Build.SourcesDirectory)\eng\config\OptProf.runsettings - displayName: 'OptProf - Build IBC training settings' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - # Publish bootstrapper info - - task: 1ES.PublishBuildArtifacts@1 - inputs: - PathtoPublish: $(Build.StagingDirectory)\MicroBuild\Output - ArtifactName: MicroBuildOutputs - ArtifactType: Container - displayName: 'OptProf - Publish Artifact: MicroBuildOutputs' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - - task: 1ES.PublishBuildArtifacts@1 - displayName: 'Publish Artifact: logs' - inputs: - PathtoPublish: 'artifacts\log\$(BuildConfiguration)' - ArtifactName: logs - condition: succeededOrFailed() - - # Publishes setup VSIXes to a drop. - # Note: The insertion tool looks for the display name of this task in the logs. - - task: 1ES.MicroBuildVstsDrop@1 - displayName: Upload VSTS Drop - inputs: - dropName: $(VisualStudio.DropName) - dropFolder: 'artifacts\VSSetup\$(BuildConfiguration)\Insertion' - dropRetentionDays: '30' # extended by insertion + VS release - accessToken: '$(System.AccessToken)' - dropServiceUri: 'https://devdiv.artifacts.visualstudio.com' - vsDropServiceUri: 'https://vsdrop.corp.microsoft.com/file/v1' - condition: succeeded() - - # Publish an artifact that the RoslynInsertionTool is able to find by its name. - - task: 1ES.PublishBuildArtifacts@1 - displayName: 'Publish Artifact: VSSetup' - inputs: - PathtoPublish: 'artifacts\VSSetup\$(BuildConfiguration)' - ArtifactName: VSSetup - condition: succeeded() - - # Archive NuGet packages to DevOps. - # Publish our NuPkgs as an artifact. The name of this artifact must be PackageArtifacts as the - # arcade templates depend on the name. - - task: 1ES.PublishBuildArtifacts@1 - displayName: 'Publish Artifact: packages' - inputs: - PathtoPublish: 'artifacts\packages\$(BuildConfiguration)' - ArtifactName: PackageArtifacts - condition: succeeded() - - # Publish "IntelliSense" XSD files to their own artifact - # so it can be consumed by the insertion-to-VS job - - task: 1ES.PublishPipelineArtifact@1 - displayName: 'Publish Artifact: xsd' - inputs: - path: 'artifacts\xsd' - artifactName: xsd - condition: succeeded() - - # Publish Asset Manifests for Build Asset Registry job - - task: 1ES.PublishBuildArtifacts@1 - displayName: Publish Asset Manifests - inputs: - PathtoPublish: '$(Build.SourcesDirectory)/artifacts/log/$(BuildConfiguration)/AssetManifest' - ArtifactName: AssetManifests - condition: succeeded() - - # Tag the build at the very end when we know it's been successful. - - task: colinsalmcorner.colinsalmcorner-buildtasks.tag-build-task.tagBuildOrRelease@0 - displayName: Tag build as ready for optimization training - inputs: - tags: 'ready-for-training' - condition: and(succeeded(), ${{ parameters.EnableOptProf }}) - - - task: ms-vseng.MicroBuildTasks.521a94ea-9e68-468a-8167-6dcf361ea776.MicroBuildCleanup@1 - displayName: Execute cleanup tasks - condition: succeededOrFailed() - - - template: /eng/common/templates-official/steps/component-governance.yml@self - parameters: - ${{ if or(startsWith(variables['Build.SourceBranch'], 'refs/heads/vs'), eq(variables['Build.SourceBranch'], 'refs/heads/main')) }}: - disableComponentGovernance: false - ${{ else }}: - disableComponentGovernance: true - - - template: /eng/common/templates-official/jobs/source-build.yml@self - parameters: - platforms: - - name: Managed - pool: - name: AzurePipelines-EO - image: 1ESPT-Ubuntu22.04 - os: linux - - - template: /eng/common/templates-official/job/publish-build-assets.yml@self + - template: /azure-pipelines/.vsts-dotnet-build-jobs.yml@self parameters: - enablePublishBuildArtifacts: true - publishUsingPipelines: true - dependsOn: - - Windows_NT - - Source_Build_Managed - pool: - name: $(DncEngInternalBuildPool) - image: $(WindowsImage) - os: windows + isExperimental: false + enableComponentGovernance: true - - template: eng\common\templates-official\post-build\post-build.yml@self + - template: /eng/common/templates-official/post-build/post-build.yml@self parameters: publishingInfraVersion: 3 enableSymbolValidation: true enableSourceLinkValidation: false enableNugetValidation: false - requireDefaultChannels: ${{ variables.requireDefaultChannels }} + enableSigningValidation: ${{ parameters.enableSigningValidation }} diff --git a/Directory.Build.targets b/Directory.Build.targets index 3538e50a581..374fe0fc145 100644 --- a/Directory.Build.targets +++ b/Directory.Build.targets @@ -31,6 +31,14 @@ + + + + false + true + true + + diff --git a/azure-pipelines/.vsts-dotnet-build-jobs.yml b/azure-pipelines/.vsts-dotnet-build-jobs.yml new file mode 100644 index 00000000000..d267de0ca42 --- /dev/null +++ b/azure-pipelines/.vsts-dotnet-build-jobs.yml @@ -0,0 +1,233 @@ +# Template for the main Windows_NT build job +parameters: +- name: isExperimental + type: boolean + default: false +- name: enableComponentGovernance + type: boolean + default: false + +jobs: +- job: Windows_NT + pool: + name: VSEngSS-MicroBuild2022-1ES + demands: + - agent.os -equals Windows_NT + + timeoutInMinutes: 180 + + variables: + - group: Publish-Build-Assets + - name: TeamName + value: MSBuild + - name: VisualStudio.MajorVersion + value: 18 + - name: VisualStudio.ChannelName + value: 'int.main' + - name: VisualStudio.DropName + value: Products/$(System.TeamProject)/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber) + + steps: + - task: NuGetToolInstaller@1 + displayName: 'Install NuGet.exe' + - pwsh: Get-MpComputerStatus + + - pwsh: Set-MpPreference -DisableRealtimeMonitoring $true + + - task: PowerShell@2 + displayName: Setup Private Feeds Credentials + inputs: + filePath: $(Build.SourcesDirectory)/eng/common/SetupNugetSources.ps1 + arguments: -ConfigFile $(Build.SourcesDirectory)/NuGet.config -Password $Env:Token + env: + Token: $(dn-bot-dnceng-artifact-feeds-rw) + + - task: NuGetCommand@2 + displayName: Restore internal tools + inputs: + command: restore + feedsToUse: config + restoreSolution: 'eng\common\internal\Tools.csproj' + nugetConfigPath: 'eng\common\internal\NuGet.config' + restoreDirectory: '$(Build.SourcesDirectory)\.packages' + + - task: MicroBuildSigningPlugin@4 + displayName: Install MicroBuild plugin + inputs: + signType: $(SignType) + zipSources: false + feedSource: https://devdiv.pkgs.visualstudio.com/_packaging/MicroBuildToolset/nuget/v3/index.json + ${{ if and(eq(variables['SignType'], 'real'), eq(variables['System.TeamProject'], 'DevDiv')) }}: + ConnectedPMEServiceName: 6cc74545-d7b9-4050-9dfa-ebefcc8961ea + env: + SYSTEM_ACCESSTOKEN: $(System.AccessToken) + condition: and(succeeded(), in(variables['SignType'], 'test', 'real')) + + - task: MicroBuildOptProfPlugin@6 + inputs: + ProfilingInputsDropName: '$(VisualStudio.DropName)' + ShouldSkipOptimize: true + AccessToken: '$(System.AccessToken)' + feedSource: 'https://devdiv.pkgs.visualstudio.com/DefaultCollection/_packaging/MicroBuildToolset/nuget/v3/index.json' + displayName: 'Install OptProf Plugin' + + # Required by MicroBuildBuildVSBootstrapper + - task: MicroBuildSwixPlugin@4 + inputs: + dropName: $(VisualStudio.DropName) + + - script: eng/CIBuild.cmd + -configuration $(BuildConfiguration) + -officialBuildId $(Build.BuildNumber) + -officialSkipApplyOptimizationData $(SkipApplyOptimizationData) + /p:RepositoryName=$(Build.Repository.Name) + /p:VisualStudioIbcSourceBranchName=$(SourceBranch) + /p:VisualStudioDropAccessToken=$(System.AccessToken) + /p:VisualStudioDropName=$(VisualStudio.DropName) + /p:DotNetSignType=$(SignType) + /p:TeamName=MSBuild + /p:DotNetPublishUsingPipelines=true + /p:VisualStudioIbcDrop=$(OptProfDrop) + /p:GenerateSbom=true + /p:SuppressFinalPackageVersion=${{ parameters.isExperimental }} + displayName: Build + condition: succeeded() + + # Required by Microsoft policy + - template: /eng/common/templates-official/steps/generate-sbom.yml@self + + # Publish OptProf configuration files + - task: 1ES.PublishArtifactsDrop@1 + inputs: + dropServiceURI: 'https://devdiv.artifacts.visualstudio.com' + buildNumber: 'ProfilingInputs/DevDiv/$(Build.Repository.Name)/$(Build.SourceBranchName)/$(Build.BuildNumber)' + sourcePath: '$(Build.SourcesDirectory)\artifacts\OptProf\$(BuildConfiguration)\Data' + toLowerCase: false + usePat: true + displayName: 'OptProf - Publish to Artifact Services - ProfilingInputs' + condition: succeeded() + + # Build VS bootstrapper + # Generates $(Build.StagingDirectory)\MicroBuild\Output\BootstrapperInfo.json + - task: MicroBuildBuildVSBootstrapper@3 + inputs: + vsMajorVersion: $(VisualStudio.MajorVersion) + channelName: $(VisualStudio.ChannelName) + manifests: $(VisualStudio.SetupManifestList) + outputFolder: '$(Build.SourcesDirectory)\artifacts\VSSetup\$(BuildConfiguration)\Insertion' + displayName: 'OptProf - Build VS bootstrapper' + condition: succeeded() + + # Publish run settings + - task: PowerShell@2 + inputs: + filePath: eng\common\sdk-task.ps1 + arguments: -configuration $(BuildConfiguration) + -task VisualStudio.BuildIbcTrainingSettings + /p:VisualStudioDropName=$(VisualStudio.DropName) + /p:BootstrapperInfoPath=$(Build.StagingDirectory)\MicroBuild\Output\BootstrapperInfo.json + /p:VisualStudioIbcTrainingSettingsPath=$(Build.SourcesDirectory)\eng\config\OptProf.runsettings + displayName: 'OptProf - Build IBC training settings' + condition: succeeded() + + # Publish bootstrapper info + - task: 1ES.PublishBuildArtifacts@1 + inputs: + PathtoPublish: $(Build.StagingDirectory)\MicroBuild\Output + ArtifactName: MicroBuildOutputs + ArtifactType: Container + displayName: 'OptProf - Publish Artifact: MicroBuildOutputs' + condition: succeeded() + + - task: 1ES.PublishBuildArtifacts@1 + displayName: 'Publish Artifact: logs' + inputs: + PathtoPublish: 'artifacts\log\$(BuildConfiguration)' + ArtifactName: logs + condition: succeededOrFailed() + + # Publishes setup VSIXes to a drop. + # Note: The insertion tool looks for the display name of this task in the logs. + - task: 1ES.MicroBuildVstsDrop@1 + displayName: Upload VSTS Drop + inputs: + dropName: $(VisualStudio.DropName) + dropFolder: 'artifacts\VSSetup\$(BuildConfiguration)\Insertion' + dropRetentionDays: '30' # extended by insertion + VS release + accessToken: '$(System.AccessToken)' + dropServiceUri: 'https://devdiv.artifacts.visualstudio.com' + vsDropServiceUri: 'https://vsdrop.corp.microsoft.com/file/v1' + condition: succeeded() + + # Publish an artifact that the RoslynInsertionTool is able to find by its name. + - task: 1ES.PublishBuildArtifacts@1 + displayName: 'Publish Artifact: VSSetup' + inputs: + PathtoPublish: 'artifacts\VSSetup\$(BuildConfiguration)' + ArtifactName: VSSetup + condition: succeeded() + + # Archive NuGet packages to DevOps. + # Publish our NuPkgs as an artifact. The name of this artifact must be PackageArtifacts as the + # arcade templates depend on the name. + - task: 1ES.PublishBuildArtifacts@1 + displayName: 'Publish Artifact: packages' + inputs: + PathtoPublish: 'artifacts\packages\$(BuildConfiguration)' + ArtifactName: PackageArtifacts + condition: succeeded() + + # Publish "IntelliSense" XSD files to their own artifact + # so it can be consumed by the insertion-to-VS job + - task: 1ES.PublishPipelineArtifact@1 + displayName: 'Publish Artifact: xsd' + inputs: + path: 'artifacts\xsd' + artifactName: xsd + condition: succeeded() + + # Publish Asset Manifests for Build Asset Registry job + - task: 1ES.PublishBuildArtifacts@1 + displayName: Publish Asset Manifests + inputs: + PathtoPublish: '$(Build.SourcesDirectory)/artifacts/log/$(BuildConfiguration)/AssetManifest' + ArtifactName: AssetManifests + condition: succeeded() + + # Tag the build at the very end when we know it's been successful. + - task: colinsalmcorner.colinsalmcorner-buildtasks.tag-build-task.tagBuildOrRelease@0 + displayName: Tag build as ready for optimization training + inputs: + tags: 'ready-for-training' + condition: succeeded() + + - task: ms-vseng.MicroBuildTasks.521a94ea-9e68-468a-8167-6dcf361ea776.MicroBuildCleanup@1 + displayName: Execute cleanup tasks + condition: succeededOrFailed() + + # Conditional component governance + - ${{ if parameters.enableComponentGovernance }}: + - template: /eng/common/templates-official/steps/component-governance.yml@self + parameters: + disableComponentGovernance: false + +- template: /eng/common/templates-official/jobs/source-build.yml@self + parameters: + platforms: + - name: Managed + pool: + name: AzurePipelines-EO + image: 1ESPT-Ubuntu22.04 + os: linux + +- template: /eng/common/templates-official/job/publish-build-assets.yml@self + parameters: + enablePublishBuildArtifacts: true + publishUsingPipelines: true + dependsOn: + - Windows_NT + - Source_Build_Managed + pool: + name: $(DncEngInternalBuildPool) + image: $(WindowsImage) + os: windows diff --git a/azure-pipelines/.vsts-dotnet-exp-perf.yml b/azure-pipelines/.vsts-dotnet-exp-perf.yml new file mode 100644 index 00000000000..a6c5460ef54 --- /dev/null +++ b/azure-pipelines/.vsts-dotnet-exp-perf.yml @@ -0,0 +1,95 @@ +trigger: +- exp/* +- perf/* + +# If defined here, these values are not overrideable +# Once they exist, we should define these as "runtime parameters" +# https://github.com/Microsoft/azure-pipelines-yaml/pull/129 +# variables: +# SignType: test +# SkipApplyOptimizationData: false + +parameters: +- name: OptProfDropName + displayName: Optional OptProfDrop Override + type: string + default: 'default' +- name: enableSigningValidation + displayName: Enable Signing Validation + type: boolean + default: false + +variables: + # if OptProfDrop is not set, string '$(OptProfDrop)' will be passed to the build script. + - name: OptProfDrop + value: '' + - name: requireDefaultChannels + value: false + - name: SourceBranch + value: $(IbcSourceBranchName) + # Use main as our optprof collection branch by default in exp branches. + - ${{ if not(startsWith(variables['Build.SourceBranch'], 'refs/heads/vs')) }}: + - name: SourceBranch + value: main + # if OptProfDropName is set as a parameter, set OptProfDrop to the parameter and unset SourceBranch + - ${{ if ne(parameters.OptProfDropName, 'default') }}: + - name: OptProfDrop + value: ${{parameters.OptProfDropName}} + - name: SourceBranch + value: '' + - name: Codeql.Enabled + value: false + - group: DotNet-MSBuild-SDLValidation-Params + - group: AzureDevOps-Artifact-Feeds-Pats + - name: cfsNugetWarnLevel + value: warn + - name: nugetMultiFeedWarnLevel + value: none + - name: NugetSecurityAnalysisWarningLevel + value: none + +resources: + repositories: + - repository: 1ESPipelineTemplates + type: git + name: 1ESPipelineTemplates/1ESPipelineTemplates + +extends: + template: v1/1ES.Official.PipelineTemplate.yml@1ESPipelineTemplates + parameters: + featureFlags: + autoBaseline: true + pool: + name: VSEngSS-MicroBuild2022-1ES + os: windows + sdl: + policheck: + enabled: true + exclusionsFile: $(Build.SourcesDirectory)\eng\policheck_exclusions.xml + tsa: + enabled: true + # We generate SBOM ourselves, so don't need steps injected by 1ES. + sbom: + enabled: false + codeSignValidation: + enabled: false + additionalTargetsGlobPattern: -|**\bootstrapper\**\vs_enterprise.exe + credscan: + suppressionsFile: $(Build.SourcesDirectory)/eng/CredScanSuppressions.json + + stages: + - stage: build + displayName: Build + jobs: + - template: /azure-pipelines/.vsts-dotnet-build-jobs.yml@self + parameters: + isExperimental: true + enableComponentGovernance: false + + - template: /eng/common/templates-official/post-build/post-build.yml@self + parameters: + publishingInfraVersion: 3 + enableSymbolValidation: true + enableSourceLinkValidation: false + enableNugetValidation: false + enableSigningValidation: ${{ parameters.enableSigningValidation }} \ No newline at end of file diff --git a/azure-pipelines/vs-insertion-experimental.yml b/azure-pipelines/vs-insertion-experimental.yml index a61586a3034..295f82f1639 100644 --- a/azure-pipelines/vs-insertion-experimental.yml +++ b/azure-pipelines/vs-insertion-experimental.yml @@ -4,9 +4,9 @@ name: $(Date:yyyyMMdd).$(Rev:r) resources: pipelines: - - pipeline: 'MSBuild' + - pipeline: 'MSBuildExpPerf' project: 'DevDiv' - source: 'MSBuild' + source: 'MSBuild/MSBuild-ExpPerf' trigger: branches: include: @@ -40,9 +40,9 @@ variables: - name: TeamEmail value: msbtm@microsoft.com - name: MSBuild_CI_BuildNumber - value: $(resources.pipeline.MSBuild.runName) + value: $(resources.pipeline.MSBuildExpPerf.runName) - name: MSBuild_CI_SourceVersion - value: $(resources.pipeline.MSBuild.sourceCommit) + value: $(resources.pipeline.MSBuildExpPerf.sourceCommit) - name: ArtifactPackagesPath value: $(Build.ArtifactStagingDirectory)/PackageArtifacts @@ -80,11 +80,11 @@ extends: templateContext: inputs: - input: pipelineArtifact - pipeline: 'MSBuild' + pipeline: 'MSBuildExpPerf' artifactName: 'xsd' targetPath: '$(Pipeline.Workspace)/xsd' - input: pipelineArtifact - pipeline: 'MSBuild' + pipeline: 'MSBuildExpPerf' artifactName: 'PackageArtifacts' targetPath: '$(Build.ArtifactStagingDirectory)/PackageArtifacts' # the CI build creates a sourcebuild intermediate package that is not signed, remove it to avoid warning from Guardian @@ -99,7 +99,7 @@ extends: targetType: inline script: | # Extract the last section after the last '/' - $fullBranch = "$(resources.pipeline.MSBuild.sourceBranch)" + $fullBranch = "$(resources.pipeline.MSBuildExpPerf.sourceBranch)" $branchSegments = $fullBranch -split '/' $branch = $branchSegments[-1] Write-Host "Setting drops branch to '$branch'" @@ -112,7 +112,7 @@ extends: targetType: inline script: | # Extract VS version from branch name if it follows exp/vsXX.Y-somename pattern - $fullBranch = "$(resources.pipeline.MSBuild.sourceBranch)" + $fullBranch = "$(resources.pipeline.MSBuildExpPerf.sourceBranch)" $parameterTargetBranch = "${{ parameters.TargetBranch }}" $detectedTarget = "main" # Default target branch diff --git a/documentation/wiki/CollectedTelemetry.md b/documentation/wiki/CollectedTelemetry.md index de290b12f15..19b09393edf 100644 --- a/documentation/wiki/CollectedTelemetry.md +++ b/documentation/wiki/CollectedTelemetry.md @@ -26,11 +26,8 @@ Expressed and collected via [LoggingConfigurationTelemetry type](https://github. | >= 8.0.100 | Default choice on terminal logger enablement. | | >= 8.0.100 | Source of default choice on terminal logger enablement. | | >= 8.0.100 | Indication if Console logger was used. | -| >= 8.0.100 | Console logger type (serial, parallel). | | >= 8.0.100 | Console logger verbosity. | | >= 8.0.100 | Indication if File logger was used. | -| >= 8.0.100 | File logger type (serial, parallel). | -| >= 8.0.100 | Number of file loggers. | | >= 8.0.100 | File logger verbosity. | | >= 8.0.100 | Indication if Binary logger was used. | | >= 8.0.100 | Indication if Binary logger used with default log name. | diff --git a/eng/Version.Details.props b/eng/Version.Details.props index 322e8a03c1a..532ac337f5c 100644 --- a/eng/Version.Details.props +++ b/eng/Version.Details.props @@ -25,12 +25,12 @@ This file should be imported by eng/Versions.props 9.0.0 9.0.0 - 10.0.0-beta.25415.5 - 10.0.0-beta.25415.5 + 10.0.0-beta.25422.3 + 10.0.0-beta.25422.3 - 7.0.0-preview.1.190 + 7.0.0-preview.1.197 - 5.0.0-2.25415.2 + 5.0.0-2.25424.8 diff --git a/eng/Version.Details.xml b/eng/Version.Details.xml index 4221ffa3e4f..036f713af27 100644 --- a/eng/Version.Details.xml +++ b/eng/Version.Details.xml @@ -1,6 +1,6 @@ - + @@ -106,21 +106,21 @@ - + https://github.com/dotnet/arcade - e58b086a85a1df5762c8980308f7e4e330edd0aa + f2cdf946c00a12a2c283835bb41ddc2255832055 - + https://github.com/nuget/nuget.client - aa6cfbc3e15f2054d4be1ffbcdd4e74777c2ba23 + ef6cddc7c977ddbfa28dba10b1ebeec77c6a3674 - + https://github.com/dotnet/roslyn - 7c7cfd9c3fee9f3be48c1f8b7f1d139cfb3f5942 + e0c717e94a83de00b8ebe340b6c66c04f4e4e639 - + https://github.com/dotnet/arcade - e58b086a85a1df5762c8980308f7e4e330edd0aa + f2cdf946c00a12a2c283835bb41ddc2255832055 diff --git a/eng/Versions.props b/eng/Versions.props index 7d932d48554..ae4d21d85cd 100644 --- a/eng/Versions.props +++ b/eng/Versions.props @@ -69,7 +69,7 @@ 5.0.0-1.25277.114 - 10.0.100-preview.7.25380.108 + 10.0.100-rc.1.25411.109