Skip to content

Use node 13 for CodeGeneration and merge pipelines #14989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pakrym merged 12 commits into from
Sep 14, 2020
Merged

Use node 13 for CodeGeneration and merge pipelines #14989

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
36d1e72
Use node 14 for CodeGeneration
pakrym Sep 8, 2020
ca7480a
13
pakrym Sep 8, 2020
3a243d4
Merge steps
pakrym Sep 8, 2020
8994690
Reorder
pakrym Sep 8, 2020
ad748b6
Reorder more
pakrym Sep 9, 2020
653d33c
More
pakrym Sep 9, 2020
612c782
timeout
pakrym Sep 9, 2020
32ffd0a
In parallel
pakrym Sep 9, 2020
e693c65
rem
pakrym Sep 9, 2020
36ea7fd
Rev
pakrym Sep 9, 2020
39810ef
rev
pakrym Sep 14, 2020
8a2f2af
Rem dep
pakrym Sep 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions eng/Directory.Build.Data.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
<ValidateReleaseNotesScriptPath Condition=" '$(ValidateReleaseNotesScriptPath)'=='' ">$(MSBuildThisFileDirectory)common/scripts/Verify-ChangeLog.ps1</ValidateReleaseNotesScriptPath>
<ChangeLogPath>$([MSBuild]::NormalizeDirectory($(MSBuildProjectDirectory)/../))CHANGELOG.md</ChangeLogPath>
</PropertyGroup>
<Exec ContinueOnError="true" ConsoleToMSBuild="true" Command="$(PowerShellExe) -NoProfile -NonInteractive -executionpolicy Unrestricted -File $(ValidateReleaseNotesScriptPath) -ChangeLogLocation $(ChangeLogPath) -VersionString $(_VersionInProject)">
<Exec ContinueOnError="true" ConsoleToMSBuild="true" StandardOutputImportance="Low" Command="$(PowerShellExe) -NoProfile -NonInteractive -executionpolicy Unrestricted -File $(ValidateReleaseNotesScriptPath) -ChangeLogLocation $(ChangeLogPath) -VersionString $(_VersionInProject)">
<Output TaskParameter="ExitCode" PropertyName="SetReleaseNotesErrorCode" />
</Exec>
<Error Condition="'$(SetReleaseNotesErrorCode)' != '0'" Text="No entry for version [$(_VersionInProject)] found in the ChangeLog [$(ChangeLogPath)]. " />
Expand Down Expand Up @@ -173,7 +173,7 @@
Text="When UseProjectReferenceToAzureClients=true all Azure.* references should be Project References, but the following are not [@(ShouldBeProjectReference)]" />
</Target>

<Target Name="RunApiCompat" AfterTargets="CoreBuild" Condition="'$(ApiCompatVersion)' != '' and '$(DesignTimeBuild)' != 'true'">
<Target Name="RunApiCompat" AfterTargets="CoreBuild" Condition="'$(ApiCompatVersion)' != '' and '$(DesignTimeBuild)' != 'true' and '$(SkipApiCompat)' != 'true'">
<MSBuild
Projects="$(MSBuildThisFileDirectory)/ApiCompat/ApiCompat.csproj"
Properties="TargetPackageName=$(PackageId);
Expand Down
50 changes: 21 additions & 29 deletions eng/pipelines/templates/jobs/archetype-sdk-client.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,17 @@ jobs:
displayName: 'Use Python 3.6'
inputs:
versionSpec: '3.6'
- template: /eng/common/pipelines/templates/steps/verify-path-length.yml
parameters:
SourceDirectory: $(Build.SourcesDirectory)
- template: /eng/common/pipelines/templates/steps/verify-links.yml
parameters:
Directory: sdk/${{ parameters.ServiceDirectory }}
- script: |
pip install setuptools wheel
pip install doc-warden==$(DocWardenVersion)
ward scan -d $(Build.SourcesDirectory) -c $(Build.SourcesDirectory)/eng/.docsettings.yml
displayName: "Verify Readmes"
- template: /eng/common/pipelines/templates/steps/replace-relative-links.yml
parameters:
TargetFolder: '.'
Expand All @@ -38,6 +49,13 @@ jobs:
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
DOTNET_CLI_TELEMETRY_OPTOUT: 1
DOTNET_MULTILEVEL_LOOKUP: 0
- task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
displayName: "Component Detection"
# ComponentGovernance is currently unable to run on pull requests of public projects. Running on
# scheduled builds should be good enough.
inputs:
ignoreDirectories: "sdk/storage/Azure.Storage.Common/swagger/Generator"
condition: and(succeededOrFailed(), ne(variables['Build.Reason'],'PullRequest'))
- task: PublishPipelineArtifact@1
condition: succeeded()
inputs:
Expand All @@ -49,41 +67,15 @@ jobs:
Artifacts: ${{parameters.Artifacts}}
DocGenerationDir: "$(Build.SourcesDirectory)/doc/ApiDocGeneration"
LibType: 'client'
- job: "Analyze"
condition: ne(variables['Skip.Analyze'], true)
- job: VerifyGenerated
Copy link
Contributor

@mitchdenny mitchdenny Sep 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason why this can't be run within the Analyze job (on Windows)? Adding this job here causes a whole agent to be consumed.

Copy link
Contributor Author

@pakrym pakrym Sep 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Analyze runs after build completes and we would like to start the code check as early as possible because it's the longest leg of the build. In my initial change, I removed the Analyze job and moved all the steps to build to reduce the agent found but @weshaggard wanted to preserve it.

Copy link
Member

@weshaggard weshaggard Sep 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mitchdenny This is a pretty long running task which would be good to run in parallel with the build and test legs. I believe the goal was to have the Analyze job run after the build job which makes the overall pipeline run take longer then it should. The original reason was because we wanted to download and analyze the packages from the build job, however it looks like we aren't even downloading the packages any more and as such the Component Governance is not running on any packages beyond what is committed in the repo (yes there are still a couple of those :(). I just looked at our CG runs and the packages are present and it looks like they are getting analyzed in our dev publishing step instead.

Given that @pakrym can you just keep this in the Analyze job and just remove the Depends on Build from that job.

I've filed #15143 to follow-up on the CG stuff.

Copy link
Contributor Author

@pakrym pakrym Sep 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that @pakrym can you just keep this in the Analyze job and just remove the Depends on Build from that job.

CodeGeneration is significantly faster on ubuntu, that's part of the speedup.

Copy link
Member

@weshaggard weshaggard Sep 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with the move to ubuntu as well.

variables:
- template: ../variables/globals.yml
dependsOn:
- Build
pool:
vmImage: "windows-2019"
vmImage: ubuntu-18.04
steps:
- task: UsePythonVersion@0
displayName: "Use Python 3.6"
inputs:
versionSpec: "3.6"
- template: /eng/common/pipelines/templates/steps/verify-links.yml
parameters:
Directory: sdk/${{ parameters.ServiceDirectory }}
- script: |
pip install setuptools wheel
pip install doc-warden==$(DocWardenVersion)
ward scan -d $(Build.SourcesDirectory) -c $(Build.SourcesDirectory)/eng/.docsettings.yml
displayName: "Verify Readmes"
- template: /eng/common/pipelines/templates/steps/verify-path-length.yml
parameters:
SourceDirectory: $(Build.SourcesDirectory)
- task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
displayName: "Component Detection"
# ComponentGovernance is currently unable to run on pull requests of public projects. Running on
# scheduled builds should be good enough.
inputs:
ignoreDirectories: "sdk/storage/Azure.Storage.Common/swagger/Generator"
condition: and(succeededOrFailed(), ne(variables['Build.Reason'],'PullRequest'))
- task: NodeTool@0
inputs:
# AutoRest doesn't work with versions 12.x
versionSpec: '10.15.0'
versionSpec: '13.13.0'
displayName: 'Install NodeJS'
- task: PowerShell@2
displayName: "Verify generated code"
Expand Down
2 changes: 1 addition & 1 deletion eng/scripts/Export-API.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ param (

$servicesProj = Resolve-Path "$PSScriptRoot/../service.proj"

dotnet build /p:GenerateApiListingOnBuild=true /p:Configuration=Release /p:IncludeSamples=false /p:IncludeTests=false /p:Scope="$ServiceDirectory" /restore $servicesProj
dotnet build /p:GenerateApiListingOnBuild=true /p:SkipApiCompat=true /p:GeneratePackageOnBuild=false /p:Configuration=Release /p:IncludeSamples=false /p:IncludeTests=false /p:Scope="$ServiceDirectory" /restore $servicesProj
Copy link
Contributor Author

@pakrym pakrym Sep 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faster API export