Skip to content
Merged
Show file tree
Hide file tree
Changes from 95 commits
Commits
Show all changes
98 commits
Select commit Hold shift + click to select a range
03e9f7c
preliminary changes to build
scbedd Oct 2, 2024
f6352dc
pulling across the improvements from original pipelinev3 branch
scbedd Oct 2, 2024
dccd0f9
fix the pullrequest caller for proper relative path
scbedd Oct 2, 2024
cf03790
is this better?
scbedd Oct 2, 2024
4c683df
handle nonexisting service when invoking for PR
scbedd Oct 2, 2024
2cb2a7b
script -> pwsh
scbedd Oct 2, 2024
6295825
fix rush runner
scbedd Oct 2, 2024
dd73653
swap from from to to when multiple services are provided
scbedd Oct 2, 2024
e750488
iterating across all the artifacts for a prdiff
scbedd Oct 2, 2024
725c6a7
working my way through the crashing build
scbedd Oct 3, 2024
733b020
we need to pass PackagePropertiesFolder not PackageInfoFolder
scbedd Oct 3, 2024
3bf5519
fix documentation run
scbedd Oct 3, 2024
61aee6d
need to pass the service to each one
scbedd Oct 3, 2024
cd78bea
$(Build.ArtifactStagingDirectory -> $(Build.ArtifactStagingDirectory)
scbedd Oct 3, 2024
b6d50de
artifactName -> artifact
scbedd Oct 3, 2024
b913f2e
pull across the artifact changes from the tools PR. use in analyze phase
scbedd Oct 10, 2024
6daa9b0
pass the package info folder in analyze
scbedd Oct 10, 2024
741a3c7
repair the calls to resolve-package-artifacts
scbedd Oct 10, 2024
f651fd1
update everywhere that we utilize ServiceDirectory with changedServic…
scbedd Oct 10, 2024
5cb6786
update analyze usage to pass changed services
scbedd Oct 10, 2024
d0f58a5
update to consume packageinfo instead of artifact list for set-artifa…
scbedd Oct 10, 2024
063520a
Merge branch 'main' into pipelinev3-v2
scbedd Oct 10, 2024
e52aba6
actually, pass it --only?
scbedd Oct 11, 2024
0d0034c
upate rush-runner to invoke --to when handling core changes, otherwis…
scbedd Oct 15, 2024
c5a6059
ensure that our new logic doesn't get activated in other service dire…
scbedd Oct 15, 2024
b5a6a65
add servicedirectoryfilter param
scbedd Oct 16, 2024
16fd9e1
fix issue with language-settings
scbedd Oct 16, 2024
da96d77
Merge branch 'main' into pipelinev3-v2
scbedd Oct 16, 2024
ece0dbf
we need to set the property of package-props. AdditionalPackages does…
scbedd Oct 16, 2024
1b23867
save ChangedServices dynamically
scbedd Oct 16, 2024
2626172
resolve changedServiced from the packageProperties files, instead of …
scbedd Oct 16, 2024
48cb9cf
resolve all the arguments here
scbedd Oct 16, 2024
ddba884
standardizing some of the code, we don't need two separate handles wh…
scbedd Oct 16, 2024
83464d4
update all of analyze for non-service-specific check
scbedd Oct 16, 2024
a3f2366
update this thing to allow non-sdk changes to still trigger properly
scbedd Oct 16, 2024
4722a54
changedServices -> $changedServices
scbedd Oct 16, 2024
4a7d646
ensure the argument is surrounded in quotes
scbedd Oct 16, 2024
b072794
fix spelling warnings
scbedd Oct 16, 2024
aa7643f
update step name
scbedd Oct 16, 2024
d0d2f1f
we weren't passing the correct package names
scbedd Oct 16, 2024
0ce2151
updating to handle the split between mgmt and not
scbedd Oct 18, 2024
07d47dc
merge main so we're not changing upstream anymore
scbedd Oct 18, 2024
1c6218b
flatten to what's in main
scbedd Oct 18, 2024
7398926
remove trailing space
scbedd Oct 18, 2024
d503e4b
fix ChangedServices on the fly
scbedd Oct 18, 2024
36140a4
remove unnecessary parameter ServiceDirectoryFilter it is hardcoded
scbedd Oct 18, 2024
2166d3b
try using double quotes instead of single, does that work?
scbedd Oct 22, 2024
01b5822
update to the runtime expression
scbedd Oct 22, 2024
37c1084
adjust the issue with verify-samples
scbedd Oct 22, 2024
599bf6e
CommaChangedServices def
scbedd Oct 22, 2024
53f5ef8
handle the specific --to vs --from requests
scbedd Oct 22, 2024
eb37ef5
ensure that we flatten to an actual array
scbedd Oct 22, 2024
8ecf061
get rid of usage of flat() in favor of simpler getPackagesWithDirection
scbedd Oct 22, 2024
2dc4112
we now split up --to vs --from
scbedd Oct 22, 2024
0805b60
directional updates
scbedd Oct 22, 2024
64f581e
clean up some comments and console logs from rush-runner.js
scbedd Oct 22, 2024
86f4dd6
clean up rushRunAllWithDirection
scbedd Oct 22, 2024
966ae33
Merge branch 'main' into pipelinev3-v2
scbedd Oct 22, 2024
b70cd1f
cleaning up
scbedd Oct 22, 2024
c69371b
we were snapping the targeted packages too soon!
scbedd Oct 23, 2024
e2ceb92
initial version of distributing packages to the matrix. still buggy f…
scbedd Oct 23, 2024
abbff35
add package distribution to the matrix
scbedd Oct 23, 2024
7dc60ba
splatting should be working. just need to ensure multiple batches are…
scbedd Oct 23, 2024
6adb684
ensure that we are able to scan for the packages
scbedd Oct 23, 2024
795a98a
restore accidentally changed matrix
scbedd Oct 23, 2024
3a975b6
update our service resolution logic to handle the ArtifactPackageName…
scbedd Oct 23, 2024
37d47de
ensure the cspell config is pulled across
scbedd Oct 23, 2024
f02bcf2
remove useless set-artifact-packages call. update variable reference
scbedd Oct 23, 2024
8a4fea9
Apply suggestions from code review
scbedd Oct 23, 2024
25c20ae
ensure we pass servicedirectory to the test job. resolve the issue wi…
scbedd Oct 23, 2024
ea53757
clean up changed service output
scbedd Oct 23, 2024
72b7401
actually remove the branch ref
scbedd Oct 25, 2024
191c749
rely on some heavier usage of common parlance
scbedd Oct 25, 2024
0f20711
we need PostGenerationScript to be optional
scbedd Oct 25, 2024
72db8de
simplify the matrix update
scbedd Oct 25, 2024
3af6f36
merge main to resolve merge conflict
scbedd Oct 25, 2024
94ba6b5
add some additional output
scbedd Oct 25, 2024
9aa8fac
dump the output right before using it
scbedd Oct 25, 2024
4f67fe6
remove useless check for engChanged when it is encapsulated by 'other…
scbedd Oct 25, 2024
2a0391e
add cmdlet binding
scbedd Oct 25, 2024
2996fdf
dramatically simplify the call to distribute packages to matrix
scbedd Oct 25, 2024
e75d156
and now that it will actually invoke the script, add back in the call…
scbedd Oct 25, 2024
1248bca
one attempt using no write-hosts or strict mode
scbedd Oct 25, 2024
5d831c9
remove weirdness...I think this was breaking somehow behind the scenes
scbedd Oct 25, 2024
3a09472
disable everything except a single write host
scbedd Oct 25, 2024
7b64344
now remove the Write-Host
scbedd Oct 25, 2024
16cd6b3
removing the write-host solved the problem. time to dump the variable…
scbedd Oct 25, 2024
cd50e71
try re-enabling and referencing a real environment setting
scbedd Oct 25, 2024
909c5b7
fix the batch save
scbedd Oct 26, 2024
e9e0e82
the parameter input MUST be a string, not an array[] type
scbedd Oct 26, 2024
1609ea3
merge main
scbedd Oct 31, 2024
cddda7f
now remove the distribute-packages-to-matrix
scbedd Oct 31, 2024
f0172ed
restore eng/common to main
scbedd Oct 31, 2024
0799806
fix the platform matrix back to main version
scbedd Oct 31, 2024
1c2891d
Merge branch 'main' into pipelinev3-v2
scbedd Oct 31, 2024
6dda23d
resolve the issue with the artifactpackagenames
scbedd Oct 31, 2024
d67e2f9
remove erroneously added audit step
scbedd Oct 31, 2024
37a4dbc
Merge branch 'main' into pipelinev3-v2
scbedd Oct 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .vscode/cspell.json
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,11 @@
"overrides": [
{
"filename": "eng/pipelines",
"words": ["azuresdkartifacts", "gdnbaselines", "policheck"]
"words": ["azuresdkartifacts", "gdnbaselines", "policheck", "issecret", "dlrw"]
},
{
"filename": "eng/tools/rush-runner.js",
"words": ["Jsons"]
},
{
"filename": "sdk/apimanagement/api-management-custom-widgets-scaffolder/review/api-management-custom-widgets-scaffolder.api.md",
Expand Down
7 changes: 7 additions & 0 deletions eng/pipelines/templates/jobs/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,9 +74,16 @@ jobs:
MatrixConfigs: ${{ parameters.MatrixConfigs }}
MatrixFilters: ${{ parameters.MatrixFilters }}
MatrixReplace: ${{ parameters.MatrixReplace }}
SparseCheckoutPaths: [ "sdk/", ".vscode"]
CloudConfig:
Cloud: Public
AdditionalParameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
Artifacts: ${{ parameters.Artifacts }}
TestProxy: ${{ parameters.TestProxy }}
${{ if eq(parameters.ServiceDirectory, 'auto') }}:
EnablePRGeneration: true
PreGenerationSteps:
- template: /eng/common/pipelines/templates/steps/save-package-properties.yml
parameters:
ServiceDirectory: ${{parameters.ServiceDirectory}}
44 changes: 25 additions & 19 deletions eng/pipelines/templates/steps/analyze.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,22 @@ parameters:
TestPipeline: false

steps:
- template: /eng/common/pipelines/templates/steps/save-package-properties.yml
parameters:
ServiceDirectory: ${{parameters.ServiceDirectory}}

- template: /eng/pipelines/templates/steps/set-artifact-packages.yml
parameters:
PackageInfo: $(Build.ArtifactStagingDirectory)/PackageInfo
Artifacts: ${{ parameters.Artifacts }}

- template: /eng/common/pipelines/templates/steps/check-spelling.yml

- task: PowerShell@2
inputs:
targetType: 'filePath'
filePath: eng/scripts/spell-check-public-api.ps1
arguments: -ServiceDirectory ${{ parameters.ServiceDirectory }}
filePath: eng/scripts/spell-check-public-apis.ps1
arguments: -ChangedServices "$(ChangedServices)"
pwsh: true
displayName: Spell check public API

Expand All @@ -26,10 +35,9 @@ steps:
ArtifactName: 'package-diffs'
SbomEnabled: false


- template: /eng/common/pipelines/templates/steps/verify-readme.yml
- template: /eng/common/pipelines/templates/steps/verify-readmes.yml
parameters:
ScanPath: $(Build.SourcesDirectory)/sdk/${{ parameters.ServiceDirectory }}
PackagePropertiesFolder: $(Build.ArtifactStagingDirectory)/PackageInfo

- template: /eng/common/pipelines/templates/steps/verify-path-length.yml
parameters:
Expand All @@ -46,7 +54,7 @@ steps:

- template: /eng/common/pipelines/templates/steps/verify-samples.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
ServiceDirectories: $(ChangedServicesCsv)

- script: |
npm ci
Expand All @@ -62,28 +70,26 @@ steps:
node common/scripts/install-run-rush.js install
displayName: "Install dependencies"

- template: /eng/pipelines/templates/steps/set-artifact-packages.yml
parameters:
Artifacts: ${{ parameters.Artifacts }}

- script: |
node eng/tools/rush-runner.js build "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
node eng/tools/rush-runner.js build $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Build libraries"

- template: /eng/pipelines/templates/steps/run-eslint.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}
ServiceDirectories: $(ChangedServices)

- pwsh: |
node eng/tools/rush-runner.js check-format "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose
node eng/tools/rush-runner.js check-format $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose
displayName: "Check Format in Libraries"

- ${{ each artifact in parameters.Artifacts }}:
- template: /eng/common/pipelines/templates/steps/verify-changelog.yml
parameters:
PackageName: ${{artifact.name}}
ServiceName: ${{parameters.ServiceDirectory}}
ForRelease: false
- script: |
node eng/tools/rush-runner.js audit $(ChangedServices) -packages "$(ArtifactPackageNames)"
condition: and(succeeded(), eq(variables['RunNpmAudit'], 'true'))
displayName: "Audit libraries"

- template: /eng/common/pipelines/templates/steps/verify-changelogs.yml
parameters:
PackagePropertiesFolder: $(Build.ArtifactStagingDirectory)/PackageInfo

- task: ms.vss-governance-buildtask.governance-build-task-component-detection.ComponentGovernanceComponentDetection@0
# ComponentGovernance is currently unable to run on pull requests of public projects. Running on non-PR
Expand Down
57 changes: 26 additions & 31 deletions eng/pipelines/templates/steps/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,32 +13,23 @@ steps:

- pwsh: |
$folder = "${{parameters.ServiceDirectory}}"
if ($folder -eq "*") { $folder = "" }
if ($folder -eq "*" -or $folder -eq "auto") { $folder = "" }
echo "##vso[task.setvariable variable=folder]$folder"
displayName: "Set folder variable for readme links"

# we are not passing service directory, so we only ever set dev build to true
- template: /eng/common/pipelines/templates/steps/daily-dev-build-variable.yml
parameters:
ServiceDirectory: ${{ parameters.ServiceDirectory }}

- script: |
npm install ./eng/tools/versioning
node eng/tools/versioning/set-dev.js --build-id "$(Build.BuildNumber)" --repo-root "$(Build.SourcesDirectory)" --service "$(folder)"
node eng/tools/versioning/set-dev.js --build-id "$(Build.BuildNumber)" --repo-root "$(Build.SourcesDirectory)"
node common/scripts/install-run-rush.js update
condition: and(succeeded(),eq(variables['SetDevVersion'],'true'))
displayName: "Update package versions for dev build"

- task: Powershell@2
inputs:
filePath: $(Build.SourcesDirectory)/eng/common/scripts/Save-Package-Properties.ps1
arguments: >
-ServiceDirectory ${{parameters.ServiceDirectory}}
-OutDirectory $(Build.ArtifactStagingDirectory)/PackageInfo
-AddDevVersion
pwsh: true
workingDirectory: $(Pipeline.Workspace)
displayName: Update package properties with dev version
condition: and(succeeded(),eq(variables['SetDevVersion'],'true'))
- template: /eng/common/pipelines/templates/steps/save-package-properties.yml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

save-package-properties.yml will call Save-Package-Properties.ps1 with appropriate -AddDevVersion by honoring the SetDevVersion set above in daily-dev-build-variable.yml

parameters:
ServiceDirectory: ${{parameters.ServiceDirectory}}

- script: |
node common/scripts/install-run-rush.js install
Expand All @@ -47,7 +38,9 @@ steps:

- template: /eng/pipelines/templates/steps/set-artifact-packages.yml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really the main thrust of the PR. PackageInfo folder becomes the set of targeted packages, versus honoring a hardcoded Artifacts list. The hardcoded artifacts will still be utilized in the release phases of the internal build definition.

parameters:
PackageInfo: $(Build.ArtifactStagingDirectory)/PackageInfo
Artifacts: ${{ parameters.Artifacts }}

- ${{ if and(eq(variables['System.TeamProject'], 'internal'), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI')) }}:
- task: AzureCLI@2
inputs:
Expand All @@ -60,26 +53,24 @@ steps:
sasToken=$(az storage container generate-sas --account-name azuresdkartifacts --name azure-sdk-for-js-rush-cache --permissions dlrw --auth-mode login --as-user --expiry $expiry --https-only -o tsv)
echo "##vso[task.setvariable variable=rushBuildCacheCred;issecret=true;]$sasToken"

# Option "-p max" ensures parallelism is set to the number of cores on all platforms, which improves build times.
# The default on Windows is "cores - 1" (microsoft/rushstack#436).
- script: |
node eng/tools/rush-runner.js build "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
- pwsh: |
node eng/tools/rush-runner.js build $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Build libraries"
env:
${{ if and(eq(variables['System.TeamProject'], 'internal'), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI')) }}:
RUSH_BUILD_CACHE_CREDENTIAL: $(rushBuildCacheCred)
RUSH_BUILD_CACHE_WRITE_ALLOWED: 1

- script: |
node eng/tools/rush-runner.js build:samples "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose
displayName: "Build samples"
node eng/tools/rush-runner.js build:samples "$(ChangedServices)" -packages "$(ArtifactPackageNames)" --verbose
displayName: "Build samples for PR"

- pwsh: |
eng/tools/check-api-warning.ps1
displayName: "Check api extractor output changes"

- script: |
node eng/tools/rush-runner.js pack "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose
node eng/tools/rush-runner.js pack "$(ChangedServices)" -packages "$(ArtifactPackageNames)" --verbose
displayName: "Pack libraries"

# Unlink node_modules folders to significantly improve performance of subsequent tasks
Expand All @@ -93,18 +84,22 @@ steps:
ServiceDirectory: ${{parameters.ServiceDirectory}}

- pwsh: |
$artifacts = '${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json
$artifacts = Get-ChildItem -Recurse -Force "$(Build.ArtifactStagingDirectory)/PackageInfo/*.json" `
| ForEach-Object { $_.Name.Replace(".json", "") }

foreach ($artifact in $artifacts)
{
$artifactName = $artifact.name
Write-Host "Copying $artifactName artifacts to $(Build.ArtifactStagingDirectory)/$artifactName"
New-Item -Type Directory -Name $artifactName -Path $(Build.ArtifactStagingDirectory) > $null
Copy-Item sdk/${{parameters.ServiceDirectory}}/**/$artifactName-[0-9]*.[0-9]*.[0-9]*.tgz $(Build.ArtifactStagingDirectory)/$artifactName
Copy-Item sdk/${{parameters.ServiceDirectory}}/**/browser/$artifactName-[0-9]*.[0-9]*.[0-9]*.zip $(Build.ArtifactStagingDirectory)/$artifactName
if ($${{ parameters.IncludeRelease }} -eq $true -and $artifact.skipPublishDocMs -ne $true)
$artifactDetails = Get-Content -Raw $(Build.ArtifactStagingDirectory)/PackageInfo/$artifact.json | ConvertFrom-Json

Write-Host "Copying $artifact artifacts to $(Build.ArtifactStagingDirectory)/$artifact"
New-Item -Type Directory -Force -Name $artifact -Path $(Build.ArtifactStagingDirectory) > $null
Copy-Item sdk/$($artifactDetails.Service)/**/$artifact-[0-9]*.[0-9]*.[0-9]*.tgz $(Build.ArtifactStagingDirectory)/$artifact
Copy-Item sdk/$($artifactDetails.Service)/**/browser/$artifact-[0-9]*.[0-9]*.[0-9]*.zip $(Build.ArtifactStagingDirectory)/$artifact

if ($${{ parameters.IncludeRelease }} -eq $true -and $artifactDetails.ArtifactDetails.skipPublishDocMs -ne $true)
{
New-Item -Type Directory -Name documentation -Path $(Build.ArtifactStagingDirectory)/$artifactName > $null
Copy-Item $(Build.SourcesDirectory)/docGen/$artifactName.zip $(Build.ArtifactStagingDirectory)/$artifactName/documentation
New-Item -Type Directory -Force -Name documentation -Path $(Build.ArtifactStagingDirectory)/$artifact > $null
Copy-Item $(Build.SourcesDirectory)/docGen/$artifact.zip $(Build.ArtifactStagingDirectory)/$artifact/documentation
}
}
displayName: 'Copy Packages'
Expand Down
12 changes: 11 additions & 1 deletion eng/pipelines/templates/steps/generate-doc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,17 @@ steps:
- pwsh: |
node $(Build.SourcesDirectory)/eng/tools/generate-doc/dist/index.js --serviceDir "${{parameters.ServiceDirectory}}"
displayName: "Run Typedoc Docs"
condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

- pwsh: |
if ('$(ChangedServices)' -and '$(ChangedServices)' -notlike '*ChangedServices*') {
foreach($service in '$(ChangedServices)'.Split(" ")) {
node $(Build.SourcesDirectory)/eng/tools/generate-doc/dist/index.js --serviceDir "$service"
}
}
displayName: "Run Typedocs for PR"
condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest'))

- pwsh: |
$(Build.SourcesDirectory)/eng/tools/compress-subfolders.ps1 "$(Build.SourcesDirectory)/docGen" "$(Build.SourcesDirectory)/docGen"
displayName: "Generate Typedoc Docs"
displayName: "Compress generated docs"
4 changes: 2 additions & 2 deletions eng/pipelines/templates/steps/run-eslint.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
parameters:
ServiceDirectory: ''
ServiceDirectories: ''

steps:
- script: |
Expand All @@ -8,5 +8,5 @@ steps:

- pwsh: |
node common/scripts/install-run-rush.js build -t @azure/eslint-plugin-azure-sdk -t @azure/monitor-opentelemetry-exporter
node eng/tools/rush-runner.js lint "${{parameters.ServiceDirectory}}" -p max
node eng/tools/rush-runner.js lint "${{parameters.ServiceDirectories}}" -p max
displayName: "Build ESLint Plugin and Lint Libraries"
51 changes: 45 additions & 6 deletions eng/pipelines/templates/steps/set-artifact-packages.yml
Original file line number Diff line number Diff line change
@@ -1,12 +1,51 @@
parameters:
PackageInfo: ''
Artifacts: []
steps:
# Package-Properties folder contains the package properties for all discovered packages that were either A) affected by the PR or
# B) explicitly present in the service directory. This repo splits the builds into two categories: "mgmt" and "dataplane". While
# a given directory may contain both, there will be a separate build definition to release the management packages. Due to this
# we need to artificially filter the packages to ensure that only the packages targeted for THAT ci.yml are built.
# When we merge the PR adding js - pullrequest, this code will also merge, meaning the public - <service> - ci builds will still operate
# EXACTLY the same as they did before the pipelinev3 change. This is important to ensure that we don't accidentally build the wrong packages
# while in the integration period. After we disable all the public `js - <service> - ci` builds, only the `js - pullrequest` build WONT provide
# an artifact list, which will allow the expand/contract. Meanwhile the internal builds will continue to provide the artifact list from their
# individual ci.yml and ci.mgmt.yml files.
- pwsh: |
$artifacts = '${{ convertToJson(parameters.Artifacts) }}' | ConvertFrom-Json
$packages = ""
foreach ($artifact in $artifacts)
{
$packages += "$($artifact.name),"
$knownArtifacts = @()
if ($artifacts) {
$knownArtifacts = $artifacts | ForEach-Object { $_.name }
}
echo "##vso[task.setvariable variable=ArtifactPackageNames]$packages"
displayName: "Find Packages to build"

$packageProperties = Get-ChildItem -Recurse "${{ parameters.PackageInfo }}" *.json `
| Where-Object { if($knownArtifacts) { $knownArtifacts -contains $_.Name.Replace(".json", "") } else { $true } }

$pkgNames = $packageProperties | ForEach-Object { $_.Name.Replace(".json", "") }

$setting = $pkgNames -join ","
Write-Host "Setting ArtifactPackageNames to: `n$setting"
Write-Host "##vso[task.setvariable variable=ArtifactPackageNames;]$setting"
displayName: Resolve Targeted Packages
condition: eq(variables['ArtifactPackageNames'], '')

- pwsh: |
# set changed services given the set of changed packages, this will mean that
# ChangedServices will be appropriate for the batched set of packages if that is indeed how
# we set the targeted artifacts
$packageProperties = Get-ChildItem -Recurse "${{ parameters.PackageInfo }}" *.json `
| Foreach-Object { Get-Content -Raw -Path $_.FullName | ConvertFrom-Json }

$packageSet = "$(ArtifactPackageNames)" -split ","

$changedServicesArray = $packageProperties | Where-Object { $packageSet -contains $_.ArtifactName }
| ForEach-Object { $_.ServiceDirectory } | Get-Unique

$changedServices = $changedServicesArray -join " "
$commaChangedServices = $changedServicesArray -join ","
Write-Host "##vso[task.setvariable variable=ChangedServices;]$changedServices"
Write-Host "##vso[task.setvariable variable=ChangedServicesCsv;]$commaChangedServices"

Write-Host "This run is targeting: `n$(ArtifactPackageNames) in [$changedServices]"
displayName: Resolve Targeted Packages
condition: ne(variables['ArtifactPackageNames'], '')
17 changes: 11 additions & 6 deletions eng/pipelines/templates/steps/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,42 +9,47 @@ steps:
parameters:
AgentImage: ${{ parameters.OSName}}

- template: /eng/common/pipelines/templates/steps/save-package-properties.yml
parameters:
ServiceDirectory: ${{parameters.ServiceDirectory}}

- script: |
node common/scripts/install-run-rush.js install
displayName: "Install dependencies"

- template: /eng/pipelines/templates/steps/set-artifact-packages.yml
parameters:
PackageInfo: $(Build.ArtifactStagingDirectory)/PackageInfo
Artifacts: ${{ parameters.Artifacts }}

# Option "-p max" ensures parallelism is set to the number of cores on all platforms, which improves build times.
# The default on Windows is "cores - 1" (microsoft/rushstack#436).
- script: |
node eng/tools/rush-runner.js build "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
node eng/tools/rush-runner.js build $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Build libraries"

# Option "-p max" ensures parallelism is set to the number of cores on all platforms, which improves build times.
# The default on Windows is "cores - 1" (microsoft/rushstack#436).
- script: |
node eng/tools/rush-runner.js build:test "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
node eng/tools/rush-runner.js build:test $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Build test assets"

- template: ../steps/use-node-test-version.yml

- ${{ if eq(parameters.TestProxy, true) }}:
- template: /eng/common/testproxy/test-proxy-standalone-tool.yml

# Option "-p max" ensures parallelism is set to the number of cores on all platforms, which improves build times.
# The default on Windows is "cores - 1" (microsoft/rushstack#436).
- script: |
node eng/tools/rush-runner.js unit-test:node "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
node eng/tools/rush-runner.js unit-test:node $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Test libraries"
condition: and(succeeded(),eq(variables['TestType'], 'node'))

# Option "-p max" ensures parallelism is set to the number of cores on all platforms, which improves build times.
# The default on Windows is "cores - 1" (microsoft/rushstack#436).
- script: |
node eng/tools/rush-runner.js unit-test:browser "${{parameters.ServiceDirectory}}" -packages "$(ArtifactPackageNames)" --verbose -p max
node eng/tools/rush-runner.js unit-test:browser $(ChangedServices) -packages "$(ArtifactPackageNames)" --verbose -p max
displayName: "Test libraries"
condition: and(succeeded(),eq(variables['TestType'], 'browser'))

Expand All @@ -53,7 +58,7 @@ steps:
copy $(Build.SourcesDirectory)/test-proxy.log $(Build.ArtifactStagingDirectory)
displayName: 'Dump Test Proxy logs'
condition: succeededOrFailed()

# Unlink node_modules folders to significantly improve performance of subsequent tasks
# which need to walk the directory tree (and are hardcoded to follow symlinks).
# Retry for 30 seconds, since this command may fail with error "Another rush command is already
Expand Down
Loading
Loading