-
Notifications
You must be signed in to change notification settings - Fork 1.4k
[Perf] Refactor tests.yml into one file per package per language #24398
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
43e5b1d
Add perf-tests.yml
mikeharder 39bfe04
Merge branch 'main' into perf-refactoring
mikeharder e6dd9bf
Merge branch 'main' into perf-refactoring
mikeharder af37d2f
Remove "Services" parameter
mikeharder 713ff13
Merge branch 'main' into perf-refactoring
mikeharder 5288cb3
Add LanguageVersion parameter
mikeharder ef76a17
Merge branch 'main' into perf-refactoring
mikeharder 6b95ece
Merge branch 'main' into perf-refactoring
mikeharder 7b0d840
Merge branch 'main' into perf-refactoring
mikeharder c82e00d
Merge branch 'main' into perf-refactoring
mikeharder 3b7fd77
Update perf.yml, add perf-tests.yml
mikeharder 20a05a9
Add displayName to LanguageVersion
mikeharder File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| Service: eventhubs | ||
|
|
||
| Project: sdk/eventhub/perf-tests/event-hubs | ||
|
|
||
| PrimaryPackage: '@azure/event-hubs' | ||
|
|
||
| PackageVersions: | ||
| - '@azure/event-hubs': 5.8.0 | ||
| '@azure/core-amqp': 3.2.0 | ||
| '@azure/core-auth': 1.4.0 | ||
| '@azure/core-http': 2.3.1 | ||
| '@azure/core-rest-pipeline': 1.10.0 | ||
| '@azure/core-tracing': 1.0.1 | ||
| '@azure/core-util': 1.1.1 | ||
| '@azure/abort-controller': 1.1.0 | ||
| '@azure/logger': 1.0.3 | ||
| - '@azure/event-hubs': source | ||
| '@azure/core-amqp': source | ||
| '@azure/core-auth': source | ||
| '@azure/core-http': source | ||
| '@azure/core-rest-pipeline': source | ||
| '@azure/core-tracing': source | ||
| '@azure/core-util': source | ||
| '@azure/abort-controller': source | ||
| '@azure/logger': source | ||
|
|
||
| Tests: | ||
| - Test: send | ||
| Class: SendTest | ||
| Arguments: | ||
| - --event-size 1024 --batch-size 100 --parallel 64 | ||
|
|
||
| - Test: subscribe | ||
| Class: SubscribeTest | ||
| Arguments: | ||
| - --event-size 1024 --max-batch-size 100 --events 1000000 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| Service: keyvault-secrets | ||
|
|
||
| Project: sdk/keyvault/perf-tests/keyvault-secrets | ||
|
|
||
| PrimaryPackage: '@azure/storage-file-datalake' | ||
|
|
||
| PackageVersions: | ||
| - '@azure/keyvault-secrets': 4.3.0 | ||
| - '@azure/keyvault-secrets': source | ||
|
|
||
| Tests: | ||
| - Test: get-secret | ||
| Class: GetSecretTest | ||
| Arguments: | ||
| - --parallel 64 | ||
|
|
||
| - Test: list-secrets | ||
| Class: ListSecretsTest | ||
| Arguments: | ||
| - --count 5 --parallel 64 | ||
| - --count 500 --parallel 32 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| Service: servicebus | ||
|
|
||
| Project: sdk/servicebus/perf-tests/service-bus | ||
|
|
||
| PrimaryPackage: '@azure/service-bus' | ||
|
|
||
| PackageVersions: | ||
| - '@azure/service-bus': 7.7.3 | ||
| '@azure/core-amqp': 3.2.1 | ||
| '@azure/core-auth': 1.4.0 | ||
| '@azure/core-client': 1.6.1 | ||
| '@azure/core-http': 2.3.1 | ||
| '@azure/core-paging': 1.4.0 | ||
| '@azure/core-rest-pipeline': 1.10.0 | ||
| '@azure/core-tracing': 1.0.1 | ||
| '@azure/core-util': 1.1.1 | ||
| '@azure/core-xml': 1.3.0 | ||
| '@azure/abort-controller': 1.1.0 | ||
| '@azure/logger': 1.0.3 | ||
| - '@azure/service-bus': source | ||
| '@azure/core-amqp': source | ||
| '@azure/core-auth': source | ||
| '@azure/core-client': source | ||
| '@azure/core-http': source | ||
| '@azure/core-paging': source | ||
| '@azure/core-rest-pipeline': source | ||
| '@azure/core-tracing': source | ||
| '@azure/core-util': source | ||
| '@azure/core-xml': source | ||
| '@azure/abort-controller': source | ||
| '@azure/logger': source | ||
|
|
||
| Tests: | ||
| - Test: send-event-batch | ||
| Class: BatchSendTest | ||
| Arguments: | ||
| - --size 10240 --batch-size 10 | ||
|
|
||
| - Test: subscribe | ||
| Class: SubscribeTest | ||
| Arguments: | ||
| - --message-size 2000 --messages 10000 --mcc 1000 | ||
|
|
||
| - Test: receive-batch | ||
| Class: BatchReceiveTest | ||
| Arguments: | ||
| - --number-of-messages 10000 --message-size 2000 --max-message-count 50 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,37 +1,41 @@ | ||
| parameters: | ||
| - name: operatingSystems | ||
| displayName: OperatingSystems (list of operating systems to run) | ||
| type: string | ||
| default: 'Linux' | ||
| - name: packageVersions | ||
| displayName: PackageVersions (regex of package versions to run) | ||
| type: string | ||
| default: '7|source' | ||
| - name: tests | ||
| displayName: Tests (regex of tests to run) | ||
| type: string | ||
| default: '^(send-event-batch|subscribe|receive-batch)$' | ||
| - name: arguments | ||
| displayName: Arguments (regex of arguments to run) | ||
| type: string | ||
| default: '.*' | ||
| - name: iterations | ||
| displayName: Iterations (times to run each test) | ||
| type: number | ||
| default: '5' | ||
| - name: additionalArguments | ||
| displayName: AdditionalArguments (passed to PerfAutomation) | ||
| type: string | ||
| default: ' ' | ||
| - name: OperatingSystems | ||
| displayName: OperatingSystems (list of operating systems to run) | ||
| type: string | ||
| default: 'Linux' | ||
| - name: LanguageVersion | ||
| displayName: LanguageVersion (14, 16, 18) | ||
| type: string | ||
| default: '18' | ||
| - name: PackageVersions | ||
| displayName: PackageVersions (regex of package versions to run) | ||
| type: string | ||
| default: '7|source' | ||
| - name: Tests | ||
| displayName: Tests (regex of tests to run) | ||
| type: string | ||
| default: '^(send-event-batch|subscribe|receive-batch)$' | ||
| - name: Arguments | ||
| displayName: Arguments (regex of arguments to run) | ||
| type: string | ||
| default: '.*' | ||
| - name: Iterations | ||
| displayName: Iterations (times to run each test) | ||
| type: number | ||
| default: '5' | ||
| - name: AdditionalArguments | ||
| displayName: AdditionalArguments (passed to PerfAutomation) | ||
| type: string | ||
| default: ' ' | ||
|
|
||
| extends: | ||
| template: /eng/pipelines/templates/jobs/perf.yml | ||
| parameters: | ||
| operatingSystems: ${{ parameters.operatingSystems }} | ||
| serviceDirectory: servicebus/service-bus | ||
| services: "^servicebus$" | ||
| packageVersions: ${{ parameters.packageVersions }} | ||
| tests: ${{ parameters.tests }} | ||
| arguments: ${{ parameters.arguments }} | ||
| iterations: ${{ parameters.iterations }} | ||
| additionalArguments: ${{ parameters.additionalArguments }} | ||
| OperatingSystems: ${{ parameters.operatingSystems }} | ||
| ServiceDirectory: servicebus/service-bus | ||
| LanguageVersion: ${{ parameters.LanguageVersion }} | ||
| PackageVersions: ${{ parameters.packageVersions }} | ||
| Tests: ${{ parameters.tests }} | ||
| Arguments: ${{ parameters.arguments }} | ||
| Iterations: ${{ parameters.iterations }} | ||
| AdditionalArguments: ${{ parameters.additionalArguments }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| Service: storage-blob | ||
|
|
||
| Project: sdk/storage/perf-tests/storage-blob | ||
|
|
||
| PrimaryPackage: '@azure/storage-blob' | ||
|
|
||
| PackageVersions: | ||
| - '@azure/storage-blob': 12.12.0 | ||
| '@azure/core-auth': 1.4.0 | ||
| '@azure/core-http': 2.3.1 | ||
| '@azure/core-lro': 2.4.0 | ||
| '@azure/core-paging': 1.4.0 | ||
| '@azure/core-rest-pipeline': 1.10.0 | ||
| '@azure/core-tracing': 1.0.1 | ||
| '@azure/core-util': 1.1.1 | ||
| '@azure/abort-controller': 1.1.0 | ||
| '@azure/logger': 1.0.3 | ||
| - '@azure/storage-blob': source | ||
| '@azure/core-auth': source | ||
| '@azure/core-http': source | ||
| '@azure/core-lro': source | ||
| '@azure/core-paging': source | ||
| '@azure/core-rest-pipeline': source | ||
| '@azure/core-tracing': source | ||
| '@azure/core-util': source | ||
| '@azure/abort-controller': source | ||
| '@azure/logger': source | ||
|
|
||
| Tests: | ||
| - Test: download | ||
| Class: StorageBlobDownloadTest | ||
| Arguments: | ||
| # TODO: Remove before merging | ||
| - --size 1 --warmup 0 --duration 1 | ||
| - --size 10240 --parallel 64 | ||
| - --size 10485760 --parallel 32 | ||
| - --size 1073741824 --parallel 1 --warmup 60 --duration 60 | ||
| - --size 1073741824 --parallel 8 --warmup 60 --duration 60 | ||
|
|
||
| - Test: upload | ||
| Class: StorageBlobUploadTest | ||
| Arguments: | ||
| # TODO: Remove before merging | ||
| - --size 1 --warmup 0 --duration 1 | ||
| - --size 10240 --parallel 64 | ||
| - --size 10485760 --parallel 32 | ||
| - --size 1073741824 --parallel 1 --warmup 60 --duration 60 | ||
| - --size 1073741824 --parallel 8 --warmup 60 --duration 60 | ||
|
|
||
| - Test: list-blobs | ||
| Class: StorageBlobListTest | ||
| Arguments: | ||
| # TODO: Remove before merging | ||
| - --count 1 --warmup 0 --duration 1 | ||
| - --count 5 --parallel 64 | ||
| - --count 500 --parallel 32 | ||
| - --count 50000 --parallel 32 --warmup 60 --duration 60 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| Service: storage-file-datalake | ||
|
|
||
| Project: sdk/storage/perf-tests/storage-file-datalake | ||
|
|
||
| PrimaryPackage: '@azure/storage-file-datalake' | ||
|
|
||
| PackageVersions: | ||
| - '@azure/storage-file-datalake': 12.11.0 | ||
| '@azure/storage-blob': 12.12.0 | ||
| '@azure/core-auth': 1.4.0 | ||
| '@azure/core-http': 2.3.1 | ||
| '@azure/core-lro': 2.4.0 | ||
| '@azure/core-paging': 1.3.0 | ||
| '@azure/core-rest-pipeline': 1.10.0 | ||
| '@azure/core-tracing': 1.0.1 | ||
| '@azure/core-util': 1.1.1 | ||
| '@azure/abort-controller': 1.1.0 | ||
| '@azure/logger': 1.0.3 | ||
| - '@azure/storage-file-datalake': source | ||
| '@azure/storage-blob': source | ||
| '@azure/core-auth': source | ||
| '@azure/core-http': source | ||
| '@azure/core-lro': source | ||
| '@azure/core-paging': source | ||
| '@azure/core-rest-pipeline': source | ||
| '@azure/core-tracing': source | ||
| '@azure/core-util': source | ||
| '@azure/abort-controller': source | ||
| '@azure/logger': source | ||
|
|
||
| Tests: | ||
| - Test: download | ||
| Class: StorageDFSReadTest | ||
| Arguments: &sizes | ||
| - --size 10240 --parallel 64 | ||
| - --size 10485760 --parallel 32 | ||
| - --size 1073741824 --parallel 1 --warmup 60 --duration 60 | ||
| - --size 1073741824 --parallel 8 --warmup 60 --duration 60 | ||
|
|
||
| - Test: upload | ||
| Class: StorageDFSUploadTest | ||
| Arguments: *sizes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file should also be taken care of by the post-release automation bot PR.
Whenever a new version of a package is released, it should update this file in the same PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think updating the baseline versions should stay a manual process for now. We should only update baseline once we have confirmed there are no regressions in the new package version (or main, assuming main is close to the latest package released).