Stress development maturity updates #3481

benbp · 2022-06-21T18:09:23Z

Adds a flag -LocalAddonsPath to the stress deploy script to use a local stress-test-addons instead of the remote one. Currently this has to be updated manually in Chart.yaml and is easy to forget to do.
Adds a -WhatIf flag to the provision script to do bicep and helm template validation.
Adds a pipeline to deploy the stress test examples and validate buildout actions on merge to main and nightly (it can't be run on PR since it uses deployment secrets).
Adds a pipeline to build out/provision a stress cluster environment.
Fixes an issue where the deployId is the same for all local users, so anyone deploying stress tests locally may see other users deployed tests listed in the output.
Fixes a minor bug with the network chaos example.
Fixes an unrelated issue in the powershell test pipeline that has the images mixed up.

azure-sdk · 2022-06-21T18:14:41Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

azure-sdk · 2022-06-21T20:25:07Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

azure-sdk · 2022-06-21T20:42:02Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

azure-sdk · 2022-06-21T21:01:42Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

eng/pipelines/templates/stages/archetype-sdk-tool-pwsh.yml

benbp · 2022-06-22T17:56:11Z

The pipeline is currently failing due to permissions on the service principal that fail validation. I am following up on this.

azure-sdk · 2022-06-22T17:56:43Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

benbp · 2022-06-22T17:57:07Z

FYI @richardpark-msft

benbp · 2022-06-22T20:28:59Z

Pipelines will be failing until I can rebase with @ckairen's changes to deploy to the new playground cluster which has been migrated to the latest versions of our cluster dependencies. Everything is looking as I expect it to otherwise.

azure-sdk · 2022-06-22T20:29:04Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

eng/common/scripts/stress-testing/stress-test-deployment-lib.ps1

ckairen · 2022-06-22T23:50:13Z

eng/pipelines/stress-cluster-provision.yml

+    default: test
+    values:
+    - prod
+    - test


Should it be pg?

Yes I was planning on rebasing/updating after your changes went in.

eng/pipelines/templates/jobs/stress-test-release.yml

tools/stress-cluster/cluster/provision.ps1

ckairen · 2022-06-23T00:02:53Z

tools/stress-cluster/cluster/provision.ps1

+$WhatIfPreference = $false
+. (Join-Path $PSScriptRoot "../../../eng/common/scripts/Helpers" PSModule-Helpers.ps1)
+Install-ModuleIfNotInstalled "powershell-yaml" "0.4.1" | Import-Module
+$WhatIfPreference = $LastWhatIfPreference


Don't quite understand why we are changing WhatIfPreference to false and then to original value again

It's somewhat annoying. There are sub-components called within Install-ModuleIfNotInstalled that don't run when -WhatIf is set that we absolutely need to run, but because the top level cmdlet doesn't have the SupportsShouldProcess binding, I don't think I can just pass -WhatIf:$false. We need this dependency installed whether we are in whatif mode or not.

Same for Import-Module. It doesn't support -WhatIf but if Install-ModuleIfNotInstalled doesn't return a module, then we throw.

Can you explicitly pass -WhatIf:$false to those cmdlets instead of mucking around with the global variable?

I updated the underlying function to add SupportsShouldProcess, so it should work now.

tools/stress-cluster/cluster/provision.ps1

azure-sdk · 2022-06-23T01:21:07Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

weshaggard · 2022-06-23T21:41:23Z

eng/pipelines/stress-cluster-provision.yml

+    - prod
+    - test
+  - name: WhatIf
+    type: boolean


This will be an interesting pattern. If you find that it works out OK can you please write up some guideline docs about this pattern.

Yep can do. Plumbing WhatIf support through all our scripts will be pretty useful for general testing and development, I think. Similarly we could run a lot more validation pipelines/scripts on PR changes if we had WhatIf enabled to catch surface level issues.

weshaggard

Took a high-level look at add a couple comments but otherwise seems reasonable.

azure-sdk · 2022-06-23T22:00:33Z

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

ghost · 2022-06-24T18:24:09Z

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.

benbp added 4 commits June 21, 2022 11:49

Support local addons path override in stress test deployment

fe43088

Support username based deployId in local stress deployment

c9c05b1

Fix network chaos example

577b5fb

Add stress test example release ci

e642009

Support WhatIf in stress infrastructure provision script

090651a

Fix image ref issues in powershell template

6e77c36

benbp force-pushed the benbp/stress-addon-dev branch from c4070ed to 50c61d7 Compare June 21, 2022 20:34

benbp force-pushed the benbp/stress-addon-dev branch from 50c61d7 to 9106a65 Compare June 21, 2022 20:54

benbp commented Jun 22, 2022

View reviewed changes

eng/pipelines/templates/stages/archetype-sdk-tool-pwsh.yml Show resolved Hide resolved

Add stress cluster provision and validation pipeline

8cb8610

benbp force-pushed the benbp/stress-addon-dev branch from 9106a65 to 8cb8610 Compare June 22, 2022 17:51

benbp marked this pull request as ready for review June 22, 2022 17:52

benbp requested review from a team, ckairen, sima-zhu and weshaggard as code owners June 22, 2022 17:52

benbp self-assigned this Jun 22, 2022

benbp added Central-EngSys This issue is owned by the Engineering System team. Stress This issue is related to stress testing, part of our reliability pillar. labels Jun 22, 2022

Simplify provision pipeline yaml and enable sub config/sp auth

84e82c0

Fix chart.lock local reference in stress example

e63b00c

ckairen reviewed Jun 22, 2022

View reviewed changes

eng/common/scripts/stress-testing/stress-test-deployment-lib.ps1 Show resolved Hide resolved

ckairen reviewed Jun 22, 2022

View reviewed changes

eng/pipelines/templates/jobs/stress-test-release.yml Outdated Show resolved Hide resolved

ckairen reviewed Jun 23, 2022

View reviewed changes

tools/stress-cluster/cluster/provision.ps1 Outdated Show resolved Hide resolved

ckairen reviewed Jun 23, 2022

View reviewed changes

tools/stress-cluster/cluster/provision.ps1 Show resolved Hide resolved

benbp added 3 commits June 22, 2022 21:16

Run helm plugin add with helper

6f3aa65

Use mms pool for stress test release

fc4bb32

Fix default parameter set for stress cluster provision script

1b8ec23

benbp force-pushed the benbp/stress-addon-dev branch from e2a75ed to 1b8ec23 Compare June 23, 2022 01:16

weshaggard reviewed Jun 23, 2022

View reviewed changes

weshaggard approved these changes Jun 23, 2022

View reviewed changes

ckairen approved these changes Jun 23, 2022

View reviewed changes

Add WhatIf support to ps module install helper function

7b46f3b

azure-sdk added the auto-merge label Jun 24, 2022

benbp merged commit 552b521 into Azure:main Jun 24, 2022

benbp deleted the benbp/stress-addon-dev branch June 24, 2022 18:24

benbp mentioned this pull request Jun 29, 2022

Stress Test Release pipeline should deploy examples to playground environment #2635

Closed

Stress development maturity updates #3481

Stress development maturity updates #3481

Uh oh!

Conversation

benbp commented Jun 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azure-sdk commented Jun 21, 2022

Uh oh!

azure-sdk commented Jun 21, 2022

Uh oh!

azure-sdk commented Jun 21, 2022

Uh oh!

azure-sdk commented Jun 21, 2022

Uh oh!

Uh oh!

benbp commented Jun 22, 2022

Uh oh!

azure-sdk commented Jun 22, 2022

Uh oh!

benbp commented Jun 22, 2022

Uh oh!

benbp commented Jun 22, 2022

Uh oh!

azure-sdk commented Jun 22, 2022

Uh oh!

Uh oh!

ckairen Jun 22, 2022

Choose a reason for hiding this comment

Uh oh!

benbp Jun 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ckairen Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

benbp Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

benbp Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

weshaggard Jun 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

benbp Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

azure-sdk commented Jun 23, 2022

Uh oh!

weshaggard Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

benbp Jun 23, 2022

Choose a reason for hiding this comment

Uh oh!

weshaggard left a comment

Choose a reason for hiding this comment

Uh oh!

azure-sdk commented Jun 23, 2022

Uh oh!

ghost commented Jun 24, 2022

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benbp commented Jun 21, 2022 •

edited

Loading

benbp Jun 22, 2022 •

edited

Loading

weshaggard Jun 23, 2022 •

edited

Loading

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (`@msftbot`) and give me an instruction to get started! Learn more here.