Bootstrap test resources when running tests#23652
Conversation
JoshLove-msft
requested review from
KrzysztofCwalina,
chidozieononiwu,
danieljurek,
pakrym and
weshaggard
as code owners
ghost
added
the
There was a problem hiding this comment.
Why do we need these in Get* methods?
There was a problem hiding this comment.
How else would we know if bootstrapping is required?
There was a problem hiding this comment.
Check that RG exists. Would handle expiring resources as well as non-existant.
There was a problem hiding this comment.
That doesn't work - you could have an existent RG and still not have the resources.
There was a problem hiding this comment.
You can have an existing variable and no resource as well.
There was a problem hiding this comment.
The concern that I have is the experience for external contributors. If I'm trying to get myself running and I encounter this because I'm on Linux or am using the documented/supported approach of environment variables on Windows, the experience isn't awesome. Will I bother to let you know or will I just get frustrated and give up without a word?
I'd prefer that we optimize for not prompting unless we can prove that the environment isn't suitable to run, rather than optimizing for the common use case of the SDK team.
There was a problem hiding this comment.
The prompt will only happen on Windows.
There was a problem hiding this comment.
Fair enough, but I think the overall concern stands.
There was a problem hiding this comment.
Yep, if you are using env vars you may find the behavior annoying if we switch to check for the existence of an env file.
There was a problem hiding this comment.
If the env vars aren't detected (or the file not found - normal behavior now) why not just point them at the instructions on how to run New-TestResources? Adding extra time to check all the time for resources existing will slow tests down even further and some of them cost by the hour. You also can't discern much from incremental deployments. Some resources might support it, while others don't because of the configuration they are in - like resources configured for Customer-Managed Keys (CMK) like sdk/search. You can't redeploy resources in that state incrementally - it messes everything up.
|
Can we plug this feature in somewhere around ? |
We don't know whether to bootstrap until the tests start running. |
What are the scenarios where we don't know? |
Someone has a resouce group env var that is unrelated to the current test suite. The Test Framework thinks the resources are already there. |
Pretty sure we are safe there https://azure.grep.app/search?q=RESOURCE_GROUP&case=true&filter[repo][0]=Azure/azure-sdk-for-net&filter[lang][0]=C%23 Checking for RG also allows us to extend resource expiration time and check for expired resources. I don't think we should overcorrect for people who are advanced enough to be setting env vars. |
Or it should be a parameter to the existing script. |
Did you mean to respond to this comment - #23652 (comment) |
Yep |
There was a problem hiding this comment.
Is this wrapper even useful? Is it only so you can do the prompting? If so can that be done in the test run context?
There was a problem hiding this comment.
Yes, it is only for the prompting.
I'm not aware of a way to provide an interactive prompt within the test run context.
There was a problem hiding this comment.
If you are not opposed to adding this directly to New-TestResources along with a new switch param, I can do that instead.
There was a problem hiding this comment.
I like keeping it separate for now and I also like it is in a .NET specific script instead of under eng/common.
|
This pull request is protected by Check Enforcer. What is Check Enforcer?Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass. Why am I getting this message?You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged. What should I do now?If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows: What if I am onboarding a new service?Often, new services do not have validation pipelines associated with them. In order to bootstrap pipelines for a new service, please perform following steps: For data-plane/track 2 SDKs Issue the following command as a pull request comment:
For track 1 management-plane SDKsPlease open a separate PR and to your service SDK path in this file. Once that PR has been merged, you can re-run the pipeline to trigger the verification. |
There was a problem hiding this comment.
RuntimeInformation.IsOsPlatform is not available on net461 or net462. (ref)
Should we consider using (or falling back to) Envvironment.OSVersion?
There was a problem hiding this comment.
Hmm, I'm not sure how this builds then.
There was a problem hiding this comment.
Because we usually don't build tests for those frameworks...but Key Vault because there are differences we need to test for.
JoshLove-msft
requested review from
ArcturusZhang,
allenjzhang,
archerzz and
cijothomas
as code owners
JoshLove-msft
removed request for
TimothyMothra,
chidozieononiwu,
fengzhou-msft,
m-nash,
reyang,
serkantkaraca,
vishweshbankwar and
yungezz
JoshLove-msft
force-pushed
the
tf-bootstrap
branch
from
4bc3037 to
7bc5784
Compare
|
Let's try this option as it's a bit safer and see how it goes. |
| @@ -0,0 +1,12 @@ | |||
| # Copyright (c) Microsoft Corporation. All rights reserved. | |||
There was a problem hiding this comment.
Why is this not in Azure/azure-sdk-tools?
There was a problem hiding this comment.
This is only going to be used by the .NET test suite.
There was a problem hiding this comment.
Because we usually don't build tests for those frameworks...but Key Vault because there are differences we need to test for.
There was a problem hiding this comment.
If the env vars aren't detected (or the file not found - normal behavior now) why not just point them at the instructions on how to run New-TestResources? Adding extra time to check all the time for resources existing will slow tests down even further and some of them cost by the hour. You also can't discern much from incremental deployments. Some resources might support it, while others don't because of the configuration they are in - like resources configured for Customer-Managed Keys (CMK) like sdk/search. You can't redeploy resources in that state incrementally - it messes everything up.
| "Make sure the test environment was initialized using the eng/common/TestResources/New-TestResources.ps1 script."; | ||
| if (_bootstrappingException != null) | ||
| { | ||
| message += Environment.NewLine + "Resource creation failed during the test run. Make sure PowerShell version 6 or higher is installed."; |
There was a problem hiding this comment.
Why PowerShell 6? Seems like tight coupling with something else, much like checking if this is Windows. If using env vars, why wouldn't other platforms work?
There was a problem hiding this comment.
It says at least powershell 6. This is the same messaging we use in the contributing.md file.
| s_bootstrappingAttemptedTypes.Contains(_type) || | ||
| Mode == RecordedTestMode.Playback || | ||
| GlobalIsRunningInCI) |
There was a problem hiding this comment.
Nit: I'd reorder these with faster checks first - basically, reverse these last three.
There was a problem hiding this comment.
GlobalIsRunningInCI does an environment variable lookup whenever invoked.
I know these are cached for the process, but isn't it still a relatively expensive system call?
|
@heaths replying here as I wasn't able to reply to your comments for some reason:
So why did the core - ci pass? If KeyVault tests run against those frameworks shouldn't they have failed?
This is the current behavior. The idea is to remove the step of manually executing the script and passing the service name.
After the initial prompt to run the script, there should be minimal perf impact - we are just checking a local bool and returning.
If you are dealing with a service like that you can just say that you won't run the script, or when you get to the prompt about an existing resource group, you can just say not to update it. |
In my experience working with partners who've had trouble running this script, doing it automatically won't find much success. First, you need to have permissions to deploy to a sub - something we're fortunate to have but many don't. Even if we went back to the same, slow, user-unfriendly method of documenting non-copy/pastable Azure CLI commands the same is still true. They have to make sure they are logged into the right sub, have the right permissions (might require a sub admin giving them an SP they can use, such as the case with one internal partner), etc. While I understand this PR doesn't fully automate it, I think doing so much cause more confusion if the user isn't at least involved in interactively selecting the right sub they want to use, etc. For example, |
|
I agree that there the default experience is not always ideal but this feature doesn't make it worse. If users don't have access to create resources both running the script manually or automatically would fail. We are just doing exactly what we are telling customers to do. I agree we should improve the sub selection (make it interactive?) but it's a bit orthogonal. |
|
Alright, I am going to merge this now. We can keep an eye out for any issues that users experience, and adjust as needed. |
6 participants
No description provided.