Remove canary from default weekly live testing clouds

[benbp]

Resolves Azure/azure-sdk-tools#1781

@weshaggard related to the discussion in ^^, I think it adds too much complexity to have a separate "additional clouds" parameter. We already have a bunch of parameters like this where you can pick one to override the default, or the other to append to the default, and it trips people up from time to time. I think in this case given that the set is very static (it will only be changed if we add another sovereign cloud to Azure), the tradeoff is in favor of a single parameter. It also makes it clear at to the tests.yml reader which clouds will actually be tested vs. having to dig into the pipeline template to get the full set.

If this PR goes through on tests and reviews, I'll put ones up for the other repos.

[weshaggard]

So if I understand correctly individual tests.yml need to specify the full list of clouds in order to get the full coverage on the test-weekly runs?

[benbp]

So if I understand correctly individual tests.yml need to specify the full list of clouds in order to get the full coverage on the test-weekly runs?

Yes, it was set up this way to enable opt-in to sovereign cloud testing as support is added.

[weshaggard]

Is the name "SupportedClouds" the right one then? I wouldn't have inferred that just by the name. Perhaps providing a snippet here of what the configuration would look like to control the daily vs the weekly would be helpful.

[benbp]

Is the name "SupportedClouds" the right one then? I wouldn't have inferred that just by the name. Perhaps providing a snippet here of what the configuration would look like to control the daily vs the weekly would be helpful.

Sorry, I misspoke a little as I was just thinking about the weekly specific flag. For a weekly run it will check inclusion in either Clouds or SupportedClouds.

So to run weekly in ALL clouds, you could do:

Clouds: Public
SupportedClouds: Canary,UsGov,China

or more verbosely:

Clouds: Public
SupportedClouds: Public,Canary,UsGov,China

Or to run weekly for public/usgov/china but exclude canary:

UnsupportedClouds: Canary

[benbp]

Currently SupportedClouds defaults to Public,Canary, but I think the original intent was so that we could create all weekly pipelines in one effort, and then opt-in via config, as opposed to creating each new weekly pipeline as we added sovereign cloud support.

[weshaggard]

Ok no matter what we do here we should write the docs on how to configure this as part of the change so we can understand, as currently it is still a little unclear to how to figure out which will run for nightly vs weekly.

[benbp]

Ok no matter what we do here we should write the docs on how to configure this as part of the change so we can understand, as currently it is still a little unclear to how to figure out which will run for nightly vs weekly.

I have some documentation on this here and here.

We've chatted before about renaming these variables to something along the lines of Clouds and WeeklyPipelineClouds, but it's another area where it's a tedious amount of work for a low-touch area so I haven't prioritized it highly.

[weshaggard]

It might be worth adding some of those docs in the MD files in the repo instead of in the internal wiki.

[benbp]

/check-enforcer override