Consolidate live test cloud/platform configuration, use stages, support weekly tests

[benbp]

The overarching intent is to give us more control over the test matrix, and enable us to consolidate as much live test pipeline yaml as possible into azure-sdk-tools. Right now there are multiple different solutions across the language repos that could all be consolidated into one place. This will make it easier to roll out testing support for sovereign clouds, etc. across all the languages in the future. The changes here outline the proposed approach going forward for all live test pipelines.

This PR aims to achieve the above via the following:

1. Simplify the cloud and platform configuration
   • Offload arm parameters and environment variables into the keyvault subscription configuration. This will prevent future duplication and also simplifies the pipeline
   • Make it easier to specify which clouds a pipeline should run (in the case of a daily trigger, which defaults to public clouds), and which clouds a pipeline should not run (in the case of a weekly trigger, which defaults to all clouds).
   • Enable easier definition of a sparse matrix (i.e. which platforms run on which clouds).
2. Remove as many if conditionals as possible, such as TestCanary, weekly test, etc. When possible, we should prefer to check these values only in a single place instead of littering the template with if conditionals, to reduce complexity.
3. Split out cloud/platform configuration and test yaml specific to a language as much as possible from the rest of the pipeline logic, so that the latter can be relocated to azure-sdk-tools/eng/common.

I've also pulled in the changes from Azure/azure-sdk-tools#1163 in order to support the new subscription configuration schema while keeping the PR pipelines working. This should be merged soon and then I'll rebase.

The new subscription configuration schema looks like:

{
  "SubscriptionId": "<subscription id>",
  "ProvisionerApplicationId": "<provisioner app id>",
  "ProvisionerApplicationSecret": "<provisioner app secret>",
  "Environment": "AzureCloud | AzureGov | AzureChina | <other environment>"
  "EnvironmentVariables": {
      "SERVICE_MANAGEMENT_URL": "<service management url>",
      "STORAGE_ENDPOINT_SUFFIX": "<storage endpoint suffix>",
      "RESOURCE_MANAGER_URL": "<resource manager url>",
      "SEARCH_ENDPOINT_SUFFIX": "<search endpoint suffix>",
      "COSMOS_TABLES_ENDPOINT_SUFFIX": "<cosmos tables endpoint suffix>"
  },
  "ArmTemplateParameters": {
      "keyVaultDomainSuffix": "<keyVaultDomainSuffix>",
      "storageEndpointSuffix": "<storageEndpointSuffix>",
      "endpointSuffix": "<endpointSuffix>",
      "azureAuthorityHost": "<azureAuthorityHost>",
      "keyVaultEndpointSuffix": "<keyVaultEndpointSuffix>"
  }
}

The cloud selection experience for manually running pipelines is also improved. Clouds are represented as stages, which can be selected for manual runs against the forthcoming weekly pipeline:

[benbp]

/azp run net - identity - tests

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[weshaggard]

@benbp after thinking about this a little I think we should wait until after releases this week before we merge this change as this change has the potential to cause a number of issues for folks trying to get their tests working for release. Lets hold this change until next week.

[benbp]

@weshaggard I agree, but we may want to work with @heaths to figure out a workaround for hsm.

[weshaggard]

What in this PR was he depending on? I thought he already had a workaround based on just the script changes which were part of the eng/common change you merged earlier.

[benbp]

Nvm you're right.

[benbp]

Some passing pipeline tests with this version:
identity: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=621321&view=results
appconfig: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=621313&view=results
storage: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=621444&view=results

[weshaggard]

@benbp did you intend these eng/common changes directly into the .NET repo? We should make sure these get into the tools repo.

[benbp]

Ah! Missed the directory for this one. I'll make a sync PR.

[benbp]

@weshaggard Azure/azure-sdk-tools#1220