Allow specifying default versions on ClientTestFixture

[pakrym]

No description provided.

[kasobol-msft]

Looking good.

[weshaggard]

cc @benbp
@pakrym do you you have thoughts on how we would setup which service version we want to run the tests against?

[pakrym]

Adding an environment variable (or something that flows into the .env file) with a list of supported service versions would be an easy solution.

We would then intersect that with the list we have in code and run only supported tests.

[benbp]

@maorleger and I were just talking about a pattern along the same lines for js. I think we can pass the service version in via the matrix, and plumb through the env variables in the yaml.

[weshaggard]

Wouldn't we also need to use that information to setup the clients we are creating in our tests? By default the clients usually use the latest api version so we would need to either pass in a version or maybe start to have a pattern where our client libraries could read some well known env variable as the default api version when being constructed.

[pakrym]

Yes, that would only work for tests that opted into our multi-service-version experience.

They would be already reading the parameter:

https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/keyvault/Azure.Security.KeyVault.Keys/tests/KeysTestBase.cs#L15-L18

https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/keyvault/Azure.Security.KeyVault.Keys/tests/KeysTestBase.cs#L40

[maorleger]

@pakrym I'm doing something similar for JS, maybe we can sync up offline to see if we can align on the approach?

Do you have any concerns about testing every service version across every matrix / build configuration (as far as it adds to the run time)? One thing I was considering is having every live test run against the latest service version plus one live test that runs against every service version. So our environment variable might be more along the lines of "enable/disable multi-version testing" and library authors would need to opt-in by reading that variable and setting the service versions accordingly.

We might have something like:

• Node 14 Windows (latest service version)
• Node 10 Windows (latest service version)
• ....
• Node 14 OS X (latest service version)
• ...
• (One additional) Node 14 Windows (every service version)

What do you think about that?

[pakrym]

maybe we can sync up offline to see if we can align on the approach?

Sure.

Do you have any concerns about testing every service version across every matrix / build configuration (as far as it adds to the run time)?

Yeah, testing all service versions an all operation systems seems excessive. We already have a flag to only run the latest service version and are considering adding an additional flag to specify which exact service version to run. But I also think it's useful to be able to run a set of tests against all versions in development.

We are trying to build up additional features when requirements come up and see how they work out.

What do you think about that?

I think it's reasonable.