From 862f66825e27ecf7ff1296cf56ff332cc5a05f8b Mon Sep 17 00:00:00 2001 From: mccoyp Date: Mon, 24 Oct 2022 16:26:49 -0700 Subject: [PATCH 1/3] Clarify EnvironmentVariableLoader use --- doc/dev/test_proxy_migration_guide.md | 40 ++++++++++++++++----------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/doc/dev/test_proxy_migration_guide.md b/doc/dev/test_proxy_migration_guide.md index 5cb550dbb7d3..dfd04a2484ba 100644 --- a/doc/dev/test_proxy_migration_guide.md +++ b/doc/dev/test_proxy_migration_guide.md @@ -267,32 +267,40 @@ This loader is meant to be paired with the PowerShell test resource management c [/eng/common/TestResources][test_resources]. It's recommended that all test suites use these scripts for live test resource management. -For an example of using the EnvironmentVariableLoader with the test proxy, you can refer to the Tables SDK. The -CosmosPreparer and TablesPreparer defined in this [preparers.py][tables_preparers] file each define an instance of the -EnvironmentVariableLoader, which are used to fetch environment variables for Cosmos and Tables, respectively. These -preparers can be used to decorate test methods directly; for example: +The EnvironmentVariableLoader accepts a positional `directory` argument and arbitrary keyword-only arguments: +- `directory` is the name of your package's service as it appears in the Python repository; i.e. `service` in `azure-sdk-for-python/sdk/service/azure-service-package`. + - For example, for `azure-keyvault-keys`, the value of `directory` is `keyvault`. +- For each environment variable you want to provide to tests, pass in a keyword argument with the pattern `environment_variable_name="sanitized-value"`. + - For example, to fetch the value of `STORAGE_ENDPOINT` and sanitize this value in recordings as `fake-endpoint`, provide `storage_endpoint="fake-endpoint"` to the EnvironmentVariableLoader constructor. + +Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these +values will automatically have sanitizers registered with the test proxy. More specifically, the true values of +requested variables will be provided to tests in live mode, and the sanitized values of these variables will be provided +in playback mode. + +The most common way to use the EnvironmentVariableLoader is to declare a callable specifying arguments by using +`functools.partial` and then decorate test methods with that callable. For example: ```python -from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy -from .preparers import TablesPreparer +import functools +from devtools_testutils import AzureRecordedTestCase, EnvironmentVariableLoader, recorded_by_proxy + +ServicePreparer = functools.partial( + EnvironmentVariableLoader, + "service", + service_endpoint="fake-endpoint", + service_account_name="fake-account-name", +) class TestExample(AzureRecordedTestCase): - @TablesPreparer() + @ServicePreparer() @recorded_by_proxy def test_example_with_preparer(self, **kwargs): - tables_storage_account_name = kwargs.pop("tables_storage_account_name") - tables_primary_storage_account_key = kwargs.pop("tables_primary_storage_account_key") + service_endpoint = kwargs.pop("service_endpoint") ... ``` -Or, they can be used in a custom decorator, as they are in the `cosmos_decorator` and `tables_decorator` defined in -[preparers.py][tables_preparers]. `@tables_decorator`, for instance, is then used in place of `@TablesPreparer()` for -the example above (note that the method-style `tables_decorator` is used without parentheses). - -Decorated test methods will have the values of environment variables passed to them as keyword arguments, and these -values will automatically have sanitizers registered with the test proxy. - ### Record test variables To run recorded tests successfully when there's an element of non-secret randomness to them, the test proxy provides a From 6766bfabcaeaa04dddee07012d0efd0452067dcb Mon Sep 17 00:00:00 2001 From: mccoyp Date: Mon, 24 Oct 2022 19:28:13 -0700 Subject: [PATCH 2/3] Miscellaneous improvements --- doc/dev/test_proxy_migration_guide.md | 28 +++++++++++++-------------- doc/dev/tests.md | 18 +++++++++-------- 2 files changed, 23 insertions(+), 23 deletions(-) diff --git a/doc/dev/test_proxy_migration_guide.md b/doc/dev/test_proxy_migration_guide.md index dfd04a2484ba..eb71333bc5e9 100644 --- a/doc/dev/test_proxy_migration_guide.md +++ b/doc/dev/test_proxy_migration_guide.md @@ -95,8 +95,8 @@ Resource preparers need a management client to function, so test classes that us ### Perform one-time setup -1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install]. -2. After installing, make sure Docker is running and is using Linux containers before running tests. +1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command. +2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests. 3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in order to communicate with the test proxy over a secure connection. @@ -110,6 +110,7 @@ In a `conftest.py` file for your package's tests, add a session-level fixture th `devtools_testutils.test_proxy` as a parameter (and has `autouse` set to `True`): ```python +import pytest from devtools_testutils import test_proxy # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method @@ -119,9 +120,7 @@ def start_proxy(test_proxy): ``` The `test_proxy` fixture will fetch the test proxy Docker image and create a new container called -`ambitious_azsdk_test_proxy` if one doesn't exist already. If the container already exists, the fixture will start the -container if it's currently stopped. The container will be stopped after tests finish running, but will stay running if -test execution is interrupted. +`ambitious_azsdk_test_proxy`, which will be deleted after test execution unless interrupted. If your tests already use an `autouse`d, session-level fixture for tests, you can accept the `test_proxy` parameter in that existing fixture instead of adding a new one. For an example, see the [Register sanitizers](#register-sanitizers) @@ -144,17 +143,13 @@ need old `.yml` recordings. > **Note:** support for configuring live or playback tests with a `testsettings_local.cfg` file has been > deprecated in favor of using just `AZURE_TEST_RUN_LIVE`. -> **Note:** the recording storage location is determined when the proxy Docker container is created. If there are -> multiple local copies of the `azure-sdk-for-python` repo on your machine, you will need to delete any existing -> `ambitious_azsdk_test_proxy` container before recordings can be stored in a different repo copy. - ### Register sanitizers Since the test proxy doesn't use [`vcrpy`][vcrpy], tests don't use a scrubber to sanitize values in recordings. Instead, sanitizers (as well as matchers and transforms) can be registered on the proxy as detailed in [this][sanitizers] section of the proxy documentation. Sanitizers can be registered via `add_*_sanitizer` methods in `devtools_testutils`. For example, the general-use method for sanitizing recording bodies, headers, and URIs is -`add_general_regex_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at +`add_general_string_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at [devtools_testutils/sanitizers.py][py_sanitizers]. Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that @@ -162,8 +157,8 @@ are shared by different tests, using a session fixture declared in a `conftest.p [pytest's scoped fixture documentation][pytest_fixtures] for more details. As a simple example, to emulate the effect registering a name pair with a `vcrpy` scrubber, you can provide the exact -value you want to sanitize from recordings as the `regex` in the general regex sanitizer. With `vcrpy`, you would likely -do something like the following: +value you want to sanitize from recordings as the `target` in the general string sanitizer. With `vcrpy`, you would +likely do something like the following: ```python import os @@ -180,19 +175,21 @@ To do the same sanitization with the test proxy, you could add something like th ```python import os -from devtools_testutils import add_general_regex_sanitizer, test_proxy +from devtools_testutils import add_general_string_sanitizer, test_proxy # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method @pytest.fixture(scope="session", autouse=True) def add_sanitizers(test_proxy): - add_general_regex_sanitizer(regex=os.getenv("AZURE_KEYVAULT_NAME"), value="fake-vault") + # The default value for the environment variable should be the value you use in playback + vault_name = os.getenv("AZURE_KEYVAULT_NAME", "fake-vault") + add_general_string_sanitizer(target=vault_name, value="fake-vault") ``` Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started beforehand. For a more advanced scenario, where we want to sanitize the account names of all Tables endpoints in recordings, we -could instead call +could instead use the `add_general_regex_sanitizer` method: ```python add_general_regex_sanitizer( @@ -441,6 +438,7 @@ For more details on proxy startup, please refer to the [proxy documentation][det [pipelines_ci]: https://github.com/Azure/azure-sdk-for-python/blob/5ba894966ed6b0e1ee8d854871f8c2da36a73d79/sdk/eventgrid/ci.yml#L30 [pipelines_live]: https://github.com/Azure/azure-sdk-for-python/blob/e2b5852deaef04752c1323d2ab0958f83b98858f/sdk/textanalytics/tests.yml#L26-L27 +[podman]: https://podman.io/ [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/test-proxy/trusting-cert-per-language.md [py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py [pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery diff --git a/doc/dev/tests.md b/doc/dev/tests.md index 7944f601f092..4be3fbb65562 100644 --- a/doc/dev/tests.md +++ b/doc/dev/tests.md @@ -151,8 +151,8 @@ To migrate an existing test suite to use the test proxy, or to learn more about ### Perform one-time test proxy setup -1. Docker is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install]. -2. After installing, make sure Docker is running and is using Linux containers before running tests. +1. Docker (or Podman) is a requirement for using the test proxy. You can install Docker from [docs.docker.com][docker_install], or install Podman at [podman.io][podman]. To use Podman, set an alias for `podman` to replace the `docker` command. +2. After installing, make sure Docker/Podman is running and is using Linux containers before running tests. 3. Follow the instructions [here][proxy_cert_docs] to complete setup. You need to trust a certificate on your machine in order to communicate with the test proxy over a secure connection. @@ -213,6 +213,7 @@ Create a `conftest.py` file within your package's test directory (`sdk/{service} session-level fixture that accepts `devtools_testutils.test_proxy` as a parameter (and has `autouse` set to `True`): ```python +import pytest from devtools_testutils import test_proxy # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method @@ -358,27 +359,27 @@ There are two primary ways to keep secrets from being written into recordings: 1. The `EnvironmentVariableLoader` will automatically sanitize the values of captured environment variables with the provided fake values. 2. Sanitizers can be registered via `add_*_sanitizer` methods in `devtools_testutils`. For example, the general-use - method for sanitizing recording bodies, headers, and URIs is `add_general_regex_sanitizer`. Other sanitizers are + method for sanitizing recording bodies, headers, and URIs is `add_general_string_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at [devtools_testutils/sanitizers.py][py_sanitizers]. As a simple example of registering a sanitizer, you can provide the exact value you want to sanitize from recordings as -the `regex` in the general regex sanitizer. To replace all instances of the string "my-key-vault" with "fake-vault" in +the `target` in the general string sanitizer. To replace all instances of the string "my-key-vault" with "fake-vault" in recordings, you could add something like the following in the package's `conftest.py` file: ```python -from devtools_testutils import add_general_regex_sanitizer, test_proxy +from devtools_testutils import add_general_string_sanitizer, test_proxy # autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method @pytest.fixture(scope="session", autouse=True) def add_sanitizers(test_proxy): - add_general_regex_sanitizer(regex="my-key-vault", value="fake-vault") + add_general_string_sanitizer(target="my-key-vault", value="fake-vault") ``` Note that the sanitizer fixture accepts the `test_proxy` fixture as a parameter to ensure the proxy is started beforehand (see [Start the test proxy server](#start-the-test-proxy-server)). For a more advanced scenario, where we want to sanitize the account names of all storage endpoints in recordings, we -could instead call +could instead use `add_general_regex_sanitizer`: ```python add_general_regex_sanitizer( @@ -640,7 +641,7 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should [engsys_wiki]: https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/48/Create-a-new-Live-Test-pipeline?anchor=test-resources.json [env_var_loader]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/envvariable_loader.py -[generate_sas]: https://github.com/Azure/azure-sdk-for-python/blob/6e1f7c02af0c28d5725a532ebe4fc7125256858c/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L200 +[generate_sas]: https://github.com/Azure/azure-sdk-for-python/blob/bf4749babb363e2dc972775f4408036e31f361b4/tools/azure-sdk-tools/devtools_testutils/azure_recorded_testcase.py#L196 [generate_sas_example]: https://github.com/Azure/azure-sdk-for-python/blob/3e3fbe818eb3c80ffdf6f9f1a86affd7e879b6ce/sdk/tables/azure-data-tables/tests/test_table_entity.py#L1691 [kv_test_resources]: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/keyvault/test-resources.json @@ -650,6 +651,7 @@ Tests that use the Shared Access Signature (SAS) to authenticate a client should [mgmt_settings_fake]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_settings_fake.py [packaging]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md +[podman]: https://podman.io/ [proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md [proxy_general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md [proxy_migration_guide]: https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/test_proxy_migration_guide.md From 86bb1a7540f581d5b9b0f4f2015f6b947bee237d Mon Sep 17 00:00:00 2001 From: mccoyp Date: Tue, 25 Oct 2022 13:19:44 -0700 Subject: [PATCH 3/3] Guidance for parametrization --- doc/dev/test_proxy_migration_guide.md | 61 +++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/doc/dev/test_proxy_migration_guide.md b/doc/dev/test_proxy_migration_guide.md index eb71333bc5e9..3f342b6d87de 100644 --- a/doc/dev/test_proxy_migration_guide.md +++ b/doc/dev/test_proxy_migration_guide.md @@ -425,6 +425,64 @@ container if it's not already running. For more details on proxy startup, please refer to the [proxy documentation][detailed_docs]. +### Use `pytest.mark.parametrize` with migrated tests + +Migrating tests to use basic `pytest` tools allows us to take advantage of helpful features such as +[parametrization][parametrize]. Parametrization allows you to share test code by re-running the same test with varying +inputs. For example, [`azure-keyvault-keys` tests][parametrize_example] are parametrized to run with multiple API +versions and multiple Key Vault configurations. + +Because of how the `pytest.mark.parametrize` mechanism works, the `recorded_by_proxy(_async)` decorators aren't +compatible without an additional decorator that handles the arguments we want to parametrize. The callable that +`pytest.mark.parametrize` decorates needs to have positional parameters that match the arguments we're parametrizing; +for example: + +```python +import pytest +from devtools_testutils import recorded_by_proxy + +test_values = [ + ("first_value_a", "first_value_b"), + ("second_value_a", "second_value_b"), +] + +# Works because `parametrize` decorates a method with positional `a` and `b` parameters +@pytest.mark.parameterize("a, b", test_values) +def test_function(a, b, **kwargs): + ... + +# Doesn't work; raises collection error +# `recorded_by_proxy`'s wrapping function doesn't accept positional `a` and `b` parameters +@pytest.mark.parameterize("a, b", test_values) +@recorded_by_proxy +def test_recorded_function(a, b, **kwargs): + ... +``` + +To parametrize recorded tests, we need a decorator between `pytest.mark.parametrize` and `recorded_by_proxy` that +accepts the expected arguments. We can do this by declaring a class with a custom `__call__` method: + +```python +class ArgumentPasser: + def __call__(self, fn): + # _wrapper accepts the `a` and `b` arguments we want to parametrize with + def _wrapper(test_class, a, b, **kwargs): + fn(test_class, a, b, **kwargs) + return _wrapper + +# Works because `ArgumentPasser.__call__`'s return value has the expected parameters +@pytest.mark.parameterize("a, b", test_values) +@ArgumentPasser() +@recorded_by_proxy +def test_recorded_function(a, b, **kwargs): + ... +``` + +You can also introduce additional logic into the `__call__` method of your intermediate decorator. In the aforementioned +[`azure-keyvault-keys` test example][parametrize_example], the decorator between `parametrize` and `recorded_by_proxy` +is actually a [client preparer][parametrize_class] that creates a client based on the parametrized input and passes this +client to the test. + [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md [docker_install]: https://docs.docker.com/get-docker/ @@ -436,6 +494,9 @@ For more details on proxy startup, please refer to the [proxy documentation][det [mgmt_recorded_test_case]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/mgmt_recorded_testcase.py +[parametrize]: https://docs.pytest.org/latest/example/parametrize.html +[parametrize_example]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/test_key_client.py#L182 +[parametrize_class]: https://github.com/Azure/azure-sdk-for-python/blob/d92b63b9976b0025b274016c49a250fb7c4d7333/sdk/keyvault/azure-keyvault-keys/tests/_test_case.py#L59 [pipelines_ci]: https://github.com/Azure/azure-sdk-for-python/blob/5ba894966ed6b0e1ee8d854871f8c2da36a73d79/sdk/eventgrid/ci.yml#L30 [pipelines_live]: https://github.com/Azure/azure-sdk-for-python/blob/e2b5852deaef04752c1323d2ab0958f83b98858f/sdk/textanalytics/tests.yml#L26-L27 [podman]: https://podman.io/