update branch

.github/CODEOWNERS

@@ -17,7 +17,7 @@
 # Service team
 ####
 # PRLabel: %Azure.Identity
-/sdk/identity/                                                       @chlowell @mccoyp @schaabs
+/sdk/identity/                                                       @chlowell @mccoyp @schaabs @xiangyan99
 
 # PRLabel: %Event Hubs
 /sdk/eventhub/                                                       @annatisch @yunhaoling @swathipil @rakshith91
@@ -39,6 +39,9 @@
 
 # PRLabel: %Communication
 /sdk/communication/                                                  @acsdevx-msft
+/sdk/communication/azure-communication-phonenumbers/                 @RoyHerrod @danielav7 @whisper6284 @AlonsoMondal
+/sdk/communication/azure-communication-sms/                          @RoyHerrod @arifibrahim4
+/sdk/communication/azure-communication-identity/                     @Azure/acs-identity-sdk
 
 # PRLabel: %KeyVault
 /sdk/keyvault/                                                       @schaabs @chlowell @mccoyp @YalinLi0312
@@ -63,7 +66,7 @@
 /sdk/containerservice/                                               @samkreter @zqingqing1 @GaneshaThirumurthi
 
 # PRLabel: %Cosmos
-/sdk/cosmos/                                                         @southpolesteve @zfoster
+/sdk/cosmos/                                                         @kushagraThapar @simorenoh @simplynaveen20 @xinlian12 @moderakh 
 
 # PRLabel: %Data Factory
 /sdk/datafactory/                                                    @hvermis
@@ -101,7 +104,7 @@
 /sdk/servicebus/                                                     @annatisch @yunhaoling @swathipil @rakshith91
 
 # PRLabel: %Synapse
-/sdk/synapse/                                                        @aim-for-better @idear1203
+/sdk/synapse/                                                        @wonner @idear1203 @xiangyan99
 
 # PRLabel: %EngSys
 /sdk/template/                                                       @Azure/azure-sdk-eng

.gitignore

@@ -95,12 +95,17 @@ venv/
 code_reports
 
 # Azure Storage test credentials
+sdk/storage/azure-storage-blob/tests/settings/settings_real.py
+sdk/storage/azure-storage-queue/tests/settings/settings_real.py
+sdk/storage/azure-storage-file-share/tests/settings/settings_real.py
+sdk/storage/azure-storage-file-datalake/tests/settings/settings_real.py
+
+
+# The locations below are deprecated - keep to prevent any accidental secrets leakage ==========
 sdk/storage/azure-storage-blob/tests/_shared/settings_real.py
 sdk/storage/azure-storage-queue/tests/_shared/settings_real.py
 sdk/storage/azure-storage-file-share/tests/_shared/settings_real.py
 sdk/storage/azure-storage-file-datalake/tests/settings_real.py
-
-# The locations below are deprecated - keep to prevent any accidental secrets leakage ==========
 sdk/storage/azure-storage-blob/tests/settings_real.py
 sdk/storage/azure-storage-queue/tests/settings_real.py
 sdk/storage/azure-storage-file-share/tests/settings_real.py

README.md

@@ -4,6 +4,10 @@
 
 This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our [public developer docs](https://docs.microsoft.com/python/azure/) or our versioned [developer docs](https://azure.github.io/azure-sdk-for-python).
 
+## _Disclaimer_
+
+_Azure SDK Python packages support for Python 2.7 is ending 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691_
+
 ## Getting started
 
 For your convenience, each service has a separate set of libraries that you can choose to use instead of one, large Azure package. To get started with a specific library, see the `README.md` (or `README.rst`) file located in the library's project folder.
@@ -32,7 +36,7 @@ You can find the [most up to date list of all of the new packages on our page](h
 
 ### Client: Previous Versions
 
-Last stable versions of packages that have been provided for usage with Azure and are production-ready. These libraries provide you with similar functionalities to the Preview ones as they allow you to use and consume existing resources and interact with them, for example: upload a blob. They might not implement the [guidelines](https://azure.github.io/azure-sdk/python/guidelines/index.html) or have the same feature set as the Novemeber releases. They do however offer wider coverage of services.
+Last stable versions of packages that have been provided for usage with Azure and are production-ready. These libraries provide you with similar functionalities to the Preview ones as they allow you to use and consume existing resources and interact with them, for example: upload a blob. They might not implement the [guidelines](https://azure.github.io/azure-sdk/python/guidelines/index.html) or have the same feature set as the November releases. They do however offer wider coverage of services.
 
 ### Management: New Releases
 A new set of management libraries that follow the [Azure SDK Design Guidelines for Python](https://azure.github.io/azure-sdk/python/guidelines/) are now available. These new libraries provide a number of core capabilities that are shared amongst all Azure SDKs, including the intuitive Azure Identity library, an HTTP Pipeline with custom policies, error-handling, distributed tracing, and much more.

doc/dev/README.md

@@ -6,8 +6,8 @@ Overview of the documents:
 - [Developer Set-Up](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/dev_setup.md) : How to create a development environment for this repo
 - [Release](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/release.md) : How to release a package when ready
 - [Packaging](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md) : How to organize packaging information for packages under `azure`
-- [Testing](./tests.md): How to write unit and functional tests for a library
-- [Docstrings and Type hints](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring_typehint.md): How to document an SDK for compatability with internal tools (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site.
+- [Testing](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md): How to write unit and functional tests for a library
+- [Docstrings and Type hints](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring_typehint.md): How to document an SDK for compatibility with internal tools (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site.
 
 The [mgmt](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`)
 

doc/dev/docstring_typehint.md

@@ -4,7 +4,7 @@ All public methods should have docstrings to document the parameters, keywords,
 
 * [Docstrings](#docstrings)
     * [Method Docstrings](#method_docstrings)
-    * [Model and Client Docstrings](#Model_and_Client_Docstrings)
+    * [Model and Client Docstrings](#model_and_client_docstrings)
 * [Type Hints](#type_hints)
     * [Type Hints for Python 2.7 and 3.5+](#type_hints_for_python_2.7_and_3.5+)
     * [Type Hints for Python 3.5+](#type_hints_for_python_3.5+)

doc/dev/get_unreleased_package_guide.md

@@ -0,0 +1,19 @@
+# Get the Unreleased Package
+
+This guide is to help Python SDK users to get unreleased package.
+
+(1) Submit a draft PR to https://github.com/Azure/azure-rest-api-specs.
+
+(Any modification is OK, because only the pipeline needs to be triggered)
+![img.png](unreleased_package_guide_example1.png)
+
+(2) Wait until swagger generation artifacts is complete.
+
+The following figure shows the wheel and zip of the package.Click to download them.
+![img.png](unreleased_package_guide_example2.png)
+
+## Note
+(1) If there is no link in the figure above, it may be folded. You can also find it in the check.
+![img.png](unreleased_package_guide_example3.png)
+
+(2) The private Azure/azure-rest-api-specs-pr repo can only be triggered when the target branch is `main`

doc/dev/perfstress_tests.md

@@ -82,7 +82,7 @@ The framework has a series of common command line options built in:
 - `-w --warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
 - `--sync` Whether to run the tests in sync or async. Default is False (async).
 - `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).
-- `-x --test-proxy` Whether to run the tests against the test proxy server. Specfiy the URL for the proxy endpoint (e.g. "https://localhost:5001").
+- `-x --test-proxies` Whether to run the tests against the test proxy server. Specify the URL(s) for the proxy endpoint(s) (e.g. "https://localhost:5001").
 - `--profile` Whether to run the perftest with cProfile. If enabled (default is False), the output file of the **last completed single iteration** will be written to the current working directory in the format `"cProfile-<TestClassName>-<TestID>-<sync/async>.pstats"`.
 
 

doc/dev/test_proxy_migration_guide.md

@@ -0,0 +1,193 @@
+# Guide for migrating to the test proxy from vcrpy
+
+This guide describes the changes that service SDKs should make to their test frameworks in order to take advantage of
+the Azure SDK test proxy.
+
+Documentation of the motivations and goals of the test proxy can be found [here][general_docs] in the azure-sdk-tools
+GitHub repository, and documentation of how to set up and use the proxy can be found [here][detailed_docs].
+
+## Update existing tests
+
+### Current test structure
+
+Test classes currently inherit from AzureTestCase, and test methods can optionally use decorators:
+
+```py
+from devtools_testutils import AzureTestCase
+
+class TestExample(AzureTestCase):
+
+    def test_example(self):
+        ...
+
+    @ExamplePreparer()
+    def test_example_with_preparer(self):
+        ...
+```
+
+### New test structure
+
+To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a
+RecordedByProxy decorator:
+
+```py
+from devtools_testutils import AzureRecordedTestCase, RecordedByProxy
+
+class TestExample(AzureRecordedTestCase):
+
+    @RecordedByProxy
+    def test_example(self):
+        ...
+
+    @ExamplePreparer()
+    @RecordedByProxy
+    def test_example_with_preparer(self):
+        ...
+```
+
+For async tests, import the RecordedByProxyAsync decorator from `devtools_testutils.aio` and use it in the same
+way as RecordedByProxy.
+
+> **Note:** since AzureRecordedTestCase doesn't inherit from `unittest.TestCase`, test class names need to start
+> with "Test" in order to be properly collected by pytest by default. For more information, please refer to
+> [pytest's documentation][pytest_collection].
+
+## Run the tests
+
+### Perform one-time setup
+
+The test proxy is made available for your tests via a Docker container. Some tests require an SSL connection to work, so
+the Docker image used for the container has a certificate imported that you need to trust on your machine. Instructions
+on how to do so can be found [here][proxy_cert_docs].
+
+### Start the proxy server
+
+There is a [PowerShell script][docker_start_proxy] in `eng/common/testproxy` that will fetch the proxy Docker image if
+you don't already have it, and will start or stop a container running the image for you. You can run the following
+command from the root of the `azure-sdk-for-python` directory to start the container whenever you want to make the test
+proxy available for running tests:
+
+```powershell
+.\eng\common\testproxy\docker-start-proxy.ps1 "start"
+```
+
+Note that the proxy is available as long as the container is running. In other words, you don't need to start and
+stop the container for each test run or between tests for different SDKs. You can run the above command in the morning
+and just stop the container whenever you'd like. To stop the container, run the same command but with `"stop"` in place
+of `"start"`. In the future, the proxy container will be set up and started automatically when tests are run, and
+starting it manually will be optional.
+
+For more details on proxy startup, please refer to the [proxy documentation][detailed_docs].
+
+### Record or play back tests
+
+Configuring live and playback tests is done with the `AZURE_TEST_RUN_LIVE` environment variable. When this variable is
+set to "true" or "yes", live tests will run and produce recordings. When this variable is set to "false" or "no", or
+not set at all, tests will run in playback mode and attempt to match existing recordings.
+
+Recordings for a given package will end up in that package's `/tests/recordings` directory, just like they currently
+do.
+
+> **Note:** at this time, support for configuring live or playback tests with a `testsettings_local.cfg` file has been
+> deprecated in favor of using just `AZURE_TEST_RUN_LIVE`.
+
+### Register sanitizers
+
+Since the test proxy doesn't use [`vcrpy`][vcrpy], tests don't use a scrubber to sanitize values in recordings.
+Instead, sanitizers (as well as matchers and transforms) can be registered on the proxy as detailed in
+[this][sanitizers] section of the proxy documentation. Sanitizers can be registered via `add_*_sanitizer` methods in
+`devtools_testutils`. For example, the general-use method for sanitizing recording bodies, headers, and URIs is
+`add_general_regex_sanitizer`. Other sanitizers are available for more specific scenarios and can be found at
+[devtools_testutils/sanitizers.py][py_sanitizers].
+
+Sanitizers, matchers, and transforms remain registered until the proxy container is stopped, so for any sanitizers that
+are shared by different tests, using a session fixture declared in a `conftest.py` file is recommended. Please refer to
+[pytest's scoped fixture documentation][pytest_fixtures] for more details.
+
+As a simple example, to emulate the effect registering a name pair with a `vcrpy` scrubber, you can provide the exact
+value you want to sanitize from recordings as the `regex` in the general regex sanitizer. To replace all instances of
+the string "my-key-vault" with "fake-vault" in recordings, you could add something like the following in the package's
+`conftest.py` file:
+
+```python
+from devtools_testutils import add_general_regex_sanitizer
+
+# autouse=True will trigger this fixture on each pytest run, even if it's not explicitly used by a test method
+@pytest.fixture(scope="session", autouse=True)
+def add_sanitizers():
+    add_general_regex_sanitizer(regex="my-key-vault", value="fake-vault")
+```
+
+For a more advanced scenario, where we want to sanitize the account names of all storage endpoints in recordings, we
+could instead call
+
+```python
+add_general_regex_sanitizer(
+    regex="(?<=\\/\\/)[a-z]+(?=(?:|-secondary)\\.(?:table|blob|queue)\\.core\\.windows\\.net)",
+    value="fakeendpoint",
+)
+```
+
+`add_general_regex_sanitizer` accepts a regex, replacement value, and capture group as keyword-only arguments. In the
+snippet above, any storage endpoint URIs that match the specified URI regex will have their account name replaced with
+"fakeendpoint". A request made to `https://tableaccount-secondary.table.core.windows.net` will be recorded as being
+made to `https://fakeendpoint-secondary.table.core.windows.net`, and URIs will also be sanitized in bodies and headers.
+
+For more details about sanitizers and their options, please refer to [devtools_testutils/sanitizers.py][py_sanitizers].
+
+## Implementation details
+
+### What does the test proxy do?
+
+The gist of the test proxy is that it stands in between your tests and the service. What this means is that test
+requests which would usually go straight to the service should instead point to the locally-hosted test proxy.
+
+For example, if an operation would typically make a GET request to
+`https://fakeazsdktestaccount.table.core.windows.net/Tables`, that operation should now be sent to
+`https://localhost:5001/Tables` instead. The original endpoint should be stored in an `x-recording-upstream-base-uri` --
+the proxy will send the original request and record the result.
+
+The RecordedByProxy and RecordedByProxyAsync decorators patch test requests to do this for you.
+
+### How does the test proxy know when and what to record or play back?
+
+This is achieved by making POST requests to the proxy server that say whether to start or stop recording or playing
+back, as well as what test is being run.
+
+To start recording a test, the server should be primed with a POST request:
+
+```
+URL: https://localhost:5001/record/start
+headers {
+    "x-recording-file": "<path-to-test>/recordings/<testfile>.<testname>"
+}
+```
+
+This will return a recording ID in an `x-recording-id` header. This ID should be sent as an `x-recording-id` header in
+all further requests during the test.
+
+After the test has finished, a POST request should be sent to indicate that recording is complete:
+
+```
+URL: https://localhost:5001/record/stop
+headers {
+    "x-recording-id": "<x-recording-id>"
+}
+```
+
+Running tests in playback follows the same pattern, except that requests will be sent to `/playback/start` and
+`/playback/stop` instead. A header, `x-recording-mode`, should be set to `record` for all requests when recording and
+`playback` when playing recordings back. More details can be found [here][detailed_docs].
+
+The RecordedByProxy and RecordedByProxyAsync decorators send the appropriate requests at the start and end of each test
+case.
+
+[detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md
+[docker_start_proxy]: https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/testproxy/docker-start-proxy.ps1
+[general_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/README.md
+[proxy_cert_docs]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/documentation/trusting-cert-per-language.md
+[py_sanitizers]: https://github.com/Azure/azure-sdk-for-python/blob/main/tools/azure-sdk-tools/devtools_testutils/sanitizers.py
+[pytest_collection]: https://docs.pytest.org/latest/goodpractices.html#test-discovery
+[pytest_fixtures]: https://docs.pytest.org/latest/fixture.html#scope-sharing-fixtures-across-classes-modules-packages-or-session
+[sanitizers]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md#session-and-test-level-transforms-sanitiziers-and-matchers
+[vcrpy]: https://vcrpy.readthedocs.io