Merge branch 'main' of https://github.com/Azure/azure-sdk-for-python into webpubsub-review-2021-10-25

* 'main' of https://github.com/Azure/azure-sdk-for-python: (99 commits)
  Add Mixed Reality Remote Rendering SDK (#16643)
  [AutoRelease] t2-monitor-2021-11-03-63776 (#21559)
  Latest API(2021-10-31-preview) changes implementation in azure-communication-identity (#21331)
  [AutoRelease] t2-eventgrid-2021-10-21-28743 (#21361)
  STG79 Preview (#21591)
  delete redundant network version (#21612)
  [WebPubSub] Use consistent service description and introduction across all languages (#21544)
  [formrecognizer] Remove get children methods (#21602)
  [SchemaRegistry] generate GA SR from swagger (#21570)
  [SchemaRegistry] update core + SR version (#21573)
  update changelog (#21608)
  Revert "Update azure-core version to unbreak Search. (#21599)" (#21610)
  [acr] regen with autorest 5.11.0, bump min core dep to 1.20.0 (#21600)
  Update azure-core version to unbreak Search. (#21599)
  [formrecognizer] Update changelog for release (#21555)
  update type hints to reflect actual document body (#21593)
  Increment package version after release of azure-core (#21594)
  adding exception to regression tests for telemetry-exporter (#21576)
  [SchemaRegistry] final api review changes (#21589)
  Add prompt before replasing latest release title on prepare release run (#21579)
  ...

Author: iscai-msft

doc/dev/package_version/package_version_rule.md

@@ -0,0 +1,44 @@
+This file claims the rules how Python decide next version number for package.
+
+The package version contains two part:
+1. the package is preview or stable?
+2. version number
+
+# How to judge preview or stable?
+Python SDK is generated with [swagger](https://github.com/Azure/azure-rest-api-specs), so if swagger content is preview, 
+the package is preview; if swagger content is stable, the package is stable.
+
+(1) For single api package(for example: [datadog](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/datadog)),
+ as long as the current tag is preview, the package version should be preview
+
+(2) For multi api package(for example: [network](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/network/azure-mgmt-network)), 
+there will be `DEFAULT_API_VERSION`(for example: [`DEFAULT_API_VERSION` of network](https://github.com/Azure/azure-sdk-for-python/blob/59709af16b7cd29a51d562137bc5bbfdf53f9327/sdk/network/azure-mgmt-network/azure/mgmt/network/_network_management_client.py#L60)). 
+As long as it is preview, then the package version is preview.
+
+(note1: If the name of tag contains 'preview' or the tag contains files of 'preview' folder, then the tag is preview tag. 
+For exampe: [preview tag](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/compute/resource-manager#tag-package-2021-06-01-preview))
+
+(note2: If the api-version contains 'preview', then it is preview api-version. for example: [preview api-version](https://github.com/Azure/azure-rest-api-specs/blob/69eacf00a36d565d3220d5dd6f4a5293664f1ae9/specification/network/resource-manager/Microsoft.Network/preview/2015-05-01-preview/network.json#L6))
+
+(note3: The difference about single api and multi api, please see the detailed file)
+
+(note4: preview package version contains `b`, for example: `1.0.0b1`)
+
+# How to decide next version number
+1\. If current version is preview version, the new tag is preview tag, then next version is `x.x.xbx+1`
+
+2\. If current version is stable version, the new tag is stable tag, then :
+ * if there is breaking change, next version is `x+1.x.x`
+ * if there is new feature but no breaking change, next version is `x.x+1.x`
+ * if there is only bugfix, next version is `x.x.x+1`
+
+3\. If current version is stable version, the new tag is preview tag, calculate version number according to `2` 
+and then append `b1` in the result
+
+
+According to the up rules, we could summarize all the possibilities in the following table:
+
+![img.png](version_summary.png)
+
+(`-` means that this item doesn't influence result)
+ 

doc/dev/package_version/version summary.xlsx

@@ -28,25 +28,25 @@ class TestExample(AzureTestCase):
 ### New test structure
 
 To use the proxy, test classes should inherit from AzureRecordedTestCase and recorded test methods should use a
-RecordedByProxy decorator:
+`recorded_by_proxy` decorator:
 
 ```py
-from devtools_testutils import AzureRecordedTestCase, RecordedByProxy
+from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
 
 class TestExample(AzureRecordedTestCase):
 
-    @RecordedByProxy
+    @recorded_by_proxy
     def test_example(self):
         ...
 
     @ExamplePreparer()
-    @RecordedByProxy
+    @recorded_by_proxy
     def test_example_with_preparer(self):
         ...
 ```
 
-For async tests, import the RecordedByProxyAsync decorator from `devtools_testutils.aio` and use it in the same
-way as RecordedByProxy.
+For async tests, import the `recorded_by_proxy_async` decorator from `devtools_testutils.aio` and use it in the same
+way as `recorded_by_proxy`.
 
 > **Note:** since AzureRecordedTestCase doesn't inherit from `unittest.TestCase`, test class names need to start
 > with "Test" in order to be properly collected by pytest by default. For more information, please refer to
@@ -135,6 +135,43 @@ made to `https://fakeendpoint-secondary.table.core.windows.net`, and URIs will a
 
 For more details about sanitizers and their options, please refer to [devtools_testutils/sanitizers.py][py_sanitizers].
 
+### Record test variables
+
+To run recorded tests successfully when there's an element of non-secret randomness to them, the test proxy provides a
+[`variables` API](https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy#storing-variables).
+This makes it possible for a test to record the values of variables that were used during recording and use the same
+values in playback mode without a sanitizer.
+
+For example, imagine that a test uses a randomized `table_name` variable when creating resources. The same random value
+for `table_name` can be used in playback mode by using this `variables` API.
+
+There are two requirements for a test to use recorded variables. First, the test method should accept `**kwargs` and/or
+a `variables` parameter. Second, the test method should `return` a dictionary with any test variables that it wants to
+record. This dictionary will be stored in the recording when the test is run live, and will be passed to the test as a
+`variables` keyword argument when the test is run in playback.
+
+Below is a code example of how a test method could use recorded variables:
+
+```python
+from devtools_testutils import AzureRecordedTestCase, recorded_by_proxy
+
+class TestExample(AzureRecordedTestCase):
+
+    @recorded_by_proxy
+    def test_example(self, variables):
+        # in live mode, variables is an empty dictionary
+        # in playback mode, the value of variables is {"table_name": "random-value"}
+        if self.is_live:
+            table_name = "random-value"
+            variables = {"table_name": table_name}
+        
+        # use variables["table_name"] when using the table name throughout the test
+        ...
+
+        # return the variables at the end of the test
+        return variables
+```
+
 ## Implementation details
 
 ### What does the test proxy do?
@@ -147,7 +184,7 @@ For example, if an operation would typically make a GET request to
 `https://localhost:5001/Tables` instead. The original endpoint should be stored in an `x-recording-upstream-base-uri` --
 the proxy will send the original request and record the result.
 
-The RecordedByProxy and RecordedByProxyAsync decorators patch test requests to do this for you.
+The `recorded_by_proxy` and `recorded_by_proxy_async` decorators patch test requests to do this for you.
 
 ### How does the test proxy know when and what to record or play back?
 
@@ -179,7 +216,7 @@ Running tests in playback follows the same pattern, except that requests will be
 `/playback/stop` instead. A header, `x-recording-mode`, should be set to `record` for all requests when recording and
 `playback` when playing recordings back. More details can be found [here][detailed_docs].
 
-The RecordedByProxy and RecordedByProxyAsync decorators send the appropriate requests at the start and end of each test
+The `recorded_by_proxy` and `recorded_by_proxy_async` decorators send the appropriate requests at the start and end of each test
 case.
 
 [detailed_docs]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/README.md

doc/dev/package_version/version_summary.png

@@ -334,7 +334,7 @@ try {
     $serviceName = if (Split-Path $ServiceDirectory) {
         Split-Path -Leaf $ServiceDirectory
     } else {
-        $ServiceDirectory
+        $ServiceDirectory.Trim('/')
     }
 
     $ResourceGroupName = if ($ResourceGroupName) {
@@ -600,7 +600,7 @@ try {
             $outputFile = "$($templateFile.originalFilePath).env"
 
             $environmentText = $deploymentOutputs | ConvertTo-Json;
-            $bytes = ([System.Text.Encoding]::UTF8).GetBytes($environmentText)
+            $bytes = [System.Text.Encoding]::UTF8.GetBytes($environmentText)
             $protectedBytes = [Security.Cryptography.ProtectedData]::Protect($bytes, $null, [Security.Cryptography.DataProtectionScope]::CurrentUser)
 
             Set-Content $outputFile -Value $protectedBytes -AsByteStream -Force

doc/dev/test_proxy_migration_guide.md

@@ -0,0 +1,15 @@
+parameters: 
+  - name: ContainerRegistryClientId
+    type: string
+  - name: ContainerRegistryClientSecret
+    type: string
+  - name: ImageId
+    type: string
+steps:
+- pwsh: |
+    $containerRegistry = ("${{parameters.ImageId}}" -split "\/")[0]
+    docker login $containerRegistry -u "${{ parameters.ContainerRegistryClientId }}" -p "${{ parameters.ContainerRegistryClientSecret }}"
+  displayName: Login container registry
+- pwsh: |
+    docker pull '${{ parameters.ImageId}}'
+  displayName: Pull docker image ${{ parameters.ImageId }}

eng/common/TestResources/New-TestResources.ps1

@@ -38,8 +38,8 @@ steps:
             } else {
               Write-Host "Repository $($repository.Name) is being initialized."
 
-              Write-Host "git clone --no-checkout --filter=tree:0 git://github.com/$($repository.Name) ."
-              git clone --no-checkout --filter=tree:0 git://github.com/$($repository.Name) .
+              Write-Host "git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) ."
+              git clone --no-checkout --filter=tree:0 https://github.com/$($repository.Name) .
 
               Write-Host "git sparse-checkout init"
               git sparse-checkout init

eng/common/pipelines/templates/steps/docker-pull-image.yml

@@ -351,4 +351,15 @@ function Remove-EmptySections {
     }
     $ChangeLogEntry.ReleaseContent = $sanitizedReleaseContent.ToArray()
   }
+}
+
+function  Get-LatestReleaseDateFromChangeLog
+{
+  param (
+    [Parameter(Mandatory = $true)]
+    $ChangeLogLocation
+  )
+  $changeLogEntries = Get-ChangeLogEntries -ChangeLogLocation $ChangeLogLocation
+  $latestVersion = $changeLogEntries[0].ReleaseStatus.Trim("()")
+  return ($latestVersion -as [DateTime])
 }

eng/common/pipelines/templates/steps/sparse-checkout.yml

@@ -157,6 +157,36 @@ function New-GitHubPullRequest {
           -MaximumRetryCount 3
 }
 
+function New-GitHubIssue {
+  param (
+    [Parameter(Mandatory = $true)]
+    $RepoOwner,
+    [Parameter(Mandatory = $true)]
+    $RepoName,
+    [Parameter(Mandatory = $true)]
+    $Title,
+    [Parameter(Mandatory = $true)]
+    $Description,
+    [ValidateNotNullOrEmpty()]
+    [Parameter(Mandatory = $true)]
+    $AuthToken
+  )
+
+  $uri = "$GithubAPIBaseURI/$RepoOwner/$RepoName/issues"
+
+  $parameters = @{
+    title = $Title
+    body = $Description
+  }
+
+  return Invoke-RestMethod `
+    -Method POST `
+    -Body ($parameters | ConvertTo-Json) `
+    -Uri $uri `
+    -Headers (Get-GitHubApiHeaders -token $AuthToken) `
+    -MaximumRetryCount 3
+}
+
 function Add-GitHubIssueComment {
   param (
     [Parameter(Mandatory = $true)]

eng/common/scripts/ChangeLog-Operations.ps1

@@ -109,11 +109,15 @@ else
 $releaseDateString = $ParsedReleaseDate.ToString("MM/dd/yyyy")
 $month = $ParsedReleaseDate.ToString("MMMM")
 
-Write-Host
 Write-Host "Assuming release is in $month with release date $releaseDateString" -ForegroundColor Green
+if (Test-Path "Function:GetExistingPackageVersions")
+{
+    $releasedVersions = GetExistingPackageVersions -PackageName $packageProperties.Name -GroupId $packageProperties.Group
+    $latestReleasedVersion = $releasedVersions[$releasedVersions.Count - 1]
+    Write-Host "Latest released version: ${latestReleasedVersion}" -ForegroundColor Green
+}
 
 $currentProjectVersion = $packageProperties.Version
-
 $newVersion = Read-Host -Prompt "Input the new version, or press Enter to use use current project version '$currentProjectVersion'"
 
 if (!$newVersion)
@@ -173,8 +177,17 @@ if ($releaseTrackingOnly)
 
 if (Test-Path "Function:SetPackageVersion")
 {
-  SetPackageVersion -PackageName $packageProperties.Name -Version $newVersion -ServiceDirectory $packageProperties.ServiceDirectory -ReleaseDate $releaseDateString `
-    -PackageProperties $packageProperties
+  $replaceLatestEntryTitle = $true
+  $latestVersion = Get-LatestReleaseDateFromChangeLog -ChangeLogLocation $packageProperties.ChangeLogPath
+  if ($latestVersion)
+  {
+    $promptMessage = "The latest entry in the CHANGELOG.md already has a release date. Do you want to replace the latest entry title? Please enter (y or n)."
+    while (($readInput = Read-Host -Prompt $promptMessage) -notmatch '^[yn]$'){ }
+    $replaceLatestEntryTitle = ($readInput -eq "y")
+  }
+  SetPackageVersion -PackageName $packageProperties.Name -Version $newVersion `
+    -ServiceDirectory $packageProperties.ServiceDirectory -ReleaseDate $releaseDateString `
+    -PackageProperties $packageProperties -ReplaceLatestEntryTitle $replaceLatestEntryTitle
 }
 else
 {