Merge branch 'master' of https://github.com/Azure/azure-sdk-for-python into regenerate_keys

* 'master' of https://github.com/Azure/azure-sdk-for-python:
  Add foundations for eng/common restructure (Azure#12725)
  Sync eng/common directory with azure-sdk-tools repository (Azure#12730)
  fix flaky test (Azure#12721)
  python link verification (Azure#12543)
  [ServiceBus] Remove legacy control client (Azure#12644)
  mirror update from MicrosoftDocs/azure-docs-sdk-python#709 (Azure#12691)
  [Service Bus] Rule supports parameters of sql expressions and properties of cor filter with multiple data types (Azure#12580)

Author: iscai-msft

README.md

@@ -47,7 +47,7 @@ Management libraries can be identified by namespaces that start with `azure-mgmt
 ## Need help?
 
 * For detailed documentation visit our [Azure SDK for Python documentation](https://aka.ms/python-docs)
-* File an issue via [Github Issues](../../issues)
+* File an issue via [Github Issues](https://github.com/Azure/azure-sdk-for-python/issues)
 * Check [previous questions](https://stackoverflow.com/questions/tagged/azure+python) or ask new ones on StackOverflow using `azure` and `python` tags.
 
 ### Community

doc/dev/mgmt/swagger/multi_api/readme.md

@@ -8,7 +8,7 @@ This is the AutoRest configuration file for Network.
 
 ## Getting Started
 
-To build the SDK for Network, simply [Install AutoRest](https://aka.ms/autorest/install) and in this folder, run:
+To build the SDK for Network, simply [Install AutoRest](https://github.com/Azure/autorest/blob/master/docs/installing-autorest.md) and in this folder, run:
 
 > `autorest`
 
@@ -66,4 +66,4 @@ swagger-to-sdk:
 
 ## Python
 
-See configuration in [readme.python.md](./readme.python.md)
+See configuration in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)

doc/dev/mgmt/swagger/single_api/readme.md

@@ -8,7 +8,7 @@ This is the AutoRest configuration file for Cdn.
 
 ---
 ## Getting Started
-To build the SDK for Cdn, simply [Install AutoRest](https://aka.ms/autorest/install) and in this folder, run:
+To build the SDK for Cdn, simply [Install AutoRest](https://github.com/Azure/autorest/blob/master/docs/installing-autorest.md) and in this folder, run:
 
 > `autorest`
 

doc/dev/mgmt/swagger_conf.md

@@ -18,38 +18,38 @@ Writing the readme is the responsibility of the Python SDK team. There is curren
 - Readme that handles only one API version, and generates packages that handle one API version only
 - Readme that handles several API versions, and generates packages with multiples API and profile supports
 
-These templates can be found in the [single_api](./single_api) and the [multi_api](./multi_api) folders.
+These templates can be found in the [single_api](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api) and the [multi_api](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api) folders.
 
 
 ### Single API readmes
 
 This one is the most simple:
-- Copy the [readme.python.md](./single_api/readme.python.md) and replace `servicetoreplace` by your service name
-- Be sure the main [readme.md](./single_api/readme.md) contains a "swagger-to-sdk" section with Python
+- Copy the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api/readme.python.md) and replace `servicetoreplace` by your service name
+- Be sure the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api/readme.md) contains a "swagger-to-sdk" section with Python
 
 ### Multi API readmes
 
-When doing multi-api packages, it means you have shipping several "tags" of the main readme as one package. Autorest is calling this process a "batch" call, and this is the purpose of the "batch" section in [readme.python.md](./multi_api/readme.python.md).
+When doing multi-api packages, it means you have shipping several "tags" of the main readme as one package. Autorest is calling this process a "batch" call, and this is the purpose of the "batch" section in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md).
 
 In order to be sure the correct tags exist, you can use the following script:
 ```shell
 python ./scripts/multi_api_readme_help.py /azure-rest-api-specs/specification/service/resource-manager/
 ```
 
 This script will analyze the Swaggers available, and suggests on stdout:
-- A list of tags for the main [readme.md](./multi_api/readme.md)
-- A batch declaration for the [readme.python.md](./multi_api/readme.python.md)
+- A list of tags for the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.md)
+- A batch declaration for the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
 
 This script is not perfect that it *does* require manual review of the output and not a direct copy/paste.
 
 It's important for Python that tags represents only *ONE* unique API version. It's why it's pretty common that Python uses a set of tags that other languages don't use.
 
 Once you know the list of tags you need to generate:
 
-- Copy the [readme.python.md](./multi_api/readme.python.md) and replace `servicetoreplace` by your service name
-- Update the batch list of [readme.python.md](./multi_api/readme.python.md)
-- Be sure you have one tag section for each batch entry in [readme.python.md](./multi_api/readme.python.md)
-- Be sure the main [readme.md](./multi_api/readme.md) contains a "swagger-to-sdk" section with Python with an `afterscripts` section like the one in the template.
+- Copy the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md) and replace `servicetoreplace` by your service name
+- Update the batch list of [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
+- Be sure you have one tag section for each batch entry in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
+- Be sure the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.md) contains a "swagger-to-sdk" section with Python with an `afterscripts` section like the one in the template.
 
 The `afterscripts` will execute a Jinja template to create a client to link together all the batch generated autorest ([example](https://github.com/Azure/azure-sdk-for-python/blob/4a7c67189591b052fe2b5769847ff68f7845386d/sdk/storage/azure-mgmt-storage/azure/mgmt/storage/_storage_management_client.py))
 

eng/common/scripts/Verify-Links.ps1

@@ -87,7 +87,7 @@ function ResolveUri ([System.Uri]$referralUri, [string]$link)
     return $null
   }
 
-  if ($null -ne $ignoreLinks -and $ignoreLinks.Contains($link)) {
+  if ($null -ne $ignoreLinks -and ($ignoreLinks.Contains($link) -or $ignoreLinks.Contains($linkUri.ToString()))) {
     Write-Verbose "Ignoring invalid link $linkUri because it is in the ignore file."
     return $null
   }

eng/ignore-links.txt

@@ -0,0 +1 @@
+https://docs.microsoft.com/python/api/overview/azure/{{package_doc_id}}

eng/pipelines/aggregate-reports.yml

@@ -15,6 +15,10 @@ jobs:
     vmImage: 'windows-2019'
 
   steps:
+  - template: ../common/pipelines/templates/steps/verify-links.yml
+    parameters:
+      Directory: ""
+
   - template: ./templates/steps/analyze.yml
 
   - task: AzureFileCopy@2

eng/pipelines/templates/jobs/archetype-sdk-client.yml

@@ -68,6 +68,10 @@ jobs:
       vmImage: 'ubuntu-18.04'
 
     steps:
+    - template: ../../../common/pipelines/templates/steps/verify-links.yml
+      parameters:
+        Directory: "sdk/${{ parameters.ServiceDirectory }}"
+
     - template: ../steps/analyze.yml
       parameters: 
         ServiceDirectory: ${{ parameters.ServiceDirectory }}

eng/scripts/Language-Settings.ps1

@@ -0,0 +1,108 @@
+$Language = "python"
+$Lang = "python"
+$PackageRepository = "PyPI"
+$packagePattern = "*.zip"
+$MetadataUri = "https://raw.githubusercontent.com/Azure/azure-sdk/master/_data/releases/latest/python-packages.csv"
+
+function Extract-python-PkgProperties ($pkgPath, $serviceName, $pkgName)
+{
+  $pkgName = $pkgName.Replace('_', '-')
+  if (Test-Path (Join-Path $pkgPath "setup.py"))
+  {
+    $setupLocation = $pkgPath.Replace('\','/')
+    pushd $RepoRoot
+    $setupProps = (python -c "import sys; import os; sys.path.append(os.path.join('scripts', 'devops_tasks')); from common_tasks import parse_setup; obj=parse_setup('$setupLocation'); print('{0},{1}'.format(obj[0], obj[1]));") -split ","
+    popd
+    if (($setupProps -ne $null) -and ($setupProps[0] -eq $pkgName))
+    {
+      return [PackageProps]::new($setupProps[0], $setupProps[1], $pkgPath, $serviceName)
+    }
+  }
+  return $null
+}
+
+# Returns the pypi publish status of a package id and version.
+function IsPythonPackageVersionPublished($pkgId, $pkgVersion) {
+  try 
+  {
+    $existingVersion = (Invoke-RestMethod -MaximumRetryCount 3 -Method "Get" -uri "https://pypi.org/pypi/$pkgId/$pkgVersion/json").info.version
+    # if existingVersion exists, then it's already been published
+    return $True
+  }
+  catch 
+  {
+    $statusCode = $_.Exception.Response.StatusCode.value__
+    $statusDescription = $_.Exception.Response.StatusDescription
+
+    # if this is 404ing, then this pkg has never been published before
+    if ($statusCode -eq 404) 
+    {
+      return $False
+    }
+    Write-Host "PyPI Invocation failed:"
+    Write-Host "StatusCode:" $statusCode
+    Write-Host "StatusDescription:" $statusDescription
+    exit(1)
+  }
+}
+
+# Parse out package publishing information given a python sdist of ZIP format.
+function Parse-python-Package($pkg, $workingDirectory) {
+  $pkg.Basename -match $SDIST_PACKAGE_REGEX | Out-Null
+
+  $pkgId = $matches["package"]
+  $pkgVersion = $matches["versionstring"]
+
+  $workFolder = "$workingDirectory$($pkg.Basename)"
+  $origFolder = Get-Location
+  $releaseNotes = ""
+  $readmeContent = ""
+
+  New-Item -ItemType Directory -Force -Path $workFolder
+  Expand-Archive -Path $pkg -DestinationPath $workFolder
+
+  $changeLogLoc = @(Get-ChildItem -Path $workFolder -Recurse -Include "CHANGELOG.md")[0]
+  if ($changeLogLoc) {
+    $releaseNotes = Get-ChangeLogEntryAsString -ChangeLogLocation $changeLogLoc -VersionString $pkgVersion
+  }
+
+  $readmeContentLoc = @(Get-ChildItem -Path $workFolder -Recurse -Include "README.md") | Select-Object -Last 1
+
+  if ($readmeContentLoc) {
+    $readmeContent = Get-Content -Raw $readmeContentLoc
+  }
+
+  Remove-Item $workFolder -Force  -Recurse -ErrorAction SilentlyContinue
+
+  return New-Object PSObject -Property @{
+    PackageId      = $pkgId
+    PackageVersion = $pkgVersion
+    Deployable     = $forceCreate -or !(IsPythonPackageVersionPublished -pkgId $pkgId -pkgVersion $pkgVersion)
+    ReleaseNotes   = $releaseNotes
+    ReadmeContent  = $readmeContent
+  }
+}
+
+# Stage and Upload Docs to blob Storage
+function StageAndUpload-python-Docs()
+{
+  $PublishedDocs = Get-ChildItem "$DocLocation" | Where-Object -FilterScript {$_.Name.EndsWith(".zip")}
+
+  foreach ($Item in $PublishedDocs)
+  {
+    $PkgName = $Item.BaseName
+    $ZippedDocumentationPath = Join-Path -Path $DocLocation -ChildPath $Item.Name
+    $UnzippedDocumentationPath = Join-Path -Path $DocLocation -ChildPath $PkgName
+    $VersionFileLocation = Join-Path -Path $UnzippedDocumentationPath -ChildPath "version.txt"
+
+    Expand-Archive -Force -Path $ZippedDocumentationPath -DestinationPath $UnzippedDocumentationPath
+
+    $Version = $(Get-Content $VersionFileLocation).Trim()
+
+    Write-Host "Discovered Package Name: $PkgName"
+    Write-Host "Discovered Package Version: $Version"
+    Write-Host "Directory for Upload: $UnzippedDocumentationPath"
+
+    Upload-Blobs -DocDir $UnzippedDocumentationPath -PkgName $PkgName -DocVersion $Version
+  }
+}

sdk/advisor/azure-mgmt-advisor/README.md

@@ -2,7 +2,7 @@
 
 This is the Microsoft Azure Advisor Client Library.
 This package has been tested with Python 2.7, 3.5, 3.6, 3.7 and 3.8.
-For a more complete view of Azure libraries, see the [Github repo](https://github.com/Azure/azure-sdk-for-python/)
+For a more complete view of Azure libraries, see the [azure sdk python release](https://aka.ms/azsdk/python/all).
 
 
 # Usage