Merge pull request Azure#347 from navalev/feature/merge-upstream-22

Merge from upstream repo 12/17
navalev · Dec 18, 2019 · 140dddd · 140dddd
2 parents 8dd3b09 + 12f96fc
commit 140dddd
Show file tree

Hide file tree

Showing 343 changed files with 15,857 additions and 8,924 deletions.
diff --git a/.gitignore b/.gitignore
@@ -61,4 +61,8 @@ Thumbs.db
 
 # reduced pom files should not be included
 dependency-reduced-pom.xml
-package-lock.json
+package-lock.json
+
+# VS Code
+.vscode/
+.factorypath
diff --git a/README.md b/README.md
@@ -22,32 +22,33 @@ Java 8 or later is required to use the November 2019 client libraries, otherwise
 
 Each service might have a number of libraries available from each of the following categories:
 
-- [Client: GA November 2019 Releases](#Client-GA-November-2019-Releases)
+- [Client: GA Decemeber 2019 Releases](#Client-GA-Decemeber-2019-Releases)
 - [Client - Previous Versions](#Client-Previous-Versions)
 - [Management](#Management)
 
-### Client: GA November 2019 Releases
+### Client: GA December 2019 Releases
 
-New wave of packages that were released in November 2019 client library as General Availability (GA) and several others that were released in **preview**. These libraries follow the [Azure SDK Design Guidelines for Java](https://azure.github.io/azure-sdk/java/guidelines/) and share a number of core features such as HTTP retries, logging, transport protocols, authentication protocols, etc., so that once you learn how to use these features in one client library, you will know how to use them in other client libraries. You can learn about these shared features [here](sdk/core/README.md).
+New wave of packages that were released in December 2019 client library as General Availability (GA) and several others that were released in **beta**. These libraries follow the [Azure SDK Design Guidelines for Java](https://azure.github.io/azure-sdk/java/guidelines/) and share a number of core features such as HTTP retries, logging, transport protocols, authentication protocols, etc., so that once you learn how to use these features in one client library, you will know how to use them in other client libraries. You can learn about these shared features [here](sdk/core/README.md).
 
 These libraries can be easily identified by sdk/ folder, package, and namespaces names starting with `azure-`, e.g. `azure-keyvault`.
 
-The libraries released in the GA November 2019 release:
-- [Identity](https://github.com/Azure/azure-sdk-for-java/blob/azure-identity_1.0.0/sdk/identity/azure-identity/README.md)
-- [Key Vault Keys](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-keys_4.0.0/sdk/keyvault/azure-security-keyvault-keys/README.md)
-- [Key Vault Secrets](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-secrets_4.0.0/sdk/keyvault/azure-security-keyvault-secrets/README.md)
-- [Storage Blobs](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.0.0/sdk/storage/azure-storage-blob/README.md)
-- [Storage Blobs Batch](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.0.0/sdk/storage/azure-storage-blob-batch/README.md)
-- [Storage Blobs Cryptography](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.0.0/sdk/storage/azure-storage-blob-cryptography/README.md)
-- [Storage Queues](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.0.0/sdk/storage/azure-storage-queue/README.md)
-
-The libraries released in the November 2019 preview:
-- [App Configuration](https://github.com/Azure/azure-sdk-for-java/tree/azure-data-appconfiguration_1.0.0-preview.6/sdk/appconfiguration/azure-data-appconfiguration)
-- [Event Hubs](https://github.com/Azure/azure-sdk-for-java/blob/azure-messaging-eventhubs_5.0.0-preview.5/sdk/eventhubs/azure-messaging-eventhubs/README.md)
-- [Event Hubs Checkpoint Store](https://github.com/Azure/azure-sdk-for-java/blob/azure-messaging-eventhubs-checkpointstore-blob_1.0.0-preview.3/sdk/eventhubs/azure-messaging-eventhubs-checkpointstore-blob/README.md)
-- [Storage File Share](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-file-share_12.0.0-preview.5/sdk/storage/azure-storage-file-share/README.md)
-- [Key Vault Certificates](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-certificates_4.0.0-preview.5/sdk/keyvault/azure-security-keyvault-certificates/README.md)
-- [OpenCensus Tracing](https://github.com/Azure/azure-sdk-for-java/blob/azure-core-tracing-opencensus_1.0.0-preview.4/sdk/core/azure-core-tracing-opencensus/README.md)
+The libraries released in the GA December 2019 release:
+- [Identity](https://github.com/Azure/azure-sdk-for-java/blob/azure-identity_1.0.1/sdk/identity/azure-identity/README.md)
+- [Key Vault Keys](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-keys_4.0.1/sdk/keyvault/azure-security-keyvault-keys/README.md)
+- [Key Vault Secrets](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-secrets_4.0.1/sdk/keyvault/azure-security-keyvault-secrets/README.md)
+- [Storage Blobs](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob_12.1.0/sdk/storage/azure-storage-blob/README.md)
+- [Storage Blobs Batch](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob-batch_12.1.0/sdk/storage/azure-storage-blob-batch/README.md)
+- [Storage Blobs Cryptography](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-blob-cryptography_12.1.0/sdk/storage/azure-storage-blob-cryptography/README.md)
+- [Storage Queues](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-queue_12.1.0/sdk/storage/azure-storage-queue/README.md)
+- [Storage File Share](https://github.com/Azure/azure-sdk-for-java/blob/azure-storage-file-share_12.0.0/sdk/storage/azure-storage-file-share/README.md)
+
+The libraries released in the December 2019 beta:
+- [App Configuration](https://github.com/Azure/azure-sdk-for-java/tree/azure-data-appconfiguration_1.0.0-beta.7/sdk/appconfiguration/azure-data-appconfiguration/README.md)
+- [Event Hubs](https://github.com/Azure/azure-sdk-for-java/blob/azure-messaging-eventhubs_5.0.0-beta.6/sdk/eventhubs/azure-messaging-eventhubs/README.md)
+- [Event Hubs Checkpoint Store](https://github.com/Azure/azure-sdk-for-java/blob/azure-messaging-eventhubs-checkpointstore-blob_1.0.0-beta.4/sdk/eventhubs/azure-messaging-eventhubs-checkpointstore-blob/README.md)
+- [Key Vault Certificates](https://github.com/Azure/azure-sdk-for-java/blob/azure-security-keyvault-certificates_4.0.0-beta.6/sdk/keyvault/azure-security-keyvault-certificates/README.md)
+- [OpenCensus Tracing](https://github.com/Azure/azure-sdk-for-java/blob/azure-core-tracing-opencensus_1.0.0-beta.5/sdk/core/azure-core-tracing-opencensus/README.md)
+- [OpenTelemetry Tracing](https://github.com/Azure/azure-sdk-for-java/blob/azure-core-tracing-opentelemetry_1.0.0-beta.1/sdk/core/azure-core-tracing-opentelemetry/README.md)
 
 > NOTE: If you need to ensure your code is ready for production use one of the stable, non-preview libraries.
 

diff --git a/eng/docgeneration/service-mapper.json b/eng/docgeneration/service-mapper.json
@@ -4,6 +4,7 @@
   "apimanagement": "Api Management",
   "appconfiguration": "App Configuration",
   "applicationinsights": "Application Insights",
+  "appplatform": "App Platform",
   "appservice": "App Service",
   "authorization": "Authorization",
   "automation": "Automation",
@@ -23,11 +24,14 @@
   "core": "Core",
   "cosmos": "Cosmos",
   "cosmosdb": "Cosmos DB",
+  "costmanagement":"Cost Management",
   "customer-insights": "Customer Insights",
   "databox": "Data Box",
   "databricks": "Databricks",
   "datacatalog": "Data Catalog",
   "datafactory": "Data Factory",
+  "datalakeanalytics":"Data Lake Analytics",
+  "datalakestore":"Data Lake Store",
   "datamigration": "Database Migration",
   "deploymentmanager": "Deployment Manager",
   "deviceprovisioningservices": "Device Provisioning Services",
@@ -58,6 +62,7 @@
   "machinelearningcompute": "Machine Learning Compute",
   "machinelearningexperimentation": "Machine Learning Experimentation",
   "machinelearningservices": "Machine Learning Services",
+  "maintenance": "Maintenance",
   "managedapplications": "Managed Applications",
   "managementgroups": "Management Groups",
   "managementpartner": "Management Partner",
@@ -83,8 +88,8 @@
   "powerbiembedded": "PowerBI Embedded",
   "privatedns": "Private DNS",
   "recoveryservices": "Recovery Services",
-  "recoveryservicesbackup": "Recovery Services Backup",
-  "recoveryservicessiterecovery": "Recovery Services Site Recovery",
+  "recoveryservices.backup": "Recovery Services Backup",
+  "recoveryservices.siterecovery": "Recovery Services Site Recovery",
   "redis": "Redis",
   "relay": "Relay",
   "reservations": "Reservations",
@@ -102,6 +107,7 @@
   "sql": "SQL",
   "sqlvirtualmachine": "SQL Virtual Machine",
   "storage": "Storage",
+  "storagecache": "Storage Cache",
   "storageimportexport": "Storage Import Export",
   "storagesync": "Storage Sync",
   "storsimple1200series": "Storsimple 1200 Series",

diff --git a/eng/mgmt/gulpfile.js b/eng/mgmt/gulpfile.js
@@ -39,9 +39,10 @@ gulp.task('default', function() {
     console.log("\tThe version of AutoRest. E.g. 2.0.9, or the location of AutoRest repo, e.g. E:\\repo\\autorest");
 
     console.log("--autorest-java");
-    console.log("\tPath to an autorest.java generator to pass as a --use argument to AutoRest.");
+    console.log("\tOption#1: Path to an autorest.java generator to pass as a --use argument to AutoRest.");
+    console.log("\tOption#2: The version of AutoRest.Java. E.g. 2.0.9. You can also pass latest or preview.");
     console.log("\tUsually you'll only need to provide this and not a --autorest argument in order to work on Java code generation.");
-    console.log("\tSee https://github.com/Azure/autorest/blob/master/docs/developer/autorest-extension.md");
+    console.log("\tSee https://github.com/Azure/autorest/blob/master/.attic/developer/autorest-extension.md");
 
     console.log("--debug");
     console.log("\tFlag that allows you to attach a debugger to the autorest.java generator.");
@@ -62,15 +63,19 @@ var autoRestVersion = 'preview'; // default
 if (args['autorest'] !== undefined) {
     autoRestVersion = args['autorest'];
 }
+var autoRestJavaVersion = ''; // default
+if (args['autorest-java'] !== undefined) {
+    autoRestJavaVersion = args['autorest-java'];
+}
 var debug = args['debug'];
 var autoRestArgs = args['autorest-args'] || '';
 var autoRestExe;
 const mgmtPomFilename = 'pom.mgmt.xml'
 
 gulp.task('codegen', function(cb) {
     if (autoRestVersion.match(/[0-9]+\.[0-9]+\.[0-9]+.*/) ||
-        autoRestVersion == 'preview') {
-            autoRestExe = 'autorest ---version=' + autoRestVersion;
+        autoRestVersion == 'preview' || autoRestVersion == 'latest') {
+            autoRestExe = 'autorest --version=' + autoRestVersion;
             handleInput(projects, cb);
     } else {
         autoRestExe = "node " + path.join(autoRestVersion, "src/autorest-core/dist/app.js");
@@ -107,9 +112,10 @@ var codegen = function(project, cb) {
 
     console.log('Generating "' + project + '" from spec file ' + specRoot + '/' + mappings[project].source);
 
-    const generatorPath = args['autorest-java']
-        ? `--use=${path.resolve(args['autorest-java'])} `
-        : '';
+    const generatorPath = autoRestJavaVersion == 'preview' || autoRestJavaVersion == 'latest' 
+		|| autoRestJavaVersion.match(/^[0-9]+\.[0-9]+\.[0-9a-zA-Z]+$/)
+        ? `[email protected]/autorest.java@` + autoRestJavaVersion +` `
+        : (autoRestJavaVersion == '' ? '' : `--use=${path.resolve(args['autorest-java'])} `);
 
     const regenManager = args['regenerate-manager'] ? ' --regenerate-manager=true ' : '';
 

diff --git a/eng/pipelines/client.yml b/eng/pipelines/client.yml
@@ -12,7 +12,7 @@ resources:
       endpoint: azure
 
 variables:
-  DefaultOptions: '--batch-mode -Dmaven.wagon.http.pool=false --settings eng/settings.xml'
+  DefaultOptions: '--batch-mode --fail-at-end -Dmaven.wagon.http.pool=false --settings eng/settings.xml'
   LoggingOptions: '-Dorg.slf4j.simpleLogger.defaultLogLevel=error -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn'
 
 jobs:

diff --git a/eng/pipelines/docindex.yml b/eng/pipelines/docindex.yml
@@ -160,7 +160,7 @@ jobs:
                 if ($mgmtArr.Count -gt 0) 
                 {
                     Add-Content -Path "$($YmlPath)/$($Dir.Name).md" -Value "# Management Libraries"
-                    foreach($lib in $clientArr) 
+                    foreach($lib in $mgmtArr) 
                     {
                         Write-Output "Write $($lib) to $($Dir.Name).md"
                         Add-Content -Path "$($YmlPath)/$($Dir.Name).md" -Value "#### $lib"

diff --git a/eng/pipelines/templates/jobs/archetype-sdk-client-search.yml b/eng/pipelines/templates/jobs/archetype-sdk-client-search.yml
@@ -46,10 +46,10 @@ jobs:
 
     strategy:
       matrix:
-        Java 8:
-          ArtifactName: 'packages'
-          ProfileFlag: ''
-          JavaVersion: '1.8'
+#        Java 8:
+#          ArtifactName: 'packages'
+#          ProfileFlag: ''
+#          JavaVersion: '1.8'
         Java LTS:
           ArtifactName: 'java-lts-packages'
           ProfileFlag: '-Djava-lts'

diff --git a/eng/pipelines/templates/jobs/archetype-sdk-client.yml b/eng/pipelines/templates/jobs/archetype-sdk-client.yml
@@ -101,6 +101,10 @@ jobs:
         displayName: 'Publish Artifacts $(ArtifactName)'
         artifact: '$(ArtifactName)'
 
+      - publish: $(System.DefaultWorkingDirectory)
+        condition: and(eq(variables['CaptureRepositoryOnFailure'], 'true'), failed())
+        displayName: 'Capture repo state for analysis'
+        artifact: repository
 
   - job: 'Analyze'
     condition: ne(variables['Skip.Analyze'], 'true')
@@ -286,7 +290,7 @@ jobs:
           goals: ${{ parameters.TestGoals }}
 
       - task: PublishTestResults@2
-        condition: succeededOrFailed()
+        condition: always()
         inputs:
           mergeTestResults: true
           testRunTitle: '$(OSName) on Java $(JavaVersion)'
diff --git a/eng/pipelines/templates/jobs/archetype-sdk-tests-pre-sdk.yml b/eng/pipelines/templates/jobs/archetype-sdk-tests-pre-sdk.yml
@@ -27,4 +27,4 @@ jobs:
         condition: succeededOrFailed()
         inputs:
           mergeTestResults: true
-          testRunTitle: 'Live tests for ${{ parameters.PomFilePath }}'
+          testRunTitle: 'Live tests for ${{ parameters.PomFilePath }}'
diff --git a/eng/pipelines/templates/jobs/archetype-sdk-tests.yml b/eng/pipelines/templates/jobs/archetype-sdk-tests.yml
@@ -3,17 +3,44 @@ parameters:
   EnvVars: {}
   MaxParallel: 0
   Matrix:
-    Win2016:
+    Linux - Java 8:
+      OSName: 'Linux'
+      OSVmImage: 'ubuntu-16.04'
+      JavaVersion: '1.8'
+      DisplayName: 'Linux - Java 8'
+    macOS - Java 8:
+      OSName: 'macOS'
+      OSVmImage: 'macOS-10.13'
+      JavaVersion: '1.8'
+      DisplayName: 'macOS - Java 8'
+    Windows - Java 8:
+      OSName: 'Windows'
       OSVmImage: 'windows-2019'
-      DisplayName: 'Run Live tests'
+      JavaVersion: '1.8'
+      DisplayName: 'Windows - Java 8'
+    Linux - Java 11:
+      OSName: 'Linux'
+      OSVmImage: 'ubuntu-16.04'
+      JavaVersion: '1.11'
+      DisplayName: 'Linux - Java 11'
+    macOS - Java 11:
+      OSName: 'macOS'
+      OSVmImage: 'macOS-10.13'
+      JavaVersion: '1.11'
+      DisplayName: 'macOS - Java 11'
+    Windows - Java 11:
+      OSName: 'Windows'
+      OSVmImage: 'windows-2019'
+      JavaVersion: '1.11'
+      DisplayName: 'Windows - Java 11'
   PreRunSteps: []
   TestName: LiveTest
   TimeoutInMinutes: 60
   TestStepMavenInputs:
-    options: '--batch-mode --fail-at-end -Dmaven.wagon.http.pool=false -Dsurefire.rerunFailingTestsCount=3 --settings eng/settings.xml'
-    mavenOptions: '-Xmx3072m -Dorg.slf4j.simpleLogger.defaultLogLevel=error -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn'
+    options: '-Dmaven.wagon.http.pool=false -Dsurefire.rerunFailingTestsCount=3 $(DefaultOptions)'
+    mavenOptions: '$(MemoryOptions) $(LoggingOptions)'
     javaHomeOption: 'JDKVersion'
-    jdkVersionOption: '1.8'
+    jdkVersionOption: '$(JavaVersion)'
     jdkArchitectureOption: 'x64'
     publishJUnitResults: false
     goals: 'test'
@@ -43,7 +70,7 @@ jobs:
         env: ${{ parameters.EnvVars }}
 
       - task: PublishTestResults@2
-        condition: succeededOrFailed()
+        condition: always()
         inputs:
           mergeTestResults: true
           testRunTitle: 'Live tests for ${{ parameters.ServiceDirectory }} $(DisplayName)'

diff --git a/eng/pipelines/templates/variables/globals.yml b/eng/pipelines/templates/variables/globals.yml
@@ -4,5 +4,5 @@ variables:
   LoggingOptions: '-Dorg.slf4j.simpleLogger.defaultLogLevel=error -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn'
   MemoryOptions: '-Xmx3072m'
   #Agent.Source.Git.ShallowFetchDepth: 1
-  skipComponentGovernanceDetection: true  
+  skipComponentGovernanceDetection: true
   EmulatorMsiUrl: 'https://aka.ms/cosmosdb-emulator'
diff --git a/eng/versioning/external_dependencies.txt b/eng/versioning/external_dependencies.txt
@@ -17,7 +17,7 @@ com.microsoft.azure:azure-client-runtime;1.6.15
 com.microsoft.azure:azure-core;0.9.8
 com.microsoft.azure:azure-keyvault-cryptography;1.2.2
 com.microsoft.azure:client-runtime;1.6.15
-com.microsoft.azure:qpid-proton-j-extensions;1.2.1
+com.microsoft.azure:qpid-proton-j-extensions;1.2.2
 com.microsoft.azure:spotbugs-maven-plugin;1.2.1
 com.microsoft.rest:client-runtime;1.6.15
 com.microsoft.rest.v2:client-runtime;2.1.1
@@ -211,4 +211,4 @@ media_com.microsoft.azure:adal4j;1.2.0
 servicebus_com.microsoft.azure:azure-client-authentication;1.6.7
 
 # sdk\storage\azure-storage-blob-cryptography\pom.xml
-storage_com.microsoft.azure:azure-storage;8.4.0
+storage_com.microsoft.azure:azure-storage;8.4.0
diff --git a/eng/versioning/update_versions.py b/eng/versioning/update_versions.py
@@ -23,7 +23,7 @@
 # necessary here, since our README.md files don't contain externaly dependency versions, there's no point in scanning files
 # that shouldn't require changes.
 #    python eng/versioning/update_versions.py --ut external_dependency --sr
-# 
+#
 # The script must be run at the root of azure-sdk-for-java.
 
 import argparse
@@ -54,16 +54,16 @@ def update_versions(version_map, target_file):
                 if match:
                     module_name, version_type = match.group(1), match.group(2)
                     repl_thisline = True
-                else:	
-                    match = version_update_start_marker.search(line)	
-                    if match:	
-                        module_name, version_type = match.group(1), match.group(2)	
-                        repl_open, repl_thisline = True, True	
-                    else:	
-                        match = version_update_end_marker.search(line)	
-                        if match:	
+                else:
+                    match = version_update_start_marker.search(line)
+                    if match:
+                        module_name, version_type = match.group(1), match.group(2)
+                        repl_open, repl_thisline = True, True
+                    else:
+                        match = version_update_end_marker.search(line)
+                        if match:
                             repl_open, repl_thisline = False, False
-                
+
                 if repl_thisline:
                     # If the module isn't found then just continue. This can
                     # happen if we're going through and replacing only library
@@ -121,7 +121,7 @@ def load_version_map_from_file(the_file, version_map):
 def display_version_info(version_map):
     for value in version_map.values():
         print(value)
-    
+
 def update_versions_all(update_type, build_type, target_file, skip_readme):
     version_map = {}
     # Load the version and/or external dependency file for the given UpdateType
@@ -145,10 +145,10 @@ def update_versions_all(update_type, build_type, target_file, skip_readme):
         for root, _, files in os.walk("."):
             for file_name in files:
                 file_path = root + os.sep + file_name
-                if (file_name == 'README.md' and not skip_readme) or (file_name.startswith('pom.') and file_name.endswith('.xml')):
+                if (file_name.endswith('.md') and not skip_readme) or (file_name.startswith('pom.') and file_name.endswith('.xml')):
                     update_versions(version_map, file_path)
 
-    # This is a temporary stop gap to deal with versions hard coded in java files. 
+    # This is a temporary stop gap to deal with versions hard coded in java files.
     # Everything within the begin/end tags below can be deleted once
     # https://github.com/Azure/azure-sdk-for-java/issues/3141 has been fixed.
     # version_*_java_files.txt
@@ -184,4 +184,4 @@ def main():
     print('Total time for replacement: {}'.format(str(timedelta(seconds=elapsed_time))))
 
 if __name__ == '__main__':
-    main()
+    main()