[CI] Run TimeZoneInfo tests in parallel. (#4161)

Create RunAllTests variable

Creates a new RunAllTests variable that defaults to false. This variable is used as a condition to trigger running our extended regression test suite.

This variable is set to true in the following ways:

    The master pipeline sets the value to true so all commits to master run the full suite.
    The value is set to true for a PR build if:
        The branch name contains mono-
        The branch is not a fork
        This allows our Mono bump branches PRs (which take the form of mono-2019-12) to run the full suite, as there is a much higher chance of breakage for these commits.
    If you are queuing a new run from the UI, clicking the Variables option will allow you to set it to true for that run. (The variable shows up with the default value so you don't have to remember its name.)

Put extended suites behind RunAllTests variable

The following test suites are behind this new variable:

    Integrated Regression - MacOS
    Integrated Regression - Windows
    TimeZoneInfo - MacOS

The effective changes are:

    Previously the integrated regression suite also ran for non-fork PR builds.
    Previously the TZI suite ran on all PR builds.

We have determined that these suites are unlikely to break for normal day-to-day changes, so by default they will now only run on master commits and Mono bump PRs.

As stated above, if you feel your change may be risky you can manually kick off a full test in the Pipelines UI.

Updates the regression test stage introduced in commit 47e00d7 to only
run when the mac_build stage completes successfully, as it depends
on artifacts produced by that stage.
Parallelize TimeZoneInfo tests

The TimeZoneInfo test suite has been split across 3 build nodes to reduce each run below the target of 1 hour. (Previously this step took over 2 hours.)

Author: jpobst

build-tools/automation/azure-pipelines.yaml

@@ -53,6 +53,12 @@ variables:
   HostedMac: Hosted Mac Internal
   HostedWinVS2019: Hosted Windows 2019 with VS2019
   VSEngWinVS2019: VSEng-Xamarin-Android
+  # Run all tests if:
+  # - User who queued the job requested it (They set XA.RunAllTests to true)
+  # - This is the master integration branch (Pipeline defaults XA.RunAllTests to true)
+  # - This is a non-fork branch with name containing "mono-" (for Mono bumps)
+  IsMonoBranch: $[and(eq(variables['XA.Commercial.Build'], 'true'), ne(variables['System.PullRequest.IsFork'], 'True'), contains(variables['Build.SourceBranchName'], 'mono-'))]
+  RunAllTests: $[or(eq(variables['XA.RunAllTests'], true), eq(variables['IsMonoBranch'], true))]
 
 # Stage and Job "display names" are shortened because they are combined to form the name of the corresponding GitHub check.
 stages:
@@ -819,7 +825,7 @@ stages:
       parameters:
         testRunTitle: MSBuildDeviceIntegration On Device - macOS
         testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/MSBuildDeviceIntegration.dll
-        nunitConsoleExtraArgs: --where "test != Xamarin.Android.Build.Tests.DeploymentTest.CheckTimeZoneInfoIsCorrect"
+        nunitConsoleExtraArgs: --where "cat != TimeZoneInfo"
         testResultsFile: TestResult-MSBuildDeviceIntegration-$(XA.Build.Configuration).xml
 
     - task: MSBuild@1
@@ -836,51 +842,6 @@ stages:
       parameters:
         artifactName: Test Results - MSBuild With Emulator - macOS
 
-  # Check - "Xamarin.Android (Test TimeZoneInfo With Emulator - macOS)"
-  - job: mac_timezonedevice_tests
-    displayName: TimeZoneInfo With Emulator - macOS
-    pool: $(HostedMac)
-    timeoutInMinutes: 240
-    cancelTimeoutInMinutes: 5
-    workspace:
-      clean: all
-    steps:
-    - template: yaml-templates/setup-test-environment.yaml
-
-    - task: DownloadPipelineArtifact@1
-      inputs:
-        artifactName: $(TestAssembliesArtifactName)
-        downloadPath: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)
-
-    - task: MSBuild@1
-      displayName: start emulator
-      inputs:
-        solution: src/Mono.Android/Test/Mono.Android-Tests.csproj
-        configuration: $(XA.Build.Configuration)
-        msbuildArguments: >
-          /t:AcquireAndroidTarget /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/start-emulator.binlog
-
-    - template: yaml-templates/run-nunit-tests.yaml
-      parameters:
-        testRunTitle: TimeZoneInfoTests On Device - macOS
-        testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/MSBuildDeviceIntegration.dll
-        nunitConsoleExtraArgs: --where "test == Xamarin.Android.Build.Tests.DeploymentTest.CheckTimeZoneInfoIsCorrect"
-        testResultsFile: TestResult-TimeZoneInfoTests-$(XA.Build.Configuration).xml
-
-    - task: MSBuild@1
-      displayName: shut down emulator
-      inputs:
-        solution: src/Mono.Android/Test/Mono.Android-Tests.csproj
-        configuration: $(XA.Build.Configuration)
-        msbuildArguments: >
-          /t:AcquireAndroidTarget,ReleaseAndroidTarget
-          /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/shutdown-emulator.binlog
-      condition: always()
-
-    - template: yaml-templates/upload-results.yaml
-      parameters:
-        artifactName: Test Results - TimeZoneInfo With Emulator - macOS
-
   # Check - "Xamarin.Android (Test Designer - macOS)"
   - job: designer_integration_mac
     displayName: Designer - macOS
@@ -984,12 +945,12 @@ stages:
 - stage: integrated_regression_test
   displayName: Regression Tests
   dependsOn: mac_build
-  condition: and(eq(variables['XA.Commercial.Build'], 'true'), ne(variables['System.PullRequest.IsFork'], 'True'))
+  condition: and(eq(dependencies.mac_build.result, 'Succeeded'), eq(variables['RunAllTests'], true))
   jobs:
 
   # Check - "Xamarin.Android (Test Integrated Regression - macOS)"
   - job: integrated_regression_mac_1
-    displayName: macOS - Node 1
+    displayName: Mac-1
     pool:
       name: VSEng-Xamarin-Mac-Devices
       demands:
@@ -1005,7 +966,7 @@ stages:
 
   # Check - "Xamarin.Android (Test Integrated Regression - macOS)"
   - job: integrated_regression_mac_2
-    displayName: macOS - Node 2
+    displayName: Mac-2
     pool:
       name: VSEng-Xamarin-Mac-Devices
       demands:
@@ -1021,7 +982,7 @@ stages:
 
   # Check - "Xamarin.Android (Test Integrated Regression - macOS)"
   - job: integrated_regression_mac_3
-    displayName: macOS - Node 3
+    displayName: Mac-3
     pool:
       name: VSEng-Xamarin-Mac-Devices
       demands:
@@ -1037,7 +998,7 @@ stages:
 
   # Check - "Xamarin.Android (Test Integrated Regression - Windows)"
   - job: integrated_regression_Win_1
-    displayName: Windows - Node 1
+    displayName: Windows-1
     pool:
       name: VSEng-Xamarin-Win-XMA
       demands:
@@ -1059,7 +1020,7 @@ stages:
 
   # Check - "Xamarin.Android (Test Integrated Regression - Windows)"
   - job: integrated_regression_Win_2
-    displayName: Windows - Node 2
+    displayName: Windows-2
     pool:
       name: VSEng-Xamarin-Win-XMA
       demands:
@@ -1081,7 +1042,7 @@ stages:
 
   # Check - "Xamarin.Android (Test Integrated Regression - Windows)"
   - job: integrated_regression_Win_3
-    displayName: Windows - Node 3
+    displayName: Windows-3
     pool:
       name: VSEng-Xamarin-Win-XMA
       demands:
@@ -1100,3 +1061,16 @@ stages:
         node_id: 3
 
     - template: remove-visualstudio.yml@yaml
+
+  # TimeZoneInfo tests
+  - template: yaml-templates\run-timezoneinfo-tests.yaml
+    parameters:
+      node_id: 1
+
+  - template: yaml-templates\run-timezoneinfo-tests.yaml
+    parameters:
+      node_id: 2
+
+  - template: yaml-templates\run-timezoneinfo-tests.yaml
+    parameters:
+      node_id: 3

build-tools/automation/yaml-templates/run-timezoneinfo-tests.yaml

@@ -0,0 +1,49 @@
+# Runs TimeZoneInfo tests against an emulator running on macOS
+
+parameters:
+  node_id: 0
+
+jobs:
+  - job: mac_timezoneinfo_tests_${{ parameters.node_id }}
+    displayName: TimeZone - Mac-${{ parameters.node_id }}
+    pool: $(HostedMac)
+    timeoutInMinutes: 90
+    cancelTimeoutInMinutes: 5
+    workspace:
+      clean: all
+    steps:
+    - template: setup-test-environment.yaml
+
+    - task: DownloadPipelineArtifact@1
+      inputs:
+        artifactName: $(TestAssembliesArtifactName)
+        downloadPath: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)
+
+    - task: MSBuild@1
+      displayName: start emulator
+      inputs:
+        solution: src/Mono.Android/Test/Mono.Android-Tests.csproj
+        configuration: $(XA.Build.Configuration)
+        msbuildArguments: >
+          /t:AcquireAndroidTarget /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/start-emulator.binlog
+
+    - template: run-nunit-tests.yaml
+      parameters:
+        testRunTitle: TimeZoneInfoTests On Device - macOS
+        testAssembly: $(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/MSBuildDeviceIntegration/MSBuildDeviceIntegration.dll
+        nunitConsoleExtraArgs: --where "test == Xamarin.Android.Build.Tests.DeploymentTest.CheckTimeZoneInfoIsCorrectNode${{ parameters.node_id }}"
+        testResultsFile: TestResult-TimeZoneInfoTests-Node${{ parameters.node_id }}-$(XA.Build.Configuration).xml
+
+    - task: MSBuild@1
+      displayName: shut down emulator
+      inputs:
+        solution: src/Mono.Android/Test/Mono.Android-Tests.csproj
+        configuration: $(XA.Build.Configuration)
+        msbuildArguments: >
+          /t:AcquireAndroidTarget,ReleaseAndroidTarget
+          /bl:$(System.DefaultWorkingDirectory)/bin/Test$(XA.Build.Configuration)/shutdown-emulator.binlog
+      condition: always()
+
+    - template: upload-results.yaml
+      parameters:
+        artifactName: Test Results - TimeZoneInfo With Emulator - macOS - ${{ parameters.node_id }}

tests/MSBuildDeviceIntegration/Tests/DeploymentTest.cs

@@ -182,7 +182,9 @@ public void CheckXamarinFormsAppDeploysAndAButtonWorks ()
 			}, Path.Combine (Root, builder.ProjectDirectory, "button-logcat.log")), "Button Should have been Clicked.");
 		}
 
-		static object [] GetTimeZoneTestCases ()
+		private const int NODE_COUNT = 3;
+
+		static object [] GetTimeZoneTestCases (int node)
 		{
 			List<object> tests = new List<object> ();
 			var ignore = new string [] {
@@ -197,12 +199,27 @@ static object [] GetTimeZoneTestCases ()
 					tz,
 				});
 			}
-			return tests.ToArray ();
+			return tests.Where (p => tests.IndexOf (p) % NODE_COUNT == node).ToArray ();
 		}
 
 		[Test]
-		[TestCaseSource (nameof (GetTimeZoneTestCases))]
+		[TestCaseSource (nameof (GetTimeZoneTestCases), new object [] { 0 })]
+		[Retry (1)]
+		[Category ("TimeZoneInfo")]
+		public void CheckTimeZoneInfoIsCorrectNode1 (string timeZone) => CheckTimeZoneInfoIsCorrect (timeZone);
+
+		[Test]
+		[TestCaseSource (nameof (GetTimeZoneTestCases), new object [] { 1 })]
 		[Retry (1)]
+		[Category ("TimeZoneInfo")]
+		public void CheckTimeZoneInfoIsCorrectNode2 (string timeZone) => CheckTimeZoneInfoIsCorrect (timeZone);
+
+		[Test]
+		[TestCaseSource (nameof (GetTimeZoneTestCases), new object [] { 2 })]
+		[Retry (1)]
+		[Category ("TimeZoneInfo")]
+		public void CheckTimeZoneInfoIsCorrectNode3 (string timeZone) => CheckTimeZoneInfoIsCorrect (timeZone);
+
 		public void CheckTimeZoneInfoIsCorrect (string timeZone)
 		{
 			if (!HasDevices)