CI Failures

"Rando" Failures Seen On CI

Everybody wants a green build, but sometimes "things happen" which prevent a build or test from completing successfully. Frequently we'll just restart the offending job and it'll work on the next run, but… Can we improve the reliability of our builds and tests on CI? What issues are we actually encountering, and what are their causes? Let's start formally tracking this.

Networking

Networking: the solution to, and cause of, all life's problems. (…wait.)

Linux & NuGet

A Linux build for PR #7448 failed because of the network:

/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error : Failed to download package 'System.Memory.4.5.5' from 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/system.memory/4.5.5/system.memory.4.5.5.nupkg'. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error : The SSL connection could not be established, see inner exception. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :   Authentication failed, see inner exception. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :   Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :     at /build/mono-6.12.0.182/external/boringssl/ssl/handshake_client.c:1132 [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error : Failed to retrieve information about 'System.Memory' from remote source 'https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/9332014c-138b-4c5b-927b-c5044ec5d809/nuget/v3/flat2/system.memory/index.json'. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :   The SSL connection could not be established, see inner exception. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :   Authentication failed, see inner exception. [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :   Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]
/usr/lib/mono/msbuild/Current/bin/NuGet.targets(131,5): error :     at /build/mono-6.12.0.182/external/boringssl/ssl/handshake_client.c:1132 [/mnt/vss/_work/1/s/xamarin-android/tools/xabuild/xabuild.csproj]

502 Bad Gateway

The BindingCheckHiddenFiles test for commit 52d85154 failed with:

System.Net.WebException : The remote server returned an error: (502) Bad Gateway.
   at System.Net.HttpWebRequest.GetResponse()
   at System.Net.WebClient.GetWebResponse(WebRequest request)
   at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
   at System.Net.WebClient.DownloadFile(Uri address, String fileName)
   at Xamarin.ProjectTools.DownloadedCache.GetAsFile(String url) in /Users/runner/work/1/s/xamarin-android/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/DownloadedCache.cs:line 36
   at Xamarin.ProjectTools.BuildItem.<>c__DisplayClass70_0.<set_WebContent>b__0() in /Users/runner/work/1/s/xamarin-android/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/BuildItem.cs:line 129
   at Xamarin.ProjectTools.XamarinProject.<>c.<Save>b__109_1(BuildItem s) in /Users/runner/work/1/s/xamarin-android/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/XamarinProject.cs:line 257
   at System.Linq.Enumerable.SelectListIterator`2.MoveNext()
   at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection)
   at Xamarin.ProjectTools.XamarinProject.Save(Boolean saveProject) in /Users/runner/work/1/s/xamarin-android/src/Xamarin.Android.Build.Tasks/Tests/Xamarin.ProjectTools/Common/XamarinProject.cs:line 275
…

1. Should our unit test harness really be using WebClient and not HttpClient?
2. Regardless of (1), should our test fixture check for errors >= HTTP 500 and automatically attempt to retry (after some delay)?

Test Job Timeouts

For commit 9159acaa9142d0e5857f9595de73ebb8dfb3975f, the macOS > Tests > MSBuild+Emulator One .NET #1 job timed out.

When a job times out, if there are any test failures from before the timeout, are they recorded? If not, can they be recorded? Do we have a policy for reviewing job timeouts?

Signing Job Timeouts

For commit 9159acaa9142d0e5857f9595de73ebb8dfb3975f, the Sign Archives > Sign Nuget Packages step timed out. Should these be reported to a relevant team? (Is there a relevant team?)

ANDZA0000

Tracked as: https://github.com/xamarin/xamarin-android/issues/6067

Haven't seen this one recently, but something we've seen historically.