Getting HTTP 500 errors sending tests to Helix

[jaredpar]

Error text

      error Code .packages/microsoft.dotnet.helix.sdk/5.0.0-beta.20151.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(54,5): error : (NETCORE_ENGINEERING_TELEMETRY=Helix) RestApiException: The response contained an invalid status code 500 Internal Server Error

Body: {"Message":"An error has occurred."}
   at Microsoft.DotNet.Helix.Client.Job.OnPassFailFailed(Request req, Response res) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 322
   at Microsoft.DotNet.Helix.Client.Job.PassFailAsync(String job, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Client/CSharp/generated-code/Job.cs:line 286
   at Microsoft.DotNet.Helix.Sdk.WaitForHelixJobCompletion.WaitForHelixJobAsync(String jobName, CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/WaitForHelixJobCompletion.cs:line 38
   at Microsoft.DotNet.Helix.Sdk.WaitForHelixJobCompletion.ExecuteCore(CancellationToken cancellationToken) in /_/src/Microsoft.DotNet.Helix/Sdk/WaitForHelixJobCompletion.cs:line 26
   at Microsoft.DotNet.Helix.Sdk.HelixTask.Execute() in /_/src/Microsoft.DotNet.Helix/Sdk/HelixTask.cs:line 51

Build	Kind	Timeline Record
543996	Rolling	Send to Helix
543996	Rolling	Send to Helix
543935	PR #810	Send tests to Helix
543978	PR #31701	Send tests to Helix
543893	PR #33060	Send tests to Helix
543871	PR #33052	Send to Helix

This is impacting 4% of our builds

runfo search-timeline -d runtime -c 100 -pr -v "TELEMETRY=Helix.*500"

FR issue https://github.com/dotnet/core-eng/issues/9201

[MattGal]

This is not 8591, rather https://github.com/dotnet/core-eng/issues/9201 . I've already merged a fix for this to Arcade so it should radically drop HTTP 500s once this flows to you via arcade.

[jaredpar]

Closing as there are no recent occurrences of this issue.