Occasional restore failures

[mmitche]

• This issue is blocking
• This issue is causing unreasonable pain

I'm seeing restore failures in some cases, across a number of build pipelines. The failures usually succeed on retry. Example (I've retained this build):

https://dev.azure.com/dnceng/public/_build/results?buildId=826657&view=results

The packages in question are in dotnet5 and dotnet5-transport and were pushed prior to the dependency update PR starting. The most interesting thing about this is that the first attempt to get info about at least some of these packages from the feed is that we're hitting the cache:

  CACHE https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/7d9f5c21-0d79-403f-bfe3-9a4506529760/nuget/v3/flat2/microsoft.netcore.dotnethostresolver/index.json

These are not hosted pool machines, so it's quite possible we are hitting the cache. But why is the cache up to date? For sure there is a new package that should be found.

I've seen this in a variety of other places:

https://dev.azure.com/dnceng/public/_build/results?buildId=827136&view=logs&j=09811346-8274-5b72-2d96-1dd38f87c84b&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c

Microsoft.DotNet.Cli.Utils is one package was not found in this case

[riarenas]

Hmm, as a workaround for when the credential provider used to crash all the time, we were clearing all the nuget caches on every single invocation of msbuild. I wonder if we're seeing this now as I removed that, and we should now add a single http cache clear at the start of a build?

[riarenas]

#5952 is where I removed this. I don't think it should be reintroduced as we were clearing it multiple times per build, but maybe a single clear is still required?

[mmitche]

A single clear could work...though I think still we should follow up with the right people to get to the root cause,

[riarenas]

NuGet/Home#3116 seems relevant?

It seems like NuGet just bails when a feed is in the http cache and doesn't find a package in it. We were being guarded from this behavior by clearing the http cache every single time.

I broke this, so I'll prepare a change so we still clear the http cache.

[riarenas]

The nuget issue provides a workaround of using a restoreNoCache environment variable in the build machines. I'm testing setting that when we're in CI mode now.

[riarenas]

Merged a change that sets the environment variable in the build machines. Will keep an eye out to see if it resolves the issue completely, or if we need to go back to clearing the http cache multiple times per job.

#6237