Floating package references don't get updated because of the no-op and http caches

[mpetito]

Details about Problem

I have a .NET Core project which references nuget packages published on a private feed in Visual Studio Team Services. These references use floating versions, such as:

    <PackageReference Include="My.Library" Version="4.0.*" />

During the first restore operation using dotnet restore, the latest patch version is retrieved from the nuget feed as expected. However, if a new patch version is published, subsequent restores on the same machine will never see the update unless I explicitly run dotnet restore --no-cache or alternatively run dotnet nuget locals http-cache --clear.

Other suggested things

I understand there is an http-cache in use, where restore operations may use a cached list of versions for each package. I can see this cache and the stale version list on disk which explains why dotnet restore isn't restoring the new version.

However, it seems like this makes the usage of floating package versions confusing. Even if I force the restore of a newer version locally using --no-cache, other developers or build servers may be stuck with an older version. Is the intention really that the http-cache should never expire, and new versions will never be automatically restored unless explicitly called with --no-cache?

My expectation is that the http-cache for the version list should be short lived, and dotnet restore (performed explicitly or as part of a build) would eventually see new versions for floating package versions. Note that the http-cache also does not appear to respect the HTTP headers returned by the feed, as in Fiddler I can see that VSTS returns no-cache directives on the nuget feed responses.

If a short-lived cache is not advisable for some other reasons, then I would request a feature that allows disabling the cache per feed in nuget.config. This way the --no-cache flag might be automatically applied for private package feeds as appropriate.

---

NuGet product used (NuGet.exe | VS UI | Package Manager Console | dotnet.exe):

NuGet version: 4.6.0

dotnet.exe --version: 2.1.302

VS version: Visual Studio 2017 15.7.6

OS version: Windows 10

Worked before? If so, with which NuGet version: Not sure.

[zhili1208]

current expire time for http-cache is 30 mins. we can consider either short this time or create a feature like no-cache flag

[mpetito]

@zhili1208 I'm not seeing new packages restored after 30 minutes, so either the 30 minute expiration is not operating as expected or is unrelated to this issue.

In a recent case of this happening, I had a build server fail to restore a newer version that was published 4 days earlier.

I've also just reproduced this in a project where I ran dotnet restore --no-cache at 5PM ET while documenting this issue. Now at 6:45PM ET I have published a new patch version, verified the new version appeared in the feed, and ran dotnet restore on the same project. It did not find the new version.

[zhili1208]

@mpetito Could you check the Fiddler log, when you repro the issue, does nuget send http request to VSTS? also does VSTS return correct version information? I saw a very similar issue which is related to VSTS months ago. So please check this first

[mpetito]

@zhili1208 In Fiddler, when running dotnet restore (without the --no-cache flag) I am only seeing a POST request to https://dc.services.visualstudio.com/v2/track. I am not seeing any other HTTP requests from nuget. I believe this POST request is related to Application Insights for the dotnet cli.

I do see the expected HTTP requests to the feed if I run dotnet restore --no-cache.

[mpetito]

As more evidence that caching is not working as expected, the next day (16 hours later), dotnet restore still does not see the updated packages.

If I download the latest nuget.exe cli (4.7.1) and run .\nuget.exe list -Source vsts-source the output shows the updated version. If I run .\nuget.exe restore MyProject.csproj it looks like it invokes msbuild and it still does not see the updated version.

[mpetito]

@zhili1208 Is this an issue I should raise with the dotnet cli team? It seems like this issue is miscategorized as the expected behavior for the http cache is not working when nuget is invoked via msbuild.

[zhili1208]

thanks, it looks like a nuget bug, I will mark it as bug

[nkolev92]

Do we have a repro here @zhili1208, @mpetito?

list and restore use different endpoints, so it's not a perfect analogy to compare those 2.

Does restore work on any other machine consistently?
If the problem was as simple to reproduce as discussed here, I would have expected to see more issues filed about it.

[mpetito]

@nkolev92 I've reproduced / witnessed this behavior on four workstations and a build server now. Each have installed latest versions of .net sdk and Visual Studio. However, each are also building the same projects, so it could be something wrong with the project setup.

I've thought about how best to provide a working repro but it's a little tricky because:

• I'm using a private feed in vsts (not sure if it matters)
• The steps to repro include publishing a new package version.

I will try to recreate the environment for testing this if it's useful.

[nkolev92]

Thanks @mpetito
We really do appreciate it.
If we can reproduce this on a simple project it will go long way toward helping us analyze this.

[mpetito]

@nkolev92 @zhili1208 I've put together a sample repository at https://github.com/mpetito/RestoreTest which reproduces this issue. Instructions are included there, and note the api key included is only valid for the package mpetito.nuget.test for 1 year.

I was successful in reproducing the issue using new projects for both the nuget package and the referencing project publishing to nuget.org. I also happened to update to Visual Studio 15.8.0 which did not resolve the issue (I'm not sure if this update changed my versions of msbuild or nuget).

[nkolev92]

@mpetito
Is it possible that you are hitting #5445 ?

30 mins after the initial restore and after the package has been published on nuget.org, does running dotnet restore --force download the latest version?

[mpetito]

@nkolev92 dotnet restore --force does appear to download and install the latest version after the http-cache 30 min expiry.

If I understand correctly then, the workarounds are:

• In Visual Studio, use rebuild with restore and hope the http-cache has expired.
• Using CLI, use --no-cache (since it doesn't seem --force is any faster but still uses http-cache).

I get why this behavior makes sense as you're trying to minimize network calls during restore. However, I'll probably re-evaluate the usage of floating versions for my particular use-case or minimally force --no-cache restores on build servers. In this regard, disabling the cache per feed using NuGet.config would be a useful feature even if used solely for build servers.

[gregbair]

My team has also run into this exact scenario. We've had to update our build scripts to use --no-cache as well.

Even after the 30 minutes has expired, new versions are not pulled.

[nkolev92]

@mpetito
Yes, that summarizes the current behavior.
//cc @anangaur @rrelyea

In Visual Studio, use rebuild with restore and hope the http-cache has expired.

@anangaur has been looking into improving this behavior. It's likely that soon enough we have an explicit option to force the reevaluation, rather than relying on rebuild.

Using CLI, use --no-cache (since it doesn't seem --force is any faster but still uses http-cache).

Can you clarify what you mean by --force not being any faster? If the http cache is older than 30 minutes, force will hit the sources again.

However, I'll probably re-evaluate the usage of floating versions for my particular use-case or minimally force --no-cache restores on build servers. In this regard, disabling the cache per feed using NuGet.config would be a useful feature even if used solely for build servers.

Both no-op and the http caching strategy are erring on the side of performance as in a lot of cases, this led to too many network calls that made nuget feel sluggish.

We understand that this leads to the situations that we've called out in #5445 and what you're articulating.

I'll amend this issue a bit to reflect that.

@gregbair

My team has also run into this exact scenario. We've had to update our build scripts to use --no-cache as well.

Even after the 30 minutes has expired, new versions are not pulled.

After 30 minutes you should be able to run --force, no need to to do --no-cache.

[mpetito]

Can you clarify what you mean by --force not being any faster? If the http cache is older than 30 minutes, force will hit the sources again.

@nkolev92 in my informal testing comparing the timing output between --no-cache and --force, without waiting for the 30 min http-cache expiry, I found the timings to be nearly identical (within tens of milliseconds). In both cases the restore time was about 2.3s for the project I tested.

This could have been due to my connection speed, and other restore work significantly outweighing the extra network call for --no-cache.

[nkolev92]

Ah, that's what you meant :)
Thanks for clarifying it @mpetito

Yeah, it's influenced by the connection speed and the fact that most of the packages don't need updated, so there's likely only a few package downloads happening.

[nkolev92]

Related
https://developercommunity.visualstudio.com/content/problem/320953/nuget-restore-doesnt-get-latest-wildcard-version-a.html
and
#6987

[timotei]

Do we have any news on the way to go for this? Can't we have at least a way (which can be automated) to disable/ignore the cache mechanism altogether in certain cases?

[timotei]

@zhili1208 @nkolev92 Would you accept a PR that tries to fix/enable disabling the cache on rebuild/explicitly?

[nkolev92]

@timotei

I don't think it's a good idea to disable the caching mechanisms on rebuild.
See #6987.

Doing a restore without any caching in place is very costly.
I think we'd need to design for scenarios like these explicitly, rather than making them the default.

[timotei]

@nkolev92 OK, but I am thinking of the scenario that if I want to explicitly do a rebuild ignoring case, I should be able to do it, without having to wait X minutes for the cache to expire. Maybe I am missing something?

[nkolev92]

@timotei

There are multiple caching strategies in place.
The no-op cache will always have a strategy to defeat it. See, rebuild today.
The http-cache on the other hand can't be bypassed, rather just cleaned. We're talking only about a very small minority of use-cases that will work better without the http caching.

However, I'd be in favor of adding switches to do that.

//cc @anangaur fyi.

[anangaur]

Don’t we already have a —no-cache switch already?

[nkolev92]

We do.
I should've more specific.

The VS experience needs some thought.
The equivalent is more costly there, especially since it'd be difficult to disable caching for just that operation.

[timotei]

The use case is pretty simple: say, I merged something on a dependent library, and in my main project, where I reference the package via floating reference, I want to do a rebuild now, taking into account the newly deployed version. How can I do that from:

1. VS
2. command line build (e.g., building on CI/CD)

[tugrulelmas]

Any update on this?

As @mpetito has suggested before, making a feed non-cacheable would make more sense for floating package references because, in my opinion, this feature is mostly used for packages which get lots of update in a day and is published to a different feed. In that scenario, I shouldn't wait for 30 minutes to download latest version of package.

disabling the cache per feed using NuGet.config would be a useful feature even if used solely for build servers.

@nkolev92 is there an equivalent of dotnet restore --force for MSBuild or nuget.exe? How can I force nuget to download latest packages for .NET Framework projects which use floating package references?

[nkolev92]

@tugrulelmas

NuGet.exe has a -Force switch.
MSBuild /t:restore will read the /p:RestoreForce="true"
dotnet.exe restore is msbuild restore under the hood.

[tugrulelmas]

Then, can we say following documents are outdated?
https://docs.microsoft.com/en-us/nuget/reference/msbuild-targets#restore-target
https://docs.microsoft.com/en-us/nuget/reference/cli-reference/cli-ref-restore

[nkolev92]

Seems like they are.
Sorry about that, I will update them.

[nkolev92]

Closing this issue as it's captured in #3389 and #5445

[drdamour]

@nkolev92 don't agree with the assessment that it's a small amount of use cases, anyone coming from maven/java has been relying on this type of behaviour with -SNAPSHOT for the last decade. Any time you are working with a shared library and making mods on it that you want to immediately use on your main project this type of flow is vital to rapid dev.