dotnet restore gives "The repository primary signature validity period has expired".

[bobleujr]

Description

It's been almost 3 weeks that my CI/CDs fail intermittently when restoring the nuget packages. Here's one example

 1>C:\Program Files\dotnet\sdk\5.0.302\NuGet.targets(131,5): error : The given key '(Hangfire.Core, 1.6.19)' was not present in the dictionary. [D:\a\1\s\MyProject.Api.sln]
       System.Collections.Generic.KeyNotFoundException: The given key '(Hangfire.Core, 1.6.19)' was not present in the dictionary.
          at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
          at NuGet.Commands.LockFileBuilder.CreateLockFile(LockFile previousLockFile, PackageSpec project, IEnumerable`1 targetGraphs, IReadOnlyList`1 localRepositories, RemoteWalkContext context)
          at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token)
          at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
          at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, CancellationToken token)
          at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks)
          at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreContext, CancellationToken token)
          at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token)
          at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken)
          at NuGet.Build.Tasks.RestoreTask.ExecuteAsync(ILogger log)
     Done executing task "RestoreTask" -- FAILED.
   1>Done building target "Restore" in project "MyProject.Api.sln" -- FAILED.
   1>Done Building Project "D:\a\1\s\MyProject.Api.sln" (Restore target(s)) -- FAILED.

Build FAILED.

     "D:\a\1\s\MyProject.Api.sln" (Restore target) (1) ->
     (Restore target) -> 
       D:\a\1\s\src\sites\MyProject.Api\MyProject.Api.csproj : error NU3037: Package 'System.Threading.Thread 4.0.0-rc2-24027' from source 'https://api.nuget.org/v3/index.json': The repository primary signature validity period has expired. [D:\a\1\s\MyProject.Api.sln]
       D:\a\1\s\src\sites\MyProject.Api\MyProject.Api.csproj : error NU3028: Package 'System.Threading.Thread 4.0.0-rc2-24027' from source 'https://api.nuget.org/v3/index.json': **The repository primary signature's timestamping certificate is not trusted by the trust provider. [**D:\a\1\s\MyProject.Api.sln]
       D:\a\1\s\src\lib\MyProject.Core\MyProject.Core.csproj : error NU3037: Package 'Microsoft.NETCore.Windows.ApiSets 1.0.1-rc2-24027' from source 'https://api.nuget.org/v3/index.json': The repository primary signature validity period has expired. [D:\a\1\s\MyProject.Api.sln]
       D:\a\1\s\src\lib\MyProject.Core\MyProject.Core.csproj : error NU3028: Package 'Microsoft.NETCore.Windows.ApiSets 1.0.1-rc2-24027' from source 'https://api.nuget.org/v3/index.json': The repository primary signature's timestamping certificate is not trusted by the trust provider. [D:\a\1\s\MyProject.Api.sln]
       D:\a\1\s\src\lib\MyProject.Core\MyProject.Core.csproj : error NU3037: Package 'System.IO.FileSystem.Watcher 4.0.0-rc2-24027' from source 'https://api.nuget.org/v3/index.json': The repository primary signature validity period has expired. [D:\a\1\s\MyProject.Api.sln]
       D:\a\1\s\src\lib\MyProject.Core\MyProject.Core.csproj : error NU3028: Package 'System.IO.FileSystem.Watcher 4.0.0-rc2-24027' from source 'https://api.nuget.org/v3/index.json': The repository primary signature's timestamping certificate is not trusted by the trust provider. [D:\a\1\s\MyProject.Api.sln]
       C:\Program Files\dotnet\sdk\5.0.302\NuGet.targets(131,5): error : The given key '(Hangfire.Core, 1.6.19)' was not present in the dictionary. [D:\a\1\s\MyProject.Api.sln]

It fails trying to restore different packages. At one given moment it's Hangfire, the next run, is a different package. After a few times trying to rerun the same pipelines, it works.

Configuration

My .csproj / .sln files are very straightforward. I will paste an example .csproj from one of the projects failing here

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="Responses\" />
    <Folder Include="wwwroot\" />
  </ItemGroup>

  <ItemGroup>
    <None Remove="wkhtmlx32\libwkhtmltox.dll" />
    <None Remove="wkhtmlx64\libwkhtmltox.dll" />
    <None Remove="wkhtmlx64\libwkhtmltox.dylib" />
    <None Remove="wkhtmlx64\libwkhtmltox.so" />
  </ItemGroup>

  <ItemGroup>
    <Content Include="wkhtmlx32\libwkhtmltox.dll">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </Content>
    <Content Include="wkhtmlx64\libwkhtmltox.dll" />
    <Content Include="wkhtmlx64\libwkhtmltox.dylib" />
    <Content Include="wkhtmlx64\libwkhtmltox.so" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="HangFire.Redis.StackExchange" Version="1.7.2" />
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.AspNetCore.Razor.Design" Version="2.1.2" PrivateAssets="All" />
    <PackageReference Include="Microsoft.Extensions.Logging.ApplicationInsights" Version="2.15.0" />
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="2.1.9" />
  </ItemGroup>


  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <CodeAnalysisRuleSet>..\..\..\root\default.ruleset</CodeAnalysisRuleSet>
    <PlatformTarget>AnyCPU</PlatformTarget>
  </PropertyGroup>

  <ItemGroup>
    <Folder Include="Extensions\" />
    <Folder Include="wkhtmlx64\" />
    <Folder Include="Services\" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="wwwroot\" />
  </ItemGroup>

</Project>

I am using VSTS agents (Microsoft-hosted) via Azure DevOps, so I don't have control over the dotnet installation.

Regression?

Other information

I saw @chgill-MSFT talking about author signing certificates here but we are not enforcing

<config>
  <add key="signatureValidationMode" value="require" />
</config>

Thanks for the help !

[mairaw]

@omajid posted this on the first issue:

Sounds like NuGet/Home#10712
What OS and .NET version are you using?

[bobleujr]

The agent running the job is specified as

Which should contain these sdks

the agent OS is

As the .csproj file shows, it's targeting

<Project Sdk="Microsoft.NET.Sdk.Web">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
  </PropertyGroup>

[richlander]

I cannot reproduce those errors using the 5.0.302 SDK (using docker) with the given project.

Are you using that the signature validation nuget.config file you shared? That's not clear from what you wrote.

/cc @aortiz-msft

[bobleujr]

thank you for the answer @richlander. In one of the projects, we don't have a nuget.config file. In the other one it looks like (it's also failing from time to time):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="Nuget" value="https://api.nuget.org/v3/index.json" />
    <add key="privatefeed" value="our private feed" />
  </packageSources>
</configuration>

Do you think adding the certificate's fingerprint to a nuget.config explicitly could solve the problem? Shouldn't not having a nuget.config yield the same behavior?

[richlander]

For kicks, can you try moving to a later version of the SDK? 5.0.302 is now old.

[richlander]

Another thing I noticed is that you are using a VS2017 image with .NET 5. .NET 5 is supported with VS2019+. You should also upgrade your image to VS 2019. That may be the bigger issue at play.

[jeromelaban]

I'm getting this error on CI on macOS specifically, since a few hours.

Here's an example: https://dev.azure.com/uno-platform/Uno%20Platform/_build/results?buildId=32328&view=logs&j=eb40ee8c-c15a-5b86-e8f8-c957f416d765&t=7b4a93a7-f0fb-5b1e-a9ea-609a25e9f0ec&l=17980

There are two errors, in fact:

The repository primary signature's timestamp found a chain building issue: ExplicitDistrust: The trust setting for this policy was set to Deny.

and

error NU3037: Package 'System.Security.Cryptography.Csp 4.0.0' from source 'https://api.nuget.org/v3/index.json': The repository primary signature validity period has expired.

[richlander]

Your SDK is too old. I recommend jumping up to 5.0.400.

This is happening on Linux, macOS, and Windows for a series of reasons. If you upgrade to the latest .NET 5 or .NET 6 SDK, it will likely resolved. I'll explain.

• In January Debian made a certificate change to distrust Symantec certificates (which nuget.org uses pervasively). We asked Debian to revert this, which they did.
• In April the same change became pervasive and we changed .NET 5 and 6 at that time to no longer validate packages (which was super unfortunate). As long as you are using .NET 5.0.202 or higher, then you should not be seeing this issue.
• In August (today), we saw that macOS made the same change as we saw on Linux. Again, if you are using .NET 5.0.202 or higher, you should not experience this.
• Very separately, there is an issue on Windows where certificates are not updated correctly on boot. We are working with the Windows team on that. It has no affinity to the Symantec certificates. NuGet package validation is still in force on Windows since the Symantec certificates we need are still trusted.
• We are working on a plan to re-enable certificate validation on Linux and macOS and will soon share a spec on that. It is somewhat complex.

As a result of today's certificate changes on macOS, .NET 5.0.100 series SDKs are now unsupported (on macOS).

[dtivel]

there is an issue on Windows where certificates are not updated correctly

This is tracked via NuGet/Home#11099. Note that this is Windows only and intermittent.

[jeromelaban]

@richlander thanks for the update and explanations, upgrading to 5.0.400 fixed it for me.

[bobleujr]

As a follow up, I update our agents on Azure DevOps to use a newer SDK on win-2019 and it seems to have stabilized the issue.

[richlander]

I'm talking to the team about adding logging to Azure DevOps and GH Actions to tell you that you are on an old SDK and should move. That may help in future.

[bobleujr]

Absolutely. Thank you @richlander . I am going to close the issue.

[mario-galindo]

Your SDK is too old. I recommend jumping up to 5.0.400.

This is happening on Linux, macOS, and Windows for a series of reasons. If you upgrade to the latest .NET 5 or .NET 6 SDK, it will likely resolved. I'll explain.

• In January Debian made a certificate change to distrust Symantec certificates (which nuget.org uses pervasively). We asked Debian to revert this, which they did.
• In April the same change became pervasive and we changed .NET 5 and 6 at that time to no longer validate packages (which was super unfortunate). As long as you are using .NET 5.0.202 or higher, then you should not be seeing this issue.
• In August (today), we saw that macOS made the same change as we saw on Linux. Again, if you are using .NET 5.0.202 or higher, you should not experience this.
• Very separately, there is an issue on Windows where certificates are not updated correctly on boot. We are working with the Windows team on that. It has no affinity to the Symantec certificates. NuGet package validation is still in force on Windows since the Symantec certificates we need are still trusted.
• We are working on a plan to re-enable certificate validation on Linux and macOS and will soon share a spec on that. It is somewhat complex.

As a result of today's certificate changes on macOS, .NET 5.0.100 series SDKs are now unsupported (on macOS).

This save my life, thanks!