Restore cannot be completed for a large solution

[marcin-krystianc]

NuGet Product Used

dotnet.exe

Product Version

SDK Version: 10.0.100-alpha.1.24630.4

Worked before?

SDK Version: 9.0.101

Impact

I'm unable to use this version

Repro Steps & Context

• Start a clean Linux container:
  docker.exe run -t -i --rm ubuntu
• Get daily build of dotnet and checkout test solutions:

apt-get update && apt-get install -y wget git tar &&  \
mkdir /dotnet \
 && wget https://aka.ms/dotnet/10.0.1xx/daily/dotnet-sdk-linux-x64.tar.gz \
 && tar -C /dotnet -xvzf dotnet-sdk-linux-x64.tar.gz \
 && rm dotnet-sdk-linux-x64.tar.gz \
 && echo "export DOTNET_ROOT=/dotnet" >> $HOME/.bashrc \
 && echo "export PATH=$PATH:/dotnet:$HOME/.dotnet/tools" >> $HOME/.bashrc \
 && source ~/.bashrc \
 && git clone https://github.com/marcin-krystianc/TestSolutions.git \
 && cd TestSolutions \
 && git reset --hard 142722bebfe90c4e5c98303fa1598db6a760adae

• Run the restore:

dotnet restore -clp:PerformanceSummary -clp:summary --force /p:RestoreUseStaticGraphEvaluation=true LargeAppWithPrivatePackagesCentralisedNGBVRemoved/solution/LargeAppWithPrivatePackagesCentralisedNGBVRemoved.sln

In version 9.0.101, the restore operation completes after about 10-20s.
In the recent daily build, the restore operation wasn't completed after >150s. It also consumes more and more RAM (tens of Gigabytes) until it exhausts all available resources and crashes the container.

Verbose Logs

No response

[jgonz120]

Hello! Can you provide an example repo to reproduce this issue?

[marcin-krystianc]

https://github.com/marcin-krystianc/TestSolutions.git

It is in the script above. The repo is https://github.com/marcin-krystianc/TestSolutions.git and the solution file is LargeAppWithPrivatePackagesCentralisedNGBVRemoved/solution/LargeAppWithPrivatePackagesCentralisedNGBVRemoved.sln

[jgonz120]

Must have missed that! thanks :)

[jgonz120]

I had to update the execution script to get it to work

apt-get update && apt-get install -y wget git tar &&  \
mkdir /dotnet \
 && wget https://aka.ms/dotnet/10.0.1xx/daily/dotnet-sdk-linux-x64.tar.gz \
 && tar -C /dotnet -xvzf dotnet-sdk-linux-x64.tar.gz \
 && rm dotnet-sdk-linux-x64.tar.gz \
 && echo "export DOTNET_ROOT=/dotnet" >> $HOME/.bashrc \
 && echo "export PATH=$PATH:/dotnet:$HOME/.dotnet/tools" >> $HOME/.bashrc \
 && . ~/.bashrc \
 && git clone https://github.com/marcin-krystianc/TestSolutions.git \
 && cd TestSolutions \
 && git reset --hard 142722bebfe90c4e5c98303fa1598db6a760adae \
 && export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

[marcin-krystianc]

This issue has been automatically marked as stale because we have not received a response in 14 days. It will be closed if no further activity occurs within another 14 days of this comment.

I think all the necessary information to repro that issue are provided.

[jeffkl]

This looks like a potential issue with the new dependency resolver. I haven't been able to fully debug the problem but a quick look seemed like its processing more nodes than needed.

Disabling the new resolver worked around the problem for me.

<PropertyGroup>
  <RestoreUseLegacyDependencyResolver>true</RestoreUseLegacyDependencyResolver>
</PropertyGroup>

Internally we have repos with over 2,500 projects that reference thousands of projects and packages and they complete in less than 3 minutes. Not sure exactly what's so special about this sample solution.

[jeffkl]

More debugging shows hundreds of thousands of items being queued for processing during the graph walk, adding a line that skips children that have already been chosen seems to fix the performance issue but needs to be tested more.

if (resolvedDependencyGraphItems.TryGetValue(childLibraryDependencyIndex, out ResolvedDependencyGraphItem? childResolvedDependencyGraphItem)
    && childResolvedDependencyGraphItem.LibraryRangeIndex == childLibraryRangeIndex)
{
    continue;
}