[wasm] Codespaces cannot auto-setup, no space left

[ilonatommy]

When running auto-setup from codespaces configurations (both single and multi-threaded) that force at least 16GM RAM and 32 GB memory, we are getting an error on runtime build:

/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error : No space left on device : '/home/vscode/.nuget/packages/microsoft.codeanalysis.workspaces.common/3.11.0/lib/netcoreapp3.1/cs' [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at System.IO.Directory.CreateDirectory(String path) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Packaging.PackageFileExtractor.ExtractPackageFile(String source, String target, Stream stream) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Packaging.PackageArchiveReader.CopyFiles(String destination, IEnumerable`1 packageFiles, ExtractPackageFileDelegate extractFile, ILogger logger, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Packaging.PackageExtractor.<>c__DisplayClass5_0.<<InstallFromSourceAsync>b__0>d.MoveNext() [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error : --- End of stack trace from previous location --- [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T](String filePath, Func`2 action, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Common.ConcurrencyUtilities.ExecuteWithFileLockedAsync[T](String filePath, Func`2 action, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Packaging.PackageExtractor.InstallFromSourceAsync(PackageIdentity packageIdentity, IPackageDownloader packageDownloader, VersionFolderPathResolver versionFolderPathResolver, PackageExtractionContext packageExtractionContext, CancellationToken token, Guid parentId) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.ProjectRestoreCommand.InstallPackageAsync(RemoteMatch installItem, NuGetv3LocalRepository userPackageFolder, PackageExtractionContext packageExtractionContext, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.ProjectRestoreCommand.<>c__DisplayClass15_1.<<InstallPackagesAsync>b__4>d.MoveNext() [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error : --- End of stack trace from previous location --- [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.ProjectRestoreCommand.InstallPackagesAsync(HashSet`1 uniquePackages, IEnumerable`1 graphs, IList`1 downloadDependencyInformations, NuGetv3LocalRepository userPackageFolder, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.ProjectRestoreCommand.TryRestoreAsync(LibraryRange projectRange, IEnumerable`1 frameworkRuntimePairs, NuGetv3LocalRepository userPackageFolder, IReadOnlyList`1 fallbackPackageFolders, RemoteDependencyWalker remoteWalker, RemoteWalkContext context, Boolean forceRuntimeGraphCreation, CancellationToken token, TelemetryAc2024-03-13 13:43:06.349Z: tivity telemetryActivity, String telemetryPrefix) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreCommand.ExecuteRestoreAsync(NuGetv3LocalRepository userPackageFolder, IReadOnlyList`1 fallbackPackageFolders, RemoteWalkContext context, CancellationToken token, TelemetryActivity telemetryActivity) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreCommand.ExecuteAsync(CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.ExecuteAsync(RestoreSummaryRequest summaryRequest, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.ExecuteAndCommitAsync(RestoreSummaryRequest summaryRequest, IRestoreProgressReporter progressReporter, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.CompleteTaskAsync(List`1 restoreTasks) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(IEnumerable`1 restoreRequests, RestoreArgs restoreArgs, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Commands.RestoreRunner.RunAsync(RestoreArgs restoreContext, CancellationToken token) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.BuildTasksUtility.RestoreAsync(DependencyGraphSpec dependencyGraphSpec, Boolean interactive, Boolean recursive, Boolean noCache, Boolean ignoreFailedSources, Boolean disableParallel, Boolean force, Boolean forceEvaluate, Boolean hideWarningsAndErrors, Boolean restorePC, Boolean cleanupAssetsForUnsupportedProjects, ILogger log, CancellationToken cancellationToken) [/workspaces/runtime/Build.proj]
/workspaces/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/NuGet.RestoreEx.targets(19,5): error :    at NuGet.Build.Tasks.Console.MSBuildStaticGraphRestore.RestoreAsync(String entryProjectFilePath, IDictionary`2 globalProperties, IReadOnlyDictionary`2 options) [/workspaces/runtime/Build.proj]
    0 Warning(s)
    1 Error(s)

When we track the disck space during setup time, it grows from initial 80% to 100%:

Filesystem      Size  Used Avail Use% Mounted on
overlay          32G   24G  5.9G  81% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/root        29G   22G  7.3G  75% /vscode
/dev/loop3       32G   24G  5.9G  81% /workspaces
/dev/sdb1       118G   56K  112G   1% /tmp
tmpfs           7.9G     0  7.9G   0% /proc/acpi
tmpfs           7.9G     0  7.9G   0% /proc/scsi
tmpfs           7.9G     0  7.9G   0% /sys/firmware
-------------------------------------------------------------
Filesystem      Size  Used Avail Use% Mounted on
overlay          32G   30G     0 100% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
/dev/root        29G   22G  7.2G  76% /vscode
/dev/loop3       32G   30G     0 100% /workspaces
/dev/sdb1       118G  240K  112G   1% /tmp
tmpfs           7.9G     0  7.9G   0% /proc/acpi
tmpfs           7.9G     0  7.9G   0% /proc/scsi
tmpfs           7.9G     0  7.9G   0% /sys/firmware

inodes are not a problem:

Filesystem      Inodes  IUsed   IFree IUse% Mounted on
overlay        2097152 606676 1490476   29% /
tmpfs          2045628     16 2045612    1% /dev
shm            2045628      1 2045627    1% /dev/shm
/dev/root      3870720 190741 3679979    5% /vscode
/dev/loop3     2097152 606676 1490476   29% /workspaces
/dev/sdb1      7864320     17 7864303    1% /tmp
tmpfs          2045628      1 2045627    1% /proc/acpi
tmpfs          2045628      1 2045627    1% /proc/scsi
tmpfs          2045628      1 2045627    1% /sys/firmware
---------------------------------------------------------------------
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
overlay        2097152 652304 1444848   32% /
tmpfs          2045628     16 2045612    1% /dev
shm            2045628      1 2045627    1% /dev/shm
/dev/root      3870720 190741 3679979    5% /vscode
/dev/loop3     2097152 652304 1444848   32% /workspaces
/dev/sdb1      7864320   2263 7862057    1% /tmp
tmpfs          2045628      1 2045627    1% /proc/acpi
tmpfs          2045628      1 2045627    1% /proc/scsi
tmpfs          2045628      1 2045627    1% /sys/firmware

ram is not a problem:

               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.1Gi       683Mi       1.0Mi        13Gi        14Gi
Swap:             0B          0B          0B
-----------------------------------------------------------------------------
               total        used        free      shared  buff/cache   available
Mem:            15Gi       6.3Gi       229Mi       101Mi       9.1Gi       8.9Gi
Swap:             0B          0B          0B

When running the commands manually, there are no problems with memory.
Happens on the 1st run only, I cannot reproduce when using "full rebuild container" option. Dump logs from 1st run and n-th run:
wasm_codespaces_nth_run.txt
wasm_codespaces_1st_run.txt

We cannot repro because it seems the "rebuild" and the initial build of codespaces are done from different permission levels. On rebuild with these steps:

• remove /workspace/runtime directory to let it re-clone
• choose "full container rebuild"
  we are hitting Restored /home/vscode/.nuget/packages/microsoft.dotnet.arcade.sdk/9.0.0-beta.24161.5/tools/Tools.proj (in 2.65 sec). chmod: changing permissions of '/workspaces/runtime/eng/common/dotnet-install.sh': Operation not permitted

Possible fixes:

• Chop up restore and clean cache in between - helps with ~10% of storage drop which is not enough.
• Add "storage": "...." to hostRequirements in devcontainer, choose optimal size - temporarily till the below discussion's issue is not resolved - keeping this issue opened till then.
• This discussion: https://github.com/orgs/community/discussions/57767 - jetbrains directory takes ~12GB (out of 32).