Downloading artifacts before publishing is exceeding 10gb, in runtime

[epananth]

• This issue is blocking
• This issue is causing unreasonable pain

Build -> https://dev.azure.com/dnceng/internal/_build/results?buildId=1002147&view=results

Symbols packages are copied to a temporary directory and then uploaded to Symbol server, this was done cos we wanted blobUpload and symbolUpload have to work parallelly and SymUploader need read permission when uploading to symbol server.

But now in Runtime

• We are downloading 6.3 gigs of artifacts to publish: https://dev.azure.com/dnceng/internal/_build/results?buildId=1002323&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=cecdd0ae-1666-570d-4521-d44005725bca&l=3166
• The symbol packages are 4.36 GB, so when we do that new intermediate copy step, we’re going over the 10 gigs in a hosted machine.

cc: @mmitche @riarenas

ckages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : There is not enough space on the disk. : 'D:\a\1\a\tempSymbols\Microsoft.NETCore.App.Runtime.Mono.osx-x64.6.0.0-preview.2.21118.9.symbols.nupkg'
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error :    at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.CopySymbolFilesToTemporaryLocation(BuildModel buildModel, String symbolTemporaryLocation) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 228
D:\a\1\s\.packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error :    at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.ExecuteAsync() in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 95
##[error].packages\microsoft.dotnet.arcade.sdk\6.0.0-beta.21117.1\tools\SdkTasks\PublishArtifactsInManifest.proj(105,5): error : (NETCORE_ENGINEERING_TELEMETRY=Publish) There is not enough space on the disk. : 'D:\a\1\a\tempSymbols\Microsoft.NETCore.App.Runtime.Mono.osx-x64.6.0.0-preview.2.21118.9.symbols.nupkg'
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite)
   at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.CopySymbolFilesToTemporaryLocation(BuildModel buildModel, String symbolTemporaryLocation) in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 228
   at Microsoft.DotNet.Build.Tasks.Feed.PublishArtifactsInManifestV3.ExecuteAsync() in /_/src/Microsoft.DotNet.Build.Tasks.Feed/src/PublishArtifactsInManifestV3.cs:line 95

[Edit : renamed this as its not blocking but we need a better way to handle this scenario ]

[epananth]

I have a PR to remove symbol publishing to azure storage -> #6973 , so I was thinking we might be able to remove the temp dir altogether and test in runtime.. But this will fail for nuget feed uploads for stable builds.

I am going remove the temp dir and test it against runtime to see if helps at all. Then prob for stable builds we can come up with a different solution.

[epananth]

Till we fig this out, I have opened a FR issue to unblock runtime -> #6988

[epananth]

I tested this fix against arcade and that worked :)
https://dev.azure.com/dnceng/internal/_build/results?buildId=1003974&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=ba23343f-f710-5af9-782d-5bd26b102304

[epananth]

Runtime promotion also worked ->https://dev.azure.com/dnceng/internal/_build/results?buildId=1003984&view=logs&j=ba23343f-f710-5af9-782d-5bd26b102304&t=6e277ba4-1c1e-552d-b96f-db0aeb4be20a

Need to figure out the best way for uploading symbols to Nuget Feed. Probably only for stable builds, we do not parallelly upload symbols to Nuget feed and Symbolserver, what if we do it one after the other?

[mmitche]

@epananth I would look instead into not having to have the nuget feed for symbol publishing at all. Instead, forcing all the symbol packages to have non-stable versions. This would mean they could continue just to go to azure blob storage.

[epananth]

@mmitche mentioned in email

Some ideas:
-	@Epsitha Ananth maybe we can alter the publishing process so that it doesn’t bulk download artifacts prior to publishing? What if instead it downloaded individual artifacts as they are needed? That actually might be faster than the existing publishing methodology (using both the upstream and downstream pipes at the same time).
-	We could introduce a configuration setting into the publishing job which allows a repo to opt-in to a larger machine This would help runtime but not hurt others.

I’m more a fan of the first item there, as it would ensure a reasonably minimal amount of artifacts locally on disk at any one time.

[epananth]

I did some investigation on downloading the artifacts one by one

Here is the API I used to list all the artifacts from the storage :

1. List Api -> https://dev.azure.com/dnceng/internal/_apis/build/builds/1003925/artifacts?api-version=6.0&token=

here is the result snippet

{
	"count": 9,
	"value": [
		{
			"id": 20054593,
			"name": "AssetManifests",
			"source": "2f0d093c-1064-5c86-fc5b-b7b1eca8e66a",
			"resource": {
				"type": "Container",
				"data": "#/6297361/AssetManifests",
				"properties": {
					"artifactsize": "4316"
				},
				"url": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=AssetManifests&api-version=6.0",
				"downloadUrl": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=AssetManifests&api-version=6.0&%24format=zip"
			}
		},
		{
			"id": 20054594,
			"name": "PackageArtifacts",
			"source": "2f0d093c-1064-5c86-fc5b-b7b1eca8e66a",
			"resource": {
				"type": "Container",
				"data": "#/6297361/PackageArtifacts",
				"properties": {
					"artifactsize": "12413558"
				},
				"url": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=PackageArtifacts&api-version=6.0",
				"downloadUrl": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=PackageArtifacts&api-version=6.0&%24format=zip"
			}
		},
		{
			"id": 20054595,
			"name": "BlobArtifacts",
			"source": "2f0d093c-1064-5c86-fc5b-b7b1eca8e66a",
			"resource": {
				"type": "Container",
				"data": "#/6297361/BlobArtifacts",
				"properties": {
					"artifactsize": "12413558"
				},
				"url": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=BlobArtifacts&api-version=6.0",
				"downloadUrl": "https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=BlobArtifacts&api-version=6.0&%24format=zip"
			}
		},

So we can download the PackageArtifacts folder using https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_apis/build/builds/1003925/artifacts?artifactName=PackageArtifacts&api-version=6.0&%24format=zip but there is no way to list the contents of the PackageArtifacts, to download it one file at a time. We can only download entire folder.

Let me know if I am missing something here.

[epananth]

There is this API https://docs.microsoft.com/en-us/rest/api/azure/devops/build/artifacts/get%20file?view=azure-devops-rest-6.0, to download one file at a time. But we have to have the file name and id before hand to do so..

[ChadNedzlek]

Weirdly that API doesn't support "Container" artifacts. I'm not sure what sort of weird artifact we are producing that it doesn't work in. :-(

[epananth]

For downloading artifact from the container -> https://dev.azure.com/dnceng/_apis/resources/Containers/{containId}?itemPath=/BlobArtifacts/MergedManifest.xml&isShallow=true&api-version=4.1-preview.4

Eg: https://dev.azure.com/dnceng/_apis/resources/Containers/6297361?itemPath=/BlobArtifacts/MergedManifest.xml&isShallow=true&api-version=4.1-preview.4

[ChadNedzlek]

Yeah, we just have to sort of... infer that second path, but it's what all the support people said to do for every instance I found (they also said to stop using Publish Build Artifacts, but that's not an argument worth having right now, I think... the other API is equally useless, because you have to pass it "fileId", which isn't documented anywhere as to what that even means)

[mmitche]

Yeah agreed. This isn't ideal, but we were told to use this, and it's also shown in OSS code right now (the tasks that download artifacts to the machine use the API). Presumably they have to keep supporting it to not break the task.

[epananth]

Also adding a flag, to support both the existing functionality and the new API supported one. So that if the API fails, we have a fall back mechanism.

[lukas-lansky]

There were failures like this again in runtime, it seems an Arcade update overwrote the bandaid. Applying again.

[riarenas]

I've been doing my best to jump on any arcade dependency updates to change this, but seems like I missed yesterday's update.

@epananth do you have a sense for when the fix for this will be ready? Should we be thinking about disabling the arcade subscription until then or something more sustainable to avoid this from popping up again?

[ViktorHofer]

Just following up on mails as I was out last week.

@epananth do you have a sense for when the fix for this will be ready?

I would like to know that as well.

[akoeplinger]

We could add the workaround dotnet/runtime@2ab247a to dotnet/arcade and activate it via a if (repo == dotnet/runtime) check in the meantime so we don't need to constantly reapply it.

[epananth]

I got my pr approved => #7086, will merge today...

[mmitche]

[michellemcdaniel]

@epananth I closed this based on discussions in stand up. Please reopen if needed.