Investigate PR build failure #25293

ajaybhargavb · 2020-08-26T23:24:03Z

This PR had failures due to build ordering issues. Need to investigate.

More context, #25283 (comment)

cc @dotnet/aspnet-build

JunTaoLuo · 2020-08-29T07:38:40Z

Another example https://dev.azure.com/dnceng/public/_build/results?buildId=792496&view=results

dougbu · 2020-08-29T17:53:19Z

Is the problem usually or always showing up in Microsoft.AspNetCore.App.Runtime❔

sebastienros · 2020-09-01T22:38:18Z

Update:

From the binlog viewer, this node

Project Name=Project "Microsoft.AspNetCore.App.Runtime.csproj" (GetNativeManifest target(s)):

Has a property _BuildOutputInPackage that has the pdbs duplicated:

F:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.Antiforgery\Release\net5.0\Microsoft.AspNetCore.Antiforgery.pdb
F:\workspace\_work\1\s\artifacts\bin\Microsoft.AspNetCore.App.Runtime\Release\net5.0\win-x64\Microsoft.AspNetCore.Antiforgery.pdb

@dougbu says:

_GetBuildOutputFilesWithTfm is a target in the Runtime project. Sounds like it's either (a) wasting time adding duplicates to an iterm group or (b) not removing duplicates when it should. Either way, the root cause is likely a change somewhere in the .NET SDK that rejiggered the inputs to that target.

They might be copied to the same directory. The bin/Microsoft.AspNetCore.App.Runtime folder may also be intended as a staging area for the real layout. You'll need to look more at the project and perhaps a dotnet msbuild /pp:pp.out file for the project to figure out the project's expectations and whether the SDK is doing something different now. I often find myself fliipping back 'n forth betwen the log viewer and VS Code to try and figure these problems out.

dougbu · 2020-09-03T03:43:34Z

The PDB already in a package problem is impacting a lot of builds:

> runfo search-buildlog --definition=aspnet --pr --value="pdb' is not added because the package already contains file" --count 50
https://dev.azure.com/dnceng/public/_build/results?buildId=799262
https://dev.azure.com/dnceng/public/_build/results?buildId=799186
https://dev.azure.com/dnceng/public/_build/results?buildId=799129
https://dev.azure.com/dnceng/public/_build/results?buildId=799129
https://dev.azure.com/dnceng/public/_build/results?buildId=799079
https://dev.azure.com/dnceng/public/_build/results?buildId=797240
https://dev.azure.com/dnceng/public/_build/results?buildId=798807
https://dev.azure.com/dnceng/public/_build/results?buildId=798686
https://dev.azure.com/dnceng/public/_build/results?buildId=797948
https://dev.azure.com/dnceng/public/_build/results?buildId=797948
https://dev.azure.com/dnceng/public/_build/results?buildId=798502
https://dev.azure.com/dnceng/public/_build/results?buildId=798307
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=797993
https://dev.azure.com/dnceng/public/_build/results?buildId=797993
https://dev.azure.com/dnceng/public/_build/results?buildId=798020
https://dev.azure.com/dnceng/public/_build/results?buildId=798020
https://dev.azure.com/dnceng/public/_build/results?buildId=798020
https://dev.azure.com/dnceng/public/_build/results?buildId=798018
https://dev.azure.com/dnceng/public/_build/results?buildId=798018
https://dev.azure.com/dnceng/public/_build/results?buildId=798018
https://dev.azure.com/dnceng/public/_build/results?buildId=797973
https://dev.azure.com/dnceng/public/_build/results?buildId=797928
https://dev.azure.com/dnceng/public/_build/results?buildId=797928
https://dev.azure.com/dnceng/public/_build/results?buildId=797896
https://dev.azure.com/dnceng/public/_build/results?buildId=797896
https://dev.azure.com/dnceng/public/_build/results?buildId=797616
https://dev.azure.com/dnceng/public/_build/results?buildId=797887
https://dev.azure.com/dnceng/public/_build/results?buildId=797887
https://dev.azure.com/dnceng/public/_build/results?buildId=797772
https://dev.azure.com/dnceng/public/_build/results?buildId=797772
https://dev.azure.com/dnceng/public/_build/results?buildId=797813
https://dev.azure.com/dnceng/public/_build/results?buildId=797644
https://dev.azure.com/dnceng/public/_build/results?buildId=797724
https://dev.azure.com/dnceng/public/_build/results?buildId=797724

Evaluated 50 builds
Impacted 23 builds
Impacted 37 jobs
Searched 14002 records

Strangely, though we have warn-as-error enabled, this isn't failing our builds or at least not all of our builds. I picked https://dev.azure.com/dnceng/public/_build/results?buildId=797973&view=logs&j=1b89928a-2219-5ef9-602f-f95beb3da4dc&t=bcce746a-741e-59b7-adec-a4a82d8c5e52 somewhat randomly and see lots of error NU5118 but also Build succeeded.. @rainersigwald any ideas why dotnet msbuild is fine w/ these particular errors❔ We aren't suppressing them in the repo.

TanayParikh · 2020-09-03T20:28:26Z

/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible. [/datadisks/disk1/workspace/_work/1/s/src/Components/benchmarkapps/Wasm.Performance/TestApp/Wasm.Performance.TestApp.csproj]
##[error]src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: (NETCORE_ENGINEERING_TELEMETRY=Build) Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible.
/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible. [/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/testassets/HostedInAspNet.Client/HostedInAspNet.Client.csproj]
##[error]src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: (NETCORE_ENGINEERING_TELEMETRY=Build) Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible.
/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible. [/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/testassets/StandaloneApp/StandaloneApp.csproj]
##[error]src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: (NETCORE_ENGINEERING_TELEMETRY=Build) Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible.

#25577 (comment)

dougbu · 2020-09-04T04:17:44Z

More on the PDB duplicates: May be able to tell from the following (which ignores PRs) when the problem started…

> runfo search-buildlog --definition=aspnet --value="pdb' is not added because the package already contains file" --count 50
https://dev.azure.com/dnceng/public/_build/results?buildId=799262
https://dev.azure.com/dnceng/public/_build/results?buildId=799262
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=798151
https://dev.azure.com/dnceng/public/_build/results?buildId=798020
https://dev.azure.com/dnceng/public/_build/results?buildId=798020
https://dev.azure.com/dnceng/public/_build/results?buildId=797887
https://dev.azure.com/dnceng/public/_build/results?buildId=797644
https://dev.azure.com/dnceng/public/_build/results?buildId=796964
https://dev.azure.com/dnceng/public/_build/results?buildId=796399
https://dev.azure.com/dnceng/public/_build/results?buildId=795369
https://dev.azure.com/dnceng/public/_build/results?buildId=795151
https://dev.azure.com/dnceng/public/_build/results?buildId=795131
https://dev.azure.com/dnceng/public/_build/results?buildId=794466
https://dev.azure.com/dnceng/public/_build/results?buildId=794020
https://dev.azure.com/dnceng/public/_build/results?buildId=794020
https://dev.azure.com/dnceng/public/_build/results?buildId=793901

Evaluated 50 builds
Impacted 13 builds
Impacted 18 jobs
Searched 14426 records

No, I haven't checked whether that's every build since some date.

dougbu · 2020-09-04T20:08:47Z

The missing JavaScript file problem seems to have affected just one non-PR build:

> runfo search-buildlog --definition=aspnet --value="blazor.webassembly.js' because it does not exist or is inaccessible" --count 50
https://dev.azure.com/dnceng/public/_build/results?buildId=799302
https://dev.azure.com/dnceng/public/_build/results?buildId=799302

Evaluated 50 builds
Impacted 1 builds
Impacted 2 jobs
Searched 14426 records

dougbu · 2020-09-04T22:28:49Z

that issue also affected one PR:

> runfo search-buildlog --definition=aspnet --value="because it does not exist or is inaccessible" --count 50 --pr
https://dev.azure.com/dnceng/public/_build/results?buildId=802001
https://dev.azure.com/dnceng/public/_build/results?buildId=802001

Evaluated 50 builds
Impacted 1 builds
Impacted 2 jobs
Searched 13940 records

dougbu · 2020-09-06T02:57:47Z

@Pilchie just got really unlucky w/ four PRs opened at about the same time:

Add Public API baselines for Security #25651 hit another flaky Razor SDK test (BundlesScopedCssFiles_UpdatesBundleWhenContentsChange)
- this test failure impacted most, if not all, of these PRs
Add public API baselines for Servers #25652 hit the PDB duplication build problem in both aspnetcore-ci and aspnetcore-quarantined-pr
Add public API baselines to SignalR #25653 hit the gradlew build problem @halter73 is investigating with Debug gradle #25649
Add public API baselines for SiteExtensions, Testing, WebEncoders #25654 hit the PDB duplication build problem but only in the aspnetcore-quarantined-pr pipeline

dougbu · 2020-09-08T18:05:15Z

I'm able to reproduce the PDB duplication problem locally and seeing the build fail when it occurs. Still can't see why we sometimes hit those NU5118 errors without failing the build. Any ideas on that strange part of this @rainersigwald❔

dougbu · 2020-09-08T19:42:53Z

Assigned @BrennanConroy because he's got #25707 out and @wtgodbe because he's delving into the duplicate PDB problem

dougbu · 2020-09-08T19:48:40Z

@SteveSandersonMS is anyone looking at the Components ordering issue that leads to occasional errors like the following❔ This isn't as urgent as the other problems but we should try to get to the bottom of it.

@halter73 if nobody gets to this part of the issue today, please include it in your hand-off notes for the next person in the rotation.

/datadisks/disk1/workspace/_work/1/s/src/Components/WebAssembly/Sdk/src/targets/Microsoft.NET.Sdk.BlazorWebAssembly.Current.targets(256,5): error MSB3954: Failed to compute hash for file '/datadisks/disk1/workspace/_work/1/s/src/Components/Web.JS/dist/Release/blazor.webassembly.js' because it does not exist or is inaccessible. [/datadisks/disk1/workspace/_work/1/s/src/Components/benchmarkapps/Wasm.Performance/TestApp/Wasm.Performance.TestApp.csproj]

SteveSandersonMS · 2020-09-08T20:10:34Z

It's the first I've heard of it. I don't recall seeing this happen on local builds. Is there any known context, such as when it started happening? Might it be a side-effect of whatever ordering issue causes the duplicate PDB problem?

dougbu · 2020-09-08T21:04:58Z

Is there any known context, such as when it started happening?

The exact timing isn't clear because the problem happens sporadically. I reported a couple of runfo results for builds failing that way in #25293 (comment) and #25293 (comment)

Might it be a side-effect of whatever ordering issue causes the duplicate PDB problem?

No, that only occurs when building the Runtime project.

dougbu · 2020-09-08T21:05:38Z

And,

It's the first I've heard of it.

Ooopsie

wtgodbe · 2020-09-08T22:26:22Z

Where I'm at so far -

https://github.com/NuGet/NuGet.Client/blob/3f573425ac74d90b96a61e5e85ed39a39aad8a7f/src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.targets#L207-L223

It looks like BuildOutputInPackage is getting duplicate .pdb entries - one from the output folder of Microsoft.AspNetCore.App.Runtime, and one from the output of the project that produced the .pdb. That item group gets populated here:

https://github.com/NuGet/NuGet.Client/blob/3f573425ac74d90b96a61e5e85ed39a39aad8a7f/src/NuGet.Core/NuGet.Build.Tasks.Pack/NuGet.Build.Tasks.Pack.targets#L420-L429

The size of the binlog has made this hard to trace back locally, and building locally hasn't repro'd it yet for me. But, I did notice that the duplicates appear to be present even in green CI builds. Still poking around.

wtgodbe · 2020-09-08T22:42:02Z

Looks like in 3.1, _GetBuildOutputFilesWithTfm created output that was populated with just .dll's and .pdb's from the Microsoft.AspNetCore.App.Runtime output folder, whereas today it also has .pdb's from the output folders of the individual projects

wtgodbe · 2020-09-08T23:00:34Z

Not sure when this started, but what's happening:

aspnetcore/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj

Lines 282 to 293 in 3d042ac

    
           <Target Name="_ResolveRuntimePackBuildOutput" DependsOnTargets="ResolveReferences;Crossgen"> 
        
             <ItemGroup> 
        
               <BuiltProjectOutputGroupOutput Include="$(ProjectDepsFilePath)" /> 
        
               <BuiltProjectOutputGroupOutput Include="@(ReferenceCopyLocalPaths)" Condition=" '%(ReferenceCopyLocalPaths.IsNativeImage)' != 'true' " /> 
        
               <!-- Include all .pdbs in build output. Some .pdb files are part of ReferenceCopyLocalPaths, but this can change when running /t:Pack /p:NoBuild=true. --> 
        
               <BuiltProjectOutputGroupOutput Include="$(TargetDir)*.pdb" Exclude="@(ReferenceCopyLocalPaths)" /> 
        
               <!-- Crossgen symbols for Linux include a GUID in the file name which cannot be predicted. --> 
        
               <BuiltProjectOutputGroupOutput Include="$(TargetDir)*.map" Condition="'$(CrossGenSymbolsType)' == 'PerfMap'" /> 
        
               <NuGetPackInput Include="@(BuiltProjectOutputGroupOutput)" /> 
        
             </ItemGroup> 
        
           </Target>

ResolveReferences adds the .pdb's from the individual project output dirs to ReferenceCopyLocalPaths, then it appears the condition on

aspnetcore/src/Framework/App.Runtime/src/Microsoft.AspNetCore.App.Runtime.csproj

Line 287 in 3d042ac

    
           <BuiltProjectOutputGroupOutput Include="$(TargetDir)*.pdb" Exclude="@(ReferenceCopyLocalPaths)" />

isn't working, since the .pdb's in the output directory of the SharedFx project don't have the same %(Identity) as the ones in the in the output dirs of the individual projects. Tweaking that condition should fix the issue

wtgodbe · 2020-09-09T21:34:20Z

Resolved by #25733

wtgodbe · 2020-09-10T21:01:48Z

Looks like this is still recurring. #25733 did make NuGetPackInput free of duplicates, but we're still seeing this failure, e.g. https://dev.azure.com/dnceng/public/_build/results?buildId=809582&view=results. Looking into it.

wtgodbe · 2020-09-11T17:27:40Z

Fixed by #25785

ajaybhargavb added investigate area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework labels Aug 26, 2020

ajaybhargavb mentioned this issue Aug 26, 2020

Fix SignalR typescript tests with Chrome SameSite reaction #25283

Merged

wtgodbe mentioned this issue Aug 31, 2020

Separate MessagePackHubProtocol into its own package #25253

Merged

dougbu mentioned this issue Sep 3, 2020

[master] Update dependencies from dotnet/arcade #25577

Merged

dougbu mentioned this issue Sep 7, 2020

Eliminate duplicates in Helix TestRunner #25663

Merged

dougbu assigned BrennanConroy and wtgodbe Sep 8, 2020

wtgodbe mentioned this issue Sep 9, 2020

Strip duplicate files from SharedFx input to Nuget Pack #25733

Merged

wtgodbe closed this as completed Sep 9, 2020

hoyosjs mentioned this issue Sep 10, 2020

Enable build of ARM64 installers in ASP.NET Core #25579

Merged

wtgodbe reopened this Sep 10, 2020

wtgodbe mentioned this issue Sep 10, 2020

Strip duplicates from BuiltProjectOutputGroupOutput #25785

Merged

wtgodbe closed this as completed Sep 11, 2020

ghost locked as resolved and limited conversation to collaborators Oct 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate PR build failure #25293

Investigate PR build failure #25293

ajaybhargavb commented Aug 26, 2020 •

edited

Loading

JunTaoLuo commented Aug 29, 2020

dougbu commented Aug 29, 2020

sebastienros commented Sep 1, 2020

dougbu commented Sep 3, 2020

TanayParikh commented Sep 3, 2020

dougbu commented Sep 4, 2020

dougbu commented Sep 4, 2020

dougbu commented Sep 4, 2020 •

edited

Loading

dougbu commented Sep 6, 2020

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

SteveSandersonMS commented Sep 8, 2020 •

edited

Loading

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 9, 2020

wtgodbe commented Sep 10, 2020

wtgodbe commented Sep 11, 2020

Investigate PR build failure #25293

Investigate PR build failure #25293

Comments

ajaybhargavb commented Aug 26, 2020 • edited Loading

JunTaoLuo commented Aug 29, 2020

dougbu commented Aug 29, 2020

sebastienros commented Sep 1, 2020

dougbu commented Sep 3, 2020

TanayParikh commented Sep 3, 2020

dougbu commented Sep 4, 2020

dougbu commented Sep 4, 2020

dougbu commented Sep 4, 2020 • edited Loading

dougbu commented Sep 6, 2020

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

SteveSandersonMS commented Sep 8, 2020 • edited Loading

dougbu commented Sep 8, 2020

dougbu commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 8, 2020

wtgodbe commented Sep 9, 2020

wtgodbe commented Sep 10, 2020

wtgodbe commented Sep 11, 2020

ajaybhargavb commented Aug 26, 2020 •

edited

Loading

dougbu commented Sep 4, 2020 •

edited

Loading

SteveSandersonMS commented Sep 8, 2020 •

edited

Loading