Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent failure to restore IBC optimization data #51585

Closed
runfoapp bot opened this issue Apr 20, 2021 · 11 comments
Closed

Intermittent failure to restore IBC optimization data #51585

runfoapp bot opened this issue Apr 20, 2021 · 11 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone

Comments

@runfoapp
Copy link

runfoapp bot commented Apr 20, 2021

Runfo Tracking Issue: Intermittent failure to restore IBC optimization data

Definition Build Kind Job Name
runtime 1097184 Rolling Build Linux arm Release AllSubsets_Mono
runtime 1096348 PR 51200 Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime 1095579 PR 51488 Libraries Build Linux_musl arm64 Release
runtime 1094759 PR 51439 Libraries Build Linux_musl arm Release
runtime 1094469 PR 51440 Mono Product Build Linux arm64 release

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 5 5
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 20, 2021
@ghost
Copy link

ghost commented Apr 21, 2021

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Tracking Issue: Intermittent failure to restore IBC optimization data

Definition Build Kind Job Name
runtime 1097184 Rolling Build Linux arm Release AllSubsets_Mono
runtime 1096348 PR 51200 Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime 1095579 PR 51488 Libraries Build Linux_musl arm64 Release
runtime 1094759 PR 51439 Libraries Build Linux_musl arm Release
runtime 1094469 PR 51440 Mono Product Build Linux arm64 release

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
1 5 5
Author: runfoapp[bot]
Assignees: -
Labels:

area-Infrastructure, untriaged

Milestone: -

@jeffschwMSFT
Copy link
Member

@agocke do we still have IBC in our pipeline or is the name of the stage just not been updated?

@hoyosjs
Copy link
Member

hoyosjs commented Apr 21, 2021

@jeffschwMSFT IBC comes from arcade support, it's looking for these IBC data packages:

image

These are available at https://dev.azure.com/dnceng/public/_packaging?_a=package&feed=dotnet6-transport&package=optimization.linux-x64.IBC.CoreCLR&protocolType=NuGet&version=99.99.99-master-20200806.6&view=overview. Not sure why they are not restored

@jeffschwMSFT
Copy link
Member

Thanks! The root of my question is that since we are moving from the ibc format to mibc, I was curious if we were just reusing the name or if we still were flowing ibc data.

@hoyosjs
Copy link
Member

hoyosjs commented Apr 21, 2021

These are old packages (newest is February), so there's no flow. As for if it's used, I am not sure if that part is still hooked.

@ViktorHofer ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Apr 21, 2021
@ViktorHofer ViktorHofer added this to the 6.0.0 milestone Apr 21, 2021
@ViktorHofer ViktorHofer added the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Apr 21, 2021
@agocke
Copy link
Member

agocke commented Apr 21, 2021

@davidwrighton would know for sure, but I see the same packages coming out of dotnet-optimization with what looks like fresh mibc files in them.

@hoyosjs
Copy link
Member

hoyosjs commented Apr 21, 2021

The mIBC packages are slightly different: optimization.windows_nt-x86.mibc.runtime. After talking to @ViktorHofer, this is a previously-seen NuGet issue where packages are not getting downloaded, but there's no error getting reported.

@davidwrighton
Copy link
Member

Flow is almost there, but as @hoyosjs says this is a nuget problem. If someone knows how to reduce the likelihood of this fault. I'd be happy to help, but without that, I don't think I can do anything here.

@ViktorHofer
Copy link
Member

Copying this from another issue but it might help here to share some context. In the last years I worked on reducing NuGet invocations where possible down to three:

  1. The msbuild sdk resolver (when an msbuild sdk is downloaded from nuget which happens before restore during evaluation)
  2. Arcade's pre restore when using the build.cmd/build.sh scripts.
  3. The repo's overall up-front restore (Build.proj or in other repos through a solution file)

Any extra invocation like

runtime/src/mono/mono.proj

Lines 611 to 612 in 432a459

<MSBuild Projects="$(RepoRoot)eng\empty.csproj"
Targets="Restore"/>
is adding risk as every additional restore invocation increases the chance for a misbehavior either on the client or on the server's side. Also such extra invocations reduce perf as additional NuGet invocations are actually quite slow.

CoreClr has some of such extra invocations in their build runtime scripts when shelving out to dotnet.exe to restore:

  1. powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -File "%__RepoRootDir%\eng\common\msbuild.ps1" /clp:nosummary %__ArcadeScriptArgs%^
    "%__RepoRootDir%\eng\empty.csproj" /t:GenerateRuntimeVersionFile /restore^
  2. powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -File "%__RepoRootDir%\eng\common\msbuild.ps1" /clp:nosummary %__ArcadeScriptArgs%^
    "%OptDataProjectFilePath%" /t:Restore^

Removing these invocation will make the native build faster and will improve the overall reliability.

@hoyosjs
Copy link
Member

hoyosjs commented Jul 28, 2021

Closing as there's no hits; #54921 tracks the centralized restore.

@hoyosjs hoyosjs closed this as completed Jul 28, 2021
@ghost ghost locked as resolved and limited conversation to collaborators Aug 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Projects
None yet
Development

No branches or pull requests

5 participants