Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release/7.0.2xx] Runtime fails to generate native System.Private.CoreLib in source-build bootstrap #29177

Closed
lbussell opened this issue Nov 18, 2022 · 13 comments · Fixed by dotnet/runtime#78733
Closed

[release/7.0.2xx] Runtime fails to generate native System.Private.CoreLib in source-build bootstrap #29177

lbussell opened this issue Nov 18, 2022 · 13 comments · Fixed by dotnet/runtime#78733
Assignees
@nagilson
Labels
Area-Infrastructure

Comments

@lbussell
Copy link
Contributor

I have been looking at this issue for a little bit and can't seem to make any progress, so I'm hoping someone here can point me in the right direction.

The runtime-portable build is failing in the 7.0.2xx source build bootstrap scenario only with the following error:

      Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to 
      /tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
      Specify which project file to use because this '/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr' contains more than one project file.
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
    
    Build FAILED.
    
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
        0 Warning(s)
        1 Error(s)

Inside src/coreclr are multiple project files, crossgen-corelib.proj, runtime.proj, and runtime-prereqs.proj. I checked that these are also present in the runtime that's in .NET 7.0.1xx.
I also checked that the error from the .NET CLI isn't new to .NET 7.0.2xx. I'm not sure where else to look. Maybe @ViktorHofer knows who to tag?
@ghost ghost added the untriaged Request triage from a team member label Nov 18, 2022
@ghost
Copy link

ghost commented Nov 18, 2022

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

I have been looking at this issue for a little bit and can't seem to make any progress, so I'm hoping someone here can point me in the right direction.

The runtime-portable build is failing in the 7.0.2xx source build bootstrap scenario only with the following error:

      Generating native image of System.Private.CoreLib for Linux.x64.Release. Logging to 
      /tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
      Specify which project file to use because this '/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr' contains more than one project file.
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
    
    Build FAILED.
    
    /tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/crossgen-corelib.proj(104,5): error MSB3073: The command "/tarball/src/runtime/artifacts/source-build/self/src/dotnet.sh /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/tarball/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/" exited with code 1.
        0 Warning(s)
        1 Error(s)

Inside src/coreclr are multiple project files, crossgen-corelib.proj, runtime.proj, and runtime-prereqs.proj. I checked that these are also present in the runtime that's in .NET 7.0.1xx.
I also checked that the error from the .NET CLI isn't new to .NET 7.0.2xx. I'm not sure where else to look. Maybe @ViktorHofer knows who to tag?

Author: lbussell
Assignees: -
Labels:

area-Infrastructure-coreclr
Milestone: -

@ViktorHofer
Copy link
Member

cc @MichaelSimons @jkotas

@lbussell
Copy link
Contributor Author

Also, here is a link to the failure in CI: https://dev.azure.com/dnceng/internal/_build/results?buildId=2048260&view=logs&j=05a957c8-6537-5d6d-69e3-d72786d35c31&t=8f2f8c4f-4a49-5a74-0521-8049d6bde132&l=25268

And the binlog:
runtime.zip

@jkotas
Copy link
Member

jkotas commented Nov 19, 2022

This is one of the first places where the runtime that was just built runs an app. This failure likely means that the runtime that was just built is badly broken.

What are the steps to reproduce this locally?

@lbussell
Copy link
Contributor Author

This can be reproduced in a source-build bootstrap context. The fastest way is to

  1. Download dotnet-sdk-source-7.0.200.tar.gz from here
  2. Download dotnet-sdk-7.0.200-fedora.36-x64.tar.gz, Private.SourceBuilt.Artifacts.7.0.200.tar.gz from here.
  3. Extract the source tarball
  4. Extract the SDK to <extracted source tarball>/.dotnet
  5. Copy the Private.SourceBuilt.Artifacts archive to <extracted source tarball>/packages/archive
  6. In the container mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-36, run ./build.sh --with-sdk .dotnet/ from the root of the extracted source tarball.

@jkotas
Copy link
Member

jkotas commented Nov 21, 2022

When I re-run the failing command, I see that it is trying to download .NET SDK 7.0.200 and failing:

[root@a1d109dfbb94 ~]# /root/src/runtime/artifacts/source-build/self/src/dotnet.sh /root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll -o:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/System.Private.CoreLib.dll -r:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/*.dll --targetarch:x64 -O /root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/IL/System.Private.CoreLib.dll --perfmap-format-version:1 --perfmap --perfmap-path:/root/src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/
Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
Attempting to install dotnet from public_location.
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Attempting to download using primary link https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetcli.azureedge.net/dotnet/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet-install: Attempting to download using primary link https://dotnetbuilds.azureedge.net/public/Sdk/7.0[root@a1d109dfbb94 ~]# cd ..
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet_install: Error: Could not find `.NET Core SDK` with version = 7.0.200
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet from public_location.
Attempting to install dotnet from dotnetbuilds.
dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.

dotnet-install: Attempting to download using primary link https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz
curl: (22) The requested URL returned error: 404
dotnet-install: The resource at primary link 'https://dotnetbuilds.azureedge.net/public/Sdk/7.0.200/dotnet-sdk-7.0.200-linux-x64.tar.gz' is not available.
dotnet_install: Error: Could not find `.NET Core SDK` with version = 7.0.200
dotnet_install: Error: Refer to: https://aka.ms/dotnet-os-lifecycle for information on .NET Core support
Failed to install dotnet from dotnetbuilds.
Failed to install dotnet SDK from any of the specified locations.
[root@a1d109dfbb94 ~]#

What is the runtime that you expect /root/src/runtime/artifacts/source-build/self/src/dotnet.sh script to use?

@lbussell
Copy link
Contributor Author

It should be using the SDK from .dotnet/, which I believe it does when run from build.sh. I've seen the same thing on my end when trying to run just the failing command by itself . my guess is running the command alone like that skips some of the scaffolding needed to run the build correctly. @crummel or @MichaelSimons any debugging tips here?

@crummel
Copy link
Contributor

crummel commented Nov 22, 2022

These variables are what we set to get everything using the correct SDK, I'm not sure which in particular is the right one for runtime but just setting all of them should work fine.

@jkotas
Copy link
Member

jkotas commented Nov 22, 2022

I have set these variables to point to .dotnet directory created by step "4. Extract the SDK to /.dotnet" above.

The problem is that the path to crossgen2 that this is trying to execute is .../src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/crossgen2.dll . This file does not exist. The actual path to this file is .../src/runtime/artifacts/source-build/self/src/artifacts/bin/coreclr/Linux.x64.Release/crossgen2/linux-x64/crossgen2.dll. Notice extra linux-x64.

Is looks like there is a breaking change in the .NET SDK 7.0.xxx that added linux-x64 to the build output path. What's the right way to disable the new behavior? The crossgen2 build output path is RID specific already, it does not make sense to nest it further.

cc @dsplaisted

@jkotas
Copy link
Member

jkotas commented Nov 22, 2022

The problem was introduced by changes in sdk\7.0.200\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets . I have confirmed that the issue goes away if I copy Microsoft.NET.RuntimeIdentifierInference.targets from .NET SDK 7.0.100. @lbussell If you need to unblock the source build, you should be able to add temporary source build patch that reverts Microsoft.NET.RuntimeIdentifierInference.targets to .NET SDK 7.0.100 version.

@nagilson Could you please investigate whether the behavior change is a bug in .NET SDK, or whether this behavior is by design and we need to workaround it in dotnet/runtime build?

@jkotas jkotas transferred this issue from dotnet/runtime Nov 22, 2022
@nagilson
Copy link
Member

Thanks @jkotas for the ping and investigation. It was probably this PR: #28484

Before I can give a definitive answer, I need a bit more information ( see Questions). But if I understand the situation correctly, this is by design. We can certainly talk about if it should or shouldn't be by design if 'ya want.

Workarounds

First: @lbussell as a work-around, setting UseCurrentRuntimeIdentifier = false should disable this behavior. Please let me know how that goes.

Another option may be: AppendRuntimeIdentifierToOutputPath = false which you can set, but that will just change the pathing; you'll still get the RID, so it's a hacky workaround.

Explanation

I'm guessing this is caused by the following: There is a point in time in which crossgen2 uses RuntimeIdentifiers but does not set a RuntimeIdentifier. If crossgen2 depends on any of the following: SelfContained, or any Publish* property like PublishSingleFile, the SDK detects this and gives it a RuntimeIdentifier. Now it has a RuntimeIdentifiers and RuntimeIdentifier.

Questions

Is the above information correct, may someone from crossgen comment on the claim that it sets RuntimeIdentifiers but no RuntimeIdentifier? And do you use one of the following SelfContained or Publish* properties? If so, I'd imagine you had some special logic before to use them, because those properties require a RuntimeIdentifier to be set?

@nagilson nagilson self-assigned this Nov 22, 2022
@nagilson nagilson removed the untriaged Request triage from a team member label Nov 22, 2022
jkotas added a commit to jkotas/runtime that referenced this issue Nov 22, 2022
@jkotas
Explicitly omit RID from crossgen2 binary path
7f74fac 
Fixes dotnet/sdk#29177
@jkotas jkotas mentioned this issue Nov 22, 2022
Merged
@jkotas
Copy link
Member

jkotas commented Nov 22, 2022

it sets RuntimeIdentifiers but no RuntimeIdentifier? And do you use one of the following SelfContained or Publish* properties?

Yes, crossgen project does exactly that: https://github.com/dotnet/runtime/tree/main/src/coreclr/tools/aot/crossgen2

Another option may be: AppendRuntimeIdentifierToOutputPath = false

This sounds like the most appropriate fix. Submitted dotnet/runtime#78733 (verified locally that it fixes the build break)

github-actions bot pushed a commit to dotnet/runtime that referenced this issue Nov 22, 2022
@jkotas
Explicitly omit RID from crossgen2 binary path
77ffc87 
Fixes dotnet/sdk#29177
jkotas added a commit to dotnet/runtime that referenced this issue Nov 23, 2022
@jkotas
Explicitly omit RID from crossgen2 binary path (#78733)
563297d 
Fixes dotnet/sdk#29177
carlossanlop pushed a commit to dotnet/runtime that referenced this issue Nov 23, 2022
@github-actions @jkotas
Explicitly omit RID from crossgen2 binary path (#78735)
de84cf9 
Fixes dotnet/sdk#29177

Co-authored-by: Jan Kotas <[email protected]>
@jkotas jkotas mentioned this issue Nov 24, 2022
Open
@akoeplinger
Copy link
Member

it sets RuntimeIdentifiers but no RuntimeIdentifier? And do you use one of the following SelfContained or Publish* properties?

This is also the default setup for Android/iOS builds, see e.g.:

https://github.com/xamarin/xamarin-android/blob/ac643dddb5e9ef968fa758a6bc77a084242780e7/src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/targets/Microsoft.Android.Sdk.DefaultProperties.targets#L71-L73

nagilson added a commit to dotnet/docs that referenced this issue Nov 29, 2022
@nagilson
Document breaking changes for workloads and RID inference
bcc3115 
The additional information is included above. We didn't expect RID Inference to be a breaking change but there were after-effects, like here: dotnet/sdk#29164 or here dotnet/sdk#29177
@nagilson nagilson mentioned this issue Nov 29, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nagilson nagilson

Labels
Area-Infrastructure
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Explicitly omit RID from crossgen2 binary path jkotas/runtime
6 participants
@akoeplinger @jkotas @ViktorHofer @crummel @nagilson @lbussell