Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bootstrap is broken in rust-1.75.0-beta with: thread 'main' panicked at src/core/builder.rs:1795:26: std::fs::read_dir(registry_src) failed with No such file or directory (os error 2) #117885

Closed
stefson opened this issue Nov 13, 2023 · 8 comments · Fixed by #119582
Closed

bootstrap is broken in rust-1.75.0-beta with: thread 'main' panicked at src/core/builder.rs:1795:26: std::fs::read_dir(registry_src) failed with No such file or directory (os error 2) #117885

stefson opened this issue Nov 13, 2023 · 8 comments · Fixed by #119582
Labels
C-bug Category: This is a bug. P-medium Medium priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

Comments

@stefson
Copy link

stefson commented Nov 13, 2023

this is my attempt to bootstrap rust-1.75.0 beta, which fails with this beauty:

running sanity check
learning about cargo
> Assemble { target_compiler: Compiler { stage: 1, host: x86_64-gentoo-linux-musl } }
  > Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
  < Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
  > Rustc { target: x86_64-gentoo-linux-musl, compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, crates: [] }
    > Std { target: x86_64-gentoo-linux-musl, compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, crates: [], force_recompile: false, extra_rust_args: [] }
      > StartupObjects { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      < StartupObjects { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Assemble { target_compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl } }
      > Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
        > Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
Removing sysroot /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/build/x86_64-gentoo-linux-musl/stage0-sysroot to avoid caching bugs
        < Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
Removing sysroot /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/build/x86_64-gentoo-linux-musl/stage0-sysroot/lib/rustlib/x86_64-gentoo-linux-musl/lib to avoid caching bugs
        c Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
      < Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Libdir { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, target: x86_64-gentoo-linux-musl }
      c Sysroot { compiler: Compiler { stage: 0, host: x86_64-gentoo-linux-musl }, force_recompile: false }
thread 'main' panicked at src/core/builder.rs:1795:26:
std::fs::read_dir(registry_src) failed with No such file or directory (os error 2)
stack backtrace:
   0:     0x564e4c278d2c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h05bd3f3cfb4e4d10
   1:     0x564e4c2a732c - core::fmt::write::h48e8631484ee755d
   2:     0x564e4c27402e - std::io::Write::write_fmt::h0a6193b9593cb997
   3:     0x564e4c278b14 - std::sys_common::backtrace::print::he91a2fc2cccbc13b
   4:     0x564e4c27ac93 - std::panicking::default_hook::{{closure}}::h7e7cca84ab3b6a5a
   5:     0x564e4c27a9b4 - std::panicking::default_hook::hb9acc10797035fd2
   6:     0x564e4c27b215 - std::panicking::rust_panic_with_hook::habf92a251463da55
   7:     0x564e4c27b111 - std::panicking::begin_panic_handler::{{closure}}::ha11af79f81437361
   8:     0x564e4c279256 - std::sys_common::backtrace::__rust_end_short_backtrace::hbbd2d14762e87dfc
   9:     0x564e4c27ae62 - rust_begin_unwind
  10:     0x564e4b71d8f5 - core::panicking::panic_fmt::he0005c8db2b53995
  11:     0x564e4b95a0d9 - bootstrap::core::builder::Builder::cargo::h81a7b3de28be0597
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:1795:26
  12:     0x564e4b784d13 - <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::run::hef99c80683adc61d
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:209:25
  13:     0x564e4bc86e44 - bootstrap::core::builder::Builder::ensure::haf82eda529e87164
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  14:     0x564e4b78c76c - <bootstrap::core::build_steps::compile::Rustc as bootstrap::core::builder::Step>::run::h64561f053b21cfc9
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:778:9
  15:     0x564e4bbf0e54 - bootstrap::core::builder::Builder::ensure::h58213f8b5515ed44
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  16:     0x564e4b794564 - <bootstrap::core::build_steps::compile::Assemble as bootstrap::core::builder::Step>::run::h8d0bf6ea1c5a27d1
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:1594:9
  17:     0x564e4bc4ee83 - bootstrap::core::builder::Builder::ensure::h8cec5a594764982d
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:2142:23
  18:     0x564e4b950291 - bootstrap::core::builder::Builder::compiler::hfa7dd6787e9d598f
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:974:9
  19:     0x564e4b7843f4 - <bootstrap::core::build_steps::compile::Std as bootstrap::core::builder::Step>::make_run::h6d87d340d03044f2
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/build_steps/compile.rs:107:23
  20:     0x564e4b94a258 - bootstrap::core::builder::StepDescription::maybe_run::hb650c23b3ff31307
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:316:13
  21:     0x564e4b94afce - bootstrap::core::builder::StepDescription::run::hd04123386ea2e415
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:355:21
  22:     0x564e4b950223 - bootstrap::core::builder::Builder::run_step_descriptions::h6f290e94efac2e8f
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:966:9
  23:     0x564e4b94ffc6 - bootstrap::core::builder::Builder::execute_cli::hc0858ad5f355b7ee
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/core/builder.rs:947:9
  24:     0x564e4b9fb4cd - bootstrap::Build::build::h3bab7e9d07c88e90
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/lib.rs:670:17
  25:     0x564e4b720dcd - bootstrap::main::h1f0cea8e1c30d35c
                               at /var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/src/bin/main.rs:74:5
  26:     0x564e4b71eb83 - core::ops::function::FnOnce::call_once::h43f31234d165b8e7
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/core/src/ops/function.rs:250:5
  27:     0x564e4b71e406 - std::sys_common::backtrace::__rust_begin_short_backtrace::h59eef5b99cf3c0cb
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/sys_common/backtrace.rs:154:18
  28:     0x564e4b71e4f9 - std::rt::lang_start::{{closure}}::had666b1fe66fcdca
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/rt.rs:166:18
  29:     0x564e4c26b0d2 - std::rt::lang_start_internal::h0da414b7d71fa485
  30:     0x564e4b71e4d7 - std::rt::lang_start::hf899673c0c6d1ce0
                               at /rustc/22d4a4ff135fd79c06cf0274754bd571b1fff400/library/std/src/rt.rs:165:17
  31:     0x564e4b721c3e - main
  32:     0x7f18b57f2f1b - <unknown>
Traceback (most recent call last):
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/./x.py", line 50, in <module>
    bootstrap.main()
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1134, in main
    bootstrap(args)
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 1110, in bootstrap
    run(args, env=env, verbose=build.verbose, is_bootstrap=True)
  File "/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc-beta-src/src/bootstrap/bootstrap.py", line 187, in run
    raise RuntimeError(err)

this is my command:

python3.11 ./x.py build -vv --config="/var/tmp/portage/dev-lang/rust-1.75.0_beta/work/rustc_beta_src//config.toml -j12 || die

I can workaround this by reverting #115872

@pietroalbini can you please have a look at this error and the backtrace? I don't get why this error comes up. Thanks
@stefson stefson added the C-bug Category: This is a bug. label Nov 13, 2023
@rustbot rustbot added the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Nov 13, 2023
@stefson
Copy link
Author

stefson commented Nov 13, 2023

this is my config.toml:

[llvm]
download-ci-llvm = false
optimize = true
release-debuginfo = false
assertions = false
ninja = true
targets = "X86"
experimental-targets = ""
link-shared = true
[build]
build = "x86_64-gentoo-linux-musl"
host = ["x86_64-gentoo-linux-musl"]
target = ["x86_64-gentoo-linux-musl"]
cargo = "/usr/lib/rust/1.74.0/bin/cargo"
rustc = "/usr/lib/rust/1.74.0/bin/rustc"
docs = false
compiler-docs = false
submodules = false
python = "python3.11"
locked-deps = true
vendor = true
extended = true
tools = ["rustfmt","cargo","rustdoc"]
verbose = 2
sanitizers = false
profiler = false
cargo-native-static = false
[install]
prefix = "/usr/lib/rust/1.75.0_beta"
sysconfdir = "etc"
docdir = "share/doc/rust"
bindir = "bin"
libdir = "lib"
mandir = "share/man"
[rust]
# https://github.com/rust-lang/rust/issues/54872
codegen-units-std = 1
optimize = true
debug = false
debug-assertions = false
debuginfo-level-rustc = 0
backtrace = true
incremental = false
default-linker = "x86_64-gentoo-linux-musl-gcc"
parallel-compiler = false
channel = "stable"
rpath = false
verbose-tests = true
optimize-tests = true
codegen-tests = true
dist-src = false
remap-debuginfo = true
lld = false
backtrace-on-ice = true
jemalloc = false
lto = "off"
[dist]
src-tarball = false
compression-formats = ["xz"]
[target.x86_64-gentoo-linux-musl]
cc = "x86_64-gentoo-linux-musl-gcc"
cxx = "x86_64-gentoo-linux-musl-g++"
linker = "x86_64-gentoo-linux-musl-gcc"
ar = "x86_64-gentoo-linux-musl-ar"
crt-static = false
llvm-config = "/usr/lib/llvm/17/bin/llvm-config"

@bjorn3
Copy link
Member

bjorn3 commented Nov 14, 2023

The fixme here seems to be the issue:

rust/src/bootstrap/src/core/builder.rs

Lines 1792 to 1795 in 5526682

// FIXME: handle vendored sources
let registry_src = t!(home::cargo_home()).join("registry").join("src");
let mut env_var = OsString::new();
for entry in t!(std::fs::read_dir(registry_src)) {

As workaround you can try to run mkdir -p ~/.cargo/registry/src before the build.

cc @pietroalbini

@bjorn3 bjorn3 added regression-from-stable-to-beta Performance or correctness regression from stable to beta. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Nov 14, 2023
@rustbot rustbot added the I-prioritize Issue: Indicates that prioritization has been requested for this issue. label Nov 14, 2023
@stefson
Copy link
Author

stefson commented Nov 14, 2023

I'm compiling offline, from the beta src tarball.

@bjorn3
Copy link
Member

bjorn3 commented Nov 14, 2023

I'm aware. It seems that there is a bug when using vendored sources in combination with remap-debuginfo. Creating the ~/.cargo/registry/src directory should help even if it is kept empty.

@apiraino
Copy link
Contributor

WG-prioritization assigning priority (Zulip discussion).

@rustbot label -I-prioritize +P-medium

@rustbot rustbot added P-medium Medium priority and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Nov 14, 2023
@stefson
Copy link
Author

stefson commented Nov 14, 2023

Well, the cargo eclass is already exporting CARGO_HOME env var, and a CARGO_REGISTRY_DIR as well, and I'm doubtfull about the package manager allowing x.py to escape it's sandbox. Neverthelss, I'll try your suggestion when I'm back home on the machine.

@bjorn3
Copy link
Member

bjorn3 commented Nov 14, 2023

If you have CARGO_HOME set the directory you need to create is $CARGO_HOME/.cargo/registry/src.

@Noratrieb Noratrieb removed the needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. label Nov 16, 2023
@stefson
Copy link
Author

stefson commented Dec 28, 2023

@xen0n for your information only

@akiernan akiernan mentioned this issue Jan 2, 2024
Closed
@arlosi arlosi mentioned this issue Jan 4, 2024
Merged
@arlosi arlosi added the regression-from-stable-to-stable Performance or correctness regression from one stable version to another. label Jan 4, 2024
@bors bors closed this as completed in 1caa419 Jan 19, 2024
rust-timer added a commit to rust-lang-ci/rust that referenced this issue Jan 19, 2024
@rust-timer
Unrolled build for rust-lang#119582
ad29c6e 
Rollup merge of rust-lang#119582 - arlosi:bootstrap-vendor-remap, r=wesleywiser

bootstrap: handle vendored sources when remapping crate paths

rust-lang#115872 introduced a feature to add path remapping for crate dependencies, but only when they came from Cargo's registry cache, not a vendor directory.

This caused builds that used remapped debuginfo and vendor directories to fail with:
```
std::fs::read_dir(registry_src) failed with No such file or directory (os error 2)
```
or (if the `registry/src` directory exists but is empty)
```
error: --remap-path-prefix must contain '=' between FROM and TO
```

Fixes rust-lang#117885 by explicitly supporting the `vendor` directory and adding it to `RUSTC_CARGO_REGISTRY_SRC_TO_REMAP`.

Note that `bootstrap.py` already assumes that `./vendor` within the rust repo is the only supported vendoring location.

r? `@pietroalbini`
msrd0 added a commit to msrd0/docker-abuild-arm that referenced this issue Jan 19, 2024
@msrd0
work around rust-lang/rust#117885
bdf4b57
@apiraino apiraino removed the regression-from-stable-to-beta Performance or correctness regression from stable to beta. label Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Category: This is a bug. P-medium Medium priority regression-from-stable-to-stable Performance or correctness regression from one stable version to another. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bootstrap: handle vendored sources when remapping crate paths arlosi/rust
6 participants
@arlosi @apiraino @stefson @bjorn3 @rustbot @Noratrieb