x test core fails when download-rustc is enabled

[jyn514]

I tried this code: ./configure --set download-rustc && x test core

I expected to see this happen: All tests pass

Instead, this happened:

Testing {core} stage2 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling core v0.0.0 (/home/jyn/src/rust/library/core)
error[E0773]: attempted to define built-in macro more than once
    --> /home/jyn/src/rust/library/core/src/macros/mod.rs:1310:5
     |
1310 |     macro_rules! cfg {
     |     ^^^^^^^^^^^^^^^^
     |
note: previously defined here
    --> /rustc/ce1073ba9d894b2e351b2a85fcd39f2c99b78974/library/core/src/macros/mod.rs:1310:5

error[E0152]: duplicate lang item in crate `core` (which `corebenches` depends on): `sized`.
  |
  = note: the lang item is first defined in crate `core` (which `std` depends on)
  = note: first definition in `core` loaded from /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-05898138a596088a.rlib
  = note: second definition in `core` loaded from /home/jyn/src/rust/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/release/deps/libcore-83f7e02d8b0410a3.rlib

Some errors have detailed explanations: E0152, E0277, E0307, E0773.
error: could not compile `core` (test "coretests") due to 372 previous errors; 10 warnings emitted

Meta

HEAD is branched from ce1073b

[jyn514]

Here is where std is loaded from in stage 0:

INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7217c7a623948b1f.rlib
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7217c7a623948b1f.so
INFO rustc_metadata::locator rlib reading metadata from: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-7217c7a623948b1f.rlib

which requires loading core from the sysroot:

INFO rustc_metadata::creader resolving dep crate core hash: `34298ac765912203` extra filename: `-999232a2b2133b8a`
INFO rustc_metadata::creader resolving crate `core`
INFO rustc_metadata::creader falling back to a load
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-999232a2b2133b8a.rmeta
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-999232a2b2133b8a.rlib
INFO rustc_metadata::locator rmeta reading metadata from: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/libcore-999232a2b2133b8a.rmeta
INFO rustc_metadata::creader register crate `core` (cnum = 2. private_dep = false)

and in stage 2:

INFO rustc_metadata::creader resolving crate `std`
INFO rustc_metadata::creader falling back to a load
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-cd20bc9d771091ea.so
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-cd20bc9d771091ea.rlib
INFO rustc_metadata::locator rlib reading metadata from: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-cd20bc9d771091ea.rlib

INFO rustc_metadata::creader resolving dep crate core hash: `ca783ee9d7dc024f` extra filename: `-05898138a596088a`
INFO rustc_metadata::creader falling back to a load
INFO rustc_metadata::locator lib candidate: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-05898138a596088a.rlib
INFO rustc_metadata::locator rlib reading metadata from: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-05898138a596088a.rlib
INFO rustc_metadata::creader register crate `core` (cnum = 2. private_dep = false)

So far so good, core/std are both loaded from the sysroot at first. But then, core gets loaded again in stage 2:

INFO rustc_metadata::creader resolving crate `rustc_std_workspace_core`
INFO rustc_metadata::creader resolving dep crate core hash: `ca783ee9d7dc024f` extra filename: `-05898138a596088a`
INFO rustc_metadata::creader resolving dep crate unicode_width hash: `1ee9c73903764d5e` extra filename: `-e2c026a3fbc51a2a`
INFO rustc_metadata::creader resolving crate `unicode_width`
INFO rustc_metadata::creader resolving crate `core`
INFO rustc_metadata::creader resolving crate `core`
INFO rustc_metadata::creader resolving dep crate compiler_builtins hash: `8adb4d62f5b608ec` extra filename: `-35b8a4bd2de4e62e`
INFO rustc_metadata::creader falling back to a load
INFO rustc_metadata::creader resolving crate `compiler_builtins`
INFO rustc_metadata::creader resolving dep crate core hash: `ca783ee9d7dc024f` extra filename: `-05898138a596088a`
INFO rustc_metadata::creader resolving crate `core`
INFO rustc_metadata::creader falling back to a load
INFO rustc_metadata::locator rlib reading metadata from: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2-std/x86_64-unknown-linux-gnu/release/deps/libcore-ac23cab5970c56fe.rlib

I think what's going wrong here is that rustc-std-workspace-core explicitly has #![no_core] and a path dependency, so it uses the version in the workspace instead of from the sysroot. In stage0 this works ok, because libcore.rlib is exactly the same in stage0-std and stage0-sysroot. But in stage2, the two differ, and resolve gives a duplicate macro error.

[jyn514]

I had the idea to not download rust-std at all when download-rustc is enabled, only rustc, and unconditionally build std from source. Unfortunately that doesn't work because Sysroot returns the same thing for both stages 0 and 1:

rust/src/bootstrap/compile.rs

Lines 1247 to 1268 in da48140

	impl Step for Sysroot {
	type Output = Interned<PathBuf>;
	fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
	run.never()
	}
	/// Returns the sysroot for the `compiler` specified that *this build system
	/// generates*.
	///
	/// That is, the sysroot for the stage0 compiler is not what the compiler
	/// thinks it is by default, but it's the same as the default for stages
	/// 1-3.
	fn run(self, builder: &Builder<'_>) -> Interned<PathBuf> {
	let compiler = self.compiler;
	let host_dir = builder.out.join(&compiler.host.triple);
	let sysroot_dir = |stage| {
	if stage == 0 {
	host_dir.join("stage0-sysroot")
	} else if builder.download_rustc() && compiler.stage != builder.top_stage {
	host_dir.join("ci-rustc-sysroot")

so we end up copying two different stages of std into the same ci-rustc-sysroot, which gives an error:

error[E0464]: multiple candidates for `rlib` dependency `core` found
  |
  = note: candidate #1: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/li
bcore-05898138a596088a.rlib
  = note: candidate #2: /home/jyn/src/rust3/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/li
bcore-ac23cab5970c56fe.rlib

I think the right fix looks something like "in impl Step for Std, build std from source instead of using the download-rustc artifacts BUT only if compiler.stage == builder.top_stage.

anyway, I'm out of time for tonight.

[jyn514]

I think the right fix looks something like "in impl Step for Std, build std from source instead of using the download-rustc artifacts BUT only if compiler.stage == builder.top_stage.

For this to work, we need to avoid copying rust-std into stage2 in the Sysroot step unless it's asked for, just how we avoided it for rustc-dev in #110229.