Specific code layout can cause compiler panic with lto=true

[GinjaNinja32]

The specific setup I'm seeing to cause this is a three-crate workspace:

• parent, a binary, depending on shared
• child, a library, with crate-type = ["dylib"], depending on shared
• shared, a library, also with crate-type = ["dylib"]

For my specific use case, there are many crates like child, and parent loads them dynamically via libloading, but this is not required to reproduce the panic.

Script to create this (place in empty directory and run)

#! /bin/bash

set -euo pipefail

mkdir -p child/src
cat <<EOF > child/Cargo.toml
[package]
name = "child"
version = "0.1.0"

[lib]
crate-type = ["dylib"]

[dependencies]
shared = { path = "../shared" }
EOF
cat <<EOF > child/src/lib.rs
extern crate shared;
EOF

mkdir -p parent/src
cat <<EOF > parent/Cargo.toml
[package]
name = "parent"
version = "0.1.0"

[dependencies]
shared = { path = "../shared" }
EOF
cat <<EOF > parent/src/main.rs
extern crate shared;

fn main() {}
EOF

mkdir -p shared/src
cat <<EOF > shared/Cargo.toml
[package]
name = "shared"
version = "0.1.0"

[lib]
crate-type = ["dylib"]
EOF
cat <<EOF > shared/src/lib.rs
EOF

cat <<EOF > Cargo.toml
[workspace]
members = ["parent", "child", "shared"]

[profile.release]
lto = false
EOF

set -x

cat Cargo.toml

# OK
cargo build

# OK
cargo build --release

sed -i 's/lto = false/lto = true/' Cargo.toml

cat Cargo.toml

# Fails
RUST_BACKTRACE=full cargo build --release

I expected to see this happen: Either a successful build, or a standard compiler error.

Instead, this happened: LTO broke the compiler:

$ cargo build --release
   Compiling shared v0.1.0 (/home/nyx/rustbug/shared)
   Compiling child v0.1.0 (/home/nyx/rustbug/child)
   Compiling parent v0.1.0 (/home/nyx/rustbug/parent)
thread 'rustc' panicked at 'assertion failed: !are_upstream_rust_objects_already_included(sess)', src/librustc_codegen_llvm/back/link.rs:1402:9
stack backtrace:
   0:     0x7f7e1f91e793 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hf8722b0178fb1b63
                               at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
   1:     0x7f7e1f916c48 - std::sys_common::backtrace::_print::hc40139e5f1d656ee
                               at src/libstd/sys_common/backtrace.rs:70
   2:     0x7f7e1f91a922 - std::panicking::default_hook::{{closure}}::h993d43465919c16a
                               at src/libstd/sys_common/backtrace.rs:58
                               at src/libstd/panicking.rs:200
   3:     0x7f7e1f91a694 - std::panicking::default_hook::h73d2c2ec3d9ba5a4
                               at src/libstd/panicking.rs:215
   4:     0x7f7e1b3f631f - rustc::util::common::panic_hook::he83c1d31ee2662ae
   5:     0x7f7e1f91b109 - std::panicking::rust_panic_with_hook::h744417edfe714d72
                               at src/libstd/panicking.rs:482
   6:     0x7f7e153e8374 - std::panicking::begin_panic::hf2f2c845ae9fde35
   7:     0x7f7e15441d85 - rustc_codegen_llvm::back::link::link_natively::had7c6ec0331ae050
   8:     0x7f7e1543badf - rustc_codegen_llvm::back::link::link_binary::h190e3be9b1179a44
   9:     0x7f7e15323882 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link::{{closure}}::h378fa211691cf192
  10:     0x7f7e1531d843 - rustc::util::common::time::hba8b261a48e7f916
  11:     0x7f7e153570b4 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_utils::codegen_backend::CodegenBackend>::join_codegen_and_link::h1b338c342f4da9e8
  12:     0x7f7e1fcf48cd - rustc_driver::driver::compile_input::h65aaa3180b3e84e4
  13:     0x7f7e1fc4fdd0 - rustc_driver::run_compiler_with_pool::h7731b79ae97e4d21
  14:     0x7f7e1fc5bc05 - <scoped_tls::ScopedKey<T>>::set::h10a9d6691c6667b2
  15:     0x7f7e1fc4ec1a - rustc_driver::run_compiler::hccae18cc847caa81
  16:     0x7f7e1fc5bdba - <scoped_tls::ScopedKey<T>>::set::h71212949a1d6fa63
  17:     0x7f7e1fccd802 - std::sys_common::backtrace::__rust_begin_short_backtrace::h76607ec057ea37b9
  18:     0x7f7e1f92c529 - __rust_maybe_catch_panic
                               at src/libpanic_unwind/lib.rs:92
  19:     0x7f7e1fce4fa0 - <F as alloc::boxed::FnBox<A>>::call_box::h33b88f71d009a576
  20:     0x7f7e1f92b2ed - std::sys::unix::thread::Thread::new::thread_start::h6fb76bcf544b6181
                               at /rustc/2aa4c46cfdd726e97360c2734835aa3515e8c858/src/liballoc/boxed.rs:744
                               at src/libstd/sys_common/thread.rs:14
                               at src/libstd/sys/unix/thread.rs:81
  21:     0x7f7e1f6a06b9 - start_thread
  22:     0x7f7e1efca41c - clone
  23:                0x0 - <unknown>
query stack during panic:
end of query stack

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.33.0 (2aa4c46cf 2019-02-28) running on x86_64-unknown-linux-gnu

note: compiler flags: -C opt-level=3 -C lto --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: Could not compile `parent`.

To learn more, run the command again with --verbose.

Meta

rustc --version --verbose:

rustc 1.33.0 (2aa4c46cf 2019-02-28)
binary: rustc
commit-hash: 2aa4c46cfdd726e97360c2734835aa3515e8c858
commit-date: 2019-02-28
host: x86_64-unknown-linux-gnu
release: 1.33.0
LLVM version: 8.0

[hellow554]

Can you setup a repo with the example?

[GinjaNinja32]

https://github.com/GinjaNinja32/rust-issues/tree/59137

[hellow554]

Also happens on debug when lto=true is enabled.

[hellow554]

Has nothing to do with child. When you delete it, the ICE is still present.
Has nothing to do with incremental compiling.

[hellow554]

https://github.com/rust-lang/rust/blob/master/src/librustc_codegen_llvm/back/link.rs#L1399-L1401

rust/src/librustc_codegen_llvm/back/link.rs

Lines 1399 to 1401 in 2a8f6a7

	// If we're performing LTO, then it should have been previously required
	// that all upstream rust dependencies were available in an rlib format.
	assert!(!are_upstream_rust_objects_already_included(sess));

CC #53031
CC @michaelwoerister @alexcrichton

[michaelwoerister]

Thanks for the bug report, @GinjaNinja32! And thanks for already taking a look, @hellow554!

I suspect this is related to using dylib (i.e. "Rust dylibs") dependencies.

Nominating for priority assignment.

[pnkfelix]

triage: P-high. Leaving nominated for discussion at meeting.

[pnkfelix]

discussed at meeting. assigning to @michaelwoerister

[pnkfelix]

and removing nomination tag

[michaelwoerister]

The assertion here seems to be out of place. #59696 removes it which should fix the issue.