Cargo Build bricks, "Error: future cannot be sent between threads safely", on two different baremetal Linux

[NicholasTroutman]

Describe the bug
A clear and concise description of what the bug is.
On new installations in non-wsl environments, Ubuntu 22.04 and Arch Linux, cargo build breaks with 7 "error: future cannot be sent between threads safely" errors.
This occurs on two different machines, and two different x86_64 operating systems.

Indicate any of these common scenarios that apply:

• a program hangs under hermit
• hermit panics internally
• hermit runs the program but divergence (nondeterminism) occurs

To Reproduce
Minimal input to reproduce the behavior.
On baremetal x86_64 linux, git clone, update

Expected behavior
A clear and concise description of what you expected to happen.
Cargo build should pass all building/compiling checks and end.

Environment

• Linux kernel version (uname -a): 6.4.10-arch1-1 & Ubuntu 22.04
• CPU version (/proce/cpuinfo):
• Linux distro flavor (/etc/issue, /etc/redhat-release):

Additional context
Attach the logs to this issue as a text file generated by hermit --log=trace --log-file=FOO run.

Add any other context about the problem here.

USER: cargo clean
     Removed 2334 files, 1.1GiB total
USER: cargo build
   Compiling proc-macro2 v1.0.79
   Compiling unicode-ident v1.0.12
   Compiling libc v0.2.153
   Compiling cfg-if v1.0.0
   Compiling autocfg v1.1.0
   Compiling serde v1.0.197
   Compiling version_check v0.9.4
   Compiling memchr v2.7.1
   Compiling once_cell v1.19.0
   Compiling syn v1.0.109
   Compiling bitflags v1.3.2
   Compiling pin-project-lite v0.2.13
   Compiling parking_lot_core v0.9.9
   Compiling smallvec v1.13.1
   Compiling libm v0.2.8
   Compiling scopeguard v1.2.0
   Compiling io-lifetimes v1.0.11
   Compiling pin-utils v0.1.0
   Compiling lazy_static v1.4.0
   Compiling regex-syntax v0.8.2
   Compiling futures-sink v0.3.30
   Compiling futures-core v0.3.30
   Compiling ahash v0.7.8
   Compiling crc32fast v1.4.0
   Compiling lock_api v0.4.11
   Compiling num-traits v0.2.18
   Compiling slab v0.4.9
   Compiling memoffset v0.6.5
   Compiling aho-corasick v1.1.2
   Compiling syscalls v0.6.17
   Compiling convert_case v0.4.0
   Compiling typenum v1.17.0
   Compiling adler v1.0.2
   Compiling quote v1.0.35
   Compiling miniz_oxide v0.7.2
   Compiling syn v2.0.52
   Compiling futures-channel v0.3.30
   Compiling indexmap v1.9.3
   Compiling generic-array v0.14.7
   Compiling tracing-core v0.1.32
   Compiling futures-task v0.3.30
   Compiling iana-time-zone v0.1.60
   Compiling thiserror v1.0.58
   Compiling futures v0.1.31
   Compiling futures-io v0.3.30
   Compiling flate2 v1.0.28
   Compiling proc-macro-error-attr v1.0.4
   Compiling getrandom v0.2.12
   Compiling mio v0.8.11
   Compiling socket2 v0.5.6
   Compiling num_cpus v1.16.0
   Compiling parking_lot v0.12.1
   Compiling signal-hook-registry v1.4.1
   Compiling regex-automata v0.4.6
   Compiling hashbrown v0.12.3
   Compiling rand_core v0.6.4
   Compiling nix v0.25.1
   Compiling rustix v0.37.27
   Compiling byteorder v1.5.0
   Compiling static_assertions v1.1.0
   Compiling paste v1.0.14
   Compiling twox-hash v1.6.3
   Compiling proc-macro-error v1.0.4
   Compiling serde_json v1.0.114
   Compiling rustix v0.36.17
   Compiling itoa v1.0.10
   Compiling cpp_demangle v0.4.3
   Compiling linux-raw-sys v0.3.8
   Compiling anyhow v1.0.81
   Compiling ppv-lite86 v0.2.17
   Compiling unicase v2.7.0
   Compiling rand_chacha v0.3.1
   Compiling colored v2.1.0
   Compiling hex v0.4.3
   Compiling async-trait v0.1.77
   Compiling unicode-width v0.1.11
   Compiling linux-raw-sys v0.1.4
   Compiling stable_deref_trait v1.2.0
   Compiling procfs v0.15.1
   Compiling ryu v1.0.17
   Compiling gimli v0.28.1
   Compiling rand v0.8.5
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.6
   Compiling regex v1.10.3
   Compiling memmap2 v0.5.10
   Compiling rustix v0.38.31
   Compiling typed-arena v2.0.2
   Compiling heck v0.4.1
   Compiling os_str_bytes v6.6.1
   Compiling terminal_size v0.2.6
   Compiling rustc-demangle v0.1.23
   Compiling fallible-iterator v0.3.0
   Compiling clap_lex v0.2.4
   Compiling textwrap v0.16.1
   Compiling digest v0.10.7
   Compiling object v0.29.0
   Compiling atty v0.2.14
   Compiling raw-cpuid v10.7.0
   Compiling cpufeatures v0.2.12
   Compiling strsim v0.10.0
   Compiling termcolor v1.4.1
   Compiling either v1.10.0
   Compiling never-say-never v6.6.666
   Compiling linux-raw-sys v0.4.13
   Compiling radium v0.3.0
   Compiling sha2 v0.10.8
   Compiling fastrand v2.0.1
   Compiling arrayvec v0.5.2
   Compiling bytesize v1.3.0
   Compiling shell-words v1.1.0
   Compiling diff v0.1.13
   Compiling yansi v0.5.1
   Compiling pretty v0.12.3
   Compiling serde_derive v1.0.197
   Compiling serde_repr v0.1.18
   Compiling tokio-macros v2.2.0
   Compiling futures-macro v0.3.30
   Compiling thiserror-impl v1.0.58
   Compiling tracing-attributes v0.1.27
   Compiling pretty_assertions v1.4.0
   Compiling rand_distr v0.4.3
   Compiling log v0.4.21
   Compiling pkg-config v0.3.30
   Compiling regex-syntax v0.6.29
   Compiling derive_more v0.99.17
   Compiling clap_derive v3.2.25
   Compiling powerfmt v0.2.0
   Compiling time-core v0.1.2
   Compiling overload v0.1.1
   Compiling deranged v0.3.11
   Compiling num-conv v0.1.0
   Compiling futures-util v0.3.30
   Compiling num_threads v0.1.7
   Compiling unwind-sys v0.1.4
   Compiling nu-ansi-term v0.46.0
   Compiling tracing-log v0.2.0
   Compiling scroll_derive v0.11.1
   Compiling foreign-types-macros v0.2.3
   Compiling sharded-slab v0.1.7
   Compiling thread_local v1.1.8
   Compiling time v0.3.34
   Compiling crossbeam-utils v0.8.19
   Compiling tracing v0.1.40
   Compiling unwind v0.4.2
   Compiling regex-automata v0.1.10
   Compiling foreign-types-shared v0.3.1
   Compiling scroll v0.11.0
   Compiling env_logger v0.8.4
   Compiling foreign-types v0.5.0
   Compiling plain v0.2.3
   Compiling goblin v0.5.4
   Compiling fbinit_macros v0.1.2 (https://github.com/facebookexperimental/rust-shed.git?branch=main#4a0b610c)
   Compiling matchers v0.1.0
   Compiling quickcheck v1.0.3
   Compiling perf-event-open-sys v4.0.0
   Compiling iced-x86 v1.21.0
   Compiling clap v3.2.25
   Compiling crossbeam-channel v0.5.12
   Compiling dirs-sys v0.3.7
   Compiling futures-executor v0.3.30
   Compiling fbinit v0.1.2 (https://github.com/facebookexperimental/rust-shed.git?branch=main#4a0b610c)
   Compiling cfg-if v0.1.10
   Compiling futures v0.3.30
   Compiling sha1_smol v1.0.0
   Compiling atomic v0.5.3
   Compiling dirs v2.0.2
   Compiling test-allocator v0.0.0 (/home/kaiju/Documents/hermit/common/test-allocator)
   Compiling similar v2.4.0
   Compiling hermetic_infra_hermit_flaky-tests v0.0.0 (/home/kaiju/Documents/hermit/flaky-tests)
   Compiling ruzstd v0.5.0
   Compiling object v0.32.2
   Compiling bitflags v2.4.2
   Compiling bytes v1.5.0
   Compiling chrono v0.4.35
   Compiling bincode v1.3.3
   Compiling linked-hash-map v0.5.6
   Compiling bitvec v0.17.4
   Compiling rand_pcg v0.3.1
   Compiling digest v0.0.0 (/home/kaiju/Documents/hermit/common/digest)
   Compiling tracing-serde v0.1.3
   Compiling edit-distance v0.0.0 (/home/kaiju/Documents/hermit/common/edit-distance)
   Compiling uuid v1.7.0
   Compiling tokio v1.36.0
   Compiling reverie-memory v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
   Compiling reverie-syscalls v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
   Compiling addr2line v0.21.0
   Compiling tracing-subscriber v0.3.18
   Compiling tempfile v3.10.1
   Compiling tracing-appender v0.2.3
   Compiling reverie-process v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
   Compiling tokio-util v0.7.10
   Compiling tokio-stream v0.1.14
   Compiling reverie v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
   Compiling detcore-model v0.0.0 (/home/kaiju/Documents/hermit/detcore-model)
   Compiling safeptrace v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
warning: the item `FromIterator` is imported redundantly
  --> detcore-model/src/collections/replay_cursor.rs:11:5
   |
11 | use std::iter::FromIterator;
   |     ^^^^^^^^^^^^^^^^^^^^^^^
  --> /rustc/3cbb93223f33024db464a4df27a13c7cce870173/library/std/src/prelude/mod.rs:129:13
   |
   = note: the item `FromIterator` is already defined here
   |
   = note: `#[warn(unused_imports)]` on by default

   Compiling reverie-ptrace v0.1.0 (https://github.com/facebookexperimental/reverie.git?branch=main#29ce9faa)
   Compiling detcore v0.0.0 (/home/kaiju/Documents/hermit/detcore)
warning: `detcore-model` (lib) generated 1 warning
warning: the item `RawFd` is imported redundantly
  --> detcore/src/fd.rs:23:5
   |
23 | use crate::types::RawFd;
   |     ^^^^^^^^^^^^^^^^^^^
24 | use crate::types::*;
   |     --------------- the item `RawFd` is already imported here
   |
   = note: `#[warn(unused_imports)]` on by default

warning: the item `clap` is imported redundantly
  --> detcore/src/logdiff.rs:23:5
   |
23 | use clap;
   |     ^^^^ the item `clap` is already defined by prelude

warning: the item `FromIterator` is imported redundantly
  --> detcore/src/preemptions.rs:13:5
   |
13 | use std::iter::FromIterator;
   |     ^^^^^^^^^^^^^^^^^^^^^^^
  --> /rustc/3cbb93223f33024db464a4df27a13c7cce870173/library/std/src/prelude/mod.rs:129:13
   |
   = note: the item `FromIterator` is already defined here

warning: the item `SigWrapper` is imported redundantly
   --> detcore/src/lib.rs:104:5
    |
94  | use types::*;
    |     -------- the item `SigWrapper` is already imported here
...
104 | use crate::types::SigWrapper;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:607:37
    |
607 |       ) -> Result<CpuIdResult, Errno> {
    |  _____________________________________^
608 | |         trace!("handle_cpuid_event: eax: {}, ecx: {}", eax, ecx);
609 | |         self.pre_handler_hook(guest, false).await;
610 | |         let res = if self.cfg.virtualize_cpuid {
...   |
641 | |         Ok(res)
642 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:607:37: 642:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:607:37: 642:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<CpuIdResult, reverie::Errno>> + std::marker::Send>>`

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:648:37
    |
648 |       ) -> Result<RdtscResult, Errno> {
    |  _____________________________________^
649 | |         trace!("handle_rdtsc_event: {:?}", request);
650 | |         self.pre_handler_hook(guest, false).await;
651 | |         let result = if guest.config().virtualize_time {
...   |
688 | |         result
689 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:648:37: 689:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:648:37: 689:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<RdtscResult, reverie::Errno>> + std::marker::Send>>`

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:696:40
    |
696 |       ) -> Result<Option<Signal>, Errno> {
    |  ________________________________________^
697 | |         if signal == Signal::SIGINT && self.cfg.sigint_instakill {
698 | |             warn!("Fatal: Exiting hermit container immediately upon SIGINT");
699 | |             unrecoverable_shutdown(guest).await
...   |
741 | |         }
742 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:696:40: 742:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:696:40: 742:6}>>` to `Pin<Box<dyn Future<Output = Result<Option<Signal>, Errno>> + Send>>`
    = note: the full name for the type has been written to '/home/kaiju/Documents/hermit/target/debug/deps/detcore-392b287cb643b2af.long-type-3993446771905929726.txt'
    = note: consider using `--verbose` to print the full type name to the console

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:812:93
    |
812 |       async fn handle_thread_start<G: Guest<Self>>(&self, guest: &mut G) -> Result<(), Error> {
    |  _____________________________________________________________________________________________^
813 | |         let detpid = DetPid::from_raw(guest.pid().into());
814 | |         trace!(
815 | |             "[tid {}] detcore handle_thread_start, pid={}",
...   |
857 | |         Ok(())
858 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:812:93: 858:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:812:93: 858:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<(), reverie::Error>> + std::marker::Send>>`

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:860:90
    |
860 |       async fn handle_post_exec<G: Guest<Self>>(&self, guest: &mut G) -> Result<(), Errno> {
    |  __________________________________________________________________________________________^
861 | |         guest.thread_state_mut().past_global_first_execve = true;
862 | |         self.pre_handler_hook(guest, false).await;
863 | |
...   |
878 | |         Ok(())
879 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:860:90: 879:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:860:90: 879:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<(), reverie::Errno>> + std::marker::Send>>`

error: future cannot be sent between threads safely
   --> detcore/src/lib.rs:882:71
    |
882 |       async fn handle_timer_event<G: Guest<Self>>(&self, guest: &mut G) {
    |  _______________________________________________________________________^
883 | |         info!(
884 | |             "[detcore, dtid {}] inbound timer preemption event",
885 | |             guest.thread_state().dettid
...   |
917 | |         self.post_handler_hook(guest).await;
918 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:882:71: 918:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
   --> detcore/src/lib.rs:244:30
    |
241 |                 "(pre) registers [dtid {}][rcbs {}]. {}",
    |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244 |                 guest.regs().await.display()
    |                              ^^^^^ await occurs here, with `{}` maybe used later
    = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:882:71: 918:6}>>` to `Pin<Box<dyn futures::Future<Output = ()> + std::marker::Send>>`

error: future cannot be sent between threads safely
    --> detcore/src/lib.rs:924:29
     |
924  |       ) -> Result<i64, Error> {
     |  _____________________________^
925  | |         self.pre_handler_hook(guest, false).await;
926  | |
927  | |         let dettid = guest.thread_state().dettid;
...    |
1161 | |         res
1162 | |     }
     | |_____^ future created by async block is not `Send`
     |
     = help: the trait `std::marker::Sync` is not implemented for `core::fmt::rt::Opaque`, which is required by `{async block@detcore/src/lib.rs:924:29: 1162:6}: std::marker::Send`
note: future is not `Send` as this value is used across an await
    --> detcore/src/lib.rs:244:30
     |
241  |                 "(pre) registers [dtid {}][rcbs {}]. {}",
     |                                        -- has type `core::fmt::rt::Argument<'_>` which is not `Send`
...
244  |                 guest.regs().await.display()
     |                              ^^^^^ await occurs here, with `{}` maybe used later
     = note: required for the cast from `Pin<Box<{async block@detcore/src/lib.rs:924:29: 1162:6}>>` to `Pin<Box<dyn futures::Future<Output = Result<i64, reverie::Error>> + std::marker::Send>>`

warning: `detcore` (lib) generated 4 warnings
error: could not compile `detcore` (lib) due to 7 previous errors; 4 warnings emitted
USER: rustc --version
rustc 1.78.0-nightly (3cbb93223 2024-03-13)