i686-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libatomic.so.1

[messense]

When running cargo in a quay.io/pypa/manylinux2014_i686:latest Docker container, it gives the following error

/root/.rustup/toolchains/nightly-i686-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

Is libatomic.so.1 a hard requirement for cargo now?

Meta

rustc --version --verbose:

rustc 1.78.0-nightly (d18480b84 2024-03-04)

[Noratrieb]

That is an old year in the image name. What's the glibc version? We only support kernel 3.2+, glibc 2.17+ as listed in https://doc.rust-lang.org/nightly/rustc/platform-support.html

[messense]

It's a CentOS 7 which has glibc 2.17, since it's running in Docker on Ubuntu 22.04, it should have a newer kernel version than 3.2.

# uname -a
Linux 2d9d45df5509 5.19.0-1029-aws rust-lang/rust#30~22.04.1-Ubuntu SMP Thu Jul 13 17:17:32 UTC 2023 i686 i686 i386 GNU/Linux

# rustc -vV
rustc 1.78.0-nightly (d18480b84 2024-03-04)
binary: rustc
commit-hash: d18480b84fdbf1efc34f62070951334aa833d761
commit-date: 2024-03-04
host: i686-unknown-linux-gnu
release: 1.78.0-nightly
LLVM version: 18.1.0

# # cargo
/root/.rustup/toolchains/nightly-i686-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

# # ldd /root/.rustup/toolchains/nightly-i686-unknown-linux-gnu/bin/cargo
        linux-gate.so.1 =>  (0xf7f0e000)
        libdl.so.2 => /lib/libdl.so.2 (0xf621c000)
        libatomic.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf6201000)
        librt.so.1 => /lib/librt.so.1 (0xf61f7000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf61dc000)
        libm.so.6 => /lib/libm.so.6 (0xf619a000)
        libc.so.6 => /lib/libc.so.6 (0xf5fcf000)
        /lib/ld-linux.so.2 (0xf7f10000)

# ldd --version
ldd (GNU libc) 2.17

[messense]

cc @weihanglo in case you know something about this.

[messense]

For comparison, Rust stable (1.76.0 ATM) works fine:

# ~/.rustup/toolchains/stable-i686-unknown-linux-gnu/bin/cargo -vV
cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: i686-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.5.0-DEV (sys:0.4.70+curl-8.5.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: CentOS 7.0.0 [32-bit]

# ldd ~/.rustup/toolchains/stable-i686-unknown-linux-gnu/bin/cargo
        linux-gate.so.1 =>  (0xf7f2a000)
        libdl.so.2 => /lib/libdl.so.2 (0xf6156000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf613b000)
        librt.so.1 => /lib/librt.so.1 (0xf6132000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf6116000)
        libm.so.6 => /lib/libm.so.6 (0xf60d4000)
        libc.so.6 => /lib/libc.so.6 (0xf5f09000)
        /lib/ld-linux.so.2 (0xf7f2c000)

[weihanglo]

This is the bad Cargo submodule update rust-lang/rust#121214. It is very likely that OpenSSL v3 did something differently than v1.

[weihanglo]

Probably this openssl/openssl#23376?

[weihanglo]

sfackler/rust-openssl#2163 and I mentioned it earlier in #13449 (comment)
(I forgot everything older than three days ago)

[messense]

This seems to happen again in rust version 1.79.0-nightly (8b2459c1f 2024-04-09)

 info: downloading installer
  Warning: Not enforcing strong cipher suites for TLS, this is potentially less secure
  info: profile set to 'minimal'
  info: default host triple is i686-unknown-linux-gnu
  info: syncing channel updates for 'nightly-i686-unknown-linux-gnu'
  info: latest update on 2024-04-10, rust version 1.79.0-nightly (8b2459c1f 2024-04-09)
  info: downloading component 'cargo'
  info: downloading component 'rust-std'
  info: downloading component 'rustc'
  info: installing component 'cargo'
  info: installing component 'rust-std'
  info: installing component 'rustc'
  
  info: default toolchain set to 'nightly-i686-unknown-linux-gnu'
    nightly-i686-unknown-linux-gnu installed - rustc 1.79.0-nightly (8b2459c1f 2024-04-09)
  
  
  Rust is installed now. Great!
  
  To get started you may need to restart your current shell.
  This would reload your PATH environment variable to include
  Cargo's bin directory ($HOME/.cargo/bin).
  
  To configure your current shell, you need to source
  the corresponding env file under $HOME/.cargo.
  
  This is usually done by running one of the following (note the leading DOT):
  . "$HOME/.cargo/env"            # For sh/bash/zsh/ash/dash/pdksh
  source "$HOME/.cargo/env.fish"  # For fish
  Install Rust toolchain nightly
  info: override toolchain for '/home/runner/work/rjsonnet-py/rjsonnet-py' set to 'nightly-i686-unknown-linux-gnu'
  info: downloading component 'llvm-tools'
  info: installing component 'llvm-tools'
/root/.rustup/toolchains/nightly-i686-unknown-linux-gnu/bin/cargo: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

[messense]

Seems like this is happening again in nightly-i686-unknown-linux-gnu installed - rustc 1.82.0-nightly (3e9bd8b56 2024-08-08)

See PyO3/maturin-action#283 and https://github.com/Warthunder-Open-Source-Foundation/wt_blk/actions/runs/10321002590/job/28572852583#step:4:196

[weihanglo]

That was me overlooking it again: #14299

[weihanglo]

@messense should have been fixed via #14391 and rust-lang/rust#129066.
(Sorry I should have paid more attention to manual cargo update 🙇🏾‍♂️ )

[weihanglo]

openssl/openssl#23376 has been fixed via openssl/openssl@2fd6c12 and released with OpenSSL v3.4.0. If that is the root cause, the atomic problem should be resolved. The next step will be

• Update https://crates.io/crates/openssl-src (or openssl-sys?) to 3.4.0.
• Remove pinned OpenSSL version from Cargo.toml.

[weihanglo]

Reopen this to track the progress of upgrading to OpenSSL v3.

[weihanglo]

Let me jot down what I've found so far, for building OpenSSL v3.4¹ into Cargo with rust-lang CI.

• Similar to sfackler/rust-openssl#2163 and openssl/openssl#23376, the dist-i686-linux build job from rust-lang/rust CI produces the cargo executable that dynamically links to libatomic.so.1. The ldd output looks like

  ldd obj/dist-i686-linux/build/i686-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-gate.so.1 (0xf7fb6000)
        libdl.so.2 => /lib/libdl.so.2 (0xf583d000)
        libatomic.so.1 => not found
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf5822000)
        librt.so.1 => /lib/librt.so.1 (0xf5819000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf57fb000)
        libm.so.6 => /lib/libm.so.6 (0xf573e000)
        libc.so.6 => /lib/libc.so.6 (0xf55b0000)
        /lib/ld-linux.so.2 (0xf7fb8000)
  

  On some distributions there is no libatomic available, so it failed to
• Setting -DBROKEN_CLANG_ATOMICS in openssl-src build didn't help.
• Removing -latomic (cargo:rustc-link-lib=atomic) from rust-openssl didn't help.
• In contrast, cargo from the dist-x86_64-linux job doesn't dynamically link to libatomic at all. The ldd output is like

  ldd obj/dist-x86_64-linux/build/x86_64-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-vdso.so.1 (0x00007ffe9d249000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f250d26d000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f250d057000)
        librt.so.1 => /lib64/librt.so.1 (0x00007f250ce4f000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f250cc31000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f250c8f1000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f250c544000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f250f507000)
  

• Switching CC to gcc eliminates the dynamic links. The ldd output looks like

  ldd obj/dist-i686-linux/build/i686-unknown-linux-gnu/stage2-tools-bin/cargo
        linux-gate.so.1 (0xf7f79000)
        libdl.so.2 => /lib/libdl.so.2 (0xf574f000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xf5734000)
        librt.so.1 => /lib/librt.so.1 (0xf572b000)
        libpthread.so.0 => /lib/libpthread.so.0 (0xf570d000)
        libm.so.6 => /lib/libm.so.6 (0xf5650000)
        libc.so.6 => /lib/libc.so.6 (0xf54c2000)
        /lib/ld-linux.so.2 (0xf7f7b000)
  

The potential reason is that we set CC in the dist-i686-linux job image, so it builds a cargo linking to libatomic.so instead of statically linkage. And clang doesn't provide built-in atomic lib, according to its doc. It has also be discussed in llvm/llvm-project#73361 and mstorsjo/llvm-mingw#384. I've tried adding -DCOMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF² and -DCOMPILER_RT_USE_ATOMIC_LIBRARY=ON when bootstrapping the first clang in the dist-i686-linudex job. Nothing changed.

It has been more than one year since OpenSSL v1.1.1 EOL (2023-09-11), we should take action on this, either try harder upgrading to OpenSSL v3, or find an alternative like rustls or aws-lc. See also how rustup struggles to upgrade OpenSSL rust-lang/rustup#3790.

Footnotes

1. [email protected], [email protected], and [email protected]+3.4.0 respectively ↩

2. requires gcc10 to fix preprocessor errors and -DCOMPILER_RT_HAS_WBUILTIN_DECLARATION_MISMATCH_FLAG=OFF to surpress some errors. ↩

[washanhanzi]

is it possible to temporarily pin openssl to 1.0.66? openssl has a vulnerability prior to 1.0.66, i don't know if this affect the cargo code base.

• PR in openssl
• rustsec

[weihanglo]

is it possible to temporarily pin openssl to 1.0.66?

I don't think so. [email protected] has a hard requirement of OpenSSL v3, and just one comment above you can see how convoluted it is for the bump (or how dumb I am 😞).

In Cargo we only call openssl::version::version(). None of our dependencies require openssl (only openssl-sys through libgit2-sys/libssh2-sys), so I doubt Cargo is affected.