Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add caveats for glibc version targeting feature #232

Merged
merged 1 commit into from
Mar 11, 2024
Merged

docs: Add caveats for glibc version targeting feature #232

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,16 @@ to compile for glibc 2.17 with the `aarch64-unknown-linux-gnu` target:
cargo zigbuild --target aarch64-unknown-linux-gnu.2.17
```

> [!NOTE]
> There are [various caveats](https://github.com/rust-cross/cargo-zigbuild/issues/231#issuecomment-1983434802) with the glibc version targeting feature:
> - If you do not provide a `--target`, Zig is not used and the command effectively runs a regular `cargo build`.
> - If you specify an invalid glibc version, `cargo zigbuild` will not relay the warning emitted from `zig cc` about the fallback version selected.
> - This feature does not necessarily match the behaviour of dynamically linking to a specific version of glibc on the build host.
> - Version 2.32 can be specified, but runs on a host with only 2.31 available when it should instead abort with an error.
Copy link
Member

@messense messense Mar 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is expected because even if you want to use glibc 2.32, you can still end up being compatible with glibc 2.31 because all of the versioned symbols you linked are < 2.32.

glibc doesn't increase symbols versions for every functions in a new minor version, only these with breaking changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is expected because even if you want to use glibc 2.32, you can still end up being compatible with glibc 2.31 because all of the versioned symbols you linked are < 2.32.

My intent is just to highlight the difference in behaviour from cargo build when dynamically linking that same glibc version, cargo zigbuild is being more lenient.

That may be an improvement if it's not impacting functionality. I just wanted to highlight that there was a difference, so there isn't any confusion if it built with the intended version when it doesn't fail under the same conditions that a build via cargo build would.

How would you word it differently?

EDIT: As covered in last section below

  • The difference was Zig doesn't align with glibc 2.32 which raised the symbol version for pthread_getattr_np due to moving the symbol from libpthread to libc.
  • This is also apparently important for Rust to handle proper stack overflow detection, so I'm not sure if Zig should be misaligned with that? 🤷‍♂️

Reproduction (static vs dynamic vs zig dynamic)

Below is a reproduction for cargo build based on rust-lang/libc#2054 (comment) regarding openpty usage with glibc 2.32.

  • A static build with a lower version works correctly on older glibc versions.
  • A static build with newer versions works correctly on older glibc versions, and also on newer glibc (instead of segfaulting).
  • Dynamic linking with glibc 2.32 will prevent it running on earlier glibc, but it'll work fine on newer versions.
  • Dynamic linking with a version lower than glibc 2.32 becomes compatible with earlier versions of glibc too.
  • However cargo zigbuild can dynamically link with:
    • glibc 2.32 target and the failure on earlier glibc is avoided.
    • glibc >2.32 target will introduce failure with a dependency requirement on glibc 2.33.
Reproduction example 
use libc;
use std::{mem,ptr};
fn main() {
    let mut slave = mem::MaybeUninit::<libc::c_int>::uninit();
    let mut master = mem::MaybeUninit::<libc::c_int>::uninit();
    let p;

    unsafe {
       p = libc::openpty(
            master.as_mut_ptr(),
            slave.as_mut_ptr(),
            ptr::null_mut(),
            ptr::null_mut(),
            ptr::null_mut()
        );
    }

    println!("p:{}",p);
}
[package]
name = "example"
version = "0.1.0"
edition = "2021"

[dependencies]
libc = "0.2.153"
# Fedora 33 container with glibc-static package:
$ ldd --version
ldd (GNU libc) 2.32

$ RUSTFLAGS="-C target-feature=+crt-static" cargo build --release --target x86_64-unknown-linux-gnu
# Fedora 31 container:
$ ldd --version
ldd (GNU libc) 2.30
# Fedora 32 container:
$ ldd --version
ldd (GNU libc) 2.31

# Same runtime output below for both containers

# static build:
# https://github.com/rust-lang/libc/issues/2054#issuecomment-829119942
$ ./example
example: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.

# dynamic build (doesn't fail with `cargo zigbuild`):
$ ./example
./example: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./example)
# Fedora 40 container:
$ ldd --version
ldd (GNU libc) 2.39

# static build:
$ ./example
Segmentation fault

# dynamic build:
$ ./example
p:0

When building on Fedora 32 (glibc 2.31) container instead:

  • Static builds will:
    • Segfault on both Fedora 33 and 40.
    • No failure on Fedora 31 container (glibc 2.30)
  • Dynamic builds:
    • No GLIBC_2.31 not found on Fedora 31, works correctly.
    • Fedora 33 and 40 both work correctly too.
    • cargo zigbuild for 2.32

However when a Fedora 31 (glibc 2.30) container runs a build via Fedora 34 (glibc 2.33):

  • Static is successful.
  • Dynamic fails: 
    ./example: /lib64/libc.so.6: version `GLIBC_2.33' not found (required by ./example)
./example: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./example)

A static build with glibc 2.33 is successful across all of these Fedora containers.

Dynamic linking differences

TL;DR:

  • cargo zigbuild will match cargo build when linking to glibc 2.33 with the built program failing to run on hosts with earlier glibc.
  • Meanwhile only cargo build would also have that same failure experience when building with glibc 2.32 version too, but otherwise fine with glibc 2.31 build running on host with glibc 2.30 🤔

I tracked it down, the difference doesn't appear related to openpty. I'm not entirely sure why the symbol versions differ between cargo build and cargo zigbuild, presumably Zig knows better to not match the pthread_getattr_np version for glibc 2.32? 🤷‍♂️ (or it's a bug)

Dynamic linking differences (resolved)

UPDATE: I've learned how to inspect the symbol versions to understand what was enforcing the min supported glibc version between cargo build and cargo zigbuild:

  • cargo build had the min glibc versions of 2.32 because of pthread_getattr_np. Prior versions of glibc version this symbol much lower, which matches cargo zigbuild.
  • glibc 2.33 is enforced by cargo zigbuild as it shares the same symbol version bump for stat64 (technically renamed, not just version bump?)

What introduces a difference to cargo zigbuild between glibc 2.32 vs 2.33 version targeting? Presumably something related to this commit (that was identified as relevant to the compatibility for static builds) (UPDATE: findings below)

From the Fedora mailing list June 2020:

I am surprised that this is not handled by symbol versioning.
This function seems to exist for long long time, so why did it break just now?

The symbol was moved from libpthread to libc.
The new symbol version is required so that newly linked applications depend on glibc 2.32 as the minimum glibc version.

Related bug report:

Rust does need that symbol for stack overflow detection. I found the specific glibc change here

While this may seem Fedora specific due to the container image choice and reference links, Debian 12 (Bookworm) via rust:latest with glibc 2.36 also produces this min requirement.

# Build hosts with dynamic linking:

# Fedora 33:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.32 (5)
  008:   2 (GLIBC_2.2.5)   5 (GLIBC_2.32)    4 (GLIBC_2.2.5)   6 (GLIBC_2.18)
  0x0110:   Name: GLIBC_2.32  Flags: none  Version: 5
# For comparison to glibc 2.33 Fedora 34 change:
readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep stat64
    19: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __xstat64@GLIBC_2.2.5 (2)
    33: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __fxstat64@GLIBC_2.2.5 (2)
    59: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __lxstat64@GLIBC_2.2.5 (2)

# Fedora 34:
# Same:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
     9: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.32 (5)
  008:   2 (GLIBC_2.2.5)   5 (GLIBC_2.32)    4 (GLIBC_2.2.5)   6 (GLIBC_2.18)
  0x0110:   Name: GLIBC_2.32  Flags: none  Version: 5
# glibc 2.33 requirement added due to stat64 + fstat64 symbol versions:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.33
    20: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND stat64@GLIBC_2.33 (9)
    59: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fstat64@GLIBC_2.33 (9)
  014:   9 (GLIBC_2.33)    4 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)   2 (GLIBC_2.2.5)
  038:   2 (GLIBC_2.2.5)   4 (GLIBC_2.2.5)   4 (GLIBC_2.2.5)   9 (GLIBC_2.33)
  0x00e0:   Name: GLIBC_2.33  Flags: none  Version: 9


# Fedora 33 with `cargo zigbuild`:
$ cargo zigbuild --release --target x86_64-unknown-linux-gnu.2.33
# Nothing versioned for glibc 2.32:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.32
# glibc 2.33 target carries the same stat64 change:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep 2.33
    25: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fstat64@GLIBC_2.33 (7)
    37: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND stat64@GLIBC_2.33 (7)
  018:   2 (GLIBC_2.2.5)   7 (GLIBC_2.33)    5 (GLIBC_2.2.5)   5 (GLIBC_2.2.5)
  024:   9 (GLIBC_2.3)     7 (GLIBC_2.33)    5 (GLIBC_2.2.5)   5 (GLIBC_2.2.5)
  0x00c0:   Name: GLIBC_2.33  Flags: none  Version: 7
# The `cargo build` pthread symbol linked to glibc 2.32, while zig doesn't:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep pthread_getattr
    18: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.2.5 (5)

# Fedora 32, also has low pthread version like `cargo zigbuild` produced:
$ readelf -W --version-info --dyn-syms target/x86_64-unknown-linux-gnu/release/example | grep pthread_getattr
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_getattr_np@GLIBC_2.2.5 (4)
# Reference
# https://linux.die.net/man/1/nm
# https://johannst.github.io/notes/development/symbolver.html

# Fedora 33 + 32,
# `@@` denotes default version to link for static build:
# `U` => Undefined symbol
$ nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty@@GLIBC_2.2.5

# Fedora 34:
$ nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty@GLIBC_2.2.5

# Fedora 33 with `cargo zigbuild`, no symbol version:
nm target/x86_64-unknown-linux-gnu/release/example | grep openpty
                 U openpty

For static builds, _dl_open will be always present when linking glibc, regardless if your program would use it or not:

$ readelf -a target/x86_64-unknown-linux-gnu/release/example | grep dl_open
  3512: 00000000000a4fb4     0 NOTYPE  LOCAL  HIDDEN    11 .annobin__dl_open.end
  3513: 00000000000a5120  2623 FUNC    LOCAL  DEFAULT   11 dl_open_worker
  3517: 00000000000a511f     0 NOTYPE  LOCAL  HIDDEN    11 .annobin_dl_open[...]
  3518: 00000000000a5b5f     0 NOTYPE  LOCAL  HIDDEN    11 .annobin_dl_open[...]
  3688: 000000000014b700    32 OBJECT  LOCAL  DEFAULT   26 _dl_open_hook
  5059: 00000000000a4d60   596 FUNC    LOCAL  DEFAULT   11 _dl_open

# Roughly equivalent information:
$ nm -a target/x86_64-unknown-linux-gnu/release/example | grep _dl_open
00000000000aa903 t .annobin___libc_register_dl_open_hook.end
00000000000aa8bf t .annobin___libc_register_dl_open_hook.start
00000000000a4fb4 t .annobin__dl_open.end
00000000000a4d52 t .annobin__dl_open.start
00000000000a5b5f t .annobin_dl_open_worker.end
00000000000a511f t .annobin_dl_open_worker.start
00000000000aa8c0 t __libc_register_dl_open_hook
00000000000a4d60 t _dl_open
000000000014b700 d _dl_open_hook

> - Meanwhile specifying 2.33 will correctly be detected as incompatible when run on a host with glibc 2.31.
> - Certain `RUSTFLAGS` like `-C linker` opt-out of using Zig, while `-L path/to/files` will have Zig ignore `-C target-feature=+crt-static`.
> - `-C target-feature=+crt-static` for statically linking to a glibc version is **not supported** (_upstream `zig cc` lacks support_)

### macOS universal2 target

`cargo zigbuild` supports a special `universal2-apple-darwin` target for building macOS universal2 binaries/libraries on Rust 1.64.0 and later.
Expand Down
Loading