Document how to work around dynamic linking problems caused by glibc updates

[sternenseemann]

With the recent update to glibc 2.34, a lot of people have been running into problems caused by nixpkgs and their non-NixOS distribution switching glibc version at a different pace (and the problem is probably also possible to run into on NixOS, albeit rarer). I've seen two different class of issues recently:

• Missing internal glibc versions (of version GLIBC_PRIVATE): Cannot run test with raw-strings-qq - GHC 8.10.7, nixpkgs unstable #169332
• Missing versioned (up to date) public symbols, in this case GLIBC_2.34 when an older version gets loaded by accident: Prisma 3.12.0 is broken #170550 (comment), Teamviewer panic on Steam Deck and Ubuntu #169040, https://discourse.nixos.org/t/opening-links-in-brave-from-discord-is-broken-version-glibc-2-34-not-found/18764, https://discourse.nixos.org/t/glibc-2-33-vs-2-34-rust-madness/18909/3, …

Related discussion (on solving the technical problem):

• https://discourse.nixos.org/t/could-we-robustly-protect-against-errors-version-glibc-2-33/18343
• Solve the libGL ABI problem #31189

I think a great help short of solving the problem altogether would be documenting it and how to work around it in a central place, i.e. the nixpkgs manual. Then it becomes easier to link to an answer to the recurring question. I lack the time and precise understanding of the problem at the moment, so it'd be great if someone could pick this up!

I think most important is to document the causes and how to diagnose / guess the appropriate one: Usually this issue is caused by the use of something impure (e.g. OpenGL drivers, compiled artifacts in some build directory, ...) that assume the wrong kind of ABI compatibility.

[tpwrules]

I didn't realize there was actually a way to work around this? When I encounter it I just try to use less from my non-NixOS distro so everything I need is glibc 2.34.

Are you referring to messing with LD_PRELOAD as described in the related discussion you provided? I'd love to work on addressing this but I would appreciate a few more clues on how if you have them.

[gilescope]

"Yes, glibc has great ABI compatibility, but only one way. ". If that's the case then why is it so hard to persuade unstable nixos that maybe it should be using the latest version?

export LD_PRELOAD=/nix/store/jd21wzn6z4sw68gpk01xj33l2j4rzbjy-glibc-2.34-115/lib/libc.so.6

I am randomly preloading things that look sorta like libc in the hope that one of them migh have the right kind of elf header. We shouldn't fall in a heap when libc gets upgraded. home-manager won't even downgrade. If anyone knows what magic I should be pasting where it would be much appreciated.
(hint the one pictured above is not the magic one)

[kjeremy]

I'm pretty sure that this is the cause of #169464 too.

[msackman]

Very likely cause of #167735 too.

[tobiasBora]

@gilescope As I understand, people want to avoid export LD_PRELOAD to strengthen purity, for instance if the software is newer than the host and needs a recent glibc to work, or a fixed glibc to avoid bugs coming from different versions. As you say, this comes at the high price of having issues when the distribution is newer than the installed packages, as the drivers will require a GLIBC that is too recent compared to the one hardcoded in the old sofware, leading to the GLIBC_PRIVATE error. In my opinion, I'd say that more errors comes from this last case, and I'd be in favor of changing the way nix works.

I don't know much of the internals of glibc, but I can imagine a generic solution that combines the best of both world (runnable on newer systems, and purity when running on older systems): if we wrap all packages, we could check in the wrapper the GLIBC version used by the drivers of the system. If the version is newer than the GLIBC used by the current system, then we export LD_PRELOAD, and if it is older or equal, then we do nothing more. Note that this comes at the price of wrapping all executables, but I don't see any other solution and I'm not skilled enough to understand the solution provided in #31189 and I have no idea if this works with libwayland-client. Note that to handle wrappers correctly, we may also want to fix #150841

[kjeremy]

@sternenseemann Is the fix for this just to randomly try LD_PRELOAD?

[sternenseemann]

No, ideally you just update the impure stuff on your system to line up with whatever nixpkgs provides at the moment. Preloading the right version for a specific binary (not globbaly) can be a good workaround if that's not attainable.

[domenkozar]

Right now on 22.05 trying to downgrade to 21.05:

$ /nix/store/c1gvl27x3jnzc1z842r24xj37b7rlraw-nixos-system-virtualbox-21.05.4678.932ec35ff8a/bin/switch-to-configuration switch
Can't load '/nix/store/s6rwwwgyh7lyay0l1j33p7ai9l8w6q89-perl-5.32.1-env/lib/perl5/site_perl/5.32.1/x86_64-linux-thread-multi/auto/Net/DBus/DBus.so' for module Net::DBus: /nix/store/jsp3h3wpzc842j0rz61m5ly71ak6qgdn-glibc-2.32-54/lib/libc.so.6: version `GLIBC_2.34' not found (required by /nix/store/jl1b87dsbdy1xai0sjfqpmp4fhgy8dv8-xz-5.2.5/lib/liblzma.so.5) at /nix/store/wmpankr1aiwv9qd2533fav779yrcicry-perl-5.32.1/lib/perl5/5.32.1/XSLoader.pm line 93

[domenkozar]

An attempt to solve this: #177788

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/buildfhsuserenv-creates-a-small-root-directory/31220/1

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/help-understanding-the-libgl-abi-problem-and-possible-solutions/42022/1