Cannot run test with raw-strings-qq - GHC 8.10.7, nixpkgs unstable

[samhh]

Describe the bug

undefined symbol: __libc_siglongjmp, version GLIBC_PRIVATE running cabal test with a particular test dependency, raw-strings-qq.

Steps To Reproduce

Repro repo. Per the README there it only happens with GHC 8.10.7, not 9.0.2.

Expected behavior

cabal test should work. I believe it did for me previously on 21.11. It worked for someone else on IRC on an unspecified stable channel but likewise not on unstable.

Additional context

nixpkgs unstable ff9efb0724de5ae0f9db9df2debefced7eb1571d. Similarly recent Cabal database. Dependencies managed by Cabal, not Nix.

Notify maintainers

Recent activity in hackage-packages.nix: @sternenseemann @gebner @maralorn

Metadata

$ nix-info -m
 - system: `"x86_64-linux"`
 - host os: `Linux 5.17.2, NixOS, 22.05 (Quokka), 22.05.20220413.ff9efb0`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.7.0`
 - channels(root): `"nixos"`
 - channels(sam): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[cdepillabout]

I was able to reproduce this.

I changed the shell.nix to the following:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

mkShell {
  nativeBuildInputs = [
    haskell.compiler.ghc8107
    cabal-install
  ];
}

I got into this shell with pkgs at 1a763b8.

I cloned https://git.sr.ht/~samhh/nix-qq-repro and ran cabal build. It failed with the following output:

$ cabal build
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - clock-0.8.3 (lib) (requires build)
 - raw-strings-qq-1.1 (lib) (requires build)
 - hspec-core-2.9.4 (lib) (requires build)
 - hspec-2.9.4 (lib) (requires build)
 - repro-0.0.0 (test:test) (first run)
Starting     raw-strings-qq-1.1 (lib)
...
Installing   hspec-2.9.4 (lib)
Completed    hspec-2.9.4 (lib)
Configuring test suite 'test' for repro-0.0.0..
Preprocessing test suite 'test' for repro-0.0.0..
Building test suite 'test' for repro-0.0.0..
[1 of 1] Compiling Main             ( test/Spec.hs, /home/illabout/temp/nix-qq-repro/dist-newstyle/build/x86_64-linux/ghc-8.10.7/repro-0.0.0/t/test/build/test/test-tmp/Main.o, /home/illabout/temp/nix-qq-repro/dist-newstyle/build/x86_64-linux/ghc-8.10.7/repro-0.0.0/t/test/build/test/test-tmp/Main.dyn_o )
<command line>: /nix/store/z56jcx3j1gfyk4sv7g8iaan0ssbdkhz1-glibc-2.33-56/lib/libpthread.so.0: undefined symbol: __libc_siglongjmp, version GLIBC_PRIVATE

I've also tested this with the latest 21.11 (which is currently 9887f02), and the above does work fine.

Also, if I create a development environment using callCabal2nix on current master (again at 1a763b8), building seems to work fine:

{ pkgs ? import <nixpkgs> {} }:

with pkgs;

mkShell {
  inputsFrom = [
    (haskell.packages.ghc8107.callCabal2nix "repro" ./. {}).env
  ];
  nativeBuildInputs = [
    cabal-install
  ];
}

Building:

$ cabal build
Warning: The package list for 'hackage.haskell.org' is 37 days old.
Run 'cabal update' to get the latest list of available packages.
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - repro-0.0.0 (test:test) (first run)
Configuring test suite 'test' for repro-0.0.0..
Preprocessing test suite 'test' for repro-0.0.0..
Building test suite 'test' for repro-0.0.0..
[1 of 1] Compiling Main             ( test/Spec.hs, /home/illabout/temp/nix-qq-repro/dist-newstyle/build/x86_64-linux/ghc-8.10.7/repro-0.0.0/t/test/build/test/test-tmp/Main.o, /home/illabout/temp/nix-qq-repro/dist-newstyle/build/x86_64-linux/ghc-8.10.7/repro-0.0.0/t/test/build/test/test-tmp/Main.dyn_o )
Linking /home/illabout/temp/nix-qq-repro/dist-newstyle/build/x86_64-linux/ghc-8.10.7/repro-0.0.0/t/test/build/test/test ...

@sternenseemann Have any ideas on what could be going on here?

@samhh Also, if you wanted to bisect to figure out exactly when this stopped working, that would probably be helpful.

[samhh]

Also, if you wanted to bisect to figure out exactly when this stopped working, that would probably be helpful.

I'm 90% sure it's when I migrated to flakes which is also when I migrated from 21.11 to unstable. Unfortunately I didn't work on the project that presented this issue for a week or two whilst that was happening so I can't be completely sure.

[cdepillabout]

@samhh Ah, sorry, by "exactly when this stopped working", I mean "what commit caused this to stop working". My guess is that there is a single commit between 21.11 and current master you can point to and say, "this commit is when haskell.compiler.ghc8107 stopped working for me".

It can often take a lot of time / effort to git bisect in Nixpkgs, so if you're willing to do this and can find the single commit that caused this regression, it can often really help with the maintainers figuring out what the problem is.

[samhh]

@cdepillabout It's going to be difficult for me to test my system config going back very far, there will be all sorts of breaking changes to contend with (it's unfortunate I've only observed this coming from 21.11 rather than unstable). Unless there's a subset of system packages I can bisect against?

[cdepillabout]

@samhh Unless I'm misunderstanding, in general this shouldn't require you to test your whole system config, but instead just jump into the shell in #169332 (comment) and try running cabal build with commits from Nixpkgs between 21.11 and current master.

[sternenseemann]

I can't reproduce, nixpkgs 1ffba9f (and entire system built against this version). My suspicion would be that different executables interacting here expect a different version of glibc possibly, leading to this internal symbol missing. Maybe nix-shell is the troublemaker as cabal-install and ghc should be using the same glibc version?

Edit: That it's 8.10.7 only kind of contradicts my theory though…

[samhh]

just jump into the shell in #169332 (comment) and try running cabal build with commits from Nixpkgs between 21.11 and current master.

That was my first thought but it still fails for me with the following (a recent revision in nixos-21.11). I'm very new to Nix though, maybe there's an obvious mistake I'm making?:

let
  pkgs = import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/9887f024766aa27704d1f89f623efd1d063da92a.tar.gz";
    sha256 = "151rgnr2ip5i4fk7hllirk0gchwavgca6bk4cqk7agxr4j1pnqgf";
  }) {};

in pkgs.mkShell {
  buildInputs = with pkgs; [
    haskell.compiler.ghc8107
    haskell.packages.ghc8107.cabal-install
  ];
}

[cdepillabout]

My suspicion would be that different executables interacting here expect a different version of glibc possibly, leading to this internal symbol missing. Maybe nix-shell is the troublemaker as cabal-install and ghc should be using the same glibc version?

@sternenseemann That sounds like a very good guess to me. My NixOS system is 21.11, and I was able to reproduce the issue. It sounds like your NixOS system is on master, and you weren't able to reproduce the issue.

[cdepillabout]

Oh, I might have figured out the problem.

I just followed the same steps from #169332 (comment), but in between testing with 21.11 and current master, I did rm -rf ~/.cabal to get rid of all built artifacts in the Cabal store. This got cabal build working for both 21.11 and current master for me.

@samhh Can you try rm -rf ~/.cabal and then cabal update again after jumping into the nix-shell?

My guess is that cabal-install's "nix-store-like" functionality doesn't take into account system libraries like glibc, etc. So you're likely to get some shared objects in the Cabal store that are linked against incompatible things. (This is a similar problem I've observed with stack.)

[samhh]

Can you try rm -rf ~/.cabal and then cabal update again after jumping into the nix-shell?

Success! 🌟

I did try to remove the specific problem dependency in there before but I guess I missed something important.

[sternenseemann]

So let's close this, then? I think such issues are a little beyond our control.

[samhh]

May be worth mentioning in the wiki? Presumably this can happen whenever there's a system glibc update.

[sternenseemann]

#170897