clang18 linker error: 'libc++.so.1: no such file or directory'

[jameslamb]

Thanks for these great images! They've been really helpful in replicating some of the harder-to-replicate CRAN checks.

I believe there's a configuration issue in the clang18 image, although I'm not sure where precisely it is.

Given a small C++ program like this...

cat > conftest.cpp <<EOF
#include <stdio.h>
int main(){
    printf("this ran successfully\n");
    return 0;
}
EOF

... compiled like this ...

CXX17=`R CMD config CXX17`
CXX17STD=`R CMD config CXX17STD`
CXX="${CXX17} ${CXX17STD}"
CPPFLAGS=`R CMD config CPPFLAGS`
CXXFLAGS=`R CMD config CXX17FLAGS`
${CXX} ${CPPFLAGS} ${CXXFLAGS} -o conftest conftest.cpp

... the produced executable does not have linking information for libc++.so.1, and so fails like this when run:

./conftest: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

I found that there's only 1 copy of libc++.so.1 in the image, at /usr/lib/llvm-18/lib.

find / -type f -name 'libc++.so*'
# /usr/lib/llvm-18/lib/libc++.so.1.0
# /usr/lib/llvm-18/lib/libc++.so

Adding that to LD_LIBRARY_PATH allows loading to succeed.

LD_LIBRARY_PATH=/usr/lib/llvm-18/lib \
    ./conftest 
# this ran successfully

I did not expect to have to configure anything to compile a C++ program with clang-18 in the clang18 image.

Reproducible Example

Created a test script, test.sh.

test.sh (click me)

#!/bin/bash

CXX17=`R CMD config CXX17`
CXX17STD=`R CMD config CXX17STD`
CXX="${CXX17} ${CXX17STD}"
CPPFLAGS=`R CMD config CPPFLAGS`
CXXFLAGS=`R CMD config CXX17FLAGS`

cat > conftest.cpp <<EOF
#include <stdio.h>
int main(){
    printf("this ran successfully\n");
    return 0;
}
EOF

set -x
${CXX} ${CPPFLAGS} ${CXXFLAGS} -o conftest conftest.cpp
set +x

echo "--- ldd output ---"
ldd -v ./conftest
echo "---"

echo "--- ./conftest ---"
./conftest
echo "---"

echo "--- LD_LIBRARY_PATH=/usr/lib/llvm-18/lib ./conftest ---"
LD_LIBRARY_PATH=/usr/lib/llvm-18/lib \
    ./conftest
echo "---"

Ran that with each of the clang{nn} images, like this:

docker run \
    --rm \
    -v $(pwd):/opt/work \
    -w /opt/work \
    --platform linux/amd64 \
    -it ghcr.io/r-hub/containers/clang18:latest \
    ./test.sh

It succeeded on clang16, clang17, and clang19 ... but not clang18.

full logs from clang17 run (click me)

+ clang++-17 -stdlib=libc++ -std=gnu++17 -I/usr/local/include -O3 -Wall -pedantic -frtti -Wp,-D_FORTIFY_SOURCE=3 -o conftest conftest.cpp
+ set +x
--- ldd output ---
        libc++.so.1 => /lib/x86_64-linux-gnu/libc++.so.1 (0x00007fffff6bb000)
        libc++abi.so.1 => /lib/x86_64-linux-gnu/libc++abi.so.1 (0x00007fffff684000)
        libunwind.so.1 => /lib/x86_64-linux-gnu/libunwind.so.1 (0x00007fffff675000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff58e000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fffff56e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fffff345000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffffffc4000)

        Version information:
        ./conftest:
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc++.so.1:
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.17) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.8) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.33) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.6) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.7) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.3) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc++abi.so.1:
                libc.so.6 (GLIBC_2.3.2) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.18) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.16) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libunwind.so.1:
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.3.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
---
--- ./conftest ---
this ran successfully
---
--- LD_LIBRARY_PATH=/usr/lib/llvm-18/lib ./conftest ---
this ran successfully
---

full logs from clang18 run (click me)

+ clang++-18 -stdlib=libc++ -std=gnu++17 -I/usr/local/include -O3 -Wall -pedantic -frtti -Wp,-D_FORTIFY_SOURCE=3 -o conftest conftest.cpp
+ set +x
--- ldd output ---
        libc++.so.1 => not found
        libc++abi.so.1 => not found
        libunwind.so.1 => not found
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffff6d1000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fffff6b1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fffff488000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffffffc4000)

        Version information:
        ./conftest:
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libm.so.6:
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
                libc.so.6 (GLIBC_2.4) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_PRIVATE) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libgcc_s.so.1:
                libc.so.6 (GLIBC_2.35) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.14) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.34) => /lib/x86_64-linux-gnu/libc.so.6
                libc.so.6 (GLIBC_2.2.5) => /lib/x86_64-linux-gnu/libc.so.6
        /lib/x86_64-linux-gnu/libc.so.6:
                ld-linux-x86-64.so.2 (GLIBC_2.2.5) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_2.3) => /lib64/ld-linux-x86-64.so.2
                ld-linux-x86-64.so.2 (GLIBC_PRIVATE) => /lib64/ld-linux-x86-64.so.2
---
--- ./conftest ---
./conftest: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
---
--- LD_LIBRARY_PATH=/usr/lib/llvm-18/lib ./conftest ---
this ran successfully
---

Notes

I did look in the output of R CMD config --all and tried to compare it to the details of the clang{nn} checks described at https://cran.r-project.org/web/checks/check_issue_kinds.html. I don't see any obvious issues.

I just started seeing this around 3-5 days ago... before that, similar code ran without issue in the clang18 image produced here.

For context, this pattern of gathering C++17 flags has been used in the {lightgbm}'s configure script on CRAN for 2+ years, and attempts to follow the advice in https://cran.r-project.org/doc/manuals/R-exts.html#Using-C_002b_002b-code.

[gaborcsardi]

Thanks for the detailed report! Nothing has changed in this image for a while, so this may be a change in clang 18 or in R-devel.

I suggest we leave it alone for (say) a week, maybe it is transient and they'll fix it. If it is still happening in a week please ping me here and I'll take a closer look.

[gaborcsardi]

Btw. all older containers are available at https://github.com/r-hub/containers/pkgs/container/containers%2Fclang18/versions

Can you please double check that the one (say) a week ago works with the current version of your package and/or your reprex? Thanks.

AFAICT there were no recent changes in R that could cause this. It also seems that there were no clang 18 changes, either.

[jameslamb]

Thanks for that! I didn't know it was possible to see the old versions there, that's great.

I tried each of those with my reproducible example, found that this one from 3 days ago is the first one that exhibits this problematic behavior:

# (3 days ago)
ghcr.io/r-hub/containers/clang18@sha256:5cacc2461b5ab765a4d1315de1250a36e9b7f5397591879b2ef873cd8773bb3b

I see the same issue in the image from 2 days ago.

sha256:75a7fed7bb116e1ba54cde6f6077d76731868a563ff3385d28264b2226336bbd

BUT... it appears to already be fixed in the latest image 🎉

# (latest image, published 21 hours ago)
sha256:d3ab5ddce85e4d30e5e72a1974ab305ec42c1a51fdd10c724fb7dee61d67bae6

I tried writing out the environment and output of R CMD config --all for that latest (working) image and the one from 2 days ago where I saw this example fail, and checking the differences with a text differ.

how I did that (click me)

docker run \
    --rm \
    --platform linux/amd64 \
    -it ghcr.io/r-hub/containers/clang18@sha256:75a7fed7bb116e1ba54cde6f6077d76731868a563ff3385d28264b2226336bbd \
    R CMD config --all \
> ./broken.txt

docker run \
    --rm \
    --platform linux/amd64 \
    -it ghcr.io/r-hub/containers/clang18@sha256:75a7fed7bb116e1ba54cde6f6077d76731868a563ff3385d28264b2226336bbd \
    env \
>> ./broken.txt

docker run \
    --rm \
    --platform linux/amd64 \
    -it ghcr.io/r-hub/containers/clang18@sha256:d3ab5ddce85e4d30e5e72a1974ab305ec42c1a51fdd10c724fb7dee61d67bae6 \
    R CMD config --all \
> ./latest.txt

docker run \
    --rm \
    --platform linux/amd64 \
    -it ghcr.io/r-hub/containers/clang18@sha256:d3ab5ddce85e4d30e5e72a1974ab305ec42c1a51fdd10c724fb7dee61d67bae6 \
    env \
>> ./latest.txt

There were 0 differences, so I guess it must have been something else. Since this appears to be working, I'm not planning to investigate it further... hopefully it was just something temporary we won't see again.

Thanks very much for the help, and again... thanks for maintaining these images! They've allowed us to significantly improve the test coverage of the {lightgbm} package, while reducing the complexity of our CI setup.