Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"error initializing LibGit2 module" when CA certificates not found #18693

Closed
colbec opened this issue Sep 27, 2016 · 76 comments · Fixed by #19390
Closed

"error initializing LibGit2 module" when CA certificates not found #18693

colbec opened this issue Sep 27, 2016 · 76 comments · Fixed by #19390
Labels
libgit2 The libgit2 library or the LibGit2 stdlib module regression Regression in behavior compared to a previous version system:linux Affects only Linux

Comments

@colbec
Copy link

colbec commented Sep 27, 2016

Using openSUSE Leap 42.1:
24 hours ago a git pull followed by a make && make testall ran fine. This morning I had an error in make:

...
patching file tests/online/badssl.c
CMake Error: The current CMakeCache.txt directory /home/colin/Downloads/julia6/deps/scratch/libgit2-211e117a0590583a720c53172406f34186c543bd/CMakeCache.txt is different than the directory /home/colin/Downloads/julia/deps/scratch/libgit2-211e117a0590583a720c53172406f34186c543bd where CMakeCache.txt was created. This may result in binaries being created in the wrong place. If you are not sure, reedit the CMakeCache.txt
CMake Error: The source "/home/colin/Downloads/julia6/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/CMakeLists.txt" does not match the source "/home/colin/Downloads/julia/deps/srccache/libgit2-211e117a0590583a720c53172406f34186c543bd/CMakeLists.txt" used to generate cache.  Re-run cmake with a different source directory.
/home/colin/Downloads/julia6/deps/libgit2.mk:66: recipe for target 'scratch/libgit2-211e117a0590583a720c53172406f34186c543bd/build-configured' failed
make[1]: *** [scratch/libgit2-211e117a0590583a720c53172406f34186c543bd/build-configured] Error 1
Makefile:81: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

I thought I had fixed this with make -C deps distclean-libgit2 distclean-mbedtls distclean-libssh2 which allowed make to run to the end. However now Julia 0.6 fails to launch with:

> julia6
fatal: error thrown and no exception handler available.
Base.InitError(mod=:LibGit2, error=ErrorException("error initializing LibGit2 module"))
rec_backtrace at /home/colin/Downloads/julia6/src/stackwalk.c:84
record_backtrace at /home/colin/Downloads/julia6/src/task.c:238
jl_throw at /home/colin/Downloads/julia6/src/task.c:560
__init__ at ./libgit2/libgit2.jl:539
unknown function (ip: 0x7f60d03e3e08)
jl_call_method_internal at /home/colin/Downloads/julia6/src/julia_internal.h:218 [inlined]
jl_apply_generic at /home/colin/Downloads/julia6/src/gf.c:1852
jl_apply at /home/colin/Downloads/julia6/src/julia.h:1376 [inlined]
jl_module_run_initializer at /home/colin/Downloads/julia6/src/toplevel.c:83
_julia_init at /home/colin/Downloads/julia6/src/init.c:702
julia_init at /home/colin/Downloads/julia6/src/task.c:289
unknown function (ip: 0x4014cd)
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x40151f)

Prebuilt Julia 0.5 works fine.

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

What does ldd usr/lib/libgit2.so say?

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

If you see anything in there about libssl or libcrypto (but not of the mbed variety), try make cleanall && make

@colbec
Copy link
Author

colbec commented Sep 27, 2016

One curiosity is that if I do a find ./ -name libgit2.so from julia main directory I find eight different files matched, some clearly symlinks, but others may be different residual versions causing confusion.

> ldd usr/lib/libgit2.so
    linux-vdso.so.1 (0x00007ffeced63000)
    libcurl.so.4 => /home/colin/Downloads/julia6/usr/lib/libcurl.so.4 (0x00007f5926bf8000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f59269b2000)
    libmbedtls.so.10 => /home/colin/Downloads/julia6/usr/lib/libmbedtls.so.10 (0x00007f592678b000)
    libmbedx509.so.0 => /home/colin/Downloads/julia6/usr/lib/libmbedx509.so.0 (0x00007f5926577000)
    libmbedcrypto.so.0 => /home/colin/Downloads/julia6/usr/lib/libmbedcrypto.so.0 (0x00007f5926323000)
    libssh2.so.1 => /home/colin/Downloads/julia6/usr/lib/libssh2.so.1 (0x00007f59260f0000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f5925ee8000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5925cca000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f5925922000)
    /lib64/ld-linux-x86-64.so.2 (0x00005572745ee000)

Then:
make cleanall is successful, but make finishes with:

...
configure: creating ./config.status
config.status: creating libuv.pc
config.status: creating Makefile
config.status: executing depfiles commands
config.status: executing libtool commands
make[2]: *** No rule to make target '/home/colin/Downloads/julia/deps/srccache/libuv-8d5131b6c1595920dd30644cd1435b4f344b46c8/src/fs-poll.c', needed by 'src/libuv_la-fs-poll.lo'.  Stop.
/home/colin/Downloads/julia6/deps/libuv.mk:37: recipe for target 'scratch/libuv-8d5131b6c1595920dd30644cd1435b4f344b46c8/build-compiled' failed
make[1]: *** [scratch/libuv-8d5131b6c1595920dd30644cd1435b4f344b46c8/build-compiled] Error 2
Makefile:81: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

I'm not sure what to make of the libuv failure. Does it persist after make -C deps distclean-libuv ?

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

Can you also check ldd on some of the other libraries, libcurl, libssh2, and the libmbed* ? Is anything obviously missing?

@colbec
Copy link
Author

colbec commented Sep 27, 2016

After make -C deps distclean-libuv make still failing.

> make
 cd /home/colin/Downloads/julia/deps/srccache/libunwind-1.1-julia2 && /bin/sh /home/colin/Downloads/julia/deps/srccache/libunwind-1.1-julia2/config/missing automake-1.15 --gnu Makefile
/bin/sh: line 10: cd: /home/colin/Downloads/julia/deps/srccache/libunwind-1.1-julia2: No such file or directory
Makefile:477: recipe for target '/home/colin/Downloads/julia/deps/srccache/libunwind-1.1-julia2/Makefile.in' failed
make[2]: *** [/home/colin/Downloads/julia/deps/srccache/libunwind-1.1-julia2/Makefile.in] Error 1
/home/colin/Downloads/julia6/deps/unwind.mk:31: recipe for target '/home/colin/Downloads/julia6/usr-staging/libunwind-1.1-julia2.tgz' failed
make[1]: *** [/home/colin/Downloads/julia6/usr-staging/libunwind-1.1-julia2.tgz] Error 2
Makefile:81: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

Something is very wrong, but I'm not sure if it's in Julia's makefiles or local to your particular clone. Does a build from scratch work correctly (maybe in a new clone to at least keep this build's copies of llvm and openblas around if they're still working)?

@colbec
Copy link
Author

colbec commented Sep 27, 2016

The directory usr/lib does not contain a libcurl,so or libssh2.so or libmbed*.so. Should they be there? Doing a find for libcurl.so finds them in .deps/build and scratch.

I will wipe the cache and try again.

@colbec
Copy link
Author

colbec commented Sep 27, 2016

@tkelman Just finished a re-clone (everything deleted and recreated from scratch) of latest v 0.6 dev, and make is successful but again, will not launch as above.

@vchuravy
Copy link
Member

I just hit this on power:

    JULIA test/all
fatal: error thrown and no exception handler available.
Base.InitError(mod=:LibGit2, error=ErrorException("error initializing LibGit2 module"))
rec_backtrace at /home/valentin/juliatest/src/stackwalk.c:84
record_backtrace at /home/valentin/juliatest/src/task.c:238
jl_throw at /home/valentin/juliatest/src/task.c:560
__init__ at ./libgit2/libgit2.jl:539
unknown function (ip: 0x3fffa68cf1bf)
jl_call_method_internal at /home/valentin/juliatest/src/julia_internal.h:239 [inlined]
jl_apply_generic at /home/valentin/juliatest/src/gf.c:1852
jl_apply at /home/valentin/juliatest/src/julia.h:1376 [inlined]
jl_module_run_initializer at /home/valentin/juliatest/src/toplevel.c:83
_julia_init at /home/valentin/juliatest/src/init.c:702
julia_init at /home/valentin/juliatest/src/task.c:289
main at /home/valentin/juliatest/ui/repl.c:247
unknown function (ip: 0x3fffb226457f)
__libc_start_main at /lib64/power8/libc.so.6 (unknown line)
make: *** [all] Error 1
make: Leaving directory `/home/valentin/juliatest/test'

@maleadt
Copy link
Member

maleadt commented Sep 27, 2016

Same here, Arch Linux x86-64, both libgit2 and libssh2 are properly linked to mbedtls. Seems like git_mbedtls_stream_global_init is returning -1.

@maleadt
Copy link
Member

maleadt commented Sep 27, 2016

It isn't finding the CA certificates, after trying:

  • ENV[X509_CERT_FILE_EVP="SSL_CERT_FILE"]
  • ENV[X509_CERT_DIR_EVP="SSL_CERT_DIR"]
  • "/usr/lib/ssl/cert.pem"
  • "/usr/lib/ssl/certs"

In other words, this is failing.

EDIT: on my Debian systems, "/usr/lib/ssl/certs" symlinks to "/etc/ssl/certs", but on Arch this is not the case. Maybe we should add "/etc/ssl/certs" to the search path? As a workaround, exporting SSL_CERT_DIR makes Julia work again.

@vchuravy
Copy link
Member

On the my redhat machine at hand I have to use SSL_CERT_FILE=/etc/pki/tls/cert.pem or SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt

https://www.happyassassin.net/2015/01/12/a-note-about-ssltls-trusted-certificate-stores-and-platforms/

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

Looks like we need a few non-debian-shaped defaults too for the cert locations. There's something in make install or make binary-dist that copies certs around, things might work after that step runs. If so we could move it earlier.

@colbec
Copy link
Author

colbec commented Sep 27, 2016

On openSUSE Leap 42.1 SSL_CERT_FILE=/etc/ssl/certs/certSIGN_ROOT_CA.pem allows v0.6-dev to launch correctly. Not sure if this is the right .pem to use, just chose the closest to alternatives found above. Thanks.

Edit: I guess it is not the correct .pem. Pkg.update() immediately complains the certificate is wrong.

Edit 2: SSL_CERT_FILE=/var/lib/ca-certificates/ca-bundle.pem gets things working again. The alternate setting of SSL_CERT_DIR suggested above does not work on openSUSE.

@tkelman
Copy link
Contributor

tkelman commented Sep 27, 2016

We should also really try to make the error message clearer here if we can.

@colbec colbec changed the title Make fails in 0.6-dev: "error initializing LibGit2 module" Make succeeds with v 0.6-dev, but cannot launch when CA certificates not found Sep 27, 2016
@tkelman tkelman changed the title Make succeeds with v 0.6-dev, but cannot launch when CA certificates not found "error initializing LibGit2 module" when CA certificates not found Sep 27, 2016
@TotalVerb
Copy link
Contributor

I think I hit this error on Ubuntu, so it applies to Debian-based distros too:

fatal: error thrown and no exception handler available.
Base.InitError(mod=:LibGit2, error=ErrorException("error initializing LibGit2 module"))
rec_backtrace at /home/fengyang/julia/src/stackwalk.c:84
record_backtrace at /home/fengyang/julia/src/task.c:238
jl_throw at /home/fengyang/julia/src/task.c:560
__init__ at ./libgit2/libgit2.jl:539
unknown function (ip: 0x7fdc8d179f38)
jl_call_method_internal at /home/fengyang/julia/src/julia_internal.h:239 [inlined]
jl_apply_generic at /home/fengyang/julia/src/gf.c:1852
jl_apply at /home/fengyang/julia/src/julia.h:1377 [inlined]
jl_module_run_initializer at /home/fengyang/julia/src/toplevel.c:83
_julia_init at /home/fengyang/julia/src/init.c:702
julia_init at /home/fengyang/julia/src/task.c:289
main at /home/fengyang/julia/ui/repl.c:247
__libc_start_main at /build/glibc-GKVZIf/glibc-2.23/csu/../csu/libc-start.c:291
unknown function (ip: 0x401438)

Setting the SSL_CERT_FILE to /usr/lib/ssl/certs/ca-certificates.crt and SSL_CERT_DIR to /usr/lib/ssl/certs repaired the issue.

@tkelman
Copy link
Contributor

tkelman commented Sep 29, 2016

It may also depend on which particular ca-certificates package sets you have installed, since those come in a few different variants.

@rickhg12hs
Copy link
Contributor

rickhg12hs commented Sep 29, 2016

Same problem on Fedora 24.

$ ./julia
fatal: error thrown and no exception handler available.
Base.InitError(mod=:LibGit2, error=ErrorException("error initializing LibGit2 module"))
rec_backtrace at /home/rick/src/julia/julia/src/stackwalk.c:84
record_backtrace at /home/rick/src/julia/julia/src/task.c:238
jl_throw at /home/rick/src/julia/julia/src/task.c:560
__init__ at ./libgit2/libgit2.jl:539
unknown function (ip: 0x7fb07647cf08)
jl_call_method_internal at /home/rick/src/julia/julia/src/julia_internal.h:239 [inlined]
jl_apply_generic at /home/rick/src/julia/julia/src/gf.c:1852
jl_apply at /home/rick/src/julia/julia/src/julia.h:1377 [inlined]
jl_module_run_initializer at /home/rick/src/julia/julia/src/toplevel.c:83
_julia_init at /home/rick/src/julia/julia/src/init.c:702
julia_init at /home/rick/src/julia/julia/src/task.c:289
main at /home/rick/src/julia/julia/ui/repl.c:247
__libc_start_main at /lib64/libc.so.6 (unknown line)
unknown function (ip: 0x4013c8)

@tkelman
Copy link
Contributor

tkelman commented Sep 29, 2016

We'll have to find where the default cert search path is set and add more. Someone who has a source build in one of the problematic configurations will be best placed to test proposed additions.

@vchuravy
Copy link
Member

The patch we are carrying against mbedtls is setting these defaults.

+# define OPENSSLDIR "/usr/lib/ssl"

@tkelman
Copy link
Contributor

tkelman commented Sep 29, 2016

It honors SSL_CERT_DIR and SSL_CERT_FILE though. If those aren't set, perhaps we should do some searching. Kind of unfortunate that this initialization is happening eagerly though, would be better for startup time if we could defer it until an https connection is first initiated.

@tkelman
Copy link
Contributor

tkelman commented Sep 29, 2016

We also have some code here

julia/Makefile

Lines 456 to 459 in b36141f

# We need to bundle ca certs on linux now that we're using libgit2 with ssl
ifeq ($(shell [ -e $(shell openssl version -d | cut -d '"' -f 2)/cert.pem ] && echo exists),exists)
-cp $(shell openssl version -d | cut -d '"' -f 2)/cert.pem $(DESTDIR)$(datarootdir)/julia/
endif
that runs during make binary-dist to copy the build system's certs into the binary-dist tarball. If that snippet works more broadly, we could move it into deps/libgit2.mk and have it always run?

@vchuravy
Copy link
Member

At least on Linux the system's certs are meant to be upgraded over-time and
this would mean that the certs could get outdated.
It is quite sad that these locations are so non-standard. In my opinion the
correct course of action is to extend the lists of plausible defaults
(right now we are looking into a Debian specific place) and make libgit
init lazy.

On Fri, 30 Sep 2016 at 05:53 Tony Kelman [email protected] wrote:

We also have some code here

julia/Makefile

Lines 456 to 459 in b36141f

# We need to bundle ca certs on linux now that we're using libgit2 with ssl
ifeq ($(shell [ -e $(shell openssl version -d | cut -d '"' -f 2)/cert.pem ] && echo exists),exists)
-cp $(shell openssl version -d | cut -d '"' -f 2)/cert.pem $(DESTDIR)$(datarootdir)/julia/
endif

that runs during make binary-dist to copy the build system's certs into
the binary-dist tarball. If that snippet works more broadly, we could move
that code snippet into deps/libgit2.mk and have it always run?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#18693 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAI3avayUgvwoRbUUVQuMtN6gEYbBRS6ks5qvCU1gaJpZM4KHZum
.

@Ismael-VC
Copy link
Contributor

@TotalVerb I'm going to try to build Julia from source, using your fix if I get the same error while building, but the error I report above comes from the julia from the ubuntu package repositories, I think your fix doesn't work on that, since that julia is already built.

@staticfloat
Copy link
Member

@Ismael-VC Now that is an install case I never thought of when making the Julia PPA.

What does openssl version -d say for you?

@Ismael-VC
Copy link
Contributor

Ismael-VC commented Nov 25, 2016

@staticfloat the output of that and also which and julia info:

ismaelvc@TOYBOX ~ % openssl version -d
OPENSSLDIR: "/usr/lib/ssl"

ismaelvc@TOYBOX ~ % which openssl
/usr/bin/openssl

ismaelvc@TOYBOX ~ % apt show julia
Package: julia
Priority: extra
Section: science
Installed-Size: 540 MB
Maintainer: Debian Julia Team <[email protected]>
Version: 0.6.0-3000~201611231441~ubuntu14.04.1
Suggests: ess (>= 12.09-1~), julia-doc
Depends: libc6 (>= 2.17), libgcc1 (>= 1:4.1.1), libgfortran3 (>= 4.6), liblapack3 | liblapack.so.3, libopenblas-base, libstdc++6 (>= 4.8), zlib1g (>= 1:1.2.0), liblapack3 | libatlas3-base, libfftw3-3, libcholmod1.7.1 | libcholmod2.1.2 | libcholmod3.0.6 | libcholmod3, libumfpack5.4.0 | libumfpack5.6.2 | libumfpack5.7.1 | libumfpack5, libgmp10 (>= 6.1.0), libarpack2, librmath-julia-dev, libmpfr4, git, libssl1.0.0
Pre-Depends: multiarch-support
Download-Size: 121 MB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/staticfloat/julianightlies/ubuntu/ trusty/main amd64 Packages
Description: high-performance programming language for technical computing
 Julia is a high-level, high-performance dynamic programming language for
 technical computing, with syntax that is familiar to users of other technical
 computing environments. It provides a sophisticated compiler, distributed
 parallel execution, numerical accuracy, and an extensive mathematical function
 library. The library, mostly written in Julia itself, also integrates mature,
 best-of-breed C and Fortran libraries for linear algebra, random number
 generation, FFTs, and string processing. Julia programs are organized around
 defining functions, and overloading them for different combinations of
 argument types (which can also be user-defined).
 .
 This package provides a complete Julia installation (JIT compiler, standard
 library, text-based user interface).

N: There are 2 additional records. Please use the '-a' switch to see them.

@staticfloat
Copy link
Member

Can you show me what files are sitting in /usr/lib/ssl? Specifically, I'm looking for something that looks like a bundle of SSL certificates, hopefully something named certs.pem or ca.crt or something like that.

@TotalVerb
Copy link
Contributor

It might also be named certs/ca-certificates.crt.

@Ismael-VC
Copy link
Contributor

@staticfloat here is what's in that path:

ismaelvc@TOYBOX ~ % tree /usr/lib/ssl
/usr/lib/ssl
├── certs -> /etc/ssl/certs
├── misc
│   ├── CA.pl
│   ├── CA.sh
│   ├── c_hash
│   ├── c_info
│   ├── c_issuer
│   ├── c_name
│   └── tsget
├── openssl.cnf -> /etc/ssl/openssl.cnf
└── private -> /etc/ssl/private

3 directories, 8 files

@staticfloat
Copy link
Member

staticfloat commented Nov 27, 2016 via email

@Ismael-VC
Copy link
Contributor

Ismael-VC commented Nov 27, 2016

Lots of .pems I don't see anything with git:

ismaelvc@TOYBOX ~ % tree /usr/lib/ssl/certs | grep -i git     
1 ismaelvc@TOYBOX ~ %

@staticfloat
Copy link
Member

staticfloat commented Nov 27, 2016 via email

@JaredCrean2
Copy link
Contributor

I'm getting the same error with the latest master, but setting SSL_CERT_DIR and SSL_CERT_FILE did not fix it:

creanj@mastermind:~/fasttmp/build/julia_0.6_2$ ./julia
fatal: error thrown and no exception handler available.
Base.InitError(mod=:LibGit2, error=ErrorException("error initializing LibGit2 module"))
rec_backtrace at /fasttmp/creanj/build/julia_0.6_2/src/stackwalk.c:84
record_backtrace at /fasttmp/creanj/build/julia_0.6_2/src/task.c:239
jl_throw at /fasttmp/creanj/build/julia_0.6_2/src/task.c:565
__init__ at ./libgit2/libgit2.jl:628
unknown function (ip: 0x7fe7e84e9698)
jl_call_method_internal at /fasttmp/creanj/build/julia_0.6_2/src/julia_internal.h:248 [inlined]
jl_apply_generic at /fasttmp/creanj/build/julia_0.6_2/src/gf.c:2217
jl_apply at /fasttmp/creanj/build/julia_0.6_2/src/julia.h:1410 [inlined]
jl_module_run_initializer at /fasttmp/creanj/build/julia_0.6_2/src/toplevel.c:85
_julia_init at /fasttmp/creanj/build/julia_0.6_2/src/init.c:704
julia_init at /fasttmp/creanj/build/julia_0.6_2/src/task.c:292
main at /fasttmp/creanj/build/julia_0.6_2/ui/repl.c:259
__libc_start_main at /home/aurel32/debian/co-packages/eglibc/squeeze/eglibc-2.11.3/csu/libc-start.c:244
unknown function (ip: 0x4016ac)

ca-certificates.crt is located in /etc/ssl/certs/, and /usr/lib/ssl/certs/ is a symlink to it.

@staticfloat
Copy link
Member

Please gist the output of strace ./julia.

@JaredCrean2
Copy link
Contributor

Here it is.

@andreasnoack
Copy link
Member

Not solved. On one of the clusters I use, I just got the error again and I can fix it with SSL_CERT_FILE=/etc/ssl/certs/certSIGN_ROOT_CA.pem. This is on master.

@andreasnoack andreasnoack reopened this Mar 23, 2017
@staticfloat
Copy link
Member

What distribution is it? I've never seen one with a name so strange. Is that the cert file that gets used by openssl if you do an s_client test?

@JeffBezanson JeffBezanson modified the milestones: 0.6.x, 0.6.0 Mar 30, 2017
@tkelman
Copy link
Contributor

tkelman commented Mar 30, 2017

Anyone who still has an issue here, try writing a patch similar to #19390 adding a case for wherever your system has the certs, since almost everyone else is covered by the existing cases.

@andreasnoack
Copy link
Member

I don't see this issue anymore

@JaredCrean2
Copy link
Contributor

I investigated this further and think there is a different problem in my case. Opened a new issue #24625.

@vtjnash
Copy link
Member

vtjnash commented Jan 30, 2021

Is this still an issue?

@colbec
Copy link
Author

colbec commented Jan 30, 2021

All OK here.

@colbec colbec closed this as completed Jan 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
libgit2 The libgit2 library or the LibGit2 stdlib module regression Regression in behavior compared to a previous version system:linux Affects only Linux
Projects
None yet
Development

Successfully merging a pull request may close this issue.