panic in addTLS in cgo code

[jmgvfr04]

Hi, has anyone seen a stack like this from a go panic? This is using version 1.22.5-1

SIGSEGV: segmentation violation
PC=0x7faddcc4acb1 m=5 sigcode=1 addr=0x40
signal arrived during cgo execution

goroutine 11437845 gp=0xc0005c41c0 m=5 mp=0xc000100008 [syscall]:
runtime.cgocall(0xc1d5c0, 0xc001327858)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/cgocall.go:157 +0x4b fp=0xc001327830 sp=0xc0013277f8 pc=0x41b7cb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_PKEY_derive(0x7fadc8001400, 0xc000f3c0f0, 0xc002bb0010)
        _cgo_gotypes.go:1539 +0x4b fp=0xc001327858 sp=0xc001327830 pc=0x5a986b
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF.func6(0xc000f3c030?, {0xc000f3c0f0?, 0x0?, 0xc000f3c030?}, 0xc002bb0010)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x67 fp=0xc001327898 sp=0xc001327858 pc=0x5bf8e7
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0x60067b?, {0xc000f3c030, 0x30, 0x30}, {0x0, 0x0, 0x0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:140 +0x1a9 fp=0xc0013278f0 sp=0xc001327898 pc=0x5bf749
crypto/internal/backend.ExtractHKDF(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x19e71c0?, {0x0?, 0xc0013279b8?, 0x41b825?}, {0x0?, 0x0?, 0x0?})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/key_schedule.go:93 +0x145 fp=0xc001327970 sp=0xc0013278f0 pc=0x68dba5
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc001327bd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:384 +0xd3 fp=0xc001327ac0 sp=0xc001327970 pc=0x674193
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc001327bd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:86 +0x274 fp=0xc001327b00 sp=0xc001327ac0 pc=0x6727f4
crypto/tls.(*Conn).clientHandshake(0xc00144a008, {0x12e9c30, 0xc0013dbb30})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client.go:265 +0x594 fp=0xc001327d30 sp=0xc001327b00 pc=0x66d034
crypto/tls.(*Conn).clientHandshake-fm({0x12e9c30?, 0xc0013dbb30?})
        <autogenerated>:1 +0x33 fp=0xc001327d58 sp=0xc001327d30 pc=0x693633
crypto/tls.(*Conn).handshakeContext(0xc00144a008, {0x12e9ca0, 0xc002184700})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1553 +0x3cb fp=0xc001327f70 sp=0xc001327d58 pc=0x66aa6b
crypto/tls.(*Conn).HandshakeContext(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1493
net/http.(*persistConn).addTLS.func2()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1573 +0x6e fp=0xc001327fe0 sp=0xc001327f70 pc=0x6f32ce
runtime.goexit({})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc001327fe8 sp=0xc001327fe0 pc=0x486be1
created by net/http.(*persistConn).addTLS in goroutine 11437527
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1569 +0x309

there is one other goroutine in cgo code in the panic log, not sure if that is relevant:

goroutine 11437828 gp=0xc000231180 m=nil [runnable]:
runtime.cgocall(0xc1db90, 0xc0011cf718)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/cgocall.go:157 +0x4b fp=0xc0011cf6f0 sp=0xc0011cf6b8 pc=0x41b7cb
vendor/github.com/golang-fips/openssl/v2._Cfunc_go_openssl_EVP_MD_CTX_new()
        _cgo_gotypes.go:1263 +0x48 fp=0xc0011cf718 sp=0xc0011cf6f0 pc=0x5a8948
vendor/github.com/golang-fips/openssl/v2.newEvpHash(0x6, 0x30, 0x80)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hash.go:134 +0x6e fp=0xc0011cf770 sp=0xc0011cf718 pc=0x5bbc6e
vendor/github.com/golang-fips/openssl/v2.NewSHA384(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hash.go:541
crypto/internal/backend.NewSHA384(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:139
crypto/sha512.New384()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/sha512/sha512.go:234 +0x25 fp=0xc0011cf7a0 sp=0xc0011cf770 pc=0x60a765
crypto.Hash.New(0x5b3d29?)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/crypto.go:131 +0x3d fp=0xc0011cf7e8 sp=0xc0011cf7a0 pc=0x5a30bd
crypto.Hash.New-fm()
        <autogenerated>:1 +0x25 fp=0xc0011cf800 sp=0xc0011cf7e8 pc=0x6936a5
vendor/github.com/golang-fips/openssl/v2.newHKDF(0xc001587140?, 0x1)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:24 +0x92 fp=0xc0011cf898 sp=0xc0011cf800 pc=0x5be8b2
vendor/github.com/golang-fips/openssl/v2.ExtractHKDF(0x60067b?, {0xc001587140, 0x30, 0x30}, {0x0, 0x0, 0x0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/vendor/github.com/golang-fips/openssl/v2/hkdf.go:109 +0x45 fp=0xc0011cf8f0 sp=0xc0011cf898 pc=0x5bf5e5
crypto/internal/backend.ExtractHKDF(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/internal/backend/openssl_linux.go:261
crypto/tls.(*cipherSuiteTLS13).extract(0x19e71c0?, {0x0?, 0xc0011cf9b8?, 0x41b825?}, {0x0?, 0x0?, 0x0?})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/key_schedule.go:93 +0x145 fp=0xc0011cf970 sp=0xc0011cf8f0 pc=0x68dba5
crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc0011cfbd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:384 +0xd3 fp=0xc0011cfac0 sp=0xc0011cf970 pc=0x674193
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc0011cfbd0)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client_tls13.go:86 +0x274 fp=0xc0011cfb00 sp=0xc0011cfac0 pc=0x6727f4
crypto/tls.(*Conn).clientHandshake(0xc000a2ca88, {0x12e9c30, 0xc0013652c0})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/handshake_client.go:265 +0x594 fp=0xc0011cfd30 sp=0xc0011cfb00 pc=0x66d034
crypto/tls.(*Conn).clientHandshake-fm({0x12e9c30?, 0xc0013652c0?})
        <autogenerated>:1 +0x33 fp=0xc0011cfd58 sp=0xc0011cfd30 pc=0x693633
crypto/tls.(*Conn).handshakeContext(0xc000a2ca88, {0x12e9ca0, 0xc0000ec700})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1553 +0x3cb fp=0xc0011cff70 sp=0xc0011cfd58 pc=0x66aa6b
crypto/tls.(*Conn).HandshakeContext(...)
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/crypto/tls/conn.go:1493
net/http.(*persistConn).addTLS.func2()
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1573 +0x6e fp=0xc0011cffe0 sp=0xc0011cff70 pc=0x6f32ce
runtime.goexit({})
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0011cffe8 sp=0xc0011cffe0 pc=0x486be1
created by net/http.(*persistConn).addTLS in goroutine 11437448
        /net/code/extsrc/go/ms1.22.5-1/linuxR_x86/src/net/http/transport.go:1569 +0x309

[qmuntal]

Thanks for reporting. I have some questions:

• Which Linux distro and OpenSSL version are you using?
• Is this panic sporadic? How often it happens?
• Can you provide a reproducer?

[jmgvfr04]

Thanks for the response! This happened once on SLES. I need to check on the openssl version.
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

[jmgvfr04]

In this case GO_OPENSSL_VERSION_OVERRIDE is not set so I believe it is using the standard version on the system:

ls -l /usr/lib64/libcrypto.so.1.1

-rwxr-xr-x 1 root root 3389800 May 17 2023 /usr/lib64/libcrypto.so.1.1

[jmgvfr04]

As I dig into it I will try to come up with a reproducer but I am not hopeful that I will be able to do so. Thanks

[dagood]

A note for repro attempts (on our side): an easy way to start that probably has the same OpenSSL as SLES 15.5:

Start with a opensuse/leap:15.5 Docker container and then:

zypper install -y wget tar gzip git gcc
wget https://download.visualstudio.microsoft.com/download/pr/766eefd8-51c7-431c-8b58-5136273eced8/d6f0ed417acc7881cb620a7c1bdd0358/go1.22.5-20240702.3.linux-amd64.tar.gz
tar -xf go1.22.5-20240702.3.linux-amd64.tar.gz

[jmgvfr04]

FYI - going to try SLES 15 SP6 which has an update to the default OpenSSL as per the release notes:
https://www.suse.com/releasenotes/x86_64/SUSE-SLES/15-SP6/index.html

5.7.3 OpenSSL 3.1.4 is now default #
In SLES 15 SP6, OpenSSL has been updated to version 3.1.4, replacing OpenSSL 1.1.1.

Because the development packages of different versions are mutually exclusive and automatic conflict resolution is not performed during updates, libopenssl1_1-devel should be manually selected for de-installation.