SSL core dump in v6.9.1

[rdkgit]

• v6.9.1
• Red Hat Enterprise Linux Server release 7.3
• 3.10.0-514.el7.x86_64

Created simple https server and try to connect to it with various clients. node core dumps w/o any stack trace. GDB output below. Sorry if this is duplicate. I was not find anything related via searching. Code works fine on Fedora system running 4.6.1. Also, system it is core-dumping on is an EC2 instance thus virtual.

#0  0x00007f228053a610 in asn1_enc_restore () from /lib64/libcrypto.so.10
Missing separate debuginfos, use: debuginfo-install glibc-2.17-157.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-26.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libselinux-2.5-6.el7.x86_64 libstdc++-4.8.5-11.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) where
#0  0x00007f228053a610 in asn1_enc_restore () from /lib64/libcrypto.so.10
#1  0x00007f228053786b in ASN1_item_ex_i2d () from /lib64/libcrypto.so.10
#2  0x00007f2280537d11 in asn1_template_ex_i2d () from /lib64/libcrypto.so.10
#3  0x00007f2280537aeb in ASN1_item_ex_i2d () from /lib64/libcrypto.so.10
#4  0x00007f2280537bef in asn1_item_flags_i2d () from /lib64/libcrypto.so.10
#5  0x00007f22808478cd in ssl3_add_cert_to_buf () from /lib64/libssl.so.10
#6  0x00007f2280847fa3 in ssl3_output_cert_chain () from /lib64/libssl.so.10
#7  0x00007f228083b755 in ssl3_send_server_certificate ()
   from /lib64/libssl.so.10
#8  0x00007f228083cbcd in ssl3_accept () from /lib64/libssl.so.10
#9  0x00007f228084a478 in ssl23_accept () from /lib64/libssl.so.10
#10 0x00007f228084b612 in ssl23_read () from /lib64/libssl.so.10
#11 0x0000000000e668c5 in node::TLSWrap::ClearOut (this=this@entry=0x2b1fc80)
    at ../src/tls_wrap.cc:420
#12 0x0000000000e66b93 in Cycle (this=0x2b1fc80) at ../src/tls_wrap.h:103
#13 node::TLSWrap::DoRead (this=0x2b1fc80, nread=201, buf=<optimized out>, 
    pending=<optimized out>) at ../src/tls_wrap.cc:727
#14 0x0000000000e38c05 in OnRead (pending=UV_UNKNOWN_HANDLE, 
    buf=0x7ffe9b499820, nread=201, this=0x29d4fe8) at ../src/stream_base.h:180
#15 node::StreamWrap::OnReadCommon (handle=<optimized out>, nread=201, 
    buf=0x7ffe9b499820, pending=UV_UNKNOWN_HANDLE) at ../src/stream_wrap.cc:245
#16 0x00007f22812dc774 in uv__read (stream=stream@entry=0x29d5040)
    at src/unix/stream.c:1192
#17 0x00007f22812dcedc in uv__stream_io (loop=<optimized out>, w=0x29d50c8, 
    events=1) at src/unix/stream.c:1259
#18 0x00007f22812e1db8 in uv__io_poll (
    loop=loop@entry=0x7f22814ec900 <default_loop_struct>, timeout=119999)
    at src/unix/linux-core.c:380
#19 0x00007f22812d3998 in uv_run (loop=0x7f22814ec900 <default_loop_struct>, 
    mode=UV_RUN_ONCE) at src/unix/core.c:354
#20 0x0000000000df81d8 in node::Start (argc=2, argv=<optimized out>)
    at ../src/node.cc:4608
#21 0x00007f227db25b35 in __libc_start_main () from /lib64/libc.so.6
#22 0x00000000006f4299 in _start ()

[mscdex]

/cc @indutny

[sam-github]

@rdkgit can you provide a standalone example?

[indutny]

@rdkgit could you please provide a certificate chain that you used?

[rdkgit]

Here is an example I just now tested and it core dumps node when I connect to it using openssl s_client.
Test with: /bin/openssl s_client -connect hostname:8443

// test.js
// test to see if node core dumps

var fs = require('fs');
var constants = require('constants');
var http = require('http');
var https = require('https');
var httpPort = 8080;
var httpsPort = 8443;

var sslOptions = {

  key: fs.readFileSync('/tmp/exterus-key.pem'),
  cert: fs.readFileSync('/tmp/exterus-cert.pem'),
  // startssl sub.class1.server.ca.pem append >> cert pem
  secureProtocol: 'SSLv23_method',
  secureOptions: constants.SSL_OP_NO_SSLv3 | constants.SSL_OP_NO_SSLv2,
};

// set up proxy server and start listening!!
http.createServer().listen(httpPort);
console.log("Test listening on "+httpPort);

https.createServer(sslOptions).listen(httpsPort);
console.log("Test listening on "+httpsPort);

[rdkgit]

The cert I'm using is from startssl. Do you want me to send you the cert itself? Or, what kind of output would you like?

Thanks,

Bobby

[indutny]

@rdkgit may I ask you to put the exterus-cert.pem contents to https://gist.github.com/ ? It seems like it is crashing when it tries to encode the certificate chain, so having the full certificate chain may be a key to understanding the cause of the crash.

[rdkgit]

OK, uploaded. Are you able to reference it? Sorry if thats a dumb question. Havent used this feature of github before.
rdkgit/exterus-cert.pem

[indutny]

Thanks! Here is the link: https://gist.github.com/rdkgit/130b3476d82a61fab9a2ed164ac102ee

[indutny]

@rdkgit have you built node.js from source? Or have you downloaded the binary?

[rdkgit]

I'm using the pre-built platform packages for both Fedora, RHEL, and Centos. It core dumps on my ec2 instance but not my Fedora desktop running older version of node.

[indutny]

Oh wait! I just realized that you are using shared openssl library: /lib64/libssl.so.10. Do you know which version of OpenSSL that EC2 system has?

[rdkgit]

OpenSSL 1.0.1e-fips 11 Feb 2013

On my Fedora system where no core dumps (but older nodejs version), I have 1.0.2h though.

[indutny]

@rdkgit what does node -pe process.versions.openssl say?

[rdkgit]

/bin/node -pe process.versions.openssl
1.0.1e-fips

[rdkgit]

ldd /bin/node
linux-vdso.so.1 => (0x00007ffc839ef000)
libz.so.1 => /lib64/libz.so.1 (0x00007fea4829d000)
libuv.so.1 => /lib64/libuv.so.1 (0x00007fea48079000)
librt.so.1 => /lib64/librt.so.1 (0x00007fea47e70000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fea47c54000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fea47a3b000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fea47836000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fea475c8000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fea471de000)
libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007fea46ddf000)
libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007fea46a66000)
libicudata.so.50 => /lib64/libicudata.so.50 (0x00007fea45492000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fea45189000)
libm.so.6 => /lib64/libm.so.6 (0x00007fea44e87000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fea44c71000)
libc.so.6 => /lib64/libc.so.6 (0x00007fea448af000)
/lib64/ld-linux-x86-64.so.2 (0x00007fea484bc000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fea44661000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fea4437a000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fea44175000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fea43f43000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fea43d33000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fea43b2f000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fea43915000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fea436ed000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fea4348c000)

[indutny]

ok, at least it matches...

[indutny]

Is it the same on your desktop Fedora?

[rdkgit]

No, my desktop is running. No core dump on desktop config.

/bin/node --version
v4.6.1

/bin/openssl version
OpenSSL 1.0.2h-fips 3 May 2016

On my own webserver, it also runs fine, no core dump.

/bin/node --version
v0.10.36
/bin/openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

[bnoordhuis]

I think you should report this to RedHat unless you also get crashes with the official v6.9.1 binaries from https://nodejs.org/. Please post a backtrace if that is the case.

[rdkgit]

Back from a weekend of backpacking. I will download/install the official nodejs binaries and see if that fixes it.

Thanks,

Bobby

[rdkgit]

Hi!

I completed uninstalled nodejs rpm and npm rpm from the RHEL system. I downloaded and installed nodejs tar file from website.

Node now works with my test script and with my original script both with http and https.

Whats the best way to report this to RedHat?

I'm using RHEL with an EC2 instance and don't have a direct relationship with RedHat.

Will they accept a bug request from a non-paying user?

Thanks,

Bobby

[bnoordhuis]

cc @sgallagher - maybe you can answer @rdkgit's question?

I'll close out the bug report since it's a downstream issue.

[rdkgit]

Thanks! Happy to provide any/all info to RH so we can fix this in the RHEL package.

Bobby

[sgallagher]

@rdkgit Where did you get the package for RHEL? Was it one provided by Red Hat Software Collections or was it something you got from Fedora EPEL (Extra Packages for Enterprise Linux)?

The former are supported by Red Hat, the latter are community-supported.

If you got it from EPEL, then you should file a bug at https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora%20EPEL in the nodejs package.

Please also include the full output of rpm -q openssl (or rpm -q openssl-fips if you're using that one). I should note that the EPEL version hasn't been tested with FIPS; we only expect it to work with the standard OpenSSL. It may be that Node.js is not FIPS-compatible.

[rdkgit]

Hey. I believe it was from RHEL EPEL repo as I did not find it in the regular/default REPOs that were set up with my ec2 instance.

yum list nodejs
Loaded plugins: amazon-id, rhui-lb, search-disabled-repos
Available Packages
nodejs.x86_64 1:6.9.1-1.el7 epel

I will file a bug via bugzilla.

Thanks,

Bobby

[sgallagher]

For what it's worth, I just tested this with:

openssl-libs-1.0.1e-51.el7_2.7.x86_64
nodejs-6.9.1-1.el7.x86_64

I was unable to reproduce the issue in either FIPS or non-FIPS mode with that version of OpenSSL (the latest available on RHEL 7.2). Can you confirm that you're using the most recent version? Please provide the output of rpm -q openssl-libs, @rdkgit

[rdkgit]

Hi!

The ec2 instance I'm running has RHEL 7.3 and the following:

openssl-libs-1.0.1e-60.el7.x86_64
npm-3.10.8-1.6.9.1.1.el7.x86_64
nodejs-6.9.1-1.el7.x86_64

When I use this config, it core dumps reliably with my test program. I will create a bugzilla entry.

Thanks,

Bobby

[sgallagher]

@rdkgit Please include detailed information about how you generated the certificate in question (as in, exact steps). My guess is that there's something atypical about the certificate or its CA chain that's triggering a behavior I can't reproduce with a certificate generated the way I normally do.

[rdkgit]

Hi!

Here is the command I ran to generate the CSR. I then uploaded to startssl and got the cert.

/bin/openssl req -out exterus.csr -new -newkey rsa:2048 -nodes -keyout exterus.key

I generated the csr on my desktop fedora system using my local copy of openssl.

%/bin/openssl version
OpenSSL 1.0.2h-fips 3 May 2016

Bobby

[rdkgit]

Update from bugzilla investigation. https://bugzilla.redhat.com/show_bug.cgi?id=1394948

My startssl cert .pem file also had the intermediate startssl cert appended to the end of it. Apparently, this is known to crash some revs of nodejs due to either openssl bug or bug in node code. Not sure. Either way, when I remove the intermediate cert (its not needed anyway), the problem goes away.

Thanks to all for the help resolving this.

Bobby

[sam-github]

@rdkgit I can't comment on the bugzilla, but if they have questions about node, they should ask them here, I'm familiar with the referenced code.

Btw, I'm surprised you don't need the intermediate cert... how does the peer know your intermediate if you don't send it?

[rdkgit]

Hi!

I'm using the startcom class-1 server ca cert and that seems to make everyone happy. The convention that I understood is that one could append that the CA server cert to one's server cert. That works in other versions of node but this particular version of node core dumps. When I moved the start intermediate cert into the CA SSL option, everything worked.

Thanks,

Bobby

[mastergberry]

I can confirm this same issue. Have a very similar setup to the original one described and removing the second certificate in the chain did solve the issue. Seems to be a low level C library bug as reported on the redhat website.

[yuriploc]

Just for documentation sake, I had this issue with an EC2 instance, openssl and Let's Encrypt aswell:

segfault in libcrypto.so.1.0.1e

npm-3.10.10-1.6.9.4.2.el7.x86_64
nodejs-6.9.4-2.el7.x86_64
openssl-1.0.1e-60.el7_3.1.x86_64

Replaced fullchain.pem by cert.pem in httpsOptions and everything worked out great.

[rcjpisani]

@yuriploc thanks for the suggestion, I replaced my fullchain.pem file with cert.pem and everything is working now (using CentOS 7 and Let's Encrypt certs).