-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Logstash 1.5.3 SSL problems #3657
Comments
Getting all kinds of weird failures even with Logstash 1.5.2... |
Do you have the same issue with Java 1.8? |
@ph it's on my todo list to try. |
Same problem on Java 1.8.0_60-internal |
@ph can you help me test these scenarios (LSF 0.4.0 to LS 1.5.3, LS 1.5.2 to LS 1.5.3, etc)? I am still trying to figure out what's wrong with my workstation. |
I also tested in an Ubuntu 14.04 docker container. Same errors. This is bizarre. Will try on a different system tomorrow. |
Here's my setup: java 1.8.0_45, LSF 0.4.0, ubuntu 14.04 server 64bit. Starting lumberjack input listener {:address=>"172.xx.xx.5:6xx2", :level=>:info, :file=>"logstash/inputs/lumberjack.rb", :line=>"50", :method=>"register"} Later, nothing appears after "Logstash startup completed", but LSF keeps firing: 2015/07/28 09:52:16.243990 Connecting to [172.xx.xx.5]:6782 (172.xx.xx.5) With LS 1.5.2 no errors and logs are being processed. p.s. I use SAN IP addresses in certificate, not CN=somename. |
Here's configuration i use in both tests: { LS config: |
MacOS X
|
We can rule out the jruby version both 1.5.2 and 1.5.3 are shipped with the same version.
|
Linux vagrant-ubuntu-trusty-32 3.13.0-55-generic #92-Ubuntu SMP Sun Jun 14 18:33:09 UTC 2015 i686 i686 i686 GNU/Linux
|
@ph those failures are ones I expected given all the user reports. whew |
@jordansissel also consistent between jvm. |
I have downgraded |
Disabling the monkey patch from #3579 make LSF 0.4.0 correctly sends events to 1.5.3, I'll look into it. |
@ph and I paired up on Zoom to discuss this. We narrowed it down to the Ruby 1.9.3 long ago changed their "default" settings for openssl in the So the fix will be to not force |
We'll be releasing v1.5.4 as soon as we can after testing and confirming this fix. |
I still need to do more testing on my end, will have the PR tomorrow, |
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: elastic#3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: elastic#3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: #3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: #3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: #3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: elastic#3657
We have discovered that in some cases and some plaftorms configuring a default `verify_mode` when creating a SSL/TCPServer could make the certificate verification fail. Ruby default behavior is to use `NIL` when creating a new ssl context, this revert that change. keep in mind that all TCP clients using SSL **must** use `VERIFY_PEER` as their verify mode to prevent man in the middle attack. Fix: elastic#3657
May I know roughly when v1.5.4 will be out? |
@foresightyj I don't have an estimate. If I gave one, it may be wrong, and it would set your expectations in a way that would cause you upset :) If there's a bug fixed in 1.5.4 and it's already fixed, you can build from the 1.5 branch |
Thanks for the candid answer. I didn't have much experience with ruby especially the tooling around it. I can wait. |
@timukas I had a similar config to yours and was seeing the TLS handshake error. I was able to resolve the problem by removing {
"network": {
"servers": [ "172.xx.xx.5:6xx2" ],
"ssl ca": "/etc/logstash-forwarder/ssl/ca.crt"
},
"files": [
{
"paths": [ "/var/log/auth.log" ],
"fields": { "type": "syslog" }
}
]
} |
Confirmed that using only "ssl ca" works. I just updated to latest released LSF and LS v1.5.3 and seeing handshake errors. I switched to log-courier and still see same errors. Removing "ssl certificate" and "ssl key" works. |
What TinLe said above was a great help to me. All of my logstash-fowarders had stopped talking to my logstash-1.5.3 until I removed those configuration settings. FWIW, when diagnosing this problem, I discovered that 'openssl s_client -connect 10.x.x.x:5043 -tls1' worked (i.e. specifying tls1 resulted in a successful connection). |
Removing "ssl certification" and "ssl key" on the forwarder config also solved my issue. Thanks! |
Apologies for this. 1.5.3 had an issue where SSL servers within Logstash rejected connections from clients when they provided certificates. We fixed this in 1.5.4. Upgrading to LS 1.5.4 will fix this issue. |
I can also confirm that updating to 1.5.4 also solved the issue without a needed workaround. Thanks! |
i could still see "Read error looking for ack: EOF" ERROR on logstash1.5.4 and logstash-forwarder0.4.0 Here is my logstash-forwarder config "files": [ ERROR: |
same error here |
Something is funky with Logstash 1.5.3's SSL. I'm not sure what, yet.
Tests -
With a self-signed CN=localhost certificate on java 1.7.0_79
certificate verify failure
certificate verify failure
server key exchange invalid
Socket closed
RSA_EAY_PUBLIC_DECRYPT:data too large for modulus / SSL routines:ssl3_get_key_exchange:bad signature
Related tickets
The text was updated successfully, but these errors were encountered: