SSL handshake fails because IP SANs are missing

[soupdiver]

I've used the command from the README for generating a SSL certificate. I use the same cert on the server and on the forwarder. When starting the forwarder I get the following error:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

So I see it is a problem with the cert and not the forwarder itself but maybe you can update the README.

[alphazero-es]

@soupdiver Thanks for the heads-up. The readme needs an update, but what we really need is to add a flag to skip verification. We'll use this issue to push that through.

[soupdiver]

👍

[alphazero-es]

@soupdiver: Felix, pull branch ISSUE-221 and give it a whirl & ping me back on your end w/ ack|nack. It works for me.
/Danke

[soupdiver]

@alphazero-es ack
Thanks!

[alphazero-es]

@soupdiver great.
@jordansissel Jordan, what's your take on this? This can merge clean to master. README is updated so it is buyer beware.

[jordansissel]

(edited this comment to remove incorrect information)

Basically, in order to use tls in Go, now you MUST specify a CN in the subject field. Wildcards are supported, but '*' does not match '.'

If you're using self signed certs or have a closed ecosystem of SSL certs in your environment, it's possible that CN=* or CN=*.example.com might just be fine for you.

master branch currently sets the ServerName (which must match the cert CN field) to the address of the server being talked to, so be aware of this.

I'm open to exposing a setting you can choose for the cert CN setting (what Go calls ServerName), if that would be useful.

[PavelPolyakov]

@jordansissel
I've compiled my logstash-forwarder using go version go1.3 linux/amd64 and created my certs using your solution - however I still have this:

Have I've missed something? Could you give an example how could I create the certificate specifying the IP SAN ?

@alphazero-es
Am I understand right that your pull request hasn't yet been merged and we're not able to use the

    "ssl strict verify": false,

yet ?

ALL:
Am I understand right that if I would specify the domain name in the logstash.json, then it would work without updating the certificate?

Regards,

[alphazero-es]

@PavelPolyakov The consensus on our end is that it would violate expectation of security for users. @jordansissel has explored and found a viable and practical workaround.

In the interim, if using master, use Go1.2 or satisfy Go1.3's requirements per crypto/tls.

[PavelPolyakov]

@alphazero-es
thanks for the answer,

do you mean the workaround about creating the certificate using CN=* ?

openssl req -subj '/CN=*/' -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt

The issue is, that even if I've created the certificate using the command provided, and updated it both on the receiver host and on the logstash-forwarder host - I still see that message.

[alphazero-es]

@PavelPolyakov Yes, it requires a minor change to the publisher code as well. If you can wait until Friday, hopefully we'll push it through then. In the interim, please use Go1.2, or the 221 branch (fully noting that it is NOT acceptable to us.

[driskell]

@PavelPolyakov Using CN=* will only work if you connect via hostname. If you connect via IP then the CN is completely ignored and the certificate verification requires that an IP Subject Alternate Name is added to the certificate. If you connect via hostname the CN=* will work, or even CN=hostname.you.connect.with will work.

I'm in process of writing a script to do all this in Log Courier. Currently I have a "make selfsigned" which works - but it needs manual editing of openssl.cnf to add IP SAN. I will be writing a script to do all this soon. It at least means people who just want quick security can get up and running fast. I'll post here when I'm done but @jordansissel or @alphazero-es are probably already doing something similar.

[PavelPolyakov]

@driskell
When I connect via hostname it states the next:

where the blank space is my domain

@alphazerp-es
I've just recompiled it using go 1.2.2 but still have the same:

my logstash-config.json:

Tomorrow I would try to do that again :) Anyway, thanks for your help.

[alphazero-es]

@PavelPolyakov don't understand the 1.2.x issue. I can't at this moment switch focus to this -- I would have to reinstall 1.2 and all that -- so this will have to wait until Friday (or possibly @jordansissel can provide some insight). if this is absolutely a blocker for you, then I suggest you just build from the prior commit and all should be OK.

Thank you for your patience.

/R

[driskell]

@PavelPolyakov this might be due to "*" matches "localhost" and "example" but not "example.com" according to my interpretation of it.

Try using the same hostname in certificate as you do in the config to connect.

"" is then not a solution as you'd need ".*" presumedly for "example.com"

[jordansissel]

@driskell I think you'er right, I didn't test . (I tested * against 'localhost'). Testing now.

[jordansissel]

What I can confirm:

• Without Subject (or SAN, aka Subject Alternative Name), connecting to a hostname gets you: Failed to tls handshake with 127.0.0.1 x509: certificate is valid for , not localhost
• Same as above, but connecting with "servers": [ "127.0.0.1:5043" ] (ip, not hostname): Failed to tls handshake with 127.0.0.1 x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs

IPs and Hostnames are configured differently, or appears to be so.

To permit IPs as the server name, your ssl cert must include IP:a.b.c.d as a subjectAltName field, for example, from a sample ssl config:

[v3_ca]
subjectAltName = IP:127.0.0.1

Generating a cert with this allows lsf to connect to 127.0.0.1 (by 'name') and it's succesful:

% openssl req -config ssl.conf -x509  -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt
# ... run logstash with this key/cert
% 2014/07/08 19:47:38.215155 Setting trusted CA from file: ./lumberjack.crt
2014/07/08 19:47:38.216058 Connecting to [127.0.0.1]:5043 (127.0.0.1)
2014/07/08 19:47:38.221611 Connected to 127.0.0.1

In this respect, DNS seems much easier to work with, but I won't burden you with requiring dns vs ip server names.

[jordansissel]

Also confirmed:

• * does not match '.' so you can't just do a simple wildcard.
• *.* does match foo.com
• *.*.pants.domain matches super.fancy.pants.domain

[jordansissel]

@PavelPolyakov Your SSL cert, if you are pointing atn an IP address for the server name, must include a subjectAlternativeName field that has IP:1.2.3.4 or whatever your IP is.

[PavelPolyakov]

ok guys, sorry, but I still don't have it working :(

1. I've tried the next command:

openssl req -subj '/CN=subdomain.domain.com/' -x509 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

The in config I had the next:

  "servers": [
            "subdomain.domain.com:12345"
        ],

The output was the next:

2014/07/09 08:37:36.341373 Connecting to [XX.XX.XX.XX]:12345 (subdomain.mydomain.com) 
2014/07/09 08:37:36.348066 Failed to tls handshake with XX.XX.XX.XX x509: certificate is valid for *, not subdomain.mydomain.com

So... looks like it still doesn't want to connect.

The certificate looks the next way:

1.
@jordansissel

Then I've tried your variant with IP.
I've created the ssl.conf with the next content:

[v3_ca]
subjectAltName = IP:XX.XX.XX.XX

Then I'm trying to generate the keys, and here is the output:

Pavels-Mac-mini:new_cert pavelp$ openssl req -config ./ssl.conf -x509  -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt
Generating a 2048 bit RSA private key
...............................+++
..........+++
writing new private key to 'logstash-forwarder.key'
-----
unable to find 'distinguished_name' in config
problems making Certificate Request
140735222621024:error:0E06D06C:configuration file routines:NCONF_get_string:no value:conf_lib.c:335:group=req name=distinguished_name

So I'm not able even to create the crt file (while .key is created).

I've tried to launch everything with the binary compiled using the go-1.3 .

However, currently I'm ok having the logstash-forwarder built using the go-1.2 . And I wait patiently for Friday, when the official solution would be exposed, hope it would work for me :)

Much thanks for the help.

Regards,

[ponny]

Same issue. Followed this guide and it broke: https://www.digitalocean.com/community/tutorials/how-to-use-logstash-and-kibana-to-centralize-and-visualize-logs-on-ubuntu-14-04

[ponny]

Tried with my ip: sudo openssl req -subj '/CN=xx.xx.xx.xxx/' -x509 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

No luck. Same error.

[ponny]

Added "ssl strict verify": false

No IP SANs errors any more. No data either ಠ_ಠ

[jordansissel]

@PavelPolyakov Your certificate error says certificate is valid for *, not subdomain.mydomain.com which indicates your certificate is still for "*" not for "subdomain.mydomain.com"

@ponny Based on what I have tested, you cannot add IP addresses in the CN field, you MUST add a subjectAltName as described in this comment: #221 (comment)

Allowing folks to disable ssl cert verification disables the very security that SSL provides.

We CAN solve this in a way that doesn't continuously confuse users. However, for NOW you must use the instructions I gave in the comment here: #221 (comment) - generate a correct certificate and it works.

[ponny]

Where will I find this ssl config on Ubuntu 14.04? Do I need to do this on the logstash server or the forwarder nodes?

[jordansissel]

@ponny what ssl config? I'm confused now. You mean an ssl config file? You can copy whatever your distro ships with (probably in /etc/ssl/openssl.cnf?) and add the subjectAltName lines.

[ponny]

@jordansissel That did it. Thanks.

[ponny]

The reason it wasn't sending data before was my clumsy fingers typed "falSE" which threw the config in to a spin.

[jimmod]

@driskell
Thank you for the lc-tlscert.go
I success using lc-tlscert on cloud environment with DNS
But when I tried to setup local testing environment without DNS, only IP, logstash forwarder error

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

Got any hint?

[driskell]

@jimmod Make sure you include the IP address in the SANs when running lc-tlscert and not common name. Never put the IP in the common name - always make up a DNS if you don't have one. Then IP always should go in the SAN. I'll make an update soon to make this clear in the instructions lc-tlscert gives.

[gebi]

@jimmod use DNS and add the records to /etc/hosts file of your local test machine, no special tool needed (the openssl command from README works just fine)

[jimmod]

@driskell ok it work now.
I found some error in the logstash configuration (my bad).
And confirm it work with IP or DNS via /etc/hosts file.
Thank you

[SJ04]

Hi there!!

I read this whole thread but it i still have the same Failed to tls handsake Error.
I used this command to create a Cert.

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=logstash.example.com

Just got a little Confusion about the CN, Should it be the Host Logstash Server or Client.

[RiaanPretorius]

Seems the main issue here is SSL does not work. And you can not disable it either. So no logstash-forwarder

[driskell]

@RiaanPretorius It does work - it's just easy to setup wrong. In every case I've helped we've found the problem and fixed it. If you'd like to share your config and steps followed there'll be someone happy to help I'm sure.

If you want without SSL there are alternatives available: beaver, nxlog, log-courier (which I maintain) I think all support plain tcp

[RiaanPretorius]

@driskell Ifollowed the guides i found on internet e.g. https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-4-on-ubuntu-14-04 and https://github.com/elastic/logstash-forwarder#generating-an-ssl-certificate and a lot more. My biggest problem is multiple domains, sub domains. Plain TCP would be welcomed.

I would gladly share my configs, but also keep in mind i followed the conversaion from start to end and nothing worked.

I am going try log-courier it seems to best option.

[ashwgupt]

All of the alternatives to logstash-forwarder look good but it still appears to me that none come with as easy and simple integration with Logstash as the forwarder - No plugin or extra installations on Logstash server side, no other dependencies, and ease to use config file. Please correct me if wrong.

Hence, I am very keen to make my forwarder work out but not able to due to the ssl handshake issue.

Can any one suggest what are the possible reasons for my forwarder (on RHEL 6) not able to work with my Logstash server cert file and throwing the below error?

2015/06/07 15:47:23.964418 Setting trusted CA from file: /usr/bin/mw_lumberjack.crt
2015/06/07 15:47:23.966107 Connecting to [10.51.167.10]:10010 (ukx01234.ab.dc-intl.com)
2015/06/07 15:47:23.971762 Failed to tls handshake with 10.51.167.10 x509: certificate is valid for , not ukx01234.ab.dc-intl.com

Pasting the config and cert details below.

Cert Issuer:-
Signature Algorithm: sha1WithRSAEncryption
Issuer: CN=.ab.dc-intl.com
Validity
Not Before: Jun 5 12:24:20 2015 GMT
Not After : Jun 4 12:24:20 2016 GMT
Subject: CN=.ab.dc-intl.com

Forwarder Config File:-
"network": {
# A list of downstream servers listening for our messages.
# logstash-forwarder will pick one at random and only switch if
# the selected one appears to be dead or unresponsive
"servers": [ "ukx01234.ab.dc-intl.com:10010" ],

# The path to your client ssl certificate (optional)
##"ssl certificate": "./logstash-forwarder.crt",
# The path to your client ssl key (optional)
##"ssl key": "./logstash-forwarder.key",

# The path to your trusted ssl CA file. This is used
# to authenticate your downstream server.
"ssl ca": "/usr/bin/mw_lumberjack.crt", 

I've already tried other options of mentioning the issuer's FQDN differently in the cert but NO luck.

I also understand GO can be used to create new set of certs but my first concern is that the instructions provided in forwarder's documentation should work for me unless am making a mistake or there is a technical issue.

[quickwind]

I am still wondering why not giving the option to turn off SSL, all the solutions so far are not ok for your deployment, we are using Docker for our micro-services, and we would like each container (per service instance) will have a logstash-forwarder to forward the logs to a centralized logstash shipper (also in Docker container) in the same host, these Docker containers are using bridge mode (NAT), there is not easy way for us to generate SSL cert in my understanding, as the IP address is dynamic, Docker container itself is not in domain, hostname is dynamic as well (although we can fix it by specifying the hostname for the logstash-shipper container and linking it to the fowarder containers, but still is it a valid option here as we only have /etc/hosts covered but not under any domain?).

I hate not having choices, it is like in China....

[tymik]

I was reading through the comments and what I see is that there is a need to update README file and manual on page in point that stands for genering certs - it is not mentioned there, that there is a need to set valid address in config file for ssl, where it should be mentioned and explained, what address (fqdn or num IP) should be used. So this is the first place, where the "bug" should get fixed.

[spargos]

I've tried to resolve this issue by regenerating the cert and key every which way and I am still getting the handshake issue:

Failed to tls handshake with x.x.x.x x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs

I've tried the following open ssl commands:

openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

openssl req -x509 -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

openssl req -subj '/CN=logstash.cloud.xxx.xxx.xxx/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/logstash-forwarder.key -out /etc/pki/tls/logstash-forwarder.crt -subj /CN=logstash.cloud.xxx.xxx.xxx > /dev/null 2>&1; openssl x509 -text -in lumberjack.crt | grep CN

When using the DNS CN option I have created an entry in /etc/hosts for the dns name pointing to the logstash server ip. I have also used the lc-tlscert utility for both DNS and IP with no success.

Specify the Common Name for the certificate. The common name
can be anything, but is usually set to the server's primary
DNS name. Even if you plan to connect via IP address you
should specify the DNS name here.

Common name: logstash.cloud.x.x.x.x

The next step is to add any additional DNS names and IP
addresses that clients may use to connect to the server. If
you plan to connect to the server via IP address and not DNS
then you must specify those IP addresses here.
When you are finished, just press enter.

DNS or IP address 1: x.x.x.x
DNS or IP address 2:

How long should the certificate be valid for? A year (365
days) is usual but requires the certificate to be regenerated
within a year or the certificate will cease working.

Number of days: 365
Common name: logstash.cloud.huit.harvard.edu
DNS SANs:
None
IP SANs:
x.x.x.x

The certificate can now be generated
Press any key to begin generating the self-signed certificate.

Successfully generated certificate
Certificate: selfsigned.crt
Private Key: selfsigned.key

Copy and paste the following into your Log Courier
configuration, adjusting paths as necessary:
"transport": "tls",
"ssl ca": "path/to/selfsigned.crt",

Copy and paste the following into your LogStash configuration,
adjusting paths as necessary:
ssl_certificate => "path/to/selfsigned.crt",
ssl_key => "path/to/selfsigned.key",

Any suggestions?

[driskell]

@jdleehuit If you use the lc-tlscert method - can you verify the error message? It should be different - if lc-tlscert says it added IP SAN, then the error message shouldn't be because there is no IP SAN. Are you sure the message is not changing between your different tests?

[driskell]

Also - when putting the hostname in /etc/hosts and connecting using the hostname it does not make sense that it would mention an IP address in the error message. Can you check again?

[spargos]

@driskell Here's the error I get with the lc-tlscert method:

2015/07/08 17:13:58.821467 Connecting to [x.x.x.x]:5000 (logstash.cloud.x.x.x.x)
2015/07/08 17:13:58.823598 Failed to tls handshake with x.x.x.x x509: certificate is valid for , not logstash.cloud.x.x.x.x

[spargos]

@driskell It is working now! Silly error, restarted logstash server after recopying over key and crt and all is well. Thanks for your help!

[jordansissel]

Creating SSL certificates is somewhat out of scope for this project. There are tools and documentation for doing this online in various places.

My strongest recommendation is to avoid the difficulty of certificate Subject Alternatives (IP SAN, etc) and just use DNS so you can use a CN=hostname subject. The choice is up to you :)

[w0www]

After hours i cant connect to my logstash server with logstash forwarding.

I tried a lot of things.

Now i have a certificate in server. Then i copy the certificate to the client side and i received error signed by unknown agent.

ANY SOLUTION? The server is ubuntu 14.04 and the client is ubuntu 14.04 too. thanks

[giantninja]

I've been fighting with this ssl failed to handshake issue for a while now, but finally got it working. For me the key was what @jordansissel said earlier. Also, you only need the ssl_sa field pointing to the .crt in the logstash-forwarder config. Like so:

logstash-forwarder.conf

{
  "network": {
    "servers": [ "dev.server.com:5043" ],
    "ssl ca": "/opt/logstash-forwarder/lumberjack.crt",

    "timeout": 15
  },
  "files": [
    {
      "paths": [
        "/var/log/php/*.log",
        "/var/log/php-fpm/*.log"
      ],
      "fields": { "type": "lumberjack" }
    }
  ]
}

logstash.conf

input {
  lumberjack {
    port => 5043

    ssl_certificate => "/opt/logstash-forwarder/lumberjack.crt"
    ssl_key => "/opt/logstash-forwarder/lumberjack.key"

    type => "lumberjack"
  }
}
output {
    stdout {}
    elasticsearch {
        cluster => "elasticsearch"
    }
}

and this worked for generating the crt/key:

openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout lumberjack.key -out lumberjack.crt -subj /CN=dev.server.com

[ozbillwang]

If we set cluster as "servers": [ "server1:6782", "server2:6782" ], in logstash forwarder config file, how can I generate the certificate?

Do I have to add domain name in the server hostname?

[brunof399]

lc-tlscert.go has worked perfectly for my Logstash deployment. However, it appears the certificates are created with SHA 1 hash instead of SHA 2. Is there any modification I can make to lc-tlscert so that the certificates I generate are SHA 2? Thanks!

[florianajir]

logstash and filebeat ssl configuration are a lot more strict than elasticsearch or kibana configuration.
IMO we should have an option to skip hostname verification like in the other products.
This could avoid a lot of headaches

[vhosakot]

I saw the same error when using mysql client in golang:

Failed to connect to database:  x509: cannot validate certificate for 10.111.202.229 because it doesn't contain any IP SANs

and setting InsecureSkipVerify to true (to skip verification of certificate) resolved it for me:

https://godoc.org/crypto/tls#Config