ETCD TLS still supports weak 64-bit block ciphers #8320

msi-normr · 2017-07-27T20:08:02Z

Description:
ETCD TLS still supports weak 64-bit block ciphers
Nessus security scanner indicates ETCD TLS port can still communicate using weak 64-bit block ciphers which is a security vaulnerabilty (SWEET32).

Steps to Reproduce:
1.) Setup ETCD for TLS
2.) Initiate a query to the ETCD https port and specify a weak cipher (ECDHE-RSA-DES-CBC3-SHA, DES-CBC3-SHA)
curl -v https://lama1-1-1:4001/v2/members --cert /etc/ssl/local_certs/etcd2/etcd2.crt --key /etc/ssl/local_certs/etcd2/etcd2.key --ciphers ECDHE-RSA-DES-CBC3-SHA

Actual Results:
ETCD responded from the query using the weak cipher

cs1-1-1 ~ # curl -v https://lama1-1-1:4001/v2/members --cert /etc/ssl/local_certs/etcd2/etcd2.crt --key /etc/ssl/local_certs/etcd2/etcd2.key --ciphers ECDHE-RSA-DES-CBC3-SHA

Trying 10.10.48.224...
TCP_NODELAY set
Connected to lama1-1-1 (10.10.48.224) port 4001 (#0)
ALPN, offering http/1.1
Cipher selection: ECDHE-RSA-DES-CBC3-SHA
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.2 (OUT), TLS header, Certificate Status (22):
TLSv1.2 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS change cipher, Client hello (1):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-DES-CBC3-SHA
ALPN, server did not agree to a protocol
Server certificate:
subject: C=US; ST=Illinois; L=Chic; O=Mot_S Inc; OU=Test; CN=lama1-1-1
start date: Jul 12 23:28:38 2017 GMT
expire date: Jul 6 23:28:38 2042 GMT
subjectAltName: host "lama1-1-1" matched cert's "lama1-1-1"
issuer: C=US; ST=Illinois; L=Chic; O=Mot_S Inc; OU=Test; CN=Sxxxxxxx_CA_xxxxxxxxxx
SSL certificate verify ok.

GET /v2/members HTTP/1.1
Host: lama1-1-1:4001
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 200 OK
< Content-Length: 183
< Content-Type: application/json
< Date: Thu, 27 Jul 2017 18:56:52 GMT
< X-Etcd-Cluster-Id: 7417be5c0ca6813f
<
{"members":[{"id":"b8c5b89e3b6c0487","name":"f0bc2c5cee094fb7997f02f07cd80a55","peerURLs":["https://cs1-1-1:2380"],"clientURLs":["http://10.10.48.223:2379","https://cs1-1-1:4001"]}]}

Connection #0 to host lama1-1-1 left intact
cs1-1-1 ~ # echo $?
0

Expected Results:
Expecting ETCD to fail query.

Occurs On:
CoreOS 1409.5.0 (Ladybug)
ETCD v3.1.6
openssl 1.0.2k

Additional Information:

Openssl indicates this is fixed in v1.1.0 current CoreOS versions still use OpenSSL 1.0.2k
Is there any plans for ETCD to have a http server that will have ciphers configurable.

/usr/lib/systemd/system/etcd-member.service

[Unit]
Description=etcd (System Application Container)
Documentation=https://github.com/coreos/etcd
Wants=network.target
Conflicts=etcd.service
Conflicts=etcd2.service

[Service]
Type=notify
Restart=on-failure
RestartSec=10s
TimeoutStartSec=0
LimitNOFILE=40000

Environment="ETCD_IMAGE_TAG=v3.1.6"
Environment="ETCD_NAME=%m"
Environment="ETCD_USER=etcd"
Environment="ETCD_DATA_DIR=/var/lib/etcd"
Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"

ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos
ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS
ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid

[Install]
WantedBy=multi-user.target

/etc/systemd/system/etcd-member.service.d/10-require-certs.conf

[Unit]
Requires=link_etcd2_cert.service
After=link_etcd2_cert.service

/etc/systemd/system/etcd-member.service.d/50-options.conf

[Service]
Environment="ETCD_DISCOVERY=https://discovery.etcd.io/5fe45a47005f6cd5853d4639236b74b9"
Environment="ETCD_ADVERTISE_CLIENT_URLS=http://10.10.48.223:2379,https://cs1-1-1:4001"
Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,https://0.0.0.0:4001"
Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://cs1-1-1:2380"
Environment="ETCD_LISTEN_PEER_URLS=https://cs1-1-1:2380"
Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd2/etcd2.crt"
Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd2/etcd2.key"
Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd2/etcd2-chain.pem"
lines 1-43

cs1-1-1 ~ # cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1409.5.0
VERSION_ID=1409.5.0
BUILD_ID=2017-06-22-2222
PRETTY_NAME="Container Linux by CoreOS 1409.5.0 (Ladybug)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

cs1-1-1 ~ # openssl version
OpenSSL 1.0.2k 26 Jan 2017

heyitsanthony · 2017-07-27T20:23:01Z

If etcd is going to worry about weak ciphers, maybe there should be a FIPS-140 mode? It seems the suggested way is cgo out to a certified library. /cc @xiang90

xiang90 · 2017-07-27T20:24:39Z

@heyitsanthony

Agreed.

swestcott · 2017-11-09T12:07:12Z

Weak cipher support in etcd was flagged up in a security test on our Kubernetes cluster too. We can reduce the risk via security group controls, but being able to whitelist ciphers would address the root cause.

swestcott · 2017-12-20T15:46:07Z

Further to the above, I feel that allowing users to choose ciphers themselves would offer greater flexibility and allow cipher configuration to be tailored for different regulatory environments. For example, PCI DSS refer to NIST SP 800-52 for TLS configuration.

This is the approach Kubernetes have taken via kubernetes/kubernetes#48859

xiang90 · 2017-12-22T02:13:40Z

@swestcott

It seems k8s approach is simpler, but it requires users to understand exactly what they expect, and it is go specific.

We probably should adopt the similar approach. Would you want to help on this?

Or would @spzala ?

swestcott · 2017-12-22T15:59:40Z

@xiang90 Understood, but I think it's responsible to expect users working in regulated environments to understand such settings. This also means users can response more quickly to any future cipher compromises.

We'll look to get a PR raised in the new year /cc @ThatsMrTalbot

philips · 2017-12-22T21:35:20Z

The Kubernetes solution looks reasonable to me.

I feel we should pick some default set of ciphers too. What about this list? https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#23-use-secure-cipher-suites

Did Kubernetes select a default set? I couldn't tell from glancing at the PR.

xiang90 · 2017-12-22T21:38:46Z

@philips

I feel we should pick some default set of ciphers too. What about this list?

This might break existing users after a upgrade. Every time we tighten TLS/security or fix security related bugs, we break some users though...

philips · 2017-12-22T21:48:29Z

True. Is Kubernetes just rolling with all ciphers?

On Fri, Dec 22, 2017 at 1:38 PM Xiang Li ***@***.***> wrote: @philips <https://github.com/philips> I feel we should pick some default set of ciphers too. What about this list? This might break existing users after a upgrade. Every time we tighten TLS/security or fix security related bugs, we break some users though... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8320 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACDCNWxY14csJ-MlwqCbQZ0FIbGa3x8ks5tDCFngaJpZM4OlxmM> .

-- CTO, CoreOS, Inc Tectonic is enterprise Kubernetes https://coreos.com/tectonic

xiang90 · 2017-12-22T21:51:01Z

Is Kubernetes just rolling with all ciphers?

The default one is all available ones in Go TLS as far as I can tell.

@swestcott probably have more details to fill in here.

swestcott · 2017-12-26T15:48:04Z

The default one is all available ones in Go TLS as far as I can tell.

@xiang90 Yes, that's my understanding too. All ciphers by default, with an optional whitelist.

victorgp · 2018-01-08T02:27:36Z

Yes, in Kubernetes, if the ciphers are not specified, the Go default ones will be used.

xiang90 · 2018-01-08T19:00:20Z

@swestcott kindly ping. any progress?

ThatsMrTalbot · 2018-01-12T11:30:15Z

Have the start of something over at https://github.com/ThatsMrTalbot/etcd - it only seems to work when the TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 cipher suite is provided. If it isnt in the list then I get an error in the etcd logs (see below) and openssl s_client -connect 127.0.0.1:2379 -tls1_2 refuses to connect

Error from etcd logs when TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 is not provided:

Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"; please retry.

xiang90 · 2018-01-19T02:05:33Z

@ThatsMrTalbot

can you send a WIP pr to the etcd upstream repo? then we can discuss there.

knisbet · 2018-04-24T16:10:16Z

@ThatsMrTalbot are you still working on this?

I'm also looking for this feature, as right now I have some settings hacked into a fork. So I can work on this if needed.

fmazoyer · 2018-05-31T09:50:18Z

Hi everyone,
I have no etcd/security knowledge :-), so I might be asking dumb questions.

Our Security team ran a scan on all our local systems and reported this:
Linux,2379,TLS/SSL Server Supports The Use of Static Key Ciphers,"Disable TLS/SSL support for static key

Is this ectd issue linked to 'Sweet32' vulnerability?
Is there a known workaround?

Thanks a lot for your help :-)

hexfusion · 2018-05-31T11:07:35Z

Our Security team ran a scan on all our local systems and reported this:
Linux,2379,TLS/SSL Server Supports The Use of Static Key Ciphers,"Disable TLS/SSL support for static key

Hi @fmazoyer what version of etcd was this scan against?

fmazoyer · 2018-05-31T11:14:06Z

Hi @hexfusion thanks for the swift reply :-)

It's docker's quay.io/coreos/etcd:v3.2.4

vhosakot · 2018-11-28T23:34:36Z

I checked all the TLS 1.2 ciphers with etcd 3.3 and found that etcd 3.3 by default (without the --cipher-suites argument) supports the following TLS cipher suites sorted by strength (encryption algorithm key length):

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA

All the TLS 1.2 ciphers were found using the openssl command:

openssl ciphers 'ALL:eNULL:@STRENGTH'

gyuho · 2018-11-28T23:42:32Z

@vhosakot Right, we default to whatever Go runtime provides (depending on the machine that you run etcd with). You can specify the cipher suites to exclude weaker ones.

vhosakot · 2018-11-28T23:54:56Z

Thanks @gyuho. Are the ciphers in

etcd/pkg/tlsutil/cipher_suites.go

Lines 21 to 44 in dedae6e

    
           var cipherSuites = map[string]uint16{ 
        
           	"TLS_RSA_WITH_RC4_128_SHA":                tls.TLS_RSA_WITH_RC4_128_SHA, 
        
           	"TLS_RSA_WITH_3DES_EDE_CBC_SHA":           tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_128_CBC_SHA":            tls.TLS_RSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_256_CBC_SHA":            tls.TLS_RSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_128_CBC_SHA256":         tls.TLS_RSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_RSA_WITH_AES_128_GCM_SHA256":         tls.TLS_RSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_RSA_WITH_AES_256_GCM_SHA384":         tls.TLS_RSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_ECDSA_WITH_RC4_128_SHA":        tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_RC4_128_SHA":          tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA":     tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384":   tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384": tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305":    tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, 
        
           	"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305":  tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, 
        
           }

the default ciphers Go runtime provides to etcd?

gyuho · 2018-11-29T00:00:37Z

@vhosakot I meant to say, default would be empty cipher suite, which then is populated by Go runtime :) Those are only used to check if specified cipher is valid, but we do not provider cipher suite ordering by default (let Go handle it).

vhosakot · 2018-11-29T21:44:15Z

@gyuho I have etcd up with --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 and the required keys and certs.

Now, how can I test with a client (like etcdctl or openssl) to verify that these ciphers are supported by etcd? Should the client send one of these ciphers when it connects to etcd?

tremble · 2018-11-30T09:02:15Z

I've found https://testssl.sh/ ( https://github.com/drwetter/testssl.sh ) helpful for testing ciphers, it doesn't explicitly check that etcd is working behind the ciphers, but it does test the initial SSL/TLS handshakes.

vhosakot · 2018-12-03T18:27:56Z

@gyuho I started etcd 3.3.10 with --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. Etcd is running at 127.0.0.1:12379.

See --cipher-suites at the end in:

/usr/local/bin/etcd --name=contiv-etcd --data-dir=/var/etcd/contiv-data \
  --client-cert-auth --trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --peer-client-cert-auth --peer-trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --peer-cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --peer-key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --advertise-client-urls=https://0.0.0.0:12379 \
  --listen-client-urls=https://0.0.0.0:12379 --listen-peer-urls=https://0.0.0.0:12380 \
  --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

But, when I test these three ciphers using openssl, I see just the ciphers TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 are accepted by etcd and not TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. See outputs below:

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 work:

$ openssl s_client -cipher ECDHE-RSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Secure Renegotiation IS supported
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256

$ openssl s_client -cipher ECDHE-RSA-AES256-GCM-SHA384 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Secure Renegotiation IS supported
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

But, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 does not work:

$ openssl s_client -cipher ECDHE-ECDSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
    Cipher    : 0000

Let me know if I need to open an issue against etcd for this. I'm using openssl 1.1.0g.

Full outputs below:

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 work:

$ openssl s_client -cipher ECDHE-RSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379
CONNECTED(00000003)
depth=0 CN = contiv-vpp-etcd-server
---
Certificate chain
 0 s:/CN=contiv-vpp-etcd-server
   i:/CN=contiv-vpp
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmzCCAoOgAwIBAgIJAPL98Lhvl9VwMA0GCSqGSIb3DQEBCwUAMBUxEzARBgNV
BAMMCmNvbnRpdi12cHAwHhcNMTgxMTI3MjAwNTQwWhcNMjExMTI2MjAwNTQwWjAh
MR8wHQYDVQQDDBZjb250aXYtdnBwLWV0Y2Qtc2VydmVyMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA7CFcVcdQprkM5eOkPT1CKTyoIs5q1ghqpRknyMyb
joAsfW5uYXR261546WtjbspMcqj5jRdkxt12jLVW5ftmQUfETgGD+RO7GetK0j+G
ULnBDn9nx5rvT1RaStliBc8m6QCmnSWrRXxr8KfENHwzLqvEfH4G7oKCQ1+iR3C4
8LbTBi/ZiZjnF9hlzcXxEBOCEj9BEZ2yz5m9Wirb9a4k20mgvd/NOdebNFvkFEh9
nghed4DTD0gMTt/8mCda2NbJBJI6TqsvA42fBEjyzt3Fy6/p8Cl1wudiCHh75j1N
qiFfFEbi9V2Dtz1V543LaxQ/5sPbAkeBalXvi79bZHLpeQIDAQABo4HhMIHeMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQUDpBvd+jPqYm1raWOQCXIqt5x1fAwRQYDVR0j
BD4wPIAUNyNVb8YSqKol+YFlgpfzrUMqMyKhGaQXMBUxEzARBgNVBAMMCmNvbnRp
di12cHCCCQCRTCQQha8XqTA5BgNVHREEMjAwgh12aG9zYWtvdC1kZXYtbWFzdGVy
MTE3MzdlMGJkMIIJbG9jYWxob3N0hwR/AAABMA0GCSqGSIb3DQEBCwUAA4IBAQB9
ypYXz4+iK0h7ncM1GQoeA9c/oHyXDJ0d2S9rtnlefIRaUdIVEJmLd0UzmRmac8ka
JjDuaoKkkR/DFp5vvkujwPUj/uqhM7A97R1SiNHJEo0sau53A6KRJivu3o5tyVhw
pf2S5b+RWaAUUuRS9KXbt7/gCm573cUbknlw7XRO+l78VITsiMyHSt2ahx7K9i/f
qaWNy6IABw+lDJsd3MOdBayEBcoX3/Nsh+JSyUQPAxNd4e2FoMP6eH7Atjm0V4Gi
Xe7KQqmiVG19TNvG9nPELXVw6g2yGoT8qGha3hnzngoyDpVnxS1GTtxf79KVtguv
oZbNG5U+6+dqxddxr3YW
-----END CERTIFICATE-----
subject=/CN=contiv-vpp-etcd-server
issuer=/CN=contiv-vpp
---
Acceptable client certificate CA names
/CN=contiv-vpp
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1378 bytes and written 229 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 51383BDD9643331A5D7DF6BABF279FE854581C048B40B2FD63690BD79BD9B27FD73B134CA099AF81CC4569920AEDE35A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543860957
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

$ openssl s_client -cipher ECDHE-RSA-AES256-GCM-SHA384 -connect 127.0.0.1:12379
CONNECTED(00000003)
depth=0 CN = contiv-vpp-etcd-server
---
Certificate chain
 0 s:/CN=contiv-vpp-etcd-server
   i:/CN=contiv-vpp
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmzCCAoOgAwIBAgIJAPL98Lhvl9VwMA0GCSqGSIb3DQEBCwUAMBUxEzARBgNV
BAMMCmNvbnRpdi12cHAwHhcNMTgxMTI3MjAwNTQwWhcNMjExMTI2MjAwNTQwWjAh
MR8wHQYDVQQDDBZjb250aXYtdnBwLWV0Y2Qtc2VydmVyMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA7CFcVcdQprkM5eOkPT1CKTyoIs5q1ghqpRknyMyb
joAsfW5uYXR261546WtjbspMcqj5jRdkxt12jLVW5ftmQUfETgGD+RO7GetK0j+G
ULnBDn9nx5rvT1RaStliBc8m6QCmnSWrRXxr8KfENHwzLqvEfH4G7oKCQ1+iR3C4
8LbTBi/ZiZjnF9hlzcXxEBOCEj9BEZ2yz5m9Wirb9a4k20mgvd/NOdebNFvkFEh9
nghed4DTD0gMTt/8mCda2NbJBJI6TqsvA42fBEjyzt3Fy6/p8Cl1wudiCHh75j1N
qiFfFEbi9V2Dtz1V543LaxQ/5sPbAkeBalXvi79bZHLpeQIDAQABo4HhMIHeMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQUDpBvd+jPqYm1raWOQCXIqt5x1fAwRQYDVR0j
BD4wPIAUNyNVb8YSqKol+YFlgpfzrUMqMyKhGaQXMBUxEzARBgNVBAMMCmNvbnRp
di12cHCCCQCRTCQQha8XqTA5BgNVHREEMjAwgh12aG9zYWtvdC1kZXYtbWFzdGVy
MTE3MzdlMGJkMIIJbG9jYWxob3N0hwR/AAABMA0GCSqGSIb3DQEBCwUAA4IBAQB9
ypYXz4+iK0h7ncM1GQoeA9c/oHyXDJ0d2S9rtnlefIRaUdIVEJmLd0UzmRmac8ka
JjDuaoKkkR/DFp5vvkujwPUj/uqhM7A97R1SiNHJEo0sau53A6KRJivu3o5tyVhw
pf2S5b+RWaAUUuRS9KXbt7/gCm573cUbknlw7XRO+l78VITsiMyHSt2ahx7K9i/f
qaWNy6IABw+lDJsd3MOdBayEBcoX3/Nsh+JSyUQPAxNd4e2FoMP6eH7Atjm0V4Gi
Xe7KQqmiVG19TNvG9nPELXVw6g2yGoT8qGha3hnzngoyDpVnxS1GTtxf79KVtguv
oZbNG5U+6+dqxddxr3YW
-----END CERTIFICATE-----
subject=/CN=contiv-vpp-etcd-server
issuer=/CN=contiv-vpp
---
Acceptable client certificate CA names
/CN=contiv-vpp
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1378 bytes and written 229 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 71D4C68150B5A92ACD20FE9CAB3FA6C81F7D2C775C98AA626B9B63065F33A42D939DE62A504514C6C0F9D84317D67A87
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543861072
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

But, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 does not work:

$ openssl s_client -cipher ECDHE-ECDSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 124 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543861174
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

hexfusion · 2018-12-03T18:39:40Z

But, when I test these three ciphers using openssl, I see just the ciphers TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 are accepted by etcd and not TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. See outputs below:

@vhosakot I believe your openssl needs to support the cipher locally. openssl ciphers will generate a list. Are the ciphers that fail on that list?

vhosakot · 2018-12-03T18:53:48Z

@hexfusion Yes, the cipher TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 that fails is supported locally by openssl and is set in --cipher-suites for etcd, but still fails. Isn't this an etcd issue?

$ openssl ciphers -v -V | grep ECDHE-ECDSA-AES128-GCM-SHA256
          0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017

hexfusion · 2018-12-03T19:28:14Z

@vhosakot I guess the server key algorithm is probably rsa instead of ecdsa? I don't believe it can be both. Curious if changing that gives success.

My guess is your server cert will have similar output.

openssl x509 -in server.pem -text -noout 
[..]
Signature Algorithm: sha256WithRSAEncryption 
[..]

example using cfssl CA format for ecdsa algorithm

  "key": {
    "algo": "ecdsa", <-- not rsa
    "size": 256
 },

vhosakot · 2018-12-03T19:39:33Z

Thanks @hexfusion. So, should the following certs and keys I'm using for etcd server support ecdsa instead of rsa (they are using rsa currently). If so, I'll test with ecdsa certs and keys for etcd server.

  --trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --peer-trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --peer-cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --peer-key-file=/var/contiv/etcd-secrets/etcd-server.key

hexfusion · 2018-12-03T19:45:28Z

Thanks @hexfusion. So, should the following certs and keys I'm using for etcd server support ecdsa instead of rsa (they are using rsa currently). If so, I'll test with ecdsa certs and keys for etcd server.

If you want to use ecdsa vs rsa ciphers that makes sense to me.

ECDHE-ECDSA-AES128-GCM-SHA256

vs

ECDHE_RSA_WITH_AES_128_GCM_SHA256

vhosakot · 2018-12-03T19:47:56Z

Thanks @hexfusion! I'll test with ecdsa certs and keys for etcd server.

vhosakot · 2018-12-05T01:01:21Z

@gyuho I see the following ciphers do not work and etcd fails to start with the following errors when I configure these ciphers for --cipher-suites. Aren't the following cipher suites supported by etcd?

2018-12-04 23:59:23.323820 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256"
2018-12-05 00:12:23.287448 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
2018-12-05 00:39:15.266940 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
2018-12-05 00:41:38.286827 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"
2018-12-05 00:45:04.253193 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
2018-12-05 00:47:55.280075 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
2018-12-05 00:51:46.264873 C | etcdmain: unexpected TLS cipher suite "TLS_RSA_WITH_AES_256_CBC_SHA256"

hexfusion · 2018-12-05T15:26:50Z

Hi @vhosakot did the other ciphers work with ecdsa cert?

@gyuho I see the following ciphers do not work and etcd fails to start with the following errors when I configure these ciphers for --cipher-suites. Aren't the following cipher suites supported by etcd?

The support or non support of a cipher is most likely going to be openssl vs etcd issue. For example my openssl 1.1.1 does not support any of the ciphers that are failing can you verify both the client/server do in your case also that you are using rsa certs?

vhosakot · 2018-12-06T20:24:39Z

@hexfusion I'm not using ecdsa certs and keys for etcd currently, just rsa as etcd is using rsa certs and keys currently.

In the following cipher suites that fail, I see just DHE-RSA-AES128-GCM-SHA256 and DHE-RSA-AES256-GCM-SHA384 supported by openssl (in the output of openssl ciphers 'ALL:eNULL:@STRENGTH') and not the others. I am using openssl 1.1.0g:

2018-12-04 23:59:23.323820 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256"
2018-12-05 00:12:23.287448 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
2018-12-05 00:39:15.266940 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
2018-12-05 00:41:38.286827 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"
2018-12-05 00:45:04.253193 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
2018-12-05 00:47:55.280075 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
2018-12-05 00:51:46.264873 C | etcdmain: unexpected TLS cipher suite "TLS_RSA_WITH_AES_256_CBC_SHA256"

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017

So, are cipher suites not supported by openssl not supported by etcd too?

hexfusion · 2018-12-06T21:15:24Z

So, are cipher suites not supported by openssl not supported by etcd too?

@vhosakot yeah sorry that not incorrect I took a deeper look. It seems that currently the only supported list of definable ciphers is listed in your previous comment.

etcd/pkg/tlsutil/cipher_suites.go

Lines 21 to 44 in dedae6e

    
           var cipherSuites = map[string]uint16{ 
        
           	"TLS_RSA_WITH_RC4_128_SHA":                tls.TLS_RSA_WITH_RC4_128_SHA, 
        
           	"TLS_RSA_WITH_3DES_EDE_CBC_SHA":           tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_128_CBC_SHA":            tls.TLS_RSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_256_CBC_SHA":            tls.TLS_RSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_RSA_WITH_AES_128_CBC_SHA256":         tls.TLS_RSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_RSA_WITH_AES_128_GCM_SHA256":         tls.TLS_RSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_RSA_WITH_AES_256_GCM_SHA384":         tls.TLS_RSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_ECDSA_WITH_RC4_128_SHA":        tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA":    tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_RC4_128_SHA":          tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA":     tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, 
        
           	"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA":      tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256":   tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256": tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, 
        
           	"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384":   tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384": tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, 
        
           	"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305":    tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, 
        
           	"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305":  tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, 
        
           }

etcd/embed/config.go

Lines 529 to 535 in 829c9b2

    
           for i, s := range ss { 
        
           	var ok bool 
        
           	cs[i], ok = tlsutil.GetCipherSuite(s) 
        
           	if !ok { 
        
           		return fmt.Errorf("unexpected TLS cipher suite %q", s) 
        
           	} 
        
           }

vhosakot · 2018-12-06T21:47:18Z

Thanks @hexfusion for confirming that those keys aren't supported by etcd. I'll let you know once I test with ecdsa certs and keys for etcd.

heyitsanthony added this to the unplanned milestone Aug 1, 2017

xiang90 modified the milestones: unplanned, v3.4.0 Dec 22, 2017

xiang90 added the area/performance label Dec 22, 2017

ThatsMrTalbot mentioned this issue Jan 24, 2018

tls: add cipher flag #9216

Closed

gyuho added type/feature area/tls and removed area/performance labels Feb 25, 2018

joelegasse self-assigned this May 7, 2018

gyuho self-assigned this Jun 1, 2018

gyuho mentioned this issue Jun 4, 2018

*: support TLS cipher suite whitelist #9801

Merged

gyuho closed this as completed in #9801 Jun 5, 2018

bbeaudoin mentioned this issue Aug 31, 2018

Set etcd pod cipher suites per OpenShift 3.10 documentation openshift/openshift-ansible#9869

Closed

vhosakot unassigned joelegasse Nov 28, 2018

hexfusion mentioned this issue Dec 5, 2018

*: define strict TLS 1.2 cipher suite default #10304

Closed

sureshsubramaniam mentioned this issue May 7, 2019

Secure socket communication protocols and ciphers should be configurable sensu/sensu-go#2914

Closed

vadimeisenbergibm mentioned this issue Jul 28, 2020

Fix Sweet32 vulnerability for microk8s versions before 1.18 canonical/microk8s#1440

Closed

superseb mentioned this issue Feb 4, 2021

Kubernetes etcd vulnerability (Ports 2380) 64-bit block cipher 3DES vulnerable to SWEET32 attack rancher/rke#2451

Closed

ETCD TLS still supports weak 64-bit block ciphers #8320

ETCD TLS still supports weak 64-bit block ciphers #8320

Comments

msi-normr commented Jul 27, 2017 • edited Loading

/usr/lib/systemd/system/etcd-member.service

/etc/systemd/system/etcd-member.service.d/10-require-certs.conf

/etc/systemd/system/etcd-member.service.d/50-options.conf

heyitsanthony commented Jul 27, 2017

xiang90 commented Jul 27, 2017

swestcott commented Nov 9, 2017

swestcott commented Dec 20, 2017 • edited Loading

xiang90 commented Dec 22, 2017 • edited Loading

swestcott commented Dec 22, 2017

philips commented Dec 22, 2017

xiang90 commented Dec 22, 2017

philips commented Dec 22, 2017 via email

xiang90 commented Dec 22, 2017

swestcott commented Dec 26, 2017

victorgp commented Jan 8, 2018

xiang90 commented Jan 8, 2018

ThatsMrTalbot commented Jan 12, 2018

xiang90 commented Jan 19, 2018

knisbet commented Apr 24, 2018

fmazoyer commented May 31, 2018

hexfusion commented May 31, 2018

fmazoyer commented May 31, 2018

vhosakot commented Nov 28, 2018 • edited Loading

gyuho commented Nov 28, 2018

vhosakot commented Nov 28, 2018

gyuho commented Nov 29, 2018

vhosakot commented Nov 29, 2018

tremble commented Nov 30, 2018

vhosakot commented Dec 3, 2018

hexfusion commented Dec 3, 2018

vhosakot commented Dec 3, 2018

hexfusion commented Dec 3, 2018

vhosakot commented Dec 3, 2018

hexfusion commented Dec 3, 2018

vhosakot commented Dec 3, 2018

vhosakot commented Dec 5, 2018

hexfusion commented Dec 5, 2018

vhosakot commented Dec 6, 2018

hexfusion commented Dec 6, 2018

vhosakot commented Dec 6, 2018

msi-normr commented Jul 27, 2017 •

edited

Loading

swestcott commented Dec 20, 2017 •

edited

Loading

xiang90 commented Dec 22, 2017 •

edited

Loading

vhosakot commented Nov 28, 2018 •

edited

Loading