3.2/3.3 etcd server with TLS would start with error "tls: bad certificate" #9398

hongchaodeng · 2018-03-07T22:50:58Z

While debugging issues (might be relevant):

dep: update etcd 3.2.16, kubernetes 1.10.0, k8s codegen 1.10 coreos/etcd-operator#1727
ETCD with TLS showing error "transport: remote error: tls: bad certificate" for certificates generated using OpenSSL #8603

I have found that a single member etcd server on bootstrap will show error:

2018-03-07 22:36:51.136699 I | embed: rejected connection from "127.0.0.1:35160" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2018/03/07 22:36:51 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

Reproduce steps:
https://gist.github.com/hongchaodeng/7d62f3b5d30b58c783c382d9b629b819

Note that 3.1.11 didn't have this error log.

The text was updated successfully, but these errors were encountered:

gyuho · 2018-03-08T01:18:55Z

I can reproduce with 3.2 and 3.3. Will take a look.

hongchaodeng · 2018-03-08T03:01:21Z

This comment is relevant here: coreos/etcd-operator#1727 (comment)

itnikita · 2018-03-09T05:54:47Z

I meet the same issue etcd version 3.3.1
logs shown as follow
Mar 09 13:37:33 master1 etcd[4197]: rejected connection from "192.168.9.186:31833" (error "remote error: tls: bad certificate", ServerName "")

workhardcc · 2018-03-13T12:46:07Z

I have the same issue in 3.3.1

3月 13 12:38:47 172-20-24-117 etcd[9508]: rejected connection from "127.0.0.1:55480" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
3月 13 12:38:47 172-20-24-117 bash[9508]: WARNING: 2018/03/13 12:38:47 Failed to dial 0.0.0.0:4001: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
3月 13 12:38:47 172-20-24-117 bash[9508]: WARNING: 2018/03/13 12:38:47 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
3月 13 12:38:47 172-20-24-117 etcd[9508]: rejected connection from "127.0.0.1:46640" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")

bsctl · 2018-03-14T17:55:31Z

Hello,
I've the same for 3.2.16

-- The start-up result is done. Mar 14 18:48:09 kubem01 etcd[3089]: WARNING: 2018/03/14 18:48:09 Failed to dial 10.101.0.81:2379: connection error: desc = "transport: authentication handshake failed: remote error Mar 14 18:48:09 kubem01 etcd[3089]: WARNING: 2018/03/14 18:48:09 Failed to dial 10.101.0.81:2379: connection error: desc = "transport: authentication handshake failed: remote error [root@kubem01 ~]#

gintautassulskus · 2018-04-17T10:58:57Z

I am having the same issue after updating v3.2.11 to v3.3.3.

tkellen · 2018-05-21T17:27:14Z

Seeing the same with v3.3.5 after debugging coreos/etcd-operator#1962.

tkellen · 2018-05-21T17:34:10Z

Resolved by adding client auth as an extended key usage in my cfssl config as recommended here (and evidently missing based on error output):
https://coreos.com/os/docs/latest/generate-self-signed-certificates.html

manyagoyal12 · 2018-09-28T09:41:32Z

@tkellen What have yopu exactly added to resolve this? I am using etcd version 3.2.17 and getting below error-
Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

pengyao · 2018-10-09T02:51:10Z

I have the same issue in 3.3.9

der-ali · 2018-10-17T12:50:44Z

i have the same issue in 3.3.8 on openshift

KIVagant · 2018-10-23T22:05:41Z

How to fix the issue: #9785 (comment) (add "client auth" to "server" profile in CA config and regenerate server cert).

nejtr0n · 2018-11-02T10:46:57Z

How to fix the issue: #9785 (comment) (add "client auth" to "server" profile in CA config and regenerate server cert).

Not working for me.
CA

cat csr_ROOT_CA.json 
{
 "CN": "dev",
 "key": {
    "algo": "rsa",
    "size": 4096
 },
 "names": [
 {
    "C": "RU",
    "L": "Sauronsk",
    "O": "kubernetes",
    "ST": "Mordor"
 }
 ],
 "ca": {
    "expiry": "8760h"
 }
}

Generate

cfssl gencert -initca csr_ROOT_CA.json | cfssljson -bare root_ca

Intermediate for etcd

cat csr_INTERMEDIATE_CA.json
{
 "CN": "etcd.dev",
 "key": {
    "algo": "rsa",
    "size": 4096
 },
 "names": [
 {
    "C": "RU",
    "L": "Sauronsk",
    "O": "kubernetes",
    "ST": "Mordor"
 }
 ],
 "ca": {
    "expiry": "8760h"
 }
}

Generating intermediate

cfssl gencert -initca csr_INTERMEDIATE_CA.json | cfssljson -bare intermediate_ca

Intermediate sign config

cat root_to_intermediate_ca.json
{ 
"signing": {
 "default": {
 "usages": ["digital signature","cert sign","crl sign","signing"],
 "expiry": "8760h",
 "ca_constraint": {"is_ca": true, "max_path_len":0, "max_path_len_zero": true}
 }
 }

Sign intermediate

cfssl sign -ca ../root_ca.pem -ca-key ../root_ca-key.pem -config root_to_intermediate_ca.json intermediate_ca.csr | cfssljson -bare intermediate_ca

Etcd certificate config

cat csr_END_CA.json
{
 "CN": "etcd.kub1.cloud",
 "hosts": [
   "10.10.10.101", "etcd.kub1.cloud"
 ],
 "key": {
    "algo": "rsa",
    "size": 4096
 },
 "names": [
 {
    "C": "RU",
    "L": "Sauronsk",
    "O": "kubernetes",
    "ST": "Mordor"
 }
 ]
}

Intermediate to end config

cat intermediate_to_end.json                                                   
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "client-server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }

Generating etcd config

cfssl gencert -ca ../intermediate_ca.pem -ca-key ../intermediate_ca-key.pem -config intermediate_to_end.json -profile=client-server csr_END_CA.json | cfssljson -bare etcd

And starting etcd like

cat /etc/systemd/system/etcd.service 
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd

[Service]
User=etcd
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0

ExecStart=/usr/local/bin/etcd --name kub1_etcd \
    --data-dir /var/lib/etcd \
    --client-cert-auth \
    --trusted-ca-file /etc/ssl/certs/etcd/ca.pem \
    --cert-file /etc/ssl/certs/etcd/etcd.pem \
    --key-file /etc/ssl/certs/etcd/etcd-key.pem \
    --peer-client-cert-auth \
    --peer-trusted-ca-file /etc/ssl/certs/etcd/ca.pem \
    --peer-cert-file /etc/ssl/certs/etcd/etcd.pem \
    --peer-key-file /etc/ssl/certs/etcd/etcd-key.pem \
    --listen-client-urls https://10.10.10.101:2379 \
    --advertise-client-urls https://10.10.10.101:2379 \
    --listen-peer-urls https://10.10.10.101:2380 \
    --initial-advertise-peer-urls https://10.10.10.101:2380 \
    --initial-cluster kub1_etcd=https://10.10.10.101:2380,kub2_etcd=https://10.10.10.104:2380 \
    --initial-cluster-token my-etcd-token \
    --initial-cluster-state new

[Install]
WantedBy=multi-user.target

Same config for second node (10.10.10.104)

Erorr log still the same ((

rejected connection from "10.10.10.104:43816" (error "remote error: tls: bad certificate", ServerName "")

hexfusion · 2018-11-02T11:16:41Z

/assign

KIVagant · 2018-11-09T16:29:33Z

@nejtr0n , I don't see "client auth" permission in your config for "server" key:

      "server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },

How to fix the issue: #9785 (comment) (add "client auth" to "server" profile in CA config and regenerate server cert).

nejtr0n · 2018-11-13T10:24:02Z

I reran ansible scripts one more time, and etcd is up with 3.3 version. Dunno how it was solved... Do not change nothng.

2rs2ts · 2019-10-17T19:51:32Z

I didn't have this problem until I upgraded to 3.4. I think the golang upgrade is the cause but if y'all were having problems with 3.3 then I don't think my issues are the same as everyone's here.

BongoEADGC6 · 2019-12-10T15:32:06Z

This is still a problem with no clarity in the documentation for 3.4.

mac119 · 2020-01-22T03:06:35Z

I have the same issue in 3.4.3

stale · 2020-04-21T03:23:37Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

Avatat · 2020-05-31T09:54:31Z

I had the same issue in v3.4.9.
I resolved it by adding clientAuth (TLS Web Client Authentication) to ETCD server certificate (used in ETCD_CERT_FILE).
I'm not sure if, and why a server should have clientAuth flag in its certificate...

povsister · 2021-01-25T08:27:43Z

I think this is probably a bug.
Serving certs do not need clientAuth extend keyUsage because it is used as server certificate and should be authed by clients.
I will check it later.

XANi · 2021-06-04T15:36:05Z

I guess this ticket is another example of https://blog.benwinding.com/github-stale-bots/

please reopen it, still happens in latest stable

gyuho added the area/tls label Mar 7, 2018

gyuho self-assigned this Mar 8, 2018

raoofm mentioned this issue Jul 5, 2018

etcd: full TLS support on client, peer and operator level bank-vaults/bank-vaults#81

Merged

gyuho-bot assigned hexfusion Nov 2, 2018

anson627 mentioned this issue Mar 21, 2020

example: add client auth to server profile in CA config coreos/etcd-operator#2165

Open

discostur mentioned this issue Apr 8, 2020

etcd 3.3: (error "remote error: tls: bad certificate", ServerName "") #11770

Closed

stale bot added the stale label Apr 21, 2020

stale bot closed this as completed May 12, 2020

seanmwinn mentioned this issue Jun 23, 2020

Unable to reach service backend running on remote node from host network cilium/cilium#12244

Closed

spacewander mentioned this issue Jun 9, 2021

request help: build OpenResty for APISIX apache/apisix#4383

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.2/3.3 etcd server with TLS would start with error "tls: bad certificate" #9398

3.2/3.3 etcd server with TLS would start with error "tls: bad certificate" #9398

hongchaodeng commented Mar 7, 2018 •

edited

Loading

gyuho commented Mar 8, 2018

hongchaodeng commented Mar 8, 2018

itnikita commented Mar 9, 2018

workhardcc commented Mar 13, 2018

bsctl commented Mar 14, 2018

gintautassulskus commented Apr 17, 2018

tkellen commented May 21, 2018

tkellen commented May 21, 2018

manyagoyal12 commented Sep 28, 2018

pengyao commented Oct 9, 2018

der-ali commented Oct 17, 2018

KIVagant commented Oct 23, 2018 •

edited

Loading

nejtr0n commented Nov 2, 2018

hexfusion commented Nov 2, 2018

KIVagant commented Nov 9, 2018

nejtr0n commented Nov 13, 2018

2rs2ts commented Oct 17, 2019

BongoEADGC6 commented Dec 10, 2019

mac119 commented Jan 22, 2020

stale bot commented Apr 21, 2020

Avatat commented May 31, 2020

povsister commented Jan 25, 2021

XANi commented Jun 4, 2021

3.2/3.3 etcd server with TLS would start with error "tls: bad certificate" #9398

3.2/3.3 etcd server with TLS would start with error "tls: bad certificate" #9398

Comments

hongchaodeng commented Mar 7, 2018 • edited Loading

gyuho commented Mar 8, 2018

hongchaodeng commented Mar 8, 2018

itnikita commented Mar 9, 2018

workhardcc commented Mar 13, 2018

bsctl commented Mar 14, 2018

gintautassulskus commented Apr 17, 2018

tkellen commented May 21, 2018

tkellen commented May 21, 2018

manyagoyal12 commented Sep 28, 2018

pengyao commented Oct 9, 2018

der-ali commented Oct 17, 2018

KIVagant commented Oct 23, 2018 • edited Loading

nejtr0n commented Nov 2, 2018

hexfusion commented Nov 2, 2018

KIVagant commented Nov 9, 2018

nejtr0n commented Nov 13, 2018

2rs2ts commented Oct 17, 2019

BongoEADGC6 commented Dec 10, 2019

mac119 commented Jan 22, 2020

stale bot commented Apr 21, 2020

Avatat commented May 31, 2020

povsister commented Jan 25, 2021

XANi commented Jun 4, 2021

hongchaodeng commented Mar 7, 2018 •

edited

Loading

KIVagant commented Oct 23, 2018 •

edited

Loading