cannot turn off client cert auth in etcd using v0.5.2 and v3.1.10 #1416

raoofm · 2017-09-19T20:14:13Z

I'm unable to access etcd via LoadBalancer after enabling tls. I thought this might be due to client-cert-auth=true flag being set. So I tried to disable client auth but the flag is being ignored by etcd as they have logic in place that ignores flag if already set.

2017-09-19 20:01:52.520944 I | pkg/flags: recognized environment variable ETCD_CLIENT_CERT_AUTH, but unused: shadowed by corresponding flag

https://github.com/coreos/etcd/blob/f1509a102c192035a5467304b34f1226e100d88a/pkg/flags/flag.go#L119

if alreadySet[kv[0]] {
			// TODO: exit with error in v3.4
			plog.Warningf("recognized environment variable %s, but unused: shadowed by corresponding flag", kv[0])
			continue
		}
		if strings.HasPrefix(env, prefix+"_") {
			plog.Warningf("unrecognized environment variable %s", env)
		}

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.etcdCluster.name }}-client-service-lb
  labels:
    app: "{{ template "name" . }}"
    prometheus-app: "{{ template "name" . }}"
    dns: elb-route53
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
    dns.dev.io/domainName: {{ .Values.dns | quote }}
{{- if .Values.httpsEnable }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values.awsCert | quote }}
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 2379
{{- end }}
spec:
  selector:
#    app: etcd
    etcd_cluster: {{ .Values.etcdCluster.name }}
  ports:
  - name: client
    port: 2379
    targetPort: 2379
    protocol: TCP
  type: "LoadBalancer"

The text was updated successfully, but these errors were encountered:

hongchaodeng · 2017-09-19T20:19:10Z

@raoofm
How does the deployment look like? Why do you want to turn off client-auth ? If this is valid case, we might provide an option to disable client-auth.

raoofm · 2017-09-19T20:25:56Z

How does the deployment look like?

I'm not sure what you need. Do you want me to paste the yaml files.

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "name" . }}
  labels:
    app: {{ template "name" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version }}
    heritage: {{ .Release.Service }}
    release: {{ .Release.Name }}
    operator: etcd
spec:
  replicas: {{ .Values.replicaCount }}
  template:
    metadata:
      labels:
        app: {{ template "name" . }}
        name: {{ template "name" . }}
        chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
        heritage: "{{ .Release.Service }}"
        release: "{{ .Release.Name }}"
        operator: etcd
    spec:
      containers:
      - name: {{ template "name" . }}
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: "{{ .Values.image.pullPolicy }}"
        command:
        - "/usr/local/bin/etcd-operator"
        - "--pv-provisioner={{ .Values.etcdCluster.backup.provisioner }}"
        {{- range $key, $value := .Values.commandArgs }}
        - "--{{ $key }}={{ $value }}"
        {{- end }}
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        resources:
{{ toYaml .Values.resources | indent 10 }}
{{- if .Values.nodeSelector }}
      nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.rbacEnable }}
      serviceAccountName: {{ template "name" . }}
{{- end }}

Why do you want to turn off client-auth ?

@hongchaodeng As I need to talk to cluster from any machine using etcdctl or curl to add auth/kv

raoofm · 2017-09-19T20:29:54Z

I mean if the etcd cluster is just being used for kubernetes then it makes sense to have only clients with proper certs to talk to it.

What if any microservice which doesn't have any certs but just needs to talk to the load balancer on https.

raoofm · 2017-09-19T20:30:37Z

my setup is as described here

hongchaodeng · 2017-09-19T20:38:07Z

So by deployment I mean how do you use etcd cluster. From the context I saw that apps are talking to etcd cluster via a LB. Is it network (L4) load balancer, right?

hongchaodeng · 2017-09-19T21:01:28Z

What if any microservice which doesn't have any certs but just needs to talk to the load balancer on https.

Without certs, this will only provide transport security, not authenticity.
I don't know any use case for it.. Otherwise, "auto-tls" in etcd would also work and is much easier solution for no certs needed from users.

raoofm · 2017-09-19T21:51:51Z

@hongchaodeng As I need to talk to cluster from any machine using etcdctl or curl to add auth/kv

@hongchaodeng how about the above use case, say for maintainence

From the context I saw that apps are talking to etcd cluster via a LB. Is it network (L4) load balancer, right?

Yes

Without certs, this will only provide transport security, not authenticity.

etcd has v3 auth to talk to for authentication.

raoofm · 2017-09-19T21:56:31Z

i need a public facing aws load balancer for the apps to talk to etcd

hongchaodeng · 2017-09-19T22:12:36Z

Yeah. Your use case sounds good. I'm making sure it is clear here.

So you want to disable "client-cert-auth", right? Ref: etcd docs.

In your expected deployment, it has:

key, cert in etcd server
CA cert in etcd client. This actually confuses me. Because you already need to distribute CA cert, why not give key/cert anyway?

raoofm · 2017-09-19T22:13:00Z

@hongchaodeng the problem is after creating the load balancer I'm unable to call the endpoint with curl or etcdctl (using the client certs that were generated for operator)

so the lb is a layer4 and it just forwards the tcp traffic

What is the intention here, the clients should call lb using http or https. That gets forwarded to etcd-dev-cluster-client-service-lb.etcd:2379 TCP

When I used etcdctl with etcd-dev-cluster-client-service-lb.etcd:2379 endpoint from the etcd-operator pod (using sh) then it worked but not publicly.

hongchaodeng · 2017-09-19T22:22:28Z

the clients should call lb using http or https

Did you add the LB domain name into the SAN of the cert? Check by:

openssl x509 -in <etcd-server.crt> -text -noout

The address that etcd client (e.g. etcdctl) talks to is checked against with the server cert's SAN.
ELB is L4. It wouldn't understand https, but just forward packets.

raoofm · 2017-09-19T22:23:01Z

In your expected deployment, it has:
key, cert in etcd server
CA cert in etcd client. This actually confuses me. Because you already need to distribute CA cert, why not give key/cert anyway?

So if I understand correctly, you mean to distribute etcd-client-crt, etcd-client-key, etcd-client-ca (ref https://github.com/coreos/etcd-operator/blob/master/doc/user/cluster_tls.md#operatorsecret) to all the clients that need to talk to etcd

Thats what I tried so below works from a pod, when I sh into one of the etcd-cluster pods but not from outside

/usr/local/bin # etcdctl --endpoints=etcd-dev-cluster-0000.etcd-dev-cluster.etcd.svc:2379,etcd-dev-cluster-0001.etcd-dev-cluster.etcd.svc:2379,etcd-dev-cluster-0002.etcd-dev-cluster.etcd.svc:2379,etcd-dev-cluster-client-service-lb.etcd:2379,etcd-dev-cluster.etcd:2379,etcd-dev-cluster-client.etcd:2379  --cacert=/etc/etcdtls/operator/etcd-tls/etcd-client-ca.crt --cert=/etc/etcdtls/operator/etcd-tls/etcd-client.crt --key=/etc/etcdtls/operator/etcd-tls/etcd-client.key endpoint health
2017-09-19 18:25:24.613285 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
etcd-dev-cluster.etcd:2379 is healthy: successfully committed proposal: took = 18.906649ms
etcd-dev-cluster-client.etcd:2379 is healthy: successfully committed proposal: took = 115.415855ms
etcd-dev-cluster-0000.etcd-dev-cluster.etcd.svc:2379 is healthy: successfully committed proposal: took = 6.966288ms
etcd-dev-cluster-0002.etcd-dev-cluster.etcd.svc:2379 is healthy: successfully committed proposal: took = 7.30715ms
etcd-dev-cluster-0001.etcd-dev-cluster.etcd.svc:2379 is healthy: successfully committed proposal: took = 11.614153ms
etcd-dev-cluster-client-service-lb.etcd:2379 is healthy: successfully committed proposal: took = 1.79915ms
/usr/local/bin #

It doesn't work from outside when I run below

$ ./etcdctl --endpoints=ac28c1dc79d8511e7ba3b12a04e6fcc4-2119000283.us-east-1.elb.amazonaws.com:2379 --cacert=tls/client-ca.crt --cert=tls/client.crt --key=tls/client.key endpoint health
2017-09-19 22:22:10.478543 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
ac28c1dc79d8511e7ba3b12a04e6fcc4-2119000283.us-east-1.elb.amazonaws.com:2379 is unhealthy: failed to connect: grpc: timed out when dialing

hongchaodeng · 2017-09-19T22:37:01Z

--endpoints=ac28c1dc79d8511e7ba3b12a04e6fcc4-2119000283.us-east-1.elb.amazonaws.com:2379

So this is what I mean "the address etcd client talks to".
First it should be "https://". Second, is this address in the SAN of etcd server cert?

raoofm · 2017-09-19T22:46:58Z

@hongchaodeng awesome. So as the lb name is dynamic, i created a rout53 entry for it and added the name to the SAN.

etcdctl worked fine

 ./etcdctl --endpoints=https://etcd.ctnrva0.dev.example.net:2379 --cacert=tls/client-ca.crt --cert=tls/client.crt --key=tls/client.key endpoint health
2017-09-19 22:38:40.411752 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
https://etcd.ctnrva0.dev.example.net:2379 is healthy: successfully committed proposal: took = 8.133843ms

👍

On a separate note, can you consider the usecase of disabling client-cert-auth? (In case anyone wants to configure it that way)

hongchaodeng · 2017-09-19T23:07:03Z

Well. It's easy to add new features but it's hard to maintain. That's why we need to justify when we add new features.

I am not sure that disabling client auth would simplify things since CA cert still needs to be distributed. So that's like giving up a part of security but no gaining.. Maybe there are use cases I don't know. But so far that is not a wise choice.

hongchaodeng · 2017-09-19T23:14:13Z

And actually disabling client auth won't solve your previous problem.. this issue should be renamed to about TLS setting. One improvement would be that server side returns better message like your address is not in SAN.

raoofm · 2017-09-20T15:12:04Z

I am not sure that disabling client auth would simplify things since CA cert still needs to be distributed.

I'm not sure if I understand this. If CA is trusted by everyone say Verisign/Digicert then why do we need to distribute anything. The tls would work as is without client cert auth. (Encrypted connection without auth, auth will be done by etcd v3 auth)

hongchaodeng · 2017-09-20T16:10:30Z

I see. We usually use self signed CA.
Just curious. Doesn't it have cost to get a domain name trusted by public CA like Verisign?
Why do you prefer that instead of self signed? Also what about certs for testing?

raoofm · 2017-09-20T16:20:29Z

@hongchaodeng agree, I'm as well using self signed certs, I was just presenting a usecase.

In my case even though I'm using self signed, I can impose the CA to be copied to every machines trust store (/etc/ssl/certs/)

We can argue that the client certs can be copied over as well, but I want only the communication to be secure, I can be free to choose the auth channel or protection(clientCert/userPass/securityGroups/cors)

raoofm · 2017-09-20T16:23:00Z

@hongchaodeng @xiang90 the other challenge that we face is if we share the same client cert with all the clients which as per example is carrying the CN as client. The v3 auth says that if client-cert-auth is enabled and client is presenting the cert then CN will be used as user and it needs no pass. But we want to configure role based access where each user has access to limited keys. If we use client-cert-auth then we need to generate certs for each user.

hongchaodeng · 2017-09-20T20:12:21Z

@raoofm
If you enable auth etcd operator will not work as today...
Health check will fail:

2m        2m        1         example-vault-etcd-0002   Pod       spec.containers{etcd}   Warning   Unhealthy   kubelet, ip-10-0-37-188.us-west-1.compute.internal   Liveness probe failed: 2017-09-20 20:07:02.139183 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
Error:  etcdserver: user name is empty

Auth support should be done first before addressing the CN name as user name issue.

ref: #1051

raoofm · 2017-09-21T14:02:26Z

@hongchaodeng I mentioned a comment here

I think both are very important features. v3 auth is a must to have and addressing the problem of CN as user, which should be optional.

raoofm · 2017-10-27T18:34:05Z

@hongchaodeng The second issue of CN as user is fixed in etcd. After the disccusion etcd team agreed that user when passed in basic auth should take priority over CN as username. Check here etcd-io/etcd#8584.

Now we just need to take care of enabling v3 auth as in #1051

raoofm · 2017-10-27T18:36:05Z

also the ability to turn of client-cert-auth remains open too along with enabling v3 auth

hongchaodeng · 2017-12-20T21:23:30Z

Part of the issue is resolved in etcd.
Let's handle the rest and converge it into #1051

raoofm · 2017-12-20T21:50:03Z

@hongchaodeng enabling v3 basic auth and option to disable client-cert-auth are two different issues.

I think #1416 and #1051 are two issues. Unless you want to copy this over there.

hongchaodeng · 2017-12-20T21:50:45Z

Yeah.. Sorry my mistake.

hongchaodeng · 2018-01-19T19:33:15Z

Note that "username over cert CN" feature in etcd is available after https://github.com/coreos/etcd/releases/tag/v3.3.0-rc.3

raoofm · 2018-01-19T19:38:17Z

@hongchaodeng does that mean we should start looking at #1731

lynic · 2018-04-16T03:33:56Z

I have the same requirements, would like etcd-operator to use cert to do livenessprobe, and use user/pass auth for each clients, which means I don't want to set "client-cert-auth" to true so that I don't need to generate client certs for each client.

schlitzered · 2019-11-22T10:01:11Z

i also also had the need to disable client cert auth, and it seems like you actually have to unset these two options:

--trusted-ca-file
--client-cert-auth

just removing client-cert-auth is not enough, you have to also remove the trusted-ca-file option.

raoofm mentioned this issue Oct 27, 2017

update milestone of open issues marked 0.6.1 to a future or 0.6.2 #1590

Closed

hongchaodeng closed this as completed Dec 20, 2017

hongchaodeng reopened this Dec 20, 2017

hongchaodeng added the priority/P3 label Dec 31, 2017

raoofm mentioned this issue Jan 19, 2018

enabling auth not supported #1051

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cannot turn off client cert auth in etcd using v0.5.2 and v3.1.10 #1416

cannot turn off client cert auth in etcd using v0.5.2 and v3.1.10 #1416

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017 •

edited

Loading

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017 •

edited

Loading

raoofm commented Sep 20, 2017

hongchaodeng commented Sep 20, 2017

raoofm commented Sep 20, 2017

raoofm commented Sep 20, 2017

hongchaodeng commented Sep 20, 2017

raoofm commented Sep 21, 2017

raoofm commented Oct 27, 2017

raoofm commented Oct 27, 2017

hongchaodeng commented Dec 20, 2017

raoofm commented Dec 20, 2017

hongchaodeng commented Dec 20, 2017

hongchaodeng commented Jan 19, 2018

raoofm commented Jan 19, 2018

lynic commented Apr 16, 2018

schlitzered commented Nov 22, 2019

cannot turn off client cert auth in etcd using v0.5.2 and v3.1.10 #1416

cannot turn off client cert auth in etcd using v0.5.2 and v3.1.10 #1416

Comments

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017 • edited Loading

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

raoofm commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017

hongchaodeng commented Sep 19, 2017 • edited Loading

raoofm commented Sep 20, 2017

hongchaodeng commented Sep 20, 2017

raoofm commented Sep 20, 2017

raoofm commented Sep 20, 2017

hongchaodeng commented Sep 20, 2017

raoofm commented Sep 21, 2017

raoofm commented Oct 27, 2017

raoofm commented Oct 27, 2017

hongchaodeng commented Dec 20, 2017

raoofm commented Dec 20, 2017

hongchaodeng commented Dec 20, 2017

hongchaodeng commented Jan 19, 2018

raoofm commented Jan 19, 2018

lynic commented Apr 16, 2018

schlitzered commented Nov 22, 2019

hongchaodeng commented Sep 19, 2017 •

edited

Loading

hongchaodeng commented Sep 19, 2017 •

edited

Loading