Getting public-cert is failing with an interactive prompt #183

alexellis · 2019-07-16T08:44:06Z

I'm using a KUBECONFIG file pointing at DigitalOcean Kubernetes and I'm getting this odd behaviour.

# Wait for the controller to come up
kubectl rollout status deploy -n kube-system ofc-sealedsecrets-sealed-secrets

./kubeseal --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets

It gives this text, then blocks indefinitely

Please enter Username:

I am not sure what's going wrong. Has there been a recent update to the binary release? Did something get nudged, or could it be a problem with the latest k8s version?

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTr
eeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTr
eeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Thanks in advance for your assistance

The text was updated successfully, but these errors were encountered:

alexellis · 2019-07-16T08:54:44Z

I've also tried deleting my .kube/config file and reinstating it with the DigitalOcean CLI, and rolling back to v0.6.0 of the SealedSecrets CLI.

alexellis · 2019-07-16T08:56:20Z

kubectl get deploy -n kube-system ofc-sealedsecrets-sealed-secrets
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
ofc-sealedsecrets-sealed-secrets   1/1     1            1           18m

^ The controller definitely exists and is in the expected namespace.

The server-side image is quay.io/bitnami/sealed-secrets-controller:v0.7.0

mkmik · 2019-07-16T12:13:16Z

I am not sure what's going wrong. Has there been a recent update to the binary release?

no; this is a very old release; I plan to release 0.9.0 soon FWIW.

anyway, if --fetch-cert you can pick up the certificate manually. If you can see the logs of the sealed-secrets pod, it's emitted right at the beginning. With that you can use kubeseal offline by passing the certificate with the --cert flag.

alexellis · 2019-07-16T12:16:17Z

This seems like bug / regression. We've used that behaviour in ofc-bootstrap for some time and it saves on Kubectl/bash commands.

Can you suggest why kubeseal might be getting confused?

mkmik · 2019-07-16T12:33:58Z

not from the top of my head; 0.7.0 is quite old and possibly it uses some old k8s client libs.

Could you try a recent build of the kubeseal client (e.g. from sources or perhaps binaries of the 0.8.0 draft release?)

alexellis · 2019-07-16T17:37:23Z

I am not sure why it would stop working for me. Could you verify that it's still working as expected for you, on your cluster?

alexellis · 2019-07-16T17:52:48Z

Here is a quick test, that has been working well since about May last by following the latest release:

#!/bin/bash

helm install --namespace kube-system --name ofc-sealedsecrets stable/sealed-secrets

if [ ! -f kubeseal ];
then
    GOOS=$(go env GOOS)
    GOARCH=$(go env GOARCH)

    release=$(curl -sI https://github.com/bitnami-labs/sealed-secrets/releases/latest | grep Location | awk -F"/" '{ printf "%s", $NF }' | tr -d '\r')

#    release=$(curl --silent "https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest" | sed -n 's/.*"tag_name": *"\([^"]*\)".*/\1/p')
    echo "SealedSecrets release: $release"

    curl -sLSf https://github.com/bitnami/sealed-secrets/releases/download/$release/kubeseal-$GOOS-$GOARCH > kubeseal && \
    chmod +x kubeseal
fi

./kubeseal --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets

Does this hang for you guys?

mkmik · 2019-07-17T07:05:37Z

Could you try a recent build of the kubeseal client (e.g. from sources or perhaps binaries of the 0.8.0 draft release?)

@alexellis just to doublecheck: did you try a more recent client build?

In the meantime I'm going to try to reproduce this issue.

mkmik · 2019-07-17T07:14:51Z

I confirm that following the official installation instructions (https://github.com/bitnami-labs/sealed-secrets/tree/v0.7.0#installation) the following works:

./kubeseal --fetch-cert --controller-name=sealed-secrets-controller

(Keep in mind that the helm chart is maintained by the community)

mkmik · 2019-07-17T07:39:40Z

I also tried the helm install method, on a GKE cluster (v1.13.7-gke.8):

$ helm install --namespace kube-system --name ofc-sealedsecrets stable/sealed-secrets
NAME:   ofc-sealedsecrets
LAST DEPLOYED: Wed Jul 17 09:25:53 2019
NAMESPACE: kube-system
STATUS: DEPLOYED
....
$ ./kubeseal --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets
-----BEGIN CERTIFICATE-----
MIIErjCCApagAwIBAgIRAMEuFK5H6U....

@alexellis could you please share the output of:

$ kubectl -n kube-system logs -l app.kubernetes.io/name=sealed-secrets

$  kubectl -n kube-system get all -l app.kubernetes.io/name=sealed-secrets

In the meantime, I'll try on DigitalOcean Kubernetes (I guess I found a good excuse to play with it) :-)

mkmik · 2019-07-17T08:16:36Z

I just deployed a DigitalOcean Kubernetes cluster (v1.14.3), installed sealed-secrets using the helm chart (helm install --namespace kube-system --name ofc-sealedsecrets stable/sealed-secrets)

and:

$ ./kubeseal --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets
-----BEGIN CERTIFICATE-----
MIIErjCCApagAwIBAgIRAMhYPyO/MGH4+WSZWMDebcswDQYJKoZIhvcNAQELB...

I used this binary: https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.7.0/kubeseal-darwin-amd64

alexellis · 2019-07-17T09:00:45Z

kubectl -n kube-system logs -l app.kubernetes.io/name=sealed-secrets
8QoDrtACANov8sGtobyt8CctSzpodiBKGJlN+Is+ST4zb/QMp6/O3MSsBKXHBHha
2mQL01RHvczlstF+P8dDtCizcFP0fuEdbkY11/tnmBLLvymJcMdtKouduNTRIVlG
Hbm3RpCIU2N95LRPs2XEuUS66Y0pZeZmg9HnpIq87q8PHZtXJs+bU2C7u75Jeu5S
v+d6kiIOxMXPHtN2xcB7cPJAonWZyoIvdJp1eANt5JQDq1LkNaYtnySnxFBti9dL
qXHLeyHwDnEItypvfd7Tkzc4b8JYIQt2mdr2Br+ki8MsugcilTNFtd97xOZJslGo
AmlSYWZCbSZiOdxkkDCton+146pC6JVwnCgheVgAt25CPhwNQd/52qn67mGPW//h
AQ==
-----END CERTIFICATE-----

2019/07/17 08:59:42 HTTP server serving on :8080
alext:ofc-bootstrap alex$ kubectl -n kube-system get all -l app.kubernetes.io/name=sealed-secrets
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/ofc-sealedsecrets-sealed-secrets-6f4db57b4b-hsj79   1/1     Running   0          12s

NAME                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/ofc-sealedsecrets-sealed-secrets   ClusterIP   10.245.164.184   <none>        8080/TCP   12s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ofc-sealedsecrets-sealed-secrets   1/1     1            1           12s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/ofc-sealedsecrets-sealed-secrets-6f4db57b4b   1         1         1       12s
alext:ofc-bootstrap alex$

alexellis · 2019-07-17T09:01:19Z

alext:ofc-bootstrap alex$ ./kubeseal-darwin-amd64 --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets
Please enter Username: 
^C
alext:ofc-bootstrap alex$ 
alext:ofc-bootstrap alex$ kubectl get deploy -n kube-system ofc-sealedsecrets-sealed-secrets
NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
ofc-sealedsecrets-sealed-secrets   1/1     1            1           92s
alext:ofc-bootstrap alex$

alexellis · 2019-07-17T09:03:03Z

Here's me removing my kubeconfig and regenerating it:

alext:ofc-bootstrap alex$ mv ~/.kube ~/.kube-old
alext:ofc-bootstrap alex$ doctl k8s cluster kubeconfig save ofc-auth-plugin-test-2
Notice: adding cluster credentials to kubeconfig file found in "/Users/alex/.kube/config"
alext:ofc-bootstrap alex$ ./kubeseal-darwin-amd64 --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets
panic: invalid configuration: no configuration has been provided

goroutine 1 [running]:
main.main()
	/Users/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:216 +0x3b9
alext:ofc-bootstrap alex$ kubectx
do-lon1-ofc-auth-plugin-test-2
alext:ofc-bootstrap alex$ kubectx do-lon1-ofc-auth-plugin-test-2
Switched to context "do-lon1-ofc-auth-plugin-test-2".
alext:ofc-bootstrap alex$ ./kubeseal-darwin-amd64 --fetch-cert --controller-name=ofc-sealedsecrets-sealed-secrets
Please enter Username: 

^C
alext:ofc-bootstrap alex$

It's like the .kube/config can't be read / parsed properly?

alexellis · 2019-07-17T09:03:56Z

@mkmik thanks a lot for helping with this. @LucasRoesler also had this working with KinD. I've added some more debug / info.

mkmik · 2019-07-17T09:13:37Z

It's like the .kube/config can't be read / parsed properly?

I tired with the standard digital ocean yaml kubeconfig yaml file (which contains a client-certificate-data etc)

Are you using some advanced config mechanism?
Could you please share the non-secret bits of the .kube/config?

DefrostedTuna · 2019-07-17T21:57:54Z

I also ran into this issue trying to get everything set up on DigitalOcean. After hammering at it for the evening last night, I may have figured out what's going on. When creating a cluster via the command line, DigitalOcean saves the created cluster's config automatically. This can also be done manually for existing clusters by running doctl kubernetes cluster kubeconfig save $CLUSTER_ID.

The problem seems to lie in the format that DigitalOcean saves the config via the command line. Taking a look at the ~/.kube/config downloaded via the command line, and the ~/.kube/config downloaded via the DigitalOcean control panel shows that there are a few differences.

Here's an example of a config file retrieved via the command line:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [REMOVED]
    server: [REMOVED]
  name: do-nyc1-lordran-dev-2
contexts:
- context:
    cluster: do-nyc1-lordran-dev-2
    user: do-nyc1-lordran-dev-2-admin
  name: do-nyc1-lordran-dev-2
current-context: do-nyc1-lordran-dev-2
kind: Config
preferences: {}
users:
- name: do-nyc1-lordran-dev-2-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - kubernetes
      - cluster
      - kubeconfig
      - exec-credential
      - --version=v1beta1
      - [REFERENCE_TO_SERVER_REMOVED]
      command: doctl
      env: null

Here's a copy that was downloaded via the DigitalOcean control panel:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [REMOVED]
    server: [REMOVED]
  name: do-nyc1-lordran-dev-2
contexts:
- context:
    cluster: do-nyc1-lordran-dev-2
    user: do-nyc1-lordran-dev-2-admin
  name: do-nyc1-lordran-dev-2
current-context: do-nyc1-lordran-dev-2
kind: Config
preferences: {}
users:
- name: do-nyc1-lordran-dev-2-admin
  user:
    client-certificate-data: [REMOVED]
    client-key-data: [REMOVED]

The config file downloaded via the control panel has the client-certificate-data and client-key-data nested under the users key. It seems as though kubeseal is using these credentials to access the cluster. When I swapped my config file to use this copy, I was able to get through to the cluster without being prompted for a username or password.

Hope this helps!

mkmik · 2019-07-18T09:04:29Z

yeah, I didn't use `doctl kubernetes cluster kubeconfig save to save the config.

Turns out that doctl kubernetes cluster kubeconfig save and doctl kubernetes cluster kubeconfig show generate a different config, the former using the exec mechanism; this way it leverages the doctl access token (which I assume doesn't expire like the TLS client certificate).

I assume DO opted for this (a bit surprising) behaviour on the grounds that show (or downloaded yaml) might be manually merged in environments that lack the doctl tool.

I can now reproduce the issue with v0.7.0.
With HEAD, that uses a more recent k8s client library the error message is a bit more informative:

exec plugin: invalid apiVersion "client.authentication.k8s.io/v1beta1"

Minor API changes (some helpers functions disappeared upstream probably because they were deemed to be trivial to write explicitly). Sore point: a roundtrip test is broken since the reflection based test trips over some non-semantic changes while comparing empty slices (nil != []foo{} in Go) Will address that ASAP but I'd like to unblock this in the meantime. Closes #183

188: Upgrade client-go to v12.0.0 r=mkmik a=mkmik Minor API changes (some helpers functions disappeared upstream probably because they were deemed to be trivial to write explicitly). Sore point: a roundtrip test is broken since the reflection based test trips over some non-semantic changes while comparing empty slices (nil != []foo{} in Go) Will address that ASAP but I'd like to unblock this in the meantime. Closes #183 Co-authored-by: Marko Mikulicic <[email protected]>

alexellis · 2019-07-25T20:52:45Z

Thank you Marko 👍

mkmik · 2019-07-25T20:57:41Z

@alexellis you can use the kubeseal binary released in https://github.com/bitnami-labs/sealed-secrets/releases/tag/v0.8.1

(keep in mind that the helm chart for v0.8.x is not yet ready, tracked in helm/charts#15837; but you can use the new client with the v0.7.0 controller)

alexellis · 2019-08-14T17:29:25Z

@mkmik I'll use the workaround for now and have documented it on the issue.

When do you expect a stable binary release and helm chart bump to be available? I'll just update the docs when these are ready.

mkmik · 2019-08-14T18:11:55Z

Uh, in surprised. I thought it had been fixed in the v0.8.1 release (mentioned in the changelog for https://github.com/bitnami-labs/sealed-secrets/releases/tag/v0.8.1: also this issue gas been tagged with the v0.8.1 milestone).

Let me know if you still have problems

mkmik mentioned this issue Jul 18, 2019

Upgrade client-go to v12.0.0 #188

Merged

mkmik added this to the v0.9.0 milestone Jul 18, 2019

mkmik assigned mkmik and unassigned mkmik Jul 19, 2019

mkmik removed this from the v0.9.0 milestone Jul 25, 2019

mkmik added this to the v0.8.1 milestone Jul 25, 2019

bors bot closed this as completed in #188 Jul 25, 2019

alexellis mentioned this issue Aug 14, 2019

SealedSecrets step failing on DigitalOcean with doctl openfaas/ofc-bootstrap#119

Closed

alexellis mentioned this issue Aug 27, 2019

rio install - fails to install on DigitalOcean Kubernetes 1.14 rancher/rio#302

Closed

mkmik mentioned this issue Oct 2, 2019

Periodic loss of secrets #83

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting public-cert is failing with an interactive prompt #183

Getting public-cert is failing with an interactive prompt #183

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 16, 2019

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

mkmik commented Jul 17, 2019

DefrostedTuna commented Jul 17, 2019

mkmik commented Jul 18, 2019

alexellis commented Jul 25, 2019

mkmik commented Jul 25, 2019 •

edited

Loading

alexellis commented Aug 14, 2019

mkmik commented Aug 14, 2019

Getting public-cert is failing with an interactive prompt #183

Getting public-cert is failing with an interactive prompt #183

Comments

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 16, 2019

alexellis commented Jul 16, 2019

alexellis commented Jul 16, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

mkmik commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

alexellis commented Jul 17, 2019

mkmik commented Jul 17, 2019

DefrostedTuna commented Jul 17, 2019

mkmik commented Jul 18, 2019

alexellis commented Jul 25, 2019

mkmik commented Jul 25, 2019 • edited Loading

alexellis commented Aug 14, 2019

mkmik commented Aug 14, 2019

mkmik commented Jul 25, 2019 •

edited

Loading