Document how to add corporate root SSL certificates

[isuftin]

This is a question/feature request.

I am currently operating minikube on a network that requires all SSL traffic to be intercepted using a root SSL certificate. Minikube is using the Virtualbox driver.

Using docker-machine and boot2docker, I was able to add my root certificate to I need to be able to append my SSL root cert to the /var/lib/boot2docker/certs directory. This does not work with the minikube ISO. What does work is appending my certificate to /etc/ssl/certs/ca-certificates.crt

I create a bootlocal.sh for this. This is the command I run from my host:

read -d '' String <<"EOF"
#!/bin/bash
grep -q '<single line from root cert>' /etc/ssl/certs/ca-certificates.crt
EXIT_CODE=\\$?
if [ \\$EXIT_CODE -ne 0 ]; then
  curl http://<rootcert location>/root.cer | sudo tee -a /etc/ssl/certs/ca-certificates.crt
  sudo systemctl restart docker.service
fi
EOF
echo "echo "\""${String}"\"" | sudo tee -a /var/lib/boot2docker/bootlocal.sh && sudo chmod +x /var/lib/boot2docker/bootlocal.sh && /var/lib/boot2docker/bootlocal.sh" | minikube ssh 

This works fine until I restart the VM. It looks like what's happening is that bootlocal may be running first or not at all because when the machine restarts, ca-certificates.crt is back to the original state without my certificate in it.

How should I add my root SSL cert into the VM so Docker can pull from registries?

minikube version: v0.18.0
OS: OSX 10.10.5
Drivername: Virtualbox
ISO: minikube-v0.18.0.iso
Install Tools: homebrew

What happened:

Corporate root SSL certificate does not exist on the VM and is not picked up by the Docker daemon

What you expected to happen:

I expect to be able to add my root SSL certificate to the VM and have Docker be able to pick it up to validate TLS communication

How to reproduce it (as minimally and precisely as possible):

$ minikube start

< run above commandset to pull down any SSL cert >

$ minikube stop && minikube start

< verify that SSL cert does not exist in ca-certificates.crt >

[r2d4]

I'm not exactly sure why this method isn't working, since it seems like it should. Either way, we should expose something similar to /var/lib/boot2docker/certs to auto load self-signed certs.

[isuftin]

@r2d4 Are you able to tell whether bootlocal.sh runs before or after the certificate configuration happens? If before, there's the issue.

I'd love to be able to just dump my cert into /var/lib/boot2docker/certs and call it a day like I do with docker-machine

[isuftin]

@r2d4 The other issue would be that /var/lib/boot2docker/certs gets wiped on VM reboot, so that would need to be persisted or maybe copied in from /data/certs or /Users/[...]. Having it persist would be preferable.

[calaniz]

The latest minikube VM isn't actually based off of boot2docker any more. You can find the buildroot documentation online.

What I did was add the appropriate crt to both the /etc/ssl/certs directory and the /etc/ssl/certs/ca-certificates.crt by adding them to the overlay in the current repo. You can find it at deploy/iso/minikube-iso/board/coreos/minikube/rootfs-overlay/etc/. Create a ssl directory and drop your crts in.

[isuftin]

@calaniz I will give this a shot.

[jwflory]

For what it's worth, I tried following the advice of @calaniz above, but it didn't work for me on boot. I dropped my certs into /etc/ssl/certs/ and appended them to ca-certificates.crt, but the system isn't picking them up by default, e.g. running curl https://www.google.com. If I specify the path manually (--cacert /etc/ssl/certs/ca-certificates.crt), then it works as expected.

Trying to figure out a way to get this to work in Buildroot.

[jwflory]

Hmm, so I tried doing this again today and I think there might have been an issue with the certs I had on my local system. Reinstalling the ca-certificates package on my own system, rebuilding the ISO with the Makefiles, and running it in Minikube solved this issue. So it seems like @calaniz's solution also works for me after all.

Thanks for your help in the Slack yesterday, @r2d4!

[chino]

Is there going to be an easy way to do this at install/runtime without the need to rebuild ISOs and other steps?

[ztripez]

We are unable to run our docker repo without a cert and it's not exposed to the internet, iow we must use a self signed cert.
If there is no way to add the cert to minikube without building a new ISO makes it very hard for us to using minikube/kubernetes.

[dkirrane]

Could someone provide the steps to create the minikube ISO with my certs and how to pass it to minikube start?

[jwflory]

@dkirrane I actually forgot I was supposed to write documentation for this. A few months ago, I added a flag to Minikube that lets you specify a remote volume for generating the ISO (if you're making it from scratch).

See: #1657

Unfortunately, I no longer have access to the workstation / network where I was doing this development work. But if I remember right, you should be able to use the ISO_DOCKER_EXTRA_ARGS environment variable to specify a path to a volume, like…

$ ISO_DOCKER_EXTRA_ARGS='-v /path/to/certs/:/etc/ssl/certs/

Hopefully this is helpful for you. Sorry I can't test this further anymore.

[ChristofferNicklassonLenswayGroup]

Hi,

Docker wants to have the certs for repository's in /etc/docker/certs.d/site/ca.crt
Would like to see that path being persistent.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[erip]

Any update on this? I'm not sure if I can do this, but /remove-lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[dkirrane]

Still an issue /remove-lifecycle rotten

[certifiedwaif]

This is still an issue for me. /remove-lifecycle rotten

[ghandim]

This is also still an issue for me. I found no real sophisticated solution for this problem at the moment.

[nrupendesai]

This is still an issue for me. If we have any solution where I can inject my SSL would be very helpful.

[mynameisD]

still an issue for me

[sean-dooher]

I tried all of the above to no avail, it's still an issue for me

[rhetzler]

this is an issue with me as well

[edwinpjacques]

This is still an issue for me. /remove-lifecycle rotten

[LupusUmbrae]

You can get the SSL into minikube by adding it to the files directory (at least on linux)

If you go to ~/.minikube/files these are copied onto the VM, so to copy a self signed SSL into docker you can just create the folder .minikube/files/etc/docker/certs.d/<domain>/ and drop in the certificate as a ca.crt

I did have this all working awhile ago, wrote some docs on it and then it all stopped working :(, so I have a feeling that I also need a way to add the self signed cert into the ca-certificates or something

[nrupendesai]

This steps worked for me.

1. Download files related to ingress from github at https://github.com/nrupendesai/minikube/tree/master/deploy/addons/ingress
2. Disable minikube ingress addon.
3. Create new ingress deployment from downloaded files.
4. Create secret for ssl certificate using file. This has to be created in kube-system name space.
   kubectl create secret tls tls-certificate --key key.pem --cert cert.pem -n kube-system
5. Specify secret information in ingress file.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: colortokens-colormaster
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "<name of services separated by comma>"
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
spec:
  tls:
    - hosts:
      secretName: tls-certificate

[smiron]

/remove-lifecycle rotten

[mvaldesdeleon]

The following just worked for me, so I'm sharing it in case anybody else can benefit from it.

Following on @LupusUmbrae recommendation (which I tried but did not work for me), I settled by providing the PEM-encoded certificate file for the private registry in the following path:

~/.minikube/files/etc/ssl/certs/private-registry.example.com.pem

I set this file with 750 permissions just in case, recreated the VM and was able to connect to the registry without any further issues.

minikube version: v0.28.2
OSX 10.11.6 (15G19009)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[LupusUmbrae]

As @mvaldesdeleon commented if you add the SSL to the files it works, for me it turns out I needed the FQDN not just the domain root.

A short document on this would help I think

[ghandim]

Seems also to work for me with the hint from @mvaldesdeleon.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[kevinpauli]

I used to have this working with minikube 1.1 by following @mvaldesdeleon 's advice, but it no longer works with minikube 1.2

E0723 09:38:44.220307   92650 start.go:403] Error caching images:  Caching images for kubeadm: caching images: caching image /Users/kcpaul/.minikube/cache/images/k8s.gcr.io/kube-scheduler_v1.15.0: fetching remote image: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority

EDIT:

Well, I take that back, somehow it pulled the image correctly:

$ kubectl describe pod kube-scheduler-minikube -n kube-system | grep Image:
    Image:         k8s.gcr.io/kube-scheduler:v1.15.0

It just couldn't put the new image in the cache for some reason... note the only one in the cache is the old 1.14.3 one from minikube 1.1:

$ ls ~/.minikube/cache/images/k8s.gcr.io/ | grep kube-scheduler
kube-scheduler_v1.14.3

So I guess this is a minor annoyance

[medyagh]

@kevinpauli @ghandim @mvaldesdeleon @certifiedwaif @LupusUmbrae @isuftin @chino @jwflory @isuftin @ztripez

I believe this issue will be solved by this PR and I am excited about it ! please let me know if this PR fixes the issue you are facing.
#5015