Importing Corporate Root CA is not working as advertised

[CrossBound]

According to merge #5015 and https://minikube.sigs.k8s.io/docs/tutorials/untrusted_root_certificate/ I should be able to drop my corporate root certificate into my $home\.minikube\certs directory and it would get copied to the minikube vm during startup. I have done this and I see evidence of this in the vm (output below), however it only creates symlinks in the /etc/ssl/certs folder and there are no files there. Attempting to use curl from within the VM returns a problem with the TLS trust.

I have confirmed that if I manually create the file in the /etc/ssl/certs folder it resolves the issue.

Expected Result:
Copying my root certificate into $home.minikube\certs folder (as PEM format) and then starting the vm should copy the file to the vm's /etc/ssl/certs folder which will resolve TLS connection issues protected by our corporate root certificate. After this ssh'ing into the vm ( minikube ssh ) and then calling the service ( curl https://service.domain:0000 ) should succeed.

Actual Results:
A symlink is created in the /etc/ssl/certs folder, but not the actual certificate file.
After this ssh'ing into the vm ( minikube ssh ) and then calling the service ( curl https://service.domain:0000 ) returns an error:

In the output below, lbrtca01-ca.pem is my corporate root certificate.

PS D:\Projects\LEAF\Kubernetes\Config> dir $home\.minikube\certs

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        1/29/2020   2:25 PM           1679 ca-key.pem
-a----        1/29/2020   2:25 PM           1046 ca.pem
-a----        1/29/2020   2:25 PM           1086 cert.pem
-a----        1/29/2020   2:25 PM           1675 key.pem
-a----        2/10/2020  11:03 AM           2062 lbrtca01-ca.pem
-a----        2/10/2020  11:04 AM           1984 legacy01-svcert01-ca.pem

PS D:\Projects\LEAF\Kubernetes\Config> minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$
$ ls -l /etc/ssl/certs | grep -i 'lbrt'
lrwxrwxrwx 1 root root     30 Feb 10 17:44 9373d18b.0 -> /etc/ssl/certs/LBRTCA01-CA.pem
$ sudo cat /etc/ssl/certs/LBRTCA01-CA.pem
cat: /etc/ssl/certs/LBRTCA01-CA.pem: No such file or directory
$
$ openssl s_client -connect dnaleafas.legacy01.legacybank.com:10001
CONNECTED(00000003)
depth=1 DC = com, DC = LegacyBank, DC = Legacy01, CN = Legacy01-SVCERT01-CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=DNALEAFAS.Legacy01.LegacyBank.com
   i:/DC=com/DC=LegacyBank/DC=Legacy01/CN=Legacy01-SVCERT01-CA
 1 s:/DC=com/DC=LegacyBank/DC=Legacy01/CN=Legacy01-SVCERT01-CA
   i:/DC=com/DC=LegacyBank/DC=Legacy01/CN=LBRTCA01-CA
---

I am running minikube version v.1.7.1 with the Hyper-V VM driver on Windows 10 Enterprise Version 1809 x64.

Thank You

[michaeljohn32]

I am also experiencing this issue with the same workaround (copied cert to both .minikube/files/etc/ssl/certs/myCA.pem and .minikube/certs/myCA.pem)

minikube v1.7.3, Virtualbox Driver on MacOSX 10.14.6

[medyagh]

I wonder if the filesync is working on mac... We had a bug about file sync but I believe we fixed it

[medyagh]

Could you please do minikube ssh
And verify the certs have not been copied ?

[michaeljohn32]

If I use a minikube ssh with just the pem file copied to ~/.minikube/certs/MyCA.pem, I see that the pem file is not copied. The symbolic link with the hash is there, but that is all. If I copy the pem to ~/.minikube/files/etc/ssl/certs/MyCA.pem, the cert is copied/synced.

[medyagh]

@michaeljohn32 as far I know you are supposed to copy to the files folder. As you noted lastly.

Anything you put in that folder will be synced to the VM.

So if that is the case that putting it in files folder fixes the corp ! Then the documentation should say that too.

Do you mind sharing the link to the wrong documentation?

[CrossBound]

@medyagh No offense, but did you even read my issue? Everything you have asked was covered in my initial post.

[CrossBound]

Interestingly, putting the root certificate in the $home\.minikube\files\etc\ssl\certs\ folder on my workstation still doesn't copy the file to the minikube vm for me.

Here is my output after using minikube ssh

$ ls -l /etc/ssl/certs | grep -i 'lbrt'
lrwxrwxrwx 1 root root     30 Feb 24 23:06 9373d18b.0 -> /etc/ssl/certs/LBRTCA01-CA.pem
$

[medyagh]

@CrossBound admitably I didnt fully read your comment ! sorry about that.

I am curious would you mind trying minkube stop , start
(or at least minikube start again)
as far I know the file sync gets triggered on each start

[medyagh]

@CrossBound also I just noticed you are using v.1.7.1 we had a file sync bug in 1.7.1
would u mind trying with 1.7.3 ?

[CrossBound]

After updating to v1.7.3 it fixes the issue where files in my .minikube\files directory are not copied, however, it still requires me to put the root certificate in .minikube\files\etc\ssl\certs for it to work.

Based on the documentation listed in my first comment it sounds like all you should have to do is put the file in the .minikube\certs folder for it to work. If that is not the case maybe the docs just need to be updated?

[medyagh]

thank you @CrossBound for trying new version and updating this issue,

indeed the docs needs to be udpated I wonder if that is the case only for windows or mac too ?

I would happily review any PR that fixes this documenation.

[michaeljohn32]

@medyagh, in at least one previous version of minikube, it was not necessary to put the cert in both ~/.minikube/files/... and ~/.minikube/certs. I do not recall what version I had installed(I believe it was 1.6.0), but it used to work (both the hash and the certificate would be copied to the correct location) with just putting the certificate file in the ~/.minkube/certs folder. The documentation was correct. This is a regression.

[CrossBound]

I might also add that I'm confused about the purpose of merge #5015 if you still have to go the other route, seems pointless if it doesn't really solve the problem.

[kristoflemmens]

I have the same problem. Did some tests and it worked in version 1.6.2 and stopped working in version 1.7.0

[medyagh]

This sounds like a regression ! we will need to fix it.

we will fix this and add integration tests for it so it never happens again.

thank you everyone for bringing this to our attention. @michaeljohn32 @kristoflemmens @CrossBound

and thanks @kristoflemmens for provoding the exact versions that works.

[tstromberg]

I'm pretty sure I broke this feature when refactoring file synchronization, though I'm not exactly sure how yet. I'll need to do some git bisect work to narrow it down. Thank you for reporting this!

[tstromberg]

Sent out a fix PR. The underlying issue was a combination of an inverted boolean check and inverting the arguments to ln. Somehow, after 25 years of using UNIX, I can never get ln -s arguments correct.

I've added a comprehensive integration test for certificate synchronization so that this feature does not unexpectedly break in the future.

Thank you @CrossBound for the comprehensive bug report.