Test Flake: calculate hash for cacert: Expecting: TRUSTED CERTIFICATE

[medyagh]

As seen on a PR on the head:
https://storage.googleapis.com/minikube-builds/logs/6953/9e72a2a/Docker_Linux.html#fail_TestStartStop%2fgroup%2fold-docker

for TestStartStop:

I0308 11:48:21.318387    5087 crypto.go:165] Writing key to /home/jenkins/minikube-integration/linux-amd64-docker-6953-3395-9e72a2ad9cfe3e9e63411de65ecae4a2f2f707eb/.minikube/proxy-client.key ...
I0308 11:48:21.318415    5087 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/linux-amd64-docker-6953-3395-9e72a2ad9cfe3e9e63411de65ecae4a2f2f707eb/.minikube/proxy-client.key: {Name:mke8369eb326ee552b61eb2006c7133b4cc6de85 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
W0308 11:48:29.371066    5087 exit.go:101] Failed to setup certs: certificate symlinks: calculate hash for cacert /usr/share/ca-certificates/5904.pem: openssl x509 -hash -noout -in /usr/share/ca-certificates/5904.pem: openssl x509 -hash -noout -in /usr/share/ca-certificates/5904.pem: exit status 1
stdout:
stderr:
unable to load certificate
140712931894464:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
* 
X Failed to setup certs: certificate symlinks: calculate hash for cacert /usr/share/ca-certificates/5904.pem: openssl x509 -hash -noout -in /usr/share/ca-certificates/5904.pem: openssl x509 -hash -noout -in /usr/share/ca-certificates/5904.pem: exit status 1
stdout:
stderr:
unable to load certificate
140712931894464:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
* 
* minikube is exiting due to an error. If the above message is not useful, open an issue:
- https://github.com/kubernetes/minikube/issues/new/choose
** /stderr **

CC: @tstromberg

[tstromberg]

apparently introduced by #6889 - March 4th

[tstromberg]

commands that fail:

      2 docker_test.go:42:
      5 start_stop_delete_test.go:124:
      2 start_stop_delete_test.go:93:
     30 start_stop_delete_test.go:95:
      1 version_upgrade_test.go:100:
      8 version_upgrade_test.go:98:

[medyagh]

seen today https://storage.googleapis.com/minikube-builds/logs/7523/688a0b2/Docker_Linux.html#fail_TestStartStop%2fgroup%2fcontainerd

[tstromberg]

I don't believe this flake to be specific to Docker.

One thing I've noted with the extra logging is that we appear to be transferring a 0-byte .pem into the container/VM:

	I0408 14:07:41.639527   28421 certs.go:120] copying: /usr/share/ca-certificates/14293.pem
...
	I0408 14:07:47.206861   28421 kic_runner.go:91] Run: ls -la /usr/share/ca-certificates/14293.pem
	I0408 14:07:47.399085   28421 certs.go:370] hashing: -rw-r--r-- 1 root root 0 Apr  8 21:07 /usr/share/ca-certificates/14293.pem

That file is created by setupFileSync() in our integration tests, and should be 1338 bytes instead.

[medyagh]

I belive this was fixed