vbox test: "rm: cannot remove '/var/lib/minikube/certs/ca.crt': Buffer I/O error on device sda1

[medyagh]

As seen in:

• virtualbox mac os https://storage.googleapis.com/minikube-builds/logs/5075/VirtualBox_macOS.txt
  and https://storage.googleapis.com/minikube-builds/logs/5082/VirtualBox_macOS.txt
• virtualbox windows Failed to setup certs: rm: cannot remove '/var/lib/minikube/certs/ca.crt': Input/output error #4342 and also another

this could be side effect of #5075
I was wrong this is not side effect of that PR,

it seems to be in both cases it has happend in Virtualbox (windows and macos)

--- FAIL: TestPersistence (1502.40s)
        sudo systemctl daemon-reload &&
        sudo systemctl start kubelet
        I0814 11:47:13.448775    1984 certs.go:49] Setting up certificates for IP: 192.168.99.101
        I0814 11:47:13.452133    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/client.crt" with filemode -rw-r--r--
        I0814 11:47:13.452383    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/client.key" with filemode -rw-------
        I0814 11:47:13.455882    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/apiserver.crt" with filemode -rw-r--r--
        I0814 11:47:13.456142    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/apiserver.key" with filemode -rw-------
        I0814 11:47:13.526871    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/proxy-client.crt" with filemode -rw-r--r--
        I0814 11:47:13.527111    1984 lock.go:33] attempting to write to file "/Users/jenkins/minikube-integration/darwin-amd64-virtualbox-5075-641-9fad24aa37f48dd18ce5e76a0714a4a642ee4fed/.minikube/proxy-client.key" with filemode -rw-------
        I0814 11:47:13.528764    1984 ssh_runner.go:101] SSH: sudo rm -f /var/lib/minikube/certs/ca.crt
        I0814 11:47:13.538258    1984 utils.go:167] ! rm: cannot remove '/var/lib/minikube/certs/ca.crt': Input/output error
        W0814 11:47:13.538530    1984 exit.go:99] Failed to setup certs: pre-copy: command failed: sudo rm -f /var/lib/minikube/certs/ca.crt
        stdout: 
        stderr: rm: cannot remove '/var/lib/minikube/certs/ca.crt': Input/output error
        : Process exited with status 1
        * 
        X Failed to setup certs: pre-copy: command failed: sudo rm -f /var/lib/minikube/certs/ca.crt
        stdout: 
        stderr: rm: cannot remove '/var/lib/minikube/certs/ca.crt': Input/output error
        : Process exited with status 1
        * 
        * Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
          - https://github.com/kubernetes/minikube/issues/new/choose

[medyagh]

I wonder if we put a retry logic around pre-copy delete that would fix this flake ?

[tstromberg]

This sounds like filesystem corruption. I would love to see the output of dmesg.

[tstromberg]

This could be a virtualbox bug, or someone stepped on the VM image:

> [  +0.000002] Buffer I/O error on device sda1, logical block 399013

[medyagh]

Seen again on two tests on virtualbox:

Again on TestPersistence and this time also on TestDocker
https://storage.googleapis.com/minikube-builds/logs/5121/VirtualBox_macOS.txt

That makes me wanna take a closer look at TestPersistence

	mk := NewMinikubeRunner(t, p, "--wait=false")
	defer mk.TearDown(t)

	stdout, stderr, err := mk.Start()
	if err != nil {
		t.Fatalf("failed to start minikube (for profile %s) failed : %v\nstdout: %s\nstderr: %s", t.Name(), err, stdout, stderr)
	}

	// Make sure everything is up before we stop.
	verifyBusybox(t)

	mk.RunCommand("stop", true)
	mk.CheckStatus(state.Stopped.String())

	stdout, stderr, err = mk.Start()

Makes me suspect our Stop for virtualbox does not really stop !