Figure out KVM failures in jenkins on Debian 10

[medyagh]

No description provided.

[priyawadhwa]

Sample failure from #8850
logs here

Looks like this is pretty exclusive to KVM

$ /home/tstromberg/bin/passrate.sh TestFunctional
-------------------------------------------------
docker_Linux      | 2020-08-07 | 100% | 12 of 12
docker_Linux      | 2020-08-08 | 100% | 12 of 12
docker_Linux      | 2020-08-09 | 100% | 12 of 12
docker_Linux      | 2020-08-10 | 100% | 12 of 12
docker_Linux      | 2020-08-11 | 100% | 4 of 4
-------------------------------------------------
Hyper-V_Windows   | 2020-08-11 |   0% | 0 of 1
-------------------------------------------------
KVM_Linux         | 2020-08-07 |  80% | 12 of 15
KVM_Linux         | 2020-08-08 |  75% | 12 of 16
KVM_Linux         | 2020-08-09 |  75% | 12 of 16
KVM_Linux         | 2020-08-10 |  83% | 10 of 12
KVM_Linux         | 2020-08-11 |  80% | 4 of 5
-------------------------------------------------
none_Linux        | 2020-08-07 | 100% | 12 of 12
none_Linux        | 2020-08-08 | 100% | 12 of 12
none_Linux        | 2020-08-09 | 100% | 12 of 12
none_Linux        | 2020-08-10 |  92% | 12 of 13
none_Linux        | 2020-08-11 | 100% | 4 of 4
-------------------------------------------------
podman_Linux      | 2020-08-07 |  50% | 12 of 24
podman_Linux      | 2020-08-08 |  50% | 12 of 24
podman_Linux      | 2020-08-09 |  50% | 12 of 24
podman_Linux      | 2020-08-10 |  50% | 12 of 24
podman_Linux      | 2020-08-11 |  50% | 4 of 8
-------------------------------------------------
VirtualBox_Linux  | 2020-08-07 | 100% | 13 of 13
VirtualBox_Linux  | 2020-08-08 | 100% | 12 of 12
VirtualBox_Linux  | 2020-08-09 | 100% | 11 of 11
VirtualBox_Linux  | 2020-08-10 | 100% | 11 of 11
VirtualBox_Linux  | 2020-08-11 | 100% | 5 of 5
-------------------------------------------------
z__ALL__          | 2020-08-07 |  80% | 61 of 76
z__ALL__          | 2020-08-08 |  79% | 60 of 76
z__ALL__          | 2020-08-09 |  79% | 59 of 75
z__ALL__          | 2020-08-10 |  79% | 57 of 72
z__ALL__          | 2020-08-11 |  78% | 21 of 27
-------------------------------------------------

Looks like the minikube VM is randomly disappearing

=== CONT  TestFunctional/parallel/ServiceCmd
    functional_test.go:700: (dbg) Non-zero exit: out/minikube -p functional-20200811105441-29795 service list: exit status 69 (11.249487886s)


$ minikube service list -p  functional-20200811105441-29795

E0811 12:12:11.198533   27652 exit.go:76] &{ID:KVM2_NO_DOMAIN Err:getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'functional-20200811105441-29795'')
state
k8s.io/minikube/pkg/minikube/machine.Status
        /home/priyawadhwa/minikube/pkg/minikube/machine/host.go:44
k8s.io/minikube/pkg/minikube/mustload.Running
        /home/priyawadhwa/minikube/pkg/minikube/mustload/mustload.go:91
k8s.io/minikube/pkg/minikube/mustload.Healthy
        /home/priyawadhwa/minikube/pkg/minikube/mustload/mustload.go:142
k8s.io/minikube/cmd/minikube/cmd.glob..func23
        /home/priyawadhwa/minikube/cmd/minikube/cmd/service_list.go:41
github.com/spf13/cobra.(*Command).execute
        /home/priyawadhwa/go/pkg/mod/github.com/spf13/[email protected]/command.go:846
github.com/spf13/cobra.(*Command).ExecuteC
        /home/priyawadhwa/go/pkg/mod/github.com/spf13/[email protected]/command.go:950
github.com/spf13/cobra.(*Command).Execute
        /home/priyawadhwa/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
k8s.io/minikube/cmd/minikube/cmd.Execute
        /home/priyawadhwa/minikube/cmd/minikube/cmd/root.go:106
main.main
        /home/priyawadhwa/minikube/cmd/minikube/main.go:71
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373 Advice:The VM that minikube is configured for no longer exists. Run 'minikube delete' URL: Issues:[3636] ShowIssueLink:false}
❌  [KVM2_NO_DOMAIN] Unable to get machine status state: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'functional-20200811105441-29795'')
💡  Suggestion: The VM that minikube is configured for no longer exists. Run 'minikube delete'
⁉️   Related issue: https://github.com/kubernetes/minikube/issues/3636

[afbjorklund]

@priyawadhwa : thanks for taking this on, never got around to setting debian up

[priyawadhwa]

minikube start is failing on our kvm VM #7, which seems to be solely responsible for running the KVM tests. Fails with:

Network is already in use by interface br-f912846614e5

I SSH'd into the VM and ran the commands suggested in this comment:

#2513 (comment)

After that, I was able to start minikube with kvm2 successfully in 70 seconds.

I think this may have fixed the issue, currently waiting on test results from #9576 to make sure.

Releated slack conversation: https://kubernetes.slack.com/archives/C1F5CT6Q1/p1604002466057500

[priyawadhwa]

Closing, as that seems to have fixed it -- only one failure on the most recent run

https://storage.googleapis.com/minikube-builds/logs/9576/3c8d3ca/KVM_Linux.html