Add support for minikube+containerd

[ErmakovDmitriy]

I tried to build my project using minikube, skaffold and kaniko.
Kaniko is used because I tried to use minikube with containerd (not Docker). The main reason is that the Kubernetes team has plans to deprecate Dockershim.

Expected behavior

Correct build and dev deploy.

Actual behavior

Error:

Listing files to watch...
 - skaffold-example
watching files for artifact "skaffold-example": listing files: parsing ONBUILD instructions: retrieving image "golang:1.12.9-alpine3.10": getting docker client: getting minikube env: running [/usr/local/bin/minikube docker-env --shell none -p minikube]
 - stdout: "\n\n"
 - stderr: "X Exiting due to MK_USAGE: The docker-env command is only compatible with the \"docker\" runtime, but this cluster was configured to use the \"containerd\" runtime.\n"
 - cause: exit status 14

Information

I thought that there is some error in my skaffold.yaml so I tried to build examples/kaniko. The same error.

• Skaffold version: v1.17.2
• Operating system: minikube on KVM2, minikube version: v1.15.1, commit: 23f40a012abb52eff365ff99a709501a61ac5876.
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta10
kind: Config
build:
  artifacts:
    - image: skaffold-example
      kaniko:
        cache: {}
  cluster:
    pullSecretName: e2esecret-with-path
    pullSecretPath:     kaniko-secret.json
    namespace: default
deploy:
  kubectl:
    manifests:
      - k8s-*

Steps to reproduce the behavior

1. clone https://github.com/GoogleContainerTools/skaffold
2. cd examples/kaniko
3. skaffold dev
4. Get the error

I personally think that Minikube detect at: skaffold/cluster/minikube.go in method newAPIClient call to cluster.GetClient().IsMinikube(kubeContext) returns true because the Kubernetes context name is "minikube" (I checked in debugger).

// newAPIClient guesses the docker client to use based on current Kubernetes context.
func newAPIClient(kubeContext string, minikubeProfile string) ([]string, client.CommonAPIClient, error) {
	if minikubeProfile != "" { // skip validation if explicitly specifying minikubeProfile.
		return newMinikubeAPIClient(minikubeProfile)
	}
	if cluster.GetClient().IsMinikube(kubeContext) {
		return newMinikubeAPIClient(kubeContext)
	}
	return newEnvAPIClient()
}

and in newMinikubeAPIClient there is a call env, err := getMinikubeDockerEnv(minikubeProfile).
In getMinikubeDockerEnv we have:

cmd, err := cluster.GetClient().MinikubeExec("docker-env", "--shell", "none", "-p", minikubeProfile)

which in case of minikube with containerd returns error:

$ minikube docker-env

❌  Exiting due to MK_USAGE: The docker-env command is only compatible with the "docker" runtime, but this cluster was configured to use the "containerd" runtime.

[tstromberg]

FWIW, this should be a relatively high priority, as minikube is expected to switch the default runtime in Q1.

For non-Docker runtimes, I recommend using the minikube cache add command, at least until a faster mechanism can be designed & supported.

[tejal29]

@ErmakovDmitriy We don't support running minikube with containerd yet. However, that said, there is a skaffold bug here. For kaniko builder, skaffold should not shell minikube docker-env command.

[tstromberg]

It's worth noting that since this issue was originally opened, minikube added a minikube image load command that may be more efficient than the older cache implementation: https://minikube.sigs.k8s.io/docs/commands/image/

[sammym1982]

We are hit by this issue when we tried moving our dev tooling to target containerd with minikube. AKS now defaults to containerd runtime post 1.19 k8s version and we want to align our tooling to target same.

I see multiple issues around same. Is fair to say that #5739 will resolve this issue and correct to track.

@tstromberg

[nkubala]

@sammym1982 yes #5739 will address this issue, and is currently on the team's radar. since that issue has become active, I'll go ahead and close this one as the dupe and we can track over there.