Kubernetes v1.16.x (default in minikube v1.4) breaks Helm Tiller (v2.14.3)

[aaronhmiller]

The exact command to reproduce the issue:
Install Minikube 1.4.0

helm init
$HELM_HOME has been configured at /Users/amiller/.helm.
Error: error installing: the server could not find the requested resource

The full output of the command that failed:

Note: downgrade to Minikube 1.3.1

helm init
$HELM_HOME has been configured at /Users/amiller/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run helm init with the --tiller-tls-verify flag.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation

The output of the minikube logs command:

sorry, needed to get this running again. didn't grab logs

The operating system version:

uname -a
Darwin slate.local 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64
sw_vers
ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G95

[afbjorklund]

Helm v2.14.3 (latest) is currently incompatible with Kubernetes version v1.16.0 (latest):

helm/helm#6374

[afbjorklund]

Workarounds (only need one of them):

1. Patch the YAML from helm init:

helm init --output yaml | sed 's@apiVersion: extensions/v1beta1@apiVersion: apps/v1@' | sed 's@  replicas: 1@  replicas: 1\n  selector: {"matchLabels": {"app": "helm", "name": "tiller"}}@' | kubectl apply -f -

@@ -1,5 +1,5 @@
 ---
-apiVersion: extensions/v1beta1
+apiVersion: apps/v1
 kind: Deployment
 metadata:
   creationTimestamp: null
@@ -10,6 +10,7 @@
   namespace: kube-system
 spec:
   replicas: 1
+  selector: {"matchLabels": {"app": "helm", "name": "tiller"}}
   strategy: {}
   template:
     metadata:

2. Use old Kubernetes version v1.15

minikube start --kubernetes-version=v1.15.4

3. Wait for helm release v2.15.0

"It looks like once we start supporting Kubernetes 1.16.0 we will have to handle that case going forward and migrate to the newer apiVersion."

Either way, it doesn't look like a minikube issue...

$ minikube kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}

Maybe it should be (include helm), though: #5302

[afbjorklund]

I think this collaboration could probably be improved somewhat:

https://helm.sh/docs/using_helm/#kubernetes-distribution-guide

MINIKUBE
Helm is tested and known to work with minikube. It requires no additional configuration.

Apparently that statement is only true one-way, new helm releases.

[bacongobbler]

This is the first time in a long time where Helm has been incompatible with the latest release of Kubernetes. That statement has held true from k8s 1.5 up until 1.16.

[afbjorklund]

@bacongobbler : what I meant was that we should make sure to test new versions of minikube with helm, during our beta phase. We had a similar last-minute experience with the dashboard as well.

I don't think it is possible for downstream projects to test with future releases, but I do think that Kubernetes (and Minikube) could add some extra regression testing for prominent third-parties.

[woodcockjosh]

helm/helm#6374
FYI for anyone trying to helm init on k8s version 1.16 this branch works https://github.com/keleustes/helm/tree/kube16. You can build the branch yourself. I also uploaded the binary here for your convenience https://s3-us-west-2.amazonaws.com/bin.cryptexlabs.com/github.com/keleustes/helm/kube16/helm. One caveat is you have to use the canary image flag helm init --canary-image since the build is unreleased

[woodcockjosh]

This plugin would also fix the problem :-D #5363

[tstromberg]

It's worth noting that this also affects older minikube releases which select Kubernetes v1.16.0. In the mean time, you can work around the incompatibility by selecting an older version of Kubernetes:

--kubernetes-version=v1.15.3

https://minikube.sigs.k8s.io/docs/reference/configuration/kubernetes/

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.