Unable to decode an event from the watch stream

[Tra5is]

After deploying the aad-pod-identity yaml to a RBAC enabled AKS cluster I'm getting these errors from the MIC:

W0115 21:10:59.656749       1 main.go:26] --kubeconfig not passed will use InClusterConfig
I0115 21:10:59.656834       1 main.go:29] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I0115 21:10:59.657061       1 mic.go:53] Starting to create the pod identity client. Version: 1.2. Build date: 2018-08-11-02:06
I0115 21:10:59.873873       1 pod.go:80] Pod cache synchronized. Took 200.435196ms
I0115 21:10:59.873996       1 pod.go:86] Pod watcher started !!
I0115 21:10:59.874054       1 crd.go:156] CRD watchers started
I0115 21:10:59.874075       1 main.go:42] AAD Pod identity controller initialized!!
I0115 21:10:59.874148       1 mic.go:126] Sync thread started
E0115 21:17:04.030350       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "Status" is registered for version "v1"
W0115 21:17:04.030389       1 reflector.go:341] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: watch of *v1.AzureIdentity ended with: very short watch: github.com/Azure/aad-pod-identity/pkg/crd/crd.go:155: Unexpected watch close - watch lasted less than a second and no items received
E0115 21:20:03.943141       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "Status" is registered for version "v1"
W0115 21:20:03.943176       1 reflector.go:341] github.com/Azure/aad-pod-identity/pkg/crd/crd.go:154: watch of *v1.AzureIdentityBinding ended with: very short watch: github.com/Azure/aad-pod-identity/pkg/crd/crd.go:154: Unexpected watch close - watch lasted less than a second and no items received

I deployed using this script (powershell)

Write-Host "Deploying azure-aad-identity infrastructure..." -ForegroundColor Green
& kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml

Write-Host "Fetching helm chart to deploy application-gateway-kubernetes-ingress chart..." -ForegroundColor Green
& helm init --client-only --tiller-namespace kube-system --service-account tiller
& helm repo add application-gateway-kubernetes-ingress https://azure.github.io/application-gateway-kubernetes-ingress/helm/
& helm repo update

if (-not $delete) {
  Write-Host "Deploying application-gateway-kubernetes-ingress chart..." -ForegroundColor Green
  & helm upgrade --install --wait `
      --set appgw.subscriptionId="$subscriptionId" `
      --set appgw.resourceGroup="$agwResourceGroupName" `
      --set appgw.name="$appgwName" `
      --set armAuth.type=aadPodIdentity `
      --set armAuth.identityResourceID="$($managedIdentity.id)" `
      --set armAuth.identityClientID="$($managedIdentity.clientId)" `
      --set aksClusterConfiguration.apiServerAddress="$($aks.Properties.fqdn)" `
      --set rbac.enabled=true `
      --set kubernetes.watchNamespace="$namespace" `
      --namespace $namespace `
      appgw-k8s-$namespace `
      application-gateway-kubernetes-ingress/ingress-azure
}

I'm expecting the service that the helm step deploys (appgw-k8s-$namespace) will get the identity created by this helm chart. But the initial error seems to prevent the MIC from picking up the new pod and associated identity.

[cherwin]

Are you using something other than the default namespace? If yes, you might have run into this issue #115.

[kkmsft]

Similar issues are reported in AKS: Azure/AKS#676

Looks like the new cluster has to be created or upgrade/scale can result in the fixe. Here is the link corresponding to that:
Azure/AKS#676 (comment)

@Tra5is @cherwin - Can you please try it on a fresh cluster or a cluster on which the fix above in AKS is applied. Please let me know how it goes.

[kkmsft]

@Tra5is @cherwin - any pointers on whether fresh clusters with the fix for AKS
Azure/AKS#676 still exhibits this issue for pod identity ?

[cherwin]

@kkmsft The cluster that I am using at the moment is merely a couple of days old so it should not be affected by Azure/AKS#676.

This problem only occurs when I deploy to a namespace other than default. Judging from the script that @Tra5is is using, it looks like he also might be deploying to a different namespace. It's not clear to me how it's possible that deploying to a namespace that is different than the hardcoded one could trigger this issue. However, I realise that #140 recently got merged so if the aforementioned was the culprit, it might have been fixed already.

[kkmsft]

@cherwin - Thank you. I am suspecting there are two issues in play here - one that the regular watches are not working properly and then the non-default namespace issue.

E0115 21:20:03.943141       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "Status" is registered for version "v1"

This seems unrelated to namespace related issues. Were you seeing the above errors as well ?

@Tra5is - can you please confirm that with the latest images in this release which includes the support for non-default namespace - https://github.com/Azure/aad-pod-identity/releases/tag/1.3.0-mic-1.4.0-nmi (and on a cluster with Azure/AKS#676 (comment) applied) works well for the issue you have raised ?

[kkmsft]

@Tra5is - was wondering if you got a chance to look at the above update and try out the new feature. Please let me know if you are still seeing the issue.

[kkmsft]

@Tra5is - Kindly let us know if this is still an issue.

[kkmsft]

Closing this issue since the newer AKS cluster and namespace work should have resolved the issues. @Tra5is - please feel free to reopen if that is not the case.