GLBC: Ingress can't be properly created: Insufficient Permission

[bowei]

From @bbzg on July 16, 2017 11:36

I recently upgraded to kubernetes 1.7 with RBAC on GKE, and I am seeing this problem:

  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  6h		6m		75	loadbalancer-controller			Warning		GCE :Quota	googleapi: Error 403: Insufficient Permission, insufficientPermissions

I have double-checked my quotas, and they are all green.

I have also tried granting the Node service account Project > Editor permissions, and I have added the Node service account to the cluster-admin ClusterRole, just in case it had anything to do with that (which it should not, right?).

GKE Cluster logs (slightly redacted):

{
 insertId:  "x"   
 jsonPayload: {
  apiVersion:  "v1"    
  involvedObject: {
   apiVersion:  "extensions"     
   kind:  "Ingress"     
   name:  "ingress-testing"     
   namespace:  "default"     
   resourceVersion:  "425826"     
   uid:  "x"     
  }
  kind:  "Event"    
  message:  "googleapi: Error 403: Insufficient Permission, insufficientPermissions"    
  metadata: {
   creationTimestamp:  "2017-07-15T12:54:37Z"     
   name:  "ingress-testing.x"     
   namespace:  "default"     
   resourceVersion:  "53520"     
   selfLink:  "/api/v1/namespaces/default/events/ingress-testing.14d1822c5ed30595"     
   uid:  "x"     
  }
  reason:  "GCE :Quota"    
  source: {
   component:  "loadbalancer-controller"     
  }
  type:  "Warning"    
 }
 logName:  "projects/x/logs/events"   
 receiveTimestamp:  "2017-07-15T19:11:59.117152623Z"   
 resource: {
  labels: {
   cluster_name:  "app-cluster"     
   location:  ""     
   project_id:  "x"     
  }
  type:  "gke_cluster"    
 }
 severity:  "WARNING"   
 timestamp:  "2017-07-15T19:11:54Z"   
}

I have tried figuring out what the cause might be, but have not found anything that was applicable.

What can I do to get Ingress working again in my cluster?

Thanks!

Copied from original issue: kubernetes/ingress-nginx#975

[bowei]

From @nicksardo on July 16, 2017 17:35

@bbzg There's a bug with 1.7 that causes this problem for the ingress controller on manual GCP networks. If you contact GKE support, they can mitigate it for you.

[bowei]

From @bbzg on July 16, 2017 18:20

Great, I will contact support!

Is there anything I can do to avoid this on the production cluster, when I upgrade it to 1.7?

@nicksardo

[bowei]

From @icereval on July 17, 2017 1:32

Will this issue be expected to go away if I wait for say 1.7.1+ or do we need to contact support in either case?