The networking.gke.io/suppress-firewall-xpn-error: true annotation doesn't in combination with kubernetes.io/ingress.global-static-ip-name

[m1kola]

I tested on 1.11.3-gke.18 and it seems like there is an issue with networking.gke.io/suppress-firewall-xpn-error annotation: it doesn't seem to work in combination with kubernetes.io/ingress.global-static-ip-name.

If you create a static IP address using gcloud and then create a new Ingress resource using a yaml similar to the one below the Ingress resource will be created without both kubernetes.io/ingress.global-static-ip-name and networking.gke.io/suppress-firewall-xpn-error: true annotations (at least this is what I see in kubectl describe ingress.extensions/app).

Also for new Ingress objects controller will get a new IP address which will be different from what you specify in kubernetes.io/ingress.global-static-ip-name.

Expected result:

1. A new Ingress resource reuses previously reserved IP address (specified by name in the kubernetes.io/ingress.global-static-ip-name annotation).
2. Controller doesn't produce XPN firewall events
3. I can see both kubernetes.io/ingress.global-static-ip-name and networking.gke.io/suppress-firewall-xpn-error annotations in kubectl describe ingress.extensions/app.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: some-static-ip-name
    networking.gke.io/suppress-firewall-xpn-error: true
  name: app
  namespace: some-namespace
spec:
  backend:
    serviceName: app
    servicePort: 80
  tls:
  - hosts:
    - some.host.com
    secretName: some-tls-secret

---

Related issue #485 and PR #506

cc @grayluck @rramkumar1

/kind bug

[grayluck]

Hmmm I can't reproduce this issue.

$ gcloud compute addresses create addr-app --global
Created [https://www.googleapis.com/compute/v1/projects/yankaiz-xpn-svc/global/addresses/addr-app].
$ gcloud compute addresses describe addr-app --global
address: 35.244.192.131
creationTimestamp: '2018-12-07T13:56:41.247-08:00'
description: ''
...
$ k version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.3-gke.18", GitCommit:"35471a6a3958f610351c67208de0f3824c4e8b3d", GitTreeState:"clean", BuildDate:"2018-11-14T05:27:44Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
$ k create -f ing-web.yaml 
ingress.extensions/ing-web created
service/app created
replicationcontroller/app created
$ k describe ingress.extensions/ing-web
Name:             ing-web
Namespace:        default
Address:          35.244.192.131
Default backend:  app:8080 (10.4.1.2:8080,10.4.1.6:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     *     app:8080 (10.4.1.2:8080,10.4.1.6:8080)
Annotations:
  ingress.kubernetes.io/backends:                 {"k8s-be-32245--9b4eb1ef95f44045":"Unknown"}
  ingress.kubernetes.io/forwarding-rule:          k8s-fw-default-ing-web--9b4eb1ef95f44045
  ingress.kubernetes.io/target-proxy:             k8s-tp-default-ing-web--9b4eb1ef95f44045
  ingress.kubernetes.io/url-map:                  k8s-um-default-ing-web--9b4eb1ef95f44045
  kubernetes.io/ingress.global-static-ip-name:    addr-app
  networking.gke.io/suppress-firewall-xpn-error:  true
Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  ADD     2m58s  loadbalancer-controller  default/ing-web
  Normal  CREATE  2m15s  loadbalancer-controller  ip: 35.244.192.131

And the yaml file that I used (.txt added since github doesn't support uploading yaml files):
ing-web.yaml.txt

[m1kola]

@grayluck hm... Can it be because in my yaml I have networking.gke.io/suppress-firewall-xpn-error: true, but you use networking.gke.io/suppress-firewall-xpn-error: "true"? Note quotes in "true" (bool vs string).

Originally documentation had networking.gke.io/suppress-firewall-xpn-error: true. Probably it's the source of my confusion.

[grayluck]

@m1kola I don't think this is the case since it only takes string as input. I tried boolean and it gave me a ruthless invalid request.
If you see here, we parse the annotation from string to boolean.
Could you follow the operations that I did and show me what you get in the output? Thanks.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[thecodejunkie]

@grayluck I'm experience the same problem on GKE 1.11.7-gke.4. I'm applying the following annotations (in the exact order as presented) to my ingress

kubernetes.io/ingress.global-static-ip-name: some-name
networking.gke.io/suppress-firewall-xpn-error: "true"
ingress.gcp.kubernetes.io/pre-shared-cert: some-other-name

And when I do a kubectl describe ingress I still see this error

Type    Reason  Age                From                     Message
----    ------  ----               ----                     -------
Normal  XPN     11s (x71 over 9h)  loadbalancer-controller  Firewall change required by network admin: ...

[grayluck]

@thecodejunkie I don't have a 1.11.7-gke.4 cluster at hand right now. Could you upgrade your cluster to 1.11.7-gke.12 first? I'll try to reproduce this issue in my 1.11.7-gke.12 cluster.
/remove-lifecycle stale

[grayluck]

@thecodejunkie I can't reproduce your issue from within my cluster. Could you tell me which the zone and the name of the cluster to facilitate troubleshooting? Thanks.

[thecodejunkie]

@grayluck the weirdies thing happened. I left it running over night and the warning vanished. I will report back if I observe it again

[grayluck]

My suspicion is that your master is auto upgraded last night and includes the implementation of the annotation.
Please reopen if issue recurs. Thanks!
/close

[k8s-ci-robot]

@grayluck: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

My suspicion is that your master is auto upgraded last night and includes the implementation of the annotation.
Please reopen if issue recur. Thanks!
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[thecodejunkie]

@grayluck I think you might be right. My master node is now on 1.11.7-gke.12

[grayluck]

Thanks for the info! I'll go check if the GKE document is accurate about the versions.

[rramkumar1]

@grayluck I'm closing this based on your comment above.