Firewall rule required message ignores existing rules (shared VPC)

[bluemalkin]

I've created my own firewall rule with custom tags on the GKE node pools to allow the Google health checks.

Despite creating the rule and that everything works, I still get the event message:
gcloud compute firewall-rules create k8s-fw-l7--b1a36f0a2cbc5f47 --network xxx --description "GCE L7 firewall rule" --allow tcp:30000-32767 --source-ranges 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 --target-tags xxx --project xxx

It should check for any existing rule first.

[rramkumar1]

@bluemalkin Why are you creating your own firewall rule? The purpose of ingress-gce is that it manages that rule for you.

[bluemalkin]

@rramkumar1 it doesn't manage firewall rules for you. It simply outputs the gcloud command for you.
The tags it wants to apply it's not satisfactory to cover all my cluster nodes hence I have my own generic one.

[rramkumar1]

@bluemalkin I'm not following... who is outputting the gcloud command? Can you paste the event you are seeing?

[bowei]

Are you using ingress-gce in a Shared VPC environment?

@grayluck

[grayluck]

External load balancer:
https://github.com/kubernetes/kubernetes/blob/54dd6e4f147f05c19de9aef4933031f0348165f7/pkg/cloudprovider/providers/gce/gce_loadbalancer_external.go#L818-L854

Internal load balancer:
https://github.com/kubernetes/kubernetes/blob/54dd6e4f147f05c19de9aef4933031f0348165f7/pkg/cloudprovider/providers/gce/gce_loadbalancer_internal.go#L352

They both require the firewall rules to be strictly as expected, including the description.
Trying to figure out whether the code above also applies to ingress-gce.

[grayluck]

See here:

ingress-gce/pkg/firewalls/firewalls.go

Line 102 in b04a367

if equal(expectedFirewall, existingFirewall) {

There might be difference between the existing and expected firewall rules.

Could you also paste the command that we want you to run, please?

[bluemalkin]

Here's what kubectl get events is telling me

LASTSEEN   FIRSTSEEN   COUNT     NAME                               KIND         SUBOBJECT                  TYPE      REASON                  SOURCE                                               MESSAGE
2m         6d          1352      xxxxxxxxxxxxxxadmin                Ingress                                 Normal    XPN                     loadbalancer-controller                              Firewall change required by network admin: `gcloud compute firewall-rules create k8s-fw-l7--b1a36f0a2cbc5f47 --network xxxxxxxxxxxx --description "GCE L7 firewall rule" --allow tcp:30000-32767 --source-ranges 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 --target-tags gke-node --project xxxx`
2m         6d          1351      xxxxxxxxxxxxxxproxy                Ingress                                 Normal    XPN                     loadbalancer-controller                              Firewall change required by network admin: `gcloud compute firewall-rules create k8s-fw-l7--b1a36f0a2cbc5f47 --network xxxxxxxxxxxx --description "GCE L7 firewall rule" --allow tcp:30000-32767 --source-ranges 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 --target-tags gke-node --project xxxx```

[bluemalkin]

@bowei yes shared VPC

[grayluck]

Thanks for the event messages.
Tom did you create your own firewall rule using the command that we give?
Do you still have the command that you run to create your firewall rule? If not, could you show me the "Equivalent REST" of the firewall rule from gcloud console?
You said you have "custom tags" in your firewall rule. I suppose the firewall rules need to be exactly the same as suggested to convince GKE that the resource is managed.

If you are using custom tags, one solution to keep you from spamming firewall change events is to create a duplicate firewall rule right as suggested, meanwhile have your own firewall rule overriding it.

[bluemalkin]

The reason why I do not run the commands provided is because I automate my whole infrastructure with Terraform.
I have a global firewall rule which custom tags I've set for my GKE clusters. This is created well before the GKE clusters are created. It does not make any sense to wait for the clusters to be created, get the random node IDs then add the rule.

gcloud compute firewall-rules describe --project xxxxxxxx common-gke-in-l7-lb
allowed:
- IPProtocol: tcp
  ports:
  - 30000-32767
creationTimestamp: 'xxxxxxxx'
description: Allow Google Health Checks on L7 lbs
direction: INGRESS
disabled: false
id: 'xxxxxxxx'
kind: compute#firewall
name: common-gke-in-l7-lb
network: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/global/networks/scg-net-stg-0
priority: 1
selfLink: https://www.googleapis.com/compute/v1/projects/xxxxxxxx/global/firewalls/common-gke-in-l7-lb
sourceRanges:
- 209.85.204.0/22
- 209.85.152.0/22
- 130.211.0.0/22
- 35.191.0.0/16
targetTags:
- uat-api
- dev-api

Here's the terraform code:

resource "google_compute_firewall" "common-gke-in-l7-lb" {
  name        = "common-gke-in-l7-lb"
  description = "Allow Google Health Checks on L7 lbs"
  network     = "${google_compute_network.default.name}"
  priority    = "001"
  direction   = "INGRESS"

  allow {
    protocol = "tcp"
    ports    = ["30000-32767"]
  }

  source_ranges = [
    "209.85.152.0/22",
    "209.85.204.0/22",
    "35.191.0.0/16",
    "130.211.0.0/22",
  ]

  target_tags = ["dev-api", "uat-api"]
}

[grayluck]

If this really bothers you, you can simply grant compute.networkAdmin and compute.securityAdmin to your GKE service account of the service project.
https://cloud.google.com/vpc/docs/shared-vpc#net_and_security_admins

I am trying to find a better way to tackle this issue. It may take some time.

---

There might be a way to manually set the node-tags even before they are created.

ingress-gce/deploy/glbc/gce.conf

Line 7 in b04a367

node-tags = NETWORK TAGS FOR YOUR CLUSTER'S INSTANCE GROUP

[bluemalkin]

Thanks, giving Admin permissions to the cluster is not going to go down well.

Looking forward on how to silent those messages, perhaps a global flag in Ingress annotations to completely ignore firewall management ?

[boredabdel]

Hi,

I come across this while digging the same issue and have a comment, i don't think "granting compute.networkAdmin and compute.securityAdmin to your GKE service account of the service project." is the solution for this as it defeats the whole purpose of SharedVPC (separating network admin and security privileges from user privileges). IMO this creates a security loop whole, So proving a way to silence those alerts is the way to go.

Cheers

[grayluck]

@abdelsource Yes I couldn't agree more.
I personally don't recommend granting security permissions. This is just a workaround for test projects, not for productions.
#506 is the solution.

[rramkumar1]

/kind bug

[m1kola]

Hi @grayluck. Thanks for working on this.

Is it available on 1.11.2-gke.18? I see your PR merged and I see it in docs here, but it doesn't seem to suppress these events for me.

Has the google cloud documentation been updated too early or am I doing something wrong? Here is an example of the Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.global-static-ip-name: some-static-ip-name
    networking.gke.io/suppress-firewall-xpn-error: true
  name: app
  namespace: some-namespace
spec:
  backend:
    serviceName: app
    servicePort: 80
  tls:
  - hosts:
    - some.host.com
    secretName: some-tls-secret

[rramkumar1]

@m1kola Unfortunately, the docs were updated before this annotation was rolled out on GKE.

Due to some delays outside of our control, this will only be rolling out next week.

[rramkumar1]

This is now available in version at or above 1.11.3-gke.18.

/close

[k8s-ci-robot]

@rramkumar1: Closing this issue.

In response to this:

This is now available in version at or above 1.11.3-gke.18.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bluemalkin]

Unfortunately this annotation doesn't stop the service from trying to grant the firewall permission.

networking.gke.io/suppress-firewall-xpn-error: "true" just removes the Kubernetes event.

But in Cloud Logging, I see an error every 10 minutes Required 'compute.firewalls.create' permission for 'projects/xxxx/global/firewalls/k8s-fw-l7--xxxx

Is there a way to totally stop this entirely with a shared VPC ?

[boredabdel]

The annotation you mentioned above only suppressed the warning when you kubectl describe the ingress.

The ingress controller continues to try to create the required firewall rules (Share vpc or not) there is no option to turn that off.

Since the ingress controller is not getting any new features anymore i don't think we will release a way to turn off firewall rules creation anytime soon.

The Good news is that the logs you see in the shared vpc host project are audit logs, you don't pay for those. But i understand they are noisy. I suggest you just filter them out when you are looking at logs

[bluemalkin]

Since the ingress controller is not getting any new features anymore i don't think we will release a way to turn off firewall rules creation anytime soon.

Thanks - is there a replacement one ?

[boredabdel]

The main focus is on the new Gateway API and Gateway Controller https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api.

But it doesn't have all the features our ingress controller has yet. It's a work in progress