GCS Storage bucket mount fails for single user pods on GKE Autopilot

[vizeit]

Bug description

Google Cloud Storage bucket can be mounted as a drive using GCS FUSE CSI driver. When a bucket is configured as a shared drive for the single user pods, the pod fails to start. Upon detailed investigation at my end, I see that current network policy for single user egress only allows to reach Google meta server IP 169.254.169.254 on port 53 but GCS FUSE CSI driver requires egress to the meta server IP on port 80. I am including the deny policy from the GCP logs here

The workaround is to add network policy to allow port 80 for the GCP meta server IP 169.254.169.254

Reference: Network policies were discussed earlier for my other issue 3167

NOTE: I have searched on Jupyter community forum for similar issues and did not find any relevant to this behavior

{
  "insertId": "knm845y9npaf5lsd",
  "jsonPayload": {
    "dest": {
      "instance": "169.254.169.254"
    },
    "node_name": "gk3-testcluster-nap-179rf7sk-0f06ef44-6ga5",
    "src": {
      "pod_name": "jupyter-test-user",
      "pod_namespace": "testclusterdev",
      "namespace": "testclusterdev"
    },
    "count": 2,
    "connection": {
      "dest_ip": "169.254.169.254",
      "dest_port": 80,
      "src_port": 42806,
      "direction": "egress",
      "src_ip": "177.25.24.532",
      "protocol": "tcp"
    },
    "disposition": "deny"
  },
  "resource": {
    "type": "k8s_node",
    "labels": {
      "node_name": "gk3-testcluster-nap-179rf7sk-0f06ef44-6ga5",
      "project_id": "my-project",
      "cluster_name": "testcluster",
      "location": "us-central1"
    }
  },
  "timestamp": "2023-10-08T00:50:04.350769546Z",
  "logName": "projects/my-project/logs/policy-action",
  "receiveTimestamp": "2023-10-08T00:50:11.632623407Z"
}

How to reproduce

1. Create GKE Autopliot cluser
2. Follow the steps from this GCP page to setup a pvc pointing to a GCS bucket
3. Configure Zero-to-JupyterHub helm chart for single user shared volume pointing to the pvc created in step 2
4. Install JupyterHub on the cluster created in step 1
5. Launch a user session

Expected behaviour

Single user pod should start successfully

Actual behaviour

Single user pod fails to start

Your personal set up

Helm chart version: 3.1.0

• OS:

• Version(s):

Full environment

# paste output of `pip freeze` or `conda list` here

Configuration

# jupyterhub_config.py

Logs

[manics]

Have you configured singleuser.networkPolicy.egressAllowRules.cloudMetadataServer and disabled singleuser.cloudMetadata.blockWithIptables?
https://z2jh.jupyter.org/en/stable/resources/reference.html#singleuser-networkpolicy-egressallowrules-cloudmetadataserver ?

If that's not working please fill in all the information requested in the issue template, especially your full configuration

[vizeit]

Please find the full template with supplied configuration

singleuser:
  cloudMetadata:
    blockWithIptables: false
  extraAnnotations:
    gke-gcsfuse/volumes: "true"
  storage:
    extraVolumeMounts:
    - mountPath: /home/jovyan/shared
      name: shareddata
    extraVolumes:
    - name: shareddata
      persistentVolumeClaim:
        claimName: shared-bucket-pvc

[vizeit]

This is not an issue. I added the following entry and it worked

singleuser:
  networkPolicy:
    egressAllowRules:
      cloudMetadataServer: true

[vizeit]

I have described detailed steps in my post if anyone wants to fully setup GCS bucket as a shared drive with zero-to-jupyterhub

https://www.vizeit.com/gcs-bucket-with-jupyterhub-on-gke/