IP based netpols can't select in cluster resources with Cilium CNI

[jacobbmay]

Environment

Device and OS: RHEL 8
App version: 0.28.0-registry1
Kubernetes distro being used: eks-d
Other: Using Cilium for CNI

Steps to reproduce

1. Deploy uds core to cluster using Cilium for CNI
2. Attempt to log into Grafana using SSO

Expected result

User is successfully logged into Grafana using keycloak

Actual Result

Grafana throws error trying to get a token from Keycloak

Visual Proof (screenshots, videos, text, etc)

logger=authn.service t=2024-10-07T17:41:18.145089994Z level=error msg="Failed to authenticate request" client=auth.client.generic_oauth error="[auth.oauth.token.exchange] failed to exchange code to token: Post \"http://keycloak-http.keycloak.svc.cluster.local:8080/realms/uds/protocol/openid-connect/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Severity/Priority

High

Cause of the issue identified

Cilium can not select in cluster resources using CIDR based selectors in netpols. Currently the allow-keycloak-ingress-keycloak-backchannel-access netpol is using a cidr based policy to match 0.0.0.0/0, which means Grafana (and any other apps that attempt to access keycloak directly via its internal service hostname) will timeout when running in an environment using Cilium for CNI.

This should be updated to include a namespace selector that matches all namespaces instead of or in addition to the 0.0.0.0/0 policy. Have confirmed that adding - namespaceSelector: {} to the from selector resolves the issue.

Additional context

This issue will affect anything relying on network policies that use an IP selector to allow ingress or egress to in cluster resources, not just keycloak backchannel access. So there may be other netpols that should be updated in the same way. The issue was just identified with Grafana SSO.

[mjnagel]

I think our intent with that specific policy was to only apply to in-cluster resources so making the swap (removing Anywhere and switching to namespaceSelector: {}) makes sense to me.

We might also/alternatively want to modify our anywhere endpoint to have both the CIDR and namespaceSelector: {} which would nicely handle all the other cases you mentioned that could be floating out there but are unknown.

[jacobbmay]

I think updating the anywhere endpoint to have both makes sense. Would resolve this issue for Cilium users and still allow traffic from anywhere external the same way that it currently does.

That specific policy does seem like it should only allow access in cluster though.