fix(network): allow CoreDNS replies through the database ingress allowlist

[Tanguille]

Root cause

database-ingress-allowlist (#4072) sets endpointSelector: {}, making ingress default-deny for every pod in database. The allowlist omits kube-system, so Cilium drops rev-NAT'd CoreDNS replies. Captured live on control-3, endpoint 1100 = pgbouncer-rw-7ff5f7bdb4-rfwgp:

drop (Policy denied) bpf_lxc.c:2472, identity 19145->9091:
  10.43.0.10:53 -> 10.42.2.107:38066 tcp SYN, ACK

cilium-dbg identity get 19145 → k8s-app=kube-dns, namespace=kube-system.

A reply from a pod IP matches the outbound conntrack entry and bypasses policy evaluation. Only the rev-NAT'd ClusterIP reply is re-evaluated as a new ingress flow carrying the CoreDNS identity. That is why direct-to-pod DNS succeeds in ~1ms while the ClusterIP times out on both UDP and TCP.

Cilium cannot match a source port (#19965) — the reply's destination is the client's ephemeral port — so toPorts: [53] would match the wrong field and silently do nothing. Full-port ingress from the kube-dns identity is the floor.

Why only some pods

Service backend pinning. rfwgp is pinned to BackendID 994 = 10.42.2.109, the CoreDNS pod on its own node; conntrack shows TxFlagsSeen=0x02 RxFlagsSeen=0x00 Packets=0 — the reply dies at ingress before it can ever update conntrack. Poolers pinned to a remote backend are unaffected, which is why deleting the pod has always looked like a fix.

Measured:

pooler	node	CoreDNS on node	DNS failures / 5m
9jvml	control-2	no	0
nl8x9	control-1	yes	61
rfwgp	control-3	yes ×2	38

Blast radius

crowdsec-lapi CrashLoopBackOff on cscli machines add; radarr/sonarr/prowlarr/bazarr taking server_login_retry through pgbouncer-rw; the CNPG primary and operator intermittently losing DNS.

Scope

Cluster-scoped policy object only. Zero pods restarted, instantly reversible. The #4135 (use-vc) revert was originally bundled here and has been split into #4149 — it mutates template.spec and would roll all 7 pooler pods, which would both inherit rollout risk into an urgent fix and destroy this PR's verification (see below).

Reviewed and rejected

An earlier revision also added host, remote-node, health to fromEntities to protect kubelet probes on a future worker node. Dropped — AllowLocalhost: always is Cilium's runtime default under k8s and already inserts an unconditional host-identity ingress allow into every endpoint's policy map, verified on a namespace with no CNP granting it. Kubelet only probes pods on its own node, so probes always arrive as the local host identity. health targets node health endpoints, never app pods.

Follow-ups, not in this PR

• fromEntities: [kube-apiserver] resolves to identity 7 = {reserved:kube-apiserver, reserved:remote-node} on this collapsed control plane, so it already grants every host-namespace process on every other node ANY-port ingress to every database pod. Scoping to toPorts: 9443 (cnpg webhook) would tighten it; counters support it (identity 7 → postgres = 0 bytes).
• Hubble is disabled (hubble.enabled: false). feat(kube-system): add cilium network segmentation policies #4072's own body specified verification via hubble observe --namespace database --verdict DROPPED — a command that has never been runnable in this cluster. Agent-only Hubble (no relay, no UI) is the cheap fix and would have surfaced this drop immediately.

Verification

kustomize build renders clean. kube-system/network-policies is not suspended and reports Applied revision: main@bc287c014, so this applies on merge.

Post-merge: policy drops for identity 19145 stop, and the poolers recover without deletion. That last point is the signal — do not merge #4149 until it is confirmed, since a pooler roll would reset the pinned backends and make the result unreadable.

Summary by CodeRabbit

• Bug Fixes
  • Updated database network access rules to allow DNS traffic from the cluster’s DNS service.
  • Improved DNS resolution reliability for applications connecting to the database.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b607c27a-3bd3-4dc0-bea6-921365caa11b

📥 Commits

Reviewing files that changed from the base of the PR and between bc287c0 and e3d4302.

📒 Files selected for processing (1)

• kubernetes/apps/kube-system/network-policies/app/database-ingress.yaml

---

📝 Walkthrough

Walkthrough

The database ingress CiliumNetworkPolicy now permits DNS traffic from kube-dns pods in the kube-system namespace and documents reverse-NAT behavior and the absence of port-specific filtering.

Changes

Database DNS ingress

Layer / File(s)	Summary
Allow kube-dns ingress kubernetes/apps/kube-system/network-policies/app/database-ingress.yaml	Adds a kube-dns source selector for database ingress and explains why port matching is omitted for DNS replies.

Estimated code review effort: 1 (Trivial) | ~2 minutes

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/database-cnp-coredns-ingress

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Checkov (3.3.8)

kubernetes/apps/kube-system/network-policies/app/database-ingress.yaml

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

---

_{Comment @coderabbitai help to get the list of available commands.}

[tanguille-cluster]

@@ spec.ingress @@
# cilium.io/v2/CiliumNetworkPolicy/database/database-ingress-allowlist
! - one list entry removed:
- - fromEndpoints:
-   - matchLabels:
-       io.kubernetes.pod.namespace: ai
-   - matchLabels:
-       io.kubernetes.pod.namespace: default
-   - matchLabels:
-       io.kubernetes.pod.namespace: media
-   - matchLabels:
-       io.kubernetes.pod.namespace: observability
-   - matchLabels:
-       io.kubernetes.pod.namespace: security
-   - matchLabels:
-       io.kubernetes.pod.namespace: database
! + one list entry added:
+ - fromEndpoints:
+   - matchLabels:
+       io.kubernetes.pod.namespace: ai
+   - matchLabels:
+       io.kubernetes.pod.namespace: default
+   - matchLabels:
+       io.kubernetes.pod.namespace: media
+   - matchLabels:
+       io.kubernetes.pod.namespace: observability
+   - matchLabels:
+       io.kubernetes.pod.namespace: security
+   - matchLabels:
+       io.kubernetes.pod.namespace: database
+   - matchLabels:
+       io.kubernetes.pod.namespace: kube-system
+       k8s-app: kube-dns