Fix #2703: replace Calico with Cilium CNI

.github/workflows/test-integration.yml

@@ -42,21 +42,21 @@ jobs:
 
       - name: Set up k3s
         # Flake guard: k3s occasionally becomes unstable during the
-        # calico-node DaemonSet rollout (API server refuses connections
-        # before the 300s timeout expires).  One restart clears the
-        # transient state; the idempotent install-calico.sh is safe to
-        # re-run after k3s recovers.
+        # CNI DaemonSet rollout (API server refuses connections before
+        # the 300s timeout expires).  One restart clears the transient
+        # state; the idempotent install-cilium.sh is safe to re-run
+        # after k3s recovers.
         run: |
           curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --write-kubeconfig-mode=644" sh -
           export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
           echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> "$GITHUB_ENV"
-          # Install Calico CNI (retry once if k3s becomes unstable mid-rollout)
-          if ! scripts/install-calico.sh; then
-            echo "Calico install failed — restarting k3s and retrying..."
+          # Install Cilium CNI (retry once if k3s becomes unstable mid-rollout)
+          if ! scripts/install-cilium.sh; then
+            echo "Cilium install failed — restarting k3s and retrying..."
             sudo systemctl restart k3s
             echo "Waiting for k3s API to recover..."
             timeout 60 sh -c 'until kubectl cluster-info 2>/dev/null; do sleep 5; done'
-            scripts/install-calico.sh
+            scripts/install-cilium.sh
           fi
           # Wait for node to be ready
           kubectl wait --for=condition=Ready node --all --timeout=120s

Makefile

@@ -74,7 +74,7 @@ help:
 	@echo "  make build              - Build Docker images"
 	@echo ""
 	@echo "Kubernetes (k3s):"
-	@echo "  make k3s-setup          - Install k3s with Calico CNI"
+	@echo "  make k3s-setup          - Install k3s with Cilium CNI"
 	@echo "  make deploy             - Deploy egg to k3s"
 	@echo "  make redeploy           - Rebuild, re-import, and redeploy in one step"
 	@echo "  make k3s-import         - Import built images into k3s"
@@ -473,11 +473,15 @@ build: sync-venv-if-uv
 # Kubernetes (k3s) targets
 # ============================================================================
 
-k3s-setup:  ## Install k3s with Calico CNI
+# k3s-setup INSTALL_K3S_EXEC flags:
+#   --flannel-backend=none: Cilium replaces flannel as the CNI dataplane.
+#   --disable-network-policy: Cilium owns NetworkPolicy enforcement; the
+#     k3s-builtin policy controller would otherwise conflict.
+k3s-setup:  ## Install k3s with Cilium CNI
 	@echo "Setting up k3s cluster..."
 	curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --write-kubeconfig-mode=644" sh -
 	export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && \
-	scripts/install-calico.sh && \
+	scripts/install-cilium.sh && \
 	echo "Waiting for k3s node to be ready..." && \
 	kubectl wait --for=condition=Ready node --all --timeout=120s
 	@echo "k3s cluster ready"

docs/architecture/kubernetes-migration.md

@@ -49,7 +49,7 @@ k3s Cluster
 │   ├── Job: agent-tester-{pipeline-id}
 │   └── Job: agent-documenter-{pipeline-id}
 │
-└── NetworkPolicies (Calico CNI)
+└── NetworkPolicies (Cilium CNI)
     ├── default-deny-all (egg-agents ingress + egress)
     ├── allow-agent-to-gateway (egress to egg-system/gateway only)
     └── allow-orchestrator-to-agents (ingress from egg-system/orchestrator)
@@ -61,7 +61,7 @@ k3s Cluster
 |---|----------|--------|-----------|
 | 1 | Manifest approach | **Kustomize overlays** | YAML-native, built into kubectl, no Helm templating complexity. `base/` + `overlays/local/` structure |
 | 2 | Network isolation | **Separate namespaces + NetworkPolicies** | `egg-system` for orchestrator+gateway, `egg-agents` for Jobs. Default-deny maps to Docker's `internal: true` |
-| 3 | CNI | **Calico** (replacing Flannel) | Flannel (k3s default) does not support NetworkPolicies. Calico is mature and well-documented for k3s |
+| 3 | CNI | **Cilium** (replacing Flannel) | Flannel (k3s default) does not support NetworkPolicies. Cilium is current; was Calico through [#2703](https://github.com/jwbron/egg/issues/2703), which swapped to Cilium after recurring on-host SA-token expiry wedged pod teardown ([#2580](https://github.com/jwbron/egg/issues/2580)) |
 | 4 | Persistent storage | **hostPath** (k3s local-path) | Standard for single-node k3s. Future GKE work uses PVCs with ReadWriteMany |
 | 5 | Agent primitive | **k8s Jobs** | Agents run to completion; exit codes matter. `backoffLimit: 0` prevents unwanted restarts. `activeDeadlineSeconds` replaces timeout mechanism |
 | 6 | Gateway auth | **Token-only** (IP binding removed) | Pod IPs are ephemeral in k8s. Token auth is simpler and more portable |
@@ -79,7 +79,7 @@ k3s Cluster
 | `DockerClient` | `KubernetesClient` | Wraps `kubernetes` Python client |
 | `ContainerSpawner` | `KubernetesSpawner` | Creates Jobs with env vars, volumes, labels |
 | `ContainerMonitor` | `KubernetesMonitor` | Uses Job watch API / polling |
-| Docker networks (`egg-isolated`, `egg-external`) | Namespaces + Calico NetworkPolicies | See [Network Isolation](#network-isolation) |
+| Docker networks (`egg-isolated`, `egg-external`) | Namespaces + NetworkPolicies (Cilium) | See [Network Isolation](#network-isolation) |
 | Docker `internal: true` | NetworkPolicy default-deny egress | Agents cannot reach internet directly |
 | Container labels | Pod/Job labels + label selectors | Same filtering model |
 | Docker bind mounts | hostPath volumes | Same for single-node; PVCs for multi-node |
@@ -151,7 +151,7 @@ Namespace: egg-system          Namespace: egg-agents
 └──────────────┘
 ```
 
-**NetworkPolicies (enforced by Calico):**
+**NetworkPolicies (enforced by Cilium):**
 
 | Policy | Namespace | Effect |
 |--------|-----------|--------|
@@ -236,7 +236,7 @@ k8s/
 │   ├── orchestrator-service.yaml      # Service on port 9849
 │   ├── gateway-deployment.yaml        # Gateway Deployment + env
 │   ├── gateway-service.yaml           # Service on ports 9848, 3129, 9851
-│   ├── network-policies.yaml          # Calico NetworkPolicies
+│   ├── network-policies.yaml          # NetworkPolicies (Cilium-enforced)
 │   └── rbac.yaml                      # ServiceAccount + RBAC for orchestrator
 
 Agent Jobs are built programmatically by ``KubernetesClient.create_container`` — there is no standalone YAML template.
@@ -293,8 +293,8 @@ egg --public                  # Start sandbox session
 ### After
 
 ```bash
-# Prerequisites: k3s + Calico CNI
-make k3s-setup                # Install k3s with Calico, wait for ready
+# Prerequisites: k3s + Cilium CNI
+make k3s-setup                # Install k3s with Cilium, wait for ready
 make deploy                   # kubectl apply -k k8s/overlays/local/
 egg --public                  # Start sandbox session (creates k8s Job)
 ```
@@ -303,7 +303,7 @@ egg --public                  # Start sandbox session (creates k8s Job)
 
 | Target | Description |
 |--------|-------------|
-| `make k3s-setup` | Install k3s with `--flannel-backend=none --disable-network-policy`, install Calico, wait for cluster ready |
+| `make k3s-setup` | Install k3s with `--flannel-backend=none --disable-network-policy`, install Cilium, wait for cluster ready |
 | `make deploy` | `kubectl apply -k k8s/overlays/local/` — deploy all resources |
 | `make k3s-teardown` | Remove k3s installation |
 | `make build` | Build images and import into k3s via `k3s ctr images import` |
@@ -314,10 +314,10 @@ k3s ships with Flannel which does **not** support NetworkPolicies. k3s must be i
 
 ```bash
 curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy" sh -
-kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.5/manifests/calico.yaml
+scripts/install-cilium.sh   # downloads cilium-cli and runs `cilium install`
 ```
 
-This is automated by `make k3s-setup` and `scripts/install-calico.sh`.
+This is automated by `make k3s-setup` and `scripts/install-cilium.sh`.
 
 ## CI/CD Changes
 

docs/architecture/network-isolation.md

@@ -56,7 +56,7 @@ Specific threats:
 │  │                               │ │                               │         │
 │  └───────────────────────────────┘ └───────────────────────────────┘         │
 │                                                     │                        │
-│  NetworkPolicies (Calico):                          │ All traffic proxied    │
+│  NetworkPolicies (Cilium):                          │ All traffic proxied    │
 │  - Default deny ingress/egress                      ▼                        │
 │  - Allow egress to gateway only         ┌─────────────┐                     │
 │                                         │  Internet   │                     │
@@ -75,7 +75,7 @@ Specific threats:
 | HTTP Proxy | Route all agent traffic through gateway | Squid in gateway pod |
 | REST API | Controlled interface for git/gh operations | Python service in gateway pod |
 | Audit Logger | Log all traffic and operations | Gateway component |
-| NetworkPolicies | Enforce network isolation | Calico CNI in k3s |
+| NetworkPolicies | Enforce network isolation | Cilium CNI in k3s |
 
 ### Key Security Properties
 
@@ -553,11 +553,11 @@ Both options preserve the key principle — credentials never enter the egg cont
 
 ## Kubernetes Network Isolation
 
-> **As of [#1553](https://github.com/jwbron/egg/issues/1553)**, the container runtime has migrated from Docker to Kubernetes (k3s). The network isolation model is preserved using Calico NetworkPolicies instead of Docker networks.
+> **As of [#1553](https://github.com/jwbron/egg/issues/1553)**, the container runtime has migrated from Docker to Kubernetes (k3s). The network isolation model is preserved using Cilium NetworkPolicies instead of Docker networks. (Cilium replaced Calico in [#2703](https://github.com/jwbron/egg/issues/2703); the policies themselves use only standard `networking.k8s.io/v1` features and are CNI-agnostic.)
 
 ### Architecture
 
-The Docker dual-network model (`egg-isolated` + `egg-external`) is replaced by Kubernetes namespace separation with Calico NetworkPolicies:
+The Docker dual-network model (`egg-isolated` + `egg-external`) is replaced by Kubernetes namespace separation with NetworkPolicies (enforced by Cilium):
 
 | Docker Concept | Kubernetes Equivalent |
 |---------------|----------------------|
@@ -655,14 +655,14 @@ spec:
 
 ### CNI Requirement
 
-**Calico is required.** k3s ships with Flannel as default CNI. Flannel does **not** support NetworkPolicies. k3s must be installed with Flannel disabled:
+**A NetworkPolicy-capable CNI is required.** k3s ships with Flannel as the default CNI; Flannel does **not** support NetworkPolicies. egg uses **Cilium** — k3s is installed with Flannel and its built-in policy controller both disabled:
 
 ```bash
 curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy" sh -
-kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.5/manifests/calico.yaml
+scripts/install-cilium.sh   # downloads the cilium-cli and runs `cilium install`
 ```
 
-This is handled automatically by `make k3s-setup`.
+This is handled automatically by `make k3s-setup`. The CNI choice is local to the install path — the policies in `k8s/base/network-policies.yaml` are standard `networking.k8s.io/v1` and work on any NetworkPolicy-capable CNI (Calico, Cilium, Antrea, Weave, kube-router).
 
 ### Security Properties Preserved
 
@@ -686,7 +686,7 @@ For full migration details, see [Kubernetes Migration](kubernetes-migration.md).
 
 | Component | Local (k3s) | GCP (GKE) | GCP (Cloud Run) |
 |-----------|-------------|-----------|-----------------|
-| Network isolation | Calico NetworkPolicies | GKE NetworkPolicies (Dataplane V2) | VPC Service Controls |
+| Network isolation | Cilium NetworkPolicies | GKE NetworkPolicies (Dataplane V2) | VPC Service Controls |
 | Gateway sidecar | k8s Deployment + Service | Same | Cloud Run sidecar |
 | Audit logs | File/stdout | Cloud Logging | Cloud Logging |
 | Proxy | Squid in gateway pod | Same or Serverless VPC | Same or Serverless VPC |

docs/architecture/orchestrator.md

@@ -596,7 +596,7 @@ EGG_AGENT_ROLE=coder  # or tester, documenter
 
 ### Network Architecture
 
-All components communicate over Kubernetes networking with namespace-based isolation enforced by Calico NetworkPolicies:
+All components communicate over Kubernetes networking with namespace-based isolation enforced by NetworkPolicies (Cilium today):
 
 | Namespace | Purpose | Components |
 |-----------|---------|------------|
@@ -608,7 +608,7 @@ Service endpoints:
 - Orchestrator: `orchestrator.egg-system.svc.cluster.local` (port 9849)
 - Agent pods: Addressed by label selector (`pipeline-id`, `agent-role`)
 
-NetworkPolicies (enforced by Calico CNI):
+NetworkPolicies (enforced by Cilium CNI):
 - Default-deny all ingress in `egg-agents` — agents cannot receive unsolicited traffic
 - Default-deny all egress in `egg-agents` — agents cannot reach internet directly
 - Allow agent egress to gateway Service only — preserves the gateway-as-single-choke-point model

docs/development/STRUCTURE.md

@@ -191,7 +191,7 @@ k8s/
 │   ├── orchestrator-service.yaml      # Service exposing port 9849
 │   ├── gateway-deployment.yaml        # Gateway Deployment + environment config
 │   ├── gateway-service.yaml           # Service exposing ports 9848, 3129, 9851
-│   ├── network-policies.yaml          # Calico NetworkPolicies for agent isolation
+│   ├── network-policies.yaml          # NetworkPolicies for agent isolation (Cilium-enforced)
 │   └── rbac.yaml                      # ServiceAccount + Role + RoleBinding for orchestrator
 (Agent Job specs are built programmatically by ``KubernetesClient.create_container`` — no standalone YAML template.)
 │

docs/guides/deployment.md

@@ -8,7 +8,7 @@ egg supports two deployment methods depending on your use case:
 
 | Method | Best For | Prerequisites |
 |--------|----------|---------------|
-| **Kubernetes (k3s)** via `bin/egg-deploy` | Local development and production deployments | k3s + Calico CNI |
+| **Kubernetes (k3s)** via `bin/egg-deploy` | Local development and production deployments | k3s + Cilium CNI |
 | **GitHub Action** | CI/CD automation | GitHub repository |
 
 > **Removal note:** The legacy `egg` CLI / `bin/egg` interactive mode
@@ -38,7 +38,7 @@ egg runs on Kubernetes using k3s for local development. The orchestrator and gat
 git clone https://github.com/jwbron/egg.git
 cd egg
 
-# Install k3s with Calico CNI
+# Install k3s with Cilium CNI
 make k3s-setup
 
 # Build and import images into k3s
@@ -55,16 +55,18 @@ kubectl get pods -n egg-system
 
 #### k3s Installation
 
-`make k3s-setup` installs k3s with Flannel disabled (required for NetworkPolicy support) and installs Calico CNI:
+`make k3s-setup` installs k3s with Flannel disabled (required for NetworkPolicy support) and installs Cilium CNI:
 
 ```bash
 # What make k3s-setup does:
 curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy" sh -
-kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.31.5/manifests/calico.yaml
+scripts/install-cilium.sh   # downloads cilium-cli and runs `cilium install`
 # Waits for cluster to become ready
 ```
 
-> **Why Calico?** k3s ships with Flannel as default CNI. Flannel does **not** support NetworkPolicies, which are required for agent network isolation. Calico replaces Flannel and enforces the NetworkPolicies that prevent agents from reaching the internet directly.
+> **Why Cilium?** k3s ships with Flannel as default CNI. Flannel does **not** support NetworkPolicies, which are required for agent network isolation. Cilium replaces Flannel and enforces the NetworkPolicies that prevent agents from reaching the internet directly. Calico filled this role until [#2703](https://github.com/jwbron/egg/issues/2703) — see that issue for the swap rationale.
+>
+> **Migrating from a pre-#2703 install:** in-place CNI swap on a live k3s cluster is not supported (host CNI binaries, conflists, CRDs, `tunl0`, and per-pod veth pairs persist after deleting the calico-node DaemonSet). Run `make k3s-teardown && make k3s-setup` for a clean install. `install-cilium.sh` will refuse if it detects leftover Calico state.
 
 #### Image Management
 
@@ -117,7 +119,7 @@ make build
 
 | Command | Description |
 |---------|-------------|
-| `make k3s-setup` | Install k3s + Calico CNI (idempotent) |
+| `make k3s-setup` | Install k3s + Cilium CNI (idempotent) |
 | `make deploy` | Deploy all k8s resources via Kustomize + `envsubst` (see [details below](#make-deploy-details)) |
 | `make build` | Build images and import into k3s |
 | `make k3s-teardown` | Remove k3s installation |
@@ -140,7 +142,7 @@ EGG_HOST_REPO_MAP='{"owner/repo":"/path"}' make deploy
 
 ### Network Topology
 
-Kubernetes uses namespace separation and Calico NetworkPolicies for network isolation:
+Kubernetes uses namespace separation and NetworkPolicies (enforced by Cilium) for network isolation:
 
 ```
 Namespace: egg-system                    Namespace: egg-agents

docs/reference/mcp-deployment-tools.md

@@ -106,7 +106,7 @@ Read-only cluster introspection.
   "kubeconfig_context": "default",
   "cluster_info": {"server_version": "v1.30.2+k3s1"},
   "namespace": "egg-system",
-  "cni": "calico",
+  "cni": "cilium",
   "network_policy_enforcement": true,
   "images": {
     "orchestrator": "egg-orchestrator:sha-abc1234",

integration_tests/test_deployment_validation_logic.py

@@ -34,8 +34,8 @@
   ``apps/daemonsets`` and ``nodes`` (ClusterRole
   ``egg-cluster-topology-reader``).** ``_detect_cni`` / ``_detect_k3s``
   now run, so ``validate-network-isolation`` launches the probe
-  against the Calico-equipped integration cluster instead of
-  short-circuiting.
+  against the Cilium-equipped integration cluster instead of
+  short-circuiting. (CNI swapped from Calico to Cilium in #2703.)
 * **#2648 — orchestrator SA gained ``list`` on ``apps/deployments``
   in ``egg-system``.** Tangential to #2641 but observed in the same
   audit: ``_collect_egg_image_tags`` now returns populated image tags
@@ -266,8 +266,8 @@ class TestValidateNetworkIsolationLogic:
 
     With #2646 fixed the orchestrator SA can now list ``kube-system``
     DaemonSets and ``nodes`` cluster-wide, so ``_detect_cni`` resolves
-    to ``("calico", True)`` against the integration cluster (which
-    installs Calico via ``scripts/install-calico.sh``). The probe Job
+    to ``("cilium", True)`` against the integration cluster (which
+    installs Cilium via ``scripts/install-cilium.sh``). The probe Job
     actually launches; the happy-path test below exercises its result
     shape. The earlier short-circuit assertion has been removed.
     """

orchestrator/mcp_tools.py

@@ -1121,7 +1121,7 @@ def _is_timeout_error(exc: BaseException) -> bool:
         "name": "validate_network_isolation",
         "description": (
             "Spawn a throwaway probe Job in the egg-agents namespace to verify "
-            "Calico NetworkPolicy enforcement. Returns a structured "
+            "NetworkPolicy enforcement (Cilium today; CNI-agnostic). Returns a structured "
             "{gateway_reachable, internet_blocked, agent_pods_unreachable, "
             "orchestrator_api_reachable} result. The route deletes the Job "
             "in a try/finally; ttlSecondsAfterFinished=30 is the backstop. "