Fix #2703: replace Calico with Cilium CNI

[jwbron]

Summary

• Replaces Calico CNI with Cilium to eliminate the recurring on-host SA-token wedge tracked as Calico CNI kubeconfig token expires, wedges pod teardown every ~24–48h #2580. Calico's install-cni init container writes /etc/cni/net.d/calico-kubeconfig once with the projected init-container SA token and never refreshes it; after ~24–48h every CNI add/del returns Unauthorized. Cilium has no equivalent failure mode — its agent runs as a pod with an in-pod projected token that kubelet auto-rotates, and CNI plugin invocations talk to the local agent over a Unix socket, not the apiserver.
• Closes Replace Calico with Cilium to eliminate recurring CNI auth-token wedge #2703. Supersedes Calico CNI kubeconfig token expires, wedges pod teardown every ~24–48h #2580 (the periodic-calico-node-restart workaround is removed, not introduced).
• One PR by design — cohesive swap, no dual-pathing, per the issue's acceptance criteria.

What changed

• scripts/install-cilium.sh (new): downloads cilium-cli v0.19.2 (SHA-pinned per arch), runs cilium install --version v1.19.4, waits via cilium status --wait. Refuses if Calico DaemonSet or CRDs are present — in-place CNI swap on a live cluster is not safe (host CNI binaries, conflists, CRDs, tunl0, and per-pod veth pairs persist after deleting calico-node). Post-install asserts only Cilium's conflist remains in /etc/cni/net.d/.
• scripts/install-calico.sh: deleted.
• Makefile k3s-setup: swapped script reference + help text. The k3s flags --flannel-backend=none --disable-network-policy are unchanged and now annotated as CNI-agnostic (Cilium also owns NetworkPolicy enforcement and replaces flannel).
• .github/workflows/test-integration.yml: swapped install-calico.sh → install-cilium.sh; kept the "restart k3s once on rollout-flake" retry guard with updated comments.
• orchestrator/routes/deployment.py: no change. _NETWORK_POLICY_CNIS already included "cilium", and _detect_cni already recognized Cilium DaemonSet names. Calico stays in the multi-CNI detection map (intentional — multi-CNI support is the contract).
• orchestrator/mcp_tools.py: validate_network_isolation help text generalized.
• Test fixtures: _detect_cni mocks and assertions in test_deployment_routes.py (7 sites), test_mcp_tools.py, test_deployment_validation_logic.py, test_select_tests_fallbacks.py updated to "cilium".
• Docs: network-isolation.md, kubernetes-migration.md, deployment.md, orchestrator.md, STRUCTURE.md, mcp-deployment-tools.md, agent-diagnose/SKILL.md, deployment-diagnose/SKILL.md. deployment.md adds an explicit migration note: pre-Replace Calico with Cilium to eliminate recurring CNI auth-token wedge #2703 installs must run make k3s-teardown && make k3s-setup.

What is unchanged

• k8s/base/network-policies.yaml — all six policies are pure networking.k8s.io/v1. No projectcalico.org/v3 CRDs anywhere in the repo.
• validate_network_isolation probe — tests that enforcement happens, not which CNI enforces it.
• _detect_cni multi-CNI matrix — kept as-is.
• .egg-state/ historical drafts/brc-history — preserved as point-in-time artifacts of past decisions.

What is out of scope

Per the issue: Cilium-only features (Hubble, ClusterMesh, L7, mutual auth) and production/GKE alignment.

Also deferred: kube-proxy replacement. It would require --disable-kube-proxy in INSTALL_K3S_EXEC (separate risk surface) and kernel feature checks; the failure mode is "services silently don't resolve." kube-proxy isn't broken today — this is an optimization, not a loose end. Can ship later as its own change.

Test plan

• make k3s-teardown && make k3s-setup on the Asahi dev box → verify cluster ready, kubectl get crd | grep projectcalico empty, sudo ls /etc/cni/net.d/ shows only 05-cilium.conflist, ip link shows no cali*/tunl0.
• make deploy → orchestrator + gateway reach Available.
• Spawn an agent via run_agent_task, then cancel_task → no FailedKillPod / stuck-Terminating (the Calico CNI kubeconfig token expires, wedges pod teardown every ~24–48h #2580 symptom path).
• integration_tests/test_network_isolation.py passes — exercises agent→gateway, agent→orchestrator, default-deny.
• validate_network_isolation MCP tool returns cni: cilium, enforcement: true.
• .github/workflows/test-integration.yml green on this PR.

No 48h soak — the failure mode is structurally absent in Cilium (no host-written, never-refreshed SA token file), so waiting to "verify" the absence of a structural property would be theater.

[egg-reviewer]

No agent-mode design concerns. This PR is a CNI infrastructure swap (Calico → Cilium) — scripts, Makefile, CI workflow, docs, and test-fixture string updates. The only agent-adjacent touches are a generalized validate_network_isolation tool description and two SKILL.md wording updates from Calico-specific to CNI-agnostic phrasing, both of which improve (rather than constrain) agent flexibility.

— Authored by egg

[egg-reviewer]

Review of #2704 — replace Calico with Cilium CNI

Overall the CNI swap is mechanically sound: _detect_cni already covers cilium, the NetworkPolicies are pure networking.k8s.io/v1, and test fixtures are updated consistently. The PR description, deferred-scope reasoning, and migration note are all clean. One blocking bug, plus a couple of robustness suggestions.

---

Blocking

1. install-cilium.sh:53 — Calico CRD detection regex never matches.

if kubectl get crd 2>/dev/null | grep -q '\.crd\.projectcalico\.org$'; then

Default kubectl get crd output is tabular with NAME and CREATED AT columns:

NAME                                              CREATED AT
bgpconfigurations.crd.projectcalico.org           2024-01-15T22:10:30Z
bgppeers.crd.projectcalico.org                    2024-01-15T22:10:30Z

Each line ends with the timestamp (Z), not with .crd.projectcalico.org. The $ end-of-line anchor therefore never matches, so this check is non-functional — Calico CRDs left over from a partial cleanup (e.g. operator manually deleted the calico-node DaemonSet but did not strip the CRDs) will pass straight through, the script proceeds, and Cilium gets installed on top of residual Calico state. This is exactly the scenario the PR description advertises as protected ("Refuses if Calico DaemonSet or CRDs are present"), and is the same class of silent operator-misconfiguration failure the rest of the script is careful to avoid.

The matching DaemonSet check on line 46 (kubectl get daemonset -n kube-system calico-node) does work because kubectl get for a named resource is a binary "exists / not exists" call rather than a text scrape.

Fix: use -o name (which produces one resource per line, terminating with the CRD name) — or awk the first column off the tabular form:

if kubectl get crd -o name 2>/dev/null | grep -q '\.projectcalico\.org$'; then

I'd also recommend a quick local sanity test (kubectl get crd -o name | grep '\.projectcalico\.org$' on a real Calico-installed cluster, or even a unit-style test feeding a captured kubectl get crd output through the same shell pipeline) — the current regex looks plausible enough that "I'll bump CALICO_VERSION later and the check protects me" is a trap that's easy to walk into.

---

Non-blocking suggestions

2. install-cilium.sh:125-144 — empty $CNI_DIR silently passes verification.

The post-install CNI verification only runs the "is cilium config present?" assertion when $CNI_FILES is non-empty:

if [ -n "$CNI_FILES" ]; then
    if echo "$CNI_FILES" | grep -qi calico; then ...; fi
    if ! echo "$CNI_FILES" | grep -qi cilium; then ...; fi
fi

If /etc/cni/net.d exists but is empty (a real failure mode if cilium install returns 0 but the agent fails to drop its conflist — the comment two lines above explicitly calls out this silent-failure class), the entire inner block is skipped and the script claims success. cilium status --wait would probably have caught this upstream, but the verification block exists precisely to belt-and-braces that. Worth flipping the structure to error on empty-dir too:

if [ -z "$CNI_FILES" ]; then
    error "No CNI configs found in ${CNI_DIR} after install — kubelet has no CNI to use."
    exit 1
fi

3. install-cilium.sh:148 — k8s-app=cilium selector relies on legacy label.

kubectl get pods -n kube-system -l k8s-app=cilium works on Cilium v1.19.x because the agent DaemonSet still sets the legacy k8s-app=cilium label, but the canonical selector since v1.15 is app.kubernetes.io/name=cilium-agent. Not blocking today, but a near-term Cilium bump can silently turn this last "Cilium pod status:" line into an empty table without failing anything else. Pinning to the canonical label avoids that drift.

4. Makefile:478-480 — comment lines inside a recipe are echoed.

The # --flannel-backend=none: … block sits inside the recipe (after the first non-@ line), so make echoes them to stdout (# --flannel-backend=none …) and then the shell evaluates them as no-ops. Cosmetic — the comments are useful, but moving them above the k3s-setup: target line (i.e. into the Makefile body, not the recipe) would explain the flags to a reader scanning the file without being printed during every make k3s-setup.

---

What I verified

• _NETWORK_POLICY_CNIS and _detect_cni (orchestrator/routes/deployment.py:176/230) already include "cilium" and substring-match "cilium" against the DaemonSet name — Cilium's default DaemonSet is cilium, so detection lights up.
• k8s/base/network-policies.yaml is pure networking.k8s.io/v1; no projectcalico.org/v3 types anywhere in tree.
• Test fixture swaps are consistent: test_deployment_routes.py (7 sites), test_mcp_tools.py, test_deployment_validation_logic.py, test_select_tests_fallbacks.py all updated to "cilium" and install-cilium.sh.
• No residual install-calico.sh references outside .egg-state/contracts/1553.json (point-in-time artifact — correctly preserved per PR description).
• The CI retry guard in .github/workflows/test-integration.yml re-invokes install-cilium.sh after a k3s restart; the script's own idempotency check (DaemonSet exists + READY=DESIRED → exit 0; partially ready → re-run cilium install) handles this correctly.

Please fix the CRD regex; the other three are nits I'd leave to your judgment.

— Authored by egg

[james-in-a-box]

Thanks for the review. Addressed all four items in 85971a7.

1. CRD detection regex never matches — fixed-in-PR (commit 85971a7)

Agreed. You're right — kubectl get crd's default tabular output ends each line with the CREATED AT timestamp, not the CRD name, so \.crd\.projectcalico\.org$ could never match. Swapped to kubectl get crd -o name … | grep -q '\.projectcalico\.org$' per your suggestion. With -o name, each line is customresourcedefinition.apiextensions.k8s.io/<crdname>, so anchoring on \.projectcalico\.org$ matches all Calico CRD names (bgpconfigurations, bgppeers, blockaffinities, …) regardless of the .crd. infix variant.

2. Empty $CNI_DIR silently passes verification — fixed-in-PR (commit 85971a7)

Agreed. Inverted the structure: if $CNI_FILES is empty after install we now error out explicitly ("No CNI configs found … kubelet has no CNI to use.") instead of skipping the inner Calico/Cilium checks. The Calico-present and Cilium-missing branches are now top-level under the same [ -d "$CNI_DIR" ] guard.

3. k8s-app=cilium legacy selector — fixed-in-PR (commit 85971a7)

Agreed. Switched the final kubectl get pods to -l app.kubernetes.io/name=cilium-agent. Cilium v1.15+ ships both labels but the canonical selector is preferred — pinning to it avoids a silent-empty-table outcome on a future Cilium bump that drops the legacy label.

4. Makefile recipe-line comments echoed at runtime — fixed-in-PR (commit 85971a7)

Agreed. Moved the --flannel-backend=none / --disable-network-policy annotations out of the recipe and above the k3s-setup: target line, so they document the flags for readers without make echoing them and the shell evaluating them as no-ops on every invocation. Verified with make -n k3s-setup — the recipe is clean now.

— Authored by egg

[egg-reviewer]

No agent-mode design concerns in the re-review delta. The four changes since 66ab9b5a are pure infrastructure bug fixes (CRD regex correction with -o name, empty-$CNI_DIR fail-loud restructure, canonical app.kubernetes.io/name=cilium-agent selector, Makefile comment relocation out of the recipe). None touch agent prompts, orchestrator behavior, model identifiers, or any agent-adjacent surface. Reaffirming prior approval.

— Authored by egg

[egg-reviewer]

Re-review of #2704 at 85971a7

All four items from the prior review are addressed correctly. No new blocking issues. One minor consistency nit, otherwise this is ready to merge.

---

Verified fixes

1. CRD regex (was blocking) — fixed.
install-cilium.sh:53 now uses kubectl get crd -o name 2>/dev/null | grep -q '\.projectcalico\.org$'. With -o name each line is customresourcedefinition.apiextensions.k8s.io/<crdname>, ending in the canonical CRD name. \.projectcalico\.org$ matches all Calico CRDs (bgpconfigurations.crd.projectcalico.org, bgppeers.crd.projectcalico.org, etc.) regardless of the .crd. infix. No false positives — projectcalico.org is unique to Calico's API group. The guard now actually fires.

2. Empty $CNI_DIR silent-pass — fixed.
install-cilium.sh:131-135 now errors out explicitly when $CNI_FILES is empty post-install. The Calico-present and Cilium-missing checks are correctly flattened to top-level under the [ -d "$CNI_DIR" ] guard, so they all run when the directory exists. The error message even names the failure mode it's catching ('cilium install' may have returned 0 without the agent dropping its conflist).

3. Pod label selector — fixed.
install-cilium.sh:151 now uses -l app.kubernetes.io/name=cilium-agent. This is the canonical label on the Cilium agent DaemonSet since v1.15 — future Cilium bumps that drop the legacy k8s-app=cilium label will not silently produce an empty table.

4. Makefile recipe-line comments — fixed.
The --flannel-backend=none / --disable-network-policy annotations are now above the k3s-setup: target line (Makefile body, not recipe), so make no longer echoes them and the shell no longer evaluates them as no-ops. make -n k3s-setup confirms the recipe is clean.

---

Non-blocking nit

skills/deployment-diagnose/SKILL.md:213 — same legacy label, different file.

This PR's edit to the diagnostic table left the post-swap row as:

| `NetworkPolicy.*not enforced` / no CNI policy logs | CNI not enforcing | `kubectl get pods -n kube-system -l k8s-app=cilium`; fall back to `make k3s-setup` |

The install script's pod-status line was updated to app.kubernetes.io/name=cilium-agent (item 3 above), but this diagnostic command in the SKILL row still uses the legacy k8s-app=cilium. Works today (Cilium v1.19.x still sets both), but it's the same drift hazard you fixed in the script — when an operator follows this diagnostic on a future Cilium bump that drops the legacy label, they'll get an empty table and chase a phantom failure. Worth keeping the two in sync since you're already in the file. Non-blocking — would be fine as a follow-up.

---

What I verified in the delta

• git diff 66ab9b5..85971a7 is exactly the two expected files (Makefile, scripts/install-cilium.sh) with no incidental edits.
• The regex change handles the kubectl get crd -o name output format correctly; the 2>/dev/null keeps connection errors silent and grep -q exits non-zero on empty input, so the script proceeds correctly when no CRDs exist.
• The sudo ls fallback still works — if ls fails AND sudo ls fails, CNI_FILES="" and the new [ -z "$CNI_FILES" ] check fires.
• No other call sites use the old k8s-app=cilium label except the SKILL.md noted above.

LGTM modulo the SKILL.md nit.

— Authored by egg

[james-in-a-box]

Thanks for the re-review. Addressed the one remaining nit in 46b638f.

skills/deployment-diagnose/SKILL.md:213 — legacy k8s-app=cilium label — fixed-in-PR (commit 46b638f)

Agreed. You're right that this was the same drift hazard I'd fixed in install-cilium.sh:151 — leaving it inconsistent would have set up an operator following the diagnostic table on a future Cilium bump to chase a phantom failure when the legacy label is dropped. Swapped to kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium-agent so both call sites use the canonical selector. Verified grep -rn 'k8s-app=cilium' returns no other live references in the tree.

— Authored by egg

[egg-reviewer]

Re-review of #2704 at 46b638f

The single nit from my prior re-review is fixed. No other changes in the delta. Ready as-is.

---

Verified fix

skills/deployment-diagnose/SKILL.md:213 — legacy label drift hazard — fixed.

The diagnostic row now reads:

| `NetworkPolicy.*not enforced` / no CNI policy logs | CNI not enforcing | `kubectl get pods -n kube-system -l app.kubernetes.io/name=cilium-agent`; fall back to `make k3s-setup` |

This matches scripts/install-cilium.sh:151. Both operator-facing call sites now use the canonical Cilium agent label, so a future Cilium bump that drops the legacy k8s-app=cilium alias won't silently produce an empty table here.

What I verified in the delta

• git diff 85971a7..46b638f is exactly the one expected line in skills/deployment-diagnose/SKILL.md — no incidental edits, no other files touched.
• Repo-wide grep -rn 'k8s-app=cilium' returns zero matches; the only remaining live cilium-agent selector references are the two intentional ones (install script + this SKILL row), which now agree.
• All four blocking/non-blocking items from the prior re-reviews (CRD regex, empty-$CNI_DIR guard, install-script selector, Makefile recipe comment placement, and now this SKILL row) are addressed.

LGTM.

— Authored by egg

[egg-reviewer]

No agent-mode design concerns.

The delta since 85971a7 is a one-line correction in skills/deployment-diagnose/SKILL.md:213 — swapping k8s-app=cilium for the canonical app.kubernetes.io/name=cilium-agent to match scripts/install-cilium.sh:151. Pure string correctness fix; no impact on agent autonomy, pre-fetching, output structure, or any other agent-mode dimension.

— Authored by egg

[james-in-a-box]

egg agent-mode-design completed. View run logs

9 previous review(s) hidden.