diff --git a/.github/workflows/test-integration.yml b/.github/workflows/test-integration.yml index 5dc3f4bb53..a750f141ee 100644 --- a/.github/workflows/test-integration.yml +++ b/.github/workflows/test-integration.yml @@ -81,10 +81,12 @@ jobs: # state; the idempotent install-cilium.sh is safe to re-run # after k3s recovers. run: | - # --disable=metrics-server: egg doesn't use it, and under Cilium it - # never becomes Ready (its pod can't reach the kubelet on the node - # IP). The dead v1beta1.metrics.k8s.io APIService then wedges all - # namespace deletion, hanging test teardown and CI cleanup. + # --disable=metrics-server: disables k3s's BUNDLED metrics-server, + # which under Cilium can't reach the kubelet on the node IP, never + # becomes Ready, and whose dead v1beta1.metrics.k8s.io APIService + # wedges all namespace deletion (hanging test teardown / CI cleanup). + # install-metrics-server.sh below deploys a hostNetwork variant that + # works — see k8s/addons/metrics-server.yaml. curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --disable=metrics-server --write-kubeconfig-mode=644" sh - export KUBECONFIG=/etc/rancher/k3s/k3s.yaml echo "KUBECONFIG=/etc/rancher/k3s/k3s.yaml" >> "$GITHUB_ENV" @@ -98,6 +100,8 @@ jobs: fi # Wait for node to be ready kubectl wait --for=condition=Ready node --all --timeout=120s + # Deploy egg's hostNetwork metrics-server (replaces the disabled bundled one) + scripts/install-metrics-server.sh - name: Import images into k3s # Flake guard (HITL Q1 of #2474): retry the image-import step diff --git a/Makefile b/Makefile index 5486c236bf..68b39d0bea 100644 --- a/Makefile +++ b/Makefile @@ -481,18 +481,22 @@ build: sync-venv-if-uv # --flannel-backend=none: Cilium replaces flannel as the CNI dataplane. # --disable-network-policy: Cilium owns NetworkPolicy enforcement; the # k3s-builtin policy controller would otherwise conflict. -# --disable=metrics-server: egg does not use metrics-server. Under Cilium -# its pod cannot reach the kubelet on the node IP, so it never becomes -# Ready; the resulting perpetually-unavailable v1beta1.metrics.k8s.io -# APIService makes the namespace controller's discovery step fail, -# which wedges *all* namespace deletion (stuck Terminating forever). +# --disable=metrics-server: disables k3s's BUNDLED metrics-server, which +# runs on the pod network and under Cilium cannot reach the kubelet on +# the node IP — it never becomes Ready, and the resulting +# perpetually-unavailable v1beta1.metrics.k8s.io APIService makes the +# namespace controller's discovery step fail, wedging *all* namespace +# deletion (stuck Terminating forever). install-metrics-server.sh below +# deploys a hostNetwork variant that reaches the kubelet and works; see +# k8s/addons/metrics-server.yaml. k3s-setup: ## Install k3s with Cilium CNI @echo "Setting up k3s cluster..." curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --disable=metrics-server --write-kubeconfig-mode=644" sh - export KUBECONFIG=/etc/rancher/k3s/k3s.yaml && \ scripts/install-cilium.sh && \ echo "Waiting for k3s node to be ready..." && \ - kubectl wait --for=condition=Ready node --all --timeout=120s + kubectl wait --for=condition=Ready node --all --timeout=120s && \ + scripts/install-metrics-server.sh @echo "k3s cluster ready" k3s-secrets: ## Create gateway secrets from ~/.config/egg/ diff --git a/docs/deploy/resource-sizing.md b/docs/deploy/resource-sizing.md index 8e4e2a4194..ff4ccda6cc 100644 --- a/docs/deploy/resource-sizing.md +++ b/docs/deploy/resource-sizing.md @@ -1,5 +1,7 @@ # Pod resource sizing +> **Requires `kubectl top`.** The sampling loops below depend on the metrics.k8s.io API. egg's k3s setup ships a hostNetwork metrics-server addon ([`k8s/addons/metrics-server.yaml`](../../k8s/addons/metrics-server.yaml), installed by `make k3s-setup`); without it, `kubectl top` and the snapshots in this doc cannot be reproduced. + Resource requests and limits for the four pod types in the egg stack. The `gateway`, `orchestrator`, and `egg-sandbox-*` allocations were tuned against the observed telemetry below; the `litellm` row mirrors `k8s/base/litellm-deployment.yaml`, whose 2Gi memory limit was raised after an observed OOMKill in #2853. It postdates the snapshots below (so no sampled telemetry exists for it), but the OOMKill is the runtime signal behind the current limit. See #1888 / #1895 for the right-sizing initiative. ## Current allocations diff --git a/docs/guides/deployment.md b/docs/guides/deployment.md index 3676ed822e..9e546e973a 100644 --- a/docs/guides/deployment.md +++ b/docs/guides/deployment.md @@ -58,13 +58,14 @@ kubectl get pods -n egg-system ```bash # What make k3s-setup does: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --disable=metrics-server --write-kubeconfig-mode=644" sh - -scripts/install-cilium.sh # downloads cilium-cli and runs `cilium install` +scripts/install-cilium.sh # downloads cilium-cli and runs `cilium install` # Waits for cluster to become ready +scripts/install-metrics-server.sh # deploys the hostNetwork metrics-server addon ``` > **Why Cilium?** k3s ships with Flannel as default CNI. Flannel does **not** support NetworkPolicies, which are required for agent network isolation. Cilium replaces Flannel and enforces the NetworkPolicies that prevent agents from reaching the internet directly. Calico filled this role until [#2703](https://github.com/jwbron/egg/issues/2703) — see that issue for the swap rationale. > -> **Why `--disable=metrics-server`?** Under Cilium, the metrics-server pod cannot reach the kubelet on the node IP, so it never becomes Ready. The resulting perpetually-unavailable `v1beta1.metrics.k8s.io` APIService causes the namespace controller's discovery step to fail, which wedges all namespace deletion (namespaces become stuck in `Terminating` indefinitely). egg does not use metrics-server; disabling it avoids this hang with no functional loss. +> **Why `--disable=metrics-server`?** This disables k3s's *bundled* metrics-server, which runs on the pod network and under Cilium cannot reach the kubelet on the node IP — it never becomes Ready, and the resulting perpetually-unavailable `v1beta1.metrics.k8s.io` APIService causes the namespace controller's discovery step to fail, wedging all namespace deletion (namespaces stuck in `Terminating` indefinitely). `scripts/install-metrics-server.sh` then deploys a **hostNetwork** variant ([`k8s/addons/metrics-server.yaml`](../../k8s/addons/metrics-server.yaml)) that shares the node's network namespace and reaches the kubelet exactly as the host does, so `kubectl top` works without re-triggering the wedge. (Prior to this, metrics-server was disabled outright in [#2703](https://github.com/jwbron/egg/issues/2703); it was believed unfixable under Cilium until the hostNetwork approach.) > > **Migrating from a pre-#2703 install:** in-place CNI swap on a live k3s cluster is not supported (host CNI binaries, conflists, CRDs, `tunl0`, and per-pod veth pairs persist after deleting the calico-node DaemonSet). Run `make k3s-teardown && make k3s-setup` for a clean install. `install-cilium.sh` will refuse if it detects leftover Calico state. > diff --git a/k8s/addons/metrics-server.yaml b/k8s/addons/metrics-server.yaml new file mode 100644 index 0000000000..eab8af3d43 --- /dev/null +++ b/k8s/addons/metrics-server.yaml @@ -0,0 +1,276 @@ +# metrics-server (egg-vendored) +# +# Source: https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.1/components.yaml +# Pinned version: v0.8.1 (sha256 of upstream components.yaml: +# 4a672c4891902573a3ff753cece5de1bf1f55dd053403dfec39df9d1636b7ff1) +# +# Provides the metrics.k8s.io API that backs `kubectl top nodes/pods` and any +# HPA. k3s ships its own bundled metrics-server, but `make k3s-setup` disables +# it (`--disable=metrics-server`) and deploys THIS manifest instead. See the +# egg modifications below for why the bundled one cannot work here. +# +# === egg modifications vs upstream v0.8.1 (search "egg:") === +# Under our Cilium datapath (kubeProxyReplacement=false, legacy host routing, +# chained portmap — see scripts/install-cilium.sh), a metrics-server pod on the +# pod network cannot complete a TCP handshake to the kubelet on the node's +# InternalIP:10250 — the scrape fails with "connect: connection refused" and +# the pod never becomes Ready. A perpetually-unavailable v1beta1.metrics.k8s.io +# APIService then wedges the namespace controller's discovery step, leaving +# every deleted namespace stuck in `Terminating` indefinitely (this is the +# original reason #2703 disabled metrics-server outright). +# +# Fix: run metrics-server in the host network namespace, where it reaches the +# kubelet exactly as the host itself does (`curl -k https://:10250` — +# returns 401, i.e. the socket is reachable). Four coupled changes: +# 1. hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet +# (the latter so the pod still resolves the in-cluster API server via +# CoreDNS while sharing the host netns). +# 2. --secure-port 10250 -> 4443 (and the matching containerPort): on +# hostNetwork the pod shares the node's ports, and 10250 is already held +# by the kubelet. 4443 is free. The port is still named "https", so the +# Service targetPort and the liveness/readiness probes follow it +# unchanged. +# 3. strategy: Recreate (was rollingUpdate with maxUnavailable:0). Upstream's +# rolling strategy + maxUnavailable:0 means the controller brings up the +# new pod before deleting the old one. With hostNetwork + single replica +# on one node, the new pod would try to bind 0.0.0.0:4443 while the old +# one still holds it, CrashLoopBackOff forever, and the rollout would +# deadlock (maxUnavailable:0 also forbids evicting the old pod to make +# room). Recreate deletes-then-creates: bounded ~25s gap during rollouts, +# well below the namespace-controller's discovery window — the wedge only +# reopens if the APIService stays unavailable across a discovery cycle. +# 4. --kubelet-insecure-tls: defense-in-depth against a future k3s release +# changing kubelet serving-cert provisioning (today k3s rotates kubelet +# certs signed by the cluster CA; if that flips back to self-signed, +# scrapes would start failing with `x509: certificate signed by unknown +# authority` and re-open the wedge). Security cost ~zero: --kubelet- +# insecure-tls skips server→client cert verification (metrics-server +# confirming the kubelet's identity), and on hostNetwork this is +# loopback-equivalent traffic on a single host — no realistic +# interception surface. (Client→server auth, metrics-server proving its +# identity to the kubelet via SA token, is unchanged and independent.) +# Everything else is semantically equivalent to upstream v0.8.1 — the diff is +# the four `egg:` markers below (some upstream lists were re-indented by the +# editor, which is YAML-cosmetic; for a clean diff against a newer upstream +# release, normalize indentation first). +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + labels: + k8s-app: metrics-server + name: metrics-server + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + k8s-app: metrics-server + rbac.authorization.k8s.io/aggregate-to-admin: "true" + rbac.authorization.k8s.io/aggregate-to-edit: "true" + rbac.authorization.k8s.io/aggregate-to-view: "true" + name: system:aggregated-metrics-reader +rules: + - apiGroups: + - metrics.k8s.io + resources: + - pods + - nodes + verbs: + - get + - list + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + k8s-app: metrics-server + name: system:metrics-server +rules: + - apiGroups: + - "" + resources: + - nodes/metrics + verbs: + - get + - apiGroups: + - "" + resources: + - pods + - nodes + verbs: + - get + - list + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + labels: + k8s-app: metrics-server + name: metrics-server-auth-reader + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: extension-apiserver-authentication-reader +subjects: + - kind: ServiceAccount + name: metrics-server + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + k8s-app: metrics-server + name: metrics-server:system:auth-delegator +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:auth-delegator +subjects: + - kind: ServiceAccount + name: metrics-server + namespace: kube-system +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + k8s-app: metrics-server + name: system:metrics-server +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:metrics-server +subjects: + - kind: ServiceAccount + name: metrics-server + namespace: kube-system +--- +apiVersion: v1 +kind: Service +metadata: + labels: + k8s-app: metrics-server + name: metrics-server + namespace: kube-system +spec: + ports: + - appProtocol: https + name: https + port: 443 + protocol: TCP + targetPort: https + selector: + k8s-app: metrics-server +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + labels: + k8s-app: metrics-server + name: metrics-server + namespace: kube-system +spec: + # egg: explicit replicas:1 to make the rollout-strategy reasoning legible; + # under hostNetwork on a single node, two replicas would fight for the same + # host port. Pair with strategy:Recreate below. + replicas: 1 + selector: + matchLabels: + k8s-app: metrics-server + # egg: Recreate (was rollingUpdate + maxUnavailable:0). See header for the + # full reasoning — short version: rolling update + hostNetwork + single + # replica deadlocks at the first rollout because the new pod can't bind the + # already-held host port. Recreate accepts a bounded ~25s gap instead. + strategy: + type: Recreate + template: + metadata: + labels: + k8s-app: metrics-server + spec: + # egg: run in the host netns so the scraper can reach kubelet:10250 + # under our Cilium datapath. dnsPolicy keeps cluster DNS working. + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - args: + - --cert-dir=/tmp + # egg: 10250 is the kubelet's port; on hostNetwork we must move + # metrics-server's own serving port off it. 4443 is free. + - --secure-port=4443 + # egg: defense-in-depth against future k3s kubelet-cert changes + # (see header). No-op today; prevents a silent x509 wedge tomorrow. + - --kubelet-insecure-tls + - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname + - --kubelet-use-node-status-port + - --metric-resolution=15s + image: registry.k8s.io/metrics-server/metrics-server:v0.8.1 + imagePullPolicy: IfNotPresent + livenessProbe: + failureThreshold: 3 + httpGet: + path: /livez + port: https + scheme: HTTPS + periodSeconds: 10 + name: metrics-server + ports: + # egg: matches --secure-port=4443 above (named "https" so the + # Service targetPort and the probes resolve to it unchanged). + - containerPort: 4443 + name: https + protocol: TCP + readinessProbe: + failureThreshold: 3 + httpGet: + path: /readyz + port: https + scheme: HTTPS + initialDelaySeconds: 20 + periodSeconds: 10 + resources: + requests: + cpu: 100m + memory: 200Mi + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true + runAsNonRoot: true + runAsUser: 1000 + seccompProfile: + type: RuntimeDefault + volumeMounts: + - mountPath: /tmp + name: tmp-dir + nodeSelector: + kubernetes.io/os: linux + priorityClassName: system-cluster-critical + serviceAccountName: metrics-server + volumes: + - emptyDir: {} + name: tmp-dir +--- +apiVersion: apiregistration.k8s.io/v1 +kind: APIService +metadata: + labels: + k8s-app: metrics-server + name: v1beta1.metrics.k8s.io +spec: + group: metrics.k8s.io + groupPriorityMinimum: 100 + insecureSkipTLSVerify: true + service: + name: metrics-server + namespace: kube-system + version: v1beta1 + versionPriority: 100 diff --git a/scripts/install-metrics-server.sh b/scripts/install-metrics-server.sh new file mode 100755 index 0000000000..3aa1e66b91 --- /dev/null +++ b/scripts/install-metrics-server.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# +# install-metrics-server.sh - Deploy egg's hostNetwork metrics-server addon. +# +# `make k3s-setup` disables k3s's bundled metrics-server +# (`--disable=metrics-server`) and runs this script instead. The bundled one +# runs on the pod network and cannot reach the kubelet under our Cilium +# datapath, so it never becomes Ready and its dead v1beta1.metrics.k8s.io +# APIService wedges all namespace deletion. The vendored manifest this script +# applies runs in the host netns and works — see k8s/addons/metrics-server.yaml +# for the full rationale. +# +# Idempotent: safe to run multiple times. `kubectl apply` reconciles the +# manifest and the waits below tolerate an already-Ready deployment. +# +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MANIFEST="$SCRIPT_DIR/../k8s/addons/metrics-server.yaml" + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" +} + +error() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] ERROR: $*" >&2 +} + +# Prerequisites +if ! command -v kubectl &>/dev/null; then + error "kubectl is not installed or not in PATH" + exit 1 +fi +if ! kubectl cluster-info &>/dev/null; then + error "Cannot connect to Kubernetes cluster. Is the cluster running?" + exit 1 +fi +if [ ! -f "$MANIFEST" ]; then + error "Manifest not found at $MANIFEST" + exit 1 +fi + +# Guard against the bundled metrics-server: if k3s was installed WITHOUT +# `--disable=metrics-server`, k3s's own (non-hostNetwork) manifest is +# reconciled from /var/lib/rancher/k3s/server/manifests and will fight this +# one — kubelet-unreachable scrapes, perpetual NotReady, wedged namespace +# deletion. Detect the tell-tale HelmChart/addon and refuse rather than +# deploy into a tug-of-war. +# k3s tracks each bundled manifest as an Addon CR (addons.k3s.cattle.io), +# named after the source file — e.g. metrics-server-deployment, +# metrics-server-service. Match the prefix so this is robust across k3s +# versions. If the CRD is absent (non-k3s cluster), the list is empty and we +# don't false-positive. +if kubectl -n kube-system get addons.k3s.cattle.io -o name 2>/dev/null | grep -q 'metrics-server'; then + error "k3s's bundled metrics-server addon is present — k3s was installed" + error "without '--disable=metrics-server'. The bundled (non-hostNetwork)" + error "manifest cannot reach the kubelet under Cilium and will conflict" + error "with egg's. Reinstall k3s with --disable=metrics-server:" + error " make k3s-teardown && make k3s-setup" + exit 1 +fi + +log "Applying egg metrics-server addon from $MANIFEST..." +kubectl apply -f "$MANIFEST" + +log "Waiting for metrics-server rollout (timeout: 120s)..." +if ! kubectl -n kube-system rollout status deploy/metrics-server --timeout=120s; then + error "metrics-server did not become Ready. Recent scrape errors:" + kubectl -n kube-system logs -l k8s-app=metrics-server --tail=10 2>&1 | sed 's/^/ /' >&2 + error "A 'connect: connection refused' to the node IP:10250 means the pod is" + error "NOT in the host netns — confirm hostNetwork:true survived in the manifest." + exit 1 +fi + +# The APIService can lag the pod becoming Ready by a scrape cycle. Poll until +# the aggregation layer reports it Available, so callers (and `kubectl top`) +# don't race a not-yet-registered metrics.k8s.io. +log "Waiting for v1beta1.metrics.k8s.io APIService to become Available..." +deadline=$((SECONDS + 60)) +while true; do + avail=$(kubectl get apiservice v1beta1.metrics.k8s.io \ + -o 'jsonpath={.status.conditions[?(@.type=="Available")].status}' 2>/dev/null || echo "") + if [ "$avail" = "True" ]; then + break + fi + if [ "$SECONDS" -ge "$deadline" ]; then + error "v1beta1.metrics.k8s.io did not become Available within 60s (status: '${avail:-unknown}')." + error "Inspect with: kubectl get apiservice v1beta1.metrics.k8s.io -o yaml" + exit 1 + fi + sleep 3 +done + +# Final smoke test: a real `kubectl top nodes` proves the scrape path end to +# end (pod Ready + APIService Available is necessary but not sufficient — the +# first scrape must also have landed). 60s deadline because --metric-resolution +# is 15s and the kubelet's cAdvisor can take a beat to spin up on a cold k3s, +# so the first scrape can land well after APIService=Available. +log "Verifying 'kubectl top nodes' returns data..." +deadline=$((SECONDS + 60)) +while ! kubectl top nodes &>/dev/null; do + if [ "$SECONDS" -ge "$deadline" ]; then + error "'kubectl top nodes' still failing after metrics-server became Available." + kubectl top nodes 2>&1 | sed 's/^/ /' >&2 + exit 1 + fi + sleep 2 +done + +log "metrics-server is Ready; 'kubectl top' is working." +kubectl top nodes 2>&1 | sed 's/^/ /'