Add Claude Code skill for reproducing CI failures on GCP VMs

.claude/skills/ci-reproduce-on-gcp-vm/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: ci-reproduce-on-gcp-vm
+description: Reproduce CI test failures on a GCP VM matching the CI environment. Use when a CI job fails and the issue cannot be reproduced locally (e.g., kernel-dependent BPF verifier failures, kernel version-specific bugs).
+---
+
+## Overview
+
+CI runs Felix tests on GCP VMs using specific Ubuntu image families. The local dev machine may have a different kernel, so some failures only reproduce on the CI kernel. This skill creates a GCP VM matching the CI environment, runs the failing test, and cleans up.
+
+## Prerequisites
+
+- `gcloud` CLI authenticated with access to the `tigera-dev` project
+- The calico repo checked out locally with the failing branch
+
+## Step 1: Identify the CI Image Family
+
+Check `felix/.semaphore/fv-prologue` to find the image family for the failing CI job. The mapping is based on `FELIX_TEST_GROUP`:
+
+| Test group pattern | IMAGE_FAMILY | Ubuntu version |
+|---|---|---|
+| `22.04` | `ubuntu-2204-lts` | 22.04 Jammy |
+| `24.04` | `ubuntu-2404-lts-amd64` | 24.04 Noble |
+| `25.10` | `ubuntu-2510-amd64` | 25.10 Plucky |
+
+For example, the `bpf-24.04-ipt-with-ut` test group uses `ubuntu-2404-lts-amd64`.
+
+If in doubt, read `felix/.semaphore/fv-prologue` and `.semaphore/vms/vm-bootstrap.sh` for the latest mappings.
+
+## Step 2: Create the VM
+
+```bash
+zone=us-central1-a
+vm_name=<user>-debug
+image_family=ubuntu-2404-lts-amd64  # from Step 1
+
+gcloud config set project tigera-dev
+gcloud --quiet compute instances create "${vm_name}" \
+  --zone=${zone} \
+  --image-family=${image_family} \
+  --image-project=ubuntu-os-cloud \
+  --machine-type=n4-highcpu-4 \
+  --boot-disk-size=20G \
+  --boot-disk-type=hyperdisk-balanced
+```
+
+The machine type and disk size above match CI defaults (see `felix/.semaphore/fv-prologue`).
+
+## Step 3: Wait for SSH and Install Dependencies
+
+The VM bootstrap in CI is done by `.semaphore/vms/vm-bootstrap.sh`. Replicate its key steps:
+
+```bash
+ssh_cmd="gcloud --quiet compute ssh --zone=${zone} ubuntu@${vm_name} --"
+
+# Wait for SSH
+for i in $(seq 1 10); do
+  ${ssh_cmd} echo "SSH ready" && break
+  sleep 2
+done
+
+# Install prerequisites
+${ssh_cmd} "sudo apt-get update -y && sudo apt-get install -y --no-install-recommends apt-transport-https ca-certificates curl software-properties-common"
+
+# Add Docker repo (DEB822 format, matching .semaphore/vms/vm-bootstrap.sh)
+${ssh_cmd} "sudo install -d -m 0755 /etc/apt/keyrings && curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /tmp/docker.gpg && sudo mv /tmp/docker.gpg /etc/apt/keyrings/docker.asc && sudo chmod 0644 /etc/apt/keyrings/docker.asc"
+${ssh_cmd} "ubuntu_codename=\$(. /etc/os-release && echo \"\${UBUNTU_CODENAME:-\$VERSION_CODENAME}\") && printf '%s\n' 'Types: deb' 'URIs: https://download.docker.com/linux/ubuntu' \"Suites: \${ubuntu_codename}\" 'Components: stable' 'Architectures: amd64' 'Signed-By: /etc/apt/keyrings/docker.asc' | sudo tee /etc/apt/sources.list.d/docker.sources > /dev/null"
+${ssh_cmd} "sudo apt-get update -y"
+
+# Install Docker and tools — pin versions to match CI (see .semaphore/vms/vm-bootstrap.sh)
+# Noble (24.04): docker-ce=5:27.5.1-1~ubuntu.24.04~noble
+# Jammy (22.04): docker-ce=5:20.10.14~3-0~ubuntu-jammy
+# If unsure, omit the version pin to get the latest.
+${ssh_cmd} "sudo apt-get install -y --no-install-recommends docker-ce docker-ce-cli docker-buildx-plugin containerd.io git make iproute2 wireguard"
+
+# Post-install setup
+${ssh_cmd} "sudo usermod -a -G docker ubuntu"
+${ssh_cmd} "sudo modprobe ipip"
+
+# Configure Docker with IPv6 (required by many FV tests)
+${ssh_cmd} 'echo "{\"ipv6\": true, \"fixed-cidr-v6\": \"2001:db8:1::/64\"}" | sudo tee /etc/docker/daemon.json'
+${ssh_cmd} "sudo systemctl restart docker"
+
+# Match CI's sysctl setting (loose reverse path filtering)
+${ssh_cmd} "sudo sysctl -w net.ipv4.conf.all.rp_filter=2"
+```
+
+## Step 4: Clone Repo and Checkout Branch
+
+```bash
+# Get the current branch name
+branch=$(git rev-parse --abbrev-ref HEAD)
+remote_url=$(git remote get-url origin)
+
+${ssh_cmd} "git clone ${remote_url} calico && cd calico && git checkout ${branch}"
+```
+
+If the remote is an SSH URL and the VM doesn't have SSH keys, use the HTTPS URL instead:
+```bash
+# Convert git@github.com:user/repo.git to https://github.com/user/repo.git
+https_url=$(echo "${remote_url}" | sed 's|git@github.com:|https://github.com/|')
+${ssh_cmd} "git clone ${https_url} calico && cd calico && git checkout ${branch}"
+```
+
+## Step 5: Run the Failing Test
+
+Check the kernel version first to confirm it differs from local:
+```bash
+${ssh_cmd} "uname -r"
+```
+
+Then run the specific test. Common patterns:
+
+```bash
+# BPF unit test (e.g., verifier loadability)
+${ssh_cmd} "cd calico/felix && make FOCUS=TestPrecompiledBinariesAreLoadable ut-bpf"
+
+# Specific BPF unit test
+${ssh_cmd} "cd calico/felix && make FOCUS=TestNATNodePortNoFWD ut-bpf"
+
+# Felix FV test
+${ssh_cmd} "cd calico/felix && make fv GINKGO_FOCUS='TestName'"
+
+# BPF FV test
+${ssh_cmd} "cd calico/felix && make fv-bpf GINKGO_FOCUS='TestName'"
+```
+
+The first run will be slow (pulls Docker build images). Subsequent runs are faster.
+
+## Step 6: Clean Up
+
+Always delete the VM when done:
+
+```bash
+gcloud --quiet compute instances delete ${vm_name} --zone=${zone}
+```
+
+## Reference: CI Configuration Files
+
+| File | Purpose |
+|---|---|
+| `felix/.semaphore/fv-prologue` | Maps test groups to image families, sets env vars |
+| `.semaphore/vms/vm-bootstrap.sh` | VM startup script (Docker install, sysctl, IPv6) |
+| `.semaphore/vms/run-tests-on-vms` | Orchestrates VM creation and test execution |
+| `.semaphore/vms/configure-test-vm` | Per-VM configuration after bootstrap |
+| `.semaphore/semaphore.yml.d/blocks/20-felix.yml` | Felix CI job definitions and test groups |