feat(infra): infrastructure TAC trees + Tailscale Docker registration #949
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| # docker-compose.tailscale.yml — Tailscale mesh client for POWERFULMOVES node | ||
| # ============================================================================ | ||
| # Userspace networking mode: no TUN device needed, works on Windows Docker | ||
| # Desktop / WSL2. Slightly slower than kernel mode but zero friction. | ||
| # | ||
| # Usage: | ||
| # make -C pmoves tailscale-up # Start and join tailnet | ||
| # make -C pmoves tailscale-status # Check connection | ||
| # make -C pmoves tailscale-ip # Show Tailscale IP | ||
| # | ||
| # Requires: TAILSCALE_AUTHKEY in env.shared (get from https://login.tailscale.com/admin/settings/keys) | ||
|
|
||
| services: | ||
| tailscale: | ||
| image: tailscale/tailscale:latest | ||
| container_name: pmoves-tailscale | ||
| hostname: pmoves-powerfulmoves | ||
| restart: unless-stopped | ||
| volumes: | ||
| - tailscale-state:/var/lib/tailscale | ||
| environment: | ||
| - TS_AUTHKEY=${TAILSCALE_AUTHKEY} | ||
| - TS_USERSPACE=true | ||
| - TS_HOSTNAME=pmoves-powerfulmoves | ||
| - TS_STATE_DIR=/var/lib/tailscale | ||
| - TS_ACCEPT_DNS=true | ||
| - TS_EXTRA_ARGS=--accept-routes | ||
| network_mode: host | ||
| env_file: | ||
| - env.shared | ||
|
|
||
| volumes: | ||
| tailscale-state: | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| # TAC Tree: Infrastructure & Networking | ||
|
|
||
| > Technology-Architecture-Context tree for the PMOVES.AI infrastructure layer — mesh VPN, node topology, CI runner fleet, and DNS routing that enables all 62 agents to discover and communicate with each other. | ||
|
|
||
| ## Service Identity | ||
|
|
||
| | Field | Value | | ||
| |-------|-------| | ||
| | **Team** | infra | | ||
| | **Agents** | mesh_agent, headscale, vps_fleet_manager | | ||
| | **Node Affinity** | kvm4-1, kvm2 (production); z890, powerfulmoves (dev) | | ||
| | **CI Runner** | vps (production); ai-lab (GPU) | | ||
| | **Tier** | api (cross-cutting) | | ||
| | **Class** | Utility | | ||
| | **Evolution** | Base (foundational, stable) | | ||
|
|
||
| ## Node Topology | ||
|
|
||
| | Node | Tailscale Hostname | Role | GPU | Runner Labels | | ||
| |------|--------------------|------|-----|---------------| | ||
| | Z890 (Windows 11) | `100.113.38.37` | Dev, GPU | RTX 3090 Ti | `self-hosted, ai-lab, gpu, cuda` | | ||
| | POWERFULMOVES (Windows 11) | `pmoves-powerfulmoves` | Dev, GPU (secondary) | — | — | | ||
| | 5090 PC | (pending onboarding) | Primary GPU | RTX 5090 | (future: `ai-lab`) | | ||
| | KVM4-1 | `pmoves-kvm4-1` | API Gateway | — | `self-hosted, vps, kvm4, production` | | ||
| | KVM4-2 | `pmoves-kvm4-2` | Data / Storage | — | `self-hosted, vps, kvm4, production` | | ||
| | KVM2 | `pmoves-kvm2` | Exit Node / Proxy | — | `self-hosted, vps, kvm2, backup` | | ||
| | Cloudflare Edge | — | DNS, Worker routing | — | — | | ||
| | GitHub Cloud | — | Lightweight CI | — | `ubuntu-latest` | | ||
|
|
||
| ## Upstream Dependencies | ||
|
|
||
| | Dependency | Type | Required | | ||
| |------------|------|----------| | ||
| | Docker Engine (all nodes) | Container runtime | Yes | | ||
| | NATS (4222) | Mesh node announcements | Yes | | ||
| | Tailscale Cloud / Headscale | VPN control plane | Yes | | ||
| | Cloudflare | DNS and edge routing | Yes (production) | | ||
| | Supabase (3010) | Node metadata storage | Optional | | ||
| | Prometheus (9090) | Metrics collection | Optional | | ||
|
|
||
| ## Downstream Consumers | ||
|
|
||
| | Consumer | Dependency | Description | | ||
| |----------|-----------|-------------| | ||
| | All 62 agents | Tailscale mesh | Inter-node connectivity | | ||
| | Agent Zero (8080) | `mesh.node.announce.v1` | Host discovery via NATS | | ||
| | Model Registry | `mesh.gpu.model.*` | GPU model lifecycle events | | ||
| | CI/CD Workflows (19) | Runner fleet | Workflow execution | | ||
| | Grafana (3002) | Prometheus | Infrastructure dashboards | | ||
|
|
||
| ## NATS Mesh Subjects | ||
|
|
||
| | Subject | Publisher | Direction | Interval | | ||
| |---------|-----------|-----------|----------| | ||
| | `mesh.node.announce.v1` | mesh_agent | Publishes | 15s heartbeat | | ||
| | `mesh.gpu.status.v1` | gpu-orchestrator | Publishes | 5s heartbeat | | ||
| | `mesh.gpu.model.loaded.v1` | gpu-orchestrator | Publishes | on-event | | ||
| | `mesh.gpu.model.unloaded.v1` | gpu-orchestrator | Publishes | on-event | | ||
| | `mesh.gpu.command.v1` | agent-zero | Publishes | on-demand | | ||
| | `mesh.gpu.command.result.v1` | gpu-orchestrator | Publishes | on-event | | ||
| | `model.registry.updated.v1` | model-registry | Publishes | on-event | | ||
|
|
||
|
Comment on lines
+30
to
+62
Contributor
There was a problem hiding this comment. Sync this TAC update into the Claude context docs. Line 30 through Line 62 add infra dependencies, ports, and NATS subjects, but this PR does not include matching updates to As per coding guidelines, " 🤖 Prompt for AI Agents |
||
| ## Mesh Networking Audit | ||
|
|
||
| - [ ] All nodes reachable via Tailscale (`tailscale ping <hostname>`) | ||
| - [ ] POWERFULMOVES registered on tailnet (Docker userspace mode) | ||
| - [ ] Z890 registered on tailnet (bare-metal install) | ||
| - [ ] KVM2 advertising exit node (`--advertise-exit-node`) | ||
| - [ ] KVM4-1 and KVM4-2 accepting routes (`--accept-routes`) | ||
| - [ ] Magic DNS resolution working across all nodes | ||
| - [ ] Headscale self-hosted deployment planned (replaces Tailscale Cloud) | ||
|
|
||
| ## CI Runner Fleet | ||
|
|
||
| | Runner | Label | Node | Mode | Start Command | | ||
| |--------|-------|------|------|---------------| | ||
| | ai-lab | `self-hosted, ai-lab` | Z890 | Docker container | `make ci-runners-local-cert-up` | | ||
| | cloudstartup | `self-hosted, cloudstartup` | KVM4-1 | Docker container | Provisioned via `hostinger-kvm-setup.sh` | | ||
| | kvm4 | `self-hosted, kvm4` | KVM4-2 | Docker container | Provisioned via `hostinger-kvm-setup.sh` | | ||
| | kvm2 | `self-hosted, kvm2` | KVM2 | Docker container | Provisioned via `hostinger-kvm-setup.sh` | | ||
|
|
||
| Runner management: `local_cert_runners.py` with `myoung34/github-runner` image. | ||
|
|
||
| ## Submodule Alignment Checklist | ||
|
|
||
| | Submodule | Branch | CLAUDE.md | CHIT Stanza | .gitmodules | | ||
| |-----------|--------|-----------|-------------|-------------| | ||
| | PMOVES-Tailscale | PMOVES.AI-Edition-Hardened | Pending | Pending | Tracked | | ||
| | PMOVES-Headscale | PMOVES.AI-Edition-Hardened | Present | Pending | Tracked | | ||
|
|
||
| ## Security Stance | ||
|
|
||
| | Finding | Severity | Status | | ||
| |---------|----------|--------| | ||
| | Headscale not deployed (using Tailscale Cloud) | P2 | Planned | | ||
| | NATS mesh traffic unencrypted between nodes | P2 | Tracked (TLS planned) | | ||
| | Runner containers need cert rotation | P2 | Tracked | | ||
| | VPS provisioning scripts use reusable auth keys | P3 | Acceptable (tagged keys) | | ||
|
|
||
| ## Production Audit Checklist | ||
|
|
||
| | Requirement | Status | Notes | | ||
| |-------------|--------|-------| | ||
| | Mesh agent heartbeat | GREEN | 15s NATS announcements | | ||
| | Tailscale connectivity | Partial | Z890 + KVM nodes connected; POWERFULMOVES pending | | ||
| | Runner fleet | GREEN | 4 containerized runners via `local_cert_runners.py` | | ||
| | Prometheus scraping | GREEN | All `/metrics` endpoints configured | | ||
| | DNS zone | Partial | Registered at Hostinger; Cloudflare migration pending | | ||
| | VPN auth | GREEN | Tailscale Cloud manages auth keys | | ||
|
|
||
| ## Make Targets | ||
|
|
||
| | Target | Description | | ||
| |--------|-------------| | ||
| | `make -C pmoves tailscale-docker-up` | Start Tailscale Docker container and join tailnet | | ||
| | `make -C pmoves tailscale-docker-down` | Stop Tailscale Docker container | | ||
| | `make -C pmoves tailscale-docker-status` | Show Tailscale Docker container connection status | | ||
| | `make -C pmoves tailscale-docker-ip` | Show Tailscale Docker container's IP | | ||
| | `make ci-runners-local-cert-up` | Start Docker-containerized CI runners | | ||
|
|
||
| ## Cross-Links | ||
|
|
||
| - **Master Topology:** `pmoves/docs/operations/TOPOLOGY.md` | ||
| - **Runner Map:** `pmoves/docs/operations/WORKFLOW_RUNNER_MAP.md` | ||
| - **Runner Strategy:** `deploy/HYBRID_RUNNER_STRATEGY.md` | ||
| - **Provisioning:** `deploy/provision/hostinger-kvm-setup.sh` | ||
| - **Agent Teams:** `pmoves/configs/agent-teams.yaml` → `infra` | ||
| - **Agent Registry:** `pmoves/config/agent_registry.yaml` | ||
| - **Tailscale TAC:** [`TAC_TAILSCALE.md`](./TAC_TAILSCALE.md) | ||
| - **Runners TAC:** [`TAC_RUNNERS.md`](./TAC_RUNNERS.md) | ||
| - **Integration Topology:** [`TAC_INTEGRATION_TOPOLOGY.md`](./TAC_INTEGRATION_TOPOLOGY.md) | ||
|
|
||
| ## Open Items | ||
|
|
||
| - Register POWERFULMOVES on tailnet (Docker userspace mode) | ||
| - Deploy Headscale self-hosted (replaces Tailscale Cloud console) | ||
| - 5090 node onboarding (blocked on Tailscale + OpenSSH setup) | ||
| - Runner certificate rotation automation | ||
| - NATS TLS between nodes (P2 security finding) | ||
| - Network segmentation per node tier | ||
|
|
||
| <!-- GRAPHITI_MARK: CLAUDE-OPUS::TAC-INFRA::2026-03-15 --> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| # TAC Tree: CI/CD Runner Fleet | ||
|
|
||
| > Technology-Architecture-Context tree for the PMOVES.AI CI/CD runner infrastructure — containerized GitHub Actions runners, workflow-to-runner routing, phase policy, and certificate management. | ||
|
|
||
| ## Service Identity | ||
|
|
||
| | Field | Value | | ||
| |-------|-------| | ||
| | **Service** | GitHub Actions Self-Hosted Runners | | ||
| | **Management Tool** | `local_cert_runners.py` | | ||
| | **Runner Image** | `myoung34/github-runner` | | ||
| | **Ports** | — (outbound HTTPS to GitHub only) | | ||
| | **Health** | Runner registration status via `gh api` | | ||
| | **Metrics** | Container health via cAdvisor (8080) | | ||
| | **Tier** | api (cross-cutting) | | ||
| | **Class** | Utility | | ||
| | **Evolution** | Base | | ||
|
|
||
| ## Runner Fleet | ||
|
|
||
| | Runner | Labels | Node | vCPU / RAM | Restart Policy | | ||
| |--------|--------|------|------------|----------------| | ||
| | ai-lab | `self-hosted, ai-lab, gpu, cuda` | Z890 | 32C / 128GB | `unless-stopped` | | ||
| | cloudstartup | `self-hosted, cloudstartup` | KVM4-1 | 8C / 16GB | `unless-stopped` | | ||
| | kvm4 | `self-hosted, kvm4, production` | KVM4-2 | 8C / 16GB | `unless-stopped` | | ||
| | kvm2 | `self-hosted, kvm2, backup` | KVM2 | 4C / 8GB | `unless-stopped` | | ||
| | (GitHub hosted) | `ubuntu-latest` | GitHub Cloud | 2C / 7GB | — | | ||
|
|
||
| **Management:** `make ci-runners-local-cert-up` starts Docker-containerized Linux runners. Containers auto-restart via `unless-stopped` policy. | ||
|
|
||
| ## Workflow-to-Runner Map | ||
|
|
||
| 19 GitHub Actions workflows mapped to runners (from `WORKFLOW_RUNNER_MAP.md`): | ||
|
|
||
| | Category | Workflows | Primary Runner | Fallback | | ||
| |----------|-----------|----------------|----------| | ||
| | GPU builds | Docker multi-arch, model training | `ai-lab` | — | | ||
| | Docker builds | Service images, Dockerfile lint | `kvm4` | `cloudstartup` | | ||
| | Tests | Python tests, smoke tests, CodeQL | `cloudstartup` | `ubuntu-latest` | | ||
| | Docs/Lint | SQL lint, CHIT contract check | `ubuntu-latest` | — | | ||
| | Deploy | Production deploy, secrets sync | `kvm4` | `kvm2` | | ||
|
|
||
| **Routing logic:** Cloudflare Worker (`deploy/cloudflare/worker.js`) analyzes changed files to select appropriate runner. | ||
|
Comment on lines
+35
to
+43
Contributor
There was a problem hiding this comment. Workflow routing table is out of sync with actual orchestrator logic. Line 38 and Line 39 describe Docker/test routing that does not match As per coding guidelines, " 🤖 Prompt for AI Agents |
||
|
|
||
| ## Runner Phase Policy | ||
|
|
||
| **File:** `runner_phase_policy.json` | ||
|
|
||
| Defines which runner labels are allowed per deployment phase: | ||
|
|
||
| | Phase | Allowed Labels | Purpose | | ||
| |-------|---------------|---------| | ||
| | `local-certification` | `ai-lab` | Local Docker container runners (both on Z890) | | ||
| | `staging` | `cloudstartup`, `kvm4` | VPS runners for staging validation | | ||
| | `production` | `kvm4`, `kvm2` | Production deployment runners | | ||
| | `lightweight` | `ubuntu-latest` | GitHub-hosted for non-sensitive tasks | | ||
|
|
||
|
Comment on lines
+47
to
+57
Contributor
There was a problem hiding this comment. Phase policy table does not match Line 53 through Line 57 diverge from
🛠️ Suggested doc correction (table-level)-| `local-certification` | `ai-lab` | Local Docker container runners (both on Z890) |
-| `staging` | `cloudstartup`, `kvm4` | VPS runners for staging validation |
-| `production` | `kvm4`, `kvm2` | Production deployment runners |
-| `lightweight` | `ubuntu-latest` | GitHub-hosted for non-sensitive tasks |
+| `local-certification` | `self-hosted,Linux,X64`; `self-hosted,Linux,X64,ai-lab,gpu` | Local Docker container runners |
+| `staging` | `self-hosted,cloudstartup,staging` | Staging deployment on cloudstartup |
+| `vps-deployment` | `self-hosted,vps,kvm4-1`; `self-hosted,vps,kvm4-2`; `self-hosted,vps,kvm2` | VPS fleet deployment |
+| `production` | `self-hosted,kvm4,production`; `self-hosted,kvm4` (optional: `self-hosted,vps,kvm2`) | Production deployment |As per coding guidelines, " 🤖 Prompt for AI Agents |
||
| ## Runner Configuration | ||
|
|
||
| **Lane hosts:** `lane_hosts.json` defines containerized topology: | ||
|
|
||
| ```json | ||
| { | ||
| "ai-lab": {"host": "localhost", "type": "docker-container"}, | ||
| "cloudstartup": {"host": "kvm4-1.internal", "type": "docker-container"}, | ||
| "kvm4": {"host": "kvm4-2.internal", "type": "docker-container"}, | ||
| "kvm2": {"host": "kvm2.internal", "type": "docker-container"} | ||
| } | ||
| ``` | ||
|
|
||
| ## Docker-in-Docker Security | ||
|
|
||
| | Concern | Mitigation | | ||
| |---------|-----------| | ||
| | Container escape | Runners use `--security-opt=no-new-privileges` | | ||
| | Volume access | Bind-mounted work directories only | | ||
| | Network access | Host network for Tailscale connectivity | | ||
| | Secrets exposure | GitHub Actions secrets injected at runtime, not stored | | ||
| | Image trust | `myoung34/github-runner` from Docker Hub (verified publisher) | | ||
|
|
||
| ## Certificate Management | ||
|
|
||
| | Certificate | Location | Rotation | Status | | ||
| |-------------|----------|----------|--------| | ||
| | Runner registration token | GitHub API | Per-registration | Auto | | ||
| | GitHub PAT (for runner API) | `env.shared` | Manual (90 day) | Tracked | | ||
| | TLS certs (VPS nodes) | Let's Encrypt on KVM2 | Auto-renewal (certbot) | GREEN | | ||
| | Docker TLS | Docker daemon | Manual | Tracked | | ||
|
|
||
| ## Upstream Dependencies | ||
|
|
||
| | Dependency | Type | Required | | ||
| |------------|------|----------| | ||
| | GitHub API | Runner registration, workflow dispatch | Yes | | ||
| | Docker Engine | Container runtime for runners | Yes | | ||
| | Tailscale mesh | Inter-node connectivity for distributed runners | Yes | | ||
| | `local_cert_runners.py` | Runner lifecycle management | Yes | | ||
| | `myoung34/github-runner` | Runner container image | Yes | | ||
|
|
||
| ## Production Audit Checklist | ||
|
|
||
| | Requirement | Status | Notes | | ||
| |-------------|--------|-------| | ||
| | All runners online | Partial | 3/4 online after `make ci-runners-local-cert-up`; KVM runners need VPS access | | ||
| | Runner labels match workflows | GREEN | Validated via `runner_phase_policy.json` | | ||
| | Docker-in-Docker security | Partial | `no-new-privileges` set; full audit pending | | ||
| | Certificate rotation | Partial | Let's Encrypt auto-renews; Docker TLS manual | | ||
| | Monitoring | GREEN | cAdvisor + Prometheus scraping container metrics | | ||
|
|
||
| ## Security Stance | ||
|
|
||
| | Finding | Severity | Status | | ||
| |---------|----------|--------| | ||
| | Runner containers run as root | P2 | Tracked — `myoung34` image requires root for Docker socket | | ||
| | `apt-get` needs `sudo` on self-hosted runners | P3 | Documented — non-root runners can't install system deps | | ||
| | Docker Bench CI fails on Windows runner | P3 | Fixed (PR #846) — `if: runner.os == 'Linux'` guard | | ||
| | GitHub PAT rotation not automated | P3 | Tracked | | ||
|
|
||
| ## Cross-Links | ||
|
|
||
| - **Infrastructure TAC:** [`TAC_INFRASTRUCTURE.md`](./TAC_INFRASTRUCTURE.md) | ||
| - **Workflow Runner Map:** `pmoves/docs/operations/WORKFLOW_RUNNER_MAP.md` | ||
| - **Hybrid Runner Strategy:** `deploy/HYBRID_RUNNER_STRATEGY.md` | ||
| - **Runner Topology:** `.claude/context/runner-topology.md` | ||
| - **Lane Hosts Config:** `lane_hosts.json` | ||
| - **Phase Policy:** `runner_phase_policy.json` | ||
| - **Local Runner Script:** `local_cert_runners.py` | ||
| - **VPS Provisioning:** `deploy/provision/hostinger-kvm-setup.sh` | ||
|
|
||
| ## Open Items | ||
|
|
||
| - Automate GitHub PAT rotation (currently manual 90-day cycle) | ||
| - Non-root runner image evaluation (security improvement) | ||
| - Runner health monitoring via NATS (currently no NATS integration) | ||
| - KVM runner cert rotation automation | ||
| - Runner auto-scaling for burst CI load (currently fixed fleet) | ||
|
|
||
| <!-- GRAPHITI_MARK: CLAUDE-OPUS::TAC-RUNNERS::2026-03-15 --> | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Usage comments reference removed Make targets.
Line 7 through Line 9 still point to
tailscale-up/status/ip, but the active targets aretailscale-docker-up/status/ipinpmoves/mk/infra.mk. This will cause operator command failures.🛠️ Suggested comment fix
📝 Committable suggestion
🤖 Prompt for AI Agents