fix(infra): production audit sweep — VPS, runners, CHIT, CI #825
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,8 +29,7 @@ concurrency: | |
|
|
||
| jobs: | ||
| tests: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| actions: read | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,8 +23,7 @@ permissions: | |
|
|
||
| jobs: | ||
| lint: | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| steps: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| { | ||
| "run": [ | ||
| { | ||
| "method": "shell.run", | ||
| "params": { | ||
| "message": [ | ||
| "echo 'Deploying KVM2 (Exit Node) via Tailscale SSH...'", | ||
| "echo 'NOTE: SSH as root — key-only auth enforced (PasswordAuthentication no)'", | ||
| "ssh root@pmoves-kvm2 'cd /opt/pmoves && git pull --ff-only origin main && cd pmoves && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml pull nginx && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml up -d nginx'", | ||
| "echo 'Verifying exit node...'", | ||
| "ssh root@pmoves-kvm2 'tailscale status | head -5'", | ||
| "echo 'KVM2 deployment complete.'" | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "run": [ | ||
| { | ||
| "method": "shell.run", | ||
| "params": { | ||
| "message": [ | ||
| "echo 'Deploying KVM4-1 (API Gateway) via Tailscale SSH...'", | ||
| "echo 'NOTE: SSH as root — key-only auth enforced (PasswordAuthentication no)'", | ||
| "ssh root@pmoves-kvm4-1 'cd /opt/pmoves && git pull --ff-only origin main && cd pmoves && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml pull tensorzero-gateway agent-zero hi-rag-gateway-v2 archon-server mesh-agent gateway-agent extract-worker && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml up -d tensorzero-gateway agent-zero hi-rag-gateway-v2 archon-server mesh-agent gateway-agent extract-worker'", | ||
| "echo 'Verifying health...'", | ||
| "ssh root@pmoves-kvm4-1 'curl -sf http://localhost:8080/healthz && echo \" Agent Zero: OK\" || echo \" Agent Zero: FAIL\"'", | ||
| "ssh root@pmoves-kvm4-1 'curl -sf http://localhost:3030/healthz && echo \" TensorZero: OK\" || echo \" TensorZero: FAIL\"'", | ||
| "echo 'KVM4-1 deployment complete.'" | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| { | ||
| "run": [ | ||
| { | ||
| "method": "shell.run", | ||
| "params": { | ||
| "message": [ | ||
| "echo 'Deploying KVM4-2 (Data Services) via Tailscale SSH...'", | ||
| "echo 'NOTE: SSH as root — key-only auth enforced (PasswordAuthentication no)'", | ||
| "ssh root@pmoves-kvm4-2 'cd /opt/pmoves && git pull --ff-only origin main && cd pmoves && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml pull supabase-db supabase-rest qdrant neo4j meilisearch nats prometheus grafana loki minio && docker compose -f docker-compose.yml -f docker-compose.vps.override.yml up -d supabase-db supabase-rest qdrant neo4j meilisearch nats prometheus grafana loki minio'", | ||
|
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Check if prometheus, grafana, loki, minio are defined in base compose
echo "=== Checking for observability services in base compose ==="
rg -n "^\s*(prometheus|grafana|loki|minio):" pmoves/docker-compose.yml
echo ""
echo "=== Checking VPS override for these services ==="
rg -n "^\s*(prometheus|grafana|loki|minio):" pmoves/docker-compose.vps.override.ymlRepository: POWERFULMOVES/PMOVES.AI Length of output: 297 🏁 Script executed: #!/bin/bash
# Search more broadly for these services - they might be defined differently
echo "=== Looking for service definitions more broadly ==="
echo "Base compose file grep for all service keys:"
rg -n "^\s+[a-z_-]+:\s*$" pmoves/docker-compose.yml | head -50
echo ""
echo "=== Checking full line context for minio definition ==="
rg -B2 -A10 "^\s*minio:\s*$" pmoves/docker-compose.yml | head -40
echo ""
echo "=== Checking if prometheus, grafana, loki exist anywhere ==="
rg -i "prometheus|grafana|loki" pmoves/docker-compose.yml | head -20Repository: POWERFULMOVES/PMOVES.AI Length of output: 2715 🏁 Script executed: #!/bin/bash
# Look specifically for top-level services (two spaces, followed by service name, colon at end of line)
echo "=== All services defined in base compose ==="
rg -n "^ [a-z_-]+:\s*$" pmoves/docker-compose.yml
echo ""
echo "=== All services defined in VPS override ==="
rg -n "^ [a-z_-]+:\s*$" pmoves/docker-compose.vps.override.ymlRepository: POWERFULMOVES/PMOVES.AI Length of output: 3919 🏁 Script executed: #!/bin/bash
# Check if prometheus, grafana, loki are referenced as services anywhere or if they're part of pmoves_monitoring
echo "=== Checking pmoves_monitoring service definition ==="
rg -A30 "^\s+pmoves_monitoring:\s*$" pmoves/docker-compose.yml
echo ""
echo "=== Double-check: can we find prometheus, grafana, loki as service definitions? ==="
grep -n "^ prometheus:" pmoves/docker-compose.yml || echo "prometheus: NOT FOUND"
grep -n "^ grafana:" pmoves/docker-compose.yml || echo "grafana: NOT FOUND"
grep -n "^ loki:" pmoves/docker-compose.yml || echo "loki: NOT FOUND"
echo ""
echo "=== Check if these services are in VPS override ==="
grep -n "prometheus\|grafana\|loki" pmoves/docker-compose.vps.override.yml || echo "NONE FOUND"Repository: POWERFULMOVES/PMOVES.AI Length of output: 1060 🏁 Script executed: #!/bin/bash
# Let's look at the actual deployment script to confirm what's being deployed
echo "=== Viewing the deployment script file ==="
cat -n pbnj/pinokio/api/pmoves-pbnj/kvm4-2-deploy.json | head -20
echo ""
echo "=== Checking if neo4j is defined ==="
grep -n "^ neo4j:" pmoves/docker-compose.yml || echo "neo4j: NOT FOUND"Repository: POWERFULMOVES/PMOVES.AI Length of output: 1381 Deployment command references non-existent services: The deployment command on line 9 attempts to pull and start Either:
Confirm whether these observability services should be deployed on KVM4-2. 🤖 Prompt for AI Agents |
||
| "echo 'Verifying health...'", | ||
| "ssh root@pmoves-kvm4-2 'curl -sf http://localhost:9090/api/v1/targets > /dev/null && echo \" Prometheus: OK\" || echo \" Prometheus: FAIL\"'", | ||
| "ssh root@pmoves-kvm4-2 'curl -sf http://localhost:6333/healthz > /dev/null && echo \" Qdrant: OK\" || echo \" Qdrant: FAIL\"'", | ||
| "echo 'KVM4-2 deployment complete.'" | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| { | ||
| "run": [ | ||
| { | ||
| "method": "shell.run", | ||
| "params": { | ||
| "message": [ | ||
| "echo '=== PMOVES.AI VPS Fleet Status ==='", | ||
| "echo ''", | ||
| "echo 'KVM4-1 (API Gateway):'", | ||
| "tailscale ping --timeout 3s pmoves-kvm4-1 > /dev/null 2>&1 && echo ' Status: ONLINE' || echo ' Status: OFFLINE'", | ||
| "echo ''", | ||
| "echo 'KVM4-2 (Data Services):'", | ||
| "tailscale ping --timeout 3s pmoves-kvm4-2 > /dev/null 2>&1 && echo ' Status: ONLINE' || echo ' Status: OFFLINE'", | ||
| "echo ''", | ||
| "echo 'KVM2 (Exit Node):'", | ||
| "tailscale ping --timeout 3s pmoves-kvm2 > /dev/null 2>&1 && echo ' Status: ONLINE' || echo ' Status: OFFLINE'", | ||
| "echo ''", | ||
| "echo 'GitHub Actions Runners:'", | ||
| "gh api repos/POWERFULMOVES/PMOVES.AI/actions/runners --jq '.runners[] | \" \\(.name): \\(.status)\"' 2>/dev/null || echo ' (unable to fetch)'", | ||
| "echo ''", | ||
| "echo 'Tailscale Mesh:'", | ||
| "tailscale status 2>/dev/null | head -10 || echo ' Tailscale not connected'" | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧩 Analysis chain
🏁 Script executed:
Repository: POWERFULMOVES/PMOVES.AI
Length of output: 4403
Remove the
kvm4label fromkvm4-2instances to prevent unintended job routing.Workflows in
.github/workflows/self-hosted-builds.yml:270and.github/workflows/self-hosted-builds-hardened.yml:381targetruns-on: [self-hosted, kvm4, production]. Bothkvm4-1andkvm4-2currently emit thekvm4label, making the data-services node (kvm4-2) eligible for these legacy jobs. Keep thekvm4alias onkvm4-1only.Fix required in two files:
kvm4,from the kvm4-2 label emission..claude/scripts/setup-runner.sh, line 25**: UpdateHOST_LABELS["kvm4-2"]to remove thekvm4` label.After these changes:
kvm4-1→ emitskvm4(eligible for legacy workflows)kvm4-2→ does NOT emitkvm4(not eligible for legacy workflows)🤖 Prompt for AI Agents