Skip to content

fix(infra): TensorZero ClickHouse and GPU Orchestrator startup fixes #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
POWERFULMOVES merged 3 commits into from
Jan 4, 2026
Merged

fix(infra): TensorZero ClickHouse and GPU Orchestrator startup fixes #451

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
26b4eef
fix(infra): Resolve Archon Supabase connection and add TensorZero config
Jan 3, 2026
bb01b08
fix(infra): TensorZero ClickHouse and GPU Orchestrator startup fixes
Jan 3, 2026
70a71ab
test: add smoke test evidence for PR #451
Jan 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 121 additions & 0 deletions pmoves/PR_EVIDENCE/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Testing Evidence for PR #451

**Date:** 2026-01-03
**Branch:** `infra-fix-tensorzero-gpu`
**Base:** `PMOVES.AI-Edition-Hardened`

---

## Summary

All core infrastructure fixes from PR #451 have been validated successfully. Services are running and responding to health checks.

---

## Services Validated

### Service Count
- **Running:** 36 containers
- **Target:** 35+ ✓

### Health Check Results

| Service | Port | Status | Notes |
|---------|------|--------|-------|
| TensorZero Gateway | 3030 | ✓ | ClickHouse+Postgres OK |
| TensorZero ClickHouse | 8123 | ✓ | Responding to /ping |
| GPU Orchestrator | 8100 | ✓ | RTX 5090 detected (40% VRAM) |
| Ollama | 11434 | ✓ | 9 models available |
| Archon | 8091 | ✓ | Supabase connection fixed |
| Agent Zero | 8080 | ✓ | NATS connected |
| Hi-RAG v2 | 8086 | ✓ | HTTP responding |
| Flute-Gateway | 8055 | ✓ | Healthy |
| SupaSerch | 8099 | ✓ | Metrics endpoint OK |
| Postgres | 5432 | ✓ | pgvector enabled |
| Qdrant | 6333 | ✓ | Vector DB responding |
| Neo4j | 7474 | ✓ | Knowledge graph OK |
| Meilisearch | 7700 | ✓ | Full-text search OK |
| PostgREST | 3010 | ✓ | API gateway OK |
| Grafana | 3002 | ✓ | Dashboards accessible |
| Prometheus | 9090 | ✓ | API responding |

### Additional Services Running
- NATS (4222), DeepResearch, Publisher-Discord, Render-Webhook, PDF-Ingest, LangExtract, Extract-Worker, Media-Video, Media-Audio, Jellyfin-Bridge, PMOVES.YT, Evo-Controller, Mesh-Agent, Session-Context-Worker, Retrieval-Eval, and more.

---

## Infrastructure Fixes Validated

### 1. Archon Supabase Connection ✓
- **Fixed:** Changed `supabase_kong_PMOVES.AI:8000` → `postgrest:3000`
- **Result:** Health endpoint returns `{"status":"ok", "supabase":{"http":200}}`

### 2. Ollama Image ✓
- **Fixed:** Changed `pmoves/ollama:0.12.6` → `ollama/ollama:latest`
- **Result:** Container running, API responding on port 11434

### 3. ClickHouse IPv6 (WSL2) ✓
- **Fixed:** Added `listen.xml` for IPv4-only binding
- **Result:** ClickHouse listening on 0.0.0.0:8123, TensorZero connecting successfully

### 4. GPU Orchestrator ✓
- **Fixed:** Using `--runtime=nvidia` with proper GPU access
- **Result:** RTX 5090 detected, health endpoint responding

---

## Test Infrastructure Findings

### Issues Identified

1. **Missing Smoke Test Source Files**
- The pytest smoke tests (`test_health_endpoints.py`, etc.) don't exist as source files in either main or worktree
- Only `.pyc` cache files were found
- **Action:** Smoke test source files need to be created/committed
Comment on lines +70 to +74

@coderabbitai coderabbitai Bot Jan 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Track the missing smoke test source files as an issue.

The absence of smoke test source files (test_health_endpoints.py, etc.) with only .pyc cache files present is a significant test infrastructure gap that should be tracked and resolved.

Do you want me to create a tracking issue for recreating the missing smoke test source files? I can draft an issue description with the expected file structure and smoke test patterns based on the test commands shown in this document.

🤖 Prompt for AI Agents 
In pmoves/PR_EVIDENCE/testing.md around lines 70 to 74, the pytest smoke test
source files (e.g., test_health_endpoints.py) are missing from the repo and only
.pyc cache files are present; create and commit the missing smoke test source
files under the expected tests/ (or designated) directory, following the
document's test command patterns and naming conventions (e.g.,
test_health_endpoints.py), include minimal assertions to exercise the health
endpoints, add appropriate imports and fixtures, and ensure they are added to
version control and referenced in CI/test config so pytest discovers them.


2. **NATS Smoke Test Bug**
- `tools/supaserch_smoke.py` has incorrect API call: `nc.unsubscribe()` should be `nc.unsubscribe(subject, sid)`
- **Action:** Bug fix needed for NATS smoke tests
Comment on lines +76 to +78

@coderabbitai coderabbitai Bot Jan 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Track the NATS smoke test bug as an issue.

The bug in tools/supaserch_smoke.py (nc.unsubscribe() missing required parameters) should be tracked and fixed to enable proper NATS connectivity testing.

Do you want me to create a tracking issue for this bug fix with the correct API signature?

🤖 Prompt for AI Agents 
In pmoves/PR_EVIDENCE/testing.md around lines 76 to 78, document and fix the
NATS smoke test bug: the call nc.unsubscribe() is missing required parameters —
change usage to nc.unsubscribe(subject, sid) (or nc.unsubscribe(sid) depending
on client API) and ensure the subscription id (sid) and subject are captured
when subscribing; update the script (tools/supaserch_smoke.py) to
store/propagate sid, pass the correct arguments to unsubscribe, and add minimal
error handling/logging around subscribe/unsubscribe to surface failures; track
this change as a bug/issue entry for the NATS smoke tests.


### Workarounds Applied

- Used manual curl-based health checks instead of pytest smoke tests
- Validated NATS connectivity via service logs (all NATS-using services running)

---

## Monitoring Dashboards

| Dashboard | URL | Status |
|-----------|-----|--------|
| Grafana | http://localhost:3002 | ✓ Accessible |
| Prometheus | http://localhost:9090 | ✓ API responding |
| TensorZero UI | http://localhost:4000 | Not started |

---

## Test Commands Used

```bash
# Service count
docker ps --format "table {{.Names}}\t{{.Status}}" | grep pmoves | wc -l

# Health checks
curl -sf http://localhost:3030/health # TensorZero
curl -sf http://localhost:8091/healthz # Archon
curl -sf http://localhost:8080/healthz # Agent Zero
curl -sf http://localhost:8100/healthz # GPU Orchestrator
curl -sf http://localhost:8086/ # Hi-RAG v2
curl -sf http://localhost:8123/ping # ClickHouse
curl -sf http://localhost:11434/api/tags # Ollama
```

---

## Conclusion

All infrastructure fixes from PR #451 are validated and working. The hardened branch is ready for review with all core services operational.

**Test Execution Date:** 2026-01-03
**Total Services:** 36 containers running
**Health Check Pass Rate:** 100% (for manually tested services)
Loading