Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .claude/context/services-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,48 @@ Comprehensive reference of all production services, ports, APIs, and integration
- **Dependencies:** Open Notebook, LangExtract, Extract Worker
- **Compose Profile:** `orchestration`

## GPU & Compute Services

### GPU Orchestrator
- **Ports:** 8200
- **Purpose:** VRAM management, model lifecycle, and GPU priority queue
- **Key APIs:**
- `GET /healthz` - Service health
- `GET /models` - List loaded models
- `POST /models/load` - Load model to GPU
- `POST /models/unload` - Unload model from GPU
- **Features:**
- Multi-model VRAM allocation
- Priority queue for GPU compute requests
- Automatic model eviction on VRAM pressure
- Integration with Ollama, vLLM, TTS backends
- **Environment:**
- `GPU_ORCHESTRATOR_MAX_MODELS=3` - Max concurrent models
- `GPU_ORCHESTRATOR_VRAM_THRESHOLD=0.9` - Eviction threshold
- `OLLAMA_BASE_URL`, `VLLM_BASE_URL`, `TTS_BASE_URL`
- **Dependencies:** NATS (optional)
- **Compose Profile:** `gpu`
- **Env Tier:** `env-tier-api`

### E2B Runner (pmz-e2b-runner)
- **Ports:** 7071
- **Purpose:** Isolated code execution sandbox (self-hosted)
- **Key APIs:**
- `GET /health` - Service health
- `POST /execute` - Execute code in sandbox
- **Features:**
- Secure code execution in isolated containers
- Python, JavaScript, shell code support
- MCP gateway integration for tool access
- GPU support on ai-lab runner (optional)
- **Deployment:**
- **ai-lab:** Self-hosted with GPU access
- **VPS/proxmox:** CPU-only mode
- **Environment:**
- `E2B_API_KEY` - E2B API key (if using cloud fallback)
- `MCP_GATEWAY_URL` - Docker MCP gateway endpoint
- **Compose Profile:** `workers`, `botz`

## Utility & Integration Services

### Presign
Expand Down Expand Up @@ -353,6 +395,41 @@ Comprehensive reference of all production services, ports, APIs, and integration
- **Stores:** Videos, audio, images, analysis results
- **Compose Profile:** Default (always required)

## External Integrations (Optional)

### Firefly III
- **Ports:** 8080 (internal only - not published to host)
- **Container:** `cataclysm-firefly`
- **Purpose:** Personal finance management and budgeting
- **Key APIs:**
- `GET /api/v1/about` - Version and instance info
- `GET /api/v1/accounts` - List accounts
- `GET /api/v1/transactions` - List transactions
- **Authentication:** OAuth2 Personal Access Token (Bearer token)
- **Dependencies:** Supabase (via `finance_*` tables)
- **Related Tables:**
- `finance_accounts` - Account records
- `finance_budgets` - Budget tracking
- `finance_transactions` - Transaction history
- **n8n Workflows:** `pmoves/integrations/firefly-iii/*.json`
- **Documentation:** https://docs.firefly-iii.org/firefly-iii/api
- **Note:** Internal service, access via Docker network or reverse proxy

### wger (Health Tracking)
- **Container:** `cataclysm-wger`
- **Purpose:** Fitness and workout tracking
- **Related Tables:**
- `health_nutrition` - Nutrition logs
- `health_weight` - Weight tracking
- `health_workouts` - Workout records
- **n8n Workflows:** `pmoves/integrations/health-wger/*.json`

### Jellyfin
- **Container:** `cataclysm-jellyfin`
- **Purpose:** Media server and library management
- **Integration:** `pmoves-jellyfin-bridge` (port 8093) syncs events to Supabase
- **Related Submodules:** `PMOVES-Jellyfin`, `Pmoves-Jellyfin-AI-Media-Stack`

## Quick Reference

### All Service Health Endpoints
Expand Down
91 changes: 91 additions & 0 deletions .claude/learnings/env-tier-consolidation-2025-12.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Environment Tier Consolidation Learnings

**Date:** 2025-12-24
**PRs:** #349-354 (Data, API, LLM, Worker, Media, Agent tiers)

## Summary

Migrated 30+ services from legacy `x-env-legacy` pattern (all secrets to all services) to 6-tier env architecture (principle of least privilege).

## 6-Tier Architecture

| Tier | File | Services | Secret Scope |
|------|------|----------|--------------|
| `data` | `env.tier-data` | postgres, qdrant, neo4j, meilisearch, minio, nats | Infrastructure creds |
| `api` | `env.tier-api` | postgrest, hi-rag-*, presign, retrieval-eval, gpu-orchestrator | Data tier URLs |
| `llm` | `env.tier-llm` | tensorzero-gateway, tensorzero-ui, pmoves-ollama | ALL external API keys |
| `worker` | `env.tier-worker` | extract-worker, pdf-ingest, langextract, notebook-sync | Gateway URLs |
| `media` | `env.tier-media` | pmoves-yt, ffmpeg-whisper, channel-monitor, tts-*, invidious-* | MinIO, NATS |
| `agent` | `env.tier-agent` | agent-zero, archon, mesh-agent, supaserch, deepresearch | Internal URLs |

## Key Patterns

### YAML Anchor Pattern
```yaml
# Definition
x-env-tier-api: &env-tier-api
env_file:
- path: env.tier-api
required: false
- path: .env.local
required: false

# Usage
hi-rag-gateway-v2:
<<: *env-tier-api
environment:
- QDRANT_URL=${QDRANT_URL:-http://qdrant:6333}
```

### Secrets Isolation
- **LLM tier** is the ONLY tier with external API keys
- Services call TensorZero internally, not providers directly
- This is the "secrets fence" architecture

### Legacy Exceptions (kept on `x-env-legacy`)
- `pmoves-ui` - Needs vars from multiple tiers for frontend
- `nats-echo-*` - Diagnostic services
- `cloudflared` - Tunnel service

## CodeRabbit Review Patterns

1. **Service Catalog Updates Required**
- When adding services to tiers, update `services-catalog.md`
- Include the `Env Tier:` field for each service

2. **Example File Headers**
- Keep service list in header comment up to date
- Example: `# Services: postgrest, presign, retrieval-eval, hi-rag-gateway-v2, gpu-orchestrator`

3. **Required Variables Pattern**
```bash
MINIO_ACCESS_KEY= # REQUIRED: Must match env.tier-data
PRESIGN_SHARED_SECRET= # REQUIRED: Generate with `openssl rand -hex 32`
```

## Testing

```bash
# Validate compose with required vars
INVIDIOUS_HMAC_KEY=test INVIDIOUS_COMPANION_KEY=test docker compose config > /dev/null && echo "Valid"

# Count tier anchors in compose
grep -E 'env-tier-(agent|api|data|llm|media|worker)' docker-compose.yml | wc -l
# Expected: 30+
```

## Future Improvements

1. **Pre-flight Validation Script** (`scripts/env_validate.sh`)
- Check required vars are set
- Validate API key formats (sk-ant-, sk-, etc.)
- Block startup on missing required secrets

2. **UI Tier Decision**
- Currently on legacy pattern for simplicity
- Consider `env-tier-ui` when frontend stabilizes

3. **Variable Canonicalization**
- `SUPABASE_URL` vs `SUPA_REST_URL` inconsistency
- `TENSORZERO_URL` vs `TENSORZERO_BASE_URL` aliases
- Document canonical names and deprecate aliases
Loading
Loading