[FEATURE] Security Phase 2 - PyTorch 2.6, Docker hardening, env migration - #436
POWERFULMOVES wants to merge 15 commits into
Conversation
Phase 2 of env consolidation: API tier migration Changes: - Fix tier anchor syntax (block-style env_file with required: false) - Migrate postgrest, presign, retrieval-eval to <<: *env-tier-api - Create env.tier-api.example with secure defaults API tier services receive data tier URLs and internal credentials only, no external API keys (OPENAI_API_KEY, etc.) Services migrated: - postgrest: PostgREST database gateway - presign: MinIO URL presigner - retrieval-eval: Hi-RAG evaluation service 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Phase 3 of env consolidation: LLM tier migration Changes: - Fix tier anchor syntax (block-style env_file with required: false) - Migrate tensorzero-ui to <<: *env-tier-llm - Create env.tier-llm.example with all LLM provider API keys CRITICAL: LLM tier is the ONLY tier with access to external API keys. All other services call TensorZero internally, not providers directly. Services in LLM tier: - tensorzero-gateway: Already using <<: *env-tier-llm - tensorzero-ui: Migrated from legacy env_file - tensorzero-clickhouse: Uses inline env (no API keys needed) - pmoves-ollama: Uses inline env (no API keys needed) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Completes the 6-tier environment consolidation by migrating: **Agent Tier (10 services):** - mesh-agent, deepresearch, supaserch, consciousness-service - archon-agent-work-orders, botz-gateway, publisher-discord - messaging-gateway, jellyfin-bridge, chat-relay **API Tier Additions (4 services):** - postgrest-cli, hi-rag-gateway, hi-rag-gateway-gpu, hi-rag-gateway-v2-gpu **Worker Tier (6 services) - included in this commit:** - render-webhook, comfy-watcher, pdf-ingest, langextract - notebook-sync, session-context-worker **Media Tier (10 services) - included in this commit:** - ultimate-tts-studio, flute-gateway, ffmpeg-whisper - media-video, media-audio, channel-monitor - invidious, invidious-companion, grayjay-plugin-host, grayjay-server Security improvement: 30 services now use tier-based env_file anchors instead of legacy x-env-legacy pattern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Phase 2 hardening: Docker security, Prometheus patterns, DNS fixes
Addresses critical CVE in torch.load with weights_only=True. Changes: - torch: 2.3.1 → 2.6.0 - torchvision: 0.18.1 → 0.21.0 - torchaudio: 2.3.1 → 2.6.0 - CUDA wheels: cu121 → cu124⚠️ Breaking: Requires CUDA 12.4 runtime (was 12.1) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…e pins, required password) (#355) * docs: address PR review comments for env tier consolidation - Add GPU Orchestrator and E2B Runner to services-catalog.md - Add gpu-orchestrator to env.tier-api.example service list - Create learnings file documenting 6-tier env architecture - Add env.tier-*.example files for worker, media, agent tiers Addresses nitpick comments from PRs #349-354. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(gpu): ensure GPU access for all CUDA-enabled services - gpu-orchestrator: Add GPU deploy section + NVIDIA_VISIBLE_DEVICES - ultimate-tts-studio: Add NVIDIA_VISIBLE_DEVICES env var - hi-rag-gateway-gpu: Add NVIDIA_VISIBLE_DEVICES env var - hi-rag-gateway-v2-gpu: Add NVIDIA_VISIBLE_DEVICES env var - media-audio: Change base image to nvidia/cuda:12.4.1-runtime-ubuntu22.04 (was python:3.11-slim which caused silent CPU fallback) This fixes silent CPU fallbacks where PyTorch CUDA packages were installed but the CUDA runtime was not available in the container. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(security): production hardening - health checks and image pinning Critical security and reliability fixes: 1. **POSTGRES_PASSWORD now required** - Changed from insecure default `:-pmoves` to required syntax `${VAR:?error}`. Compose will fail if POSTGRES_PASSWORD is not set (lines 59, 78, 937). 2. **Pinned 8 :latest images to specific versions:** - postgrest/postgrest:latest → v12.2.3 - minio/minio:latest → RELEASE.2024-12-18T13-15-44Z - ollama/ollama:latest → 0.5.4 - tensorzero/gateway:latest → 2024.12.18 - tensorzero/ui:latest → 2024.12.18 - invidious:latest → 2024.12.09 - invidious-companion:latest → 2024.12.20 - grayjay:latest → 2024.11.01 3. **Added health checks to 37 services** (52 total, up from 15): - Data tier: qdrant, meilisearch, minio - API tier: hi-rag-*, retrieval-eval, presign, render-webhook - Worker tier: extract-worker, pdf-ingest, langextract, notebook-sync, ffmpeg-whisper, media-video, media-audio, pmoves-yt, channel-monitor - Agent tier: agent-zero, mesh-agent, deepresearch, supaserch - TensorZero: gateway, ui, ollama - Others: publisher-discord, messaging-gateway, jellyfin-bridge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(health): address PR review critical issues Fixes critical issues identified by PR review: MinIO: - Change health check from `mc ready local` to curl-based (minio/minio image doesn't include mc binary) NATS: - Add `-m 8222` flag to enable HTTP monitoring port (health check was targeting port that wasn't enabled) Health checks: - comfy-watcher: verify module imports instead of just `import sys` - mesh-agent: verify main module and NATS client availability Missing start_period: - postgres: add 15s start_period - chat-relay: add 30s start_period - n8n-agent: add 30s start_period - invidious-postgres: add 15s start_period All 52 health checks now have start_period defined. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Increase Dockerfile hardening from 33/36 (91.7%) to 35/36 (97.2%): - notebook-sync: Add USER pmoves (user creation already existed) - vibevoice-realtime: Add full hardening with UID/GID 65532 Accepted exception: agent-zero uses root for initialization then drops to pmoves via 'su' for the service process. This pattern is required by upstream Agent Zero's prepare.py and /ins/copy_A0.sh. Pattern reference: flute-gateway/Dockerfile 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Normalize Supabase Kong hostnames to lowercase (DNS convention) - Remove duplicate ARCHON_SUPABASE_BASE_URL env var definition - Fix notebook-sync Dockerfile: remove USER directive so entrypoint can run chown/su as root before dropping privileges - Improve deepresearch _get_or_create_counter: use module-level cache to avoid private prometheus_client API dependency 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Changes: - Archon health check: Add Kong gateway hostname support for internal Docker networking (supabase_kong_pmoves.ai:8000) - UI dashboard: Add /dashboard redirect page to /dashboard/services - Grafana: Fix env var placeholders and job name mismatches in queries - Docs: Add placeholder for Jellyfin service documentation The Archon health check now properly handles both: - Supabase CLI endpoint (host.docker.internal:65421) - Internal Kong gateway (supabase_kong_pmoves.ai:8000) This resolves the 404 errors in Archon health checks when using internal Docker DNS names instead of host.docker.internal. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add pmoves/data/agent-zero/ to gitignore. This directory contains runtime settings (settings.json) that persist via Docker volume mount. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use registry lookup pattern instead of try/except for counter registration. Prevents 'Duplicated timeseries' error when container restarts with existing registry state. Before: try/except around Counter() creation After: REGISTRY._names_to_collectors.get() for existing lookup 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add warning log to Supabase health check exception handler - Improve persona fetch error logging with status code and truncated body - Add metrics tracking for non-200 persona fetch responses Closes #322 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* fix(voice): harden WSL2 playback and Makefile voice-say target - Add OSError handling in _play_wav_via_windows() for environments where Windows binary interop is disabled (hardened WSL, nested shells) - Fix voice-say Makefile target: proper bash -lc wrapping, jq quoting - Update doc reference: PMOVES.AI-Edition-Hardened.md → Hardened-Full.md 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(flute): add Ultimate-TTS-Studio provider - Add UltimateTTSProvider for Gradio API integration - Support multiple engines: kitten_tts, f5_tts, kokoro, indextts2 - Update synthesize endpoints to support ultimate_tts provider - Add engine parameter to SynthesizeRequest model - Register provider in health check and config endpoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(flute): correct Ultimate-TTS API parameter handling - Build full 92-parameter list for generate_unified_tts endpoint - Add ENGINE_NAMES mapping (internal -> Gradio API names) - Set required indextts2_emotion_description parameter - Configure engine-specific voice parameters at correct indices 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: notebook-sync API path and Ultimate-TTS config - Fix notebook-sync to use correct /api/{resource} endpoint path Open Notebook API requires /api/ prefix for all endpoints - Add ULTIMATE_TTS_URL to flute-gateway docker-compose config Points to Pinokio instance on port 7863 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(submodule): update PMOVES-BoTZ with n8n-agent 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(n8n): upgrade to v2.1.0 from 1.123.4 - n8n 2.x uses minimal BusyBox base (no apk) - Removed sqlite-cli customization - Updated runners to 2.1.0 Note: API key must be regenerated in n8n UI after upgrade 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(n8n): add 9 new workflow exports for PMOVES automation Add new n8n workflow definitions for: - pmoves_audio_analysis: Audio processing pipeline - pmoves_video_analysis: Video processing pipeline - pmoves_channel_monitor: YouTube channel watching - pmoves_comfy_hub: ComfyUI orchestration - pmoves_deepresearch_orchestrator: Research automation - pmoves_ingestion_hub: Content ingestion hub - pmoves_jellyfin_watcher: Jellyfin event handling - pmoves_notebook_content_feed: Open Notebook sync - pmoves_social_publisher: Social media publishing Also updates contracts/topics.json with new event subjects. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(mcp): add n8n-agent MCP config with TensorZero integration - Add pmoves/config/mcp/n8n-agent.yaml for Claude Code/Codex integration - Configure stdio transport for docker-based MCP server - Include TensorZero environment variables for local LLM inference - Update PMOVES-BoTZ submodule with TensorZero and monitor agent The n8n-agent is now registered with Claude Code CLI and can be used for workflow automation with intelligent suggestions powered by local models. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Codex Agent <codex-agent@example.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Merged after rebase to resolve Jellyfin README conflict
- Add volume mount ./data/agent-zero/tmp:/a0/tmp for settings.json Root cause: PMOVES-Agent-Zero/python/helpers/settings.py:162 stores settings at /a0/tmp/settings.json but path was not mounted - Normalize Archon SUPABASE_URL to lowercase supabase_kong_pmoves.ai Root cause: Docker DNS is case-sensitive on some configurations - Fix archon-agent-work-orders health check: /healthz → /health Root cause: Service exposes /health, not /healthz 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3983831714
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| RUN groupadd -r pmoves --gid=65532 && \ | ||
| useradd -r -g pmoves --uid=65532 --home-dir=/app --shell=/sbin/nologin pmoves && \ | ||
| mkdir -p /models && \ | ||
| chown -R pmoves:pmoves /app /app/vibevoice /models | ||
| chown -R pmoves:pmoves /app /app/vibevoice |
There was a problem hiding this comment.
Ensure /models is writable for VibeVoice at runtime
The container now runs as the non-root pmoves user, but the Dockerfile no longer creates or chowns /models. In the default compose (docker-compose.voice.yml), /models is a named volume, which is root-owned on first mount. The entrypoint writes model files under /models (and /models/.hf), so fresh installs will hit “permission denied” and the service won’t start. Consider creating/chowning /models in the image or dropping privileges after a root-owned chown in the entrypoint.
Useful? React with 👍 / 👎.
| @$(LOAD_ENV_SHARED) \ | ||
| if [ -z "$$CLOUDFLARE_TUNNEL_TOKEN" ] && [ -z "$$CLOUDFLARE_TUNNEL_NAME" ]; then \ | ||
| echo "↷ Set CLOUDFLARE_TUNNEL_TOKEN (Zero Trust token) or CLOUDFLARE_TUNNEL_NAME + credentials in $(ENV_SHARED_FILE) before running make up-cloudflare."; \ | ||
| exit 1; \ | ||
| fi; \ | ||
| exec docker compose -p $(PROJECT) --profile cloudflare up -d cloudflared' | ||
| echo "↷ Set CLOUDFLARE_TUNNEL_TOKEN (Zero Trust token) or CLOUDFLARE_TUNNEL_NAME + credentials in $(ENV_SHARED_FILE) before running make up-cloudflare."; \ |
There was a problem hiding this comment.
Guard optional env checks against set -u
LOAD_ENV_SHARED now sources scripts/with-env.sh, which enables set -u. That makes the if [ -z "$$CLOUDFLARE_TUNNEL_TOKEN" ] guard fail with “unbound variable” when the token is unset, so the target exits before showing the intended help message. This is a regression for users who haven’t configured Cloudflare yet. Use ${VAR:-} or disable set -u after loading env.
Useful? React with 👍 / 👎.
|
Closing for rebase - will create new PR with resolved conflicts |
Summary
This PR contains 13 commits from `feat/hardened-security-phase2` not yet in hardened.
Changes
Security Hardening
Environment Migration (6-Tier Architecture)
Infrastructure Fixes
Documentation
Testing
```bash
docker compose -p pmoves config --quiet
curl -sf http://localhost:8080/healthz # Agent Zero
curl -sf http://localhost:8091/healthz # Archon
```
Related
Follows env migration pattern from PR #435 (submodule-sync)