Skip to content

feat(benchmarks): integrate llama-throughput-lab into compose & observability #663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
POWERFULMOVES merged 2 commits into from
Feb 19, 2026
Merged

feat(benchmarks): integrate llama-throughput-lab into compose & observability #663

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ca2a0bf
feat(benchmarks): integrate llama-throughput-lab into compose and obs…
hunnibear Feb 19, 2026
33fb93f
fix(pr-663): correct CIPHER_MEMORY_URL to cipher-api service name
hunnibear Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion PMOVES-llama-throughput-lab
Open in desktop
Submodule PMOVES-llama-throughput-lab updated 17 files
+3 −0 .gitignore
+35 −0 Dockerfile
+33 −19 PMOVES.AI_INTEGRATION.md
+51 −2 README.md
+75 −0 analyze-data.py
+141 −0 pmoves_chit_bridge.py
+148 −0 pmoves_metrics.py
+152 −0 pmoves_nats_bridge.py
+5 −0 requirements-platform.txt
+127 −7 run_llama_tests.py
+5 −0 scripts/full_sweep.py
+147 −131 scripts/round_robin_sweep.py
+6 −1 start_llama_rr.sh
+37 −7 tests/llama_server_test_utils.py
+4 −0 tests/test_llama_server_concurrent.py
+4 −0 tests/test_llama_server_round_robin.py
+4 −0 tests/test_llama_server_threads_sweep.py
6 changes: 5 additions & 1 deletion pmoves/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1246,7 +1246,7 @@ up-legacy-both: ## Ensure v1 hi-rag gateway CPU+GPU are up
@$(DC) --profile gpu --profile legacy up -d hi-rag-gateway-gpu >/dev/null 2>&1 || true
@echo "✔ hi-rag-gateway v1: CPU(:8089) + GPU(:8090 if GPUs present)."

.PHONY: up down clean smoke smoke-prod smoke-gpu smoke-qwen-rerank manifest-audit
.PHONY: up down clean smoke smoke-prod smoke-gpu smoke-qwen-rerank llama-throughput-smoke manifest-audit
.PHONY: up down clean up-workers up-yt channel-monitor-up channel-monitor-smoke up-media up-jellyfin up-nats ps supabase-up supabase-stop supabase-clean supa-extract-remote env-setup env-check
.PHONY: up down clean up-cloudflare down-cloudflare logs-cloudflare cloudflare-url restart-cloudflare up-workers up-yt up-media up-jellyfin up-nats ps supabase-up supabase-stop supabase-clean supa-extract-remote env-setup env-check manifest-audit
.PHONY: discord-ping discord-ping-ps demo-content-published health-publisher-discord up-agents health-agent-zero health-jellyfin-bridge seed-approval seed-approval-ps m2-preflight evidence-stamp evidence-stamp-ps evidence-log evidence-log-ps m2-seed-demo n8n-webhook-demo
Expand Down Expand Up @@ -1350,6 +1350,10 @@ smoke-prod: ## Production smoke: core smoke + production monitoring/profile asse
smoke-gpu: ## Validate Hi-RAG v2 GPU rerank path (strict by default; set GPU_SMOKE_STRICT=false to relax)
@$(PYTHON) tools/smoke_gpu.py $(ARGS) || python tools/smoke_gpu.py $(ARGS) || py -3 tools/smoke_gpu.py $(ARGS)

llama-throughput-smoke: ## Quick llama throughput validation (single request test)
docker compose --profile benchmarks run --rm llama-throughput-lab \
python -m pytest tests/test_llama_server_single.py -v

smoke-qwen-rerank: ## Enforce Qwen reranker selection on v2-gpu then run strict GPU smoke
@$(PYTHON) tools/smoke_gpu.py --require-qwen --stats-only $(ARGS) || python tools/smoke_gpu.py --require-qwen --stats-only $(ARGS) || py -3 tools/smoke_gpu.py --require-qwen --stats-only $(ARGS)
@GPU_SMOKE_STRICT=true $(MAKE) --no-print-directory smoke-gpu
Expand Down
40 changes: 40 additions & 0 deletions pmoves/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2442,6 +2442,45 @@ services:
- DOCKED_MODE=${DOCKED_MODE:-true}
- PARENT_SYSTEM=${PARENT_SYSTEM:-PMOVES.AI}
- PARENT_VERSION=${PARENT_VERSION:-1.0.0-hardened}
# Llama.cpp throughput benchmarking toolkit (GPU, on-demand)
llama-throughput-lab:
build:
context: ../PMOVES-llama-throughput-lab
dockerfile: Dockerfile
image: ${LLAMA_THROUGHPUT_LAB_IMAGE:-ghcr.io/powerfulmoves/pmoves-llama-throughput-lab:latest}
restart: "no"
environment:
- LLAMA_MODEL_PATH=/models/${LLAMA_MODEL:-model.gguf}
- LLAMA_SERVER_HOST=127.0.0.1
- LLAMA_RESULTS_DIR=/results
- NATS_URL=${NATS_URL:-nats://nats:pmoves@nats:4222}
- CIPHER_MEMORY_URL=http://cipher-memory:8096

@chatgpt-codex-connector chatgpt-codex-connector Bot Feb 19, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Point CIPHER_MEMORY_URL at the actual cipher-api service

The new llama-throughput-lab wiring sets CIPHER_MEMORY_URL to http://cipher-memory:8096, but this compose stack does not define a cipher-memory service/alias (the Cipher backend service here is cipher-api in pmoves/docker-compose.yml). In any run where the benchmark lab tries to use the Cipher bridge, that hostname will fail DNS resolution on the Docker network and the integration introduced by this commit will be non-functional.

Useful? React with 👍 / 👎.

- GPU_ORCHESTRATOR_URL=http://gpu-orchestrator:8200
- METRICS_PORT=8201
volumes:
- ${LLAMA_MODELS_PATH:-./models}:/models:ro
- llama-results:/results
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
depends_on:
nats:
condition: service_healthy
required: false
profiles:
- benchmarks
- gpu
networks:
- pmoves_api
- pmoves_bus
- pmoves_monitoring
ports:
- 8201:8201

volumes:
neo4j-data: {}
qdrant-data: {}
Expand All @@ -2459,6 +2498,7 @@ volumes:
pmoves-ollama-models: {}
comfy-watcher-output: {}
comfy-watcher-state: {}
llama-results: {}
networks:
pmoves_data:
name: pmoves_data
Expand Down
12 changes: 12 additions & 0 deletions pmoves/monitoring/prometheus/prometheus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,18 @@ scrape_configs:
static_configs:
- targets: ["messaging-gateway:8101"]

# === GitHub Runner Controller metrics ===
# Exposes github_runner_* metrics matching the github-runners Grafana dashboard
- job_name: github-runner-ctl
static_configs:
- targets: ["github-runner-ctl:8104"]

# === Benchmark Lab metrics ===

- job_name: llama-throughput-lab
static_configs:
- targets: ["llama-throughput-lab:8201"]

# === Blackbox HTTP probes ===
# Health check probes for services without /metrics

Expand Down
Loading