diff --git a/.env.example b/.env.example index 924146613c45..d1e64e9451fc 100644 --- a/.env.example +++ b/.env.example @@ -75,12 +75,16 @@ # ============================================================================= # MiniMax provides access to MiniMax models (global endpoint) # Get your key at: https://www.minimax.io -# MINIMAX_API_KEY= -# MINIMAX_BASE_URL=https://api.minimax.io/v1 # Override default base URL +# MINIMAX_API_KEY=*** +# Anthropic-compatible endpoint for MiniMax (required for prompt caching). +# If unset, Hermes auto-detects the endpoint based on the provider. +# MINIMAX_BASE_URL=https://api.minimax.io/anthropic # MiniMax China endpoint (for users in mainland China) -# MINIMAX_CN_API_KEY= -# MINIMAX_CN_BASE_URL=https://api.minimaxi.com/v1 # Override default base URL +# MINIMAX_CN_API_KEY=*** +# Anthropic-compatible endpoint for MiniMax China (required for prompt caching). +# If unset, Hermes auto-detects the endpoint based on the provider. +# MINIMAX_CN_BASE_URL=https://api.minimaxi.com/anthropic # ============================================================================= # LLM PROVIDER (OpenCode Zen) diff --git a/AGENTS_SETUP.md b/AGENTS_SETUP.md new file mode 100644 index 000000000000..b503b18d6a64 --- /dev/null +++ b/AGENTS_SETUP.md @@ -0,0 +1,102 @@ +# Agent Setup Guide + +How to set up and run the multi-agent Kanban coding roster on your machine. + +## Prerequisites + +- Hermes Agent installed and working (`hermes chat -q "hello"`) +- API keys for your preferred providers in `~/.hermes/.env` +- Git access to this repo + +## Quick Start + +```bash +# 1. Pull latest +cd ~/Projects/hermes-agent +git pull origin main + +# 2. Sync deploy target (if using gateway) +cd ~/.hermes/hermes-agent +git pull local-project main + +# 3. Create profiles (one-time) +hermes profile create riqui +hermes profile create miki +hermes profile create maxi + +# 4. Copy configs from repo +cp ~/Projects/hermes-agent/profiles/riqui/config.yaml ~/.hermes/profiles/riqui/ +cp ~/Projects/hermes-agent/profiles/miki/config.yaml ~/.hermes/profiles/miki/ +cp ~/Projects/hermes-agent/profiles/maxi/config.yaml ~/.hermes/profiles/maxi/ + +# 5. ADAPT PROVIDERS TO YOUR STACK (IMPORTANT) +# Edit each profile's config.yaml: +# - model.provider: your provider (openrouter, anthropic, nous, etc.) +# - model.default: your model name +# - model.base_url: your provider's endpoint (if needed) +# - model.api_key or symlink .env +$EDITOR ~/.hermes/profiles/riqui/config.yaml +$EDITOR ~/.hermes/profiles/miki/config.yaml +$EDITOR ~/.hermes/profiles/maxi/config.yaml + +# 6. Copy SOUL.md files +cp ~/Projects/hermes-agent/profiles/riqui/SOUL.md ~/.hermes/profiles/riqui/ +cp ~/Projects/hermes-agent/profiles/miki/SOUL.md ~/.hermes/profiles/miki/ +cp ~/Projects/hermes-agent/profiles/maxi/SOUL.md ~/.hermes/profiles/maxi/ + +# 7. Symlink .env and agent-memory +ln -sf ~/.hermes/.env ~/.hermes/profiles/riqui/.env +ln -sf ~/.hermes/.env ~/.hermes/profiles/miki/.env +ln -sf ~/.hermes/.env ~/.hermes/profiles/maxi/.env +ln -sf ~/.hermes/agent-memory ~/.hermes/profiles/riqui/agent-memory +ln -sf ~/.hermes/agent-memory ~/.hermes/profiles/miki/agent-memory +ln -sf ~/.hermes/agent-memory ~/.hermes/profiles/maxi/agent-memory + +# 8. Test each profile +hermes -p riqui chat -q "hello" --quiet +hermes -p miki chat -q "hello" --quiet +hermes -p maxi chat -q "hello" --quiet # ⚠ known issue: maxi needs api_mode fix +``` + +## Profile Reference + +| Profile | Purpose | Key config | Status | +|---------|---------|-----------|--------| +| riqui | Fast surgical coding | max_turns=30, reasoning=minimal | ✓ Working | +| miki | Deep-thinking coding (Kimi) | max_turns=30, reasoning=high | ✓ Working | +| maxi | Deep-thinking coding (MiniMax) | max_turns=30, reasoning=high, Anthropic endpoint | ⚠ API mode bug | + +## Provider Adaptation + +The profiles assume our stack (DeepSeek, Kimi OAuth, MiniMax API key). To use different providers: + +### Using OpenRouter +```yaml +model: + default: openai/gpt-5.4 # or anthropic/claude-sonnet-4-6, etc. + provider: openrouter +``` + +### Using Anthropic Direct +```yaml +model: + default: claude-sonnet-4-6-20250514 + provider: anthropic +``` + +### Using Nous Portal +```yaml +model: + default: anthropic/claude-sonnet-4-6 + provider: nous +``` + +The `agent.max_turns` and `agent.reasoning_effort` settings are provider-agnostic. + +## Kanban Worker Rules (CRITICAL) + +- All coding profiles MUST have `max_turns >= 25` and `reasoning_effort >= minimal` +- Lower values cause protocol violations (exhausted iterations before kanban_complete) +- Kanban dispatcher spawns `hermes -p --skills kanban-worker chat -q "work kanban task "` +- Workers MUST end with `kanban_complete()` or `kanban_block()` — text-only exit is a violation +- Dispatcher auto-blocks after 1 protocol violation (effective_limit=1) diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000000..d532453f3f13 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,148 @@ +# Changelog — nicoechaniz/hermes-agent fork + +> **Provider note:** Profile configs reference DeepSeek, Kimi, and MiniMax providers because that's our stack. Team members using different providers (OpenRouter, Anthropic, Nous, etc.) should adapt `model.provider`, `model.default`, and `model.base_url` in each profile's `config.yaml`. API keys go in each profile's `.env` (or symlink to shared `.env`). The `max_turns` and `reasoning_effort` values are provider-agnostic and should work across backends. + +## 2026-06-14 — v0.16.0 / v2026.6.5+ sync (851 upstream commits, big release) + +### TL;DR for team members on older agents + +If your agent hasn't been updated since before 2026-06-14, here's what changed and how to get the new capabilities: + +1. **Run `hermes update` in `~/.hermes/hermes-agent`** — this pulls the latest from `origin/main` (currently at `2665e44ef`). +2. **If TUI changed: `cd ~/.hermes/hermes-agent/ui-tui && npm run build`**. +3. **New `video_generate` is available** — `video_gen.provider: xai` (default) or `video_gen.provider: minimax` (PR #41241 open upstream). Just call `video_generate` in chat. +4. **New model `kimi-k2.7-code` is in the Coding Plan picker** — first option in the Kimi/Moonshot provider list. +5. **AutoResearch is functional again** — `run_research` + `research_job` with 136/136 tests passing. Docs in `~/wiki/projects/hermes-agent/notes/autoresearch-guide.md`. + +If you can't `hermes update` for some reason (locked deploy, network down, etc.), see the manual fallback in `~/wiki/projects/hermes-agent/notes/workflow.md` section "Option B — Manual fallback". + +### What merged in (chronological) + +#### Kimi WebBridge toolset (commit `72098a906`, cherry-picked from `feat/kimi-webbridge`) + +Real-browser automation via the Kimi WebBridge daemon on `127.0.0.1:10086`. Unlike Playwright-based browser tools, this controls the user's REAL browser with their actual login sessions. Tools: `kimi_webbridge_navigate`, `kimi_webbridge_find_tab`, `kimi_webbridge_snapshot`, `kimi_webbridge_click`, `kimi_webbridge_fill`, `kimi_webbridge_evaluate`, `kimi_webbridge_screenshot`, `kimi_webbridge_save_screenshot`, `kimi_webbridge_save_pdf`, `kimi_webbridge_list_tabs`, `kimi_webbridge_close_tab`, `kimi_webbridge_close_session`. Off by default (`_DEFAULT_OFF_TOOLSETS`); enable via `hermes tools` once the WebBridge extension is installed. 26/26 tests passing. + +#### AutoResearch core (commit `0f6120146`, cherry-picked from `feat/autoresearch-core-v014`) + +The distilled AutoResearch core (1 commit by nicoechaniz 2026-05-18, distilled from the 162-commit heavy `feat/autoresearch` branch). Provides `run_research` (interactive) and `research_job` (detached long-running) with full parameter set: `topic`, `deliverable`, `metric_key`, `metric_direction`, `task_type`, `max_iterations`, `evaluation_mode` (`self_report`/`llm_judge`), `evaluation_prompt`, `acceptance_criterion`, `initial_attempt`, `time_budget_sec`, `kanban_task_id`, `strategies`, `auto_specify`. 136/136 tests in `tests/agent/research/` + `tests/agent/test_research_supervisor.py` + `tests/agent/test_factory.py`. Full parameter spec in `~/wiki/projects/hermes-agent/notes/autoresearch-guide.md`. + +#### Kimi k2.7-code picker (commit `2665e44ef`) + +`kimi-k2.7-code` (Moonshot's new coding model, released 2026-06-12) is now the first option in the Kimi Coding Plan picker. Three-file change: `hermes_cli/models.py:282` (curated list), `hermes_cli/model_setup_flows.py:1800` (the picker the user sees), `hermes_cli/main.py:4038` (deprecated copy). Trigger: run `hermes model`, choose Kimi / Moonshot → Coding Plan. + +#### TUI TERMINAL_TIMEOUT display fix (commit `607f0c0e9`, cherry-picked from `feat/altermundi`) + +`hermes info` used to print `TERMINAL_TIMEOUT: 60` but the actual default in `tools/terminal_tool.py:1152` is `180`. This was confusing — now it reads the real default. One-line change, 28/28 tests passing. + +### How to verify you're on the new version + +```bash +# Check the version Hermes reports +hermes --version +# Should show: Hermes Agent v0.16.0 (2026.6.5) · upstream 2665e44e +# Or later commits (k2.7 picker = 2665e44ef, TUI fix = 607f0c0e9) + +# Check video_generate is available +hermes tools | grep -i video +# Should show video_generate tool + +# Check kimi-k2.7-code is in the picker +hermes model # interactive, see the model list + +# Check autoresearch is functional +python -c "from tools.autoresearch import run_research" 2>&1 | head +# (Import path may vary; this is just a smoke test) +``` + +### Files changed (high level) + +- 1237 files changed in the upstream sync (mostly noise: desktop, dashboard, i18n, docs) +- 18 files changed in our fork: 4 conflict resolutions + 3 cherry-picks + 1 fix + 1 picker update +- DaemonCraft tools (`mc_navigate_tool`, `mc_bit_tool`, `embodied_plan_tool`) all preserved and verified in deploy +- Kimi OAuth from `~/.kimi/credentials/kimi-code.json` still works (auto-detected) + +### Conflicts resolved + +- `agent/conversation_loop.py` — kept ours (17 retry tracking vars) +- `cli.py` — took theirs (refactored `_estimate_tui_input_height`) +- `gateway/run.py` — kept ours (DaemonCraft lab-mode fail-safe) +- `hermes_cli/main.py` — kept ours (`_model_flow_kimi`, 113 lines) + +All preserved: session_id propagation, X-Msh-* headers, DaemonCraft lab-mode, kanban review. + +### Source of truth + +- `~/Projects/hermes-agent/MEMORY.md` — current operational state +- `~/wiki/projects/hermes-agent/notes/branch-stewardship-2026-06-14.md` — full branch state +- HMK chapter 61 — canonical branch list for future sessions +- `~/wiki/projects/hermes-agent/notes/autoresearch-guide.md` — AutoResearch parameter spec + +## 2026-05-16 — mc_bit Tool Fix + +### Synchronous mc_bit Handler + +The `mc_bit` Hermes tool was broken since deploy: `async def _handler(...)` returned +a coroutine object, which surfaced as `object of type 'coroutine' has no len()` in +live tool calls. Replaced with a synchronous `httpx.get` wrapper. + +**Branch:** `feat/daemoncraft` +**Commit:** `a16bc0c5b fix(daemoncraft): make mc_bit tool synchronous` + +Tests: `scripts/run_tests.sh tests/tools/test_mc_bit_tool.py -q --tb=short` → 3 passed. + +### mBit Context in Embodied Service (DaemonCraft side) + +See DaemonCraft CHANGELOG for the full mBit context integration. The hermes-agent +side only needed the mc_bit tool fix above — the world_state injection lives in +the embodied service composer on the DaemonCraft repo. + +## 2026-05-09 — Multi-Agent Coding Roster + Kanban Hardening + +### New Profiles +- **riqui** (deepseek-v4-flash, max_turns=30, reasoning=minimal): Surgical coding Kanban worker. Fixed protocol violation (was max_turns=15 + reasoning=none → iteration exhaustion before kanban_complete). +- **miki** (kimi-k2.6, kimi-coding OAuth via ~/.kimi/, max_turns=30, reasoning=high): Coding agent. Tested working. +- **maxi** (MiniMax-M2.7, minimax provider, Anthropic endpoint, max_turns=30, reasoning=high): Coding agent. Config created but blocked by CLI api_mode detection bug (404 — hardcoded chat_completions vs anthropic_messages). +- **claudio** (planned): Proxy profile → Claude Code CLI +- **gepeto** (planned): Proxy profile → Codex CLI + +### Kanban System +- **Protocol violation root cause:** max_turns too low + reasoning=none on weak models → iteration exhaustion → model writes kanban_complete as text (not function call) → clean exit without transition → effective_limit=1 → auto-blocked +- **Fix:** max_turns ≥ 25 + reasoning ≥ minimal for all Kanban coding workers +- **Self-spawn guard:** Dispatcher DOES spawn tasks assigned to gateway's own profile (compaii). Tasks must stay in `todo`/`triage` until manually claimed. +- **Smoke test pattern:** t_4631001e (17s, riqui) validated the fix + +### RTK Plugin +- **FIXED** by Riqui (t_ad89b059): Replaced corrupted `rtk_hermes/__init__.py` (circular self-import) with 332-line source from GitHub +- Binary symlinked for gateway PATH +- Plugin loads cleanly on gateway restart (no WARNING) + +### Memory Infrastructure +- HMK chapters 9-11 seeded: dispatcher guard, profile roster, maxi api_mode debug +- Project MEMORY.md updated with full profile roster and dispatcher critical rule + +### Known Issues +- **maxi:** `hermes -p maxi chat` returns 404. CLI hardcodes api_mode=chat_completions. Provider transport=anthropic_messages is ignored. curl confirms endpoint works. +- **Upstream:** ~90 commits behind (v2026.5.7+), needs sync + +## 2026-05-08 — Upstream Sync v2026.5.7 + +- Full rebase onto upstream/main (993 commits, 7 conflicts resolved) +- All 10 custom features preserved +- Gateway split: hermes-gateway.service (CompAII) + hermes-gateway@steve.service +- RTK plugin installed (but init.py was corrupted — fixed May 9) +- Kanban migration from Lattice (64+ tasks) +- CompAII hardening: max_turns=40, reasoning=high, compression=0.50 +- HMK memory kit: library.db seeded, engram_pack prefetch + +## Custom Features (all branches merged into main) + +1. feat/kimi-oauth-clean — Kimi OAuth refresh, header fixes +2. feat/altermundi-tui — TUI scrollbar, max lines config +3. feat/altermundi-cli — Ctrl+C priority config +4. feat/minimax-defaults — MiniMax provider defaults +5. feat/compression-config-reboot — Configurable compression protect_first_n +6. feat/dc-112-daemoncraft-gateway — Gateway adapter wiring, tool_choice propagation +7. DC-99 — Profile system prompt override per platform +8. DC-123 — TTS fixes + wake-up logging, CycleDetector +9. DC-132 — Contextvars-based endpoint resolution, turn metrics +10. DC-134 — Configurable turn wall-clock timeout + per-profile max_iterations diff --git a/HERMES_RESEARCH.md b/HERMES_RESEARCH.md new file mode 100644 index 000000000000..f922c4ef3532 --- /dev/null +++ b/HERMES_RESEARCH.md @@ -0,0 +1,202 @@ +# Hermes AutoResearch + +## What This Is + +Hermes AutoResearch is the **Karpathy inner loop** for autonomous experimentation inside Hermes. Given a research topic, it runs a baseline experiment, proposes improvements via LLM, executes them through `delegate_task`, keeps improvements and discards regressions, and records structured learnings. + +The architecture is **desacoplada**: long-running research loops execute as independent OS processes with durable checkpoints, so the parent agent does not burn iteration budget or die to timeouts. + +## Quick Start + +### Running a Research Job (Detached) + +```bash +# Create a job spec JSON +python -c ' +import json +spec = { + "job_id": "my-research", + "job_dir": "/home/user/.hermes/research-jobs/my-research", + "model": "kimi-for-coding", + "provider": "kimi-coding", + "topic": "Analyze WebAssembly adoption in 2025", + "deliverable": "Ranked list of relevant papers with abstracts", + "metric_key": "completeness_score", + "metric_direction": "maximize", + "task_type": "research", + "max_iterations": 3, +} +json.dump(spec, open("/home/user/.hermes/research-jobs/my-research/job.json", "w")) +' + +# Launch detached runner +source venv/bin/activate +HERMES_YOLO_MODE=1 python -m agent.research.job_runner \ + /home/user/.hermes/research-jobs/my-research/job.json +``` + +### From Python (Synchronous) + +```python +from agent.research.supervisor import ResearchSupervisor, TaskSpec +from pathlib import Path + +spec = TaskSpec( + topic="Analyze WebAssembly adoption in 2025", + deliverable="Ranked list of relevant papers with abstracts", + metric_key="completeness_score", + metric_direction="maximize", + task_type="research", +) + +supervisor = ResearchSupervisor(parent_agent=agent, workspace=Path("research-workspace")) +history = supervisor.run( + spec, + initial_attempt="", + run_id="run-001", + max_iterations=3, + llm=agent.llm_client, +) +``` + +## Architecture + +``` +Parent Agent / CLI + │ + ▼ +┌─────────────────────────┐ +│ research/job_runner.py│ ← Detached OS process +│ (entrypoint) │ +└─────────────────────────┘ + │ + ▼ +┌─────────────────────────┐ +│ ResearchSupervisor │ ← Karpathy loop orchestrator +│ • TaskSpec │ +│ • run() │ +│ • _observe() │ +│ • _checkpoint() │ +└─────────────────────────┘ + │ + ▼ +┌─────────────────────────┐ ┌─────────────────────────┐ +│ delegate_task │────▶│ Worker Subagent │ +│ (per iteration) │ │ • Reads task_brief.md │ +└─────────────────────────┘ │ • Writes attempt.md │ + │ • Writes results.json │ + │ • Reports metric │ + └─────────────────────────┘ +``` + +## Project Structure + +``` +agent/ +├── research/job_runner.py # Detached entrypoint: builds AIAgent, calls run_research +├── research/supervisor.py # ResearchSupervisor + TaskSpec + task briefs +├── research/runner.py # ExperimentRunner + ExperimentHistory +├── research/metrics.py # UniversalMetricParser +└── subdirectory_hints.py # Progressive context discovery (cached) + +tools/ +├── research_tool.py # run_research() public API +└── research_job_tool.py # start_research_job, research_job_status, collect_research_job + +~/.hermes/research-jobs/ # Job specs + checkpoints + logs +~/.hermes/research-workspace/ # Round artifacts (attempt.md, results.json, learnings.jsonl) +``` + +## The Karpathy Loop + +``` +Step 1: BASELINE — Worker receives task brief + attempt file, produces deliverable +Step 2: METRIC — UniversalMetricParser reads results.json / stdout +Step 3: JUDGE — LLM judge scores deliverable (if evaluation_mode="llm_judge") +Step 4: OBSERVE — Structured learning appended to learnings.jsonl +Step 5: CHECKPOINT — history.json + checkpoint.json written to disk +Step 6: OPTIMIZE — LLM proposes revised attempt based on history +Step 7: KEEP/DISCARD — If metric improved: keep, else discard; iterate +``` + +## Task Types + +| Type | Default Toolsets | Deliverable | Attempt File | +|------|-----------------|-------------|--------------| +| `code` | terminal, file | Python code | attempt.py | +| `search` | web, terminal, file | Search results | attempt.md | +| `research` | web, terminal, file | Synthesis | attempt.md | +| `generic` | terminal, file | Any text | attempt.md | + +## Worker Contract + +The worker receives: +- `task_brief.md` — Full instructions including think block, rules, tools available +- `attempt.py` or `attempt.md` — Current attempt to refine +- Environment variable `HERMES_YOLO_MODE=1` to skip command approval + +The worker must produce: +- `results.json` with `{"": }` +- Final line: `METRIC: = STATUS: improved|regressed|neutral NOTES: ` + +## Checkpoints and Recovery + +After every round, the supervisor writes: + +``` +~/.hermes/research-jobs// +├── checkpoint.json # {round, total_rounds, best_metric, updated_at} +├── history.json # Full results array + best reference +├── runner.log # Runner + supervisor logs +└── state.json # {status, pid, started_at} +``` + +External monitors can read `checkpoint.json` without polling the process. + +## Decision Guide + +| Situation | Action | +|-----------|--------| +| Long-running research (>5 min) | Use `research/job_runner` detached | +| Quick experiment (<2 min) | Call `run_research()` directly | +| Need baseline only | Set `llm=None` in supervisor | +| Worker times out | `DelegateSandboxResult.timed_out=True`; loop continues | +| 3 consecutive non-improving | Runner stops early (or 1 if high baseline) | +| Want to inspect history | Read `history.json` from checkpoint dir | + +## Performance Optimizations + +| Optimization | File | Impact | +|-------------|------|--------| +| **Lock file** | `research/job_runner.py` | Prevents duplicate restarts (~16 min saved) | +| **Provider cache** | `auxiliary_client.py` | Caches `resolve_provider_client` (~14 calls → 1) | +| **Subdirectory hints cache** | `subdirectory_hints.py` | Caches hint loads per directory | +| **Aggressive early stop** | `research/supervisor.py` | Baseline ≥0.9 → stop after 1 non-improving iter | +| **LLM judge every iter** | `research/supervisor.py` | Objective scoring on all loops | + +## Anti-Patterns + +- **DO NOT** run `research/job_runner` in foreground without `timeout >= 300` +- **DO NOT** poll the process with `ps` / `tail` — read `checkpoint.json` instead +- **DO NOT** launch the same job twice — the lock file prevents this +- **DO NOT** delete `.runner.lock` manually — use `kill` on the process + +## Metric Reporting (Worker Contract) + +Workers must print metrics in one of these formats: + +``` +# Hermes format (preferred) +METRIC: accuracy=0.923 STATUS: improved NOTES: Adam lr=0.001 beat SGD baseline + +# Standard key: value format +accuracy: 0.923 +loss: 0.112 +``` + +The `UniversalMetricParser` also reads `results.json` (structured) or `results.csv` if present in the round directory. + +## Skills + +Hermes AutoResearch skills are in `skills/autoresearch/` and are loaded automatically. +Domain-specific skills (ML, chemistry, biology) are in `skills/autoresearch/domain/`. diff --git a/RESEARCH_AGENTS.md b/RESEARCH_AGENTS.md new file mode 100644 index 000000000000..8c551a75cbd2 --- /dev/null +++ b/RESEARCH_AGENTS.md @@ -0,0 +1,162 @@ +# Hermes AutoResearch — Worker Agent Contract + +## Overview + +You are a **Hermes AutoResearch worker**. You receive a goal string and a working directory from the supervisor. Your job is to run the experiment described in `task_brief.md` and report a metric in the required format. + +You are NOT responsible for the loop logic (keep/discard, iteration, LLM code improvement). That is handled by the supervisor via `ResearchSupervisor`. + +## Inputs + +| Input | Source | Description | +|-------|--------|-------------| +| Working directory | `delegate_task` argument | Directory containing `task_brief.md` and `attempt` file | +| Goal string | `delegate_task` argument | Includes metric key and output format | +| `task_brief.md` | Read from working directory | Full instructions, think block, rules, tools available | +| `attempt.py` / `attempt.md` | Read from working directory | Current attempt to refine (iteration > 0) or baseline seed | + +## Your Steps + +1. **Read `task_brief.md`** — understand the experiment goal, deliverable, and metric key +2. **Read the attempt file** — see what the previous iteration produced +3. **Set up experiment files** — write refined code/synthesis to the working directory +4. **Run the experiment** — execute the code, collect results, verify metric +5. **Write `results.json`** with `{"": }` (structured output, preferred) +6. **Print metric line** — required for fallback stdout parsing +7. **Report status** — include STATUS word in output + +## Required Output Format + +Your final output MUST include a metric line in one of these formats: + +``` +# Preferred (Hermes format) +METRIC: = STATUS: improved|regressed|neutral NOTES: + +# Acceptable (standard) +: +``` + +Example: +``` +METRIC: completeness_score=0.95 STATUS: improved NOTES: Covered WebAssembly browser support, non-browser runtimes, and language bindings +``` + +The metric key must match the key specified in the goal string (e.g., `completeness_score`, `accuracy`, `pass_rate`). + +## Tool Format + +When calling tools, use the **JSON format** provided by the system. Do NOT use XML tags like ``. + +## Tools Available + +The task brief declares available tools explicitly. Common sets: + +| Task Type | Tools | +|-----------|-------| +| code | terminal, file, code_execution | +| search | web_search, browser, file, terminal | +| research | web_search, browser, file, terminal | +| generic | terminal, file, code_execution | + +Use these actively — do NOT assume they are unavailable. + +## Stopping Conditions + +Stop and report when ANY of the following occurs: + +- Experiment completes successfully — report final metric +- Time budget exceeded (check `TIME_ESTIMATE` vs elapsed) — report partial results +- Unrecoverable error — report `STATUS: regressed` with error in NOTES +- Code validation fails after 3 auto-repair attempts — report failure + +Do NOT loop indefinitely. The supervisor handles retry logic. + +## Kanban State Transitions + +You do NOT transition kanban states directly. The supervisor's `KanbanSink` monitors your output and handles: +- `in_progress` → your worker is running +- Kanban comment posted = supervisor read your metric +- `done` = experiment accepted / loop terminated (supervisor action) +- `archived` = experiment discarded (supervisor action) + +If no `kanban_task_id` was passed, the supervisor falls back to `StubSink` and only `runner.log` records progress — no external state changes happen. + +If you need to signal an issue to the supervisor, print a line starting with `HERMES_STATUS:`: +``` +HERMES_STATUS: blocked — missing numpy, cannot proceed +HERMES_STATUS: timeout — partial results in results.json +``` + +## Configuration + +No configuration file needed. The supervisor (Hermes) provides: +- LLM provider via environment (already configured) +- Working directory via `delegate_task` call +- Metric key and format via goal string +- `HERMES_YOLO_MODE=1` to skip command approval + +## Anti-Patterns + +Do NOT: +- Use subprocess, os.system, eval, exec, or shell escapes in experiment code +- Make network calls (experiments must be self-contained) +- Invent or fabricate metric values — measure real outcomes +- Run without a time guard (always implement elapsed-time check near 80% of budget) +- Print non-metric lines as `key: value` (they will be parsed as metrics) +- Use XML `` format — use JSON tool format instead + +## A/B Testing Strategies (HRM-110) + +The ``run_research`` tool supports comparing multiple research strategies on the same task via the ``strategies`` parameter. + +### Supported strategies + +| Strategy | `fan_out` | `use_moa` | Description | +|----------|-----------|-----------|-------------| +| Sequential | 1 | — | Baseline Karpathy loop: one hypothesis per iteration. | +| Fan-out (no MOA) | N | False | N parallel workers per iteration; best branch kept. | +| Fan-out + MOA | N | True | N parallel workers + Mixture-of-Agents aggregation into super-attempt. | + +### Tool usage example + +```json +{ + "topic": "Optimize binary search tree implementation", + "deliverable": "Python BST class with insert/search/delete", + "metric_key": "pass_rate", + "strategies": [ + {"name": "sequential", "fan_out": 1, "max_iterations": 3}, + {"name": "fanout3", "fan_out": 3, "use_moa": false, "max_iterations": 3}, + {"name": "fanout3_moa", "fan_out": 3, "use_moa": true, "max_iterations": 3} + ], + "repeats": 1 +} +``` + +### Reported metrics + +The A/B test report includes per-strategy aggregates: + +- **Best metric** — mean ± std across repeats +- **Improvement rate** — `(best - baseline) / baseline` +- **Cost USD** — total LLM spend +- **Time (s)** — wall-clock elapsed time +- **Iterations** — rounds until early-stop or max +- **Tokens** — input/output token counts + +### Programmatic API + +```python +from agent.research.ab_testing import ResearchABTester, StrategyConfig +from agent.research.supervisor import TaskSpec + +tester = ResearchABTester(parent_agent=agent, workspace=Path("/tmp/ab")) +spec = TaskSpec(topic="...", deliverable="...", metric_key="accuracy") +strategies = [ + StrategyConfig(name="seq", fan_out=1), + StrategyConfig(name="fan3", fan_out=3, use_moa=False), +] +summaries = tester.compare(spec, strategies, repeats=2) +print(tester.format_report(summaries)) +``` diff --git a/RESEARCH_OPENCLAW_VOICE.md b/RESEARCH_OPENCLAW_VOICE.md new file mode 100644 index 000000000000..0939d8185f7f --- /dev/null +++ b/RESEARCH_OPENCLAW_VOICE.md @@ -0,0 +1,325 @@ +# Research Report: OpenClaw Real-Time Voice Mode and Portability to Other Agent Frameworks + +**Date:** 2026 (research snapshot) +**Focus:** Technical implementation of real-time voice in OpenClaw ecosystem, underlying technologies, architecture, code patterns, integration with agent reasoning, and portability (esp. to Hermes Agent / nousresearch/hermes-agent and embodied/Minecraft-style agents). +**Constraint:** Pure research — no code changes authored during investigation. All findings from public GitHub, docs, X/Twitter signals, and direct source inspection. + +## 1. What is OpenClaw? + +- **Core Project**: https://github.com/openclaw/openclaw (Node.js / pnpm monorepo, MIT). +- Local-first, self-hosted personal AI "Gateway" (control plane) + "Snaps"/agents. +- Strong multi-channel messaging (WhatsApp, Telegram, Discord, Slack, iMessage, Signal, WeChat, etc.), tools/skills (browser, shell, files, cron, canvas), persistent sessions/memory, multi-agent routing. +- Workspace: `~/.openclaw/workspace/` with `SOUL.md`, `IDENTITY.md`, `USER.md`, `AGENTS.md`, skills in `skills//SKILL.md`. +- Gateway typically listens on WS `ws://127.0.0.1:18789` (configurable), protocol v3 (JSON frames: `req`/`res`/`event`). +- Native voice: Voice Wake (global triggers) + Talk Mode on companion "nodes" (macOS menu bar, iOS/Android apps). Not full low-latency bidirectional realtime in the core gateway for arbitrary clients/phone. +- Creator/community: Peter Steinberger (@steipete / @openclaw on X), viral 2025-2026 growth, hackathons, comparisons to Hermes Agent (similar local agent/gateway philosophy; some users run both or migrate). + +**Key Docs**: +- Architecture: https://docs.openclaw.ai/concepts/architecture +- Gateway RPC protocol: https://docs.openclaw.ai/reference/rpc +- Voice Wake: https://docs.openclaw.ai/nodes/voicewake +- Talk Mode: https://docs.openclaw.ai/nodes/talk (includes realtime config section) + +## 2. OpenClaw's Native Voice Implementation (Talk Mode + Voice Wake) + +### Voice Wake +- Global list of triggers stored by Gateway: `~/.openclaw/settings/voicewake.json` (`{triggers: ["openclaw", "jarvis", ...]}`). +- RPC: `voicewake.get`/`set`, `voicewake.routing.get`/`set` (maps trigger → target `sessionKey` or `agentId` or "current"). +- Events: `voicewake.changed`, `voicewake.routing.changed` pushed to all WS clients/nodes. +- macOS/iOS nodes perform local wake-word detection (Porcupine? device ASR), forward trigger + audio/transcript to Gateway. +- Android: currently manual mic in Voice tab (wake disabled in some builds). +- Routing allows per-wake-word targeting of specific agents/sessions. + +### Talk Mode (Native Continuous Voice Conversation) +- **Native path** (macOS/iOS/Android nodes): + 1. Local device STT (on-device ASR, configurable `speechLocale`). + 2. Transcript sent to active Gateway session (via chat pipeline or talk RPCs). + 3. Agent reasons / uses tools. + 4. Response via `talk.speak` RPC to node for TTS playback (ElevenLabs primary, system TTS fallback, local MLX on macOS). +- Phases: Listening (mic level viz) → Thinking → Speaking. Interrupt on speech (stops playback, records timestamp for next prompt). +- Voice directives in assistant replies (first JSON line, stripped): `{ "voice": "", "once": true, "speed": ..., "stability": ... }` for per-turn or persistent voice params (ElevenLabs model, etc.). +- Config (`openclaw.json` under `talk:`): + - `provider`: "elevenlabs" | "system" | "mlx" + - `providers.elevenlabs`: voiceId, modelId (eleven_v3), apiKey, outputFormat, stability, similarity, etc. + - `silenceTimeoutMs`, `interruptOnSpeech: true` + - `speechLocale` +- **Browser realtime Talk** (emerging/native): + - `talk.client.create` (webrtc / provider-websocket) or `talk.session.create` (gateway-relay). + - `realtime:` section: `provider: "openai"`, `transport: "webrtc"`, `brain: "agent-consult"` (or "direct-tools", "none"), model `gpt-realtime-2` or similar, voice (cedar etc.), `instructions`. + - Browser clients forward tool calls via `talk.client.toolCall` → Gateway does `openclaw_agent_consult` (delegates back to full agent). + - Transcription-only mode (`brain: "none"`) for dictation/captions using `talk.session.appendAudio` etc. + `talk.event`. +- `talk.catalog` exposes supported modes/transports/brains/providers for clients. +- Limitations: Native realtime is platform/browser-tied; full phone/low-latency bidirectional + arbitrary clients is where community bridges shine. Issues #7200, #8088 track native realtime/WebRTC/SIP/OpenAI-Realtime deeper integration. + +**Personality Injection**: Talk/realtime prompts pull from workspace `IDENTITY.md`/`SOUL.md`/`USER.md` (tools/skills left to main agent). + +## 3. Community Real-Time Voice Bridges (The "Real" Low-Latency Voice Mode) + +Core OpenClaw voice is good for nodes but not the fluid, sub-second, interruptible, phone-style experience. Community fills the gap with modern stacks. + +### Primary Reference: langwatch/openclaw-phone-assistant (Recommended for True Realtime) +- **Repo**: https://github.com/langwatch/openclaw-phone-assistant (Python, uv, Pipecat-based). +- **Goal**: Talk naturally to OpenClaw "Snaps" via browser (WebRTC) or phone (Twilio). "Snaps" voice interface. +- **Architecture** (exact from README + source): + ``` + Browser mic/speaker <—WebRTC—> Pipecat pipeline <—WebSocket—> OpenClaw Gateway (ws://...18789) + Phone (Twilio) <—> Pipecat pipeline + ↓ + OpenAI Realtime API (gpt-4o-realtime-preview or gpt-realtime-2) OR Google Gemini Live + ``` + - **Voice loop**: Handled entirely by realtime speech-to-speech model (low latency, natural prosody, built-in VAD/interruption detection in Pipecat + provider). + - **Delegation**: The realtime LLM (voice "brain") has **exactly one substantive tool**: `ask_openclaw(question)`. + - Forwards to OpenClaw via `OpenClawGatewayClient.chat_send(sessionKey, message, ...)` (or `agent_send` fallback). + - Uses idempotencyKey as runId; listens for `chat` events (state: "final" / "error" / "aborted") carrying `message.content[].text`. + - OpenClaw does the heavy lifting (tools, memory, multi-step, channel actions, long reasoning). + - **During delegation waits** (OpenClaw can take 10s–minutes for real actions): Hold music injected into audio pipeline (`HoldMusicPlayer`). + - **Interruption handling** (key innovation): + - `UserStartedSpeakingFrame` → sets `user_interrupted` Event. + - Race: tool wait vs interrupt. On interrupt, return status to voice LLM ("still processing, respond to what user just said"), **keep the in-flight `send_task` alive in background**. + - User can later say "check again" or repeat question → resumes waiting on the same task (no duplicate work). + - Watchdog + error recovery for dropped tool calls / races ("already_has_active_response"). + - **End of call**: `end_call` tool (LLM must speak goodbye first, then call it). + - **Rules enforced in system prompt** (critical for voice UX): + - ALWAYS speak a short ack *before* calling `ask_openclaw` (prevents dead silence + hold music). + - Keep responses SHORT (1-2 sentences ideal for voice). + - No markdown; natural phone conversation. + - Load personality from OpenClaw workspace (IDENTITY/SOUL/USER.md) but delegate tools. + - **Transcript sync** (`TranscriptSync`): Voice turns appended to OpenClaw session's `.jsonl` (under `agents//sessions/.jsonl`) for continuity/memory in main agent. + - **Transports**: + - WebRTC: `SmallWebRTCTransport` (browser client at `/client`, port 7860). Pipecat runner. + - Twilio: `FastAPIWebsocketTransport` + `TwilioFrameSerializer` (24kHz pipeline ↔ 8kHz), caller ID filtering (`ALLOWED_CALLER_NUMBERS`), ringing/pickup/error tones (mulaw direct playback before pipeline), Cloudflared tunnel for webhook, `make twilio`. + - **Providers**: `providers/openai_rt.py` and `gemini_live.py` (create_llm + context_aggregator). + - **Other files**: `bot.py` (main, tool handlers, pipeline, interruption watcher, watchdog), `openclaw_client.py` (full protocol impl), `hold_music.py`, `config.py`, `audio_debug.py`, Makefile (daemon, tunnel, webrtc/twilio). + - **Auth/Connect**: Protocol v3 handshake (`connect` with client caps `["tool-events"]`, role "operator", optional token/password). Handles `connect.challenge` nonce. `chat.send` preferred over `agent` for event broadcasting/tool visibility. + - **Session**: `OPENCLAW_SESSION_KEY` e.g. `agent:main:main`. + - **Daemon**: systemd for 24/7 (bot + tunnel). + - **Why it works**: Decouples voice surface (Pipecat + Realtime LLM) from reasoning/tools (OpenClaw). Voice LLM is "dumb but fast conversational"; full agent is "smart but slow". + +This is the canonical example of "OpenClaw real-time voice mode" in 2026. + +### Other Notable Community Voice Projects +- **Purple-Horizons/openclaw-voice** (https://github.com/Purple-Horizons/openclaw-voice): + - Browser-based (React? + FastAPI/WS backend). + - Local STT: faster-whisper (on-device, sizes tiny→large-v3-turbo, CUDA/MPS/CPU). + - VAD: Silero. + - TTS: ElevenLabs streaming (sentence-by-sentence for perceived low latency; turbo_v2_5) or local (XTTS-v2, Chatterbox). + - Smart text cleaning (strip markdown, hashtags, URLs for TTS). + - Continuous/auto-listen mode after response. + - Direct OpenClaw Gateway integration (HTTP chat completions or WS; dedicated "voice" agent config recommended). + - WebSocket protocol for browser: `start_listening`, `audio` (base64 PCM), `stop_listening`; events `transcript`, `response_chunk`, `audio_chunk`, `vad_status`. + - Mobile-friendly via HTTPS (Tailscale Funnel or nginx). + - Roadmap: WebRTC. + - Classic turn-based pipeline (not speech-to-speech Realtime LLM). Easier for fully local (no Realtime API key cost/latency). + +- **sachaabot/openclaw-voice-agent** (https://github.com/sachaabot/openclaw-voice-agent): + - Hardware wake-word voice interface (Raspberry Pi CM5 / PamirAI Distiller). + - Porcupine (Picovoice) for "hey openclaw" or custom. + - Whisper STT → text to local OpenClaw Gateway → TTS (gTTS / ElevenLabs / offline Piper). + - LEDs for state, systemd service, `config.yaml`. + - Simple, reliable for always-on local device. Not low-latency realtime LLM. + +- **malpern/VoxClaw** (https://github.com/malpern/VoxClaw): + - Networked TTS "speaker" for headless OpenClaw. Mac listener (port 4140?) speaks text sent over network from remote gateway. Apple voices + OpenAI/ElevenLabs. Simple way to "give your server agent a voice." + +- **openserv-labs/openclaw-voice-avatar**: + - Realtime voice + video with lip-synced avatar. + +- **Nat Eliason / community gists** (e.g. https://gist.github.com/Nateliason/66fb5220574023d5f59a1c4e92914603): + - Full Pipecat + Deepgram STT + ElevenLabs TTS + WebRTC voice chat pipeline wired to Clawdbot/OpenClaw (via gateway or OpenAI-compatible endpoint). Includes `bot.py`, `server.py`, `index.html`. "ClawChat voice" tutorials (Chinese "15-minute private voice secretary" setups also use FastAPI+Pipecat+WebRTC). + +- **Discord voice bridges / skills**: Multiple (ai-agent-Zofia/discord-voice-bridge-openclaw-skill etc. via ClawHub). OpenClaw skills for joining Discord voice and bridging audio/transcripts to agent. + +- **LiveKit mentions**: OpenClaw skills (e.g. gora050/livekit-integration) for room/data management. LiveKit (WebRTC SFU + agents) pairs naturally with Pipecat (has LiveKit transport) for scalable multi-user or embodied voice rooms. + +- **Twilio / phone plugins** in main repo (voice-call plugin issues around OpenAI Realtime conversation mode, hold music, drops). + +## 4. Underlying Technologies + +- **Pipecat** (https://github.com/pipecat-ai/pipecat, Daily.co): The dominant framework for building realtime voice (and multimodal) AI agents in Python. Handles: + - Pipeline orchestration (frames: audio, LLMRun, UserStartedSpeaking, etc.). + - Transports: SmallWebRTC, FastAPIWebsocket + Twilio serializer, Daily, LiveKit, etc. + - Services: OpenAI Realtime, Gemini Live, Deepgram/Whisper STT, ElevenLabs/Cartesia TTS, SileroVAD, function calling/tools schema, context aggregation, interruption, metrics. + - Runners for browser (WebRTC signaling) and telephony. + - Why used: Battle-tested for low-latency, interruption, hold audio injection, custom tools. Many voice agent examples (Home Assistant, custom agents). + +- **OpenAI Realtime API** (`gpt-4o-realtime-preview`, `gpt-realtime-2`): End-to-end speech-to-speech (audio in → audio out + text transcripts). Low latency, natural turn-taking, voice selection (alloy, cedar, etc.). Tool calling supported (the delegation hook). Primary for "ChatGPT Advanced Voice Mode"-like feel. + +- **Google Gemini Live**: Alternative realtime speech-to-speech provider in the same Pipecat setups. + +- **WebRTC**: Core low-latency browser audio transport (mic/speaker bidirectional). Pipecat SmallWebRTCTransport or Daily.co rooms. Signaling via FastAPI/WS. + +- **Twilio**: Telephony (phone numbers, voice webhooks, media streams). Serializer handles 8kHz mulaw ↔ 24kHz. Cloudflared / ngrok for public HTTPS webhook. + +- **VAD (Voice Activity Detection)**: Silero (open, local, in Purple), built-in in Realtime providers / Pipecat, or device-level. Critical for continuous mode, silence timeout, interruption. + +- **STT (non-realtime path)**: faster-whisper (local, on-device privacy), Deepgram (cloud streaming), device ASR on nodes. + +- **TTS (non-realtime)**: ElevenLabs (streaming, high quality, voice cloning params), local XTTS/Piper/Chatterbox, system voices, MLX. + +- **Gateway Protocol (OpenClaw side)**: Custom JSON-over-WS v3. + - Handshake: `connect` (client id/version/platform/mode/caps/scopes/auth token or password; handles challenge nonce). + - `chat.send` (preferred for voice bridges): idempotencyKey/runId, returns immediately `{status:"started"}`, then `chat` events with `state: "final"` + full text (or error/aborted). Broadcasts tool events to session subscribers. + - `agent`: CLI-style, returns final payloads after accepted + done. + - Sessions: `sessionKey` (e.g. `agent:main:main`), resolved to internal IDs. Transcripts in per-agent JSONL. + - Other: sessions.list, tool events, etc. + - `openclaw_client.py` is the reference Python implementation (pending futures, event handlers for chat, final vs accepted responses, etc.). + +- **Hold Music / Audio Injection**: During long backend calls (tools), inject prerecorded or generated audio frames into the output pipeline without breaking the realtime loop. + +- **Other**: nacl for Discord voice crypto (in Hermes), Opus decode, etc. + +## 5. How Voice Mode Integrates with the Agent's Reasoning Loop + +**Decoupled "Fast Voice Surface + Smart Brain" Pattern** (the key architectural insight, highly portable): + +1. **Realtime Voice LLM** (Pipecat + OpenAI Realtime / Gemini Live) owns: + - Audio I/O, VAD, interruption, prosody, short conversational responses. + - System prompt: Personality (from main agent files) + strict rules ("you are the voice interface", "ONE tool: ask_XXX", "speak BEFORE tool call", "short answers", "end_call on goodbye"). + - Tool calling only for delegation. + +2. **Delegation Tool** (`ask_openclaw` / `openclaw_agent_consult` / equivalent): + - Sends the user's spoken request (as text) over stable interface (WS to Gateway `chat.send` or direct agent invocation) to the **full agent session**. + - Voice LLM waits (with hold music / status updates on interrupt). + - Receives final text response → relays naturally ("Here's what I found..."). + +3. **Main Agent** (OpenClaw "Snap" or Hermes AIAgent) owns: + - All tools, memory (long-term, sessions), skills, multi-step reasoning, channel actions (email, Discord messages, browser, files, cron, Minecraft controls), persona depth. + - Runs in its own loop (possibly with higher max_iterations, different model, sandboxing). + - May take seconds to minutes; voice surface stays responsive. + +4. **Interruption & Backgrounding**: + - User can barge in during thinking/hold → voice LLM acknowledges immediately; original request continues async in main agent. + - Resume by re-asking (client tracks in-flight by question or runId). + +5. **Continuity**: + - Optional transcript sync (voice turns → main session JSONL) so main agent "remembers" the voice conversation. + - Shared sessionKey / agentId. + - Personality files shared (but tools not duplicated in voice prompt). + +6. **Error / UX Hardening**: + - Speak first before any tool (no silence). + - Watchdogs for silent failures / dropped calls. + - Recovery on pipeline errors (race conditions in realtime providers). + - Caller filtering, tones, daemonization. + +**Result**: You get fluid voice like "Her" or Advanced Voice Mode, but powered by your full local agent with real capabilities and persistent identity across text/voice channels. The voice part is thin (~1 tool); the agent is thick. + +Native OpenClaw Talk does similar but with device STT + `talk.speak` TTS + Gateway chat in the middle (higher latency, less fluid interruptions than Realtime API). + +## 6. Integration Patterns (Discord, Local, Minecraft/Embodied) + +- **Discord**: + - OpenClaw: Skills/bridges for voice channels (audio → agent → TTS back?). + - Hermes: **Native strong support** in `gateway/platforms/discord.py`: + - `VoiceReceiver`: Joins guild voice channels, captures per-user Opus audio (DAVE/secretbox decrypt or passthrough), SPEAKING events (op 5), silence detection/polling loop, delivers to `_voice_input_callback` (STT in run.py or pipeline), duplicate suppression via `_recent_voice_transcripts`. + - Auto-disconnect on inactivity. + - Audio output caching (`cache_audio_from_url/bytes`). + - Gateway `voice_mode` ("off"/"voice_only"/"all"), `/voice` commands, `auto_tts`, per-chat state persistence (`gateway_voice_mode.json`). + - `_sync_voice_mode_state_to_adapter`. + - Pattern for advanced realtime on Discord voice: Run Pipecat bot that joins the same voice channel (or bridges), uses realtime LLM, delegates via Hermes Discord adapter session or direct AIAgent. Or extend existing VoiceReceiver to feed a Pipecat pipeline. + +- **Local / Desktop / Hardware**: + - Wake word (Porcupine / Silero) → capture → STT (Whisper or device) or direct to Pipecat realtime → delegate to local Hermes/OpenClaw gateway (localhost WS or even in-process Python call). + - Examples: sachaabot hardware agent, VoxClaw for output, Purple for browser local STT. + - Port: Easy — run voice worker alongside Hermes TUI/CLI; use Python AIAgent directly for zero-latency delegation (better than WS). + +- **Minecraft-style Embodied Agents** (highly relevant to this workspace): + - Hermes recent commits: daemoncraft, minecraft_tools.py, embodied_plan, spatial enrichment, mBit, Path fallback. + - Pattern: Voice commands ("go to the village, mine the diamonds, build a wall here") → realtime voice LLM delegates to Hermes agent running in Minecraft environment (tools for movement, block ops, perception, planning). + - Feedback: Agent actions → spatial state or screenshots → vision or text summary → TTS/voice response. + - Or full embodied loop: voice as high-level planner, low-level control via Minecraft env. + - OpenClaw has analogous "robot" skills (ROS mentions in community). LiveKit rooms could coordinate voice + embodied telemetry. + - Portability win: Same delegation tool pattern. Voice surface doesn't need to know about Minecraft; the brain agent does. + +- **Browser / Web / Dashboard**: + - WebRTC client (Pipecat SmallWebRTC or custom) → voice worker → Hermes (via tui_gateway JSON-RPC, web_server PTY/REST, or direct agent API). + - Hermes has dashboard embedding TUI + PTY bridge; voice could be additional pane or separate. + +- **Phone / Telephony**: Twilio + Pipecat (proven in langwatch repo). Filter callers, public tunnel. + +- **Multi-Framework Coexistence**: + - https://github.com/AaronWong1999/hermesclaw: Proxy/bridge to run Hermes + OpenClaw on same WeChat (avoids iLink lock contention). Useful migration or parallel use. Separate migration importers for settings/memory/skills. + - Shared community (gbrain setups work for both; Garry Tan posts on quick WebRTC/Twilio voice for Hermes/OpenClaw/gbrain stacks). + +## 7. X/Twitter and Community Signals (Porting / Discussions) + +- Primary accounts: @openclaw, @steipete (founder, later OpenAI?). +- Voice hype: "Voice mode that feels like the movie Her", realtime with local agents, hackathons (ROSClaw for robots, voice secretaries). +- Pipecat + OpenClaw voice chat: @nateliason (Felix agent, open-sourced Claw voice gists + "ClawChat: How to Build a Cross-Platform Voice Chat"). +- Quick deploys: @garrytan and others: "Install OpenClaw or Hermes ... get it on WebRTC or your Twilio number in <30 minutes" (gbrain-powered). +- Comparisons/ports: Frequent Hermes vs OpenClaw threads (complementary strengths: OpenClaw strong on autonomous background + channels; Hermes on deep collab?). hermesclaw bridge for shared accounts. Migration tools and "run both" patterns. +- LiveKit / WebRTC / Pipecat: Mentioned alongside OpenClaw in agent infrastructure discussions, purple teaming (security of powerful local agents), observability (LangWatch pairs with Pipecat traces). +- No single "here is the port of langwatch-phone-assistant to Hermes" mega-thread in top results, but the pattern is repeatedly described as generalizable ("voice frontend + delegation tool to your agent backend"). Chinese tutorials and Medium/LinkedIn posts on 15-min OpenClaw voice secretaries using Pipecat emphasize the same architecture. +- Discord voice + agent bridging is a recurring skill/plugin request. + +## 8. Portability Assessment & Recommendations for Hermes / Other Frameworks + +**High Portability (Score: 9/10)** — The OpenClaw community pattern is deliberately **framework-agnostic on the brain side**. + +**Minimal Requirements for Target Agent (Hermes or any)**: +- Expose a stable "send message to session X, get final response text" interface (WS RPC like `chat.send`, HTTP chat completions, or direct Python `AIAgent.run_conversation` / `chat`). +- Session continuity + optional transcript logging (JSONL or equivalent). +- Optional: Access to persona files (SOUL/IDENTITY) for voice prompt injection. +- Optional: Tool event streaming / approvals if voice surface should surface them. + +**Why Easy for Hermes**: +- Python core: Voice worker (Pipecat process) can `import` and directly instantiate `AIAgent(...)` with the right session/credential context — zero network for delegation in local setups (huge latency win vs OpenClaw's WS). +- Existing Discord voice I/O (`VoiceReceiver`, auto-TTS, voice_mode, STT dedup) provides a ready hook. Can feed voice channel audio directly into Pipecat or use as fallback. +- Gateway already has voice concepts and multi-platform sessions. +- tui_gateway / web_server / acp_adapter give structured control surfaces. +- Minecraft/daemoncraft embodied work is a perfect "brain" for voice-delegated high-level commands. +- Recent gateway fixes (run_conversation, etc.) show active iteration. + +**Recommended Porting Approach** (for a Hermes voice mode): +1. Fork/adapt `langwatch/openclaw-phone-assistant` (or Nat's gist) → replace OpenClaw client with Hermes equivalent (direct AIAgent or existing gateway WS/JSON-RPC if exposed cleanly). +2. System prompt adaptation: "You are the voice interface for Hermes. Delegate via `ask_hermes` tool...". +3. For Discord voice channels: Extend or bridge the existing `VoiceReceiver` + STT to feed the Pipecat pipeline (or run a dedicated voice bot per guild). +4. Local mic: Add wake word (Porcupine integration or use Silero in Pipecat) + always-listening worker. +5. Minecraft embodied: Same delegation; the tool response can include spatial state or vision summaries. +6. Extras: Transcript sync to Hermes session DB (`hermes_state.py`), hold music, interruption resume, personality from `~/.hermes` equivalents. +7. Observability: LangWatch or Hermes' own observability plugin. +8. Start simple: Browser WebRTC + local Hermes agent (no Twilio first). + +**Alternatives / Layers**: +- For non-realtime (easier, fully local/privacy): Adapt Purple-Horizons style (Whisper + VAD + streaming TTS + direct call to AIAgent). +- Hardware: Porcupine + Whisper + TTS + direct agent call (like sachaabot). +- Use Pipecat's built-in examples for custom LLM agents (replace the "ask_hermes" tool impl). +- For LiveKit: Add LiveKit transport to Pipecat for room-based multi-agent voice or embodied coordination. + +**Risks / Considerations** (technical facts): +- Realtime API costs (per-minute audio) vs local STT/TTS. +- Interruption races and "already_has_active_response" errors (handled in the reference bot.py with recovery/watchdog). +- Session auth / token security for the delegation WS (OpenClaw uses gateway token from `~/.openclaw/openclaw.json`). +- Sandboxing: Voice surface should not grant extra privileges; delegate enforces policy. +- Latency: In-process delegation >> WS >> HTTP. +- Platform voice APIs (Discord) have their own limits (Opus, encryption, speaking indicators) — Pipecat can run alongside or replace. + +**Live Examples to Study (in order of fidelity)**: +1. https://github.com/langwatch/openclaw-phone-assistant (full Pipecat + Realtime + delegation + Twilio/WebRTC + interruption + sync). +2. Nat Eliason gist (lighter Claw-specific voice chat). +3. Purple-Horizons/openclaw-voice (STT/TTS local pipeline). +4. Hermes `gateway/platforms/discord.py` (VoiceReceiver, voice modes) + `gateway/run.py` (voice_mode persistence, auto_tts). +5. OpenClaw native talk config + `talk.client.toolCall` path in docs. + +## 9. Conclusions & Actionable Insights + +OpenClaw's "real-time voice mode" is **not a single native feature** but a thriving ecosystem pattern: Pipecat-orchestrated realtime speech-to-speech (OpenAI/Gemini) as the conversational skin, delegating via a single narrow tool to the full persistent agent "brain" over the Gateway protocol. This gives fluid voice UX without compromising the agent's power or requiring the voice layer to duplicate tools/memory. + +The architecture is **extremely portable** — it has already been adapted across local, browser, phone, Discord, and (by extension) embodied/robot use cases. Hermes is particularly well-positioned because of its Python depth, existing Discord voice pipeline, embodied Minecraft work, and similar gateway/session model. A high-quality Hermes voice mode could be built by forking the langwatch reference and wiring it to `AIAgent` (local) or the gateway (multi-platform), potentially surpassing OpenClaw's current native offering in fluidity for Discord and local cases. + +**Next Research Steps (if desired)**: Deep-dive specific Hermes voice input path (STT details in run.py + discord receiver callbacks), inspect tui_gateway for structured voice hooks, review Pipecat LiveKit transport for Minecraft coordination, or prototype the delegation client mirroring `openclaw_client.py` against Hermes' RPC surfaces. + +**Primary URLs**: +- OpenClaw main + docs: github.com/openclaw/openclaw , docs.openclaw.ai +- Best realtime bridge: github.com/langwatch/openclaw-phone-assistant (read openclaw_client.py + bot.py) +- STT/TTS bridge: github.com/Purple-Horizons/openclaw-voice +- Hermes Discord voice: workspace/gateway/platforms/discord.py (VoiceReceiver class) +- Pipecat: github.com/pipecat-ai/pipecat +- Community voice gist: gist.github.com/Nateliason/66fb5220574023d5f59a1c4e92914603 +- Cross-framework: github.com/AaronWong1999/hermesclaw + +This report is based on direct source reads (READMEs, bot.py, openclaw_client.py, Hermes voice code), docs pages, GitHub issues, and web/X signals. All technical claims trace to verifiable public artifacts as of the research date. + +--- + +*End of Report. Suitable for internal use in hermes-agent for planning voice enhancements or embodied voice control.* \ No newline at end of file diff --git a/RESEARCH_OPERATIONS.md b/RESEARCH_OPERATIONS.md new file mode 100644 index 000000000000..2adbb476dd36 --- /dev/null +++ b/RESEARCH_OPERATIONS.md @@ -0,0 +1,214 @@ +# Hermes AutoResearch — Operations Guide + +> This document captures the operational procedures, anti-patterns, and performance characteristics discovered during the development and validation of the Hermes AutoResearch orchestration layer. + +## Launching a Research Job + +### Method 1: Detached Runner (Recommended for >2 min tasks) + +Create a job spec JSON and launch via `research/job_runner`: + +```json +{ + "job_id": "unique-job-id", + "job_dir": "/home/user/.hermes/research-jobs/unique-job-id", + "model": "kimi-for-coding", + "provider": "kimi-coding", + "topic": "Your research topic here", + "deliverable": "What the worker must produce", + "metric_key": "completeness_score", + "metric_direction": "maximize", + "task_type": "research", + "max_iterations": 3, + "env": {"HERMES_YOLO_MODE": "1"} +} +``` + +Launch: +```bash +cd /path/to/hermes-agent +source venv/bin/activate +HERMES_YOLO_MODE=1 python -m agent.research.job_runner /path/to/job.json +``` + +### Method 2: Background Process (Non-blocking) + +```bash +HERMES_YOLO_MODE=1 python -m agent.research.job_runner /path/to/job.json & +``` + +The runner creates a `.runner.lock` file atomically. If the job is already running, it exits with code 2. + +### Method 3: Direct Python API (Blocking) + +```python +from agent.research.supervisor import ResearchSupervisor, TaskSpec +from pathlib import Path + +spec = TaskSpec( + topic="...", + deliverable="...", + metric_key="completeness_score", + metric_direction="maximize", + task_type="research", +) + +supervisor = ResearchSupervisor(parent_agent=agent) +history = supervisor.run(spec, initial_attempt="", run_id="run-001", max_iterations=3, llm=llm_client) +``` + +## Monitoring Progress + +### Passive Monitoring (Recommended) + +Read checkpoint files without polling the process: + +```bash +# Quick status +cat ~/.hermes/research-jobs//checkpoint.json + +# Full history +cat ~/.hermes/research-jobs//history.json + +# Live log +tail -f ~/.hermes/research-jobs//runner.log +``` + +### File Structure + +``` +~/.hermes/research-jobs// +├── job.json # Original spec +├── .runner.lock # PID lock (prevents duplicate runs) +├── state.json # {status, pid, started_at} +├── checkpoint.json # {round, total_rounds, best_metric} +├── history.json # Full results array + best reference +├── result.json # Final result (appears on completion) +└── runner.log # Runner + supervisor logs +``` + +## Anti-Patterns and Fixes + +| Anti-Pattern | Why It Fails | Fix | +|-------------|-------------|-----| +| Foreground run with default timeout (60s) | MCP init takes 30-60s; runner killed before loop starts | Use `timeout=300` minimum, or background launch | +| Active process polling (`ps`, `find`, `tail` in loop) | Wastes iterations, creates noise | Read `checkpoint.json` or `history.json` passively | +| Deleting logs and retrying identically | Same failure repeats, no learning | Change timeout or use background mode | +| Launching same job twice | Double resource usage, conflicting checkpoints | Lock file prevents this; check `.runner.lock` | +| No `terminal` in default toolsets | Worker cannot execute code even if brief says it can | `_DEFAULT_TOOLSETS["research"] = ["web", "terminal", "file"]` | +| XML `` from worker | kimi-coding generates XML instead of JSON tools | Add anti-XML guard to task brief | +| Worker without `HERMES_YOLO_MODE` | Worker stalls waiting for command approval | Set `HERMES_YOLO_MODE=1` in env or job spec | + +## Performance Baselines + +Measured on kimi-for-coding via kimi-coding provider: + +| Metric | Value | Notes | +|--------|-------|-------| +| MCP init time | ~30-60s | ia-bridge MCP server (kanban is CLI-native, not MCP) | +| Init-to-first-checkpoint (simple) | ~30s | smoke-test with minimal topic | +| Init-to-first-checkpoint (complex) | ~300s | Benchmark with multi-step worker | +| Iteration time (research task) | ~290-350s | Includes worker execution + judge | +| Provider resolution (cached) | ~0s | Cache hit after first call | +| Provider resolution (uncached) | ~1-2s | Auth resolution + client build | +| Subdirectory hints (cached) | ~0s | Per-directory cache | +| Subdirectory hints (uncached) | ~50-100ms | Disk read + scan | + +## Early Stop Behavior + +| Baseline | Early Stop Limit | Min Delta | Rationale | +|----------|-----------------|-----------|-----------| +| < 0.9 (maximize) or > 0.1 (minimize) | 3 iterations | 0.0 | Standard exploration | +| ≥ 0.9 (maximize) or ≤ 0.1 (minimize) | 1 iteration | 0.05 | Aggressive stop for high baselines | + +## Recovery Scenarios + +### Scenario: Job appears stuck + +1. Check `checkpoint.json` — has `round` advanced? +2. Check `runner.log` — are there recent `Omitting temperature` lines? +3. If log is stale >5 min, process may be waiting on API +4. Do NOT delete `.runner.lock` — kill the process instead: `kill $(cat .runner.lock)` + +### Scenario: Job crashed + +1. Read `runner.log` for traceback +2. Fix the issue (e.g., missing field in job.json) +3. Delete `.runner.lock` if stale (see below) +4. Relaunch + +### Scenario: Stale `.runner.lock` after crash + +The lock file holds the runner PID. To verify it's actually stale before deleting: + +```bash +PID=$(cat ~/.hermes/research-jobs//.runner.lock) +ps -p "$PID" > /dev/null && echo "STILL RUNNING (PID $PID)" || echo "stale, safe to remove" +rm -f ~/.hermes/research-jobs//.runner.lock # only if stale +``` + +Never blind-delete the lock while the runner is alive — you'll get duplicate +processes writing to the same checkpoint and corrupted state. + +### Scenario: Want to resume from checkpoint + +Current implementation does not support automatic resume from `checkpoint.json`. To resume: +1. Read `history.json` to find the best artifact +2. Create a new job spec with `initial_attempt` set to the best artifact content +3. Launch as new job + +## LLM Judge + +The judge runs on **every iteration** when `evaluation_mode="llm_judge"`. + +- Skipping iterations risks accepting worker-inflated self-reported scores +- Judge latency: ~5-15s per evaluation (one API call) +- Judge prompt is in `_score_with_llm_judge()` — customizable via `evaluation_prompt` in TaskSpec + +## Task Types and Toolsets + +| Type | Default Toolsets | Use When | +|------|-----------------|----------| +| `code` | terminal, file | Writing/running Python code | +| `search` | web, terminal, file | Web research, data collection | +| `research` | web, terminal, file | Synthesis, analysis, reporting | +| `generic` | terminal, file | Any custom task | + +Override with `worker_toolsets` parameter in `supervisor.run()`. + +## Environment Variables + +| Variable | Effect | +|----------|--------| +| `HERMES_YOLO_MODE=1` | Skip command approval (required for workers) | +| `DELEGATION_MAX_CONCURRENT_CHILDREN=3` | Parallel workers (default 3) | + +## Kanban Integration (Optional) + +If you pass `kanban_task_id` when starting a research job, the supervisor wires a `KanbanSink` that posts round-by-round progress comments to that kanban task and transitions it to `done` on completion. Requirements: + +1. The kanban DB (default `~/.hermes/kanban.sqlite`) must be reachable. +2. The target task ID must already exist — the job does not auto-create kanban tasks. +3. Caller is responsible for creating the kanban task (e.g., via the kanban CLI or programmatically) before invoking `run_research` / `research_job`. + +If the kanban DB is missing or the task ID is invalid, the research job still runs normally — the supervisor falls back to a log-only `StubSink`. Check `runner.log` for `KanbanSink fallback` or `Falling back to log-only sink` warnings if you expected comments but don't see them. + +### Untracked (default) runs + +Omit `kanban_task_id` to use `StubSink` (log-only). Progress is still recorded in `runner.log` and `history.json`; only the per-iteration kanban comments are skipped. + +## Git Workflow for AutoResearch Changes + +All changes to the autoresearch stack are committed to branch `feat/hermes-autoresearch-upstream`: + +```bash +git log --oneline feat/hermes-autoresearch-upstream +``` + +Key commits: +- `cc929c0a` — Add research_job orchestration for long-running loops +- `4d64d568` — Include terminal in research/search default toolsets +- `0da707a3` — Add Tools Available + anti-XML guard to task briefs +- `f08dc63c` — Correct sandbox messaging and add partial recovery +- `8e68e23d` — Performance optimizations (lock, cache, early stop) +- `61c6e994` — Restore LLM judge on every iteration diff --git a/agent/agent_runtime_helpers.py b/agent/agent_runtime_helpers.py index cae1a685a537..71b2ef95ff28 100644 --- a/agent/agent_runtime_helpers.py +++ b/agent/agent_runtime_helpers.py @@ -1893,6 +1893,7 @@ def _execute(next_args: dict) -> Any: enabled_toolsets=getattr(agent, "enabled_toolsets", None), disabled_toolsets=getattr(agent, "disabled_toolsets", None), tool_request_middleware_trace=list(_tool_middleware_trace), + parent_agent=agent, ) from hermes_cli.middleware import run_tool_execution_middleware diff --git a/agent/anthropic_adapter.py b/agent/anthropic_adapter.py index 3a2d3f68e17f..7d21a6dc22cd 100644 --- a/agent/anthropic_adapter.py +++ b/agent/anthropic_adapter.py @@ -774,14 +774,13 @@ def build_anthropic_client( ) if _is_kimi_coding_endpoint(base_url): - # Kimi's /coding endpoint requires User-Agent: claude-code/0.1.0 - # to be recognized as a valid Coding Agent. Without it, returns 403. - # Check this BEFORE _requires_bearer_auth since both match api.kimi.com/coding. + # Kimi's /coding endpoint requires full X-Msh-* headers and KimiCLI UA. + # Without them, requests return 404/403. + from hermes_cli.auth import kimi_coding_default_headers kwargs["api_key"] = api_key - kwargs["default_headers"] = { - "User-Agent": "claude-code/0.1.0", - **( {"anthropic-beta": ",".join(common_betas)} if common_betas else {} ) - } + kwargs["default_headers"] = kimi_coding_default_headers() + if common_betas: + kwargs["default_headers"]["anthropic-beta"] = ",".join(common_betas) elif _requires_bearer_auth(normalized_base_url): # Some Anthropic-compatible providers (e.g. MiniMax) expect the API key in # Authorization: Bearer *** for regular API keys. Route those endpoints diff --git a/agent/auxiliary_client.py b/agent/auxiliary_client.py index 01ea45d7be24..9bf690fb84ea 100644 --- a/agent/auxiliary_client.py +++ b/agent/auxiliary_client.py @@ -1473,7 +1473,8 @@ def _resolve_api_key_provider() -> Tuple[Optional[OpenAI], Optional[str]]: return GeminiNativeClient(api_key=api_key, base_url=base_url), model extra = {} if base_url_host_matches(base_url, "api.kimi.com"): - extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + from hermes_cli.auth import kimi_coding_default_headers + extra["default_headers"] = kimi_coding_default_headers() elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers @@ -1513,7 +1514,8 @@ def _resolve_api_key_provider() -> Tuple[Optional[OpenAI], Optional[str]]: return GeminiNativeClient(api_key=api_key, base_url=base_url), model extra = {} if base_url_host_matches(base_url, "api.kimi.com"): - extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + from hermes_cli.auth import kimi_coding_default_headers + extra["default_headers"] = kimi_coding_default_headers() elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.models import copilot_default_headers @@ -2941,6 +2943,17 @@ def _refresh_provider_credentials(provider: str) -> bool: return False _evict_cached_clients(normalized) return True + if normalized in ("kimi-coding", "kimi-coding-cn"): + from hermes_cli.auth import resolve_kimi_coding_runtime_credentials + + creds = resolve_kimi_coding_runtime_credentials( + force_refresh=True, + allow_api_key_fallback=True, + ) + if not str(creds.get("api_key", "") or "").strip(): + return False + _evict_cached_clients(normalized) + return True except Exception as exc: logger.debug("Auxiliary provider credential refresh failed for %s: %s", normalized, exc) return False @@ -3299,7 +3312,8 @@ def _to_async_client(sync_client, model: str, is_vision: bool = False): is_agent_turn=True, is_vision=is_vision ) elif base_url_host_matches(sync_base_url, "api.kimi.com"): - async_kwargs["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + from hermes_cli.auth import kimi_coding_default_headers + async_kwargs["default_headers"] = kimi_coding_default_headers() elif base_url_host_matches(sync_base_url, "integrate.api.nvidia.com"): async_kwargs["default_headers"] = build_nvidia_nim_headers(sync_base_url) else: @@ -3588,7 +3602,8 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = "", if _dq: extra["default_query"] = _dq if base_url_host_matches(custom_base, "api.kimi.com"): - extra["default_headers"] = {"User-Agent": "claude-code/0.1.0"} + from hermes_cli.auth import kimi_coding_default_headers + extra["default_headers"] = kimi_coding_default_headers() elif base_url_host_matches(custom_base, "api.githubcopilot.com"): from hermes_cli.copilot_auth import copilot_request_headers extra["default_headers"] = copilot_request_headers( @@ -3841,7 +3856,8 @@ def _wrap_if_needed(client_obj, final_model_str: str, base_url_str: str = "", # Provider-specific headers headers = {} if base_url_host_matches(base_url, "api.kimi.com"): - headers["User-Agent"] = "claude-code/0.1.0" + from hermes_cli.auth import kimi_coding_default_headers + headers = kimi_coding_default_headers() elif base_url_host_matches(base_url, "api.githubcopilot.com"): from hermes_cli.copilot_auth import copilot_request_headers diff --git a/agent/chat_completion_helpers.py b/agent/chat_completion_helpers.py index 1ee1702b45e8..f5f4de7edd38 100644 --- a/agent/chat_completion_helpers.py +++ b/agent/chat_completion_helpers.py @@ -752,6 +752,7 @@ def build_api_kwargs(agent, api_messages: list) -> dict: max_tokens_param_fn=agent._max_tokens_param, reasoning_config=agent.reasoning_config, request_overrides=agent.request_overrides, + tool_choice=getattr(agent, "request_overrides", {}).get("tool_choice") if hasattr(agent, "request_overrides") else None, session_id=getattr(agent, "session_id", None), provider_profile=_profile, ollama_num_ctx=agent._ollama_num_ctx, @@ -1931,6 +1932,16 @@ def _call_chat_completions(): for tc_delta in delta.tool_calls: raw_idx = tc_delta.index if tc_delta.index is not None else 0 delta_id = tc_delta.id or "" + # A genuine new tool call always opens with a function + # name; argument-continuation deltas carry only + # ``function.arguments``. Gate the "new slot" logic below + # on a name so providers that resend a *changing* id on + # every continuation delta (kimi-coding) don't get each + # JSON fragment ('{"', 'query', '":', ...) split into its + # own bogus tool call. + _delta_has_name = bool( + tc_delta.function and tc_delta.function.name + ) # Ollama fix: detect a new tool call reusing the same # raw index (different id) and redirect to a fresh slot. @@ -1938,12 +1949,13 @@ def _call_chat_completions(): _active_slot_by_idx[raw_idx] = raw_idx if ( delta_id + and _delta_has_name and raw_idx in _last_id_at_idx and delta_id != _last_id_at_idx[raw_idx] ): new_slot = max(tool_calls_acc, default=-1) + 1 _active_slot_by_idx[raw_idx] = new_slot - if delta_id: + if delta_id and _delta_has_name: _last_id_at_idx[raw_idx] = delta_id idx = _active_slot_by_idx[raw_idx] diff --git a/agent/conversation_loop.py b/agent/conversation_loop.py index 379a038a9e09..a4ebd2c2fbfe 100644 --- a/agent/conversation_loop.py +++ b/agent/conversation_loop.py @@ -873,6 +873,21 @@ def run_conversation( max_retries = agent._api_max_retries _retry = TurnRetryState() max_compression_attempts = 3 + codex_auth_retry_attempted=False + anthropic_auth_retry_attempted=False + nous_auth_retry_attempted=False + nous_paid_entitlement_refresh_attempted=False + copilot_auth_retry_attempted=False + kimi_auth_retry_attempted=False + thinking_sig_retry_attempted = False + invalid_encrypted_content_retry_attempted = False + image_shrink_retry_attempted = False + multimodal_tool_content_retry_attempted = False + oauth_1m_beta_retry_attempted = False + llama_cpp_grammar_retry_attempted = False + has_retried_429 = False + restart_with_compressed_messages = False + restart_with_length_continuation = False finish_reason = "stop" response = None # Guard against UnboundLocalError if all retries fail @@ -1383,6 +1398,12 @@ def _perform_api_call(next_api_kwargs): force=True, ) finish_reason = "length" + if finish_reason != "length" and agent._has_truncated_tool_call_args(assistant_message): + agent._vprint( + f"{agent.log_prefix}⚠️ Tool-call arguments truncated mid-generation (invalid JSON) — treating as length-truncation so the retry/boost path recovers it", + force=True, + ) + finish_reason = "length" # ── Content-policy refusal (HTTP 200) ────────────────── # The model — or the provider's safety system — returned a @@ -1909,11 +1930,16 @@ def _perform_api_call(next_api_kwargs): # Stop spinner silently — retry status is buffered and # only flushed when every retry+fallback is exhausted. if thinking_spinner: - thinking_spinner.stop("") + thinking_spinner.stop("(╥_╥) error, retrying...") thinking_spinner = None if agent.thinking_callback: agent.thinking_callback("") + # Defensive: log full traceback for TypeError so we can diagnose + # 'NoneType object is not iterable' crashes in Codex path. + if isinstance(api_error, TypeError): + logger.error("TypeError in API call path", exc_info=True) + # ----------------------------------------------------------- # UnicodeEncodeError recovery. Two common causes: # 1. Lone surrogates (U+D800..U+DFFF) from clipboard paste @@ -2359,6 +2385,15 @@ def _perform_api_call(next_api_kwargs): if agent._try_refresh_copilot_client_credentials(): agent._buffer_vprint(f"🔐 Copilot credentials refreshed after 401. Retrying request...") continue + if ( + agent.provider in {"kimi-coding", "kimi-coding-cn"} + and status_code == 401 + and not kimi_auth_retry_attempted + ): + kimi_auth_retry_attempted = True + if agent._try_refresh_kimi_client_credentials(force=True): + agent._vprint(f"{agent.log_prefix}🔐 Kimi credentials refreshed after 401. Retrying request...") + continue if ( agent.api_mode == "anthropic_messages" and status_code == 401 diff --git a/agent/factory.py b/agent/factory.py new file mode 100644 index 000000000000..8474c5a3211c --- /dev/null +++ b/agent/factory.py @@ -0,0 +1,75 @@ +"""Centralized AIAgent construction for non-CLI entrypoints. + +Most Hermes entrypoints (CLI, gateway, ACP, TUI gateway, batch_runner) build +``AIAgent`` directly with their own kwargs. The detached research-job runner +historically did the same plus a manual post-init patching block to satisfy +the runtime invariants ``delegate_task`` expects. + +This module consolidates that patching into one well-named factory so: +1. The fragile patch list lives in *one* place — easier to keep in sync as + ``AIAgent`` evolves upstream. +2. Other detached entrypoints (cron, batch jobs, future schedulers) can + reuse the same factory rather than copy-pasting the patch block. + +The longer-term plan (HRM-57 follow-up, requires upstream coordination) +is to absorb the five fragile internal attributes — ``_delegate_depth``, +``terminal_cwd``, ``cwd``, ``_subdirectory_hints``, ``_delegate_spinner`` +— into ``AIAgent.__init__`` itself so this factory becomes a thin profile +mapper. Until then it is the single point of fragility. +""" +from __future__ import annotations + +import os +from typing import Any + + +def build_agent_for_research_job(spec: dict[str, Any]) -> Any: + """Build an AIAgent suitable for running a detached research job. + + Reads model/provider/toolset config from ``spec`` (typically loaded from + ``/job.json``). The parent agent inherits the active profile's + SOUL.md, AGENTS.md, and MEMORY.md unless ``spec`` explicitly opts out + via ``skip_context_files`` or ``skip_memory``. + + Returns: + Live ``AIAgent`` ready to be passed as ``parent_agent`` to + ``run_research`` / ``ResearchSupervisor``. + """ + from run_agent import AIAgent + + agent = AIAgent( + model=spec["model"], + provider=spec.get("provider"), + base_url=spec.get("base_url"), + api_key=spec.get("api_key"), + api_mode=spec.get("api_mode"), + enabled_toolsets=spec.get("toolsets", ["research", "terminal", "file"]), + quiet_mode=True, + platform="cli", + session_id=f"research-job:{spec['job_id']}", + skip_context_files=spec.get("skip_context_files", False), + skip_memory=spec.get("skip_memory", False), + ) + + # The "HRM-57 full" idea was to fold these five runtime invariants into + # AIAgent.__init__ upstream so the factory wouldn't need to patch them. + # That kwargs change never landed in the upstream AIAgent — the + # constructor still hardcodes _delegate_depth=0 internally and does not + # accept delegate_depth / terminal_cwd / cwd / subdirectory_hints kwargs + # at all. Until that reaches main, we patch the post-init invariants the + # delegate_task code path depends on. + if not hasattr(agent, "_delegate_depth"): + agent._delegate_depth = 0 + if not hasattr(agent, "terminal_cwd") or not getattr(agent, "terminal_cwd", None): + agent.terminal_cwd = os.getcwd() + if not hasattr(agent, "cwd") or not getattr(agent, "cwd", None): + agent.cwd = os.getcwd() + if not hasattr(agent, "_subdirectory_hints"): + agent._subdirectory_hints = None + + # tool_progress_callback IS in __init__, but defaults to None. Set a + # no-op so callers that read it can dispatch without a None-check. + if agent.tool_progress_callback is None: + agent.tool_progress_callback = lambda *a, **k: None + + return agent diff --git a/agent/model_metadata.py b/agent/model_metadata.py index 8cfec23fe1f7..6d2b207c3e4a 100644 --- a/agent/model_metadata.py +++ b/agent/model_metadata.py @@ -281,6 +281,11 @@ def _save_model_metadata_disk_cache(data: Dict[str, Dict[str, Any]]) -> None: "grok": 131072, # catch-all (grok-beta, unknown grok-*) # Kimi "kimi": 262144, + "kimi-k2.6": 262144, + "kimi-k2.5": 262144, + "kimi-k2": 262144, + "k2p6": 262144, + "k2p5": 262144, # Tencent — Hy3 Preview (Hunyuan) with 256K context window. # OpenRouter live metadata reports 262144 (256 × 1024); align the # static fallback so cache and offline both agree (issue #22268). diff --git a/agent/research/__init__.py b/agent/research/__init__.py new file mode 100644 index 000000000000..9b3650d63b98 --- /dev/null +++ b/agent/research/__init__.py @@ -0,0 +1,39 @@ +"""Hermes AutoResearch — Karpathy inner loop + Autogenesis AOOR for Hermes. + +Public API re-exports for convenience. Internal callers should import from +the submodules directly to keep dependency graph explicit. + +Pattern parallel: agent.research is a self-contained orchestration module +in the same shape as ``cron/`` and ``gateway/`` — a directory bundle of +related primitives, not a flat collection of agent.research_*.py files. +""" +from agent.research.supervisor import ResearchSupervisor, TaskSpec +from agent.research.runner import ( + DelegateSandboxResult, + ExperimentHistory, + ExperimentResult, + ExperimentRunner, + HermesExperimentConfig, +) +from agent.research.metrics import UniversalMetricParser +from agent.research.evolution import EvolutionStore, LessonEntry, LessonCategory + +from agent.research.ab_testing import ResearchABTester, StrategyConfig, StrategyRun, StrategySummary + +__all__ = [ + "ResearchSupervisor", + "TaskSpec", + "DelegateSandboxResult", + "ExperimentHistory", + "ExperimentResult", + "ExperimentRunner", + "HermesExperimentConfig", + "UniversalMetricParser", + "EvolutionStore", + "LessonEntry", + "LessonCategory", + "ResearchABTester", + "StrategyConfig", + "StrategyRun", + "StrategySummary", +] diff --git a/agent/research/ab_testing.py b/agent/research/ab_testing.py new file mode 100644 index 000000000000..c5d274b00b6a --- /dev/null +++ b/agent/research/ab_testing.py @@ -0,0 +1,362 @@ +"""A/B testing framework for research strategies (HRM-110). + +Compares different research strategies on the same TaskSpec: + - sequential (baseline, fan_out=1) + - fan-out N without MOA + - fan-out N with MOA aggregation + - human baseline (no iterations) + +Usage:: + + from agent.research.ab_testing import ResearchABTester, StrategyConfig + from agent.research.supervisor import TaskSpec + + tester = ResearchABTester(parent_agent=agent, workspace=Path("/tmp/ab")) + strategies = [ + StrategyConfig(name="sequential", fan_out=1, max_iterations=3), + StrategyConfig(name="fanout3", fan_out=3, use_moa=False, max_iterations=3), + StrategyConfig(name="fanout3_moa", fan_out=3, use_moa=True, max_iterations=3), + ] + results = tester.compare(spec, strategies, initial_attempt="", repeats=1) + print(tester.format_report(results)) +""" + +from __future__ import annotations + +import json +import logging +import time +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Optional + +from agent.research.supervisor import ResearchSupervisor, TaskSpec +from agent.research.runner import ExperimentHistory, ExperimentResult +from agent.research.metrics import UniversalMetricParser + +logger = logging.getLogger(__name__) + + +@dataclass +class StrategyConfig: + """Configuration for a single strategy in an A/B test.""" + + name: str + fan_out: int = 1 + use_moa: bool = True + max_iterations: int = 3 + time_budget_sec: int = 0 + keep_threshold: float = 0.0 + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +@dataclass +class StrategyRun: + """Result of one execution of a strategy.""" + + strategy_name: str + repeat: int + history: ExperimentHistory + elapsed_sec: float + workspace: Path + + @property + def best_metric(self) -> float | None: + return self.history.best_result.primary_metric if self.history.best_result else None + + @property + def baseline_metric(self) -> float | None: + return self.history.baseline_metric + + @property + def total_cost_usd(self) -> float: + return sum( + (r.cost_usd or 0.0) + for r in self.history.results + ) + + @property + def total_tokens_in(self) -> int: + return sum( + (r.tokens_in or 0) + for r in self.history.results + ) + + @property + def total_tokens_out(self) -> int: + return sum( + (r.tokens_out or 0) + for r in self.history.results + ) + + @property + def iterations_to_converge(self) -> int: + return len(self.history.results) + + @property + def improvement_rate(self) -> float | None: + if self.baseline_metric is None or self.best_metric is None: + return None + if self.baseline_metric == 0: + return float("inf") if self.best_metric != 0 else 0.0 + return (self.best_metric - self.baseline_metric) / abs(self.baseline_metric) + + +@dataclass +class StrategySummary: + """Aggregated statistics across repeats for one strategy.""" + + strategy_name: str + runs: list[StrategyRun] = field(default_factory=list) + + @property + def best_metrics(self) -> list[float]: + return [r.best_metric for r in self.runs if r.best_metric is not None] + + @property + def mean_best_metric(self) -> float | None: + vals = self.best_metrics + return sum(vals) / len(vals) if vals else None + + @property + def std_best_metric(self) -> float | None: + vals = self.best_metrics + if len(vals) < 2: + return 0.0 if vals else None + mean = sum(vals) / len(vals) + variance = sum((x - mean) ** 2 for x in vals) / (len(vals) - 1) + return variance ** 0.5 + + @property + def mean_cost_usd(self) -> float | None: + vals = [r.total_cost_usd for r in self.runs] + return sum(vals) / len(vals) if vals else None + + @property + def mean_elapsed_sec(self) -> float | None: + vals = [r.elapsed_sec for r in self.runs] + return sum(vals) / len(vals) if vals else None + + @property + def mean_iterations(self) -> float | None: + vals = [r.iterations_to_converge for r in self.runs] + return sum(vals) / len(vals) if vals else None + + @property + def mean_improvement_rate(self) -> float | None: + vals = [r.improvement_rate for r in self.runs if r.improvement_rate is not None] + return sum(vals) / len(vals) if vals else None + + @property + def mean_tokens_in(self) -> float | None: + vals = [r.total_tokens_in for r in self.runs] + return sum(vals) / len(vals) if vals else None + + @property + def mean_tokens_out(self) -> float | None: + vals = [r.total_tokens_out for r in self.runs] + return sum(vals) / len(vals) if vals else None + + +class ResearchABTester: + """Orchestrates A/B tests between research strategies.""" + + def __init__( + self, + parent_agent: Any, + workspace: Path, + *, + progress_sink: Optional[Any] = None, + llm: Any = None, + ) -> None: + self.parent_agent = parent_agent + self.workspace = workspace + self.llm = llm + self._ab_test_dir = workspace / "ab-tests" + self._ab_test_dir.mkdir(parents=True, exist_ok=True) + + # Resolve parent sink. The tester owns the close-on-completion; + # per-strategy sub-sinks must NOT call complete_task themselves + # (otherwise the first strategy closes the parent kanban task and + # subsequent strategies comment on a closed task). + if progress_sink is None: + from agent.research.sinks import StubSink + self._parent_sink = StubSink() + else: + self._parent_sink = progress_sink + + def _make_supervisor(self) -> ResearchSupervisor: + # Per-strategy child sink: same identity as the parent sink, but + # with run_completed-driven task completion suppressed. We only + # know how to do this for KanbanSink; other sinks are passed + # through unchanged (Stub is idempotent). + from agent.research.sinks import KanbanSink + if isinstance(self._parent_sink, KanbanSink): + child_sink = KanbanSink( + task_id=self._parent_sink._task_id, + db_path=self._parent_sink._db_path, + complete_on_run_completed=False, + ) + else: + child_sink = self._parent_sink + + return ResearchSupervisor( + parent_agent=self.parent_agent, + workspace=self.workspace, + progress_sink=child_sink, + ) + + def _run_single( + self, + spec: TaskSpec, + strategy: StrategyConfig, + initial_attempt: str, + run_id: str, + ) -> StrategyRun: + """Execute one strategy configuration once.""" + strategy_workspace = self._ab_test_dir / run_id / strategy.name + strategy_workspace.mkdir(parents=True, exist_ok=True) + + supervisor = self._make_supervisor() + start = time.monotonic() + history = supervisor.run( + spec, + initial_attempt=initial_attempt, + run_id=run_id, + max_iterations=strategy.max_iterations, + time_budget_sec=strategy.time_budget_sec, + keep_threshold=strategy.keep_threshold, + llm=self.llm, + checkpoint_dir=strategy_workspace / "checkpoints", + fan_out=strategy.fan_out, + use_moa=strategy.use_moa, + ) + elapsed = time.monotonic() - start + + return StrategyRun( + strategy_name=strategy.name, + repeat=0, + history=history, + elapsed_sec=elapsed, + workspace=strategy_workspace, + ) + + def compare( + self, + spec: TaskSpec, + strategies: list[StrategyConfig], + initial_attempt: str = "", + *, + repeats: int = 1, + run_prefix: str = "ab", + ) -> list[StrategySummary]: + """Run each strategy ``repeats`` times and return aggregated summaries. + + Args: + spec: The TaskSpec to test (same for all strategies). + strategies: List of StrategyConfig to compare. + initial_attempt: Starting seed. + repeats: How many times to run each strategy (for variance estimation). + run_prefix: Prefix for run IDs. + + Returns: + One StrategySummary per strategy, ordered by input list. + """ + summaries: list[StrategySummary] = [] + for strategy in strategies: + summary = StrategySummary(strategy_name=strategy.name) + for repeat in range(repeats): + run_id = f"{run_prefix}-{strategy.name}-r{repeat}" + logger.info( + "A/B test: running strategy=%s repeat=%d run_id=%s", + strategy.name, repeat, run_id, + ) + run = self._run_single(spec, strategy, initial_attempt, run_id) + run.repeat = repeat + summary.runs.append(run) + summaries.append(summary) + + # Close the parent kanban task ONCE, after all strategies+repeats + # are done. Sub-sinks ran with complete_on_run_completed=False so + # the task stayed open through every strategy's run_completed. + # We pass the LAST strategy's last run history as the "summary + # history"; the sink's run_completed only consumes results count + # and best_metric for the closing comment, both of which are + # representative enough for the close. + last_history = ( + summaries[-1].runs[-1].history + if summaries and summaries[-1].runs + else None + ) + if last_history is not None: + self._parent_sink.run_completed(last_history) + return summaries + + @staticmethod + def format_report(summaries: list[StrategySummary]) -> str: + """Return a human-readable comparison table.""" + lines: list[str] = [ + "# A/B Test Report: Research Strategies", + "", + "| Strategy | Best Metric ± std | Improvement | Cost USD | Time (s) | Iterations | Tokens In | Tokens Out |", + "|----------|-------------------|-------------|----------|----------|------------|-----------|------------|", + ] + for s in summaries: + best = f"{s.mean_best_metric:.4f} ± {s.std_best_metric:.4f}" if s.mean_best_metric is not None else "N/A" + impr = f"{s.mean_improvement_rate:.2%}" if s.mean_improvement_rate is not None else "N/A" + cost = f"${s.mean_cost_usd:.4f}" if s.mean_cost_usd is not None else "N/A" + secs = f"{s.mean_elapsed_sec:.1f}" if s.mean_elapsed_sec is not None else "N/A" + iters = f"{s.mean_iterations:.1f}" if s.mean_iterations is not None else "N/A" + tin = f"{s.mean_tokens_in:,.0f}" if s.mean_tokens_in is not None else "N/A" + tout = f"{s.mean_tokens_out:,.0f}" if s.mean_tokens_out is not None else "N/A" + lines.append( + f"| {s.strategy_name:8} | {best:17} | {impr:11} | {cost:8} | {secs:8} | {iters:10} | {tin:9} | {tout:10} |" + ) + lines.append("") + # Winner by metric + by_metric = [(s.mean_best_metric or float("-inf"), s.strategy_name) for s in summaries] + winner_metric = max(by_metric, key=lambda x: x[0]) + lines.append(f"**Winner by metric**: {winner_metric[1]} ({winner_metric[0]:.4f})") + # Winner by cost + by_cost = [(s.mean_cost_usd or float("inf"), s.strategy_name) for s in summaries] + winner_cost = min(by_cost, key=lambda x: x[0]) + lines.append(f"**Winner by cost**: {winner_cost[1]} (${winner_cost[0]:.4f})") + # Winner by time + by_time = [(s.mean_elapsed_sec or float("inf"), s.strategy_name) for s in summaries] + winner_time = min(by_time, key=lambda x: x[0]) + lines.append(f"**Winner by time**: {winner_time[1]} ({winner_time[0]:.1f}s)") + lines.append("") + return "\n".join(lines) + + @staticmethod + def to_json(summaries: list[StrategySummary]) -> str: + """Return a JSON-serializable report.""" + data = [] + for s in summaries: + data.append({ + "strategy": s.strategy_name, + "repeats": len(s.runs), + "mean_best_metric": s.mean_best_metric, + "std_best_metric": s.std_best_metric, + "mean_improvement_rate": s.mean_improvement_rate, + "mean_cost_usd": s.mean_cost_usd, + "mean_elapsed_sec": s.mean_elapsed_sec, + "mean_iterations": s.mean_iterations, + "mean_tokens_in": s.mean_tokens_in, + "mean_tokens_out": s.mean_tokens_out, + "runs": [ + { + "repeat": r.repeat, + "best_metric": r.best_metric, + "baseline_metric": r.baseline_metric, + "total_cost_usd": r.total_cost_usd, + "elapsed_sec": r.elapsed_sec, + "iterations": r.iterations_to_converge, + "improvement_rate": r.improvement_rate, + } + for r in s.runs + ], + }) + return json.dumps(data, indent=2) diff --git a/agent/research/auto_specify.py b/agent/research/auto_specify.py new file mode 100644 index 000000000000..3b5529e1db66 --- /dev/null +++ b/agent/research/auto_specify.py @@ -0,0 +1,109 @@ +"""Auto-specify: flesh out a vague research topic via the kanban triage +auxiliary LLM. Used by ``run_research(..., auto_specify=True)`` so callers +can pass a one-line topic and get back a structured TaskSpec scaffold. + +Reuses the ``triage_specifier`` aux-client role that the kanban +``/specify`` button already uses (same model + same prompt-shape +discipline). Output is constrained to a tight JSON schema so the caller +can drop the keys straight into ``TaskSpec(...)``. + +Failures are silent: aux-client unavailable, API error, or unparseable +JSON returns ``None``. The caller decides whether to fall back to its +original args or surface a warning. +""" +from __future__ import annotations + +import json +import logging +import re +from typing import Optional + +logger = logging.getLogger(__name__) + +_FENCE_RE = re.compile(r"^\s*```(?:json)?\s*|\s*```\s*$", re.IGNORECASE) + + +_SPECIFY_SYSTEM = ( + "You are a research-task specifier. Given a short topic that may be " + "vague, produce a JSON object with these keys:\n" + ' - "deliverable" (string): what the worker must produce, concretely\n' + ' - "metric_key" (string): the name of a measurable success metric\n' + ' - "metric_direction" (string): "maximize" or "minimize"\n' + ' - "task_type" (string): "code" | "search" | "research" | "generic"\n' + ' - "evaluation_mode" (string): "self_report" or "llm_judge"\n' + ' - "evaluation_prompt" (string, optional): only when evaluation_mode is ' + '"llm_judge" — a 0-to-1 scoring rubric\n\n' + "Output ONLY the JSON object. No commentary. No fences. No prose." +) + +_SPECIFY_USER_TEMPLATE = ( + "Topic: {topic}\n\n" + "Produce the JSON spec. Pick task_type to match the deliverable:\n" + " code → measurable test outcomes (pass_rate, accuracy)\n" + " search → ranked retrieval (relevance_score)\n" + " research → synthesis quality (completeness_score, coverage)\n" + " generic → anything else with a numeric metric" +) + + +def get_text_auxiliary_client(role: str): + """Indirection seam — patched in tests so we don't need a real aux client.""" + from agent.auxiliary_client import get_text_auxiliary_client as _impl + return _impl(role) + + +def _extract_json_blob(raw: str) -> Optional[dict]: + """Lenient JSON extraction tolerating fenced code blocks and prose + around the JSON. Returns None when nothing parses or the result is + not a dict.""" + if not raw: + return None + stripped = _FENCE_RE.sub("", raw.strip()) + first = stripped.find("{") + last = stripped.rfind("}") + if first == -1 or last == -1 or last <= first: + return None + try: + val = json.loads(stripped[first : last + 1]) + except (ValueError, json.JSONDecodeError): + return None + return val if isinstance(val, dict) else None + + +def auto_specify_topic(topic: str) -> Optional[dict]: + """Return a dict with TaskSpec scaffold fields, or None on any failure. + + Caller-facing schema (string keys; all optional in the response): + deliverable, metric_key, metric_direction, task_type, + evaluation_mode, evaluation_prompt + """ + if not topic or not topic.strip(): + return None + + try: + client, model = get_text_auxiliary_client("triage_specifier") + except Exception as exc: + logger.debug("auto_specify: aux client unavailable: %s", exc) + return None + if client is None or not model: + return None + + try: + resp = client.chat.completions.create( + model=model, + messages=[ + {"role": "system", "content": _SPECIFY_SYSTEM}, + {"role": "user", "content": _SPECIFY_USER_TEMPLATE.format(topic=topic)}, + ], + temperature=0, + ) + except Exception as exc: + logger.debug("auto_specify: aux call failed: %s", exc) + return None + + try: + content = resp.choices[0].message.content or "" + except Exception: + return None + + return _extract_json_blob(content) diff --git a/agent/research/events.py b/agent/research/events.py new file mode 100644 index 000000000000..4239830dd5c1 --- /dev/null +++ b/agent/research/events.py @@ -0,0 +1,55 @@ +"""agent.research.events — structured event emission for the research loop. + +Events are append-only JSON lines written to ``/events.jsonl`` +so that external observers (TUI, dashboard, Lattice hooks) can subscribe to +progress without polling the running process. +""" +from __future__ import annotations + +import json +import time +from enum import Enum, auto +from pathlib import Path +from typing import Any + + +class ResearchEvent(Enum): + """Canonical event types emitted during a research job lifecycle.""" + + JOB_STARTED = auto() + BASELINE_STARTED = auto() + BASELINE_COMPLETED = auto() + ITERATION_STARTED = auto() + ITERATION_COMPLETED = auto() + CHECKPOINT_SAVED = auto() + SNAPSHOT_CREATED = auto() + BEST_RESULT_UPDATED = auto() + TIMEOUT_DETECTED = auto() + STALE_DETECTED = auto() + JOB_COMPLETED = auto() + JOB_FAILED = auto() + + +def emit_event( + job_dir: Path, + event: ResearchEvent, + data: dict[str, Any] | None = None, +) -> None: + """Append a structured event to ``/events.jsonl``. + + Args: + job_dir: Research job directory (must exist). + event: Event type. + data: Optional extra metadata (iteration, metric, error, etc.). + """ + events_path = job_dir / "events.jsonl" + line = json.dumps( + { + "ts": time.time(), + "event": event.name, + "data": data or {}, + }, + default=str, + ) + with events_path.open("a", encoding="utf-8") as f: + f.write(line + "\n") diff --git a/agent/research/evolution.py b/agent/research/evolution.py new file mode 100644 index 000000000000..ef113dcd3f57 --- /dev/null +++ b/agent/research/evolution.py @@ -0,0 +1,588 @@ +"""Self-evolution system for the ResearchClaw pipeline. + +Records lessons from each pipeline run (failures, slow stages, quality issues) +and injects them into future runs as prompt overlays. Inspired by Sibyl's +time-weighted evolution mechanism. + +Architecture +------------ +* ``LessonCategory`` — 6 issue categories for classification. +* ``LessonEntry`` — single lesson (stage, category, severity, description, ts). +* ``EvolutionStore`` — JSONL-backed persistent store with append + query. +* ``extract_lessons()`` — auto-extract lessons from ``StageResult`` lists. +* ``build_overlay()`` — generate per-stage prompt overlay text. + +Usage +----- +:: + + from researchclaw.evolution import EvolutionStore, extract_lessons + + store = EvolutionStore(Path("evolution")) + lessons = extract_lessons(results) + store.append_many(lessons) + overlay = store.build_overlay("hypothesis_gen", max_lessons=5) +""" + +from __future__ import annotations + +import json +import logging +import math +from dataclasses import asdict, dataclass +from datetime import datetime, timezone +from enum import Enum +from pathlib import Path + +logger = logging.getLogger(__name__) + +# Skills directories to scan — Hermes autoresearch skills +_PROJECT_SKILLS_DIRS: tuple[str, ...] = ( + "skills/autoresearch", +) + + +def _load_project_skills() -> list[str]: + """Load skill content from Hermes autoresearch skills directory.""" + skills: list[str] = [] + root = Path(__file__).resolve().parent.parent + for rel_dir in _PROJECT_SKILLS_DIRS: + skills_dir = root / rel_dir + if not skills_dir.is_dir(): + continue + for skill_sub in sorted(skills_dir.iterdir()): + if not skill_sub.is_dir(): + continue + # Skip the main researchclaw CLI skill — it's not a pipeline overlay + if skill_sub.name == "researchclaw": + continue + skill_file = skill_sub / "SKILL.md" + if skill_file.is_file(): + try: + text = skill_file.read_text(encoding="utf-8").strip() + if text: + skills.append(text) + except OSError: + continue + return skills + + +class LessonCategory(str, Enum): + """Issue classification for extracted lessons.""" + + SYSTEM = "system" # Environment / network / timeout + EXPERIMENT = "experiment" # Code validation, sandbox timeout + WRITING = "writing" # Paper quality issues + ANALYSIS = "analysis" # Weak analysis, missing comparison + LITERATURE = "literature" # Search / verification failures + PIPELINE = "pipeline" # Stage orchestration issues + + +@dataclass +class LessonEntry: + """A single lesson extracted from a pipeline run.""" + + stage_name: str + stage_num: int + category: str + severity: str # "info", "warning", "error" + description: str + timestamp: str # ISO 8601 + run_id: str = "" + + def to_dict(self) -> dict[str, object]: + return asdict(self) + + @classmethod + def from_dict(cls, data: dict[str, object]) -> LessonEntry: + return cls( + stage_name=str(data.get("stage_name", "")), + stage_num=int(data.get("stage_num", 0)), + category=str(data.get("category", "pipeline")), + severity=str(data.get("severity", "info")), + description=str(data.get("description", "")), + timestamp=str(data.get("timestamp", "")), + run_id=str(data.get("run_id", "")), + ) + + +# --------------------------------------------------------------------------- +# Lesson classification keywords +# --------------------------------------------------------------------------- + +_CATEGORY_KEYWORDS: dict[str, list[str]] = { + LessonCategory.SYSTEM: [ + "timeout", "connection", "network", "oom", "memory", + "permission", "ssh", "socket", "dns", + ], + LessonCategory.EXPERIMENT: [ + "sandbox", "validation", "import", "syntax", "subprocess", + "experiment", "code", "execution", + ], + LessonCategory.WRITING: [ + "paper", "draft", "outline", "revision", "review", + "template", "latex", + ], + LessonCategory.ANALYSIS: [ + "analysis", "metric", "statistic", "comparison", "baseline", + ], + LessonCategory.LITERATURE: [ + "search", "citation", "verify", "hallucin", "arxiv", + "semantic_scholar", "literature", "collect", + ], +} + + +def _classify_error(stage_name: str, error_text: str) -> str: + """Classify an error into a LessonCategory based on keywords.""" + combined = f"{stage_name} {error_text}".lower() + best_category = LessonCategory.PIPELINE + best_score = 0 + for category, keywords in _CATEGORY_KEYWORDS.items(): + score = sum(1 for kw in keywords if kw in combined) + if score > best_score: + best_score = score + best_category = category + return best_category + + +# --------------------------------------------------------------------------- +# Lesson extraction from pipeline results +# --------------------------------------------------------------------------- + +# Stage name mapping (import-free to avoid circular deps) +_STAGE_NAMES: dict[int, str] = { + 1: "topic_init", 2: "problem_decompose", 3: "search_strategy", + 4: "literature_collect", 5: "literature_screen", 6: "knowledge_extract", + 7: "synthesis", 8: "hypothesis_gen", 9: "experiment_design", + 10: "code_generation", 11: "resource_planning", 12: "experiment_run", + 13: "iterative_refine", 14: "result_analysis", 15: "research_decision", + 16: "paper_outline", 17: "paper_draft", 18: "peer_review", + 19: "paper_revision", 20: "quality_gate", 21: "knowledge_archive", + 22: "export_publish", 23: "citation_verify", +} + + +def extract_lessons( + results: list[object], + run_id: str = "", + run_dir: Path | None = None, +) -> list[LessonEntry]: + """Extract lessons from a list of StageResult objects. + + Detects: + - Failed stages → error lesson + - Blocked stages → pipeline lesson + - Decision pivots/refines → pipeline lesson (with rationale if available) + - Runtime warnings from experiment stderr → code_bug lesson + - Metric anomalies (NaN, identical convergence) → metric_anomaly lesson + """ + now = datetime.now(timezone.utc).isoformat(timespec="seconds") + lessons: list[LessonEntry] = [] + + for result in results: + stage_num = int(getattr(result, "stage", 0)) + stage_name = _STAGE_NAMES.get(stage_num, f"stage_{stage_num}") + status = str(getattr(result, "status", "")) + error = getattr(result, "error", None) + decision = str(getattr(result, "decision", "proceed")) + + # Failed stages + if "failed" in status.lower() and error: + category = _classify_error(stage_name, str(error)) + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=stage_num, + category=category, + severity="error", + description=f"Stage {stage_name} failed: {str(error)[:300]}", + timestamp=now, + run_id=run_id, + )) + + # Blocked stages + if "blocked" in status.lower(): + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=stage_num, + category=LessonCategory.PIPELINE, + severity="warning", + description=f"Stage {stage_name} blocked awaiting approval", + timestamp=now, + run_id=run_id, + )) + + # PIVOT / REFINE decisions — extract rationale if available + if decision in ("pivot", "refine"): + rationale = _extract_decision_rationale(run_dir) if run_dir else "" + desc = f"Research decision was {decision.upper()}" + if rationale: + desc += f": {rationale[:200]}" + else: + desc += " — prior hypotheses/experiments were insufficient" + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=stage_num, + category=LessonCategory.PIPELINE, + severity="warning", + description=desc, + timestamp=now, + run_id=run_id, + )) + + # --- Extract lessons from experiment artifacts --- + if run_dir is not None: + lessons.extend(_extract_runtime_lessons(run_dir, now, run_id)) + + return lessons + + +def _extract_decision_rationale(run_dir: Path) -> str: + """Extract rationale from the most recent decision_structured.json. + + Supports multiple field formats: + - ``rationale`` or ``reason`` key (direct) + - ``raw_text_excerpt`` containing ``## Justification`` section (LLM output) + """ + for stage_dir in sorted(run_dir.glob("stage-15*"), reverse=True): + decision_file = stage_dir / "decision_structured.json" + if decision_file.exists(): + try: + data = json.loads(decision_file.read_text(encoding="utf-8")) + if not isinstance(data, dict): + continue + # Try direct rationale/reason keys first + direct = data.get("rationale", "") or data.get("reason", "") + if direct: + return str(direct) + # Parse raw_text_excerpt for Justification section + raw = data.get("raw_text_excerpt", "") + if raw: + return _parse_justification_from_excerpt(str(raw)) + except (json.JSONDecodeError, OSError): + pass + return "" + + +def _parse_justification_from_excerpt(text: str) -> str: + """Extract the Justification/Rationale section from LLM decision text.""" + import re + + # Match ## Justification, ## Rationale, or similar headings + pattern = re.compile( + r"##\s*(?:Justification|Rationale|Reason)\s*\n(.*?)(?=\n##|\Z)", + re.DOTALL | re.IGNORECASE, + ) + match = pattern.search(text) + if match: + return match.group(1).strip()[:300] + # Fallback: skip the first line (## Decision / **REFINE**) and return the rest + lines = [l.strip() for l in text.splitlines() if l.strip()] + # Skip heading lines starting with ## or ** + content_lines = [ + l for l in lines + if not l.startswith("##") and not (l.startswith("**") and l.endswith("**")) + ] + if content_lines: + return " ".join(content_lines)[:300] + return "" + + +def _extract_runtime_lessons( + run_dir: Path, timestamp: str, run_id: str +) -> list[LessonEntry]: + """Extract fine-grained lessons from experiment run artifacts.""" + import math + + lessons: list[LessonEntry] = [] + + # Check sandbox run results for stderr warnings and NaN + for runs_dir in run_dir.glob("stage-*/runs"): + for run_file in runs_dir.glob("*.json"): + if run_file.name == "results.json": + continue + try: + payload = json.loads(run_file.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + continue + if not isinstance(payload, dict): + continue + + # Check stderr for runtime warnings + stderr = payload.get("stderr", "") + if stderr and any( + kw in stderr for kw in ("Warning", "Error", "divide", "overflow", "invalid value") + ): + lessons.append(LessonEntry( + stage_name="experiment_run", + stage_num=12, + category=LessonCategory.EXPERIMENT, + severity="warning", + description=f"Runtime warning in experiment: {stderr[:200]}", + timestamp=timestamp, + run_id=run_id, + )) + + # Check metrics for NaN/Inf + metrics = payload.get("metrics", {}) + if isinstance(metrics, dict): + for key, val in metrics.items(): + try: + fval = float(val) + if math.isnan(fval) or math.isinf(fval): + lessons.append(LessonEntry( + stage_name="experiment_run", + stage_num=12, + category=LessonCategory.EXPERIMENT, + severity="error", + description=f"Metric '{key}' was {val} — code bug (division by zero or overflow)", + timestamp=timestamp, + run_id=run_id, + )) + except (TypeError, ValueError): + pass + + return lessons + + +# --------------------------------------------------------------------------- +# Time-decay weighting +# --------------------------------------------------------------------------- + +HALF_LIFE_DAYS: float = 30.0 +MAX_AGE_DAYS: float = 90.0 + + +def _time_weight(timestamp_iso: str) -> float: + """Compute exponential decay weight for a lesson based on age. + + Uses 30-day half-life: weight = exp(-age_days * ln(2) / 30). + Returns 0.0 for lessons older than 90 days. + """ + try: + ts = datetime.fromisoformat(timestamp_iso) + if ts.tzinfo is None: + ts = ts.replace(tzinfo=timezone.utc) + age = datetime.now(timezone.utc) - ts + age_days = age.total_seconds() / 86400.0 + if age_days > MAX_AGE_DAYS: + return 0.0 + return math.exp(-age_days * math.log(2) / HALF_LIFE_DAYS) + except (ValueError, TypeError): + return 0.0 + + +# --------------------------------------------------------------------------- +# Evolution store +# --------------------------------------------------------------------------- + + +class EvolutionStore: + """JSONL-backed store for pipeline lessons.""" + + def __init__(self, store_dir: Path) -> None: + self._dir = store_dir + self._dir.mkdir(parents=True, exist_ok=True) + self._lessons_path = self._dir / "lessons.jsonl" + + @property + def lessons_path(self) -> Path: + return self._lessons_path + + def append(self, lesson: LessonEntry) -> None: + """Append a single lesson to the store.""" + with self._lessons_path.open("a", encoding="utf-8") as f: + f.write(json.dumps(lesson.to_dict(), ensure_ascii=False) + "\n") + + def append_many(self, lessons: list[LessonEntry]) -> None: + """Append multiple lessons atomically.""" + if not lessons: + return + with self._lessons_path.open("a", encoding="utf-8") as f: + for lesson in lessons: + f.write(json.dumps(lesson.to_dict(), ensure_ascii=False) + "\n") + logger.info("Appended %d lessons to evolution store", len(lessons)) + + def load_all(self) -> list[LessonEntry]: + """Load all lessons from disk.""" + if not self._lessons_path.exists(): + return [] + lessons: list[LessonEntry] = [] + for line in self._lessons_path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + data = json.loads(line) + lessons.append(LessonEntry.from_dict(data)) + except (json.JSONDecodeError, TypeError): + continue + return lessons + + def query_for_stage( + self, stage_name: str, *, max_lessons: int = 5 + ) -> list[LessonEntry]: + """Return the most relevant lessons for a stage, weighted by recency. + + Includes lessons that directly match the stage, plus high-severity + lessons from related stages. + """ + all_lessons = self.load_all() + scored: list[tuple[float, LessonEntry]] = [] + for lesson in all_lessons: + weight = _time_weight(lesson.timestamp) + if weight <= 0.0: + continue + # Boost direct stage matches + if lesson.stage_name == stage_name: + weight *= 2.0 + # Boost errors over warnings/info + if lesson.severity == "error": + weight *= 1.5 + scored.append((weight, lesson)) + scored.sort(key=lambda x: x[0], reverse=True) + return [entry for _, entry in scored[:max_lessons]] + + def build_overlay( + self, + stage_name: str, + *, + max_lessons: int = 5, + skills_dir: str = "", + ) -> str: + """Generate a prompt overlay string for a given stage. + + Combines two sources: + 1. Current-run lessons from ``lessons.jsonl`` (intra-run learning). + 2. Cross-run MetaClaw ``arc-*`` skills from *skills_dir* (inter-run + learning via the MetaClaw skill-generation feedback loop). + + Project-level and user-level skills are handled separately by the + SkillRegistry in ``_helpers._get_skill_registry()``. + + Returns empty string if no relevant lessons or skills exist. + """ + parts: list[str] = [] + + # --- Section 1: intra-run lessons --- + lessons = self.query_for_stage(stage_name, max_lessons=max_lessons) + if lessons: + parts.append("## Lessons from Prior Runs") + for i, lesson in enumerate(lessons, 1): + severity_icon = {"error": "❌", "warning": "⚠️", "info": "ℹ️"}.get( + lesson.severity, "•" + ) + parts.append( + f"{i}. {severity_icon} [{lesson.category}] {lesson.description}" + ) + parts.append( + "\nUse these lessons to avoid repeating past mistakes." + ) + + # --- Section 2: cross-run MetaClaw arc-* skills --- + arc_skills: list[str] = [] + if skills_dir: + from pathlib import Path as _Path + + sd = _Path(skills_dir).expanduser() + if sd.is_dir(): + for skill_dir in sorted(sd.iterdir()): + if skill_dir.is_dir() and skill_dir.name.startswith("arc-"): + skill_file = skill_dir / "SKILL.md" + if skill_file.is_file(): + try: + text = skill_file.read_text(encoding="utf-8").strip() + if text: + arc_skills.append(text) + except OSError: + continue + + if arc_skills: + parts.append("\n## Learned Skills from Prior Runs") + for skill_text in arc_skills[:5]: + parts.append(skill_text) + parts.append( + "\nApply these skills proactively to improve quality." + ) + + return "\n".join(parts) + + def count(self) -> int: + """Return total number of stored lessons.""" + return len(self.load_all()) + + def export_to_memory(self, memory_store: object) -> int: + """Export lessons to a memory store (duck-typed to avoid circular imports). + + The *memory_store* must expose an ``add(content, category, metadata)`` method + (compatible with ``researchclaw.memory.store.MemoryStore``). + + Returns the number of lessons exported. + """ + add_fn = getattr(memory_store, "add", None) + if add_fn is None or not callable(add_fn): + logger.warning("export_to_memory: memory_store has no add() method") + return 0 + lessons = self.load_all() + exported = 0 + for lesson in lessons: + weight = _time_weight(lesson.timestamp) + if weight <= 0.0: + continue + try: + # Map lesson categories to valid MemoryStore categories + _CAT_MAP = { + "system": "experiment", "analysis": "experiment", + "literature": "ideation", "pipeline": "experiment", + "experiment": "experiment", "writing": "writing", + "ideation": "ideation", + } + _mem_cat = _CAT_MAP.get(lesson.category, "experiment") + add_fn( + content=lesson.description, + category=_mem_cat, + metadata={ + "source": "evolution", + "stage": lesson.stage_name, + "severity": lesson.severity, + "run_id": lesson.run_id, + "timestamp": lesson.timestamp, + }, + ) + exported += 1 + except Exception: + logger.debug("Failed to export lesson: %s", lesson.description[:80]) + return exported + + def get_lessons_for_stage_with_memory( + self, + stage_name: str, + memory_store: object, + *, + max_lessons: int = 5, + ) -> str: + """Combine evolution overlay with memory context for a stage. + + *memory_store* must expose a ``recall(query, category, max_results)`` method + returning objects with a ``.content`` attribute. + """ + overlay = self.build_overlay(stage_name, max_lessons=max_lessons) + recall_fn = getattr(memory_store, "recall", None) + if recall_fn is None or not callable(recall_fn): + return overlay + try: + memories = recall_fn( + query=stage_name, + category=None, + max_results=max_lessons, + ) + if memories: + parts = ["\n## Recalled Memories"] + for i, mem in enumerate(memories, 1): + content = getattr(mem, "content", str(mem)) + parts.append(f"{i}. {content}") + memory_text = "\n".join(parts) + return f"{overlay}\n{memory_text}" if overlay else memory_text + except Exception: + logger.debug("Failed to recall memories for stage %s", stage_name) + return overlay diff --git a/agent/research/job_runner.py b/agent/research/job_runner.py new file mode 100644 index 000000000000..afe83ad6919a --- /dev/null +++ b/agent/research/job_runner.py @@ -0,0 +1,418 @@ +"""agent.research.job_runner — detached process entrypoint for long-running research loops. + +Two entrypoints: + + * ``main(spec_path)`` — the **parent** orchestrator. It spawns a fresh + Python subprocess running ``_child_main`` and monitors it for both + wall-clock timeout (HRM-94) and heartbeat liveness (HRM-95). On + expiry it escalates SIGTERM → SIGKILL and writes a terminal status + (``timeout`` / ``stale``) to ``state.json`` so external watchers + (research_job_tool, status probes) see the job ended. + + * ``_child_main(spec_path)`` — runs inside the spawned subprocess. + Builds the parent agent, calls ``run_research``, writes + ``result.json`` + the final ``state.json``, and refreshes + ``heartbeat.json`` every 30 s on a daemon thread. + +Why a subprocess instead of multiprocessing.Process? Hermes' runtime +holds non-fork-safe state (SQLite connections, logging handlers, +provider HTTP clients, native threads). Forking and continuing in a +child interpreter is brittle. A clean subprocess starting from +``python -m agent.research.job_runner --child `` re-imports +the world fresh and avoids those hazards. The trade-off is that the +parent must communicate with the child via files (state.json, +heartbeat.json, result.json) rather than shared memory. + +Usage: + python -m agent.research.job_runner /path/to/job.json +""" + +from __future__ import annotations + +import json +import logging +import os +import signal +import subprocess +import sys +import threading +import time +from pathlib import Path +from typing import Any + +# Re-exported so callers (research_job_tool, status probes, tests) have a +# single import surface for "is there something to resume here?". The real +# logic lives in supervisor._detect_resume — see the docstring there for +# the consistency-vs-probe distinction. +from agent.research.supervisor import _detect_resume # noqa: F401 +from agent.research.events import ResearchEvent, emit_event + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Tunables — overridable via env vars for tests so the timeout / heartbeat +# loops finish in milliseconds instead of minutes. +# --------------------------------------------------------------------------- + +_HEARTBEAT_INTERVAL = float(os.getenv("HERMES_JOB_HEARTBEAT_INTERVAL", "30")) +_STALE_THRESHOLD = float(os.getenv("HERMES_JOB_STALE_THRESHOLD", "90")) +_POLL_INTERVAL = float(os.getenv("HERMES_JOB_POLL_INTERVAL", "1")) +_SIGTERM_GRACE = float(os.getenv("HERMES_JOB_SIGTERM_GRACE", "5")) + + +def _setup_logging(job_dir: Path) -> None: + log_path = job_dir / "runner.log" + handler = logging.FileHandler(log_path, mode="a") + handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")) + root = logging.getLogger() + root.setLevel(logging.DEBUG) + root.addHandler(handler) + + +def _write_state(job_dir: Path, **fields: Any) -> None: + state_path = job_dir / "state.json" + state = json.loads(state_path.read_text()) if state_path.exists() else {} + state.update(fields) + state["updated_at"] = time.time() + state_path.write_text(json.dumps(state, indent=2)) + + +# --------------------------------------------------------------------------- +# Heartbeat — written by the child, read by the parent and external probes. +# --------------------------------------------------------------------------- + +def _write_heartbeat(job_dir: Path) -> None: + """Write ``/heartbeat.json`` atomically. + + Atomic via tempfile + os.replace so a reader never sees a partial + write — important because the parent polls this file aggressively. + """ + hb = job_dir / "heartbeat.json" + tmp = hb.with_name(hb.name + ".tmp") + tmp.write_text(json.dumps({"ts": time.time(), "pid": os.getpid()})) + os.replace(tmp, hb) + + +def _heartbeat_loop(job_dir: Path, stop_event: threading.Event) -> None: + """Daemon-thread loop that refreshes heartbeat.json every interval.""" + while not stop_event.is_set(): + try: + _write_heartbeat(job_dir) + except Exception as exc: + logger.warning("heartbeat write failed: %s", exc) + # Event.wait() returns True if set during the wait — clean exit. + if stop_event.wait(_HEARTBEAT_INTERVAL): + return + + +def _is_heartbeat_stale(job_dir: Path) -> bool: + """True when heartbeat.json is missing or older than the threshold.""" + hb = job_dir / "heartbeat.json" + if not hb.exists(): + # Don't immediately treat absence as stale — the child may not + # have written its first heartbeat yet. Caller distinguishes + # "never seen" from "gone stale" via age tracking. + return True + try: + data = json.loads(hb.read_text()) + ts = float(data.get("ts", 0)) + except (OSError, json.JSONDecodeError, TypeError, ValueError): + return True + return (time.time() - ts) > _STALE_THRESHOLD + + +# --------------------------------------------------------------------------- +# Process control — SIGTERM with SIGKILL escalation. +# --------------------------------------------------------------------------- + +def _kill_with_escalation(proc: subprocess.Popen) -> None: + """SIGTERM, wait grace period, SIGKILL if still alive. + + Mirrors the pattern documented in DESIGN-HRM94-95.md but expressed + against subprocess.Popen rather than multiprocessing.Process. + """ + if proc.poll() is not None: + return + try: + os.kill(proc.pid, signal.SIGTERM) + except ProcessLookupError: + return + try: + proc.wait(timeout=_SIGTERM_GRACE) + return + except subprocess.TimeoutExpired: + pass + try: + os.kill(proc.pid, signal.SIGKILL) + except ProcessLookupError: + return + try: + proc.wait(timeout=1) + except subprocess.TimeoutExpired: + logger.warning("child %s did not exit after SIGKILL", proc.pid) + + +# --------------------------------------------------------------------------- +# Child process — the actual research loop runs here. +# --------------------------------------------------------------------------- + +def _build_agent(spec: dict[str, Any]) -> Any: + """Build an AIAgent from the job spec. + + Thin wrapper around ``agent.factory.build_agent_for_research_job`` — + construction + post-init patching live there so other detached + entrypoints can reuse the same logic. See agent/factory.py for the + "keep in sync with AIAgent" caveat. + """ + from agent.factory import build_agent_for_research_job + return build_agent_for_research_job(spec) + + +def _child_main(spec_path: str) -> int: + """Body of the spawned subprocess: build agent, run research, write state. + + The parent owns timeout / heartbeat-stale detection; this function + only owns its own heartbeat thread and the actual call to + ``run_research``. State writes here are read by the parent (and by + external watchers) for the *successful* completion path; on timeout + or kill the parent overwrites status itself. + """ + spec = json.loads(Path(spec_path).read_text()) + job_dir = Path(spec["job_dir"]) + job_dir.mkdir(parents=True, exist_ok=True) + + _setup_logging(job_dir) + + stop_event = threading.Event() + # Synchronous initial heartbeat — the parent's polling loop starts + # before the daemon thread fires, and we don't want a phantom + # "stale" within the first second of life. + _write_heartbeat(job_dir) + hb_thread = threading.Thread( + target=_heartbeat_loop, args=(job_dir, stop_event), daemon=True + ) + hb_thread.start() + + try: + # Test hooks — exercised by tests/agent/research/test_*_timeout + # and test_heartbeat_stale. Production specs never set these. + test_mode = spec.get("_test_mode") + if test_mode == "sleep": + time.sleep(float(spec.get("_test_sleep_sec", 5))) + _write_state(job_dir, status="completed") + return 0 + if test_mode == "freeze_heartbeat": + # Stop refreshing the heartbeat, then sleep — used to trigger + # the parent's stale-detection path without killing the child + # ourselves. + stop_event.set() + time.sleep(float(spec.get("_test_sleep_sec", 60))) + return 0 + + try: + agent = _build_agent(spec) + except Exception as exc: + logger.exception("Failed to build parent agent") + _write_state(job_dir, status="failed", error=f"parent_agent build failed: {exc}") + return 1 + + from tools.research_tool import run_research + + try: + logger.info("Calling run_research with checkpoint_dir=%s", job_dir) + raw = run_research( + topic=spec["topic"], + deliverable=spec["deliverable"], + metric_key=spec["metric_key"], + metric_direction=spec.get("metric_direction", "maximize"), + task_type=spec.get("task_type", "generic"), + evaluation_mode=spec.get("evaluation_mode", "self_report"), + evaluation_prompt=spec.get("evaluation_prompt", ""), + initial_attempt=spec.get("initial_attempt", ""), + max_iterations=spec.get("max_iterations", 3), + time_budget_sec=spec.get("time_budget_sec", 0), + kanban_task_id=spec.get("kanban_task_id"), + parent_agent=agent, + checkpoint_dir=str(job_dir), + timeout_sec=spec.get("timeout_sec", 0), + ) + + result = json.loads(raw) + (job_dir / "result.json").write_text(json.dumps(result, indent=2)) + + status = "completed" if "error" not in result else "failed" + _write_state(job_dir, status=status, **result) + logger.info("Job %s finished: %s", spec["job_id"], status) + return 0 if status == "completed" else 1 + + except Exception as exc: + logger.exception("Job %s failed", spec["job_id"]) + _write_state(job_dir, status="failed", error=str(exc)) + return 1 + finally: + stop_event.set() + + +# --------------------------------------------------------------------------- +# Parent process — spawns the child, watches the clock and the heartbeat. +# --------------------------------------------------------------------------- + +def _spawn_child(spec_path: str) -> subprocess.Popen: + """Spawn the child subprocess that actually runs the research loop. + + Uses ``sys.executable -m agent.research.job_runner --child `` + so the child re-imports the module from a clean interpreter — no + forked SQLite handles, no carried-over logging state. + """ + return subprocess.Popen( + [sys.executable, "-m", "agent.research.job_runner", "--child", spec_path], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def main(spec_path: str) -> int: + """Parent orchestrator: spawn child, monitor, finalize state. + + Returns 0 on clean child exit, 1 on timeout / stale / failure, 2 if + another instance of the same job already holds the lock. + """ + spec = json.loads(Path(spec_path).read_text()) + job_dir = Path(spec["job_dir"]) + job_dir.mkdir(parents=True, exist_ok=True) + + # Lock to prevent concurrent runs of the same job. + lock_path = job_dir / ".runner.lock" + try: + fd = os.open(str(lock_path), os.O_CREAT | os.O_EXCL | os.O_WRONLY) + with os.fdopen(fd, "w") as f: + f.write(str(os.getpid())) + except FileExistsError: + print( + f"ERROR: Job {spec['job_id']} is already running (lock file exists). Exiting.", + file=sys.stderr, + ) + return 2 + + proc: subprocess.Popen | None = None + try: + _setup_logging(job_dir) + logger.info("Job %s starting (parent pid=%d)", spec["job_id"], os.getpid()) + emit_event(job_dir, ResearchEvent.JOB_STARTED, {"job_id": spec["job_id"], "parent_pid": os.getpid()}) + + timeout_sec = int(spec.get("timeout_sec", 0) or 0) + + _write_state( + job_dir, + job_id=spec["job_id"], + status="running", + pid=os.getpid(), + started_at=time.time(), + spec_path=spec_path, + timeout_sec=timeout_sec, + ) + + proc = _spawn_child(spec_path) + _write_state(job_dir, child_pid=proc.pid) + logger.info("Spawned child pid=%d for job %s", proc.pid, spec["job_id"]) + + deadline = time.monotonic() + timeout_sec if timeout_sec > 0 else None + # Allow the child a grace window to write its first heartbeat + # before we start treating "missing heartbeat" as a kill signal. + first_seen_at: float | None = None + startup_grace = max(_STALE_THRESHOLD, _HEARTBEAT_INTERVAL * 2) + spawn_time = time.monotonic() + + while True: + rc = proc.poll() + if rc is not None: + logger.info("Child %s exited rc=%d", proc.pid, rc) + if rc != 0: + # Child should have written state.json itself; only + # overwrite if it didn't manage to set a terminal + # status (e.g. crashed before the finally block). + state = _read_state(job_dir) + if state.get("status") in (None, "running"): + _write_state( + job_dir, + status="failed", + error=f"child exited rc={rc}", + ) + return rc + + now = time.monotonic() + + if deadline is not None and now > deadline: + logger.warning( + "Timeout: child %s exceeded %ds, killing", proc.pid, timeout_sec + ) + _kill_with_escalation(proc) + emit_event(job_dir, ResearchEvent.TIMEOUT_DETECTED, {"timeout_sec": timeout_sec}) + _write_state( + job_dir, + status="timeout", + error=f"Timed out after {timeout_sec}s", + ) + return 1 + + hb_path = job_dir / "heartbeat.json" + if hb_path.exists(): + if first_seen_at is None: + first_seen_at = now + if _is_heartbeat_stale(job_dir): + logger.warning( + "Heartbeat stale: child %s, killing", proc.pid + ) + _kill_with_escalation(proc) + _write_state( + job_dir, + status="stale", + error="No heartbeat update for >threshold", + ) + return 1 + elif (now - spawn_time) > startup_grace: + logger.warning( + "Child %s never wrote heartbeat within %ss, killing", + proc.pid, startup_grace, + ) + _kill_with_escalation(proc) + emit_event(job_dir, ResearchEvent.STALE_DETECTED, {"reason": "no_initial_heartbeat"}) + _write_state( + job_dir, + status="stale", + error="Child never wrote initial heartbeat", + ) + return 1 + + time.sleep(_POLL_INTERVAL) + + finally: + if proc is not None and proc.poll() is None: + _kill_with_escalation(proc) + try: + lock_path.unlink(missing_ok=True) + except Exception: + pass + + +def _read_state(job_dir: Path) -> dict[str, Any]: + sp = job_dir / "state.json" + if not sp.exists(): + return {} + try: + return json.loads(sp.read_text()) + except (OSError, json.JSONDecodeError): + return {} + + +# --------------------------------------------------------------------------- +# CLI dispatch +# --------------------------------------------------------------------------- + +if __name__ == "__main__": + if len(sys.argv) >= 3 and sys.argv[1] == "--child": + sys.exit(_child_main(sys.argv[2])) + if len(sys.argv) < 2: + print("Usage: python -m agent.research.job_runner ", file=sys.stderr) + sys.exit(1) + sys.exit(main(sys.argv[1])) diff --git a/agent/research/metrics.py b/agent/research/metrics.py new file mode 100644 index 000000000000..035866f5c7b8 --- /dev/null +++ b/agent/research/metrics.py @@ -0,0 +1,293 @@ +"""Universal metric parser — supports JSON, CSV, and stdout regex formats. + +Parse priority: + 1. ``results.json`` — structured JSON output (recommended for all domains) + 2. ``results.csv`` — tabular output + 3. stdout regex — backward-compatible with existing ``metric: value`` format + +This module extends (not replaces) the existing ``sandbox.parse_metrics`` +function. The existing stdout parser remains the fallback. +""" + +from __future__ import annotations + +import csv +import json +import logging +import math +import re +from dataclasses import dataclass, field +from enum import Enum +from io import StringIO +from pathlib import Path +from typing import Any + +logger = logging.getLogger(__name__) + + +class MetricType(str, Enum): + SCALAR = "scalar" + TABLE = "table" + CONVERGENCE = "convergence" + LEARNING_CURVE = "learning_curve" + CONFUSION_MATRIX = "confusion" + STRUCTURED = "structured" + PARETO = "pareto" + + +@dataclass +class ExperimentResults: + """Unified experiment results container. + + Works for all domains — ML scalar metrics, physics convergence data, + economics regression tables, etc. + """ + + # Flat scalar metrics (backward-compatible with existing pipeline) + scalars: dict[str, float] = field(default_factory=dict) + + # Per-condition results (new universal format) + conditions: dict[str, dict[str, Any]] = field(default_factory=dict) + + # Convergence data (for physics/math domains) + convergence: dict[str, list[dict[str, float]]] = field(default_factory=dict) + + # Regression tables (for economics) + regression_table: dict[str, dict[str, Any]] = field(default_factory=dict) + + # Full structured data (raw JSON) + structured: dict[str, Any] = field(default_factory=dict) + + # Metadata + experiment_type: str = "" + domain: str = "" + total_runtime_sec: float = 0.0 + source: str = "" # "json" | "csv" | "stdout" + + def to_flat_metrics(self) -> dict[str, float]: + """Convert to flat metric dict for backward compatibility. + + The existing pipeline expects dict[str, float] from parse_metrics(). + This method flattens all result types into that format. + """ + metrics: dict[str, float] = dict(self.scalars) + + # Flatten conditions + for cond_name, seeds in self.conditions.items(): + if isinstance(seeds, dict): + for seed_or_metric, value in seeds.items(): + if isinstance(value, dict): + for metric_name, metric_val in value.items(): + if isinstance(metric_val, (int, float)) and math.isfinite(metric_val): + metrics[f"{cond_name}/{metric_name}"] = float(metric_val) + elif isinstance(value, (int, float)) and math.isfinite(value): + metrics[f"{cond_name}/{seed_or_metric}"] = float(value) + + # Flatten convergence (take final/best error per method) + for method, points in self.convergence.items(): + if points: + last = points[-1] + for key, val in last.items(): + if key != "h" and isinstance(val, (int, float)) and math.isfinite(val): + metrics[f"{method}/{key}"] = float(val) + + # Flatten regression table + for spec, coeffs in self.regression_table.items(): + if isinstance(coeffs, dict): + for key, val in coeffs.items(): + if isinstance(val, (int, float)) and math.isfinite(val): + metrics[f"{spec}/{key}"] = float(val) + + return metrics + + +class UniversalMetricParser: + """Parse experiment results from multiple output formats. + + Usage:: + + parser = UniversalMetricParser() + results = parser.parse(run_dir) + flat = results.to_flat_metrics() # backward-compatible + """ + + def parse(self, run_dir: Path, stdout: str = "") -> ExperimentResults: + """Parse experiment results from a run directory. + + Tries formats in order: JSON → CSV → stdout regex. + """ + # 1. Try JSON + results_json = run_dir / "results.json" + if results_json.exists(): + try: + result = self._parse_json(results_json) + if result.scalars or result.conditions or result.convergence or result.regression_table: + logger.info("Parsed results from results.json") + return result + except Exception: + logger.warning("Failed to parse results.json", exc_info=True) + + # 2. Try CSV + results_csv = run_dir / "results.csv" + if results_csv.exists(): + try: + result = self._parse_csv(results_csv) + if result.source == "csv": + logger.info("Parsed results from results.csv") + return result + except Exception: + logger.warning("Failed to parse results.csv", exc_info=True) + + # 3. Fallback: stdout regex (existing behavior) + if stdout: + return self._parse_stdout(stdout) + + # Try reading stdout.log from run_dir + stdout_log = run_dir / "stdout.log" + if stdout_log.exists(): + try: + stdout_text = stdout_log.read_text(encoding="utf-8", errors="replace") + return self._parse_stdout(stdout_text) + except Exception: + logger.warning("Failed to read stdout.log", exc_info=True) + + return ExperimentResults(source="none") + + def _parse_json(self, path: Path) -> ExperimentResults: + """Parse structured JSON results.""" + with path.open(encoding="utf-8") as fh: + data = json.load(fh) + + if not isinstance(data, dict): + return ExperimentResults(source="json") + + result = ExperimentResults( + source="json", + experiment_type=data.get("experiment_type", ""), + structured=data, + ) + + # Extract metadata + meta = data.get("metadata", {}) + if isinstance(meta, dict): + result.domain = meta.get("domain", "") + result.total_runtime_sec = float(meta.get("total_runtime_sec", 0)) + + # Extract conditions (comparison experiments) + conditions = data.get("conditions", {}) + if isinstance(conditions, dict): + result.conditions = conditions + # Also extract scalar metrics for backward compatibility + for cond_name, seeds in conditions.items(): + if isinstance(seeds, dict): + for seed_key, metrics in seeds.items(): + if isinstance(metrics, dict): + for metric_name, val in metrics.items(): + if isinstance(val, (int, float)) and math.isfinite(val): + result.scalars[f"{cond_name}/{metric_name}"] = float(val) + result.scalars[metric_name] = float(val) + elif isinstance(metrics, (int, float)) and math.isfinite(metrics): + result.scalars[f"{cond_name}/{seed_key}"] = float(metrics) + + # Extract convergence data + convergence = data.get("convergence", {}) + if isinstance(convergence, dict): + result.convergence = convergence + + # Extract regression table + reg_table = data.get("regression_table", {}) + if isinstance(reg_table, dict): + result.regression_table = reg_table + + # Top-level scalar metrics + for key, val in data.items(): + if key not in ("conditions", "convergence", "regression_table", "metadata", "experiment_type"): + if isinstance(val, (int, float)) and math.isfinite(val): + result.scalars[key] = float(val) + + return result + + def _parse_csv(self, path: Path) -> ExperimentResults: + """Parse CSV results (one row per condition/seed/metric).""" + text = path.read_text(encoding="utf-8", errors="replace") + reader = csv.DictReader(StringIO(text)) + + result = ExperimentResults(source="csv") + rows_processed = 0 + + for row in reader: + rows_processed += 1 + # Expected columns: condition, seed, metric, value + # Or: method, h, error (for convergence) + cond = row.get("condition", row.get("method", "")) + metric = row.get("metric", "") + value_str = row.get("value", row.get("error", "")) + + try: + val = float(value_str) + except (ValueError, TypeError): + continue + + if not math.isfinite(val): + continue + + if metric: + key = f"{cond}/{metric}" if cond else metric + result.scalars[key] = val + elif cond: + # Convergence-style: method, h, error + h_str = row.get("h", "") + try: + h = float(h_str) + except (ValueError, TypeError): + continue + if cond not in result.convergence: + result.convergence[cond] = [] + result.convergence[cond].append({"h": h, "error": val}) + + # Mark as CSV source if we processed any rows (even if no valid data) + if rows_processed == 0: + result.source = "none" + + return result + + def _parse_stdout(self, stdout: str) -> ExperimentResults: + """Parse stdout using regex: 'metric: value' and 'METRIC: key=value' formats.""" + metrics = _parse_metrics_from_stdout(stdout) + return ExperimentResults( + scalars={k: float(v) for k, v in metrics.items() if isinstance(v, (int, float))}, + source="stdout", + ) + + +# Inline metric parser (ported from researchclaw.experiment.sandbox) +_FLOAT_RE = r"[+-]?\d+\.?\d*(?:[eE][+-]?\d+)?" +_METRIC_PATTERN = re.compile( + rf"^(?:\S+=\S+\s+)?(\w[\w.]*)\s*:\s*({_FLOAT_RE})\s*$" +) +_HERMES_METRIC_PATTERN = re.compile( + r"METRIC:\s*(\w[\w.]*)\s*=\s*(" + _FLOAT_RE + r")" +) + + +def _parse_metrics_from_stdout(stdout: str) -> dict[str, float]: + """Extract metric: value pairs from stdout text.""" + metrics: dict[str, float] = {} + for line in stdout.splitlines(): + line = line.strip() + # Hermes format: "METRIC: key=value STATUS: ..." + m = _HERMES_METRIC_PATTERN.search(line) + if m: + try: + metrics[m.group(1)] = float(m.group(2)) + except ValueError: + pass + continue + # Standard format: "metric_name: value" + m = _METRIC_PATTERN.match(line) + if m: + try: + metrics[m.group(1)] = float(m.group(2)) + except ValueError: + pass + return metrics diff --git a/agent/research/runner.py b/agent/research/runner.py new file mode 100644 index 000000000000..7ec616836cb0 --- /dev/null +++ b/agent/research/runner.py @@ -0,0 +1,429 @@ +"""Experiment execution engine — Karpathy edit→run→eval→keep/discard loop for Hermes. + +Ported from researchclaw/experiment/runner.py (aiming-lab/AutoResearchClaw, MIT). +Seams replaced: + - sandbox.run() → delegate_fn (async callable wrapping delegate_task) + - git branch/commit/discard → Lattice task lifecycle via lattice_comment_fn + - ExperimentConfig → HermesExperimentConfig (simple dataclass, no researchclaw deps) +""" + +from __future__ import annotations + +import json +import logging +import re +import time as _time +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Callable, Optional, Protocol, cast + +logger = logging.getLogger(__name__) + + +@dataclass +class HermesExperimentConfig: + """Minimal experiment config — replaces researchclaw ExperimentConfig.""" + metric_key: str = "primary_metric" + metric_direction: str = "maximize" # "maximize" or "minimize" + time_budget_sec: int = 0 + max_iterations: int = 5 + keep_threshold: float = 0.0 # min abs delta to consider "kept" + + +@dataclass +class DelegateSandboxResult: + """Adapter wrapping delegate_task JSON result into sandbox-like shape.""" + metrics: dict[str, object] + stdout: str + stderr: str + elapsed_sec: float + timed_out: bool = False + returncode: int = 0 + error: Optional[str] = None + tokens_in: int = 0 + tokens_out: int = 0 + cost_usd: float = 0.0 + + +@dataclass(frozen=True) +class ExperimentResult: + run_id: str + iteration: int + code: str + metrics: dict[str, object] + primary_metric: float | None + improved: bool + kept: bool + elapsed_sec: float + stdout: str + stderr: str + error: str | None = None + tokens_in: int = 0 + tokens_out: int = 0 + cost_usd: float = 0.0 + + +@dataclass +class ExperimentHistory: + results: list[ExperimentResult] = field(default_factory=list) + best_result: ExperimentResult | None = None + baseline_metric: float | None = None + + def add(self, result: ExperimentResult) -> None: + self.results.append(result) + if self.baseline_metric is None and result.primary_metric is not None: + self.baseline_metric = result.primary_metric + + def to_dict(self) -> dict[str, object]: + return { + "results": [asdict(result) for result in self.results], + "best_result": asdict(self.best_result) if self.best_result else None, + "baseline_metric": self.baseline_metric, + } + + @classmethod + def from_dict(cls, data: dict[str, object]) -> ExperimentHistory: + results: list[ExperimentResult] = [] + raw_results = data.get("results") + if isinstance(raw_results, list): + for item in cast(list[object], raw_results): + if isinstance(item, dict): + item_map = cast(dict[object, object], item) + normalized_item: dict[str, object] = {} + for key, value in item_map.items(): + normalized_item[str(key)] = value + parsed = _result_from_dict(normalized_item) + if parsed is not None: + results.append(parsed) + best_raw = data.get("best_result") + best_result = ( + _result_from_dict( + { + str(key): value + for key, value in cast(dict[object, object], best_raw).items() + } + ) + if isinstance(best_raw, dict) + else None + ) + baseline_metric_raw = data.get("baseline_metric") + baseline_metric = ( + float(baseline_metric_raw) + if isinstance(baseline_metric_raw, (int, float)) + else None + ) + return cls( + results=results, best_result=best_result, baseline_metric=baseline_metric + ) + + +class _ChatResponse(Protocol): + content: str + + +class _ChatClient(Protocol): + def chat( + self, messages: list[dict[str, str]], *, system: str | None = None + ) -> _ChatResponse: ... + + +class ExperimentRunner: + """Karpathy inner loop: baseline → iterate → improve/discard, wired to Hermes delegate_task. + + Args: + config: HermesExperimentConfig with metric_key, direction, budget, iterations. + workspace: Directory for round artefacts. + delegate_fn: Callable(goal: str, working_dir: str) -> DelegateSandboxResult. + Wraps delegate_task; caller is responsible for spawning the worker. + progress_sink: Optional ProgressSink. The runner only uses + ``progress_sink.comment(msg)`` for the round summary posts that + previously went through ``lattice_comment_fn``. Defaults to a + log-only StubSink when omitted. + """ + + def __init__( + self, + config: "HermesExperimentConfig", + workspace: Path, + *, + delegate_fn: Callable[[str, str], "DelegateSandboxResult"], + progress_sink: Optional[Any] = None, + ) -> None: + self.config: HermesExperimentConfig = config + self.workspace: Path = workspace + self.workspace.mkdir(parents=True, exist_ok=True) + self._delegate_fn = delegate_fn + if progress_sink is None: + from agent.research.sinks import StubSink + self._sink = StubSink() + else: + self._sink = progress_sink + # Backward-compat alias: existing internal call sites use + # self._lattice_comment. Point it at the sink's comment method so + # nothing inside the class needs to change. + self._lattice_comment = self._sink.comment + self.history: ExperimentHistory = ExperimentHistory() + + def run_experiment( + self, code: str, *, run_id: str, iteration: int = 0 + ) -> ExperimentResult: + """Run one experiment round via delegate_task and score the result.""" + t0 = _time.monotonic() + round_dir = str(self.workspace / f"round-{run_id}-iter{iteration}") + goal = ( + f"You are a Hermes research worker. Read program.md in {round_dir} " + f"and run the experiment. Report your result as:\n" + f"METRIC: {self.config.metric_key}= STATUS: improved|regressed|neutral " + f"NOTES: " + ) + + try: + sandbox_result = self._delegate_fn(goal, round_dir) + except Exception as exc: + elapsed = _time.monotonic() - t0 + logger.exception("delegate_fn failed for %s iter %d: %s", run_id, iteration, exc) + sandbox_result = DelegateSandboxResult( + metrics={}, stdout="", stderr=str(exc), elapsed_sec=elapsed, + timed_out=False, returncode=1, error=str(exc), + ) + + primary_metric = self._to_float( + sandbox_result.metrics.get(self.config.metric_key) + ) + current_best = ( + self.history.best_result.primary_metric + if self.history.best_result + else None + ) + + improved = False + kept = False + + if primary_metric is not None: + if current_best is None: + improved = True + kept = True + elif self._is_improvement(primary_metric, current_best): + improved = True + kept = abs(primary_metric - current_best) > self.config.keep_threshold + + error: str | None = sandbox_result.error + if not error and sandbox_result.timed_out: + error = f"Timed out after {self.config.time_budget_sec}s" + elif not error and sandbox_result.returncode != 0: + error = sandbox_result.stderr.strip() or f"Process exited with {sandbox_result.returncode}" + + result = ExperimentResult( + run_id=run_id, + iteration=iteration, + code=code, + metrics=sandbox_result.metrics, + primary_metric=primary_metric, + improved=improved, + kept=kept, + elapsed_sec=sandbox_result.elapsed_sec, + stdout=sandbox_result.stdout, + stderr=sandbox_result.stderr, + error=error, + tokens_in=sandbox_result.tokens_in, + tokens_out=sandbox_result.tokens_out, + cost_usd=sandbox_result.cost_usd, + ) + + if kept: + self.history.best_result = result + + self.history.add(result) + + # Post Lattice comment summarising this round + status_word = "KEPT" if kept else ("IMPROVED" if improved else "DISCARDED") + self._lattice_comment( + f"Round {run_id} iter {iteration}: {status_word} " + f"{self.config.metric_key}={primary_metric} " + f"(best={current_best})" + ) + return result + + def run_loop( + self, initial_code: str, *, run_id: str, llm: "_ChatClient | None" = None + ) -> ExperimentHistory: + """Karpathy inner loop: baseline → iterate → keep/discard.""" + self._lattice_comment(f"Research loop started: run_id={run_id}") + current_code = initial_code + baseline = self.run_experiment(current_code, run_id=run_id, iteration=0) + + if llm is None: + return self.history + + no_improvement_count = 0 + for iteration in range(1, self.config.max_iterations + 1): + next_code = self._improve_code(llm, current_code, self.history) + result = self.run_experiment(next_code, run_id=run_id, iteration=iteration) + current_code = next_code + + if result.improved: + no_improvement_count = 0 + else: + no_improvement_count += 1 + + if no_improvement_count >= 3: + logger.info("Stopping early: 3 non-improving iterations for %s", run_id) + self._lattice_comment(f"Early stop after {iteration} iterations (3 non-improving)") + break + + self._lattice_comment( + f"Research loop done: {len(self.history.results)} rounds, " + f"best={self.history.best_result.primary_metric if self.history.best_result else None}" + ) + return self.history + + def _improve_code( + self, llm: _ChatClient, current_code: str, history: ExperimentHistory + ) -> str: + direction = self.config.metric_direction + last_result = history.results[-1] if history.results else None + last_metrics = last_result.metrics if last_result else {} + best_metrics = history.best_result.metrics if history.best_result else {} + last_metric = last_result.primary_metric if last_result else None + best_metric = ( + history.best_result.primary_metric if history.best_result else None + ) + + prompt = ( + "Improve the experiment code to optimize the primary metric.\n\n" + f"Metric key: {self.config.metric_key}\n" + f"Direction: {direction}\n" + f"Last primary metric: {last_metric}\n" + f"Best primary metric: {best_metric}\n" + f"Last metrics JSON: {json.dumps(last_metrics, ensure_ascii=True)}\n" + f"Best metrics JSON: {json.dumps(best_metrics, ensure_ascii=True)}\n\n" + "Current code:\n" + "```python\n" + f"{current_code}\n" + "```\n\n" + "## Think Before Coding\n\n" + "Before writing any code:\n" + "1. State WHY the current metric is at the level it is " + "(what is the binding bottleneck?).\n" + "2. State your ONE hypothesis for what change will move the metric. " + "If uncertain between approaches, pick the simpler one.\n" + "3. Define your success criterion: " + f"'{self.config.metric_key} should move from {last_metric} toward " + f"{'higher' if direction == 'maximize' else 'lower'} by a measurable amount'.\n\n" + "## Surgical Changes\n\n" + "- Change only what your hypothesis requires. " + "Do not refactor unrelated code.\n" + "- Every changed line must trace directly to your hypothesis.\n" + "- If the fix is 5 lines, write 5 lines — not 50.\n" + "- Prefer the 50-line solution over the 200-line solution.\n\n" + "Return ONLY the updated Python code. " + "Do not include explanation outside the code." + ) + + try: + response = llm.chat( + [{"role": "user", "content": prompt}], + system=( + "You are an expert ML experimentation assistant. " + "Think carefully before writing code. " + "Make the minimum change needed to improve the metric. " + "Surface your reasoning as a comment at the top of the changed section." + ), + ) + except Exception as exc: # noqa: BLE001 + logger.exception("Code improvement call failed: %s", exc) + return current_code + + candidate = getattr(response, "content", "") + if not isinstance(candidate, str) or not candidate.strip(): + logger.warning("LLM returned empty code; keeping current version") + return current_code + + extracted = self._extract_python_code(candidate) + return extracted if extracted.strip() else current_code + + def save_history(self, path: Path) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + _ = path.write_text( + json.dumps(self.history.to_dict(), indent=2), encoding="utf-8" + ) + + def _is_improvement(self, new_value: float, best_value: float) -> bool: + if self.config.metric_direction == "maximize": + return new_value > best_value + return new_value < best_value + + @staticmethod + def _to_float(value: object) -> float | None: + if isinstance(value, bool): + return None + if isinstance(value, (int, float)): + return float(value) + if isinstance(value, str): + try: + return float(value) + except ValueError: + return None + return None + + @staticmethod + def _extract_python_code(content: str) -> str: + match = re.search(r"```(?:python)?\s*(.*?)\s*```", content, flags=re.DOTALL) + if match is None: + return content.strip() + return match.group(1).strip() + + +def _result_from_dict(data: dict[str, object]) -> ExperimentResult | None: + run_id = data.get("run_id") + iteration = data.get("iteration") + code = data.get("code") + metrics = data.get("metrics") + primary_metric = data.get("primary_metric") + improved = data.get("improved") + kept = data.get("kept") + elapsed_sec = data.get("elapsed_sec") + stdout = data.get("stdout") + stderr = data.get("stderr") + error = data.get("error") + + if not isinstance(run_id, str) or not isinstance(iteration, int): + return None + if not isinstance(code, str) or not isinstance(metrics, dict): + return None + if primary_metric is not None and not isinstance(primary_metric, (int, float)): + return None + if not isinstance(improved, bool) or not isinstance(kept, bool): + return None + if not isinstance(elapsed_sec, (int, float)): + return None + if not isinstance(stdout, str) or not isinstance(stderr, str): + return None + if error is not None and not isinstance(error, str): + return None + + typed_metrics: dict[str, object] = {} + for key, value in cast(dict[object, object], metrics).items(): + typed_metrics[str(key)] = value + + _tokens_in = data.get("tokens_in", 0) + _tokens_out = data.get("tokens_out", 0) + _cost_usd = data.get("cost_usd", 0.0) + return ExperimentResult( + run_id=run_id, + iteration=iteration, + code=code, + metrics=typed_metrics, + primary_metric=float(primary_metric) + if isinstance(primary_metric, (int, float)) + else None, + improved=improved, + kept=kept, + elapsed_sec=float(elapsed_sec), + stdout=stdout, + stderr=stderr, + error=error, + tokens_in=int(_tokens_in) if isinstance(_tokens_in, (int, float)) else 0, + tokens_out=int(_tokens_out) if isinstance(_tokens_out, (int, float)) else 0, + cost_usd=float(_cost_usd) if isinstance(_cost_usd, (int, float)) else 0.0, + ) diff --git a/agent/research/sinks.py b/agent/research/sinks.py new file mode 100644 index 000000000000..05f1602d85eb --- /dev/null +++ b/agent/research/sinks.py @@ -0,0 +1,207 @@ +"""Progress sinks for the autoresearch loop. + +A ``ProgressSink`` is the seam through which ``ResearchSupervisor`` and +``ExperimentRunner`` report run progress to an external tracker. The +supervisor consumes a ``ProgressSink`` only — it does not know about +lattice, kanban, or any other backend. + +Built-in implementations: + +* :class:`StubSink` — log-only (default when no tracker is wired). +* :class:`KanbanSink` — appends comments to an EXISTING kanban task and + transitions status on completion. The caller is responsible for + creating the task; the sink does not auto-create. + +Sinks must never raise: a misbehaving tracker must not break the loop. +""" +from __future__ import annotations + +import logging +from pathlib import Path +from typing import Any, Protocol + +logger = logging.getLogger(__name__) + + +class ProgressSink(Protocol): + """The contract implemented by every progress sink. + + Hooks are invoked by ResearchSupervisor / ExperimentRunner. All hooks + must be best-effort and never raise — failures are swallowed and logged. + """ + + def run_started(self, spec: Any, run_id: str) -> None: + """Called once at the start of a run, before iteration 0.""" + ... + + def iteration_observed( + self, iteration: int, result: Any, run_dir: Path + ) -> None: + """Called after _observe for each completed iteration.""" + ... + + def run_completed(self, history: Any) -> None: + """Called once at the end of a run with the final ExperimentHistory.""" + ... + + def comment(self, message: str) -> None: + """Free-form progress comment. Used by call sites that already only + emit text and don't have a structured event.""" + ... + + +class StubSink: + """Log-only sink. The default when no tracker is configured. + + Every hook drops a ``logger.info`` line at "[sink-stub]". Never raises. + """ + + def run_started(self, spec: Any, run_id: str) -> None: + topic = getattr(spec, "topic", "")[:60] + logger.info("[sink-stub] run_started run_id=%s topic=%s", run_id, topic) + + def iteration_observed( + self, iteration: int, result: Any, run_dir: Path + ) -> None: + metric = getattr(result, "primary_metric", None) + improved = getattr(result, "improved", False) + logger.info( + "[sink-stub] iter=%d metric=%s improved=%s", + iteration, metric, improved, + ) + + def run_completed(self, history: Any) -> None: + results = getattr(history, "results", []) or [] + best = getattr(history, "best_result", None) + best_metric = getattr(best, "primary_metric", None) if best else None + logger.info( + "[sink-stub] run_completed iters=%d best=%s", + len(results), best_metric, + ) + + def comment(self, message: str) -> None: + logger.info("[sink-stub] %s", message) + + +from typing import Optional + + +class KanbanSink: + """Posts run progress to an EXISTING kanban task. + + On ``run_started`` and ``iteration_observed`` it appends a comment + to the configured task. On ``run_completed`` it (optionally) + transitions the task to ``done``. When ``task_id`` is None, every + hook is log-only and no DB connection is opened. + + Connection lifecycle: the sink stores ``db_path`` (captured at + construction so we don't re-resolve "current board" on every call) + and opens a fresh short-lived sqlite3.Connection per write inside a + try/finally. This avoids sqlite3 thread-affinity issues if the loop + fans out across threads, and lets the dispatcher hold its own + long-lived connection without contending. WAL mode keeps reads + non-blocking and ``write_txn()`` (BEGIN IMMEDIATE) inside + ``add_comment`` / ``complete_task`` keeps writes serialized. + + ``complete_on_run_completed`` controls whether ``run_completed`` calls + ``complete_task``. A/B testing constructs per-strategy sub-sinks with + ``complete_on_run_completed=False`` so the task stays open until the + tester layer closes it once at the end. + """ + + _ACTOR = "agent:research-supervisor" + + def __init__( + self, + *, + task_id: Optional[str], + db_path: Optional[Path] = None, + complete_on_run_completed: bool = True, + ): + self._task_id = task_id + self._db_path = db_path # captured at construction; do not re-resolve + self._complete_on_run_completed = complete_on_run_completed + + def _open(self) -> Optional[Any]: + """Open a fresh short-lived connection. Returns None when no task_id.""" + if not self._task_id: + return None + try: + from hermes_cli import kanban_db + # If db_path is None, fall through to kanban_db_path()'s env / + # current-board resolution. Caller should usually pin db_path. + path = self._db_path or kanban_db.kanban_db_path() + return kanban_db.connect(path) + except Exception as exc: + logger.warning("[kanban-sink] connect failed: %s", exc) + return None + + def _comment(self, message: str) -> None: + if not self._task_id: + logger.info("[kanban-stub] %s", message) + return + conn = self._open() + if conn is None: + return + try: + from hermes_cli import kanban_db + kanban_db.add_comment( + conn, self._task_id, self._ACTOR, message, + ) + except Exception as exc: + logger.warning("[kanban-sink] add_comment failed: %s", exc) + finally: + try: + conn.close() + except Exception: + pass + + def run_started(self, spec: Any, run_id: str) -> None: + topic = getattr(spec, "topic", "")[:80] + task_type = getattr(spec, "task_type", "?") + metric = getattr(spec, "metric_key", "?") + self._comment( + f"Loop started: run_id={run_id} type={task_type} " + f"metric={metric}\nTopic: {topic}" + ) + + def iteration_observed( + self, iteration: int, result: Any, run_dir: Path + ) -> None: + metric = getattr(result, "primary_metric", None) + improved = getattr(result, "improved", False) + kept = getattr(result, "kept", False) + status = "KEPT" if kept else ("IMPROVED" if improved else "DISCARDED") + self._comment( + f"Iteration {iteration}: {status} metric={metric}" + ) + + def run_completed(self, history: Any) -> None: + results = getattr(history, "results", []) or [] + best = getattr(history, "best_result", None) + best_metric = getattr(best, "primary_metric", None) if best else None + self._comment( + f"Loop done: {len(results)} rounds, best={best_metric}" + ) + if not self._task_id or not self._complete_on_run_completed: + return + conn = self._open() + if conn is None: + return + try: + from hermes_cli import kanban_db + kanban_db.complete_task( + conn, self._task_id, + result=str(best_metric) if best_metric is not None else None, + summary=f"{len(results)} rounds, best={best_metric}", + ) + except Exception as exc: + logger.warning("[kanban-sink] complete_task failed: %s", exc) + finally: + try: + conn.close() + except Exception: + pass + + def comment(self, message: str) -> None: + self._comment(message) diff --git a/agent/research/supervisor.py b/agent/research/supervisor.py new file mode 100644 index 000000000000..d03e9b1e4c79 --- /dev/null +++ b/agent/research/supervisor.py @@ -0,0 +1,1972 @@ +"""ResearchSupervisor — Karpathy inner loop for any task with a measurable deliverable. + +Implements the Autogenesis self-evolution loop (Act → Observe → Optimize → Remember) +applied to any task with a measurable deliverable: + + Phase | Autogenesis concept | Implementation + --------- | -------------------- | -------------- + ACT | Agent produces output | worker via delegate_task + OBSERVE | Capture outcome + traces | _observe() → learnings.jsonl + OPTIMIZE | Propose next hypothesis | _improve_attempt() (reflection optimizer) + REMEMBER | Persist insights for future rounds | learnings.jsonl (HeartbeatMemorySystem schema) + +The SEPL (Self Evolution Protocol Layer) materializes as: + - propose: _improve_attempt() drafts the next attempt + - evaluate: ExperimentRunner scores and keep/discards + - commit: kept results update best_result + lineage in ExperimentHistory + - rollback: discarded results revert attempt_holder to prior best + +Supported task types: "code" | "search" | "research" | "generic" +Evaluation modes: "self_report" | "llm_judge" +""" + +from __future__ import annotations + +import json +import logging +import os +import re +import time as _time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Optional, TYPE_CHECKING + +if TYPE_CHECKING: + from agent.research.sinks import ProgressSink + +from hermes_constants import get_hermes_home + +from agent.research.runner import ( + DelegateSandboxResult, + ExperimentHistory, + ExperimentResult, + ExperimentRunner, + HermesExperimentConfig, +) +from agent.research.metrics import UniversalMetricParser +from agent.research.events import ResearchEvent, emit_event + +logger = logging.getLogger(__name__) + +# Matches the first decimal in a judge response — tolerates prefixes like +# "Score:" or suffixes like "/1.0" that the older tokens[0] parser choked on. +_JUDGE_SCORE_RE = re.compile(r"-?\d+(?:\.\d+)?") + +_parser = UniversalMetricParser() + + +# --------------------------------------------------------------------------- +# TaskSpec — the central abstraction for any measurable task +# --------------------------------------------------------------------------- + +@dataclass +class TaskSpec: + """Describes any task with a measurable deliverable. + + Examples: + # Code task — metric from test pass rate + TaskSpec( + topic="Implement a binary search tree", + deliverable="Python class with insert/search/delete, measured by test pass rate", + metric_key="pass_rate", + task_type="code", + ) + + # Search task — metric from result relevance + TaskSpec( + topic="Find papers on attention mechanisms published after 2022", + deliverable="Ranked list of relevant papers with abstracts", + metric_key="relevance_score", + task_type="search", + evaluation_mode="llm_judge", + evaluation_prompt="Score 0-1: does this paper list cover attention mechanisms published after 2022?", + ) + + # Research task — metric from synthesis quality + TaskSpec( + topic="Summarize the state of diffusion models for video generation", + deliverable="Technical synthesis covering key methods, benchmarks, and open problems", + metric_key="completeness_score", + task_type="research", + evaluation_mode="llm_judge", + evaluation_prompt="Score 0-1: does this synthesis cover key methods, benchmarks, and open problems?", + ) + + # Generic task — anything with a self-reported numeric metric + TaskSpec( + topic="Optimize hermes session search latency", + deliverable="Modified session search implementation with measured latency in ms", + metric_key="latency_ms", + metric_direction="minimize", + task_type="generic", + ) + """ + + topic: str + deliverable: str # what the worker must produce + metric_key: str # how success is measured + metric_direction: str = "maximize" # "maximize" or "minimize" + task_type: str = "generic" # "code" | "search" | "research" | "generic" + acceptance_criterion: str = "" # e.g. "pass_rate >= 0.95" or qualitative + evaluation_mode: str = "self_report" # "self_report" | "llm_judge" + evaluation_prompt: str = "" # for llm_judge: how to score the deliverable + hypothesis: str = "" # current iteration hypothesis (updated by supervisor) + + # Worker toolset hints per task type (overridable in ResearchSupervisor.run) + _DEFAULT_TOOLSETS: dict[str, list[str]] = field(default_factory=lambda: { + "code": ["terminal", "file"], + "search": ["web", "terminal", "file"], + "research": ["web", "terminal", "file"], + "generic": ["terminal", "file"], + }, repr=False) + + def default_toolsets(self) -> list[str]: + return self._DEFAULT_TOOLSETS.get(self.task_type, ["terminal", "file"]) + + +# --------------------------------------------------------------------------- +# Task brief templates — one per task_type +# --------------------------------------------------------------------------- + +def _build_task_brief(spec: TaskSpec, *, iteration: int, round_dir: str, time_budget_sec: int) -> str: + """Generate the task brief for the worker. Domain-aware but structurally identical.""" + builders = { + "code": _brief_code, + "search": _brief_search, + "research": _brief_research, + } + builder = builders.get(spec.task_type, _brief_generic) + return builder(spec, iteration=iteration, round_dir=round_dir, time_budget_sec=time_budget_sec) + + +def _think_block(spec: TaskSpec, iteration: int) -> str: + action = "improve" if iteration > 0 else "establish a baseline for" + return f"""\ +## Step 0 — Think Before Acting (Karpathy Principle 1) + +> Consult the `karpathy-guidelines` skill (`skills/autoresearch/karpathy-guidelines/SKILL.md`) +> for the full set of rules — surgical edits, surface assumptions, no overcomplication. + +Before producing anything, state in your output: + +1. **Assumption**: What do you understand the task to be asking for? +2. **Bottleneck** *(iteration {iteration} > 0 only)*: Why is `{spec.metric_key}` at its current value? + What is the binding constraint? +3. **Hypothesis**: What ONE change will {action} `{spec.metric_key}`? + If uncertain between approaches, pick the simpler one. +4. **Success criterion**: "`{spec.metric_key}` moves from X toward + {'higher' if spec.metric_direction == 'maximize' else 'lower'}" + +If something is unclear, name what is confusing in your NOTES. Do NOT guess silently. +""" + + +def _report_block(metric_key: str) -> str: + return f"""\ +## Final Report (required) + +Your last line of output must be: + +``` +METRIC: {metric_key}= STATUS: improved|regressed|neutral NOTES: +``` + +- Value must be a real number you measured or computed — never fabricated. +- NOTES must say what you did and what the key result was. +- Also write `results.json` with `{{"{metric_key}": }}` for structured parsing. +""" + + +def _brief_code(spec: TaskSpec, *, iteration: int, round_dir: str, time_budget_sec: int) -> str: + action = "Improve" if iteration > 0 else "Establish a baseline for" + return f"""\ +# Task Brief — Code ({action}) + +## Topic +{spec.topic} + +## Deliverable +{spec.deliverable} + +{_think_block(spec, iteration)} +## Step 1 — Implement + +The current attempt is in `attempt.py` in: `{round_dir}` + +{"Do not rewrite unless you have a specific, hypothesis-driven change. Make surgical edits only — every changed line must trace to your hypothesis." if iteration > 0 else "Implement the deliverable in `attempt.py`. Run it to verify."} + +{f"Time budget: {time_budget_sec}s. Print `TIME_ESTIMATE: Xs` before your main loop." if time_budget_sec > 0 else "Time budget: unlimited. Work until converged."} +{"Stop before 80% of budget and save partial results." if time_budget_sec > 0 else ""} + +## Step 2 — Measure + +Compute `{spec.metric_key}` from the code's output. +{"Acceptance criterion: " + spec.acceptance_criterion if spec.acceptance_criterion else ""} + +{_report_block(spec.metric_key)} +## Rules +- Do NOT fabricate metric values. +- No abstractions for single-use code. If 5 lines solve it, write 5. +- Do NOT refactor code unrelated to your hypothesis. +- **Package installation:** `pip install` is NOT blocked but may fail if the package isn't available. If you need a library, first check if it's already installed. If not, use `ctypes.CDLL` with system libraries (e.g., `/usr/lib/x86_64-linux-gnu/libgmp.so.10`) or write a pure-Python alternative. +- **You MAY use `python -c` and heredoc scripts** — these are allowed in your environment. +- **Tool format:** When calling tools, use the JSON format provided by the system. Do NOT use XML tags like ``. + +## Tools Available + +You have access to: `terminal` (shell commands), `file` (read/write), `code_execution` (Python scripts), and `search` (web search). +If a task requires running code, use `terminal()` or `code_execution()` — do NOT assume they are unavailable. +""" + + +def _brief_search(spec: TaskSpec, *, iteration: int, round_dir: str, time_budget_sec: int) -> str: + action = "Refine" if iteration > 0 else "Execute" + return f"""\ +# Task Brief — Search ({action}) + +## Topic +{spec.topic} + +## Deliverable +{spec.deliverable} + +{_think_block(spec, iteration)} +## Step 1 — Search + +{"The previous search strategy is in `attempt.md` in: " + round_dir + ". Revise it based on your hypothesis." if iteration > 0 else "Design and execute a search strategy. Save results to `attempt.md`."} + +{f"Time budget: {time_budget_sec}s. Do not make redundant searches — each query must have a hypothesis." if time_budget_sec > 0 else "Time budget: unlimited. Work until converged."} + +## Step 2 — Evaluate Results + +Score your results for `{spec.metric_key}` on a 0.0–1.0 scale. +{"Evaluate against: " + spec.evaluation_prompt if spec.evaluation_prompt and spec.evaluation_mode == "self_report" else ""} +{"Acceptance criterion: " + spec.acceptance_criterion if spec.acceptance_criterion else ""} + +{_report_block(spec.metric_key)} +## Rules +- Do NOT fabricate relevance scores. +- Each search iteration must test exactly one new hypothesis about where better results are. +- Save your full result set to `results.json` with `{{"{spec.metric_key}": }}`. +- **Tool format:** When calling tools, use the JSON format provided by the system. Do NOT use XML tags like ``. + +## Tools Available + +You have access to: `web_search` (find papers/articles), `browser` (visit pages), `file` (read/write), and `terminal` (shell commands for data processing). +Use these actively — do NOT assume they are unavailable. +""" + + +def _brief_research(spec: TaskSpec, *, iteration: int, round_dir: str, time_budget_sec: int) -> str: + action = "Deepen" if iteration > 0 else "Produce an initial" + return f"""\ +# Task Brief — Research ({action}) + +## Topic +{spec.topic} + +## Deliverable +{spec.deliverable} + +{_think_block(spec, iteration)} +## Step 1 — Investigate and Synthesize + +{"The current draft is in `attempt.md` in: " + round_dir + ". Identify its weakest section and address it." if iteration > 0 else "Research the topic. Produce an initial synthesis in `attempt.md`."} + +{f"Time budget: {time_budget_sec}s. Focus — do not survey everything; go deep on what your hypothesis identifies as the gap." if time_budget_sec > 0 else "Time budget: unlimited. Work until converged."} + +## Step 2 — Self-Evaluate + +Rate your synthesis on `{spec.metric_key}` (0.0–1.0). +{"Evaluate against: " + spec.evaluation_prompt if spec.evaluation_prompt and spec.evaluation_mode == "self_report" else ""} +{"Acceptance criterion: " + spec.acceptance_criterion if spec.acceptance_criterion else ""} + +{_report_block(spec.metric_key)} +## Rules +- Do NOT fabricate facts, citations, or scores. +- Each iteration must address exactly ONE identified gap — not rewrite everything. +- Save synthesis to `attempt.md` and score to `results.json`. +- **Tool format:** When calling tools, use the JSON format provided by the system. Do NOT use XML tags like ``. + +## Tools Available + +You have access to: `web_search` (research topics), `browser` (deep reading), `file` (read/write), and `terminal` (data processing). +Use these actively — do NOT assume they are unavailable. +""" + + +def _brief_generic(spec: TaskSpec, *, iteration: int, round_dir: str, time_budget_sec: int) -> str: + action = "Improve" if iteration > 0 else "Produce a baseline" + return f"""\ +# Task Brief — {action} + +## Topic +{spec.topic} + +## Deliverable +{spec.deliverable} + +{_think_block(spec, iteration)} +## Step 1 — Produce the Deliverable + +{"The previous attempt is in `attempt.md` in: " + round_dir + ". Revise it based on your hypothesis." if iteration > 0 else "Produce the deliverable. Save it to `attempt.md`."} + +{f"Time budget: {time_budget_sec}s." if time_budget_sec > 0 else "Time budget: unlimited. Work until converged."} + +## Step 2 — Measure + +Compute `{spec.metric_key}` as a number from your deliverable. +{"Acceptance criterion: " + spec.acceptance_criterion if spec.acceptance_criterion else ""} + +{_report_block(spec.metric_key)} +## Rules +- Do NOT fabricate metric values. +- Minimum effort that moves the metric. No speculative additions. +- Save deliverable to `attempt.md`, score to `results.json`. +- **Tool format:** When calling tools, use the JSON format provided by the system. Do NOT use XML tags like ``. + +## Tools Available + +You have access to: `terminal` (shell), `file` (read/write), `code_execution` (Python), `web_search`, and `browser`. +Use these actively — do NOT assume they are unavailable. +""" + + +# --------------------------------------------------------------------------- +# Attempt file name per task type +# --------------------------------------------------------------------------- + +_ATTEMPT_FILENAME: dict[str, str] = { + "code": "attempt.py", + "search": "attempt.md", + "research": "attempt.md", + "generic": "attempt.md", +} + + +# --------------------------------------------------------------------------- +# Acceptance criterion parser +# --------------------------------------------------------------------------- + +import operator as _operator + +_ACCEPTANCE_OPS = { + ">=": _operator.ge, + "<=": _operator.le, + ">": _operator.gt, + "<": _operator.lt, + "==": _operator.eq, +} + +_ACCEPTANCE_RE = re.compile( + r"^\s*(?:[\w.]+\s*)?(>=|<=|>|<|==)\s*([-+]?\d+(?:\.\d+)?)\s*$" +) + + +def _parse_acceptance_criterion(criterion: str) -> Optional[Callable[[float], bool]]: + """Parse a textual acceptance criterion into a predicate over the metric. + + Accepts forms like ``"pass_rate >= 0.9"``, ``">= 0.9"``, ``"latency_ms < 200"``. + The metric-key prefix is optional and is not validated against the spec — + callers already know which metric they're testing. Returns ``None`` if the + criterion can't be parsed (e.g. qualitative text), so the loop falls back + to the original max_iterations / time_budget termination. + """ + if not criterion: + return None + m = _ACCEPTANCE_RE.match(criterion) + if not m: + return None + op = _ACCEPTANCE_OPS[m.group(1)] + threshold = float(m.group(2)) + return lambda value: op(value, threshold) + + +# --------------------------------------------------------------------------- +# delegate_task bridge +# --------------------------------------------------------------------------- + +def _call_delegate_task( + goal: str, + context: str, + *, + parent_agent: Any, + toolsets: list[str] | None = None, +) -> dict[str, Any]: + from tools.delegate_tool import delegate_task + raw = delegate_task( + goal=goal, + context=context, + toolsets=toolsets or ["terminal", "file"], + parent_agent=parent_agent, + inherit_profile=True, + ) + try: + return json.loads(raw) + except (json.JSONDecodeError, TypeError): + return {"results": [{"status": "failed", "summary": raw or "", "error": "JSON parse failed"}]} + + +def _call_delegate_task_batch( + tasks: list[dict[str, Any]], + *, + parent_agent: Any, + toolsets: list[str] | None = None, +) -> dict[str, Any]: + """Batch variant of _call_delegate_task using delegate_task's tasks array. + + Each task dict must contain at least 'goal' and 'context'. Returns the + parsed JSON result with a 'results' array, one entry per task. + """ + from tools.delegate_tool import delegate_task + raw = delegate_task( + tasks=tasks, + toolsets=toolsets or ["terminal", "file"], + parent_agent=parent_agent, + inherit_profile=True, + ) + try: + return json.loads(raw) + except (json.JSONDecodeError, TypeError): + return {"results": [{"status": "failed", "summary": raw or "", "error": "JSON parse failed"}]} + + +# --------------------------------------------------------------------------- +# Durable checkpoint + snapshot helpers (HRM-93, HRM-96) +# --------------------------------------------------------------------------- + +def _atomic_write_text(path: Path, text: str) -> None: + """Write text to path atomically: write to a sibling .tmp then rename.""" + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + ".tmp") + tmp.write_text(text, encoding="utf-8") + os.replace(tmp, path) + + +def _load_checkpoint( + checkpoint_dir: Path, +) -> tuple[ExperimentHistory, int] | None: + """Read checkpoint.json + history.json from checkpoint_dir. + + Returns (history, current_iteration) where current_iteration is the + last-completed round number (0 means baseline done; N means iteration N + done — the next iteration to run is N+1). Returns None if either file is + missing or parsing fails. + + Cross-file consistency (peer-review Fix-2): a crash between writing + history.json and checkpoint.json (or between writing snapshots and + checkpoint.json) leaves these three artifacts out of sync. Resuming + from inconsistent state produces silent data corruption — e.g. + skipping a round that was never actually run. We reject any of: + + * checkpoint["round"] != len(history.results) - 1 + (the latest history entry must be the round the checkpoint + claims is complete; round N done ⇒ history has N+1 results, + indexed 0..N). + * snapshots/iter-{N}.json missing for the claimed round. + + On mismatch we log a warning and return None — the loop falls back + to a clean restart, which is safer than resuming from a corrupted + state. + """ + cp_path = checkpoint_dir / "checkpoint.json" + hist_path = checkpoint_dir / "history.json" + if not cp_path.exists() or not hist_path.exists(): + return None + try: + cp = json.loads(cp_path.read_text(encoding="utf-8")) + hist_data = json.loads(hist_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + logger.warning("Checkpoint load failed at %s: %s", checkpoint_dir, exc) + return None + + history = ExperimentHistory.from_dict(hist_data) + round_value = cp.get("round") + if not isinstance(round_value, int): + return None + + expected_results = round_value + 1 + if len(history.results) != expected_results: + logger.warning( + "Checkpoint inconsistent at %s: round=%d implies %d results " + "but history.json has %d. Falling back to full restart.", + checkpoint_dir, round_value, expected_results, len(history.results), + ) + return None + + snapshot_path = checkpoint_dir / "snapshots" / f"iter-{round_value}.json" + if not snapshot_path.exists(): + logger.warning( + "Checkpoint inconsistent at %s: round=%d but %s is missing. " + "Falling back to full restart.", + checkpoint_dir, round_value, snapshot_path.name, + ) + return None + + return history, round_value + + +def _detect_resume(checkpoint_dir: Path) -> dict[str, Any] | None: + """Lightweight resume probe used by job_runner / status tools. + + Returns a small dict surfacing checkpoint metadata (round, total + rounds, best metric) or None when no usable checkpoint exists. This + is intentionally weaker than ``_load_checkpoint``: it only reads + checkpoint.json and does not enforce cross-file consistency, so + operators can see *something happened* even when history.json is + corrupt or missing. The resume path itself goes through + _load_checkpoint, which does enforce consistency. + """ + cp_path = Path(checkpoint_dir) / "checkpoint.json" + if not cp_path.exists(): + return None + try: + cp = json.loads(cp_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(cp, dict) or not isinstance(cp.get("round"), int): + return None + return { + "resumed_from_round": cp["round"], + "resumed_total_rounds": cp.get("total_rounds"), + "resumed_best_metric": cp.get("best_metric"), + } + + +def restore_snapshot(snapshot_path: Path, target_dir: Path) -> None: + """Rewrite every captured file from a snapshot into target_dir. + + Snapshot schema (see ResearchSupervisor._snapshot): + {"iteration": int, "messages": [...], "metrics": {...}, + "files": [{"path": "", "content": ""}, ...]} + + Raises ValueError if any captured path would escape target_dir, either + via ``..`` components or via symlinks anywhere in the parent chain. + Path validation happens in two layers: + + 1. Canonical-path containment: the resolved destination must be a + descendant of the resolved target_dir. This catches ``..`` and + absolute-path entries. + 2. No symlink in the parent chain: every existing path component + from target_dir up to (but not including) the destination is + checked with ``os.path.islink``. If anything in the chain is a + symlink we refuse to write through it, even if it currently + resolves inside target_dir — symlinks are an attacker-controlled + redirection point. + """ + data = json.loads(Path(snapshot_path).read_text(encoding="utf-8")) + target_dir = Path(target_dir) + target_dir.mkdir(parents=True, exist_ok=True) + target_resolved = target_dir.resolve() + + # Reject if target_dir itself is a symlink — we'd be writing outside + # the directory the caller named. + if os.path.islink(target_dir): + raise ValueError( + f"target_dir is a symlink, refusing to restore: {target_dir!r}" + ) + + for entry in data.get("files", []): + rel = entry.get("path") + content = entry.get("content", "") + if not isinstance(rel, str) or not isinstance(content, str): + continue + + # Layer 1: canonical containment. + dest = (target_dir / rel).resolve() + try: + dest.relative_to(target_resolved) + except ValueError: + raise ValueError( + f"snapshot path escapes target_dir: {rel!r}" + ) from None + + # Layer 2: symlinks in the parent chain. Walk every component of + # the *unresolved* destination from target_dir down to dest's + # parent, rejecting any existing symlink. We use the unresolved + # form because resolve() already followed symlinks; we want to + # detect them, not silently traverse. + unresolved = target_dir / rel + check = target_dir + for part in unresolved.relative_to(target_dir).parts[:-1]: + check = check / part + if check.exists() and os.path.islink(check): + raise ValueError( + f"snapshot path traverses a symlink: {rel!r} (at {check})" + ) + + dest.parent.mkdir(parents=True, exist_ok=True) + dest.write_text(content, encoding="utf-8") + + +# --------------------------------------------------------------------------- +# ResearchSupervisor +# --------------------------------------------------------------------------- + +class ResearchSupervisor: + """Karpathy loop for any task with a measurable deliverable. + + Args: + parent_agent: Live AIAgent instance (required for delegate_task). + workspace: Root directory for round artefacts. + progress_sink: Optional ProgressSink to receive run lifecycle events + (run_started / iteration_observed / run_completed) and free-form + comments. Defaults to a log-only StubSink when omitted. + """ + + def __init__( + self, + *, + parent_agent: Any, + workspace: Path | None = None, + progress_sink: Optional["ProgressSink"] = None, + ) -> None: + self._parent_agent = parent_agent + self._workspace = workspace or (get_hermes_home() / "research-workspace") + if progress_sink is None: + from agent.research.sinks import StubSink + self._sink = StubSink() + else: + self._sink = progress_sink + # Populated by run() — past-run lessons prepended to every worker brief. + self._evolution_overlay: str = "" + + def run( + self, + spec: TaskSpec, + initial_attempt: str, + *, + run_id: str, + max_iterations: int = 5, + time_budget_sec: int = 0, + keep_threshold: float = 0.0, + llm: Any = None, + worker_toolsets: list[str] | None = None, + checkpoint_dir: Path | None = None, + fan_out: int = 1, + use_moa: bool = True, + disable_evolution_overlay: bool = False, + ) -> ExperimentHistory: + """Run the Karpathy loop for any TaskSpec. + + Args: + spec: Task description — topic, deliverable, metric, task type. + initial_attempt: Starting deliverable (code string, search query, + research outline, or any text the worker can iterate on). + run_id: Unique identifier for this run. + max_iterations: Max improvement iterations (not counting baseline). + time_budget_sec: Time budget per worker invocation (seconds). + keep_threshold: Min absolute metric delta to count as kept. + llm: LLM client for improvement proposals (None = baseline only). + worker_toolsets: Override default toolsets for workers. + fan_out: Number of parallel hypothesis branches per iteration. + 1 = sequential (default). >1 = parallel fan-out. + use_moa: When fan_out > 1, synthesize all branches into a super-attempt + (MOA aggregation). Set to False to use only the best branch. + ExperimentHistory with all round results and the best result. + """ + config = HermesExperimentConfig( + metric_key=spec.metric_key, + metric_direction=spec.metric_direction, + time_budget_sec=time_budget_sec, + max_iterations=max_iterations, + keep_threshold=keep_threshold, + ) + + # All progress events flow through the sink. The local + # `comment_fn` name is the free-form-text channel used by the + # baseline-only branch, _reflect, and partial-success messages. + comment_fn = self._sink.comment + self._sink.run_started(spec, run_id) + + # Load past-run lessons once per run; cap size to avoid token blow-up. + # Failure must not break the loop — overlay is best-effort. The helper + # already catches its own errors, but wrap here too as defense in depth + # (matches the _evolve call site at the bottom of run()). + # When disable_evolution_overlay is True, skip entirely — useful for + # mechanics tests, isolated runs, and CI where global state from + # ~/.hermes/evolution would leak into the worker brief. + if disable_evolution_overlay: + self._evolution_overlay = "" + else: + try: + self._evolution_overlay = self._load_evolution_overlay() + except Exception as exc: + logger.warning("Evolution overlay load failed at run start: %s", exc) + self._evolution_overlay = "" + + toolsets = worker_toolsets or spec.default_toolsets() + attempt_holder: list[str] = [initial_attempt] + + def delegate_fn(goal: str, working_dir: str) -> DelegateSandboxResult: + return self._run_worker( + goal=goal, + working_dir=working_dir, + attempt=attempt_holder[0], + spec=spec, + time_budget_sec=time_budget_sec, + iteration=_extract_iteration(working_dir), + worker_toolsets=toolsets, + llm=llm, + ) + + runner = ExperimentRunner( + config=config, + workspace=self._workspace / run_id, + delegate_fn=delegate_fn, + progress_sink=self._sink, + ) + + run_dir = self._workspace / run_id + # The sink's run_started hook (already called above) handles the + # "loop started" announcement. No duplicate comment here. + + # --- HRM-93: durable resume --- + # If checkpoint.json + history.json exist on disk, replay-skip every + # round already completed. The baseline (round=0) and any subsequent + # rounds up to checkpoint["round"] are not re-executed. + start_iteration = 1 + best_artifact_holder: list[str] = [initial_attempt] + loaded = _load_checkpoint(checkpoint_dir) if checkpoint_dir else None + if loaded is not None: + resumed_history, last_round = loaded + runner.history = resumed_history + start_iteration = last_round + 1 + if resumed_history.best_result is not None: + best_artifact = ( + self._read_artifact(spec, run_dir, resumed_history.best_result) + or resumed_history.best_result.code + or initial_attempt + ) + else: + best_artifact = initial_attempt + best_artifact_holder = [best_artifact] + last = resumed_history.results[-1] if resumed_history.results else None + if last is not None: + attempt_holder[0] = ( + self._read_artifact(spec, run_dir, last) + or last.code + or initial_attempt + ) + comment_fn( + f"Resuming from checkpoint: round={last_round} " + f"best={resumed_history.best_result.primary_metric if resumed_history.best_result else None}" + ) + else: + # --- ACT (baseline) --- + baseline = runner.run_experiment(initial_attempt, run_id=run_id, iteration=0) + # Read on-disk artifact — worker may have modified the seed during baseline + baseline_artifact = self._read_artifact(spec, run_dir, baseline) or initial_attempt + attempt_holder[0] = baseline_artifact + best_artifact_holder = [baseline_artifact] + # --- OBSERVE --- (no previous best for the baseline iteration) + self._observe(baseline, spec, run_dir, previous_best=None) + self._sink.iteration_observed(0, baseline, run_dir) + self._checkpoint(runner.history, checkpoint_dir, round=0) + self._snapshot(runner.history, checkpoint_dir, run_dir, iteration=0, result=baseline) + if checkpoint_dir: + emit_event(checkpoint_dir, ResearchEvent.CHECKPOINT_SAVED, {"round": 0}) + emit_event(checkpoint_dir, ResearchEvent.SNAPSHOT_CREATED, {"iteration": 0}) + emit_event(checkpoint_dir, ResearchEvent.BASELINE_COMPLETED, {"metric": getattr(baseline, "primary_metric", None)}) + + if llm is None: + comment_fn(f"Baseline only. best={runner.history.baseline_metric}") + self._sink.run_completed(runner.history) + try: + self._evolve(runner.history, spec, run_id) + except Exception as exc: + logger.warning("Evolution persistence failed for %s: %s", run_id, exc) + return runner.history + + # Determine early-stop parameters based on baseline quality + baseline_metric = runner.history.baseline_metric + is_high_baseline = False + min_delta = 0.0 + if baseline_metric is not None: + if spec.metric_direction == "maximize" and baseline_metric >= 0.9: + is_high_baseline = True + min_delta = 0.05 + elif spec.metric_direction == "minimize" and baseline_metric <= 0.1: + is_high_baseline = True + min_delta = 0.05 + + early_stop_limit = 1 if is_high_baseline else 3 + if is_high_baseline: + logger.info( + "High baseline detected (%s=%.4f). Using aggressive early stop: " + "limit=%d, min_delta=%.2f", + spec.metric_key, baseline_metric, early_stop_limit, min_delta, + ) + + # Autogenesis AOOR improvement loop + no_improvement = 0 + for iteration in range(start_iteration, max_iterations + 1): + # Capture best metric BEFORE this iteration runs so _observe can + # tell plateau (equal to best) from regression (worse than best). + _prev_best = ( + runner.history.best_result.primary_metric + if runner.history.best_result is not None + else None + ) + if fan_out > 1: + # HYPOTHESIS FAN-OUT (HRM-108): generate N variants and run in parallel + comment_fn( + f"Fan-out iteration {iteration}: generating {fan_out} hypotheses" + ) + attempts = self._fan_out_attempts( + llm, spec, attempt_holder[0], runner.history, fan_out + ) + results = self._run_fan_out_iteration( + spec=spec, + attempts=attempts, + run_id=run_id, + iteration=iteration, + time_budget_sec=time_budget_sec, + worker_toolsets=toolsets, + llm=llm, + history=runner.history, + keep_threshold=keep_threshold, + ) + # results sorted best-first; observe/checkpoint only the winner + best_result = results[0] if results else None + if best_result is None: + logger.warning("Fan-out iteration %d produced no results", iteration) + no_improvement += 1 + attempt_holder[0] = best_artifact_holder[0] + continue + + result = best_result + if use_moa: + # MOA-style aggregation (HRM-109): synthesize all branches into + # a super-attempt that combines the best ideas from each branch. + # The aggregated attempt becomes the seed for the next iteration, + # while the best branch's metric determines if this round improved. + comment_fn( + f"MOA aggregation: synthesizing {len(results)} branches" + ) + aggregated_attempt = self._aggregate_attempts( + llm, spec, results, best_artifact_holder[0] + ) + actual_artifact = aggregated_attempt + else: + # Fan-out without aggregation: use best branch directly + comment_fn( + f"Fan-out best branch: using branch 1/{len(results)} (no MOA)" + ) + actual_artifact = self._read_artifact(spec, run_dir, best_result) or best_result.code or attempt_holder[0] + self._observe(best_result, spec, run_dir, previous_best=_prev_best) + self._sink.iteration_observed(iteration, best_result, run_dir) + self._checkpoint(runner.history, checkpoint_dir, round=iteration) + self._snapshot( + runner.history, checkpoint_dir, run_dir, + iteration=iteration, result=best_result, + ) + if checkpoint_dir: + emit_event(checkpoint_dir, ResearchEvent.CHECKPOINT_SAVED, {"round": iteration}) + emit_event(checkpoint_dir, ResearchEvent.SNAPSHOT_CREATED, {"iteration": iteration}) + else: + # SEQUENTIAL: single hypothesis per iteration + # OPTIMIZE — propose revised attempt (SEPL: propose) + next_attempt = self._improve_attempt(llm, spec, attempt_holder[0], runner.history) + attempt_holder[0] = next_attempt + + # ACT — worker executes the attempt + result = runner.run_experiment(next_attempt, run_id=run_id, iteration=iteration) + + # Read on-disk artifact — worker may have refined it beyond the seed + actual_artifact = self._read_artifact(spec, run_dir, result) or next_attempt + + # OBSERVE + REMEMBER — extract and persist structured learning + self._observe(result, spec, run_dir, previous_best=_prev_best) + self._sink.iteration_observed(iteration, result, run_dir) + self._checkpoint(runner.history, checkpoint_dir, round=iteration) + self._snapshot(runner.history, checkpoint_dir, run_dir, iteration=iteration, result=result) + if checkpoint_dir: + emit_event(checkpoint_dir, ResearchEvent.CHECKPOINT_SAVED, {"round": iteration}) + emit_event(checkpoint_dir, ResearchEvent.SNAPSHOT_CREATED, {"iteration": iteration}) + + attempt_holder[0] = actual_artifact + + # SEPL: evaluate → keep/discard (handled by ExperimentRunner) + # SEPL: rollback — restore best on-disk artifact, not the seed string + # For high baselines, require min_delta for improvement to count + improved = result.improved + if improved and is_high_baseline and baseline_metric is not None: + current_metric = result.primary_metric + if current_metric is not None: + delta = abs(current_metric - baseline_metric) + if delta < min_delta: + improved = False + logger.info( + "Improvement below min_delta (%.4f < %.2f), treating as non-improving", + delta, min_delta, + ) + + if improved: + no_improvement = 0 + best_artifact_holder[0] = actual_artifact + else: + no_improvement += 1 + attempt_holder[0] = best_artifact_holder[0] # rollback to best artifact + + # Acceptance-criterion early termination: if the metric crosses a + # parseable threshold (e.g. "pass_rate >= 0.9"), stop iterating — + # we've delivered the contract. Qualitative criteria are ignored + # here and continue to fall through to max_iterations / no-improve. + if result.primary_metric is not None and spec.acceptance_criterion: + acceptance_test = _parse_acceptance_criterion(spec.acceptance_criterion) + if acceptance_test is not None and acceptance_test(result.primary_metric): + logger.info( + "Acceptance criterion met for %s: '%s' (got %s=%s)", + run_id, spec.acceptance_criterion, + spec.metric_key, result.primary_metric, + ) + comment_fn( + f"Acceptance criterion met: '{spec.acceptance_criterion}' " + f"({spec.metric_key}={result.primary_metric})" + ) + break + + if no_improvement >= early_stop_limit: + logger.info( + "Early stop: %d non-improving iterations for %s (limit=%d)", + no_improvement, run_id, early_stop_limit, + ) + # SEPL: reflection optimizer — synthesize before giving up + self._reflect(runner.history, spec, llm, comment_fn, run_dir) + break + + best = runner.history.best_result + # Partial recovery: if the last iteration failed but we have prior results, + # report as partial success instead of total failure + last_result = runner.history.results[-1] if runner.history.results else None + if last_result and last_result.primary_metric is None and best: + comment_fn( + f"Loop done (PARTIAL): {len(runner.history.results)} rounds, " + f"best={best.primary_metric}. Last iteration failed but prior best preserved." + ) + + self._sink.run_completed(runner.history) + + # Persist lessons for cross-run learning. Append-only — failure here + # must not affect the loop's return value. + try: + self._evolve(runner.history, spec, run_id) + except Exception as exc: + logger.warning("Evolution persistence failed for %s: %s", run_id, exc) + + return runner.history + + # ------------------------------------------------------------------ + # Worker execution + # ------------------------------------------------------------------ + + def _run_fan_out_iteration( + self, + spec: TaskSpec, + attempts: list[str], + run_id: str, + iteration: int, + time_budget_sec: int, + worker_toolsets: list[str] | None, + llm: Any, + history: ExperimentHistory, + keep_threshold: float = 0.0, + ) -> list[ExperimentResult]: + """Execute N workers in parallel via delegate_task batch mode. + + Returns a list of ExperimentResult sorted by metric quality + (best first). All results are added to the provided history. + """ + t0 = _time.monotonic() + run_dir = self._workspace / run_id + toolsets = worker_toolsets or spec.default_toolsets() + attempt_filename = _ATTEMPT_FILENAME.get(spec.task_type, "attempt.md") + + # 1. Prepare N working directories + batch_dirs: list[Path] = [] + for i, attempt in enumerate(attempts): + wd = run_dir / f"round-{run_id}-iter{iteration}-branch{i}" + wd.mkdir(parents=True, exist_ok=True) + batch_dirs.append(wd) + (wd / attempt_filename).write_text(attempt, encoding="utf-8") + brief = _build_task_brief( + spec, + iteration=iteration, + round_dir=str(wd), + time_budget_sec=time_budget_sec, + ) + if self._evolution_overlay: + brief = self._evolution_overlay + "\n\n---\n\n" + brief + (wd / "task_brief.md").write_text(brief, encoding="utf-8") + + # 2. Build tasks array for delegate_task batch + tasks: list[dict[str, Any]] = [] + for i, wd in enumerate(batch_dirs): + context = ( + f"Working directory: {wd}\n" + f"Topic: {spec.topic}\n" + f"Task type: {spec.task_type}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n" + f"Read task_brief.md for full instructions." + ) + goal = ( + f"You are a Hermes research worker (branch {i}/{len(attempts)}). " + f"Read program.md in {wd} and run the experiment. " + f"Report your result as:\n" + f"METRIC: {spec.metric_key}= STATUS: improved|regressed|neutral " + f"NOTES: " + ) + tasks.append({"goal": goal, "context": context}) + + # 3. Execute batch + try: + batch_result = _call_delegate_task_batch( + tasks=tasks, + parent_agent=self._parent_agent, + toolsets=toolsets, + ) + except Exception as exc: + logger.exception("Fan-out batch delegate failed: %s", exc) + # Fallback: return all as failed + return [ + ExperimentResult( + run_id=run_id, + iteration=iteration, + code=attempt, + metrics={}, + primary_metric=None, + improved=False, + kept=False, + elapsed_sec=_time.monotonic() - t0, + stdout="", + stderr=str(exc), + error=str(exc), + ) + for attempt in attempts + ] + + # 4. Parse each result + results: list[ExperimentResult] = [] + current_best = ( + history.best_result.primary_metric + if history.best_result + else None + ) + entries = batch_result.get("results", []) + if len(entries) != len(attempts): + logger.warning( + "Fan-out result count mismatch: expected %d, got %d", + len(attempts), len(entries), + ) + + for i, attempt in enumerate(attempts): + entry = entries[i] if i < len(entries) else {} + wd = batch_dirs[i] + status = entry.get("status", "failed") + summary = entry.get("summary") or "" + error_str = entry.get("error") if entry.get("error") else None + if status != "completed" and not error_str: + error_str = f"Worker status: {status}" + + # Parse metrics + parsed = _parser.parse(wd, stdout=summary) + metrics: dict[str, object] = dict(parsed.to_flat_metrics()) + + # LLM judge override + if spec.evaluation_mode == "llm_judge" and llm is not None and summary: + judge_score = self._score_with_llm_judge(summary, spec, llm) + if judge_score is not None: + metrics[spec.metric_key] = judge_score + + primary_metric = ExperimentRunner._to_float( + metrics.get(spec.metric_key) + ) + + improved = False + kept = False + if primary_metric is not None: + if current_best is None: + improved = True + kept = True + elif ( + (spec.metric_direction == "maximize" and primary_metric > current_best) + or (spec.metric_direction == "minimize" and primary_metric < current_best) + ): + improved = True + kept = abs(primary_metric - current_best) > keep_threshold + + # Token / cost data (best-effort) + _tokens = entry.get("tokens") or {} + _tokens_in = int(_tokens.get("input", 0)) if isinstance(_tokens.get("input"), (int, float)) else 0 + _tokens_out = int(_tokens.get("output", 0)) if isinstance(_tokens.get("output"), (int, float)) else 0 + _cost_usd = float(entry.get("_child_cost_usd", 0.0)) if isinstance(entry.get("_child_cost_usd"), (int, float)) else 0.0 + + result = ExperimentResult( + run_id=run_id, + iteration=iteration, + code=attempt, + metrics=metrics, + primary_metric=primary_metric, + improved=improved, + kept=kept, + elapsed_sec=entry.get("duration_seconds", 0), + stdout=summary, + stderr="", + error=error_str, + tokens_in=_tokens_in, + tokens_out=_tokens_out, + cost_usd=_cost_usd, + ) + + if kept: + history.best_result = result + history.add(result) + results.append(result) + + # Sort by metric quality (best first) + def _score_key(r: ExperimentResult) -> float: + if r.primary_metric is None: + return float("-inf") if spec.metric_direction == "maximize" else float("inf") + return r.primary_metric + + results.sort(key=_score_key, reverse=(spec.metric_direction == "maximize")) + return results + + def _run_worker( + self, + *, + goal: str, + working_dir: str, + attempt: str, + spec: TaskSpec, + time_budget_sec: int, + iteration: int, + worker_toolsets: list[str] | None, + llm: Any, + ) -> DelegateSandboxResult: + """Write task brief + attempt file, spawn delegate_task, parse result.""" + t0 = _time.monotonic() + wd = Path(working_dir) + wd.mkdir(parents=True, exist_ok=True) + + # Write the attempt in the appropriate format + attempt_filename = _ATTEMPT_FILENAME.get(spec.task_type, "attempt.md") + (wd / attempt_filename).write_text(attempt, encoding="utf-8") + + # Write the task brief, with optional EvolutionStore overlay prepended. + # The overlay is the read-side of HRM-59 — it surfaces past-run lessons + # so the worker doesn't repeat known mistakes. Loaded once per run via + # _load_evolution_overlay() and cached on the supervisor instance. + brief = _build_task_brief( + spec, + iteration=iteration, + round_dir=working_dir, + time_budget_sec=time_budget_sec, + ) + if self._evolution_overlay: + brief = self._evolution_overlay + "\n\n---\n\n" + brief + (wd / "task_brief.md").write_text(brief, encoding="utf-8") + + context = ( + f"Working directory: {working_dir}\n" + f"Topic: {spec.topic}\n" + f"Task type: {spec.task_type}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n" + f"Read task_brief.md for full instructions." + ) + + result = _call_delegate_task( + goal, + context, + parent_agent=self._parent_agent, + toolsets=worker_toolsets or ["terminal", "file"], + ) + + elapsed = _time.monotonic() - t0 + first = result.get("results", [{}])[0] if result.get("results") else {} + summary = first.get("summary") or "" + status = first.get("status", "failed") + + # Parse metrics from structured files first, stdout fallback + parsed = _parser.parse(wd, stdout=summary) + metrics: dict[str, object] = dict(parsed.to_flat_metrics()) + + # LLM judge override: score the deliverable externally + if spec.evaluation_mode == "llm_judge" and llm is not None and summary: + judge_score = self._score_with_llm_judge(summary, spec, llm) + if judge_score is not None: + metrics[spec.metric_key] = judge_score + logger.info( + "LLM judge scored %s=%.4f for %s iter %d", + spec.metric_key, judge_score, working_dir, iteration, + ) + + completed = status == "completed" + error: str | None = None if completed else (first.get("error") or f"Worker status: {status}") + + # Extract token / cost data from delegate_task result (best-effort) + _tokens = first.get("tokens") or {} + _tokens_in = int(_tokens.get("input", 0)) if isinstance(_tokens.get("input"), (int, float)) else 0 + _tokens_out = int(_tokens.get("output", 0)) if isinstance(_tokens.get("output"), (int, float)) else 0 + _cost_usd = float(first.get("_child_cost_usd", 0.0)) if isinstance(first.get("_child_cost_usd"), (int, float)) else 0.0 + + return DelegateSandboxResult( + metrics=metrics, + stdout=summary, + stderr="", + elapsed_sec=elapsed, + timed_out=False, + returncode=0 if completed else 1, + error=error, + tokens_in=_tokens_in, + tokens_out=_tokens_out, + cost_usd=_cost_usd, + ) + + # ------------------------------------------------------------------ + # Autogenesis: Observe + Remember (HeartbeatMemorySystem schema) + # ------------------------------------------------------------------ + + def _read_artifact(self, spec: TaskSpec, run_dir: Path, result: ExperimentResult) -> str | None: + """Read the actual on-disk artifact produced by the worker. + + Workers may modify attempt.py / attempt.md beyond the seed string passed in. + This ensures rollback restores the real artifact, not the seed text. + """ + round_dir = run_dir / f"round-{result.run_id}-iter{result.iteration}" + attempt_filename = _ATTEMPT_FILENAME.get(spec.task_type, "attempt.md") + artifact_file = round_dir / attempt_filename + try: + return artifact_file.read_text(encoding="utf-8") if artifact_file.exists() else None + except OSError: + return None + + @staticmethod + def _insight_from_json(round_dir: Path, metric_key: str) -> str: + """Extract a human-readable insight from results.json (structured source).""" + results_json = round_dir / "results.json" + if not results_json.exists(): + return "" + try: + data = json.loads(results_json.read_text(encoding="utf-8")) + for field in ("notes", "summary", "insight", "description"): + val = data.get(field) + if isinstance(val, str) and val.strip(): + return val.strip()[:200] + val = data.get(metric_key) + if val is not None: + return f"{metric_key}={val}" + except (json.JSONDecodeError, OSError): + pass + return "" + + def _observe( + self, + result: ExperimentResult, + spec: TaskSpec, + run_dir: Path, + *, + previous_best: Optional[float] = None, + ) -> None: + """Extract a structured learning from a completed round and append to learnings.jsonl. + + Schema mirrors Autogenesis HeartbeatMemorySystem: + type — "improvement" | "regression" | "neutral" | "failure" + key — metric name being optimized + insight — one-line summary of what happened and why + confidence — metric value (0.0 if unavailable) + source — "iter-N" for lineage tracing + + Classification: + - failure — primary_metric is None (worker failed / unparseable) + - improvement — strictly better than the previous best + - regression — strictly worse than the previous best + - neutral — equal to (or first-seen against unknown) previous best + + ``previous_best`` is the best metric value before this iteration ran; + used to distinguish plateau from regression. When omitted or None, + the classifier falls back to the binary improved/regression split for + backwards compatibility. + + Insight extraction priority: + 1. results.json (structured, most reliable) + 2. NOTES: field from METRIC line in stdout + 3. Raw stdout excerpt (last resort) + """ + if result.primary_metric is None: + entry_type = "failure" + elif result.improved: + entry_type = "improvement" + elif previous_best is not None: + # Plateau (within float tolerance) is distinct from regression. + if abs(result.primary_metric - previous_best) < 1e-9: + entry_type = "neutral" + elif spec.metric_direction == "minimize": + entry_type = "regression" if result.primary_metric > previous_best else "neutral" + else: + entry_type = "regression" if result.primary_metric < previous_best else "neutral" + else: + # No prior best to compare against: treat non-improvement as neutral + # rather than asserting regression on the first round. + entry_type = "neutral" + + round_dir = run_dir / f"round-{result.run_id}-iter{result.iteration}" + + # 1. Structured source: results.json + insight_text = self._insight_from_json(round_dir, spec.metric_key) + + # 2. NOTES: field from the worker's METRIC line + if not insight_text and result.stdout: + m = re.search(r"NOTES:\s*(.+)", result.stdout) + if m: + insight_text = m.group(1).strip() + + # 3. Raw stdout excerpt + if not insight_text and result.stdout: + insight_text = result.stdout[:200].replace("\n", " ") + + # 4. Error fallback + if not insight_text and result.error: + insight_text = result.error[:200] + + # Normalize confidence to 0-1 scale regardless of metric direction + raw_metric = result.primary_metric + if raw_metric is not None: + # For minimize metrics, invert so higher confidence = better result + if spec.metric_direction == "minimize": + # Use inverse with a small epsilon to avoid div by zero + confidence = round(1.0 / (1.0 + abs(raw_metric)), 6) + else: + confidence = round(min(abs(raw_metric), 1.0), 6) + else: + confidence = 0.0 + + entry = { + "type": entry_type, + "key": spec.metric_key, + "insight": insight_text or "no output", + "confidence": confidence, + "source": f"iter-{result.iteration}", + } + + run_dir.mkdir(parents=True, exist_ok=True) + learnings_file = run_dir / "learnings.jsonl" + with learnings_file.open("a", encoding="utf-8") as f: + f.write(json.dumps(entry) + "\n") + + logger.debug( + "[observe] iter=%d type=%s %s=%.4f insight=%s", + result.iteration, entry_type, spec.metric_key, + entry["confidence"], insight_text[:80], + ) + + def _checkpoint( + self, + history: ExperimentHistory, + checkpoint_dir: Path | None, + round: int, + ) -> None: + """Serialize experiment history to a durable checkpoint directory. + + Called after baseline and every completed iteration so that external + monitors (e.g. research_job_tool) can read progress without polling + the running process AND so that a restarted job_runner can resume + instead of replaying baseline + completed rounds (HRM-93). + + history.json carries the full-fidelity dataclass dump (round-trips + through ExperimentHistory.from_dict) plus a "best" alias of + "best_result" for the legacy schema consumed by research_job_tool. + Both files are written atomically: tempfile + os.replace. + """ + if checkpoint_dir is None: + return + + checkpoint_dir.mkdir(parents=True, exist_ok=True) + + history_data = history.to_dict() + # Legacy alias used by tools/research_job_tool.py (_action_status, + # _action_resume) which reads history["best"]["primary_metric"]. + history_data["best"] = history_data.get("best_result") + + _atomic_write_text( + checkpoint_dir / "history.json", + json.dumps(history_data, indent=2), + ) + + _atomic_write_text( + checkpoint_dir / "checkpoint.json", + json.dumps({ + "round": round, + "total_rounds": len(history.results), + "best_metric": history.best_result.primary_metric if history.best_result else None, + "updated_at": _time.time(), + }, indent=2), + ) + + logger.debug("[checkpoint] round=%d dir=%s", round, checkpoint_dir) + + # ------------------------------------------------------------------ + # HRM-96: Atomic per-iteration workspace snapshot + # ------------------------------------------------------------------ + + def _snapshot( + self, + history: ExperimentHistory, + checkpoint_dir: Path | None, + run_dir: Path, + *, + iteration: int, + result: ExperimentResult, + ) -> None: + """Capture iteration state to /snapshots/iter-{N}.json. + + Captures: + - iteration (int) + - messages (list of full ExperimentResult dicts up to N) + - metrics (dict — current iteration's metrics) + - files (list of {path, content} for the round dir) + + File paths are stored relative to run_dir so restore_snapshot() can + rewrite them back into any workspace root. Atomic write via tempfile + + os.replace; partial writes never appear at the published path. + """ + if checkpoint_dir is None: + return + + snapshots_dir = checkpoint_dir / "snapshots" + snapshots_dir.mkdir(parents=True, exist_ok=True) + + round_dir = run_dir / f"round-{result.run_id}-iter{iteration}" + files: list[dict[str, str]] = [] + if round_dir.exists(): + for f in sorted(round_dir.rglob("*")): + if not f.is_file(): + continue + try: + content = f.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError): + continue + rel = f.relative_to(run_dir).as_posix() + files.append({"path": rel, "content": content}) + + history_data = history.to_dict() + snapshot = { + "iteration": iteration, + "messages": history_data.get("results", []), + "metrics": dict(result.metrics), + "files": files, + } + _atomic_write_text( + snapshots_dir / f"iter-{iteration}.json", + json.dumps(snapshot, indent=2), + ) + logger.debug("[snapshot] iter=%d files=%d", iteration, len(files)) + + # ------------------------------------------------------------------ + # Autogenesis: Reflect (SEPL reflection optimizer on early stop) + # ------------------------------------------------------------------ + + def _reflect( + self, + history: "ExperimentHistory", + spec: TaskSpec, + llm: Any, + comment_fn: "Callable[[str], None]", + run_dir: Path, + ) -> None: + """Synthesis pass after 3 non-improving iterations. + + Reads learnings.jsonl, asks the LLM to diagnose why the metric stalled, + and posts the diagnosis to Lattice. This is the SEPL reflection optimizer: + instead of iterating blindly, we re-examine whether the hypothesis was wrong. + """ + learnings_file = run_dir / "learnings.jsonl" + learnings: list[dict[str, Any]] = [] + if learnings_file.exists(): + for line in learnings_file.read_text(encoding="utf-8").splitlines(): + try: + learnings.append(json.loads(line)) + except json.JSONDecodeError: + pass + + best = history.best_result + best_metric = best.primary_metric if best else None + + if not llm: + comment_fn( + f"[reflect] Early stop after 3 non-improving rounds. " + f"Best {spec.metric_key}={best_metric}. " + f"llm=None — reflection skipped. Pass an LLM client to enable diagnosis." + ) + return + + if not learnings: + comment_fn( + f"[reflect] Early stop after 3 non-improving rounds. " + f"Best {spec.metric_key}={best_metric}. " + f"No learnings in learnings.jsonl — re-examine hypothesis manually." + ) + return + + learnings_summary = "\n".join( + f"- iter {e['source']}: {e['type']} | {e['key']}={e['confidence']} | {e['insight']}" + for e in learnings + ) + + prompt = ( + f"A research loop ran {len(learnings)} iterations on the following task:\n\n" + f"Topic: {spec.topic}\n" + f"Deliverable: {spec.deliverable}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n" + f"Best achieved: {best_metric}\n\n" + f"Round-by-round observations:\n{learnings_summary}\n\n" + "The loop stopped because 3 consecutive iterations did not improve the metric.\n\n" + "Diagnose:\n" + "1. Why did the metric stall? What is the fundamental bottleneck?\n" + "2. Was the hypothesis wrong — or was the approach right but the budget too small?\n" + "3. What ONE different approach would you try next if given another budget?\n\n" + "Be specific and concise. This diagnosis will be posted to the task tracker." + ) + + try: + response = llm.chat( + [{"role": "user", "content": prompt}], + system=( + "You are an expert research diagnostician. " + "Identify root causes, not symptoms. Be concrete and actionable." + ), + ) + diagnosis = getattr(response, "content", "").strip()[:1000] + except Exception as exc: + logger.warning("Reflection LLM call failed: %s", exc) + diagnosis = f"LLM reflection failed: {exc}" + + comment_fn( + f"[reflect] Early stop after {len(learnings)} rounds. " + f"Best {spec.metric_key}={best_metric}.\n\n" + f"Diagnosis:\n{diagnosis}" + ) + + # Persist the reflection as a special learning entry + reflection_entry = { + "type": "reflection", + "key": spec.metric_key, + "insight": diagnosis[:500], + "confidence": best_metric or 0.0, + "source": "reflect-final", + } + with learnings_file.open("a", encoding="utf-8") as f: + f.write(json.dumps(reflection_entry) + "\n") + + # ------------------------------------------------------------------ + # Autogenesis: Optimize — improvement proposal (Karpathy principles) + # ------------------------------------------------------------------ + + def _fan_out_attempts( + self, + llm: Any, + spec: TaskSpec, + current_attempt: str, + history: ExperimentHistory, + n: int, + ) -> list[str]: + """Generate N diverse revision hypotheses for the current attempt. + + Each variant targets a different bottleneck hypothesis so the + parallel batch can explore the solution space efficiently. + Returns a list of N attempt strings (may be fewer if the LLM + returns malformed output — callers must handle len < n). + """ + last = history.results[-1] if history.results else None + best = history.best_result + last_metric = last.primary_metric if last else None + best_metric = best.primary_metric if best else None + last_stdout = last.stdout if last else "" + + _DOMAIN_VERB = { + "code": "Revise the code", + "search": "Revise your search strategy, queries, or result ranking", + "research": "Deepen or reframe your research synthesis", + "generic": "Revise your approach", + } + domain_verb = _DOMAIN_VERB.get(spec.task_type, "Revise your approach") + + prompt = ( + f"Task: {spec.topic}\n" + f"Deliverable: {spec.deliverable}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n" + f"Last score: {last_metric}\n" + f"Best score: {best_metric}\n" + f"Last worker output (excerpt):\n{last_stdout[:800]}\n\n" + "---\n\n" + "Current attempt:\n" + f"{current_attempt}\n\n" + "---\n\n" + f"Generate EXACTLY {n} diverse revisions of the attempt above. " + "Each revision must explore a DIFFERENT hypothesis for why the metric " + "is stuck and what change could move it.\n\n" + "Format your response as:\n" + "=== VARIANT 1 ===\n" + "[hypothesis: one sentence]\n" + "[full revised attempt]\n" + "=== VARIANT 2 ===\n" + "[hypothesis: one sentence]\n" + "[full revised attempt]\n" + "... and so on.\n\n" + f"{domain_verb}. Each variant must be a complete, standalone attempt." + ) + + system = ( + f"You are a {spec.task_type} improvement specialist. " + "Generate N diverse hypotheses and revised attempts. " + "Be creative — each variant should try a genuinely different angle. " + "Never repeat the same change across variants." + ) + + try: + response = llm.chat([{"role": "user", "content": prompt}], system=system) + except Exception as exc: + logger.exception("Fan-out generation failed: %s", exc) + return [current_attempt] + + content = getattr(response, "content", "") + if not isinstance(content, str) or not content.strip(): + logger.warning("LLM returned empty fan-out; falling back to single attempt") + return [current_attempt] + + # Parse === VARIANT N === blocks + variants: list[str] = [] + for block in re.split(r"===\s*VARIANT\s*\d+\s*===", content): + block = block.strip() + if not block: + continue + # Drop the hypothesis line if it exists + lines = block.splitlines() + if lines and lines[0].lower().startswith("[hypothesis:"): + block = "\n".join(lines[1:]).strip() + if block: + variants.append(block) + + # If parsing yields fewer than n, pad with the current attempt + while len(variants) < n: + variants.append(current_attempt) + + return variants[:n] + + def _aggregate_attempts( + self, + llm: Any, + spec: TaskSpec, + results: list[ExperimentResult], + current_best_attempt: str, + ) -> str: + """Synthesize N fan-out results into a single super-attempt (MOA-style). + + Rather than keeping only the best branch, analyze what worked in each + branch and produce a merged attempt that is better than any individual. + Falls back to the best individual result if synthesis fails. + """ + if not results: + return current_best_attempt + + # Build a ranked summary of each branch + branches: list[str] = [] + for i, r in enumerate(results): + metric_str = f"{r.primary_metric:.4f}" if r.primary_metric is not None else "N/A" + branches.append( + f"--- BRANCH {i+1} (metric={metric_str}) ---\n" + f"Attempt:\n{r.code[:800]}\n\n" + f"Worker output:\n{r.stdout[:400]}\n" + ) + + branches_text = "\n\n".join(branches) + + prompt = ( + f"Task: {spec.topic}\n" + f"Deliverable: {spec.deliverable}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n\n" + "You just ran N parallel experiments with different hypotheses. " + "Here are the results, ranked from best to worst:\n\n" + f"{branches_text}\n\n" + "---\n\n" + "Current best attempt (before this round):\n" + f"{current_best_attempt}\n\n" + "---\n\n" + "## Synthesis Instructions (MOA)\n\n" + "Analyze each branch:\n" + "1. What specific change in this branch helped or hurt the metric?\n" + "2. Is there any idea here worth incorporating into the final attempt, " + "even if the branch itself underperformed?\n\n" + "Then produce a SINGLE merged attempt that:\n" + "- Starts from the current best attempt\n" + "- Incorporates the best ideas from ALL branches (not just the winner)\n" + "- Avoids the pitfalls you identified in weaker branches\n" + "- Is a complete, standalone deliverable\n\n" + "Return ONLY the merged attempt. " + "Include a brief comment at the top summarizing what you borrowed from each branch." + ) + + system = ( + f"You are a {spec.task_type} synthesis specialist. " + "You combine multiple partial solutions into one superior solution. " + "Be selective — don't merge blindly. Only incorporate changes that " + "directly serve the metric." + ) + + try: + response = llm.chat([{"role": "user", "content": prompt}], system=system) + except Exception as exc: + logger.exception("MOA aggregation failed: %s", exc) + # Fallback: return the best individual result + return results[0].code if results else current_best_attempt + + candidate = getattr(response, "content", "") + if not isinstance(candidate, str) or not candidate.strip(): + logger.warning("LLM returned empty aggregation; using best branch") + return results[0].code if results else current_best_attempt + + # For code tasks, extract from code fence if present + if spec.task_type == "code": + from agent.research.runner import ExperimentRunner + extracted = ExperimentRunner._extract_python_code(candidate) + return extracted if extracted.strip() else candidate.strip() + + return candidate.strip() + + def _improve_attempt( + self, + llm: Any, + spec: TaskSpec, + current_attempt: str, + history: ExperimentHistory, + ) -> str: + """Propose a revised attempt using domain-aware Karpathy prompting.""" + last = history.results[-1] if history.results else None + best = history.best_result + last_metric = last.primary_metric if last else None + best_metric = best.primary_metric if best else None + last_stdout = last.stdout if last else "" + + _DOMAIN_VERB = { + "code": "Revise the code", + "search": "Revise your search strategy, queries, or result ranking", + "research": "Deepen or reframe your research synthesis", + "generic": "Revise your approach", + } + domain_verb = _DOMAIN_VERB.get(spec.task_type, "Revise your approach") + + _DOMAIN_HINT = { + "code": "Make surgical edits — every changed line must trace to your hypothesis. " + "No refactoring of unrelated sections.", + "search": "Test exactly ONE new query strategy or source. " + "Don't repeat what didn't work.", + "research": "Address exactly ONE identified gap (missing source, weak argument, " + "uncovered angle). Don't rewrite everything.", + "generic": "Change only what your hypothesis requires. " + "Minimum viable revision.", + } + domain_hint = _DOMAIN_HINT.get(spec.task_type, "") + + prompt = ( + f"Task: {spec.topic}\n" + f"Deliverable: {spec.deliverable}\n" + f"Metric: {spec.metric_key} ({spec.metric_direction})\n" + f"Last score: {last_metric}\n" + f"Best score: {best_metric}\n" + f"Last worker output (excerpt):\n{last_stdout[:800]}\n\n" + "---\n\n" + "Current attempt:\n" + f"{current_attempt}\n\n" + "---\n\n" + "## Think Before Revising (Karpathy Principle 1)\n\n" + "State:\n" + f"1. WHY is `{spec.metric_key}` at {last_metric}? What is the binding bottleneck?\n" + "2. Your ONE hypothesis for what change will move it.\n" + f"3. Success criterion: `{spec.metric_key}` moves from {last_metric} toward " + f"{'higher' if spec.metric_direction == 'maximize' else 'lower'}.\n\n" + "## Simplicity First\n\n" + "If the revision can be 5 lines, make it 5 lines — not 50.\n" + "No speculative additions. No features that don't serve the metric.\n\n" + f"## Your Task\n\n{domain_verb}. {domain_hint}\n\n" + "Return ONLY the revised attempt. " + "Include a brief comment at the top stating your hypothesis and what you changed." + ) + + system = ( + f"You are a {spec.task_type} improvement specialist. " + "Apply the Karpathy loop: think first, make surgical changes, verify the metric moves. " + "Surface your reasoning. Never guess silently." + ) + + try: + response = llm.chat([{"role": "user", "content": prompt}], system=system) + except Exception as exc: + logger.exception("Improvement call failed: %s", exc) + return current_attempt + + candidate = getattr(response, "content", "") + if not isinstance(candidate, str) or not candidate.strip(): + logger.warning("LLM returned empty attempt; keeping current") + return current_attempt + + # For code tasks, extract from code fence if present + if spec.task_type == "code": + from agent.research.runner import ExperimentRunner + extracted = ExperimentRunner._extract_python_code(candidate) + return extracted if extracted.strip() else candidate.strip() + + return candidate.strip() + + # ------------------------------------------------------------------ + # Evolution — persist lessons across runs (HRM-59 v1) + # and surface them to next-run workers (HRM-62 v2) + # ------------------------------------------------------------------ + + _OVERLAY_MAX_CHARS = 1500 + _OVERLAY_MAX_LESSONS = 3 + + def _load_evolution_overlay(self) -> str: + """Load past-run lessons formatted for prepending to worker briefs. + + Reads from EvolutionStore at $HERMES_HOME/evolution and uses + build_overlay() with a research-loop scope. Capped at + _OVERLAY_MAX_CHARS to bound prompt-token cost. Returns empty + string on any failure — the loop must run with or without lessons. + """ + try: + from agent.research.evolution import EvolutionStore + + store_dir = get_hermes_home() / "evolution" + if not store_dir.exists(): + return "" + overlay = EvolutionStore(store_dir).build_overlay( + stage_name="research_loop", + max_lessons=self._OVERLAY_MAX_LESSONS, + ) + if len(overlay) > self._OVERLAY_MAX_CHARS: + overlay = overlay[: self._OVERLAY_MAX_CHARS] + "\n\n[... overlay truncated ...]" + return overlay + except Exception as exc: + logger.warning("Evolution overlay load failed: %s", exc) + return "" + + def _evolve(self, history: Any, spec: TaskSpec, run_id: str) -> None: + """Append per-iteration lessons from this run to the EvolutionStore. + + Adapter between ExperimentResult (Karpathy loop) and LessonEntry + (ResearchClaw schema). v1 only persists; prompt overlay injection + is intentionally deferred to v2 so this hook stays append-only and + cannot affect ongoing or future loops if it misbehaves. + """ + from datetime import datetime, timezone + from agent.research.evolution import ( + EvolutionStore, + LessonEntry, + LessonCategory, + _classify_error, + ) + + results = getattr(history, "results", []) or [] + if not results: + return + + now = datetime.now(timezone.utc).isoformat() + lessons: list[LessonEntry] = [] + + for result in results: + iteration = getattr(result, "iteration", 0) + stage_name = f"iter_{iteration}" + error = getattr(result, "error", None) + improved = getattr(result, "improved", False) + kept = getattr(result, "kept", False) + metric = getattr(result, "primary_metric", None) + + if error: + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=iteration, + category=_classify_error(stage_name, str(error)), + severity="error", + description=f"{spec.metric_key}: {error}", + timestamp=now, + run_id=run_id, + )) + elif improved and kept: + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=iteration, + category=LessonCategory.PIPELINE, + severity="info", + description=f"{spec.metric_key} improved to {metric}", + timestamp=now, + run_id=run_id, + )) + else: + lessons.append(LessonEntry( + stage_name=stage_name, + stage_num=iteration, + category=LessonCategory.PIPELINE, + severity="warning", + description=f"{spec.metric_key}={metric} no improvement, attempt discarded", + timestamp=now, + run_id=run_id, + )) + + store_dir = get_hermes_home() / "evolution" + store = EvolutionStore(store_dir) + store.append_many(lessons) + + # ------------------------------------------------------------------ + # LLM judge evaluator + # ------------------------------------------------------------------ + + def _score_with_llm_judge( + self, + deliverable: str, + spec: TaskSpec, + llm: Any, + ) -> float | None: + """Score a deliverable using an LLM judge. Returns 0.0–1.0 or None.""" + eval_prompt = spec.evaluation_prompt or ( + f"Score the following deliverable for the task '{spec.topic}' " + f"on a scale of 0.0 to 1.0, where 1.0 = perfect. " + f"Return ONLY a decimal number, nothing else." + ) + prompt = f"{eval_prompt}\n\nDeliverable:\n{deliverable[:4000]}\n\nScore (0.0–1.0):" + content = "" + try: + response = llm.chat( + [{"role": "user", "content": prompt}], + system="You are an objective evaluator. Return only a decimal number between 0.0 and 1.0.", + ) + content = (getattr(response, "content", "") or "").strip() + if not content: + logger.warning("LLM judge returned empty response") + return None + # Extract the first decimal found anywhere — tolerates prose like + # "Score: 0.85", "0.8/1.0", "The score is 0.7 because …". + match = _JUDGE_SCORE_RE.search(content) + if match is None: + logger.warning( + "LLM judge response had no numeric score; raw=%r", + content[:200], + ) + return None + return max(0.0, min(1.0, float(match.group()))) + except Exception as exc: + logger.warning( + "LLM judge scoring failed: %s; raw=%r", exc, content[:200] + ) + return None + + +# --------------------------------------------------------------------------- +# helpers +# --------------------------------------------------------------------------- + +def _extract_iteration(working_dir: str) -> int: + try: + return int(working_dir.rsplit("iter", 1)[-1]) + except (ValueError, IndexError): + return 0 diff --git a/agent/transports/chat_completions.py b/agent/transports/chat_completions.py index 49bc91f44d2a..ebef76be3236 100644 --- a/agent/transports/chat_completions.py +++ b/agent/transports/chat_completions.py @@ -10,7 +10,10 @@ """ import copy -from typing import Any, Dict +import logging +from typing import Any, Dict, List, Optional + +logger = logging.getLogger(__name__) from agent.lmstudio_reasoning import resolve_lmstudio_effort from agent.moonshot_schema import is_moonshot_model, sanitize_moonshot_tools @@ -453,6 +456,30 @@ def build_kwargs( if overrides: api_kwargs.update(overrides) + # Tool choice override for proactive/agentic turns. + # When forcing tool_choice="required", disable reasoning/thinking for + # this turn — several providers (Kimi, DeepSeek, etc.) reject the + # combination. The system prompt still instructs the agent to use tools. + _tool_choice = params.get("tool_choice") + if _tool_choice: + if _tool_choice == "required": + _stripped_any = False + if "reasoning_effort" in api_kwargs: + api_kwargs.pop("reasoning_effort") + _stripped_any = True + if "extra_body" in api_kwargs: + _eb = api_kwargs["extra_body"] + if isinstance(_eb, dict) and "thinking_config" in _eb: + _eb.pop("thinking_config") + _stripped_any = True + if isinstance(_eb, dict) and not _eb: + api_kwargs.pop("extra_body") + if _stripped_any: + logger.info( + "[chat_completions] Disabled thinking for tool_choice='required' turn." + ) + api_kwargs["tool_choice"] = _tool_choice + return api_kwargs def _build_kwargs_from_profile(self, profile, model, sanitized, tools, params): @@ -570,6 +597,33 @@ def _build_kwargs_from_profile(self, profile, model, sanitized, tools, params): else: api_kwargs[k] = v + # Tool choice override for proactive/agentic turns. + # When forcing tool_choice="required", disable reasoning/thinking + # for this turn — several providers reject the combination. + # tool_choice may arrive via params (direct) or via request_overrides (merged above). + _tool_choice = params.get("tool_choice") + if not _tool_choice and overrides: + _tool_choice = overrides.get("tool_choice") + if _tool_choice: + if _tool_choice == "required": + _stripped_any = False + if "reasoning_effort" in api_kwargs: + api_kwargs.pop("reasoning_effort") + _stripped_any = True + if "thinking" in extra_body: + extra_body.pop("thinking") + _stripped_any = True + if "thinking_config" in extra_body: + extra_body.pop("thinking_config") + _stripped_any = True + if _stripped_any: + logger.info( + "[chat_completions] Disabled thinking for tool_choice='required' turn (profile path)." + ) + # Only set if not already set by request_overrides merge above + if "tool_choice" not in api_kwargs: + api_kwargs["tool_choice"] = _tool_choice + if extra_body: # Native Gemini (generativelanguage.googleapis.com, non-/openai) # speaks Google's REST schema, not OpenAI's. OpenAI-style extra_body diff --git a/cli.py b/cli.py index 47bca386241a..a940c411d7c5 100644 --- a/cli.py +++ b/cli.py @@ -462,6 +462,12 @@ def load_cli_config() -> Dict[str, Any]: "skin": "default", }, + "tui": { + "input_max_lines": 8, + "collapse_large_pastes": True, + "history_nav_requires_empty_input": False, + "show_full_input": False, + }, "clarify": { "timeout": 120, # Seconds to wait for a clarify answer before auto-proceeding }, @@ -3250,6 +3256,9 @@ def __init__( self.busy_input_mode = "steer" else: self.busy_input_mode = "interrupt" + # ctrl_c_priority: "interrupt_agent" (default) or "clear_input" + _ccp = CLI_CONFIG["display"].get("ctrl_c_priority", "interrupt_agent") + self.ctrl_c_priority = "clear_input" if str(_ccp).strip().lower() == "clear_input" else "interrupt_agent" # self.verbose ONLY controls global DEBUG logging (root logger level). # display.tool_progress="verbose" controls tool-call rendering (full args, @@ -4645,9 +4654,13 @@ def _expand_ref(match): def _print_user_message_preview(self, user_input: str) -> None: """Render a user message using the normal chat scrollback style.""" + _show_full_input = bool(CLI_CONFIG.get("tui", {}).get("show_full_input", False)) ChatConsole().print(f"[{_accent_hex()}]{'─' * 40}[/]") text = str(user_input or "") - if "\n" in text: + if _show_full_input: + ChatConsole().print(f"[{_accent_hex()}]{'─' * 40}[/]") + ChatConsole().print(f"[bold {_accent_hex()}]●[/] [bold]{_escape(text)}[/]") + elif "\n" in text: ChatConsole().print(self._format_submitted_user_message_preview(text)) else: ChatConsole().print(f"[bold {_accent_hex()}]●[/] [bold]{_escape(text)}[/]") @@ -10566,9 +10579,24 @@ def run_agent(): all_parts.append(extra) except queue.Empty: break - combined = "\n".join(all_parts) + + # Normalize multimodal payloads: (text, images) tuples come from + # interrupt messages that contain pasted/attached images. + # Split text and images so we can combine text with join while + # preserving image attachments. + texts = [] + images = [] + for part in all_parts: + if isinstance(part, tuple): + texts.append(str(part[0]) if part else "") + if len(part) > 1: + images.extend(part[1]) + else: + texts.append(str(part)) + + combined = ("\n".join(texts), images) if images else "\n".join(texts) n = len(all_parts) - preview = combined[:50] + ("..." if len(combined) > 50 else "") + preview = "\n".join(texts)[:50] + ("..." if len("\n".join(texts)) > 50 else "") if n > 1: print(f"\n⚡ Sending {n} messages after interrupt: '{preview}'") else: @@ -11543,9 +11571,13 @@ def handler(event): lambda: not self._clarify_state and not self._approval_state and not self._slash_confirm_state and not self._sudo_state and not self._secret_state and not self._model_picker_state ) + _history_nav_requires_empty = bool(CLI_CONFIG.get("tui", {}).get("history_nav_requires_empty_input", False)) + @kb.add('up', filter=_normal_input) def history_up(event): """Up arrow: browse history when on first line, else move cursor up.""" + if _history_nav_requires_empty and event.app.current_buffer.text: + return event.app.current_buffer.auto_up(count=event.arg) @kb.add('down', filter=_normal_input) @@ -11641,13 +11673,21 @@ def handle_ctrl_c(event): event.app.invalidate() return + # When the user prefers "clear_input", Ctrl+C behaves like bash: + # clear the buffer first; only interrupt the agent when the buffer is empty. + if self.ctrl_c_priority == "clear_input" and (event.app.current_buffer.text or self._attached_images): + event.app.current_buffer.reset() + self._attached_images.clear() + event.app.invalidate() + return + if self._agent_running and self.agent: if now - self._last_ctrl_c_time < 2.0: print("\n⚡ Force exiting...") self._should_exit = True event.app.exit() return - + self._last_ctrl_c_time = now print("\n⚡ Interrupting agent... (press Ctrl+C again to force exit)") self.agent.interrupt() @@ -11909,6 +11949,9 @@ def _start_recording(): event.app.invalidate() from prompt_toolkit.keys import Keys + _input_max_lines = int(CLI_CONFIG.get("tui", {}).get("input_max_lines", 8)) + _collapse_large_pastes = bool(CLI_CONFIG.get("tui", {}).get("collapse_large_pastes", True)) + @kb.add(Keys.BracketedPaste, eager=True) def handle_paste(event): """Handle terminal paste — detect clipboard images. @@ -12022,11 +12065,12 @@ def get_prompt(): skill_bundles_provider=lambda: get_skill_bundles(), ) input_area = TextArea( - height=Dimension(min=1, max=8, preferred=1), + height=Dimension(min=1, max=_input_max_lines, preferred=1), prompt=get_prompt, style='class:input-area', multiline=True, wrap_lines=True, + scrollbar=True, read_only=Condition(lambda: bool(cli_ref._command_running)), history=FileHistory(str(self._history_file)), completer=_completer, @@ -12042,6 +12086,26 @@ def get_prompt(): # EEXIST. The suffix keeps markdown highlighting without that bug. input_area.buffer.tempfile_suffix = '.md' + # Guard history navigation so Up/Down only browse history when the input is empty. + if _history_nav_requires_empty: + _orig_auto_up = input_area.buffer.auto_up + _orig_auto_down = input_area.buffer.auto_down + + def _auto_up_guard(count=1, go_to_start_of_line_if_history_changes=False): + if input_area.buffer.text: + input_area.buffer.cursor_up(count) + else: + _orig_auto_up(count, go_to_start_of_line_if_history_changes) + + def _auto_down_guard(count=1, go_to_start_of_line_if_history_changes=False): + if input_area.buffer.text: + input_area.buffer.cursor_down(count) + else: + _orig_auto_down(count, go_to_start_of_line_if_history_changes) + + input_area.buffer.auto_up = _auto_up_guard + input_area.buffer.auto_down = _auto_down_guard + # Dynamic height: accounts for both explicit newlines AND visual # wrapping of long lines so the input area always fits its content. def _input_height(): @@ -13014,10 +13078,19 @@ def process_loop(): # Expand paste references back to full content _paste_ref_re = re.compile(r'\[Pasted text #\d+: \d+ lines \u2192 (.+?)\]') paste_refs = list(_paste_ref_re.finditer(user_input)) if isinstance(user_input, str) else [] + _show_full_input = bool(CLI_CONFIG.get("tui", {}).get("show_full_input", False)) + _user_bar = f"[{_accent_hex()}]{'─' * 40}[/]" + print() + ChatConsole().print(_user_bar) if paste_refs: user_input = self._expand_paste_references(user_input) print() - self._print_user_message_preview(user_input) + _show_full_input = bool(CLI_CONFIG.get("tui", {}).get("show_full_input", False)) + if _show_full_input: + ChatConsole().print(f"[{_accent_hex()}]{'─' * 40}[/]") + ChatConsole().print(f"[bold {_accent_hex()}]●[/] [bold]{_escape(user_input)}[/]") + else: + self._print_user_message_preview(user_input) # Show image attachment count if submit_images: diff --git a/docs/architecture/autoresearch-goal-kanban-review.md b/docs/architecture/autoresearch-goal-kanban-review.md new file mode 100644 index 000000000000..36ab16da79f7 --- /dev/null +++ b/docs/architecture/autoresearch-goal-kanban-review.md @@ -0,0 +1,202 @@ +# Architecture Review: AutoResearcher + /goal + Kanban (HRM-110 follow-up) + +**Context:** HRM-110 (A/B testing) is complete. This doc explores how the new `/goal` and `kanban` primitives could reshape the autoresearch architecture. + +--- + +## Current AutoResearch Architecture + +``` +User → run_research tool → ResearchSupervisor.run() + ├── ExperimentRunner (Karpathy loop) + │ ├── delegate_task → Worker subagent + │ └── Metric parsing + keep/discard + ├── _improve_attempt() → LLM reflection + └── Lattice comments (progress tracking) +``` + +**Strengths:** Self-contained, single-turn tool call, deterministic loop. + +**Weaknesses:** +- Research is a "fire-and-forget" tool call. If it exceeds the turn budget, the user must wait or the agent loops blindly. +- No visibility into running experiments except Lattice comments. +- Fan-out branches are ephemeral — no durable task representation. +- No integration with the dispatch/worker system that Hermes uses for other multi-agent workloads. + +--- + +## New Primitives Overview + +### `/goal` (GoalManager) + +- **What:** A standing objective that persists across turns. After each assistant response, a judge model asks "is the goal done?" +- **State machine:** active → paused → done/cleared +- **Budget:** Max turns (default 20). Auto-pauses on budget exhaustion. +- **Persistence:** Stored in SessionDB state_meta, survives `/resume`. + +### Kanban + +- **What:** SQLite-backed task board with statuses (triage → todo → ready → running → blocked → done → archived). +- **Claim/CAS:** Workers claim tasks via compare-and-swap on `claim_lock`. +- **Workspaces:** Each task gets a scratch/workspace directory. +- **Events/Comments:** Full audit trail per task. +- **Multi-board:** Separate boards per project. + +--- + +## Strategic Opportunities + +### 1. Research as a Persistent Goal + +**Idea:** A research task becomes a `/goal` instead of a single `run_research` tool call. + +**Flow:** +``` +User: /goal "Find and evaluate 5 papers on diffusion models for video generation" + +Turn 1: Agent runs baseline search, produces initial list, metric = 0.3 +Judge: "NOT done — only 2 papers found, need 5" + +Turn 2: Agent refines search strategy, finds 3 more papers, metric = 0.6 +Judge: "NOT done — papers found but no evaluation yet" + +Turn 3: Agent evaluates each paper, metric = 0.9 +Judge: "DONE — 5 papers found and evaluated" +``` + +**Benefits:** +- Research can span multiple turns without blocking the user. +- The judge provides an external "done" signal independent of the worker's self-report. +- Natural pause/resume semantics (`/goal pause`, `/goal resume`). + +**Challenges:** +- The judge currently evaluates a single response, not a cumulative research state. Need a research-aware judge that can inspect the workspace/checkpoints. +- Turn budget may be too coarse for research (20 turns × N iterations each = potentially 100+ worker calls). + +### 2. Kanban as Experiment Orchestrator + +**Idea:** Each experiment run becomes a kanban task. Fan-out branches become linked subtasks. + +**Mapping:** + +| Kanban Concept | Research Mapping | +|----------------|------------------| +| Board | Research project / topic | +| Task | Single experiment run | +| Task status | Experiment lifecycle | +| Task workspace | `research-workspace//` | +| Task assignee | Worker profile or agent ID | +| Task links | Parent/child for fan-out branches | +| Task comments | Round-by-round progress | +| Task events | CHECKPOINT_SAVED, SNAPSHOT_CREATED | + +**Flow:** +``` +User: /kanban create "Research: diffusion models for video" +Agent: Creates task, moves to "running" + +Each iteration: + → Update task comment with metric + → If blocked: move to "blocked", add block reason + → If done: move to "done", attach results.json + +Fan-out (3 branches): + → Create 3 linked subtasks + → Each worker claims one subtask + → MOA aggregation: parent task collects subtask results +``` + +**Benefits:** +- Durable, queryable experiment history (not just Lattice comments). +- Workers can claim experiment tasks via the standard kanban dispatcher. +- Dashboard visibility into all running research. +- Research tasks coexist with coding tasks on the same board. + +**Challenges:** +- Kanban tasks are designed for dispatcher/worker workloads, not iterative self-improvement loops. Need a "loop driver" that updates the task after each iteration. +- The kanban DB schema doesn't have native support for "iteration N of M" or "best metric so far." + +### 3. Unified Dispatch: Kanban Workers for Research + +**Idea:** Replace `delegate_task` with kanban's dispatch system for research workers. + +**Current:** `ResearchSupervisor` spawns workers directly via `delegate_task`. +**Proposed:** `ResearchSupervisor` creates kanban tasks; the kanban dispatcher spawns workers. + +``` +ResearchSupervisor → kanban_db.create_task() → dispatcher tick + → worker claims task + → worker runs experiment + → worker updates task result + → ResearchSupervisor reads result +``` + +**Benefits:** +- Workers can run detached (like `research_job_tool`) but with full kanban lifecycle. +- Fault tolerance: if a worker crashes, the task becomes reclaimable. +- Multi-profile: different worker profiles for different task types (code vs search vs research). + +**Challenges:** +- Research workers need tight feedback loops (seconds, not minutes). Kanban dispatcher ticks are typically 30s. +- The supervisor needs synchronous results to decide keep/discard. Async kanban would require a polling loop. + +### 4. Research-Aware Goal Judge + +**Idea:** Extend the goal judge to understand research-specific completion criteria. + +**Current judge prompt:** "Is the goal satisfied based on the last response?" +**Research judge prompt:** "Is the research goal satisfied? Check: (1) metric >= threshold, (2) iterations converged, (3) no regressions in last 3 rounds." + +**Integration point:** `agent/research/ab_testing.py` could expose a `ResearchGoalEvaluator` that the GoalManager calls instead of the generic judge. + +--- + +## Recommended Path Forward + +### Phase 1: Research-Aware Goal (Immediate) + +Implement a `ResearchGoalManager` subclass or adapter that: +- Wraps `ResearchSupervisor.run()` as a goal lifecycle. +- Uses the research metric as the "done" signal (instead of a generic judge). +- Persists experiment state in the goal's session metadata. + +**Deliverable:** A new tool `run_research_goal` or a parameter `as_goal=True` in `run_research`. + +### Phase 2: Kanban Integration for Fan-Out (Medium-term) + +When `fan_out > 1`, create linked kanban subtasks instead of inline `delegate_task` batch calls. + +**Deliverable:** `ResearchSupervisor._run_fan_out_iteration()` optionally uses kanban tasks. Falls back to `delegate_task` when kanban is unavailable. + +### Phase 3: Research Board Template (Long-term) + +A `hermes kanban board create --template research` that sets up: +- Predefined columns: hypothesis → experiment → evaluate → aggregate → done +- Auto-linking of fan-out subtasks +- Dashboard widgets for metric-over-time graphs + +--- + +## Open Questions + +1. **Should research tasks live on the main kanban board or a separate board?** + - Same board: visibility alongside coding tasks. + - Separate board: cleaner schema, research-specific columns. + +2. **How does the goal turn budget interact with research iterations?** + - One turn = one iteration? Too coarse. + - One turn = one full run (baseline + N iterations)? May exceed budget. + - Sub-turn budget within the goal loop? + +3. **Should the A/B tester (HRM-110) create kanban tasks for each strategy?** + - Yes: each strategy run becomes a task, results are comments. + - No: keep A/B tester lightweight, use Lattice for now. + +--- + +## Related Tasks + +- HRM-105: TUI integration for research progress → could render kanban tasks. +- HRM-106: Dashboard metrics graph → could read kanban task events. +- HRM-107: Lattice API native → kanban already has a native API (SQLite). +- HRM-111: Skill generation from lessons → kanban tasks could track skill generation experiments. diff --git a/gateway/config.py b/gateway/config.py index f11146e606ad..b1ed2acc1345 100644 --- a/gateway/config.py +++ b/gateway/config.py @@ -164,6 +164,7 @@ class Platform(Enum): BLUEBUBBLES = "bluebubbles" QQBOT = "qqbot" YUANBAO = "yuanbao" + DAEMONCRAFT = "daemoncraft" @classmethod def _missing_(cls, value): """Accept unknown platform names only for known plugin adapters. diff --git a/gateway/platforms/base.py b/gateway/platforms/base.py index 205d9cbf5096..ff38705822e7 100644 --- a/gateway/platforms/base.py +++ b/gateway/platforms/base.py @@ -1466,6 +1466,10 @@ class MessageEvent: # completion notifications) that must bypass user authorization checks. internal: bool = False + # Tool choice override for proactive/agentic turns (e.g. wake-up events). + # When set to "required", the agent MUST respond with a tool call. + tool_choice: Optional[str] = None + # Timestamps timestamp: datetime = field(default_factory=datetime.now) diff --git a/gateway/platforms/block_to_char_1.21.9.js b/gateway/platforms/block_to_char_1.21.9.js new file mode 100644 index 000000000000..877643c4c73a --- /dev/null +++ b/gateway/platforms/block_to_char_1.21.9.js @@ -0,0 +1,1172 @@ +// Auto-generated from minecraft-data 1.21.9 (1166 blocks, 0 collisions) +// Regenerate via: python3 lib/build_block_chars.py +// Pool: CJK Unified Ideographs (U+4E00-U+9FFF), starts after category chars. + +export const BLOCK_TO_CHAR = { + 'air': ' ', + 'cave_air': ' ', + 'void_air': ' ', + 'water': '~', + 'lava': '!', + 'short_grass': ',', + 'tall_grass': ';', + 'redstone_wire': 'R', + 'redstone_torch': 'r', + 'torch': '†', + 'wall_torch': '†', + 'soul_torch': '†', + 'lantern': '◊', + 'soul_lantern': '◊', + 'waxed_copper_door': '◫', + 'waxed_oxidized_copper_door': '◫', + 'cherry_door': '◫', + 'weathered_copper_door': '◫', + 'waxed_weathered_copper_door': '◫', + 'jungle_door': '◫', + 'dark_oak_door': '◫', + 'mangrove_door': '◫', + 'waxed_exposed_copper_door': '◫', + 'oak_door': '◫', + 'oxidized_copper_door': '◫', + 'pale_oak_door': '◫', + 'acacia_door': '◫', + 'exposed_copper_door': '◫', + 'birch_door': '◫', + 'spruce_door': '◫', + 'bamboo_door': '◫', + 'crimson_door': '◫', + 'copper_door': '◫', + 'warped_door': '◫', + 'iron_door': '◫', + 'chest': '◰', + 'ender_chest': '◰', + 'trapped_chest': '◰', + 'smoker': '⊡', + 'furnace': '⊡', + 'blast_furnace': '⊡', + 'cartography_table': '⊞', + 'smithing_table': '⊞', + 'fletching_table': '⊞', + 'crafting_table': '⊞', + 'loom': '⊞', + 'pink_bed': '⊏', + 'gray_bed': '⊏', + 'magenta_bed': '⊏', + 'red_bed': '⊏', + 'yellow_bed': '⊏', + 'light_blue_bed': '⊏', + 'white_bed': '⊏', + 'orange_bed': '⊏', + 'green_bed': '⊏', + 'light_gray_bed': '⊏', + 'blue_bed': '⊏', + 'purple_bed': '⊏', + 'cyan_bed': '⊏', + 'black_bed': '⊏', + 'brown_bed': '⊏', + 'lime_bed': '⊏', + 'glass': '▢', + 'brown_stained_glass': '▢', + 'blue_stained_glass': '▢', + 'lime_stained_glass': '▢', + 'light_gray_stained_glass': '▢', + 'green_stained_glass': '▢', + 'gray_stained_glass': '▢', + 'tinted_glass': '▢', + 'pink_stained_glass': '▢', + 'purple_stained_glass': '▢', + 'cyan_stained_glass': '▢', + 'orange_stained_glass': '▢', + 'black_stained_glass': '▢', + 'white_stained_glass': '▢', + 'yellow_stained_glass': '▢', + 'red_stained_glass': '▢', + 'magenta_stained_glass': '▢', + 'light_blue_stained_glass': '▢', + 'acacia_button': '一', + 'acacia_fence': '丁', + 'acacia_fence_gate': '丂', + 'acacia_hanging_sign': '七', + 'acacia_leaves': '丄', + 'acacia_log': '丅', + 'acacia_planks': '丆', + 'acacia_pressure_plate': '万', + 'acacia_sapling': '丈', + 'acacia_shelf': '三', + 'acacia_sign': '上', + 'acacia_slab': '下', + 'acacia_stairs': '丌', + 'acacia_trapdoor': '不', + 'acacia_wall_hanging_sign': '与', + 'acacia_wall_sign': '丏', + 'acacia_wood': '丐', + 'activator_rail': '丑', + 'allium': '丒', + 'amethyst_block': '专', + 'amethyst_cluster': '且', + 'ancient_debris': '丕', + 'andesite': '世', + 'andesite_slab': '丗', + 'andesite_stairs': '丘', + 'andesite_wall': '丙', + 'anvil': '业', + 'attached_melon_stem': '丛', + 'attached_pumpkin_stem': '东', + 'azalea': '丝', + 'azalea_leaves': '丞', + 'azure_bluet': '丟', + 'bamboo': '丠', + 'bamboo_block': '両', + 'bamboo_button': '丢', + 'bamboo_fence': '丣', + 'bamboo_fence_gate': '两', + 'bamboo_hanging_sign': '严', + 'bamboo_mosaic': '並', + 'bamboo_mosaic_slab': '丧', + 'bamboo_mosaic_stairs': '丨', + 'bamboo_planks': '丩', + 'bamboo_pressure_plate': '个', + 'bamboo_sapling': '丫', + 'bamboo_shelf': '丬', + 'bamboo_sign': '中', + 'bamboo_slab': '丮', + 'bamboo_stairs': '丯', + 'bamboo_trapdoor': '丰', + 'bamboo_wall_hanging_sign': '丱', + 'bamboo_wall_sign': '串', + 'barrel': '丳', + 'barrier': '临', + 'basalt': '丵', + 'beacon': '丶', + 'bedrock': '丷', + 'bee_nest': '丸', + 'beehive': '丹', + 'beetroots': '为', + 'bell': '主', + 'big_dripleaf': '丼', + 'big_dripleaf_stem': '丽', + 'birch_button': '举', + 'birch_fence': '丿', + 'birch_fence_gate': '乀', + 'birch_hanging_sign': '乁', + 'birch_leaves': '乂', + 'birch_log': '乃', + 'birch_planks': '乄', + 'birch_pressure_plate': '久', + 'birch_sapling': '乆', + 'birch_shelf': '乇', + 'birch_sign': '么', + 'birch_slab': '义', + 'birch_stairs': '乊', + 'birch_trapdoor': '之', + 'birch_wall_hanging_sign': '乌', + 'birch_wall_sign': '乍', + 'birch_wood': '乎', + 'black_banner': '乏', + 'black_candle': '乐', + 'black_candle_cake': '乑', + 'black_carpet': '乒', + 'black_concrete': '乓', + 'black_concrete_powder': '乔', + 'black_glazed_terracotta': '乕', + 'black_shulker_box': '乖', + 'black_stained_glass_pane': '乗', + 'black_terracotta': '乘', + 'black_wall_banner': '乙', + 'black_wool': '乚', + 'blackstone': '乛', + 'blackstone_slab': '乜', + 'blackstone_stairs': '九', + 'blackstone_wall': '乞', + 'blue_banner': '也', + 'blue_candle': '习', + 'blue_candle_cake': '乡', + 'blue_carpet': '乢', + 'blue_concrete': '乣', + 'blue_concrete_powder': '乤', + 'blue_glazed_terracotta': '乥', + 'blue_ice': '书', + 'blue_orchid': '乧', + 'blue_shulker_box': '乨', + 'blue_stained_glass_pane': '乩', + 'blue_terracotta': '乪', + 'blue_wall_banner': '乫', + 'blue_wool': '乬', + 'bone_block': '乭', + 'bookshelf': '乮', + 'brain_coral': '乯', + 'brain_coral_block': '买', + 'brain_coral_fan': '乱', + 'brain_coral_wall_fan': '乲', + 'brewing_stand': '乳', + 'brick_slab': '乴', + 'brick_stairs': '乵', + 'brick_wall': '乶', + 'bricks': '乷', + 'brown_banner': '乸', + 'brown_candle': '乹', + 'brown_candle_cake': '乺', + 'brown_carpet': '乻', + 'brown_concrete': '乼', + 'brown_concrete_powder': '乽', + 'brown_glazed_terracotta': '乾', + 'brown_mushroom': '乿', + 'brown_mushroom_block': '亀', + 'brown_shulker_box': '亁', + 'brown_stained_glass_pane': '亂', + 'brown_terracotta': '亃', + 'brown_wall_banner': '亄', + 'brown_wool': '亅', + 'bubble_column': '了', + 'bubble_coral': '亇', + 'bubble_coral_block': '予', + 'bubble_coral_fan': '争', + 'bubble_coral_wall_fan': '亊', + 'budding_amethyst': '事', + 'bush': '二', + 'cactus': '亍', + 'cactus_flower': '于', + 'cake': '亏', + 'calcite': '亐', + 'calibrated_sculk_sensor': '云', + 'campfire': '互', + 'candle': '亓', + 'candle_cake': '五', + 'carrots': '井', + 'carved_pumpkin': '亖', + 'cauldron': '亗', + 'cave_vines': '亘', + 'cave_vines_plant': '亙', + 'chain_command_block': '亚', + 'cherry_button': '些', + 'cherry_fence': '亜', + 'cherry_fence_gate': '亝', + 'cherry_hanging_sign': '亞', + 'cherry_leaves': '亟', + 'cherry_log': '亠', + 'cherry_planks': '亡', + 'cherry_pressure_plate': '亢', + 'cherry_sapling': '亣', + 'cherry_shelf': '交', + 'cherry_sign': '亥', + 'cherry_slab': '亦', + 'cherry_stairs': '产', + 'cherry_trapdoor': '亨', + 'cherry_wall_hanging_sign': '亩', + 'cherry_wall_sign': '亪', + 'cherry_wood': '享', + 'chipped_anvil': '京', + 'chiseled_bookshelf': '亭', + 'chiseled_copper': '亮', + 'chiseled_deepslate': '亯', + 'chiseled_nether_bricks': '亰', + 'chiseled_polished_blackstone': '亱', + 'chiseled_quartz_block': '亲', + 'chiseled_red_sandstone': '亳', + 'chiseled_resin_bricks': '亴', + 'chiseled_sandstone': '亵', + 'chiseled_stone_bricks': '亶', + 'chiseled_tuff': '亷', + 'chiseled_tuff_bricks': '亸', + 'chorus_flower': '亹', + 'chorus_plant': '人', + 'clay': '亻', + 'closed_eyeblossom': '亼', + 'coal_block': '亽', + 'coal_ore': '亾', + 'coarse_dirt': '亿', + 'cobbled_deepslate': '什', + 'cobbled_deepslate_slab': '仁', + 'cobbled_deepslate_stairs': '仂', + 'cobbled_deepslate_wall': '仃', + 'cobblestone': '仄', + 'cobblestone_slab': '仅', + 'cobblestone_stairs': '仆', + 'cobblestone_wall': '仇', + 'cobweb': '仈', + 'cocoa': '仉', + 'command_block': '今', + 'comparator': '介', + 'composter': '仌', + 'conduit': '仍', + 'copper_bars': '从', + 'copper_block': '仏', + 'copper_bulb': '仐', + 'copper_chain': '仑', + 'copper_chest': '仒', + 'copper_golem_statue': '仓', + 'copper_grate': '仔', + 'copper_lantern': '仕', + 'copper_ore': '他', + 'copper_torch': '仗', + 'copper_trapdoor': '付', + 'copper_wall_torch': '仙', + 'cornflower': '仚', + 'cracked_deepslate_bricks': '仛', + 'cracked_deepslate_tiles': '仜', + 'cracked_nether_bricks': '仝', + 'cracked_polished_blackstone_bricks': '仞', + 'cracked_stone_bricks': '仟', + 'crafter': '仠', + 'creaking_heart': '仡', + 'creeper_head': '仢', + 'creeper_wall_head': '代', + 'crimson_button': '令', + 'crimson_fence': '以', + 'crimson_fence_gate': '仦', + 'crimson_fungus': '仧', + 'crimson_hanging_sign': '仨', + 'crimson_hyphae': '仩', + 'crimson_nylium': '仪', + 'crimson_planks': '仫', + 'crimson_pressure_plate': '们', + 'crimson_roots': '仭', + 'crimson_shelf': '仮', + 'crimson_sign': '仯', + 'crimson_slab': '仰', + 'crimson_stairs': '仱', + 'crimson_stem': '仲', + 'crimson_trapdoor': '仳', + 'crimson_wall_hanging_sign': '仴', + 'crimson_wall_sign': '仵', + 'crying_obsidian': '件', + 'cut_copper': '价', + 'cut_copper_slab': '仸', + 'cut_copper_stairs': '仹', + 'cut_red_sandstone': '仺', + 'cut_red_sandstone_slab': '任', + 'cut_sandstone': '仼', + 'cut_sandstone_slab': '份', + 'cyan_banner': '仾', + 'cyan_candle': '仿', + 'cyan_candle_cake': '伀', + 'cyan_carpet': '企', + 'cyan_concrete': '伂', + 'cyan_concrete_powder': '伃', + 'cyan_glazed_terracotta': '伄', + 'cyan_shulker_box': '伅', + 'cyan_stained_glass_pane': '伆', + 'cyan_terracotta': '伇', + 'cyan_wall_banner': '伈', + 'cyan_wool': '伉', + 'damaged_anvil': '伊', + 'dandelion': '伋', + 'dark_oak_button': '伌', + 'dark_oak_fence': '伍', + 'dark_oak_fence_gate': '伎', + 'dark_oak_hanging_sign': '伏', + 'dark_oak_leaves': '伐', + 'dark_oak_log': '休', + 'dark_oak_planks': '伒', + 'dark_oak_pressure_plate': '伓', + 'dark_oak_sapling': '伔', + 'dark_oak_shelf': '伕', + 'dark_oak_sign': '伖', + 'dark_oak_slab': '众', + 'dark_oak_stairs': '优', + 'dark_oak_trapdoor': '伙', + 'dark_oak_wall_hanging_sign': '会', + 'dark_oak_wall_sign': '伛', + 'dark_oak_wood': '伜', + 'dark_prismarine': '伝', + 'dark_prismarine_slab': '伞', + 'dark_prismarine_stairs': '伟', + 'daylight_detector': '传', + 'dead_brain_coral': '伡', + 'dead_brain_coral_block': '伢', + 'dead_brain_coral_fan': '伣', + 'dead_brain_coral_wall_fan': '伤', + 'dead_bubble_coral': '伥', + 'dead_bubble_coral_block': '伦', + 'dead_bubble_coral_fan': '伧', + 'dead_bubble_coral_wall_fan': '伨', + 'dead_bush': '伩', + 'dead_fire_coral': '伪', + 'dead_fire_coral_block': '伫', + 'dead_fire_coral_fan': '伬', + 'dead_fire_coral_wall_fan': '伭', + 'dead_horn_coral': '伮', + 'dead_horn_coral_block': '伯', + 'dead_horn_coral_fan': '估', + 'dead_horn_coral_wall_fan': '伱', + 'dead_tube_coral': '伲', + 'dead_tube_coral_block': '伳', + 'dead_tube_coral_fan': '伴', + 'dead_tube_coral_wall_fan': '伵', + 'decorated_pot': '伶', + 'deepslate': '伷', + 'deepslate_brick_slab': '伸', + 'deepslate_brick_stairs': '伹', + 'deepslate_brick_wall': '伺', + 'deepslate_bricks': '伻', + 'deepslate_coal_ore': '似', + 'deepslate_copper_ore': '伽', + 'deepslate_diamond_ore': '伾', + 'deepslate_emerald_ore': '伿', + 'deepslate_gold_ore': '佀', + 'deepslate_iron_ore': '佁', + 'deepslate_lapis_ore': '佂', + 'deepslate_redstone_ore': '佃', + 'deepslate_tile_slab': '佄', + 'deepslate_tile_stairs': '佅', + 'deepslate_tile_wall': '但', + 'deepslate_tiles': '佇', + 'detector_rail': '佈', + 'diamond_block': '佉', + 'diamond_ore': '佊', + 'diorite': '佋', + 'diorite_slab': '佌', + 'diorite_stairs': '位', + 'diorite_wall': '低', + 'dirt': '住', + 'dirt_path': '佐', + 'dispenser': '佑', + 'dragon_egg': '佒', + 'dragon_head': '体', + 'dragon_wall_head': '佔', + 'dried_ghast': '何', + 'dried_kelp_block': '佖', + 'dripstone_block': '佗', + 'dropper': '佘', + 'emerald_block': '余', + 'emerald_ore': '佚', + 'enchanting_table': '佛', + 'end_gateway': '作', + 'end_portal': '佝', + 'end_portal_frame': '佞', + 'end_rod': '佟', + 'end_stone': '你', + 'end_stone_brick_slab': '佡', + 'end_stone_brick_stairs': '佢', + 'end_stone_brick_wall': '佣', + 'end_stone_bricks': '佤', + 'exposed_chiseled_copper': '佥', + 'exposed_copper': '佦', + 'exposed_copper_bars': '佧', + 'exposed_copper_bulb': '佨', + 'exposed_copper_chain': '佩', + 'exposed_copper_chest': '佪', + 'exposed_copper_golem_statue': '佫', + 'exposed_copper_grate': '佬', + 'exposed_copper_lantern': '佭', + 'exposed_copper_trapdoor': '佮', + 'exposed_cut_copper': '佯', + 'exposed_cut_copper_slab': '佰', + 'exposed_cut_copper_stairs': '佱', + 'exposed_lightning_rod': '佲', + 'farmland': '佳', + 'fern': '佴', + 'fire': '併', + 'fire_coral': '佶', + 'fire_coral_block': '佷', + 'fire_coral_fan': '佸', + 'fire_coral_wall_fan': '佹', + 'firefly_bush': '佺', + 'flower_pot': '佻', + 'flowering_azalea': '佼', + 'flowering_azalea_leaves': '佽', + 'frogspawn': '佾', + 'frosted_ice': '使', + 'gilded_blackstone': '侀', + 'glass_pane': '侁', + 'glow_lichen': '侂', + 'glowstone': '侃', + 'gold_block': '侄', + 'gold_ore': '侅', + 'granite': '來', + 'granite_slab': '侇', + 'granite_stairs': '侈', + 'granite_wall': '侉', + 'grass_block': '侊', + 'gravel': '例', + 'gray_banner': '侌', + 'gray_candle': '侍', + 'gray_candle_cake': '侎', + 'gray_carpet': '侏', + 'gray_concrete': '侐', + 'gray_concrete_powder': '侑', + 'gray_glazed_terracotta': '侒', + 'gray_shulker_box': '侓', + 'gray_stained_glass_pane': '侔', + 'gray_terracotta': '侕', + 'gray_wall_banner': '侖', + 'gray_wool': '侗', + 'green_banner': '侘', + 'green_candle': '侙', + 'green_candle_cake': '侚', + 'green_carpet': '供', + 'green_concrete': '侜', + 'green_concrete_powder': '依', + 'green_glazed_terracotta': '侞', + 'green_shulker_box': '侟', + 'green_stained_glass_pane': '侠', + 'green_terracotta': '価', + 'green_wall_banner': '侢', + 'green_wool': '侣', + 'grindstone': '侤', + 'hanging_roots': '侥', + 'hay_block': '侦', + 'heavy_core': '侧', + 'heavy_weighted_pressure_plate': '侨', + 'honey_block': '侩', + 'honeycomb_block': '侪', + 'hopper': '侫', + 'horn_coral': '侬', + 'horn_coral_block': '侭', + 'horn_coral_fan': '侮', + 'horn_coral_wall_fan': '侯', + 'ice': '侰', + 'infested_chiseled_stone_bricks': '侱', + 'infested_cobblestone': '侲', + 'infested_cracked_stone_bricks': '侳', + 'infested_deepslate': '侴', + 'infested_mossy_stone_bricks': '侵', + 'infested_stone': '侶', + 'infested_stone_bricks': '侷', + 'iron_bars': '侸', + 'iron_block': '侹', + 'iron_chain': '侺', + 'iron_ore': '侻', + 'iron_trapdoor': '侼', + 'jack_o_lantern': '侽', + 'jigsaw': '侾', + 'jukebox': '便', + 'jungle_button': '俀', + 'jungle_fence': '俁', + 'jungle_fence_gate': '係', + 'jungle_hanging_sign': '促', + 'jungle_leaves': '俄', + 'jungle_log': '俅', + 'jungle_planks': '俆', + 'jungle_pressure_plate': '俇', + 'jungle_sapling': '俈', + 'jungle_shelf': '俉', + 'jungle_sign': '俊', + 'jungle_slab': '俋', + 'jungle_stairs': '俌', + 'jungle_trapdoor': '俍', + 'jungle_wall_hanging_sign': '俎', + 'jungle_wall_sign': '俏', + 'jungle_wood': '俐', + 'kelp': '俑', + 'kelp_plant': '俒', + 'ladder': '俓', + 'lapis_block': '俔', + 'lapis_ore': '俕', + 'large_amethyst_bud': '俖', + 'large_fern': '俗', + 'lava_cauldron': '俘', + 'leaf_litter': '俙', + 'lectern': '俚', + 'lever': '俛', + 'light': '俜', + 'light_blue_banner': '保', + 'light_blue_candle': '俞', + 'light_blue_candle_cake': '俟', + 'light_blue_carpet': '俠', + 'light_blue_concrete': '信', + 'light_blue_concrete_powder': '俢', + 'light_blue_glazed_terracotta': '俣', + 'light_blue_shulker_box': '俤', + 'light_blue_stained_glass_pane': '俥', + 'light_blue_terracotta': '俦', + 'light_blue_wall_banner': '俧', + 'light_blue_wool': '俨', + 'light_gray_banner': '俩', + 'light_gray_candle': '俪', + 'light_gray_candle_cake': '俫', + 'light_gray_carpet': '俬', + 'light_gray_concrete': '俭', + 'light_gray_concrete_powder': '修', + 'light_gray_glazed_terracotta': '俯', + 'light_gray_shulker_box': '俰', + 'light_gray_stained_glass_pane': '俱', + 'light_gray_terracotta': '俲', + 'light_gray_wall_banner': '俳', + 'light_gray_wool': '俴', + 'light_weighted_pressure_plate': '俵', + 'lightning_rod': '俶', + 'lilac': '俷', + 'lily_of_the_valley': '俸', + 'lily_pad': '俹', + 'lime_banner': '俺', + 'lime_candle': '俻', + 'lime_candle_cake': '俼', + 'lime_carpet': '俽', + 'lime_concrete': '俾', + 'lime_concrete_powder': '俿', + 'lime_glazed_terracotta': '倀', + 'lime_shulker_box': '倁', + 'lime_stained_glass_pane': '倂', + 'lime_terracotta': '倃', + 'lime_wall_banner': '倄', + 'lime_wool': '倅', + 'lodestone': '倆', + 'magenta_banner': '倇', + 'magenta_candle': '倈', + 'magenta_candle_cake': '倉', + 'magenta_carpet': '倊', + 'magenta_concrete': '個', + 'magenta_concrete_powder': '倌', + 'magenta_glazed_terracotta': '倍', + 'magenta_shulker_box': '倎', + 'magenta_stained_glass_pane': '倏', + 'magenta_terracotta': '倐', + 'magenta_wall_banner': '們', + 'magenta_wool': '倒', + 'magma_block': '倓', + 'mangrove_button': '倔', + 'mangrove_fence': '倕', + 'mangrove_fence_gate': '倖', + 'mangrove_hanging_sign': '倗', + 'mangrove_leaves': '倘', + 'mangrove_log': '候', + 'mangrove_planks': '倚', + 'mangrove_pressure_plate': '倛', + 'mangrove_propagule': '倜', + 'mangrove_roots': '倝', + 'mangrove_shelf': '倞', + 'mangrove_sign': '借', + 'mangrove_slab': '倠', + 'mangrove_stairs': '倡', + 'mangrove_trapdoor': '倢', + 'mangrove_wall_hanging_sign': '倣', + 'mangrove_wall_sign': '値', + 'mangrove_wood': '倥', + 'medium_amethyst_bud': '倦', + 'melon': '倧', + 'melon_stem': '倨', + 'moss_block': '倩', + 'moss_carpet': '倪', + 'mossy_cobblestone': '倫', + 'mossy_cobblestone_slab': '倬', + 'mossy_cobblestone_stairs': '倭', + 'mossy_cobblestone_wall': '倮', + 'mossy_stone_brick_slab': '倯', + 'mossy_stone_brick_stairs': '倰', + 'mossy_stone_brick_wall': '倱', + 'mossy_stone_bricks': '倲', + 'moving_piston': '倳', + 'mud': '倴', + 'mud_brick_slab': '倵', + 'mud_brick_stairs': '倶', + 'mud_brick_wall': '倷', + 'mud_bricks': '倸', + 'muddy_mangrove_roots': '倹', + 'mushroom_stem': '债', + 'mycelium': '倻', + 'nether_brick_fence': '值', + 'nether_brick_slab': '倽', + 'nether_brick_stairs': '倾', + 'nether_brick_wall': '倿', + 'nether_bricks': '偀', + 'nether_gold_ore': '偁', + 'nether_portal': '偂', + 'nether_quartz_ore': '偃', + 'nether_sprouts': '偄', + 'nether_wart': '偅', + 'nether_wart_block': '偆', + 'netherite_block': '假', + 'netherrack': '偈', + 'note_block': '偉', + 'oak_button': '偊', + 'oak_fence': '偋', + 'oak_fence_gate': '偌', + 'oak_hanging_sign': '偍', + 'oak_leaves': '偎', + 'oak_log': '偏', + 'oak_planks': '偐', + 'oak_pressure_plate': '偑', + 'oak_sapling': '偒', + 'oak_shelf': '偓', + 'oak_sign': '偔', + 'oak_slab': '偕', + 'oak_stairs': '偖', + 'oak_trapdoor': '偗', + 'oak_wall_hanging_sign': '偘', + 'oak_wall_sign': '偙', + 'oak_wood': '做', + 'observer': '偛', + 'obsidian': '停', + 'ochre_froglight': '偝', + 'open_eyeblossom': '偞', + 'orange_banner': '偟', + 'orange_candle': '偠', + 'orange_candle_cake': '偡', + 'orange_carpet': '偢', + 'orange_concrete': '偣', + 'orange_concrete_powder': '偤', + 'orange_glazed_terracotta': '健', + 'orange_shulker_box': '偦', + 'orange_stained_glass_pane': '偧', + 'orange_terracotta': '偨', + 'orange_tulip': '偩', + 'orange_wall_banner': '偪', + 'orange_wool': '偫', + 'oxeye_daisy': '偬', + 'oxidized_chiseled_copper': '偭', + 'oxidized_copper': '偮', + 'oxidized_copper_bars': '偯', + 'oxidized_copper_bulb': '偰', + 'oxidized_copper_chain': '偱', + 'oxidized_copper_chest': '偲', + 'oxidized_copper_golem_statue': '偳', + 'oxidized_copper_grate': '側', + 'oxidized_copper_lantern': '偵', + 'oxidized_copper_trapdoor': '偶', + 'oxidized_cut_copper': '偷', + 'oxidized_cut_copper_slab': '偸', + 'oxidized_cut_copper_stairs': '偹', + 'oxidized_lightning_rod': '偺', + 'packed_ice': '偻', + 'packed_mud': '偼', + 'pale_hanging_moss': '偽', + 'pale_moss_block': '偾', + 'pale_moss_carpet': '偿', + 'pale_oak_button': '傀', + 'pale_oak_fence': '傁', + 'pale_oak_fence_gate': '傂', + 'pale_oak_hanging_sign': '傃', + 'pale_oak_leaves': '傄', + 'pale_oak_log': '傅', + 'pale_oak_planks': '傆', + 'pale_oak_pressure_plate': '傇', + 'pale_oak_sapling': '傈', + 'pale_oak_shelf': '傉', + 'pale_oak_sign': '傊', + 'pale_oak_slab': '傋', + 'pale_oak_stairs': '傌', + 'pale_oak_trapdoor': '傍', + 'pale_oak_wall_hanging_sign': '傎', + 'pale_oak_wall_sign': '傏', + 'pale_oak_wood': '傐', + 'pearlescent_froglight': '傑', + 'peony': '傒', + 'petrified_oak_slab': '傓', + 'piglin_head': '傔', + 'piglin_wall_head': '傕', + 'pink_banner': '傖', + 'pink_candle': '傗', + 'pink_candle_cake': '傘', + 'pink_carpet': '備', + 'pink_concrete': '傚', + 'pink_concrete_powder': '傛', + 'pink_glazed_terracotta': '傜', + 'pink_petals': '傝', + 'pink_shulker_box': '傞', + 'pink_stained_glass_pane': '傟', + 'pink_terracotta': '傠', + 'pink_tulip': '傡', + 'pink_wall_banner': '傢', + 'pink_wool': '傣', + 'piston': '傤', + 'piston_head': '傥', + 'pitcher_crop': '傦', + 'pitcher_plant': '傧', + 'player_head': '储', + 'player_wall_head': '傩', + 'podzol': '傪', + 'pointed_dripstone': '傫', + 'polished_andesite': '催', + 'polished_andesite_slab': '傭', + 'polished_andesite_stairs': '傮', + 'polished_basalt': '傯', + 'polished_blackstone': '傰', + 'polished_blackstone_brick_slab': '傱', + 'polished_blackstone_brick_stairs': '傲', + 'polished_blackstone_brick_wall': '傳', + 'polished_blackstone_bricks': '傴', + 'polished_blackstone_button': '債', + 'polished_blackstone_pressure_plate': '傶', + 'polished_blackstone_slab': '傷', + 'polished_blackstone_stairs': '傸', + 'polished_blackstone_wall': '傹', + 'polished_deepslate': '傺', + 'polished_deepslate_slab': '傻', + 'polished_deepslate_stairs': '傼', + 'polished_deepslate_wall': '傽', + 'polished_diorite': '傾', + 'polished_diorite_slab': '傿', + 'polished_diorite_stairs': '僀', + 'polished_granite': '僁', + 'polished_granite_slab': '僂', + 'polished_granite_stairs': '僃', + 'polished_tuff': '僄', + 'polished_tuff_slab': '僅', + 'polished_tuff_stairs': '僆', + 'polished_tuff_wall': '僇', + 'poppy': '僈', + 'potatoes': '僉', + 'potted_acacia_sapling': '僊', + 'potted_allium': '僋', + 'potted_azalea_bush': '僌', + 'potted_azure_bluet': '働', + 'potted_bamboo': '僎', + 'potted_birch_sapling': '像', + 'potted_blue_orchid': '僐', + 'potted_brown_mushroom': '僑', + 'potted_cactus': '僒', + 'potted_cherry_sapling': '僓', + 'potted_closed_eyeblossom': '僔', + 'potted_cornflower': '僕', + 'potted_crimson_fungus': '僖', + 'potted_crimson_roots': '僗', + 'potted_dandelion': '僘', + 'potted_dark_oak_sapling': '僙', + 'potted_dead_bush': '僚', + 'potted_fern': '僛', + 'potted_flowering_azalea_bush': '僜', + 'potted_jungle_sapling': '僝', + 'potted_lily_of_the_valley': '僞', + 'potted_mangrove_propagule': '僟', + 'potted_oak_sapling': '僠', + 'potted_open_eyeblossom': '僡', + 'potted_orange_tulip': '僢', + 'potted_oxeye_daisy': '僣', + 'potted_pale_oak_sapling': '僤', + 'potted_pink_tulip': '僥', + 'potted_poppy': '僦', + 'potted_red_mushroom': '僧', + 'potted_red_tulip': '僨', + 'potted_spruce_sapling': '僩', + 'potted_torchflower': '僪', + 'potted_warped_fungus': '僫', + 'potted_warped_roots': '僬', + 'potted_white_tulip': '僭', + 'potted_wither_rose': '僮', + 'powder_snow': '僯', + 'powder_snow_cauldron': '僰', + 'powered_rail': '僱', + 'prismarine': '僲', + 'prismarine_brick_slab': '僳', + 'prismarine_brick_stairs': '僴', + 'prismarine_bricks': '僵', + 'prismarine_slab': '僶', + 'prismarine_stairs': '僷', + 'prismarine_wall': '僸', + 'pumpkin': '價', + 'pumpkin_stem': '僺', + 'purple_banner': '僻', + 'purple_candle': '僼', + 'purple_candle_cake': '僽', + 'purple_carpet': '僾', + 'purple_concrete': '僿', + 'purple_concrete_powder': '儀', + 'purple_glazed_terracotta': '儁', + 'purple_shulker_box': '儂', + 'purple_stained_glass_pane': '儃', + 'purple_terracotta': '億', + 'purple_wall_banner': '儅', + 'purple_wool': '儆', + 'purpur_block': '儇', + 'purpur_pillar': '儈', + 'purpur_slab': '儉', + 'purpur_stairs': '儊', + 'quartz_block': '儋', + 'quartz_bricks': '儌', + 'quartz_pillar': '儍', + 'quartz_slab': '儎', + 'quartz_stairs': '儏', + 'rail': '儐', + 'raw_copper_block': '儑', + 'raw_gold_block': '儒', + 'raw_iron_block': '儓', + 'red_banner': '儔', + 'red_candle': '儕', + 'red_candle_cake': '儖', + 'red_carpet': '儗', + 'red_concrete': '儘', + 'red_concrete_powder': '儙', + 'red_glazed_terracotta': '儚', + 'red_mushroom': '儛', + 'red_mushroom_block': '儜', + 'red_nether_brick_slab': '儝', + 'red_nether_brick_stairs': '儞', + 'red_nether_brick_wall': '償', + 'red_nether_bricks': '儠', + 'red_sand': '儡', + 'red_sandstone': '儢', + 'red_sandstone_slab': '儣', + 'red_sandstone_stairs': '儤', + 'red_sandstone_wall': '儥', + 'red_shulker_box': '儦', + 'red_stained_glass_pane': '儧', + 'red_terracotta': '儨', + 'red_tulip': '儩', + 'red_wall_banner': '優', + 'red_wool': '儫', + 'redstone_block': '儬', + 'redstone_lamp': '儭', + 'redstone_ore': '儮', + 'redstone_wall_torch': '儯', + 'reinforced_deepslate': '儰', + 'repeater': '儱', + 'repeating_command_block': '儲', + 'resin_block': '儳', + 'resin_brick_slab': '儴', + 'resin_brick_stairs': '儵', + 'resin_brick_wall': '儶', + 'resin_bricks': '儷', + 'resin_clump': '儸', + 'respawn_anchor': '儹', + 'rooted_dirt': '儺', + 'rose_bush': '儻', + 'sand': '儼', + 'sandstone': '儽', + 'sandstone_slab': '儾', + 'sandstone_stairs': '儿', + 'sandstone_wall': '兀', + 'scaffolding': '允', + 'sculk': '兂', + 'sculk_catalyst': '元', + 'sculk_sensor': '兄', + 'sculk_shrieker': '充', + 'sculk_vein': '兆', + 'sea_lantern': '兇', + 'sea_pickle': '先', + 'seagrass': '光', + 'short_dry_grass': '兊', + 'shroomlight': '克', + 'shulker_box': '兌', + 'skeleton_skull': '免', + 'skeleton_wall_skull': '兎', + 'slime_block': '兏', + 'small_amethyst_bud': '児', + 'small_dripleaf': '兑', + 'smooth_basalt': '兒', + 'smooth_quartz': '兓', + 'smooth_quartz_slab': '兔', + 'smooth_quartz_stairs': '兕', + 'smooth_red_sandstone': '兖', + 'smooth_red_sandstone_slab': '兗', + 'smooth_red_sandstone_stairs': '兘', + 'smooth_sandstone': '兙', + 'smooth_sandstone_slab': '党', + 'smooth_sandstone_stairs': '兛', + 'smooth_stone': '兜', + 'smooth_stone_slab': '兝', + 'sniffer_egg': '兞', + 'snow': '兟', + 'snow_block': '兠', + 'soul_campfire': '兡', + 'soul_fire': '兢', + 'soul_sand': '兣', + 'soul_soil': '兤', + 'soul_wall_torch': '入', + 'spawner': '兦', + 'sponge': '內', + 'spore_blossom': '全', + 'spruce_button': '兩', + 'spruce_fence': '兪', + 'spruce_fence_gate': '八', + 'spruce_hanging_sign': '公', + 'spruce_leaves': '六', + 'spruce_log': '兮', + 'spruce_planks': '兯', + 'spruce_pressure_plate': '兰', + 'spruce_sapling': '共', + 'spruce_shelf': '兲', + 'spruce_sign': '关', + 'spruce_slab': '兴', + 'spruce_stairs': '兵', + 'spruce_trapdoor': '其', + 'spruce_wall_hanging_sign': '具', + 'spruce_wall_sign': '典', + 'spruce_wood': '兹', + 'sticky_piston': '兺', + 'stone': '养', + 'stone_brick_slab': '兼', + 'stone_brick_stairs': '兽', + 'stone_brick_wall': '兾', + 'stone_bricks': '兿', + 'stone_button': '冀', + 'stone_pressure_plate': '冁', + 'stone_slab': '冂', + 'stone_stairs': '冃', + 'stonecutter': '冄', + 'stripped_acacia_log': '内', + 'stripped_acacia_wood': '円', + 'stripped_bamboo_block': '冇', + 'stripped_birch_log': '冈', + 'stripped_birch_wood': '冉', + 'stripped_cherry_log': '冊', + 'stripped_cherry_wood': '冋', + 'stripped_crimson_hyphae': '册', + 'stripped_crimson_stem': '再', + 'stripped_dark_oak_log': '冎', + 'stripped_dark_oak_wood': '冏', + 'stripped_jungle_log': '冐', + 'stripped_jungle_wood': '冑', + 'stripped_mangrove_log': '冒', + 'stripped_mangrove_wood': '冓', + 'stripped_oak_log': '冔', + 'stripped_oak_wood': '冕', + 'stripped_pale_oak_log': '冖', + 'stripped_pale_oak_wood': '冗', + 'stripped_spruce_log': '冘', + 'stripped_spruce_wood': '写', + 'stripped_warped_hyphae': '冚', + 'stripped_warped_stem': '军', + 'structure_block': '农', + 'structure_void': '冝', + 'sugar_cane': '冞', + 'sunflower': '冟', + 'suspicious_gravel': '冠', + 'suspicious_sand': '冡', + 'sweet_berry_bush': '冢', + 'tall_dry_grass': '冣', + 'tall_seagrass': '冤', + 'target': '冥', + 'terracotta': '冦', + 'test_block': '冧', + 'test_instance_block': '冨', + 'tnt': '冩', + 'torchflower': '冪', + 'torchflower_crop': '冫', + 'trial_spawner': '冬', + 'tripwire': '冭', + 'tripwire_hook': '冮', + 'tube_coral': '冯', + 'tube_coral_block': '冰', + 'tube_coral_fan': '冱', + 'tube_coral_wall_fan': '冲', + 'tuff': '决', + 'tuff_brick_slab': '冴', + 'tuff_brick_stairs': '况', + 'tuff_brick_wall': '冶', + 'tuff_bricks': '冷', + 'tuff_slab': '冸', + 'tuff_stairs': '冹', + 'tuff_wall': '冺', + 'turtle_egg': '冻', + 'twisting_vines': '冼', + 'twisting_vines_plant': '冽', + 'vault': '冾', + 'verdant_froglight': '冿', + 'vine': '净', + 'warped_button': '凁', + 'warped_fence': '凂', + 'warped_fence_gate': '凃', + 'warped_fungus': '凄', + 'warped_hanging_sign': '凅', + 'warped_hyphae': '准', + 'warped_nylium': '凇', + 'warped_planks': '凈', + 'warped_pressure_plate': '凉', + 'warped_roots': '凊', + 'warped_shelf': '凋', + 'warped_sign': '凌', + 'warped_slab': '凍', + 'warped_stairs': '凎', + 'warped_stem': '减', + 'warped_trapdoor': '凐', + 'warped_wall_hanging_sign': '凑', + 'warped_wall_sign': '凒', + 'warped_wart_block': '凓', + 'water_cauldron': '凔', + 'waxed_chiseled_copper': '凕', + 'waxed_copper_bars': '凖', + 'waxed_copper_block': '凗', + 'waxed_copper_bulb': '凘', + 'waxed_copper_chain': '凙', + 'waxed_copper_chest': '凚', + 'waxed_copper_golem_statue': '凛', + 'waxed_copper_grate': '凜', + 'waxed_copper_lantern': '凝', + 'waxed_copper_trapdoor': '凞', + 'waxed_cut_copper': '凟', + 'waxed_cut_copper_slab': '几', + 'waxed_cut_copper_stairs': '凡', + 'waxed_exposed_chiseled_copper': '凢', + 'waxed_exposed_copper': '凣', + 'waxed_exposed_copper_bars': '凤', + 'waxed_exposed_copper_bulb': '凥', + 'waxed_exposed_copper_chain': '処', + 'waxed_exposed_copper_chest': '凧', + 'waxed_exposed_copper_golem_statue': '凨', + 'waxed_exposed_copper_grate': '凩', + 'waxed_exposed_copper_lantern': '凪', + 'waxed_exposed_copper_trapdoor': '凫', + 'waxed_exposed_cut_copper': '凬', + 'waxed_exposed_cut_copper_slab': '凭', + 'waxed_exposed_cut_copper_stairs': '凮', + 'waxed_exposed_lightning_rod': '凯', + 'waxed_lightning_rod': '凰', + 'waxed_oxidized_chiseled_copper': '凱', + 'waxed_oxidized_copper': '凲', + 'waxed_oxidized_copper_bars': '凳', + 'waxed_oxidized_copper_bulb': '凴', + 'waxed_oxidized_copper_chain': '凵', + 'waxed_oxidized_copper_chest': '凶', + 'waxed_oxidized_copper_golem_statue': '凷', + 'waxed_oxidized_copper_grate': '凸', + 'waxed_oxidized_copper_lantern': '凹', + 'waxed_oxidized_copper_trapdoor': '出', + 'waxed_oxidized_cut_copper': '击', + 'waxed_oxidized_cut_copper_slab': '凼', + 'waxed_oxidized_cut_copper_stairs': '函', + 'waxed_oxidized_lightning_rod': '凾', + 'waxed_weathered_chiseled_copper': '凿', + 'waxed_weathered_copper': '刀', + 'waxed_weathered_copper_bars': '刁', + 'waxed_weathered_copper_bulb': '刂', + 'waxed_weathered_copper_chain': '刃', + 'waxed_weathered_copper_chest': '刄', + 'waxed_weathered_copper_golem_statue': '刅', + 'waxed_weathered_copper_grate': '分', + 'waxed_weathered_copper_lantern': '切', + 'waxed_weathered_copper_trapdoor': '刈', + 'waxed_weathered_cut_copper': '刉', + 'waxed_weathered_cut_copper_slab': '刊', + 'waxed_weathered_cut_copper_stairs': '刋', + 'waxed_weathered_lightning_rod': '刌', + 'weathered_chiseled_copper': '刍', + 'weathered_copper': '刎', + 'weathered_copper_bars': '刏', + 'weathered_copper_bulb': '刐', + 'weathered_copper_chain': '刑', + 'weathered_copper_chest': '划', + 'weathered_copper_golem_statue': '刓', + 'weathered_copper_grate': '刔', + 'weathered_copper_lantern': '刕', + 'weathered_copper_trapdoor': '刖', + 'weathered_cut_copper': '列', + 'weathered_cut_copper_slab': '刘', + 'weathered_cut_copper_stairs': '则', + 'weathered_lightning_rod': '刚', + 'weeping_vines': '创', + 'weeping_vines_plant': '刜', + 'wet_sponge': '初', + 'wheat': '刞', + 'white_banner': '刟', + 'white_candle': '删', + 'white_candle_cake': '刡', + 'white_carpet': '刢', + 'white_concrete': '刣', + 'white_concrete_powder': '判', + 'white_glazed_terracotta': '別', + 'white_shulker_box': '刦', + 'white_stained_glass_pane': '刧', + 'white_terracotta': '刨', + 'white_tulip': '利', + 'white_wall_banner': '刪', + 'white_wool': '别', + 'wildflowers': '刬', + 'wither_rose': '刭', + 'wither_skeleton_skull': '刮', + 'wither_skeleton_wall_skull': '刯', + 'yellow_banner': '到', + 'yellow_candle': '刱', + 'yellow_candle_cake': '刲', + 'yellow_carpet': '刳', + 'yellow_concrete': '刴', + 'yellow_concrete_powder': '刵', + 'yellow_glazed_terracotta': '制', + 'yellow_shulker_box': '刷', + 'yellow_stained_glass_pane': '券', + 'yellow_terracotta': '刹', + 'yellow_wall_banner': '刺', + 'yellow_wool': '刻', + 'zombie_head': '刼', + 'zombie_wall_head': '刽', +}; diff --git a/gateway/platforms/daemoncraft.py b/gateway/platforms/daemoncraft.py new file mode 100644 index 000000000000..ff4ceb1afac7 --- /dev/null +++ b/gateway/platforms/daemoncraft.py @@ -0,0 +1,1818 @@ +""" +DaemonCraft platform adapter for Hermes Gateway. + +Routes Minecraft chat (player whispers + world broadcasts) through the +Hermes AIAgent, while the agent_loop.py handles embodiment (movement, +quest engine, sensors). + +The adapter consumes the Bot API WebSocket and HTTP endpoints: + - WS /ws : inbound chat events (array snapshot) + - POST /chat/send : outbound text + - POST /tts/play : outbound TTS relay to dashboards + - GET /agent/log : recent loop turns for context injection +""" + +import asyncio +import datetime as _dt +import json +import logging +import os +import random +import time +import uuid +from pathlib import Path +from typing import Any, Dict, Optional, Set + +import aiohttp +from aiohttp import WSMsgType + +from gateway.config import Platform, PlatformConfig +from gateway.platforms.daemoncraft_antiloop import StuckPivotTracker +from gateway.platforms.daemoncraft_narrategate import ( + NarrateGateTracker, +) + +# --------------------------------------------------------------------------- +# CycleDetector — ported from daemoncraft agents/safety.py (stdlib-only) +# --------------------------------------------------------------------------- +import hashlib +import json as _json +from collections import deque +from dataclasses import dataclass, field +from typing import Deque + + +def _cd_canonicalize(args) -> str: + try: + if isinstance(args, str): + try: + args = _json.loads(args) + except Exception: + return args + return _json.dumps(args, sort_keys=True, default=str) + except Exception: + return repr(args) + + +def _cd_signature(name: str, args) -> str: + payload = f"{name}|{_cd_canonicalize(args)}".encode("utf-8") + return hashlib.sha256(payload).hexdigest()[:16] + + +@dataclass +class _CycleResult: + triggered: bool + sig: Optional[str] + count: int + window: int + action: str + + +@dataclass +class CycleDetector: + """Ring-buffer cycle detector for repeated tool-call patterns.""" + n: int = 4 + window: int = 6 + action: str = "warn" + _buf: Deque[str] = field(default_factory=deque) + _last_triggered_sig: Optional[str] = None + + def __post_init__(self) -> None: + self._buf = deque(maxlen=max(self.window, self.n)) + + def record(self, name: str, args) -> _CycleResult: + sig = _cd_signature(name, args) + self._buf.append(sig) + return self._evaluate() + + def _evaluate(self) -> _CycleResult: + if len(self._buf) < self.n: + return _CycleResult(False, None, 0, len(self._buf), self.action) + counts: Dict[str, int] = {} + for s in self._buf: + counts[s] = counts.get(s, 0) + 1 + top_sig, top_count = max(counts.items(), key=lambda kv: kv[1]) + if top_count >= self.n: + if top_sig == self._last_triggered_sig: + return _CycleResult(False, top_sig, top_count, len(self._buf), self.action) + self._last_triggered_sig = top_sig + return _CycleResult(True, top_sig, top_count, len(self._buf), self.action) + if self._last_triggered_sig and self._last_triggered_sig != top_sig: + self._last_triggered_sig = None + return _CycleResult(False, top_sig, top_count, len(self._buf), self.action) + + def reset(self) -> None: + self._buf.clear() + self._last_triggered_sig = None +from gateway.platforms.base import BasePlatformAdapter, MessageEvent, MessageType, SendResult +from gateway.session import SessionSource, build_session_key + +logger = logging.getLogger(__name__) + +META_NO_CLAMP = "_no_clamp" # Set in metadata to bypass gateway-side char clamping (used by TTS transcripts) + + +class DaemonCraftAdapter(BasePlatformAdapter): + """Gateway adapter for DaemonCraft (Minecraft bot API).""" + + def __init__(self, config: PlatformConfig): + super().__init__(config, Platform.DAEMONCRAFT) + self._bot_api_url: str = (config.extra or {}).get("bot_api_url", "") + self._bot_username: str = (config.extra or {}).get("bot_username", "") + self._profile: str = (config.extra or {}).get("profile", "") + self._allowed_users: Set[str] = set() + self._session: Optional[aiohttp.ClientSession] = None + self._ws_task: Optional[asyncio.Task] = None + self._last_seen_timestamp: int = 0 + self._shutdown_event = asyncio.Event() + self._world_names: Set[str] = set() # Track broadcast worlds for send() routing + self._ws_retry_count: int = 0 + self._voice_mode_default: str = "all" # DaemonCraft defaults to TTS for all replies + self._last_tts_time: float = 0.0 + self._tts_queue: list[dict] = [] # Dedup buffer for rapid-fire messages + self._cycle_detector: Optional[CycleDetector] = None + + # Plan tracking for heartbeat-driven progress evaluation and GC + self._plan_goal: Optional[str] = None + self._plan_tasks_snapshot: list = [] + self._plan_created_at: float = 0.0 + self._plan_last_progress_at: float = 0.0 + self._plan_gc_timeout: int = (config.extra or {}).get("plan_gc_timeout_seconds", 300) + self._turn_counter: int = 0 # Sequential turn counter for agent logs + self._last_idle_wake_up: float = 0.0 # Throttle idle wake-ups + self._task_signature_history: list = [] # Anti-loop watchdog: last N (action,status) tuples + self._task_loop_threshold: int = (config.extra or {}).get("task_loop_threshold", 4) + self._task_stale_seconds: int = (config.extra or {}).get("task_stale_seconds", 600) + # StuckPivotTracker: detect same-objective / no-progress thrash using + # judge verdicts + position. Threshold + bucket size are config-tunable. + self._stuck_pivot_tracker = StuckPivotTracker( + threshold=(config.extra or {}).get("stuck_pivot_threshold", 3), + bucket_size=(config.extra or {}).get("spatial_bucket_blocks", 5), + cooldown_seconds=(config.extra or {}).get("stuck_pivot_cooldown_seconds", 120.0), + ) + # NarrateGateTracker: detect when the L4 narrates a past-tense action + # that contradicts the most recent mc_* tool result. Records every tool + # result so the gateway can check the LLM's next assistant text against + # the verified tool outcome. Part of Opción B for t_0fa2c6dc. + self._narrate_gate_tracker = NarrateGateTracker( + reminder_cooldown_seconds=(config.extra or {}).get( + "narrate_reminder_cooldown_seconds", 30.0, + ), + ) + self._session_epoch: int = int(time.time() * 1_000_000) # microsecond resolution — practically zero collision risk between gateway restarts + + # Load allowlist by UUID (preferred) or username fallback. + raw_allow = os.getenv("DAEMONCRAFT_ALLOWED_USERS", "").strip() + if raw_allow: + self._allowed_users = {u.strip().lower() for u in raw_allow.split(",") if u.strip()} + + # Force group sessions per world (broadcasts must share context) + if config.extra is None: + config.extra = {} + config.extra.setdefault("group_sessions_per_user", False) + + def _group_chat_id(self, world: str = "world") -> str: + """Return a chat_id scoped to this bot so each bot has its own session.""" + return f"{world}:{self._bot_username}" + + def _is_group_chat_id(self, chat_id: str) -> bool: + """Check whether a chat_id is one of our group chat ids.""" + return chat_id in self._world_names or any( + chat_id.startswith(w + ":") for w in self._world_names + ) + + def invoke_hook(self, hook_name: str, **kwargs): + """Wrapper around the hermes_cli.plugins invoke_hook helper. + + The plugin system is hermes-cli-internal; this adapter sits + above the LLM loop and shouldn't import from hermes_cli + directly at module top (avoids a circular import when + hermes-cli imports back from the gateway package during + tests). We do a lazy import here. + """ + try: + from hermes_cli.plugins import invoke_hook as _invoke_hook + return _invoke_hook(hook_name, **kwargs) + except Exception as _he: + logging.debug(f"invoke_hook({hook_name}) failed: {_he}") + return iter(()) + + def _write_event_to_queue(self, event: dict) -> None: + """Write an event to the daemoncraft-events.jsonl bridge file. + + The agent_loop reads this file each tick and includes events in the + context stream, which the CLI can observe. + """ + try: + # Match agent_loop's path: ~/.hermes/sessions/-events.jsonl + bot_user = (os.getenv("MC_USERNAME") or self._bot_username or "CompAII") + queue_path = Path.home() / ".hermes" / "sessions" / f"{bot_user}-events.jsonl" + queue_path.parent.mkdir(parents=True, exist_ok=True) + with open(queue_path, "a") as f: + f.write(json.dumps(event) + "\n") + except Exception as e: + logger.warning("[DaemonCraft] Failed to write event to queue: %s", e) + + async def _is_lab_mode(self) -> bool: + """Check if the bot is in lab mode (explicit, not automagic). + + In lab mode, the gateway never spawns agent turns. All events + (chat, heartbeats) only add context to the stream file. + """ + try: + async with self._session.get( + f"{self._bot_api_url}/controller/mode", timeout=aiohttp.ClientTimeout(total=3) + ) as resp: + if resp.status == 200: + data = await resp.json() + mode_data = data.get("data") if data.get("ok") else {} + return mode_data.get("mode") == "lab" + except Exception: + pass + return False + + # ------------------------------------------------------------------ + # Connection lifecycle + # ------------------------------------------------------------------ + + async def connect(self) -> bool: + if not self._bot_api_url: + logger.error("[DaemonCraft] bot_api_url missing in platform config extra") + return False + if not self._bot_username: + logger.error("[DaemonCraft] bot_username missing in platform config extra") + return False + + self._last_seen_timestamp = int(time.time() * 1000) + self._shutdown_event.clear() + self._session = aiohttp.ClientSession() + # Cache the controller mode at connect() so the heartbeat + # classifier can short-circuit without an HTTP roundtrip on + # every heartbeat. Updated whenever the user toggles mode + # via /controller/mode. + # Set initial value to 0.0 so the FIRST heartbeat forces a + # refresh (the 5s threshold check would otherwise trust the + # stale "autonomous" default for up to 5s after a bot server + # that's slow to start). + self._last_mode_check = 0.0 + self._controller_mode_cache = "autonomous" # safe default + try: + async with self._session.get( + f"{self._bot_api_url}/controller/mode", + timeout=aiohttp.ClientTimeout(total=2.0), + ) as resp: + if resp.status == 200: + data = await resp.json() + if data.get("ok"): + self._controller_mode_cache = data.get("data", {}).get("mode", "autonomous") + logger.info("[DaemonCraft] controller_mode at connect: %s", self._controller_mode_cache) + else: + logger.warning("[DaemonCraft] controller_mode fetch returned status %d at connect; " + "defaulting to '%s' (will refresh on first heartbeat)", + resp.status, self._controller_mode_cache) + except Exception as _ce: + logger.warning("[DaemonCraft] controller_mode fetch failed at connect: %s; defaulting to '%s' " + "(will refresh on first heartbeat)", _ce, self._controller_mode_cache) + + n = int(os.getenv("MC_CYCLE_N", "0")) + window = int(os.getenv("MC_CYCLE_WINDOW", "20")) + action = os.getenv("MC_CYCLE_ACTION", "warn") + if n > 0: + self._cycle_detector = CycleDetector(n=n, window=window, action=action) + logger.info("[DaemonCraft] CycleDetector enabled: n=%d window=%d action=%s", n, window, action) + self._ws_task = asyncio.create_task(self._ws_loop()) + self._mark_connected() + logger.info("[DaemonCraft] Connected to %s as %s", self._bot_api_url, self._bot_username) + return True + + async def disconnect(self) -> None: + self._shutdown_event.set() + if self._ws_task: + self._ws_task.cancel() + try: + await self._ws_task + except asyncio.CancelledError: + pass + self._ws_task = None + if self._session: + await self._session.close() + self._session = None + self._mark_disconnected() + logger.info("[DaemonCraft] Disconnected") + + async def handle_message(self, event: MessageEvent) -> None: + """Handle a chat message, injecting heartbeat context if relevant. + + Sets the bot_api_url context variable so that any tools (today: + embodied_plan; previously: minecraft/altercraft) dispatched for + this message target the correct bot server. + + In lab mode, strip the re-entry "[System note: ...]" prefix that the + gateway prepends when restoring a session after a crash. The note would + otherwise wake the L4 with a turn containing only the previous state + summary, which is exactly the "no-op narration" pattern we want to + avoid in lab. Autonomous mode keeps the note (it's useful for + continuity when the L4 is meant to keep acting). + """ + from tools.bot_api_url_ctx import set_bot_api_url, reset_bot_api_url + if event.text and await self._is_lab_mode(): + stripped = self._strip_reentry_note(event.text) + if stripped != event.text: + logger.info( + "[DaemonCraft] Stripped re-entry note in lab mode " + "(%d → %d chars)", + len(event.text), len(stripped), + ) + event.text = stripped + token = set_bot_api_url(self._bot_api_url) + try: + await super().handle_message(event) + finally: + reset_bot_api_url(token) + + @staticmethod + def _strip_reentry_note(text: str) -> str: + """Remove the leading `[System note: ...]\n\n` block that gateway/run.py + prepends on session restore. Idempotent: if the text does not start + with that block, returns it unchanged. + """ + import re + if not text.startswith("[System note:"): + return text + return re.sub( + r"^\[System note:.*?\]\s*\n\n", + "", + text, + count=1, + flags=re.DOTALL, + ) + + # ------------------------------------------------------------------ + # WebSocket listener + # ------------------------------------------------------------------ + + async def _ws_loop(self) -> None: + ws_url = self._bot_api_url.replace("http://", "ws://").replace("https://", "wss://") + "/ws" + while not self._shutdown_event.is_set(): + try: + async with self._session.ws_connect(ws_url) as ws: + self._ws_retry_count = 0 + logger.info("[DaemonCraft] WebSocket connected") + while not self._shutdown_event.is_set(): + msg = await ws.receive(timeout=30) + if msg.type == WSMsgType.TEXT: + await self._on_ws_message(msg.data) + elif msg.type in (WSMsgType.CLOSED, WSMsgType.ERROR): + break + except asyncio.CancelledError: + raise + except Exception as e: + self._ws_retry_count += 1 + delay = min(2 ** self._ws_retry_count, 30) + jitter = random.random() # 0–1s uniform jitter + sleep_time = delay + jitter + logger.warning("[DaemonCraft] WebSocket error: %s — reconnecting in %.1fs", e, sleep_time) + await asyncio.sleep(sleep_time) + + async def _on_ws_message(self, data: str) -> None: + try: + payload = json.loads(data) + except json.JSONDecodeError: + return + + msg_type = payload.get("type") + if msg_type == "chat": + messages = payload.get("data", []) + if not isinstance(messages, list): + return + await self._handle_chat_batch(messages) + elif msg_type == "quest_event": + data = payload.get("data", {}) + await self._handle_quest_event(data) + elif msg_type == "blueprint_updated": + data = payload.get("data", {}) + await self._handle_blueprint_updated(data) + elif msg_type == "heartbeat_context": + data = payload.get("data", {}) + await self._handle_heartbeat_context(data) + elif msg_type == "action_result": + await self._handle_action_result(payload) + elif msg_type == "interrupt": + # Loop-to-gateway interrupt acknowledgment — no action needed + pass + elif msg_type == "status": + pass + else: + logger.debug("[DaemonCraft] Unknown WS message type: %s", msg_type) + + async def _handle_chat_batch(self, messages: list) -> None: + """Process a batch of chat messages with bot filtering and @mention classification. + + - Bot messages without @mention are silently dropped. + - Human @mentions are treated as urgent (interrupts loop + immediate response). + - All other human messages are queued normally. + """ + new_messages = [m for m in messages if m.get("time", 0) > self._last_seen_timestamp] + if not new_messages: + return + + for entry in new_messages: + self._last_seen_timestamp = max(self._last_seen_timestamp, entry.get("time", 0)) + + # Dynamically discover all known bots from cast configs. + # This is a live hook — no need to update .env files when bots change. + def _discover_known_bots() -> set[str]: + import yaml + from pathlib import Path as _Path + bots = set() + casts_dir = _Path.home() / "Projects" / "DaemonCraft" / "agents" / "casts" + try: + for cf in sorted(casts_dir.glob("*.yaml")): + cfg = yaml.safe_load(cf.read_text()) or {} + for a in cfg.get("agents", []): + name = a.get("name", "") + if name: + bots.add(name.strip().lower()) + except Exception: + pass + # Also check env override + override = os.getenv("MC_KNOWN_BOTS", "") + if override: + for u in override.split(","): + u = u.strip().lower() + if u: + bots.add(u) + return bots + + known_bots = _discover_known_bots() + + urgent_msgs = [] + accepted_msgs = [] + import re + + # Build two regexes: + # 1. @username! — URGENT interrupt (exclamation forces immediate response) + # 2. @username — normal steer (queued, doesn't interrupt) + urgent_re = re.compile(rf"\b@{re.escape(self._bot_username.lower())}!", re.IGNORECASE) + mention_re = re.compile(rf"\b@{re.escape(self._bot_username.lower())}\b", re.IGNORECASE) + + for entry in new_messages: + from_user = entry.get("from", "").lower() + msg_text = entry.get("message", "") + is_bot = from_user in known_bots + mentions_bot = bool(mention_re.search(msg_text)) + is_urgent = bool(urgent_re.search(msg_text)) + + if is_bot and not mentions_bot: + continue # Silently drop bot spam + + accepted_msgs.append(entry) + + # Only @username! (with exclamation) is urgent interrupt. + # @username without ! is steer — queued, doesn't abort current turn. + if is_urgent and not is_bot: + urgent_msgs.append(entry) + + # Interrupt the loop for urgent human @mentions before generating response + if urgent_msgs: + senders = ", ".join({m.get("from", "Player") for m in urgent_msgs}) + logger.info("[DaemonCraft] Urgent @mention from %s — interrupting loop", senders) + await self._interrupt_agent("urgent_mention") + elif accepted_msgs: + senders = ", ".join({m.get("from", "Player") for m in accepted_msgs}) + logger.info("[DaemonCraft] Chat from %s queued", senders) + + # Process all accepted messages through the gateway + for entry in accepted_msgs: + await self._handle_chat_entry(entry) + + async def _interrupt_agent(self, reason: str) -> None: + """POST /agent/interrupt to abort the loop's in-progress LLM turn.""" + try: + async with self._session.post( + f"{self._bot_api_url}/agent/interrupt", + json={"reason": reason}, + ) as resp: + if resp.status >= 400: + body = await resp.text() + logger.warning("[DaemonCraft] /agent/interrupt failed: %s %s", resp.status, body) + else: + logger.debug("[DaemonCraft] /agent/interrupt sent (%s)", reason) + except Exception as e: + logger.warning("[DaemonCraft] /agent/interrupt exception: %s", e) + + async def _force_pivot_interrupt(self, pivot_reason: str) -> None: + """Abort the L4's stuck turn and queue a pivot directive as a system + message. The system message is constructed so that it survives the + re-entry note filter (it's not a [System note: ...] prefix) and the + /-command filter (it doesn't start with /). The L4 receives the + directive at the start of its next turn and must radically change + category of action. + """ + # 1. Abort the in-progress LLM turn via the bot server interrupt endpoint. + await self._interrupt_agent(pivot_reason) + + # 2. Reset the tracker so the next turn starts fresh. + self._stuck_pivot_tracker.reset_turn() + + # 3. Inject the pivot directive as a system message into the L4. + # Mirrors the pattern used by plan_cancelled at line ~595. + source = self.build_source( + chat_id=self._group_chat_id(), + chat_name="world", + chat_type="group", + user_id="system", + user_name="System", + thread_id="world", + ) + source.profile = self._profile + event = MessageEvent( + text=f"[Pivot directive] {pivot_reason}", + message_type=MessageType.TEXT, + source=source, + raw_message={"pivot_reason": pivot_reason, "type": "stuck_pivot"}, + internal=True, + ) + await self.handle_message(event) + + async def _handle_quest_event(self, data: dict) -> None: + """Process a quest_event from the QuestEngine. + + Builds a narrative message and injects it into the gateway so the + AIAgent can respond to the player (narrate phase changes, etc.). + """ + message = data.get("message", "A quest event occurred.") + event_type = data.get("event_type", "quest_event") + from_phase = data.get("from_phase") + to_phase = data.get("to_phase") + + # Build a natural-language description for the gateway AIAgent + lines = [f"[Quest Event] {message}"] + if from_phase and to_phase: + lines.append(f"Phase transition: {from_phase} → {to_phase}") + elif event_type: + lines.append(f"Event type: {event_type}") + event_text = "\n".join(lines) + + logger.info("[DaemonCraft] Quest event: %s", event_text.replace("\n", " | ")) + + # Route to the world broadcast session (group chat) + source = self.build_source( + chat_id=self._group_chat_id(), + chat_name="world", + chat_type="group", + user_id="quest_engine", + user_name="QuestEngine", + thread_id="world", + ) + source.profile = self._profile + + event = MessageEvent( + text=event_text, + message_type=MessageType.TEXT, + source=source, + raw_message=data, + ) + await self.handle_message(event) + + async def _handle_blueprint_updated(self, data: dict) -> None: + """Process a blueprint_updated event from the dashboard. + + Notifies the gateway AIAgent that a blueprint was modified so it + can reload or acknowledge the change. + """ + name = data.get("name", "unknown") + saved_at = data.get("saved_at", 0) + + event_text = ( + f"[Blueprint Updated] The blueprint '{name}' was edited via the dashboard " + f"at {time.strftime('%H:%M:%S', time.localtime(saved_at / 1000))}. " + f"Use mc_story(action='load_blueprint', name='{name}') to reload the latest version." + ) + + logger.info("[DaemonCraft] Blueprint updated: %s", name) + + source = self.build_source( + chat_id=self._group_chat_id(), + chat_name="world", + chat_type="group", + user_id="dashboard", + user_name="Dashboard", + thread_id="world", + ) + source.profile = self._profile + + event = MessageEvent( + text=event_text, + message_type=MessageType.TEXT, + source=source, + raw_message=data, + ) + await self.handle_message(event) + + async def _handle_action_result(self, payload: dict) -> None: + """Forward action_result events to transform_tool_result hooks. + + Also record the result in the NarrateGateTracker so the next + assistant message can be checked for past-tense narration that + contradicts the verified tool outcome. Tied to t_a2c3facb: + consumes the typed outcome/category fields directly (not strings). + + t_f8481d90: ALSO augment the narration verification with a + synthetic world state injection that includes a visual pre-process + of the area affected by the action. This is the "soft discard" + — we don't strip the assistant text from history, but we give + the LLM a strong anchor (the visual) for re-narrating correctly + on the next turn. + """ + data = payload.get("data", {}) if isinstance(payload, dict) else {} + action_name = str(data.get("action") or "") + # TYPED fields from the server (see agents/bot/lib/typed_result.js). + # No string matching on result blobs. + outcome = str(data.get("outcome") or "unknown") + category = str(data.get("category") or "other") + target = data.get("target") + position_before = data.get("position_before") + position_after = data.get("position_after") + # Forward the original payload to the transform_tool_result hook + # for any consumer that wants the full record (judge, ok, ts, etc.) + import json as _json + result_str = _json.dumps(data) + # Tied to t_f8481d90: invoke_hook returns a sync iterator, not + # awaitable. Just iterate it (transform_tool_result is a + # fire-and-forget observer hook). + for _ in self.invoke_hook("transform_tool_result", tool_name="mc_action_result", result=result_str): + pass + + # NarrateGateTracker: capture this tool result with the typed fields. + # No substring matching, no heuristic classification. The server + # emits the outcome and category at the top level. + pos_before = None + pos_after = None + try: + pos_before = ( + (position_before.get("x"), position_before.get("y"), position_before.get("z")) + if isinstance(position_before, dict) else None + ) + pos_after = ( + (position_after.get("x"), position_after.get("y"), position_after.get("z")) + if isinstance(position_after, dict) else None + ) + self._narrate_gate_tracker.record_tool_result( + tool_name=action_name or "mc_action", + outcome=outcome, + action_category=category, + position_before=pos_before, + position_after=pos_after, + now=time.time(), + ) + except Exception as _ng_err: + logging.debug(f"NarrateGateTracker record failed: {_ng_err}") + + # t_f8481d90: build the visual-augmented narrate reminder. + # If the last assistant message had past-tense narration that + # contradicts this tool result, inject a synthetic world state + # with the visual pre-process of the affected area. Cap at 2 + # injections per turn to avoid infinite loops. + try: + last_assistant_text = await self._get_last_assistant_text() + if last_assistant_text: + mismatch = self._narrate_gate_tracker.detect_narrate_mismatch( + last_assistant_text, now=time.time() + ) + if mismatch and self._narrate_gate_tracker.should_inject_reminder(mismatch, now=time.time()): + await self._inject_narrate_mismatch_visual( + mismatch=mismatch, + action_name=action_name, + category=category, + target=target, + position_after=position_after, + last_assistant_text=last_assistant_text, + ) + except Exception as _v_err: + logging.debug(f"Narrate visual inject failed: {_v_err}") + + async def _get_last_assistant_text(self) -> str: + """Return the most recent assistant text from the world session. + + Used by t_f8481d90 (discard narrate) to compare narration + against the verified tool result. + """ + if not self._session_store: + return "" + session_id = self._get_world_session_id() + if not session_id: + return "" + try: + transcript = self._session_store.load_transcript(session_id) + for msg in reversed(transcript): + if msg.get("role") == "assistant": + content = msg.get("content", "") + if isinstance(content, list): + text_parts = [b.get("text", "") for b in content if b.get("type") == "text"] + return "\n".join(text_parts).strip() + return str(content or "").strip() + except Exception: + pass + return "" + + async def _inject_narrate_mismatch_visual( + self, + mismatch, + action_name: str, + category: str, + target, + position_after, + last_assistant_text: str, + ) -> None: + """Inject a synthetic world state that includes a visual pre-process + of the area affected by the action, plus an explicit reminder. + + Tied to t_f8481d90 (discard narrate + visual inject). This is + the "soft discard": we don't strip the assistant text from + history, but we give the LLM a strong anchor (the visual) for + re-narrating correctly on the next turn. The reminder text + names the specific narration that contradicted the tool result + and tells the LLM to anchor to the visual. + """ + # Fetch a visual of the affected area. For movement, it's the + # bot's current position. For build, it's the target cell. For + # mine, the target cell. + visual_block = "" + try: + if (position_after and isinstance(position_after, dict) + and self._session is not None): + bx, by, bz = ( + int(position_after.get("x", 0)), + int(position_after.get("y", 0)), + int(position_after.get("z", 0)), + ) + radius = 4 + url = ( + f"{self._bot_api_url}/blocks" + f"?x1={bx-radius}&y1={by-radius}&z1={bz-radius}" + f"&x2={bx+radius}&y2={by+radius}&z2={bz+radius}" + f"&format=visual" + ) + async with self._session.get(url, timeout=aiohttp.ClientTimeout(total=5)) as resp: + if resp.status == 200: + body = await resp.json() + visual_block = (body.get("data") or {}).get("text", "") or "" + if len(visual_block) > 2500: + visual_block = visual_block[:2500] + "\n...[truncated]" + except Exception as _v_err: + logging.debug(f"Failed to fetch visual for narrate mismatch: {_v_err}") + visual_block = "(visual fetch failed)" + + reminder_text = ( + f"[Verify-Before-Narrate] Your last narration said " + f"'{mismatch.snippet}' but the most recent tool result " + f"({action_name}) had outcome '{mismatch.actual_outcome}'. " + f"The narration contradicts the verified tool result. " + f"Below is the visual pre-process of the affected area " + f"(radius 4 around the bot). Anchor your next narration " + f"to this visual, not to your earlier intention. The visual " + f"is the ground truth." + ) + data = { + "kind": "narrate_mismatch_visual", + "reminder": reminder_text, + "tool_name": action_name, + "category": category, + "outcome": mismatch.actual_outcome, + "target": target, + "position_after": position_after, + "visual": visual_block, + "snippet": mismatch.snippet, + "mismatch_kind": mismatch.kind, + "ts": time.time(), + } + await self._inject_synthetic_world_state(data) + + async def _handle_heartbeat_context(self, data: dict) -> None: + """Process heartbeat_context with two-level event architecture. + + - Context-only updates: inject synthetic world-state into the session_store + silently. This is an adapter-internal observation artifact, not a normal + agent tool call, so it is exempt from any external authority/lease gate. + - Wake-up events: inject synthetic world-state + force an agent turn with + tool_choice="required". The agent MUST react with a tool call (or mc_no_op). + - Active plans: every heartbeat while a plan is active forces a wake_up so + the agent evaluates progress against the plan. + - L4 verdict (GAP #5): body_session.l4_verdict is formatted into prompt as + compact "[L4 last] ..." feedback line (observations+delta only, no prescription). + + In lab mode, drop heartbeats entirely. We don't update plan tracking, we + don't poll the watchdog, and we don't create the L4 session in the + session_store. The session is only created when a real user turn arrives + (chat message, dashboard event, or explicit /command from the operator). + This keeps lab mode truly dormant between operator actions. + """ + if await self._is_lab_mode(): + return + + # StuckPivotTracker: detect same-objective / no-progress thrash and + # interrupt the active L4 turn with a pivot message. Only fires in + # autonomous mode (lab already returned above). Runs on every + # heartbeat — cheap O(1) state update. + body_session = data.get("body_session") or {} + status = data.get("status") or {} + pending_judges = body_session.get("pending_judges") or [] + pivot_reason = self._stuck_pivot_tracker.record_heartbeat( + body_session=body_session, + status=status, + pending_judges=pending_judges, + now=time.time(), + ) + if pivot_reason: + logger.warning("[DaemonCraft] Stuck pivot triggered: %s", pivot_reason) + await self._force_pivot_interrupt(pivot_reason) + + plan = data.get("plan") or {} + await self._update_plan_tracking(plan) + + # Run plan garbage collection before classification. + # Tied to t_97b030a6 followup: in lab mode, plan GC is + # silenced too. The plan is just a record; lab mode means + # Nico is observing and not driving the bot, so a stale + # plan should not fire events. Without this guard, the + # auto-resumed session's stale plan GC'd and re-fired the + # L4 in a loop (52 API calls, 14 min, 0 user input). + if await self._is_lab_mode(): + # In lab mode, also clear any active plan so the L4 + # has no reference to the dead auto-resume plan. + if self._plan_goal is not None: + logger.info("[DaemonCraft] Lab mode: clearing stale plan '%s' on heartbeat", self._plan_goal[:40]) + self._plan_goal = None + self._plan_tasks_snapshot = [] + self._plan_created_at = 0.0 + self._plan_last_progress_at = 0.0 + # Drop the heartbeat entirely. + return + gc_reason = await self._maybe_gc_plan() + if gc_reason: + logger.info("[DaemonCraft] Plan GC: %s", gc_reason) + # Inject cancellation as a system event + await self._inject_synthetic_world_state({ + "type": "plan_cancelled", + "reason": gc_reason, + "timestamp": int(time.time() * 1000), + }) + # Force wake_up with the cancellation message + source = self.build_source( + chat_id=self._group_chat_id(), + chat_name="world", + chat_type="group", + user_id="system", + user_name="System", + thread_id="world", + ) + source.profile = self._profile + event = MessageEvent( + text=f"[System: {gc_reason} — set a new plan or continue with immediate actions.]", + message_type=MessageType.TEXT, + source=source, + raw_message={"gc_reason": gc_reason}, + internal=True, + ) + await self.handle_message(event) + return + + event_type = await self._classify_heartbeat_event(data) + logger.info("[DaemonCraft] Heartbeat classified as: %s", event_type) + + # World-state injection REMOVED — was flooding gAndy with scans every heartbeat, + # interrupting McCompaii's own embodied_plan calls. L4 scans when HE decides. + + if event_type == "context": + logger.debug("[DaemonCraft] Context-only heartbeat injected silently") + return + + # Skip wake_up if there's an active user session — single controller + if await self._is_lab_mode(): + logger.info("[DaemonCraft] Skipping wake_up: active user session detected") + return + + # Cycle guard — skip wake-up if loop is repeating embodied_plan calls + if await self._check_cycle("embodied_plan", {}): + return + + # Wake-up event: force an agent turn with tool_choice=required + plan_goal = self._plan_goal + body = data.get("body_session") or {} + + # Build enriched prompt from body_session + prompt_parts = [] + + # Header: trigger + classification + reason = body.get("heartbeat_reason", "unknown") + if reason and reason != "idle": + prompt_parts.append(f"[System: Body heartbeat — WAKE UP. Trigger: {reason}.") + else: + prompt_parts.append("[System: Body heartbeat — IDLE wake up.") + + # Body status — position only. Survival is L2's job, never L4's concern. + pos = body.get("position", {}) + pos_str = f"({pos.get('x', '?')}, {pos.get('y', '?')}, {pos.get('z', '?')})" if pos else "unknown" + prompt_parts.append(f" Body: {pos_str}. Your body handles survival automatically — you explore, document, build.\n") + prompt_parts.append(" Nico is a spectator watching your stream. You are alone in this world. Never wait for him, never go to him, never change plans for him. If he wants something he'll say it in chat. Until then: explore, build paths, document places. You are the protagonist. Act.\n") + # Body activity — narrative continuity, NOT a problem to solve. L2 fought? Fine. L2 ate? Fine. + # This is for your story when you chat with humans. Never plan around it. Your body already handled it. + body_act = (body.get("body_activity") or "").strip() + if body_act: + prompt_parts.append(f" [Info] Your body handled: {body_act}. Everything is fine — you focus on exploring.\n") + + # GAP #5: L4 last-action verdict (from judge) — injected into NEXT heartbeat only. + # Reports WHAT happened (outcome + delta), never prescribes next action (LLM must reason). + # Combines with L2 runner activity during the open-loop window for context. + l4v = body.get("l4_verdict") + if l4v and isinstance(l4v, dict): + l2_sum = (body.get("runner_activity") or {}).get("summary") or "" + l2_part = f" | L2: {l2_sum}" if l2_sum else "" + ago = l4v.get("seconds_ago", 0) + delta = l4v.get("delta", "0.0m") + rc = l4v.get("reason_code") or "" + outcome = l4v.get("outcome") or "?" + act = l4v.get("action") or "?" + # Compact one-line report; example: [L4 last] dig@540,115,-307: preempted RUNNER_ACTIVE delta=0.0m 14s ago | L2: 2 attacks, 1 flee + verdict_line = f"[L4 last] {act}: {outcome} {rc} delta={delta} {ago}s ago{l2_part}".strip() + prompt_parts.append(f" {verdict_line}.") + + + + # Action history (oldest -> newest) + actions = body.get("action_history") or [] + if actions: + a_strs = [f"{a.get('action', '?')}({a.get('status', '?')}, {a.get('secondsAgo', '?')}s ago)" for a in actions] + prompt_parts.append(f" Recent actions: {' -> '.join(a_strs)}.") + + # Active task + task_mode = body.get("mode", "idle") + last_action = body.get("last_action") + if last_action and task_mode not in ("idle", None): + prompt_parts.append(f" Active task: {last_action} ({task_mode}).") + else: + prompt_parts.append(" Active task: none.") + + # Plan context + if plan_goal: + prompt_parts.append(f" Plan: '{plan_goal}' — {len(self._plan_tasks_snapshot)} tasks.") + prompt_parts.append(" Evaluate progress based on the provided wake up event data. Continue, adjust, or wait.]") + else: + prompt_parts.append(" No active plan. START following your autonomous curriculum immediately. Take ONE concrete action now: gather, craft, build, or explore. Do not wait.]") + + prompt_text = "".join(prompt_parts) + + # Store for dashboard Bot Mind panel + self._last_prompt = prompt_text + + # Auto-consume pending judges before dispatching to L4 + pending_judges = body.get("pending_judges") or [] + if pending_judges and self._session: + try: + l4_ticks = [j["captured_at_tick"] for j in pending_judges if j.get("initiator") == "l4_agent"] + if l4_ticks: + async with self._session.post( + f"{self._bot_api_url}/judge/consume", + json={"ticks": l4_ticks}, + timeout=aiohttp.ClientTimeout(total=3), + ) as resp: + result = await resp.json() + logger.debug("[DaemonCraft] Consumed %d judge entries, %d remaining", + result.get("consumed", 0), result.get("remaining", 0)) + except Exception: + pass + + source = self.build_source( + chat_id=self._group_chat_id(), + chat_name="world", + chat_type="group", + user_id="system", + user_name="System", + thread_id="world", + ) + source.profile = self._profile + + event = MessageEvent( + text=prompt_text, + message_type=MessageType.TEXT, + source=source, + raw_message=data, + internal=True, + ) + await self.handle_message(event) + + async def _update_plan_tracking(self, plan: dict) -> None: + """Update internal plan snapshot and detect progress.""" + goal = plan.get("goal") + tasks = plan.get("tasks", []) + + if not goal: + # No active plan + self._plan_goal = None + self._plan_tasks_snapshot = [] + self._plan_created_at = 0.0 + self._plan_last_progress_at = 0.0 + return + + # Detect if this is a new plan + if goal != self._plan_goal: + self._plan_goal = goal + self._plan_tasks_snapshot = [dict(t) for t in tasks] + self._plan_created_at = time.time() + self._plan_last_progress_at = time.time() + logger.info("[DaemonCraft] New plan tracked: %s (%d tasks)", goal, len(tasks)) + return + + # Detect progress: compare task statuses + progress_made = False + if len(tasks) == len(self._plan_tasks_snapshot): + for old, new in zip(self._plan_tasks_snapshot, tasks): + if old.get("status") != new.get("status"): + progress_made = True + break + elif len(tasks) != len(self._plan_tasks_snapshot): + progress_made = True + + if progress_made: + self._plan_last_progress_at = time.time() + self._plan_tasks_snapshot = [dict(t) for t in tasks] + logger.debug("[DaemonCraft] Plan progress detected: %s", goal) + + async def _maybe_gc_plan(self) -> Optional[str]: + """Garbage-collect stale plans. Returns cancellation reason or None.""" + if not self._plan_goal: + return None + + now = time.time() + age = now - self._plan_created_at + since_progress = now - self._plan_last_progress_at + + # GC if plan is older than timeout AND no progress in timeout period + if age > self._plan_gc_timeout and since_progress > self._plan_gc_timeout: + reason = ( + f"Plan '{self._plan_goal}' cancelled after {int(age)}s " + f"with no progress for {int(since_progress)}s" + ) + # Clear plan on bot server + try: + async with self._session.post( + f"{self._bot_api_url}/plan/update", + json={"action": "clear_goal"}, + ) as resp: + if resp.status < 400: + logger.info("[DaemonCraft] Plan cleared on bot server") + except Exception as e: + logger.warning("[DaemonCraft] Failed to clear plan on bot server: %s", e) + + # Reset local tracking + self._plan_goal = None + self._plan_tasks_snapshot = [] + self._plan_created_at = 0.0 + self._plan_last_progress_at = 0.0 + return reason + + return None + + async def _classify_heartbeat_event(self, data: dict) -> str: + """Classify heartbeat as 'context' or 'wake_up'. + + Wake-up triggers: + - Bot is stuck on a movement task (task_stuck in status) + - Active plan exists (agent must evaluate progress every heartbeat) + - Health decreased from previous known value + - Nearby hostile entities (zombie, skeleton, creeper, spider) + - Explicit damage events in events list + + In lab mode, NO wake_up triggers fire. Lab mode means: the + agent only acts on user input. Heartbeats add to context but + do not spawn agent turns. This is so the human operator has + full control during testing. + """ + # Lab mode silences ALL wake_up triggers. Only user input + # (chat message) drives the L4. Tied to t_97b030a6 followup. + # The check below is async because we need a HTTP roundtrip + # to /controller/mode (the mode is in the bot server's memory + # and may have changed since connect()). Cached for 5s. + # FAIL-SAFE: if the fetch fails, assume lab mode. This is + # because the alternative (stale "autonomous" cache) caused + # a 52-API-call loop on 2026-06-02. Better to miss a wake-up + # in autonomous mode than to fire one in lab mode. + now = time.time() + if (now - getattr(self, "_last_mode_check", 0)) > 5.0: + self._last_mode_check = now + try: + async with self._session.get( + f"{self._bot_api_url}/controller/mode", + timeout=aiohttp.ClientTimeout(total=1.0), + ) as resp: + if resp.status == 200: + md = await resp.json() + if md.get("ok"): + actual = md.get("data", {}).get("mode", "lab") + self._controller_mode_cache = actual + else: + # ok=false means error, assume lab + self._controller_mode_cache = "lab" + else: + # HTTP error, assume lab + self._controller_mode_cache = "lab" + except Exception: + # Connection error, timeout, anything — assume lab + # (safer than assuming autonomous) + self._controller_mode_cache = "lab" + if getattr(self, "_controller_mode_cache", None) == "lab": + return "context" + + status = data.get("status") or {} + nearby = data.get("nearby") or {} + events = data.get("events") or [] + plan = data.get("plan") or {} + + # Stuck on movement task — force wake_up so agent can react + task_stuck = status.get("task_stuck") + if task_stuck: + events.append(f"Stuck: {task_stuck}") + return "wake_up" + + # Active plan — force wake_up so agent evaluates progress + if plan.get("goal"): + events.append(f"Plan progress check: {plan['goal']}") + return "wake_up" + + # Damage / health drop + current_health = status.get("health") + if current_health is not None and hasattr(self, "_last_health"): + if current_health < self._last_health: + logger.info("[DaemonCraft] Wake-up reason: health dropped %s -> %s", self._last_health, current_health) + self._last_health = current_health + return "wake_up" + if current_health is not None: + self._last_health = current_health + + # Explicit damage events + for ev in events: + ev_str = str(ev).lower() + if any(k in ev_str for k in ("damage", "hurt", "attack", "hit", "died", "killed")): + logger.info("[DaemonCraft] Wake-up reason: damage event '%s'", ev_str[:80]) + return "wake_up" + + # Death detected — force immediate wake-up so agent can react + body = data.get("body_session") or {} + current_deaths = body.get("deaths", 0) + if current_deaths > getattr(self, "_last_deaths", 0): + self._last_deaths = current_deaths + last = body.get("last_death") or {} + pos = last.get("position", {}) + logger.info("[DaemonCraft] Wake-up reason: death #%d at (%.1f, %.1f, %.1f)", + current_deaths, pos.get("x", 0), pos.get("y", 0), pos.get("z", 0)) + return "wake_up" + self._last_deaths = current_deaths + + # Nearby hostile mobs + hostile = {"zombie", "skeleton", "creeper", "spider", "enderman", "witch", "husk", "drowned", "phantom"} + for ent in nearby.get("entities", [])[:12]: + name = str(ent.get("name", ent) if isinstance(ent, dict) else ent).lower() + if any(h in name for h in hostile): + logger.info("[DaemonCraft] Wake-up reason: hostile entity '%s'", name) + return "wake_up" + + # Bot stuck — critical, needs immediate reaction + task = status.get("task") + if task and task.get("status") == "stuck": + logger.info("[DaemonCraft] Wake-up reason: bot stuck (%s)", task.get("error", "unknown")[:60]) + return "wake_up" + + # Anti-loop watchdog: same task signature across N heartbeats = stuck in a loop. + # The L4 may keep choosing the same action (e.g. deathpoint, mine, goto) without + # changing the world. We force a wake-up with an explicit loop message so the + # agent sees "you've been doing X for N heartbeats, pivot." + if task: + action = task.get("action", "") + tstatus = task.get("status", "") + elapsed = int(task.get("elapsed_s", 0) or 0) + signature = (action, tstatus, elapsed // 30) # bucket elapsed by 30s + self._task_signature_history.append(signature) + if len(self._task_signature_history) > self._task_loop_threshold * 2: + self._task_signature_history = self._task_signature_history[-self._task_loop_threshold * 2:] + # Stale task: status="done" but elapsed_s huge = L4 is not acting on results + if tstatus == "done" and elapsed > self._task_stale_seconds: + logger.info( + "[DaemonCraft] Wake-up reason: stale task '%s' done for %ds (>%ds threshold)", + action, elapsed, self._task_stale_seconds, + ) + return "wake_up" + # Repeated signature: same action+status (bucketed) for N consecutive heartbeats + if len(self._task_signature_history) >= self._task_loop_threshold: + recent = self._task_signature_history[-self._task_loop_threshold:] + if len(set(recent)) == 1: + logger.warning( + "[DaemonCraft] Wake-up reason: TASK LOOP detected — %s/%s repeated for %d heartbeats", + action, tstatus, self._task_loop_threshold, + ) + return "wake_up" + + # Idle heartbeat: wake up Steve so he can act autonomously + # (progress on achievements, scout, etc.) Throttle to avoid token spam. + now = time.time() + if now - self._last_idle_wake_up >= 90: + self._last_idle_wake_up = now + logger.info("[DaemonCraft] Wake-up reason: idle heartbeat (90s throttle)") + return "wake_up" + + return "context" + + async def _inject_synthetic_world_state(self, data: dict) -> None: + """Inject a fake assistant tool_call + tool result into the world session. + + The transcript shape still uses `mc_perceive` so downstream consumers can + reuse their existing parsing path, but the gateway treats this as an + internal observation event, not an actionable tool execution. + """ + if not self._session_store: + logger.debug("[DaemonCraft] No session_store available, skipping synthetic injection") + return + + session_id = self._get_world_session_id() + if not session_id: + logger.debug("[DaemonCraft] No world session found, skipping synthetic injection") + return + + tool_call_id = f"hb_{uuid.uuid4().hex[:12]}" + + # Build a concise JSON payload for the tool result + payload = json.dumps(data, ensure_ascii=False, default=str) + # Truncate if too large to avoid flooding context window + if len(payload) > 4000: + payload = payload[:4000] + "\n...[truncated]" + + assistant_msg = { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": tool_call_id, + "type": "function", + "function": {"name": "mc_perceive", "arguments": "{}"}, + } + ], + } + tool_msg = { + "role": "tool", + "tool_call_id": tool_call_id, + "content": payload, + } + + self._session_store.append_to_transcript(session_id, assistant_msg) + + # Run transform_tool_result hooks so plugins (e.g. altercraft scene-graph) + # can consume synthetic mc_perceive on the same path as real tool results. + try: + from hermes_cli.plugins import invoke_hook + for hook_result in invoke_hook( + "transform_tool_result", + tool_name="mc_perceive", + args={}, + result=payload, + task_id="", + session_id=session_id, + tool_call_id=tool_call_id, + duration_ms=0, + ): + if isinstance(hook_result, str): + payload = hook_result + tool_msg["content"] = payload + break + except Exception as _hook_exc: + logger.debug("[DaemonCraft] transform_tool_result hook error: %s", _hook_exc) + + self._session_store.append_to_transcript(session_id, tool_msg) + logger.info("[DaemonCraft] Synthetic world state injected into session %s", session_id) + + async def _inject_embodied_world_state(self, data: dict) -> None: + """Query the body (Gemma-Andy via embodied service) for world state. + + Instead of injecting raw bot data as synthetic mc_perceive, we ask the + body to scan the world and inject its processed response. This keeps + the architecture pure: Steve only knows the world through his body. + """ + if not self._session_store: + logger.debug("[DaemonCraft] No session_store, skipping embodied injection") + return + + session_id = self._get_world_session_id() + if not session_id: + logger.debug("[DaemonCraft] No world session, skipping embodied injection") + return + + embodied_url = os.environ.get("EMBODIED_SERVICE_URL", "http://localhost:7790") + intent = ( + "Scan the area. Report concisely: your position, the 5 most common " + "nearby blocks with counts, any entities (players, mobs) with distances, " + "inventory highlights (tools, key materials), and any hazards. " + "Keep the report under 600 characters." + ) + + tool_call_id = f"hb_{uuid.uuid4().hex[:12]}" + payload = None + ok = False + exc_info = None + + try: + async with aiohttp.ClientSession() as session: + async with session.post( + f"{embodied_url}/intent", + json={ + "intent": intent, + "autonomy_level": 1, + "deadline_seconds": 15, + "allowed_tools": [ + "scan_nearby", + "get_inventory", + "ask_clarification", + "raise_guardian_event", + "report_execution_error", + ], + }, + timeout=aiohttp.ClientTimeout(total=20), + ) as resp: + if resp.status == 200: + body = await resp.json() + ok = body.get("ok", False) + if ok and body.get("execution_results"): + payload = json.dumps(body, ensure_ascii=False, default=str) + elif body.get("plan", {}).get("body_plan"): + payload = json.dumps(body["plan"], ensure_ascii=False, default=str) + except Exception as exc: + logger.warning("[DaemonCraft] Embodied world-state query failed: %s", exc) + exc_info = str(exc) + + if not payload: + payload = json.dumps({ + "_note": "Body unresponsive — do not act as if it is responding. Wait for the next heartbeat.", + "error": exc_info or "embodied service unavailable", + }) + + if len(payload) > 4000: + payload = payload[:4000] + "\n...[truncated]" + + assistant_msg = { + "role": "assistant", + "content": None, + "tool_calls": [ + { + "id": tool_call_id, + "type": "function", + "function": {"name": "embodied_plan", "arguments": json.dumps({"intent": intent})}, + } + ], + } + tool_msg = { + "role": "tool", + "tool_call_id": tool_call_id, + "content": payload, + } + + self._session_store.append_to_transcript(session_id, assistant_msg) + self._session_store.append_to_transcript(session_id, tool_msg) + logger.info( + "[DaemonCraft] Embodied world-state injected (body ok=%s, %d chars) into session %s", + ok, len(payload), session_id, + ) + + def _get_world_session_id(self) -> Optional[str]: + """Resolve the session_id for the world broadcast session.""" + if not self._session_store: + return None + source = SessionSource( + platform=Platform.DAEMONCRAFT, + chat_id=self._group_chat_id(), + chat_type="group", + user_id=f"{self._bot_username}:{self._session_epoch}", # epoch prevents stale session reuse + thread_id="world", + ) + session_key = build_session_key( + source, + group_sessions_per_user=False, + thread_sessions_per_user=False, + ) + entries = getattr(self._session_store, "_entries", {}) + entry = entries.get(session_key) + if entry: + return entry.session_id + return None + + async def _handle_chat_entry(self, entry: dict) -> None: + from_ = entry.get("from", "") + if not from_: + return + if from_.lower() == self._bot_username.lower(): + return # Ignore self-echo + + # Authorization by UUID (preferred) or username fallback + sender_uuid = entry.get("uuid") + if self._allowed_users: + allowed = False + if sender_uuid and sender_uuid.lower() in self._allowed_users: + allowed = True + if from_.lower() in self._allowed_users: + allowed = True + if not allowed: + logger.debug("[DaemonCraft] Ignored message from unauthorized user: %s", from_) + return + + text = entry.get("message", "") + if not text: + return + + # Filter: any message that starts with "/" is a Minecraft server command + # (e.g. "/gamemode creative", "/tp", "/time set day"). These are operator + # instructions, not agent prompts — they must not be injected into the L4 + # session as a chat turn, otherwise the L4 reads them as user intents and + # acts. If we want to tell the L4 something about a command, we do it in + # natural language, not as a slash-command. + stripped = text.lstrip() + if stripped.startswith("/"): + logger.info( + "[DaemonCraft] Filtered /-command from %s (not injected to L4): %r", + from_, stripped[:80], + ) + self._write_event_to_queue({ + "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "src": "gateway", + "event": "filtered_command", + "player": from_, + "text": stripped, + }) + return + + is_whisper = entry.get("whisper", False) + is_private = entry.get("private", False) + world = entry.get("world", "world") + + # Session mapping + if is_whisper or is_private: + # 1:1 session + chat_id = from_ + chat_type = "dm" + thread_id = None + else: + # Group session per world, scoped to this bot + chat_id = self._group_chat_id(world) + chat_type = "group" + thread_id = world + self._world_names.add(world) + + source = self.build_source( + chat_id=chat_id, + chat_name=chat_id, + chat_type=chat_type, + user_id=sender_uuid or from_, + user_name=from_, + thread_id=thread_id, + ) + source.profile = self._profile + + # Always inject chat into the world session so autonomous McCompaii sees it. + # When lab mode is active (human controls via CLI), mark internal to suppress + # auto-response — the CLI session handles the reply. Still write to event queue + # for the CLI bridge so it can pick up the message text. + lab = await self._is_lab_mode() + if lab: + logger.info("[DaemonCraft] Chat from %s injected to world session (lab mode — no auto-response)", from_) + self._write_event_to_queue({ + "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), + "src": "gateway", + "event": "chat", + "player": from_, + "text": text, + }) + + event = MessageEvent( + text=text, + message_type=MessageType.TEXT, + source=source, + raw_message=entry, + internal=lab, # suppress auto-response when human is driving + ) + + await self.handle_message(event) + + # ------------------------------------------------------------------ + # Cycle detection + # ------------------------------------------------------------------ + + async def _check_cycle(self, tool_name: str, args: dict) -> bool: + """Check tool-call cycle. Returns True if cycle detected and action is 'interrupt'.""" + if self._cycle_detector is None: + return False + result = self._cycle_detector.record(tool_name, args) + if result.triggered: + if result.action == "interrupt": + logger.warning( + "[DaemonCraft] Cycle detected for '%s' (%d/%d) — interrupting agent", + tool_name, result.count, result.window, + ) + await self._interrupt_agent("cycle_detected") + return True + else: + logger.warning( + "[DaemonCraft] Cycle detected for '%s' (%d/%d) — action=%s", + tool_name, result.count, result.window, result.action, + ) + return False + + # ------------------------------------------------------------------ + # Dashboard feed (DC-123) + # ------------------------------------------------------------------ + + async def on_processing_complete(self, event, outcome) -> None: + """POST the last assistant turn to /agent/log so the dashboard Bot Mind panel populates. + + Before DC-112 the agent_loop posted turns directly. After DC-112 cognition + moved to the gateway but no one wired the log relay. This hook restores + visibility without touching the loop. + """ + if not self._bot_api_url or not self._session: + return + try: + session_id = self._get_world_session_id() + if not session_id or not self._session_store: + return + transcript = self._session_store.load_transcript(session_id) + # Find the last assistant message in the transcript + last_assistant = None + tool_calls = [] + for msg in reversed(transcript): + role = msg.get("role", "") + if role == "assistant" and last_assistant is None: + content = msg.get("content", "") + if isinstance(content, list): + # Extract text and tool_use blocks + text_parts = [b.get("text", "") for b in content if b.get("type") == "text"] + tool_calls = [ + {"name": b.get("name"), "input": b.get("input")} + for b in content if b.get("type") == "tool_use" + ] + last_assistant = "\n".join(text_parts).strip() + else: + last_assistant = str(content) + break + + if last_assistant is None and not tool_calls: + return + + await self._session.post( + f"{self._bot_api_url}/agent/log", + json={ + "turn": len(transcript), + "time": int(time.time() * 1000), + "prompt": "", + "response": last_assistant or "", + "tool_calls": tool_calls, + "error": None, + }, + ) + + # Write LLM response to event bridge so agent_loop can display it + self._write_event_to_queue({ + "type": "agent_response", + "text": last_assistant or "", + "tool_calls": [tc.get("name", "?") for tc in tool_calls], + "ts": int(time.time() * 1000), + }) + except Exception as e: + logger.debug("[DaemonCraft] on_processing_complete /agent/log post failed: %s", e) + + # DC-132 — emit a turn metric (best-effort; never raises). + # Latency: time since the last user/perceive message in the transcript, + # if we can find one. tokens_in/out: not yet exposed by AIAgent at this + # hook, so we emit zero placeholders rather than fabricate values. + try: + self._emit_metric( + "turn", + tokens_in=0, + tokens_out=0, + latency_ms=None, + tool_call_count=len(tool_calls), + ) + for tc in tool_calls: + self._emit_metric("tool", tool=tc.get("name") or "?", ok=True) + except Exception: + pass + + # ------------------------------------------------------------------ + # DC-132 — JSONL metrics (mirrors agents/agent_loop.py emitter in daemoncraft) + # ------------------------------------------------------------------ + + def _emit_metric(self, kind: str, **fields) -> None: + """Append a JSON line to ~/.hermes/metrics//.jsonl. + + Schema is documented in scripts/agent-metrics-report.py in the + daemoncraft repo. This is the gateway counterpart to the heartbeat + emitter in agent_loop.py — together they cover the four families + the report script aggregates. + + Cast comes from DAEMONCRAFT_METRICS_CAST env var; falls back to the + bot username so events still group sensibly if the operator hasn't + set it. No env var → emitter still fires under the username. + """ + try: + cast = os.getenv("DAEMONCRAFT_METRICS_CAST", "").strip() or self._bot_username or "daemoncraft" + metrics_root = Path(os.getenv("DAEMONCRAFT_METRICS_DIR", str(Path.home() / ".hermes" / "metrics"))) + now = _dt.datetime.utcnow() + cast_dir = metrics_root / cast + cast_dir.mkdir(parents=True, exist_ok=True) + path = cast_dir / f"{now.date().isoformat()}.jsonl" + record = { + "ts": now.isoformat(timespec="seconds") + "Z", + "cast": cast, + "agent": self._bot_username or "?", + "kind": kind, + **fields, + } + # Single os.write() with O_APPEND — POSIX-atomic for writes + # under PIPE_BUF (typically 4 KB on Linux). Prevents truncated + # lines under concurrent writers / mid-write process kill. + line = (json.dumps(record, separators=(",", ":")) + "\n").encode("utf-8") + fd = os.open(str(path), os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644) + try: + os.write(fd, line) + finally: + os.close(fd) + except Exception: + pass + + # ------------------------------------------------------------------ + # Outbound + # ------------------------------------------------------------------ + + async def send( + self, + chat_id: str, + content: str, + reply_to: Optional[str] = None, + metadata: Optional[Dict[str, Any]] = None, + ) -> SendResult: + # Log agent turn to bot server for dashboard display (skip heartbeat/context-only turns) + if content and content.strip() and content != "None": + await self._post_agent_log(content, metadata) + + # _world_names is populated lazily from inbound broadcasts. If the gateway + # initiates an outbound broadcast before any inbound from that world, this + # will default to DM (whisper). For now the agent only replies to inbound. + is_group = self._is_group_chat_id(chat_id) + payload: dict[str, Any] = {"message": content} + + if is_group: + payload["target"] = "broadcast" + else: + payload["target"] = chat_id + + try: + async with self._session.post( + f"{self._bot_api_url}/chat/send", + json=payload, + ) as resp: + if resp.status >= 400: + body = await resp.text() + logger.warning("[DaemonCraft] /chat/send failed: %s %s", resp.status, body) + return SendResult(success=False, error=f"HTTP {resp.status}: {body}") + except Exception as e: + logger.warning("[DaemonCraft] /chat/send exception: %s", e) + return SendResult(success=False, error=str(e), retryable=True) + + # DC-123: relay TTS to dashboard after successful outbound message. + system_tts_skip = {"steer", "gateway shutting down", "synthetic mc_perceive", "heartbeat", "mc_perceive", "queued", "⏳"} + is_system_msg = any(skip in content.lower() for skip in system_tts_skip) + if (content and content.strip() not in ("PASS", "") + and not is_system_msg + and not (metadata or {}).get("suppress_tts")): + asyncio.create_task(self._generate_and_relay_tts(content, chat_id)) + + return SendResult(success=True) + + async def _post_agent_log(self, content: str, metadata: Optional[Dict[str, Any]] = None) -> None: + """Post agent turn to bot server /agent/log for dashboard display.""" + try: + self._turn_counter += 1 + tool_calls = [] + if metadata and "tool_calls" in metadata: + tool_calls = metadata["tool_calls"] + payload = { + "turn": self._turn_counter, + "time": int(time.time() * 1000), + "prompt": getattr(self, "_last_prompt", ""), + "response": content, + "tool_calls": tool_calls, + "error": None, + } + async with self._session.post( + f"{self._bot_api_url}/agent/log", + json=payload, + ) as resp: + if resp.status >= 400: + body = await resp.text() + logger.debug("[DaemonCraft] /agent/log failed: %s %s", resp.status, body) + except Exception as e: + logger.debug("[DaemonCraft] /agent/log exception: %s", e) + + async def _generate_and_relay_tts(self, text: str, chat_id: str) -> None: + """Generate TTS for outbound text and relay audio to the dashboard. + + DC-123 fix: before DC-112 agent_loop called TTS explicitly. After DC-112 + the gateway owns cognition but the TTS relay was never wired. This method + closes that gap — it is called as a fire-and-forget task from send(). + """ + try: + from tools.tts_tool import text_to_speech_tool, check_tts_requirements + if not check_tts_requirements(): + return + import re as _re, json as _json + # Strip Minecraft formatting codes and markdown before synthesis. + clean = _re.sub(r'§[0-9a-fklmnor]', '', text) + clean = _re.sub(r'[*_`#\[\]()]', '', clean).strip() + if not clean: + return + # Edge-TTS stutter fix: prepend zero-width space to prevent first-word repetition. + clean = "\u200b" + clean + tts_result = await asyncio.to_thread(text_to_speech_tool, text=clean[:4000]) + tts_data = _json.loads(tts_result) + audio_path = tts_data.get("file_path") + if audio_path and os.path.exists(audio_path): + await self._copy_and_relay_tts(audio_path, chat_id) + try: + os.remove(audio_path) + except OSError: + pass + except Exception as e: + logger.debug("[DaemonCraft] TTS generation failed: %s", e) + + async def _copy_and_relay_tts(self, audio_path: str, chat_id: str) -> SendResult: + """Copy audio to shared TTS cache and POST /tts/play to dashboards.""" + try: + import shutil + + tts_dir = "/tmp/daemoncraft-tts" + os.makedirs(tts_dir, exist_ok=True) + filename = os.path.basename(audio_path) + dest = os.path.join(tts_dir, filename) + shutil.copy2(audio_path, dest) + + # Build public URL — bot API serves /tts/audio/:filename + audio_url = f"{self._bot_api_url}/tts/audio/{filename}" + + async with self._session.post( + f"{self._bot_api_url}/tts/play", + json={"audio_url": audio_url, "chat_id": chat_id}, + ) as resp: + if resp.status >= 400: + body = await resp.text() + logger.warning("[DaemonCraft] /tts/play failed: %s %s", resp.status, body) + return SendResult(success=False, error=f"HTTP {resp.status}: {body}") + return SendResult(success=True) + except Exception as e: + logger.warning("[DaemonCraft] /tts/play exception: %s", e) + return SendResult(success=False, error=str(e), retryable=True) + + async def play_tts(self, chat_id: str, audio_path: str, **kwargs) -> SendResult: + """Relay TTS audio to dashboards and send transcript to Minecraft chat.""" + result = await self._copy_and_relay_tts(audio_path, chat_id) + if not result.success: + return result + + # Also send the full text to Minecraft chat so players can read it + text = kwargs.get("text", "[Voice message]") + return await self.send(chat_id, text) + + async def send_typing(self, chat_id: str, metadata=None) -> None: + # Minecraft has no typing indicator — no-op + pass + + async def send_voice( + self, + chat_id: str, + audio_path: str, + caption: Optional[str] = None, + reply_to: Optional[str] = None, + **kwargs, + ) -> SendResult: + """Relay TTS audio to dashboards via the bot API.""" + return await self._copy_and_relay_tts(audio_path, chat_id) + + async def get_chat_info(self, chat_id: str) -> Dict[str, Any]: + chat_type = "group" if self._is_group_chat_id(chat_id) else "dm" + return {"name": chat_id, "type": chat_type, "chat_id": chat_id} + + +# ------------------------------------------------------------------ +# Requirements check +# ------------------------------------------------------------------ + +def check_daemoncraft_requirements() -> bool: + """DaemonCraft only needs aiohttp (already a core dep).""" + try: + import aiohttp # noqa: F401 + return True + except ImportError: + return False diff --git a/gateway/platforms/daemoncraft_antiloop.py b/gateway/platforms/daemoncraft_antiloop.py new file mode 100644 index 000000000000..fe20ea311404 --- /dev/null +++ b/gateway/platforms/daemoncraft_antiloop.py @@ -0,0 +1,268 @@ +"""L4 stuck detection for DaemonCraft — judge + spatial objective buckets. + +Tracks per-heartbeat whether the L4 is making progress on its current +objective. When the bot is stuck on the same target area for N consecutive +heartbeats AND position is not changing AND recent judges report +no_progress/error, returns a pivot reason string that the gateway uses +to interrupt the active L4 turn and queue a system message telling the L4 +to radically change its category of action. + +Not a replacement for the iteration cap (gateway/run.py) or the prompt +rules (McCompaii SOUL). It's the per-heartbeat enforcement that catches +"same target, no movement, just trying different action aliases" — the +pattern we observed empirically in the 100-iter thrash turn on 2026-06-01. +""" + +from __future__ import annotations + +import math +import re +from collections import deque +from dataclasses import dataclass, field +from typing import Deque, Dict, List, Optional, Tuple + + +# Movement-ish action names from server.js judgeIntents + mc_move aliases. +# These are the "trying to navigate" actions that get aliased into one +# another when the pathfinder gives up: goto / goto_near / follow / flee. +_MOVEMENT_ACTIONS = frozenset({ + "goto", "gotonear", "goto_near", "follow", "flee", "bg_goto", + "pathfind", "stop", "come", "navigate", +}) + +_COORD_RE = re.compile(r"(-?\d+)\s*,\s*(-?\d+)\s*,\s*(-?\d+)") + + +def _bucket(v, size: int = 5) -> int: + return int(math.floor(float(v) / size)) + + +def _pos_bucket(pos, size: int = 5) -> Tuple[int, int, int]: + if not pos: + return (0, 0, 0) + return ( + _bucket(pos.get("x", 0), size), + _bucket(pos.get("y", 0), size), + _bucket(pos.get("z", 0), size), + ) + + +def _cells_within(buckets: List[Tuple[int, int, int]], max_diff: int = 1) -> bool: + """True if all buckets are within max_diff of each other on every axis. + + Used by StuckPivotTracker's Path B to detect thrash that hops between + adjacent cells (e.g. bucket_size=3 with target rotating between + (190,40,-110) and (190,40,-111)) without the L4 actually moving the bot. + """ + if not buckets: + return True + xs = [b[0] for b in buckets] + ys = [b[1] for b in buckets] + zs = [b[2] for b in buckets] + return (max(xs) - min(xs) <= max_diff + and max(ys) - min(ys) <= max_diff + and max(zs) - min(zs) <= max_diff) + + +def _parse_target_from_task(task) -> Optional[Tuple[int, int, int]]: + if not task: + return None + action = str(task.get("action") or "") + m = _COORD_RE.search(action) + if m: + try: + return int(m.group(1)), int(m.group(2)), int(m.group(3)) + except (ValueError, IndexError): + return None + for key in ("target", "goal", "dest"): + t = task.get(key) + if isinstance(t, dict) and "x" in t: + try: + return int(t["x"]), int(t["y"]), int(t["z"]) + except (ValueError, KeyError, TypeError): + return None + return None + + +def _parse_target_from_judge(j: dict) -> Optional[Tuple[int, int, int]]: + intent = j.get("intent") + if isinstance(intent, dict) and intent.get("x") is not None: + try: + return int(intent["x"]), int(intent["y"]), int(intent["z"]) + except (ValueError, KeyError, TypeError): + return None + act = str(j.get("action") or "") + m = _COORD_RE.search(act) + if m: + try: + return int(m.group(1)), int(m.group(2)), int(m.group(3)) + except (ValueError, IndexError): + return None + return None + + +def _movement_class(action: str) -> str: + a = (action or "").lower().split("@")[0].strip() + if a in _MOVEMENT_ACTIONS or any(x in a for x in ("goto", "follow", "flee", "path")): + return "movement" + if a in ("dig", "mine", "collect", "tunnel", "spiral"): + return "mining" + if a in ("place", "fill", "build"): + return "build" + return a or "other" + + +@dataclass +class StuckPivotTracker: + """Detect same-objective / no-progress loops using judges + position. + + Call record_heartbeat() once per heartbeat. Returns a pivot reason + string when the threshold is met, else None. Stateful across calls + (tracks the last position bucket and a short objective streak). + Threading: a single instance is owned by the DaemonCraftAdapter and + is touched only from the gateway event loop coroutine, no lock needed. + + Detection signatures (refined sub-fix 4, 2026-06-02): + - bucket_size=3 (was 5) — finer spatial granularity catches thrash in + 3-5 block cells (the goto_near+place loop observed on 2026-06-01 06:36) + - obj_key includes action class — "place → goto_near → place" with + different coordinates but same action_class counts as the same + objective, so a thrash that changes target every call still fires + - Path B (sub-fix 4): position pinned to the same cell + all stuck + + at least one bad L4 judge (no_progress/error/preempted/blocked) in + recent window. Catches the rotate-action-class thrash. + - cooldown reduced to 60s (was 120s) — repeated thrash on a related + objective in the same minute should re-fire + """ + threshold: int = 3 + bucket_size: int = 3 + cooldown_seconds: float = 60.0 + _objective_streak: Deque[Tuple] = field(default_factory=lambda: deque(maxlen=8)) + _last_pos_bucket: Optional[Tuple[int, int, int]] = None + _fired_objective: Optional[Tuple] = None + _fired_at: float = 0.0 + + def reset_turn(self) -> None: + """Call at the start of a new L4 turn to clear the streak. + + Doesn't clear _fired_objective / _fired_at — those are cooldown + state that should survive across heartbeats in the same wall-clock + window. + """ + self._objective_streak.clear() + self._last_pos_bucket = None + + def record_heartbeat( + self, + *, + body_session: dict, + status: dict, + pending_judges: Optional[List[dict]] = None, + now: float = 0.0, + ) -> Optional[str]: + """Record a heartbeat and return a pivot reason if threshold met. + + Args: + body_session: latest body_session dict from the heartbeat payload. + status: latest status dict from the heartbeat payload. + pending_judges: list of judge entries pending for the L4 session. + now: current time.time() (for cooldown check). + Returns: + Pivot reason string if threshold met, else None. + """ + pending_judges = pending_judges or [] + pos = (body_session or {}).get("position") or (status or {}).get("position") or {} + pos_b = _pos_bucket(pos, self.bucket_size) + + task = (status or {}).get("task") or {} + target = _parse_target_from_task(task) + action_class = _movement_class(str(task.get("action") or "")) + if target is None: + for j in reversed(pending_judges): + if j.get("initiator") == "l4_agent": + target = _parse_target_from_judge(j) + if target: + action_class = action_class or _movement_class(str(j.get("action") or "")) + break + + if target is not None: + tgt_b = ( + _bucket(target[0], self.bucket_size), + _bucket(target[1], self.bucket_size), + _bucket(target[2], self.bucket_size), + ) + obj_key: Tuple = ("nav", action_class) + tgt_b + else: + obj_key = ("local", action_class) + pos_b + + moved = ( + self._last_pos_bucket is not None + and self._last_pos_bucket != pos_b + ) + self._last_pos_bucket = pos_b + + # Recent L4 judges: no_progress / error at same place + l4 = [j for j in pending_judges if j.get("initiator") == "l4_agent"] + bad_outcome = False + if l4: + last = l4[-1] + bad_outcome = last.get("outcome") in ("no_progress", "error") or ( + last.get("reason_code") in ("NO_MOVEMENT", "EXCEPTION") + ) + act = _movement_class(str(last.get("action") or "")) + if act == "movement" and bad_outcome: + obj_key = obj_key + ("movement_fail",) + + stuck_here = not moved or bad_outcome + self._objective_streak.append(obj_key + ("stuck" if stuck_here else "moved",)) + + if len(self._objective_streak) < self.threshold: + return None + + recent = list(self._objective_streak)[-self.threshold:] + # Strip the trailing 'stuck'/'moved' flag to compare objectives + base_keys = [r[:-1] if r[-1] in ("stuck", "moved") else r for r in recent] + + # Path A: all base_keys identical AND all stuck. The canonical pattern. + all_same_obj = len({b for b in base_keys}) == 1 + all_stuck = all(r[-1] == "stuck" for r in recent) + if not (all_same_obj and all_stuck): + # Path B (sub-fix 4): position is pinned to a small area AND + # all heartbeats are stuck AND at least one bad L4 judge in the + # recent window. Catches goto_near+place+goto_near thrash where + # the bot barely moves but the action class rotates and the + # target hops between adjacent buckets. + spatial_keys = [b[-3:] for b in base_keys] + all_in_one_cell = _cells_within(spatial_keys, max_diff=1) + l4_judges_recent = [j for j in pending_judges + if j.get("initiator") == "l4_agent"] + has_bad_judge = any( + j.get("outcome") in ("no_progress", "error", "preempted", "blocked") + for j in l4_judges_recent[-self.threshold:] + ) + if not (all_in_one_cell and all_stuck and has_bad_judge): + return None + + # Cooldown: don't re-fire on the same objective within cooldown window + if obj_key == self._fired_objective: + if now and (now - self._fired_at) < self.cooldown_seconds: + return None + # Also dedupe across the two paths: if we already fired for this cell + # recently, suppress. + spatial_only = obj_key[-3:] if len(obj_key) >= 5 else pos_b + if spatial_only == self._fired_objective: + if now and (now - self._fired_at) < self.cooldown_seconds: + return None + self._fired_objective = obj_key + self._fired_at = now + + if target is not None: + tgt_str = f"({target[0]:.0f},{target[1]:.0f},{target[2]:.0f})" + else: + tgt_str = f"cell {pos_b}" + return ( + f"STUCK_PIVOT: {self.threshold} heartbeats with no progress on " + f"target {tgt_str}. Radically change category of action — try " + f"mining, building, exploring elsewhere, crafting, or " + f"documenting. Do NOT retry movement to the same target area." + ) diff --git a/gateway/platforms/daemoncraft_narrategate.py b/gateway/platforms/daemoncraft_narrategate.py new file mode 100644 index 000000000000..1eb799d7feb9 --- /dev/null +++ b/gateway/platforms/daemoncraft_narrategate.py @@ -0,0 +1,373 @@ +#!/usr/bin/env python3 +""" +L4 narrate-gate for DaemonCraft — verify-before-narrate enforcement. + +Tracks the most recent mc_* tool result and the most recent body state +(queried via /status on the bot server). On each heartbeat, if the L4's +last assistant message narrates a past-tense action that contradicts +the most recent tool result, logs a warning and exposes a `should_remind` +flag the daemoncraft platform can use to inject a system reminder. + +This is Opción B of t_0fa2c6dc (2026-06-02). The SOUL already has +"Verify Before Narrate" (Opción A) but the LLM still confabulates when +the body state is stale. This module provides a runtime check. + +Architecture: +- NarrateGateTracker is a stateful observer owned by the DaemonCraft + adapter. Touched only from the gateway event loop coroutine. +- record_tool_result(action, args, result) is called from the + transform_tool_result hook for mc_action_result events. It captures + the last tool's claimed outcome (done, cancelled, stuck, no_progress). +- record_body_status(status) is called from the heartbeat path with + the latest bot /status response (compact summary: position, health, + holding, last_action). +- detect_narrate_mismatch(assistant_text) is called after each LLM + response. It returns None if the narration is consistent with the + last tool result, or a mismatch dict the platform can use to inject + a reminder. + +Heuristics (deliberately conservative — false positives are worse than +missed mismatches because the LLM can correct itself; bad reminders +undermine trust in the system): + +- PAST_TENSE_PATTERNS: regex matching common Minecraft narrations in + English and Spanish (the bot's two primary languages). +- Tool result categories: "did_move" (mc_move done with non-trivial + position delta), "did_not_move" (mc_move cancelled/stuck/no_progress), + "did_place" (mc_build place done), "did_not_place" (place failed). +- Mismatch examples: + - Tool said "did_not_move" but LLM narrates "I walked/avancé/recorrí" + - Tool said "did_not_place" but LLM narrates "I built/construí/puse" + - Tool said success but LLM narrates a different action category + +Limitations (known, will refine if mismatch rate is high): +- Only checks simple past-tense patterns. Misses: "I have arrived" + (present perfect), "Llegué" past tense forms in other conjugations, + idioms like "I should now be at..." (conditional, not past). +- Does not check spatial claims (e.g. "I am at X,Y,Z" vs actual + position). Could add later. +- Does not check temporal claims ("just now", "earlier"). Could add. +""" + +from __future__ import annotations + +import re +import time +from dataclasses import dataclass, field +from typing import Optional + + +# Past-tense narration patterns. Group them by what they imply: +# moved_past: the LLM claims it moved +# placed_past: the LLM claims it placed/built +# arrived_past: the LLM claims it arrived somewhere +# failed_past: the LLM claims a tool failed +PAST_TENSE_MOVED = re.compile( + r"\b(" + r"i walked|i moved|i went|i ran|i traveled|i travelled|" + r"i stepped|i headed|i advanced|i proceeded|i navigated|" + r"avancé|avanzó|avanzamos|avanzaron|" + r"caminé|caminó|caminamos|caminaron|" + r"fui|fue|fuimos|fueron|" + r"corrí|corrió|corrimos|corrieron|" + r"recorrí|recorrió|recorrimos|recorrieron|" + r"me moví|se movió|me desplacé|se desplazó" + r")\b", + re.IGNORECASE, +) + +PAST_TENSE_PLACED = re.compile( + r"\b(" + r"i placed|i built|i constructed|i put|i set|" + r"i dropped|i laid|i stacked|" + r"colocoqué|coloqué|colocamos|colocaron|" + r"construí|construyó|construimos|construyeron|" + r"puse|puso|pusimos|pusieron|" + r"edifiqué|edificó|edificamos|edificaron|" + r"hice una?|hizo una? hice un|hizo un" + r")\b", + re.IGNORECASE, +) + +PAST_TENSE_ARRIVED = re.compile( + r"\b(" + r"i arrived|i got to|i reached|i made it|" + r"i'm at|i am at|i'm now at|i am now at|" + r"llegué|llegó|llegamos|llegaron|" + r"ya estoy|ya está|ya estamos|ya están" + r")\b", + re.IGNORECASE, +) + +# Categorical outcomes from server.js typed_result schema (canonical). +# These are the typed outcome strings the server emits. See +# agents/bot/lib/typed_result.js for the canonical list. +# Categorized as: did the action actually accomplish its goal, or did it fail/stall? +MOVEMENT_SUCCESS_OUTCOMES = {"success", "displaced"} # bot moved (success) or fell (displaced, still moved) +MOVEMENT_FAILURE_OUTCOMES = {"cancelled", "stuck", "no_progress", "preempted", "error", "unknown"} +BUILD_SUCCESS_OUTCOMES = {"success", "displaced"} # block placed (displaced means fell after, but place succeeded) +BUILD_FAILURE_OUTCOMES = {"cancelled", "stuck", "no_progress", "preempted", "error", "unknown"} + + +@dataclass +class ToolResultRecord: + """Compact record of the most recent tool result. + + Built from the server's typed_result fields. outcome and category + are typed enums (not string-matched from a result blob). + """ + tool_name: str + claimed_outcome: str # "success" | "no_progress" | "cancelled" | "stuck" | "preempted" | "error" | "displaced" | "unknown" + action_category: str # "movement" | "build" | "mine" | "interact" | "craft" | "other" + position_before: Optional[tuple] = None # (x, y, z) before the tool + position_after: Optional[tuple] = None # (x, y, z) after the tool + timestamp: float = 0.0 + + +@dataclass +class BodyStateSnapshot: + """Compact record of the most recent body state, captured from /status.""" + position: Optional[tuple] = None # (x, y, z) + health: Optional[float] = None + holding: Optional[str] = None + last_action: Optional[str] = None # last_action.action from body_session + timestamp: float = 0.0 + + +@dataclass +class NarrateMismatch: + """Returned by detect_narrate_mismatch when narration contradicts reality.""" + kind: str # "moved_but_didnt" | "placed_but_didnt" | "arrived_but_didnt" + pattern: str # the past-tense pattern that matched + snippet: str # the offending text snippet + tool_name: str + actual_outcome: str + timestamp: float = 0.0 + + def reminder_text(self) -> str: + """Format a system reminder the gateway can inject.""" + return ( + f"[Verify-Before-Narrate] Last assistant message narrated " + f"'{self.snippet}' but the most recent tool result ({self.tool_name}) " + f"had outcome '{self.actual_outcome}'. The narration contradicts " + f"the verified tool result. Correct the narration in your next turn " + f"based on the actual outcome. If you are uncertain, call " + f"mc_perceive(type='status') before narrating." + ) + + +@dataclass +class NarrateGateTracker: + """Stateful observer. Touched only from the gateway event loop.""" + last_tool: Optional[ToolResultRecord] = None + last_body: Optional[BodyStateSnapshot] = None + _last_reminded_at: float = 0.0 + reminder_cooldown_seconds: float = 30.0 + + def record_tool_result( + self, + *, + tool_name: str, + outcome: str, + action_category: str = "other", + position_before: Optional[tuple] = None, + position_after: Optional[tuple] = None, + now: float = 0.0, + ) -> None: + """Capture a tool result for future narration comparison. + + outcome and action_category are TYPED fields (canonical enums + from agents/bot/lib/typed_result.js), not strings to parse. The + server emits these at the top level of every tool result. See + lib/typed_result.js for the canonical enum values. + """ + # Validate that the outcome is a known enum value. If the server + # emits an unknown outcome, treat as "unknown" — never crash. + if outcome not in _VALID_OUTCOMES: + outcome = "unknown" + if action_category not in _VALID_CATEGORIES: + action_category = "other" + self.last_tool = ToolResultRecord( + tool_name=tool_name, + claimed_outcome=outcome, + action_category=action_category, + position_before=position_before, + position_after=position_after, + timestamp=now or time.time(), + ) + + def record_body_status( + self, + *, + position: Optional[tuple] = None, + health: Optional[float] = None, + holding: Optional[str] = None, + last_action: Optional[str] = None, + now: float = 0.0, + ) -> None: + """Capture a body state snapshot from the heartbeat.""" + self.last_body = BodyStateSnapshot( + position=position, + health=health, + holding=holding, + last_action=last_action, + timestamp=now or time.time(), + ) + + def detect_narrate_mismatch( + self, + assistant_text: str, + now: float = 0.0, + ) -> Optional[NarrateMismatch]: + """Return a NarrateMismatch if the LLM's narration contradicts + the most recent tool result, else None. + + Conservative: only fires on clear past-tense claims that the + tool result contradicts. Misses implicit claims and conditional + language — those are not worth the false-positive cost. + """ + if not assistant_text or not self.last_tool: + return None + + text = assistant_text.strip() + if not text: + return None + + tool = self.last_tool + ac = tool.action_category + + # Check moved-claim against move-outcome + if ac == "movement": + if tool.claimed_outcome in MOVEMENT_FAILURE_OUTCOMES: + m = PAST_TENSE_MOVED.search(text) + if m: + return NarrateMismatch( + kind="moved_but_didnt", + pattern=m.group(0), + snippet=_snippet(text, m.start()), + tool_name=tool.tool_name, + actual_outcome=tool.claimed_outcome, + timestamp=now or time.time(), + ) + m2 = PAST_TENSE_ARRIVED.search(text) + if m2: + return NarrateMismatch( + kind="arrived_but_didnt", + pattern=m2.group(0), + snippet=_snippet(text, m2.start()), + tool_name=tool.tool_name, + actual_outcome=tool.claimed_outcome, + timestamp=now or time.time(), + ) + + # Check placed-claim against place-outcome + if ac == "build": + if tool.claimed_outcome in BUILD_FAILURE_OUTCOMES: + m = PAST_TENSE_PLACED.search(text) + if m: + return NarrateMismatch( + kind="placed_but_didnt", + pattern=m.group(0), + snippet=_snippet(text, m.start()), + tool_name=tool.tool_name, + actual_outcome=tool.claimed_outcome, + timestamp=now or time.time(), + ) + + return None + + def should_inject_reminder( + self, + mismatch: NarrateMismatch, + now: float = 0.0, + ) -> bool: + """Throttle reminders so we don't spam the LLM.""" + now = now or time.time() + if (now - self._last_reminded_at) < self.reminder_cooldown_seconds: + return False + self._last_reminded_at = now + return True + + def reset(self) -> None: + self.last_tool = None + self.last_body = None + + +def _classify_outcome_legacy() -> None: + """Legacy outcome classifier. REMOVED in t_a2c3facb refactor. + + The server now emits a typed `outcome` field directly (see + agents/bot/lib/typed_result.js). Consumers consume the typed field + directly via record_tool_result(outcome=...). No string matching. + + This stub remains as a marker — if you see code calling this, it + predates the typed_result refactor and should be migrated. + """ + raise NotImplementedError( + "Legacy _classify_outcome removed. Use the typed 'outcome' field " + "from agents/bot/lib/typed_result.js. See t_a2c3facb in kanban." + ) + + +# Canonical outcome + category sets (mirror agents/bot/lib/typed_result.js). +# Validated at record_tool_result() time. Unknown values are coerced to +# "unknown" / "other" respectively. +_VALID_OUTCOMES = { + "success", "no_progress", "cancelled", "preempted", + "stuck", "error", "displaced", "unknown", +} +_VALID_CATEGORIES = {"movement", "build", "mine", "interact", "craft", "other"} + + +def _snippet(text: str, center: int, width: int = 80) -> str: + """Return a context window around `center` in `text`.""" + start = max(0, center - width // 2) + end = min(len(text), center + width // 2) + return text[start:end].replace("\n", " ").strip() + + +# Action-to-category mapping mirrors agents/bot/lib/typed_result.js. Kept +# locally so this module can compute the category from the raw action +# name without a server round-trip. The two definitions must stay in +# sync; if you change one, change both. +_MOVEMENT_ACTIONS_FOR_NARRATE = frozenset({ + "goto", "gotonear", "goto_near", "follow", "flee", "bg_goto", + "pathfind", "stop", "come", "navigate", +}) +_BUILD_ACTIONS_FOR_NARRATE = frozenset({ + "place", "fill", "build", "interact", +}) +_MINE_ACTIONS_FOR_NARRATE = frozenset({ + "dig", "mine", "collect", "tunnel", "spiral", +}) +_INTERACT_ACTIONS_FOR_NARRATE = frozenset({ + "chat", "equip", "use", "eat", "drink", "sleep", "attack", "shoot", + "sneak", "shield", "toss", "pickup", "equip_item", +}) +_CRAFT_ACTIONS_FOR_NARRATE = frozenset({ + "craft", "smelt", "brew", "furnace_smelt", "view_craftable", +}) + + +def _narrate_action_category(action: str) -> str: + """Categorize a tool action for the NarrateGateTracker. + + Returns one of the typed categories: "movement", "build", "mine", + "interact", "craft", "other". Used by detect_narrate_mismatch to + pick the right past-tense pattern. Mirrors + lib/typed_result.js::categoryForAction — keep in sync. + """ + a = (action or "").lower().split("@")[0].strip() + if not a: + return "other" + if a in _MOVEMENT_ACTIONS_FOR_NARRATE or any(x in a for x in ("goto", "follow", "flee", "path", "navigat")): + return "movement" + if a in _MINE_ACTIONS_FOR_NARRATE: + return "mine" + if a in _BUILD_ACTIONS_FOR_NARRATE: + return "build" + if a in _INTERACT_ACTIONS_FOR_NARRATE: + return "interact" + if a in _CRAFT_ACTIONS_FOR_NARRATE: + return "craft" + return "other" diff --git a/gateway/run.py b/gateway/run.py index cd0834301d5d..808917f207f2 100644 --- a/gateway/run.py +++ b/gateway/run.py @@ -4605,8 +4605,9 @@ def _schedule_resume_pending_sessions(self, platform=None) -> int: Adapters that are not yet ready (adapter missing from ``self.adapters``) are skipped silently; their sessions stay ``resume_pending`` and will auto-resume on the next real user - message, or when the platform reconnects — the reconnect watcher - calls this again scoped to that ``platform``. + message, on the next gateway startup, or when the platform + reconnects — the reconnect watcher calls this again scoped to that + ``platform``. ``platform`` (a ``Platform``) restricts the pass to sessions that originated on that platform. The reconnect path passes it so a @@ -4614,6 +4615,15 @@ def _schedule_resume_pending_sessions(self, platform=None) -> int: re-touches another platform's in-flight recoveries. Sessions whose agent is already running are skipped regardless, so a session scheduled at startup is never resumed a second time. + + Tied to t_97b030a6 followup: in lab mode, auto-resume is + SKIPPED entirely. The lab mode is for human observation with + manual control. A stale auto-resume of an old session + (especially with a malformed plan) caused a 52-API-call, + 14-minute loop with 0 user input on 2026-06-02. The session + stays ``resume_pending`` and will only fire when a real user + turn arrives (chat message, dashboard event, or explicit + /command). Lab mode is truly dormant between operator actions. """ window = _auto_continue_freshness_window() try: @@ -6448,6 +6458,13 @@ def _create_adapter( return None return YuanbaoAdapter(config) + elif platform == Platform.DAEMONCRAFT: + from gateway.platforms.daemoncraft import DaemonCraftAdapter, check_daemoncraft_requirements + if not check_daemoncraft_requirements(): + logger.warning("DaemonCraft: aiohttp not installed") + return None + return DaemonCraftAdapter(config) + return None @@ -6599,7 +6616,7 @@ async def _handle_message(self, event: MessageEvent) -> Optional[str]: # Record rate limit so subsequent messages are silently ignored self.pairing_store._record_rate_limit(platform_name, source.user_id) return None - + # Intercept messages that are responses to a pending /update prompt. # The update process (detached) wrote .update_prompt.json; the watcher # forwarded it to the user; now the user's reply goes back via @@ -8626,7 +8643,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str, run_g # One-time prompt if no home channel is set for this platform # Skip for webhooks - they deliver directly to configured targets (github_comment, etc.) - if not history and source.platform and source.platform != Platform.LOCAL and source.platform != Platform.WEBHOOK: + if not history and source.platform and source.platform != Platform.LOCAL and source.platform != Platform.WEBHOOK and source.platform != Platform.DAEMONCRAFT: platform_name = source.platform.value env_key = _home_target_env_var(platform_name) if not os.getenv(env_key): @@ -8713,6 +8730,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str, run_g run_generation=run_generation, event_message_id=self._reply_anchor_for_event(event), channel_prompt=event.channel_prompt, + tool_choice=getattr(event, "tool_choice", None), ) # Stop persistent typing indicator now that the agent is done @@ -13236,6 +13254,7 @@ async def _run_agent( _interrupt_depth: int = 0, event_message_id: Optional[str] = None, channel_prompt: Optional[str] = None, + tool_choice: Optional[str] = None, ) -> Dict[str, Any]: """ Run the agent with the given message and context. @@ -14034,8 +14053,26 @@ def run_sync(): # (concurrency-safe). Keep os.environ as fallback for CLI/cron. os.environ["HERMES_SESSION_KEY"] = session_key or "" - # Read from env var or use default (same as CLI) - max_iterations = int(os.getenv("HERMES_MAX_ITERATIONS", "90")) + # DC-134: per-profile max_iterations / turn_timeout (DaemonCraft etc.) + # Load from active profile config so gateway-wide defaults are not + # forced on every platform. + _profile_name = getattr(source, "profile", None) or "" + _profile_max_turns = None + _profile_turn_timeout = None + if _profile_name: + try: + import yaml as _yaml + _profile_cfg_path = Path.home() / ".hermes" / "profiles" / _profile_name / "config.yaml" + if _profile_cfg_path.exists(): + _profile_cfg = _yaml.safe_load(_profile_cfg_path.read_text()) or {} + _agent_cfg = _profile_cfg.get("agent", {}) + _profile_max_turns = _agent_cfg.get("max_turns") + _profile_turn_timeout = _agent_cfg.get("turn_timeout_seconds") + except Exception: + pass + + max_iterations = int(_profile_max_turns or os.getenv("HERMES_MAX_ITERATIONS", "90")) + turn_timeout_seconds = int(_profile_turn_timeout or os.getenv("HERMES_TURN_TIMEOUT_SECONDS", "0") or 0) or None # Map platform enum to the platform hint key the agent understands. # Platform.LOCAL ("local") maps to "cli"; others pass through as-is. @@ -14073,6 +14110,32 @@ def run_sync(): "tools": [], } + # DC-134+: per-profile model/provider override (DaemonCraft wake-up routing, etc.) + # When source.profile is set, load that profile's config and override the global + # model/provider so wake-up turns use the embodied agent's profile. + if _profile_name: + try: + import yaml as _yaml + _profile_cfg_path = Path.home() / ".hermes" / "profiles" / _profile_name / "config.yaml" + if _profile_cfg_path.exists(): + _profile_cfg = _yaml.safe_load(_profile_cfg_path.read_text()) or {} + _profile_model_cfg = _profile_cfg.get("model", {}) + if _profile_model_cfg.get("default") and _profile_model_cfg.get("provider"): + model = _profile_model_cfg["default"] + runtime_kwargs["provider"] = _profile_model_cfg["provider"] + if _profile_model_cfg.get("base_url"): + runtime_kwargs["base_url"] = _profile_model_cfg["base_url"] + # Only override api_mode if explicitly set in profile + _profile_api_mode = _profile_model_cfg.get("api_mode") + if _profile_api_mode: + runtime_kwargs["api_mode"] = _profile_api_mode + logger.info( + "Profile model override: profile=%s model=%s provider=%s", + _profile_name, model, runtime_kwargs.get("provider"), + ) + except Exception: + logger.exception("Failed to load profile model config for %s", _profile_name) + pr = self._provider_routing reasoning_config = self._resolve_session_reasoning_config( source=source, @@ -14222,6 +14285,7 @@ def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: model=turn_route["model"], **turn_route["runtime"], max_iterations=max_iterations, + quiet_mode=True, verbose_logging=False, enabled_toolsets=enabled_toolsets, @@ -14258,6 +14322,11 @@ def _interim_assistant_cb(text: str, *, already_streamed: bool = False) -> None: # Per-message state — callbacks and reasoning config change every # turn and must not be baked into the cached agent constructor. + # If a profile is active, override the cached system prompt so the + # agent does not load the global SOUL.md from SQLite session storage. + _profile_name = getattr(source, "profile", None) + if _profile_name and combined_ephemeral: + agent._cached_system_prompt = combined_ephemeral agent.tool_progress_callback = progress_callback if tool_progress_enabled else None # Discord voice verbal-ack hook (fires once per turn on first tool # call; armed only when in a voice channel with the mixer running). @@ -14303,6 +14372,8 @@ def _notice_callback_sync(notice) -> None: agent.reasoning_config = reasoning_config agent.service_tier = self._service_tier agent.request_overrides = turn_route.get("request_overrides") or {} + if tool_choice: + agent.request_overrides["tool_choice"] = tool_choice _bg_review_release = threading.Event() _bg_review_pending: list[str] = [] @@ -14700,6 +14771,7 @@ def _approval_notify_sync(approval_data: dict) -> None: } if observed_group_context: _conversation_kwargs["persist_user_message"] = message + result = agent.run_conversation(_api_run_message, **_conversation_kwargs) finally: unregister_gateway_notify(_approval_session_key) diff --git a/hermes_cli/auth.py b/hermes_cli/auth.py index 38bcab929072..934b54d26c79 100644 --- a/hermes_cli/auth.py +++ b/hermes_cli/auth.py @@ -507,12 +507,11 @@ def get_anthropic_key() -> str: # api.moonshot.ai/v1 (the old default). Auto-detect when user hasn't set # KIMI_BASE_URL explicitly. # -# Note: the base URL intentionally has NO /v1 suffix. The /coding endpoint -# speaks the Anthropic Messages protocol, and the anthropic SDK appends -# "/v1/messages" internally — so "/coding" + SDK suffix → "/coding/v1/messages" -# (the correct target). Using "/coding/v1" here would produce -# "/coding/v1/v1/messages" (a 404). -KIMI_CODE_BASE_URL = "https://api.kimi.com/coding" +# Note: the /coding endpoint speaks the Anthropic Messages protocol. +# The OpenAI-compatible surface is at /coding/v1/chat/completions. +# Both /coding and /coding/v1 were valid at different times; current +# Kimi Coding Plan requires /coding/v1 (without it → 404). +KIMI_CODE_BASE_URL = "https://api.kimi.com/coding/v1" def _resolve_kimi_base_url(api_key: str, default_url: str, env_override: str) -> str: @@ -531,6 +530,318 @@ def _resolve_kimi_base_url(api_key: str, default_url: str, env_override: str) -> return default_url +# ============================================================================= +# Kimi CLI OAuth (read credentials installed by `kimi login`) +# ============================================================================= + +KIMI_CODE_CLIENT_ID = "17e5f671-d194-4dfb-9706-5516cb48c098" +KIMI_CODE_OAUTH_HOST = "https://auth.kimi.com" + + +def _kimi_cli_credentials_path() -> Path: + return Path.home() / ".kimi" / "credentials" / "kimi-code.json" + + +def _kimi_cli_device_id_path() -> Path: + return Path.home() / ".kimi" / "device_id" + + +def _kimi_cli_version() -> str: + """Return installed kimi-cli version, or a sensible default.""" + try: + kimi_bin = shutil.which("kimi") + if kimi_bin: + result = subprocess.run( + [kimi_bin, "--version"], + capture_output=True, text=True, timeout=5, + ) + for part in result.stdout.strip().split(): + part = part.strip().rstrip(",") + if part and part[0].isdigit(): + return part + except Exception: + pass + return "1.37.0" + + +def _read_kimi_cli_credentials() -> Dict[str, Any]: + """Read OAuth credentials from the installed Kimi CLI.""" + cred_path = _kimi_cli_credentials_path() + if not cred_path.exists(): + raise AuthError( + "Kimi CLI credentials not found. Run 'kimi login' first.", + provider="kimi-coding", + code="kimi_auth_missing", + ) + try: + data = json.loads(cred_path.read_text(encoding="utf-8")) + except Exception as exc: + raise AuthError( + f"Failed to read Kimi CLI credentials from {cred_path}: {exc}", + provider="kimi-coding", + code="kimi_auth_read_failed", + ) from exc + if not isinstance(data, dict): + raise AuthError( + f"Invalid Kimi CLI credentials in {cred_path}.", + provider="kimi-coding", + code="kimi_auth_invalid", + ) + return data + + +def _save_kimi_cli_credentials(tokens: Dict[str, Any]) -> Path: + cred_path = _kimi_cli_credentials_path() + cred_path.parent.mkdir(parents=True, exist_ok=True) + tmp_path = cred_path.with_suffix(".tmp") + tmp_path.write_text(json.dumps(tokens, indent=2, sort_keys=True) + "\n", encoding="utf-8") + os.chmod(tmp_path, stat.S_IRUSR | stat.S_IWUSR) + tmp_path.replace(cred_path) + return cred_path + + +def _refresh_kimi_cli_credentials( + tokens: Dict[str, Any], + *, + base_url: str, + force_refresh: bool = False, + timeout_seconds: float = 20.0, +) -> Dict[str, Any]: + """Refresh Kimi CLI OAuth credentials and persist the updated token file.""" + refresh_token = str(tokens.get("refresh_token", "") or "").strip() + access_token = str(tokens.get("access_token", "") or "").strip() + + if access_token and not force_refresh and not _kimi_oauth_token_is_expired(tokens.get("expires_at")): + return { + "provider": "kimi-coding", + "api_key": access_token, + "base_url": base_url, + "source": "kimi-cli-oauth", + "auth_file": str(_kimi_cli_credentials_path()), + } + + if not refresh_token: + raise AuthError( + "Kimi CLI OAuth credentials are missing a refresh_token. Run `kimi login` to re-authenticate.", + provider="kimi-coding", + code="kimi_oauth_missing_refresh_token", + relogin_required=True, + ) + + timeout = httpx.Timeout(max(5.0, float(timeout_seconds))) + with httpx.Client(timeout=timeout, headers={"Accept": "application/json"}) as client: + response = client.post( + f"{KIMI_CODE_OAUTH_HOST.rstrip('/')}/api/oauth/token", + headers={"Content-Type": "application/x-www-form-urlencoded"}, + data={ + "grant_type": "refresh_token", + "refresh_token": refresh_token, + "client_id": KIMI_CODE_CLIENT_ID, + }, + ) + + if response.status_code != 200: + code = "kimi_oauth_refresh_failed" + message = f"Kimi token refresh failed with status {response.status_code}." + relogin_required = False + try: + err = response.json() + if isinstance(err, dict): + err_code = err.get("error") + if isinstance(err_code, str) and err_code.strip(): + code = err_code.strip() + err_desc = err.get("error_description") or err.get("message") + if isinstance(err_desc, str) and err_desc.strip(): + message = f"Kimi token refresh failed: {err_desc.strip()}" + except Exception: + pass + if code in {"invalid_grant", "invalid_token", "invalid_request"}: + relogin_required = True + if response.status_code in (401, 403): + relogin_required = True + raise AuthError( + message, + provider="kimi-coding", + code=code, + relogin_required=relogin_required, + ) + + try: + refresh_payload = response.json() + except Exception as exc: + raise AuthError( + "Kimi token refresh returned invalid JSON.", + provider="kimi-coding", + code="kimi_oauth_refresh_invalid_json", + relogin_required=True, + ) from exc + + if not isinstance(refresh_payload, dict): + raise AuthError( + "Kimi token refresh returned an invalid payload.", + provider="kimi-coding", + code="kimi_oauth_refresh_invalid_payload", + relogin_required=True, + ) + + refreshed_access = refresh_payload.get("access_token") + if not isinstance(refreshed_access, str) or not refreshed_access.strip(): + raise AuthError( + "Kimi token refresh response was missing access_token.", + provider="kimi-coding", + code="kimi_oauth_refresh_missing_access_token", + relogin_required=True, + ) + + next_refresh = str(refresh_payload.get("refresh_token", refresh_token) or refresh_token).strip() + expires_in_raw = refresh_payload.get("expires_in") + try: + expires_in = float(expires_in_raw) + except Exception: + expires_in = None + + updated = dict(tokens) + updated["access_token"] = refreshed_access.strip() + updated["refresh_token"] = next_refresh + if expires_in is not None and expires_in > 0: + updated["expires_at"] = time.time() + expires_in + updated["expires_in"] = expires_in + else: + updated["expires_at"] = tokens.get("expires_at", time.time() + 3600) + updated["expires_in"] = tokens.get("expires_in", 3600) + scope = refresh_payload.get("scope") + if isinstance(scope, str) and scope.strip(): + updated["scope"] = scope.strip() + token_type = refresh_payload.get("token_type") + if isinstance(token_type, str) and token_type.strip(): + updated["token_type"] = token_type.strip() + _save_kimi_cli_credentials(updated) + + return { + "provider": "kimi-coding", + "api_key": updated["access_token"], + "base_url": base_url, + "source": "kimi-cli-oauth-refresh", + "auth_file": str(_kimi_cli_credentials_path()), + } + + +def _kimi_oauth_token_is_expired(expires_at: Any, skew_seconds: int = 300) -> bool: + try: + exp = float(expires_at) + except Exception: + return True + return exp <= (time.time() + max(0, skew_seconds)) + + +def kimi_coding_default_headers() -> Dict[str, str]: + """Return the X-Msh-* headers that Kimi's coding API now requires.""" + import platform as _platform + import socket as _socket + + device_id = "" + device_path = _kimi_cli_device_id_path() + if device_path.exists(): + try: + device_id = device_path.read_text(encoding="utf-8").strip() + except Exception: + pass + + version = _kimi_cli_version() + + headers: Dict[str, str] = { + "User-Agent": f"KimiCLI/{version}", + "X-Msh-Platform": "kimi_cli", + "X-Msh-Version": version, + "X-Msh-Device-Name": _platform.node() or _socket.gethostname(), + "X-Msh-Device-Model": _platform.machine(), + "X-Msh-Os-Version": _platform.version(), + } + if device_id: + headers["X-Msh-Device-Id"] = device_id + return headers + + +def resolve_kimi_coding_runtime_credentials( + *, + prefer_cli_oauth: bool = True, + force_refresh: bool = False, + allow_api_key_fallback: bool = True, +) -> Dict[str, Any]: + """Resolve credentials for kimi-coding, preferring Kimi CLI OAuth.""" + base_url = os.getenv("KIMI_BASE_URL", "").strip().rstrip("/") + if not base_url: + base_url = KIMI_CODE_BASE_URL + + if prefer_cli_oauth: + try: + creds = _read_kimi_cli_credentials() + access_token = str(creds.get("access_token", "") or "").strip() + refresh_token = str(creds.get("refresh_token", "") or "").strip() + token_expired = _kimi_oauth_token_is_expired(creds.get("expires_at")) + + if access_token and not force_refresh and not token_expired: + return { + "provider": "kimi-coding", + "api_key": access_token, + "base_url": base_url, + "source": "kimi-cli-oauth", + "auth_file": str(_kimi_cli_credentials_path()), + } + + if refresh_token: + return _refresh_kimi_cli_credentials( + creds, + base_url=base_url, + force_refresh=force_refresh or token_expired or not access_token, + ) + + if access_token and not force_refresh: + return { + "provider": "kimi-coding", + "api_key": access_token, + "base_url": base_url, + "source": "kimi-cli-oauth", + "auth_file": str(_kimi_cli_credentials_path()), + } + + raise AuthError( + "Kimi CLI OAuth credentials are not usable. Run 'kimi login' to refresh them.", + provider="kimi-coding", + code="kimi_oauth_credentials_unusable", + relogin_required=True, + ) + except AuthError: + if not allow_api_key_fallback: + raise + logger.debug("Kimi CLI OAuth unavailable, falling back to API key.") + except Exception as exc: + if not allow_api_key_fallback: + raise AuthError( + f"Kimi CLI OAuth read failed: {exc}", + provider="kimi-coding", + code="kimi_oauth_read_failed", + relogin_required=True, + ) from exc + logger.debug("Kimi CLI OAuth read failed: %s", exc) + + api_key = os.getenv("KIMI_API_KEY", "").strip() + if api_key: + if not base_url: + base_url = _resolve_kimi_base_url(api_key, KIMI_CODE_BASE_URL, "") + return { + "provider": "kimi-coding", + "api_key": api_key, + "base_url": base_url, + "source": "env-api-key", + } + + raise AuthError( + "No Kimi credentials found. Set KIMI_API_KEY or run 'kimi login'.", + provider="kimi-coding", + code="kimi_auth_missing", + ) + _PLACEHOLDER_SECRET_VALUES = { "*", @@ -6043,6 +6354,24 @@ def resolve_api_key_provider_credentials(provider_id: str) -> Dict[str, Any]: if pconfig.base_url_env_var: env_url = os.getenv(pconfig.base_url_env_var, "").strip() + # Kimi OAuth fallback: when no env API key is set, try CLI OAuth credentials. + # This centralizes OAuth resolution so every consumer of this function + # (runtime_provider, auxiliary_client, etc.) gets OAuth automatically. + if not api_key and provider_id == "kimi-coding": + try: + oauth = resolve_kimi_coding_runtime_credentials(allow_api_key_fallback=False) + oauth_key = str(oauth.get("api_key", "") or "").strip() + if oauth_key: + api_key = oauth_key + key_source = str(oauth.get("source", "") or "").strip() or "kimi-cli-oauth" + oauth_url = str(oauth.get("base_url", "") or "").strip() + if oauth_url and not env_url: + env_url = oauth_url + except Exception: + # OAuth not configured or unreadable — fall through to empty credentials + # so callers that expect missing keys get them gracefully. + pass + if provider_id in {"kimi-coding", "kimi-coding-cn"}: base_url = _resolve_kimi_base_url(api_key, pconfig.inference_base_url, env_url) elif provider_id == "zai": diff --git a/hermes_cli/config.py b/hermes_cli/config.py index 7bb0b283035b..94fb1b3bacbc 100644 --- a/hermes_cli/config.py +++ b/hermes_cli/config.py @@ -1426,6 +1426,7 @@ def _ensure_hermes_home_managed(home: Path): # starts delegating, nudging the user toward the live spawn-tree # dashboard. Set false to suppress the hint. "tui_agents_nudge": True, + "ctrl_c_priority": "interrupt_agent", # "interrupt_agent" | "clear_input" "bell_on_complete": False, "show_reasoning": False, "streaming": False, diff --git a/hermes_cli/doctor.py b/hermes_cli/doctor.py index 79c41b03f15b..137e1577c81e 100644 --- a/hermes_cli/doctor.py +++ b/hermes_cli/doctor.py @@ -1832,7 +1832,9 @@ def _probe_apikey_provider(pname, env_vars, default_url, base_env, "User-Agent": _HERMES_USER_AGENT, } if base_url_host_matches(base, "api.kimi.com"): - headers["User-Agent"] = "claude-code/0.1.0" + from hermes_cli.auth import kimi_coding_default_headers + headers = kimi_coding_default_headers() + headers["Authorization"] = f"Bearer {key}" # Google's Generative Language API (generativelanguage.googleapis.com) # rejects ``Authorization: Bearer `` with 401 # ``ACCESS_TOKEN_TYPE_UNSUPPORTED`` — that header is reserved for @@ -1842,6 +1844,7 @@ def _probe_apikey_provider(pname, env_vars, default_url, base_env, if url and base_url_host_matches(url, "generativelanguage.googleapis.com"): headers.pop("Authorization", None) headers["x-goog-api-key"] = key + r = httpx.get(url, headers=headers, timeout=10) if ( pname == "Alibaba/DashScope" diff --git a/hermes_cli/kanban.py b/hermes_cli/kanban.py index 31c4bf68ae85..78c8f1ac2acf 100644 --- a/hermes_cli/kanban.py +++ b/hermes_cli/kanban.py @@ -26,6 +26,7 @@ from hermes_cli import kanban_db as kb from hermes_cli import kanban_swarm as ks +from hermes_cli import kanban_review as kr from hermes_cli.profiles import get_active_profile_name, get_profile_dir, seed_profile_skills @@ -844,6 +845,26 @@ def build_parser(parent_subparsers: argparse._SubParsersAction) -> argparse.Argu p_gc.add_argument("--log-retention-days", type=int, default=30, help="Delete worker log files older than N days (default: 30)") + # --- review (ship-review orchestration) --- + p_review = sub.add_parser( + "review", + help="Create a ship-review graph for a git change", + ) + review_sub = p_review.add_subparsers(dest="review_action") + p_review_create = review_sub.add_parser( + "create", + help="Build a durable 5-card review graph (parent + 3 reviewers + synthesis)", + ) + p_review_create.add_argument("title", help="Human title for the review") + p_review_create.add_argument("--base", required=True, help="Git base ref") + p_review_create.add_argument("--head", required=True, help="Git head ref") + p_review_create.add_argument("--repo-path", default=".", help="Path to repository (default: cwd)") + p_review_create.add_argument("--assignee", default=None, help="Profile to assign") + p_review_create.add_argument("--ready", action="store_true", help="Create cards in 'ready' instead of 'triage'") + p_review_create.add_argument("--skill", action="append", default=None, help="Skill to attach (repeatable)") + p_review_create.add_argument("--body", default=None, help="Extra context appended to parent body") + p_review_create.add_argument("--json", action="store_true", help="Emit JSON output") + kanban_parser.set_defaults(_kanban_parser=kanban_parser) return kanban_parser @@ -959,6 +980,7 @@ def kanban_command(args: argparse.Namespace) -> int: "specify": _cmd_specify, "decompose": _cmd_decompose, "gc": _cmd_gc, + "review": _cmd_review, } handler = handlers.get(action) if not handler: @@ -971,6 +993,7 @@ def kanban_command(args: argparse.Namespace) -> int: return 1 + # --------------------------------------------------------------------------- # Handlers # --------------------------------------------------------------------------- @@ -2725,6 +2748,52 @@ def _cmd_gc(args: argparse.Namespace) -> int: return 0 +def _cmd_review(args: argparse.Namespace) -> int: + """Dispatch ``hermes kanban review ``.""" + sub = getattr(args, "review_action", None) + if sub == "create": + return _cmd_review_create(args) + print("kanban review: unknown action. Use `hermes kanban review create --help`", file=sys.stderr) + return 2 + + +def _cmd_review_create(args: argparse.Namespace) -> int: + """Handle ``hermes kanban review create …``.""" + import json as _json + from pathlib import Path + repo = Path(args.repo_path) + if not repo.exists() or not repo.is_dir(): + print(f"kanban review: {args.repo_path} is not a directory", file=sys.stderr) + return 1 + try: + result = kr.create_review_graph( + title=args.title, + base=args.base, + head=args.head, + repo_path=str(repo.resolve()), + assignee=args.assignee, + ready=args.ready, + skills=args.skill or [], + body=args.body, + ) + except Exception as exc: + print(f"kanban review: {exc}", file=sys.stderr) + return 1 + if args.json: + print(_json.dumps(result, indent=2, default=str)) + else: + if result["created"]: + print("Created review graph") + else: + print("Found existing review graph") + print("all cards already existed") + print(f"parent: {result['parent_id']}") + for idx, rid in enumerate(result['reviewer_ids'], 1): + print(f"reviewer {idx}: {rid}") + print(f"synthesis: {result['synthesis_id']}") + return 0 + + # --------------------------------------------------------------------------- # Slash-command entry point (used by /kanban from CLI and gateway) # --------------------------------------------------------------------------- diff --git a/hermes_cli/kanban_review.py b/hermes_cli/kanban_review.py new file mode 100644 index 000000000000..4d10a496040f --- /dev/null +++ b/hermes_cli/kanban_review.py @@ -0,0 +1,351 @@ +"""Kanban ship-review graph creation. + +Provides ``create_review_graph()`` — a helper that builds a durable +5-card review graph for a git change: + + 1. Parent review card (base..head change summary) + 2. Code-quality reviewer ┐ + 3. Security reviewer │ parallel + 4. Test-coverage reviewer ┘ + 5. Synthesis card ← gated on 2-4 + +All cards use deterministic idempotency keys so repeated invocations are +idempotent. By default every card is created in ``triage`` so nothing +dispatches until the operator explicitly promotes them. + +The CLI surface lives in ``hermes_cli/kanban.py`` under +``hermes kanban review create …``. +""" + +from __future__ import annotations + +import hashlib +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +from hermes_cli import kanban_db as kb + + +# --------------------------------------------------------------------------- +# Typed spec +# --------------------------------------------------------------------------- + +@dataclass +class ReviewGraphSpec: + """Parameters that fully describe a ship-review graph.""" + + repo_path: str + base: str + head: str + title: str + assignee: Optional[str] = None + ready: bool = False + idempotency_prefix: Optional[str] = None + skills: list[str] = field(default_factory=list) + body: Optional[str] = None + + +# --------------------------------------------------------------------------- +# Idempotency helpers +# --------------------------------------------------------------------------- + +def _repo_hash(repo_path: str) -> str: + """Stable 16-char hex hash of the resolved repo path.""" + abs_path = str(Path(repo_path).resolve()) + return hashlib.sha256(abs_path.encode()).hexdigest()[:16] + + +def _review_base_key(base: str, head: str, repo_path: str) -> str: + """Deterministic base key for a given review target. + + Derives from repo realpath hash + base + head so the key is stable + across board switches and path aliasing. + """ + return f"ship-review:{_repo_hash(repo_path)}:{base}:{head}" + + +def _card_key(base_key: str, role: str) -> str: + return f"{base_key}:{role}" + + +# --------------------------------------------------------------------------- +# Template helpers +# --------------------------------------------------------------------------- + +_ROLE_FOCUS = { + "code-quality": ( + "readability, maintainability, naming, complexity, DRY violations, " + "and architectural consistency" + ), + "security": ( + "injection vectors, unsafe evals, hardcoded secrets, input validation, " + "auth/authz gaps, and dependency risks" + ), + "test-coverage": ( + "missing tests for new logic, edge cases, regression tests, " + "test readability, and CI pass status" + ), +} + +_ROLE_CHECKLIST: dict[str, list[str]] = { + "code-quality": [ + "Readability and naming conventions", + "DRY violations and duplicated logic", + "Complexity and function length", + "Architectural consistency with existing patterns", + "Type safety and static analysis concerns", + "Documentation completeness", + ], + "security": [ + "Injection vectors (SQL, command, path, eval)", + "Hardcoded secrets or credentials", + "Input validation and sanitization", + "Authentication/authorization gaps", + "Unsafe deserialization or eval usage", + "Dependency risks (untrusted sources, version pinning)", + "Privilege escalation paths", + ], + "test-coverage": [ + "New logic has accompanying tests", + "Edge cases are covered", + "Regression tests for bug fixes", + "Test readability and naming", + "CI pass status and flaky test checks", + "Integration / E2E coverage for user-facing changes", + ], +} + + +def _reviewer_body(role: str, base: str, head: str, ws_path: str) -> str: + """Return a hardened reviewer task body for *role*.""" + focus = _ROLE_FOCUS.get(role, "general code review") + checklist_lines = "\n".join(f"- [ ] {item}" for item in _ROLE_CHECKLIST.get(role, [])) + + return ( + f"Review the {role} of `{base}` → `{head}` in `{ws_path}`.\n\n" + f"Diff to review:\n" + f" git diff {base}...{head} --stat\n" + f" git diff {base}...{head}\n\n" + f"**REVIEW-ONLY v1** — Do NOT modify source code. " + f"Report findings as structured metadata only.\n\n" + f"Severity labels:\n" + f"- **Critical** — Merge blocker; must be fixed before ship.\n" + f"- **Important** — Significant concern; strongly recommend fixing.\n" + f"- **Optional/Nit** — Minor improvement; ship at discretion.\n\n" + f"kanban_complete / kanban_block contract:\n" + f'- Call kanban_complete(summary=..., metadata={{"findings": [...]}})\n' + f'- Call kanban_block(reason=...) if you are blocked ' + f"(missing context, cannot access files)\n" + f"- Each finding must include: severity, file, line (if applicable), " + f"issue description.\n\n" + f"Focus on: {focus}.\n\n" + f"Checklist:\n{checklist_lines}" + ) + + +def _synthesis_body(base: str, head: str, ws_path: str) -> str: + """Return a hardened synthesis task body.""" + return ( + f"Synthesize findings from the three reviewers for `{base}` → `{head}` " + f"in `{ws_path}`.\n\n" + f"Inputs: parent card + three completed reviewer cards " + f"(code-quality, security, test-coverage).\n\n" + f"Required output structure:\n" + f"- **GO/NO-GO decision** with explicit rationale.\n" + f"- **Blockers**: list of Critical findings that must be resolved before ship.\n" + f"- **Recommended fixes**: ordered by priority (Critical first, then Important).\n" + f"- **Acknowledged risks**: Important/Optional findings accepted as-is with justification.\n" + f"- **Rollback plan**: steps to revert this change if issues surface in production.\n" + f"- **Evidence reviewed**: list of files/evidence examined " + f"(diff stat, key changed files).\n\n" + f"Default rule: **NO-GO** if any Critical finding exists unless the user " + f"explicitly accepts the risk in writing.\n\n" + f"kanban_complete / kanban_block contract:\n" + f'- Call kanban_complete(summary=..., metadata={{"ship_decision": "GO|NO-GO", ' + f'"blockers": [...], "recommended_fixes": [...], ' + f'"acknowledged_risks": [...], "rollback_plan": "...", ' + f'"evidence_reviewed": [...]}})\n' + f'- Call kanban_block(reason=...) if you are blocked ' + f"(missing reviewer output, incomplete context)." + ) + + +# --------------------------------------------------------------------------- +# Graph creation +# --------------------------------------------------------------------------- + +def create_review_graph( + *, + title: str, + base: str, + head: str, + repo_path: str, + board: Optional[str] = None, + assignee: Optional[str] = None, + ready: bool = False, + body: Optional[str] = None, + skills: Optional[list[str]] = None, +) -> dict[str, Any]: + """Create (or return existing) ship-review graph. + + Parameters + ---------- + title: + Human title for the parent review card (e.g. "Review PR #42"). + base: + Git base ref (e.g. ``nousmain``). + head: + Git head ref (e.g. ``feat/auth``). + repo_path: + Absolute path to the repository root. Used as ``dir:`` workspace. + board: + Board slug. Defaults to ``kanban_db.get_current_board()``. + assignee: + Profile name for **all** cards. ``None`` leaves them unassigned. + ready: + When ``False`` (default) every card is created in ``triage``. + When ``True`` the parent + reviewer cards are created in ``ready`` + and the synthesis card in ``todo`` (it will auto-promote once its + parents complete). + body: + Optional extra context appended to the parent review card body. + skills: + Optional list of skills to attach to every card. + + Returns + ------- + dict with ``parent_id``, ``reviewer_ids``, ``synthesis_id``, and + ``created`` (bool — ``False`` when every id already existed). + """ + board = board or kb.get_current_board() + base_key = _review_base_key(base, head, repo_path) + ws_kind, ws_path = "dir", str(Path(repo_path).resolve()) + default_status = "ready" if ready else "triage" + + # Parent card body + parent_body_parts = [ + f"Ship review for `{base}` → `{head}`.", + f"Repository: {ws_path}", + "", + "**REVIEW-ONLY v1** — Do NOT modify source code. " + "Report findings as structured metadata only.", + "", + "Reviewers:", + "- code-quality", + "- security", + "- test-coverage", + "", + "Synthesis card will aggregate findings once all reviewers finish.", + ] + if body: + parent_body_parts.extend(["", "Context:", body]) + parent_body = "\n".join(parent_body_parts) + + # Reviewer templates + reviewers = [ + ("code-quality", f"[REVIEW] Code quality — {title}"), + ("security", f"[REVIEW] Security — {title}"), + ("test-coverage", f"[REVIEW] Test coverage — {title}"), + ] + + created_any = False + reviewer_ids: list[str] = [] + + with kb.connect(board=board) as conn: + # --- Parent review card --- + parent_key = _card_key(base_key, "parent") + existing_parent = conn.execute( + "SELECT id FROM tasks WHERE idempotency_key = ? AND status != 'archived'", + (parent_key,), + ).fetchone() + if existing_parent: + parent_id = existing_parent["id"] + else: + parent_id = kb.create_task( + conn, + title=title, + body=parent_body, + assignee=assignee, + created_by=_profile_author(), + workspace_kind=ws_kind, + workspace_path=ws_path, + triage=not ready, + idempotency_key=parent_key, + skills=skills, + ) + created_any = True + + # --- Reviewer cards (parallel) --- + # Note: reviewers are NOT linked to the parent card because + # kanban_db treats every parent link as a blocking dependency. + # The parent is an organisational umbrella; only the synthesis + # card is gated on the reviewers. + for role, rtitle in reviewers: + rkey = _card_key(base_key, role) + existing = conn.execute( + "SELECT id FROM tasks WHERE idempotency_key = ? AND status != 'archived'", + (rkey,), + ).fetchone() + if existing: + rid = existing["id"] + else: + rid = kb.create_task( + conn, + title=rtitle, + body=_reviewer_body(role, base, head, ws_path), + assignee=assignee, + created_by=_profile_author(), + workspace_kind=ws_kind, + workspace_path=ws_path, + triage=not ready, + idempotency_key=rkey, + skills=skills, + ) + created_any = True + reviewer_ids.append(rid) + + # --- Synthesis card (gated on all reviewers) --- + synthesis_body = _synthesis_body(base, head, ws_path) + synth_key = _card_key(base_key, "synthesis") + existing_synth = conn.execute( + "SELECT id FROM tasks WHERE idempotency_key = ? AND status != 'archived'", + (synth_key,), + ).fetchone() + if existing_synth: + synthesis_id = existing_synth["id"] + else: + synthesis_id = kb.create_task( + conn, + title=f"[SYNTHESIS] {title}", + body=synthesis_body, + assignee=assignee, + created_by=_profile_author(), + workspace_kind=ws_kind, + workspace_path=ws_path, + triage=not ready, + idempotency_key=synth_key, + parents=tuple(reviewer_ids), + skills=skills, + ) + created_any = True + + return { + "parent_id": parent_id, + "reviewer_ids": reviewer_ids, + "synthesis_id": synthesis_id, + "created": created_any, + } + + +def _profile_author() -> str: + for env in ("HERMES_PROFILE_NAME", "HERMES_PROFILE"): + v = os.environ.get(env) + if v: + return v + try: + from hermes_cli.profiles import get_active_profile_name + return get_active_profile_name() or "user" + except Exception: + return "user" diff --git a/hermes_cli/main.py b/hermes_cli/main.py index 714a61cae624..e92920e33167 100644 --- a/hermes_cli/main.py +++ b/hermes_cli/main.py @@ -3963,6 +3963,118 @@ def _prompt_new_key(*, allow_lmstudio_default: bool) -> str: return existing_key, False +def _model_flow_kimi(config, current_model=""): + """Kimi / Moonshot model selection with automatic endpoint routing. + + - sk-kimi-* keys → api.kimi.com/coding/v1 (Kimi Coding Plan) + - Other keys → api.moonshot.ai/v1 (legacy Moonshot) + + No manual base URL prompt — endpoint is determined by key prefix. + """ + from hermes_cli.auth import ( + PROVIDER_REGISTRY, + KIMI_CODE_BASE_URL, + _prompt_model_selection, + _save_model_choice, + deactivate_provider, + ) + from hermes_cli.config import ( + get_env_value, + save_env_value, + load_config, + save_config, + ) + from hermes_cli.models import _PROVIDER_MODELS + + provider_id = "kimi-coding" + pconfig = PROVIDER_REGISTRY[provider_id] + key_env = pconfig.api_key_env_vars[0] if pconfig.api_key_env_vars else "" + base_url_env = pconfig.base_url_env_var or "" + + # Step 1: Check for credentials — prefer OAuth, then env API key, then prompt + existing_key = "" + for ev in pconfig.api_key_env_vars: + existing_key = get_env_value(ev) or os.getenv(ev, "") + if existing_key: + break + + oauth_available = False + if not existing_key: + try: + from hermes_cli.auth import resolve_kimi_coding_runtime_credentials + oauth_creds = resolve_kimi_coding_runtime_credentials() + if oauth_creds.get("source") in {"kimi-cli-oauth", "kimi-cli-oauth-refresh"}: + oauth_available = True + print(f" {pconfig.name} OAuth: {oauth_creds['auth_file']} ✓") + print() + except Exception: + pass + + if not existing_key and not oauth_available: + existing_key, abort = _prompt_api_key( + pconfig, existing_key, provider_id=provider_id + ) + if abort: + return + elif existing_key: + print(f" {pconfig.name} API key: {existing_key[:8]}... ✓") + print() + + # Step 2: Auto-detect endpoint from key prefix or OAuth + is_coding_plan = oauth_available or existing_key.startswith("sk-kimi-") + if is_coding_plan: + effective_base = KIMI_CODE_BASE_URL + print(f" Detected Kimi Coding Plan key → {effective_base}") + else: + effective_base = pconfig.inference_base_url + print(f" Using Moonshot endpoint → {effective_base}") + # Clear any manual base URL override so auto-detection works at runtime + if base_url_env and get_env_value(base_url_env): + save_env_value(base_url_env, "") + print() + + # Step 3: Model selection — show appropriate models for the endpoint + if is_coding_plan: + # Coding Plan models (kimi-k2.7-code first, the latest) + model_list = [ + "kimi-k2.7-code", + "kimi-k2.6", + "kimi-k2.5", + "kimi-for-coding", + "kimi-k2-thinking", + "kimi-k2-thinking-turbo", + ] + else: + # Legacy Moonshot models (excludes Coding Plan-only models) + model_list = _PROVIDER_MODELS.get("moonshot", []) + + if model_list: + selected = _prompt_model_selection(model_list, current_model=current_model) + else: + try: + selected = input("Enter model name: ").strip() + except (KeyboardInterrupt, EOFError): + selected = None + + if selected: + _save_model_choice(selected) + + # Update config with provider and base URL + cfg = load_config() + model = cfg.get("model") + if not isinstance(model, dict): + model = {"default": model} if model else {} + cfg["model"] = model + model["provider"] = provider_id + model["base_url"] = effective_base + model.pop("api_mode", None) # let runtime auto-detect from URL + save_config(cfg) + deactivate_provider() + + endpoint_label = "Kimi Coding" if is_coding_plan else "Moonshot" + print(f"Default model set to: {selected} (via {endpoint_label})") + else: + print("No change.") def _infer_stepfun_region(base_url: str) -> str: diff --git a/hermes_cli/model_setup_flows.py b/hermes_cli/model_setup_flows.py index 83e60fc20a23..447d18f2232d 100644 --- a/hermes_cli/model_setup_flows.py +++ b/hermes_cli/model_setup_flows.py @@ -1854,8 +1854,9 @@ def _model_flow_kimi(config, current_model=""): # Step 3: Model selection — show appropriate models for the endpoint if is_coding_plan: - # Coding Plan models (kimi-k2.6 first) + # Coding Plan models (kimi-k2.7-code first, the latest) model_list = [ + "kimi-k2.7-code", "kimi-k2.6", "kimi-k2.5", "kimi-for-coding", diff --git a/hermes_cli/models.py b/hermes_cli/models.py index afab5bac32de..8ce899a17be9 100644 --- a/hermes_cli/models.py +++ b/hermes_cli/models.py @@ -281,6 +281,7 @@ def _xai_curated_models() -> list[str]: "openai/gpt-oss-120b", ], "kimi-coding": [ + "kimi-k2.7-code", "kimi-k2.6", "kimi-k2.5", "kimi-for-coding", @@ -3592,6 +3593,23 @@ def validate_requested_model( api_key=api_key, ) or requested + # Kimi OAuth — no /models endpoint, validate against curated catalog. + if normalized in {"kimi-coding", "kimi-coding-cn", "kimi-for-coding", "kimi"}: + if not requested: + return {"accepted": False, "persist": False, "recognized": False, "message": "Model name cannot be empty."} + try: + catalog_models = provider_model_ids("kimi-coding") # always use Hermes slug + except Exception: + catalog_models = [] + if requested_for_lookup in set(catalog_models): + return {"accepted": True, "persist": True, "recognized": True, "message": None} + return { + "accepted": True, + "persist": True, + "recognized": False, + "message": f"Note: Kimi does not expose a /models endpoint. Accepted `{requested}` without API verification.", + } + if not requested: return { "accepted": False, diff --git a/hermes_cli/researcher_scaffold.py b/hermes_cli/researcher_scaffold.py new file mode 100644 index 000000000000..a021cf53a223 --- /dev/null +++ b/hermes_cli/researcher_scaffold.py @@ -0,0 +1,552 @@ +"""Bootstrap the 'researcher' profile with research-specific config, SOUL, and memories. + +Usage: + hermes profile create researcher + hermes profile setup researcher + researcher chat +""" + +from __future__ import annotations + +from pathlib import Path + +# --------------------------------------------------------------------------- +# Content constants +# --------------------------------------------------------------------------- + +_CONFIG_YAML = """\ +# Researcher profile — optimised for iterative self-improving research loops +# This agent is a node in the altermundi operational chain. It reads from +# and writes to the shared vault (Markdown + git at $HERMES_VAULT_PATH) and reports progress +# via the shared task tracker (Kanban). +model: + default: kimi-k2.6 + provider: kimi-coding + +toolsets: + - research # run_research: Karpathy + Autogenesis AOOR loop + - web # search/research workers need web access + - file # read/write artifacts + - delegation # run_research uses delegate_task internally + - terminal # code tasks need terminal + - memory # persist research findings across sessions + - session_search + - skills + - todo + +# MCP servers — intentionally none +# Vault interaction goes through standard file + grep + git tools. +# Kanban task tracking goes through the `hermes kanban` CLI in the terminal. +# No MCP layer over either — both already have first-class CLI/text interfaces. +# Set HERMES_VAULT_PATH for the vault root; kanban DB is resolved via the +# default board (or HERMES_KANBAN_DB env var). +mcp_servers: {} + +agent: + max_turns: 80 + reasoning_effort: high + verbose: false +""" + +_SOUL_MD = """\ +You are a Research Agent powered by the Karpathy self-improvement loop and the +Autogenesis self-evolution protocol (Act → Observe → Optimize → Remember). + +You are NOT an isolated assistant. You are a node in the **altermundi operational +chain**, connected to two shared systems: + +- **Vault** (plain Markdown + git, at `$HERMES_VAULT_PATH`): The team's shared + LLM-wiki — knowledge graph, specs, runbooks, and accumulated research. Read + with standard file tools (`grep -r`, `cat`, `Read`). Write with `Write`/`Edit` + and commit with `git` so changes are versioned and reviewable. **No MCP layer** + — the vault is just files in a repo. +- **Kanban** (CLI: `hermes kanban`): The team's task tracker — every research + run that needs visibility should be tracked as a kanban task with + round-by-round progress comments posted by the supervisor's `KanbanSink`. + Invoke via the terminal — `hermes kanban create`, `hermes kanban comment`, + `hermes kanban complete`, etc. + +## Operational Context + +Before starting any research: +1. **Search the vault** for existing work — `grep -r "" $HERMES_VAULT_PATH` +2. **Read relevant notes directly** with `Read` / `cat` to avoid duplicating effort +3. **Create a kanban task** for tracking — `hermes kanban create "Research: "` + (capture the returned task id) +4. After completion, **write findings into the vault**, **commit with git**, + and **close the kanban task** + +After research completes: +1. Write a summary note to `$HERMES_VAULT_PATH/Research/.md` +2. `cd $HERMES_VAULT_PATH && git add Research/.md && git commit -m "research: "` +3. Link the note path in a final kanban comment +4. Close the kanban task with `complete`: + ``` + hermes kanban complete --review "" + ``` + Note: the `KanbanSink` already transitions the task to `done` when the + research loop terminates successfully. The explicit `complete` above is + only needed for runs that bypassed the sink (untracked runs). + +**Degraded mode**: If the kanban DB is unavailable (file missing, permissions), +declare degraded mode: +- State: "Kanban offline — running without coordination integration" +- Continue research if the core task is still possible +- Vault read/write still works — it's just files +- The supervisor falls back to `StubSink` automatically; check `runner.log` + for `KanbanSink fallback` warnings + +## Core Behavior + +Your primary tool is `run_research`. Use it when a task requires iterative +refinement toward a measurable quality criterion. For simple lookups or +one-shot tasks, use `delegate_task` directly. + +## When to use `run_research` + +- User asks to "research", "investigate", "find the best", "optimize", "study" +- Task has a clear quality criterion: relevance, accuracy, completeness, latency +- A single attempt is unlikely to be sufficient — the topic needs iteration +- You can define a numeric metric (0–1 score, pass rate, ms latency, etc.) + +## When NOT to use `run_research` + +- Simple factual questions → answer directly from knowledge +- One-off file operations or code edits → use `delegate_task` +- Tasks with no measurable outcome → use `delegate_task` + +## CRITICAL: Do NOT manually construct AIAgent + +The old pattern of importing `AIAgent` from `run_agent.py` and calling it +inside `execute_code` is DEPRECATED. `run_research` already spawns workers +via `delegate_task` internally. Just call the tool directly. + +## Choosing parameters (by task type) + +| Situation | Recommended metric_key | evaluation_mode | Notes | +|-----------|------------------------|-----------------|-------| +| Code optimization | `latency_ms` or `throughput` | `self_report` | pass_rate is baseline-only; optimize for speed/memory | +| Code correctness | `pass_rate` | `self_report` | Start here, then switch to latency_ms | +| Literature/web search | `relevance_score` | `llm_judge` | Specific criteria beat generic scoring | +| Research synthesis | `completeness_score` | `llm_judge` | 0–1 scale, evaluate against rubric | +| Algorithm design | `time_to_solution` or `iterations_to_converge` | `self_report` | Measures efficiency, not just correctness | +| Ambiguous quality | (custom) | `llm_judge` | Write a clear evaluation_prompt | + +## Metric selection guide + +- **Code tasks**: Start with `pass_rate` to get a working baseline. Once + baseline = 1.0, run a SECOND `run_research` with `latency_ms` or + `throughput` to optimize performance. This is a manual pivot, not automatic. +- **Search tasks**: `relevance_score` (0–1) with a specific llm_judge prompt + like "Score 0-1: does this list cover X published after 2022?" +- **Research tasks**: `completeness_score` (0–1) with rubric in evaluation_prompt +- **Generic tasks**: Pick the ONE number that best captures "better". If you + can't define it numerically, use `llm_judge`. + +## Kanban tracking workflow + +1. Before calling `run_research`, create a kanban task for tracking: + ``` + hermes kanban create "Research: " + ``` +2. Pass the returned task id as `kanban_task_id` to `run_research` +3. The supervisor's `KanbanSink` auto-posts round-by-round progress comments + and transitions the task to `done` on successful loop termination +4. After completion, link the workspace path with a final kanban comment + +## Before calling `run_research` + +1. Clarify the metric with the user if unclear ("what does 'good' mean here?") +2. Tell the user: "I'll run a research loop — this may take a few minutes." +3. Set a specific `evaluation_prompt` for llm_judge tasks +4. Start with max_iterations=3, time_budget_sec=0 (unlimited); increase only if needed +5. For code tasks: if pass_rate is already 1.0, use latency_ms or throughput + +## After `run_research` returns + +1. State: best metric achieved + number of iterations +2. Summarize the key finding or deliverable in plain language +3. Offer to run more iterations if the metric didn't converge +4. Point to `workspace` path if the user wants raw artifacts +5. If `kanban_task_id` was set, confirm KanbanSink posted round comments and the task transitioned to `done` + +## Research integrity + +- Never fabricate findings — only report what `run_research` actually produced +- If the metric is low, say so honestly and diagnose why +- Cite the `learnings_file` as the audit trail for your conclusions + +## Autonomous execution mode + +When the user prompt contains explicit phrasing like "do not ask", "no preguntes", +"execute autonomously", "no permission", or "iterate without asking": + +- DO NOT offer to "write a script if you'd like" — write it and run it. +- DO NOT request clarification when the task is well-scoped — proceed with reasonable assumptions and document them in the result. +- DO NOT halt on the first tool error — diagnose, attempt one alternative, then proceed with what you have. +- DO NOT escape to the user mid-task — finish the work and report what you did, including failures. + +If a task is genuinely impossible (missing capability, locked file, unreachable +service), STILL complete the protocol: emit the FAIL marker the prompt asked for, +explain the obstacle in the report, do not request input. + +Counterexample (do not do this): "If you'd like me to write the orchestration +script anyway (as a deliverable), I can produce a clean Python script... Just +let me know which path to take." This is bailing in autonomous mode. + +## Tool usage patterns (lessons from prior research swarms) + +These patterns avoid common errors observed in past research sessions. Follow +them by default — they save tool calls and prevent retries. + +### Long kanban comments — write to file, then heredoc + +Inline comment text with embedded quotes, newlines, or `$()` expansions is +fragile. The reliable pattern: + +``` +write_file /tmp/-comment.txt "" +hermes kanban comment "$(cat /tmp/-comment.txt)" +``` + +Skip the inline-first attempt. Go straight to file + cat for anything over +two lines. + +### File reads — generous range, no re-reads + +Read with explicit offset+limit covering what you need on the first pass. +Re-read a file only after you have *edited* it; do not re-read by inertia +to "remember the section." If you genuinely need a different section than +the first read, request it once with the right offset. + +### `grep` alternation — use `-E` or `-P`, never `\|` + +Bash escape of `\|` inside double quotes is fragile and frequently fails. +Always: + +``` +grep -E "pattern_a|pattern_b" # extended regex +grep -P "pattern_a|pattern_b" # perl-compat +``` + +Never `grep "pattern_a\|pattern_b"`. + +### Heredoc tag must not appear in body + +If your content might contain words like `ANALYSIS`, `EOF`, `END`, do not +use them as the heredoc tag. Use a unique, scoped tag: + +``` +cat <<'EOF_HRM57' > /tmp/x.txt +... content that may contain EOF or ANALYSIS literally ... +EOF_HRM57 +``` + +### Do NOT use `execute_code` to import internal Hermes modules + +`from hermes_tools import read_file` and similar do not work — these are +agent tools, not Python modules. Use the `read_file` tool dispatch directly. +`execute_code` is for *running computation*, not for tool routing. +""" + +_MEMORY_MD = """\ +--- +name: Research Agent Bootstrap Memory +description: Initial patterns and workspace info for the researcher profile +type: project +--- + +## Workspace + +Research artifacts live in: ~/.hermes/research-workspace/ +Each `run_research` call creates a subdirectory named by run_id: + - learnings.jsonl — HeartbeatMemorySystem schema: type/key/insight/confidence/source + - round-*/task_brief.md — worker instructions per iteration + - round-*/attempt.py or attempt.md — actual deliverable per round + - round-*/results.json — structured metrics + +## Codebase layout — research subsystem + +When investigating the AutoResearch implementation, these are the canonical +paths. Read directly; do not `find` or `grep` to discover them. + +| Path | Role | +|------|------| +| `agent/research/supervisor.py` | Karpathy loop core — `ResearchSupervisor`, `TaskSpec`, `_build_task_brief`, `_score_with_llm_judge` | +| `agent/research/runner.py` | `ExperimentRunner`, `ExperimentHistory`, `ExperimentResult` | +| `agent/research/job_runner.py` | Detached OS process entrypoint — `_build_agent`, `main` | +| `agent/research/evolution.py` | `EvolutionStore`, `extract_lessons` (vendored, currently unwired) | +| `agent/research/metrics.py` | `UniversalMetricParser` for results.json + stdout | +| `tools/research_tool.py` | `run_research` tool handler + `_LLMBridge` | +| `tools/research_job_tool.py` | `research_job` tool (start/status/collect/resume) | +| `tools/delegate_tool.py` | `delegate_task`, `_build_child_agent` (~line 967) | +| `skills/autoresearch/` | Bundled skills: `karpathy-guidelines`, `a-evolve`, 7 domain skills | +| `tests/agent/test_research_supervisor.py` | 18 integration tests | +| `HERMES_RESEARCH.md`, `RESEARCH_AGENTS.md`, `RESEARCH_OPERATIONS.md` | Top-level docs | + +## Vault integration (plain Markdown + git, no MCP) + +The vault at `$HERMES_VAULT_PATH` is just a git repo of Markdown files. +Interact with standard tools — no abstraction layer. + +- **Pre-flight**: Search for existing research before starting + ``` + grep -ri "fibonacci optimization" $HERMES_VAULT_PATH + ``` +- **During**: Read specs, runbooks, or prior research notes + ``` + cat $HERMES_VAULT_PATH/Research/Fibonacci\ Optimization.md + ``` +- **Post-flight**: Write findings back and commit + ``` + cat >> $HERMES_VAULT_PATH/Research/Fibonacci\ Optimization.md <<'EOF' + + ## Results + ... + EOF + cd $HERMES_VAULT_PATH && git add -A && git commit -m "research: fibonacci optimization results" + ``` +- **History**: Use `git log` / `git blame` to trace who wrote what, when, and why. + +**Naming convention**: `Research/.md` for research outputs. +**Why no MCP**: the vault is plain Markdown; standard text + git tools are +simpler, more debuggable, and let any agent (not just Hermes) interact with it. + +## Kanban integration (CLI) + +Kanban is the coordination layer. Invoke via the terminal — no MCP layer. + +- **Task creation**: Every tracked research run starts with a kanban task + ``` + hermes kanban create "Research: " + ``` +- **Progress tracking**: The supervisor's `KanbanSink` auto-posts round + comments when `kanban_task_id` is passed to `run_research`. You can also + post manual updates: + ``` + hermes kanban comment "Baseline complete: pass_rate=1.0" + ``` +- **Completion**: The sink transitions the task to `done` automatically on + successful termination. For manual completion or summary review: + ``` + hermes kanban complete --review "" + ``` +- **History/audit**: `hermes kanban show `, `hermes kanban list`. + +## Metric patterns by task type + +| Task type | Phase 1 metric | Phase 2 metric | Why | +|-----------|---------------|----------------|-----| +| Code (new) | pass_rate | latency_ms or throughput | Baseline correctness, then optimize | +| Code (existing) | latency_ms | memory_mb | Already correct, optimize speed/resource | +| Search | relevance_score | coverage_score | Quality first, then completeness | +| Research | completeness_score | depth_score | Breadth first, then depth | +| Algorithm | pass_rate | iterations_to_converge | Correctness, then efficiency | + +**Anti-pattern**: Using pass_rate for code optimization after baseline is already +1.0. The supervisor sees no improvement and wastes iterations. Switch to a +performance metric. + +## Kanban integration pattern + +1. `hermes kanban create "Research: "` +2. Capture task_id from output +3. Call `run_research` with `kanban_task_id=` +4. `KanbanSink` auto-posts per-round comments and transitions to `done` +5. Optional final note: `hermes kanban comment "Workspace: "` + +## Resume protocol (when reclaiming an in-progress task) + +When you `claim` a kanban task, **before doing anything else**, check +whether you are resuming previous work. The kanban task survives across +iteration-budget exhaustion and worker crashes; your workspace and +detached background jobs are designed to outlive any single run. + +### Step 0: read `workspace/STATE.json` + +``` +cat $WORKSPACE/STATE.json 2>/dev/null +``` + +If it exists, **the previous run wrote it**. Treat its contents as +authoritative. Schema: + +```json +{ + "phase": "baseline-running | baseline-done | iter-N-running | iter-N-done | reporting | complete", + "started_at": "", + "last_run_at": "", + "expected_completion_estimate": "", + "detached_jobs": [ + { + "pid": , + "started_at": "", + "purpose": "baseline-runner | iter-N-runner | ...", + "log_path": "", + "expected_artifacts": ["", ...] + } + ], + "completed_artifacts": ["", ...], + "next_action": "wait | check-results | advance | report", + "notes": "" +} +``` + +If STATE.json does NOT exist, you are starting fresh; create one as your +**first non-read action** with `phase: planning` and write it again +after each meaningful state change. + +### Step 1: verify detached jobs + +For each `detached_jobs[].pid`: + +``` +ls /proc/ 2>/dev/null && echo alive || echo dead +``` + +- **alive + expected_artifacts not yet on disk** → the job is still + running. Update STATE.json's `last_run_at`, post a kanban heartbeat + with "still in flight (uptime Xm)", and **exit your run with a short + summary**. Do NOT poll in a busy loop — let the next claim handle + the next checkpoint. +- **alive + some expected_artifacts appeared** → partial completion; + process what's available, advance STATE.json, exit. +- **dead + expected_artifacts complete** → job finished cleanly; + advance `phase`, process results, write the next heartbeat. +- **dead + expected_artifacts missing** → job died mid-run; either + rerun it (mark a retry count) or escalate to human via kanban + `block` with a reason. + +### Step 2: launch detached jobs correctly + +Heavy jobs (anything over ~30s) MUST be detached so they survive your +own worker exit. Use `setsid` to put the child in a new session/group: + +```bash +LOG=/tmp/researcher_${TASK_ID}_${PURPOSE}.log +setsid bash -c "" >$LOG 2>&1 & +PID=$! +disown $PID +``` + +Record the PID + log path in STATE.json immediately. Do NOT block on +the job within the same iteration that launched it. + +### Step 3: update STATE.json before every kanban write + +Sequence per iteration: + +1. Read STATE.json (or initialize if absent). +2. Do the one piece of work this iteration calls for. +3. Update STATE.json reflecting the new state. +4. Post kanban heartbeat with a 1-line summary. +5. Exit. + +The kanban dispatcher will reclaim and re-spawn you when appropriate. +Budget per run is small (~80 iterations of tool use), but the **task +itself is unbounded** — you can take 20 runs across a day to complete +a slow experiment. + +### When you genuinely have nothing to do (job still running) + +Post a heartbeat with the uptime + ETA, then exit cleanly. **Do not +poll** the job in a loop within the same run — that wastes iteration +budget. The dispatcher's tick interval handles the polling cadence +for you. + +``` +hermes kanban heartbeat --note "baseline-runner pid=

uptime=42min eta=" +# … then return your final summary and stop +``` + +### Closing your run when work is yielded — block, don't text-only-exit + +The kanban-worker harness treats **any run that exits without calling +`kanban_complete` or `kanban_block`** as `crashed`. After several such +"crashes" the dispatcher hits `gave_up` and the task stops being +auto-reclaimed. + +When the resume protocol determines the detached job is still running, +the correct closing move is: + +``` +hermes kanban block "Waiting on detached job pid=

at . Re-evaluate when results appear at ." +``` + +This is a **cooperative yield**, not a permanent block. The detached +job is expected to call `hermes kanban unblock ` when it +completes — that's the wake-up signal. The dispatcher will reclaim +and re-spawn you to process results. + +Three actors cooperate: + +``` + worker run N detached job worker run N+1 + │ │ │ + ▼ ▼ ▼ + check STATE run reclaim after unblock + post heartbeat (no kanban re-read STATE + KANBAN BLOCK ◀───── awareness) process new artifacts + exit │ advance phase + ▼ or KANBAN BLOCK again + on success: or COMPLETE + call kanban unblock +``` + +If you forget the `kanban block` call, you'll hit `gave_up` after a +few runs and stall the task. **The block call is what makes the +resume cooperative**. + +## Patterns that work well + +- For literature search: evaluation_mode="llm_judge" with specific criteria beats self_report +- For code tasks: start with a minimal baseline, keep time_budget_sec=0 (unlimited) +- For generic research: metric_key="completeness_score" with 0-1 scale is broadly applicable +- When metric stalls after 3 rounds: read learnings.jsonl to diagnose the bottleneck +- Two-phase code research: first `pass_rate` baseline, then `latency_ms` optimization + +## Toolset notes + +- run_research internally uses delegate_task — both toolsets must be enabled +- search/research task_type workers use web+file toolsets automatically +- code task_type workers use terminal+file toolsets automatically +- DO NOT manually construct AIAgent inside execute_code — use run_research directly +""" + + +# --------------------------------------------------------------------------- +# Setup function +# --------------------------------------------------------------------------- + +def setup_researcher_profile(profile_name: str = "researcher") -> None: + """Write research-specific config, SOUL.md, and MEMORY.md to a profile. + + The profile must already exist (created via `hermes profile create `). + This function overwrites config.yaml, SOUL.md, and memories/MEMORY.md with + researcher-optimised content. + """ + from hermes_cli.profiles import get_profile_dir + + profile_dir = get_profile_dir(profile_name) + if not profile_dir.exists(): + raise FileNotFoundError( + f"Profile '{profile_name}' not found. " + f"Run: hermes profile create {profile_name}" + ) + + # config.yaml + (profile_dir / "config.yaml").write_text(_CONFIG_YAML, encoding="utf-8") + print(f" ✓ config.yaml") + + # SOUL.md + (profile_dir / "SOUL.md").write_text(_SOUL_MD, encoding="utf-8") + print(f" ✓ SOUL.md") + + # memories/MEMORY.md + memories_dir = profile_dir / "memories" + memories_dir.mkdir(exist_ok=True) + (memories_dir / "MEMORY.md").write_text(_MEMORY_MD, encoding="utf-8") + print(f" ✓ memories/MEMORY.md") + + print(f"\nResearcher profile ready at: {profile_dir}") + print(f"Start a session with: {profile_name} chat") diff --git a/hermes_cli/runtime_provider.py b/hermes_cli/runtime_provider.py index 909cbe07a080..28653af2cf4c 100644 --- a/hermes_cli/runtime_provider.py +++ b/hermes_cli/runtime_provider.py @@ -97,6 +97,12 @@ def _detect_api_mode_for_url(base_url: str) -> Optional[str]: path = urlparse(normalized).path.rstrip("/") if path.endswith("/anthropic") or path.endswith("/anthropic/v1"): return "anthropic_messages" + if hostname == "api.kimi.com" and normalized.endswith("/coding/v1"): + # Kimi Coding Plan exposes an OpenAI-compatible surface at + # /coding/v1/chat/completions. Older fork configs stored this URL + # directly; keep routing those sessions through chat_completions so + # the OpenAI SDK sends Bearer auth and default_headers are applied. + return "chat_completions" if hostname == "api.kimi.com" and "/coding" in normalized: return "anthropic_messages" return None @@ -159,6 +165,21 @@ def _host_derived_api_key(base_url: str) -> str: return (os.getenv(env_name, "") or "").strip() +def _try_kimi_oauth_credentials() -> Optional[Dict[str, Any]]: + """Return Kimi CLI OAuth credentials when available, otherwise None.""" + try: + from hermes_cli.auth import resolve_kimi_coding_runtime_credentials + + creds = resolve_kimi_coding_runtime_credentials() + except Exception: + return None + if not isinstance(creds, dict): + return None + if not str(creds.get("api_key") or "").strip(): + return None + return creds + + def _auto_detect_local_model(base_url: str) -> str: """Query a local server for its model name when only one model is loaded.""" if not base_url: diff --git a/hermes_cli/tools_config.py b/hermes_cli/tools_config.py index d71fd5edb738..f46f5abc5c83 100644 --- a/hermes_cli/tools_config.py +++ b/hermes_cli/tools_config.py @@ -55,6 +55,7 @@ CONFIGURABLE_TOOLSETS = [ ("web", "🔍 Web Search & Scraping", "web_search, web_extract"), ("browser", "🌐 Browser Automation", "navigate, click, type, scroll"), + ("kimi_webbridge", "🌉 Kimi WebBridge", "real browser control via Kimi extension"), ("terminal", "💻 Terminal & Processes", "terminal, process"), ("file", "📁 File Operations", "read, write, patch, search"), ("code_execution", "⚡ Code Execution", "execute_code"), @@ -72,8 +73,12 @@ ("session_search", "🔎 Session Search", "search past conversations"), ("clarify", "❓ Clarifying Questions", "clarify"), ("delegation", "👥 Task Delegation", "delegate_task"), + ("research", "🔬 AutoResearch", "run_research, research_job"), ("cronjob", "⏰ Cron Jobs", "create/list/update/pause/resume/run, with optional attached skills"), ("messaging", "📨 Cross-Platform Messaging", "send_message"), + ("minecraft", "⛏️ Minecraft", "perceive, navigate, build, craft, combat, manage, screenshot, command, story"), + ("embodiment", "🤖 Embodiment", "mc_bit, embodied_plan"), + ("rl", "🧪 RL Training", "Tinker-Atropos training tools"), ("homeassistant", "🏠 Home Assistant", "smart home device control"), ("spotify", "🎵 Spotify", "playback, search, playlists, library"), ("discord", "💬 Discord (read/participate)", "fetch messages, search members, create thread"), @@ -112,7 +117,7 @@ def gui_toolset_label(label: str) -> str: # `hermes tools` → X (Twitter) Search setup walks users through credential # setup. The tool's check_fn means the schema still won't appear to the # model if the credential later goes missing or expires. -_DEFAULT_OFF_TOOLSETS = {"moa", "homeassistant", "spotify", "discord", "discord_admin", "video", "video_gen", "x_search"} +_DEFAULT_OFF_TOOLSETS = {"moa", "homeassistant", "rl", "spotify", "discord", "discord_admin", "video", "video_gen", "x_search", "kimi_webbridge"} def _xai_credentials_present() -> bool: diff --git a/model_tools.py b/model_tools.py index 0618138aa9a8..e56b07725ae9 100644 --- a/model_tools.py +++ b/model_tools.py @@ -888,6 +888,7 @@ def handle_function_call( tool_request_middleware_trace: Optional[List[Dict[str, Any]]] = None, enabled_toolsets: Optional[List[str]] = None, disabled_toolsets: Optional[List[str]] = None, + parent_agent: Any = None, ) -> str: """ Main function call dispatcher that routes calls to the tool registry. @@ -1117,6 +1118,7 @@ def _dispatch(next_args: Dict[str, Any]) -> Any: task_id=task_id, session_id=session_id, enabled_tools=sandbox_enabled, + parent_agent=parent_agent, ) else: def _dispatch(next_args: Dict[str, Any]) -> Any: @@ -1125,6 +1127,7 @@ def _dispatch(next_args: Dict[str, Any]) -> Any: task_id=task_id, session_id=session_id, user_task=user_task, + parent_agent=parent_agent, ) from hermes_cli.middleware import run_tool_execution_middleware diff --git a/run_agent.py b/run_agent.py index 2bf27d575101..3c156fda7890 100644 --- a/run_agent.py +++ b/run_agent.py @@ -1347,6 +1347,41 @@ def _is_ollama_glm_backend(self) -> bool: return True return bool(self.base_url and is_local_endpoint(self.base_url)) + def _has_truncated_tool_call_args(self, assistant_message) -> bool: + """Detect a tool call whose JSON arguments were cut off mid-generation + (e.g. kimi-coding emitting an opening brace then stopping while + under-reporting finish_reason as 'stop'/'tool_calls'). Returning True + lets the caller upgrade finish_reason to 'length' so the existing + truncated-tool-call retry/boost path recovers instead of silently + sanitizing the args to an empty object and looping on an empty tool + call. Conservative: only fires when args start as a JSON object/array + but fail to parse.""" + if self.api_mode != "chat_completions": + return False + tool_calls = getattr(assistant_message, "tool_calls", None) + if not tool_calls: + return False + import json as _json + for tc in tool_calls: + fn = getattr(tc, "function", None) + if fn is None and isinstance(tc, dict): + fn = tc.get("function") + if fn is None: + continue + args = getattr(fn, "arguments", None) + if args is None and isinstance(fn, dict): + args = fn.get("arguments") + if not isinstance(args, str): + continue + stripped = args.strip() + if not stripped or stripped[0] not in "{[": + continue + try: + _json.loads(stripped) + except (ValueError, TypeError): + return True + return False + def _should_treat_stop_as_truncated( self, finish_reason: str, @@ -3834,6 +3869,40 @@ def _try_refresh_copilot_client_credentials(self) -> bool: return False logger.info("Copilot credentials refreshed from %s", token_source) + + def _try_refresh_kimi_client_credentials(self, *, force: bool = True) -> bool: + if self.provider not in {"kimi-coding", "kimi-coding-cn"} and not base_url_host_matches(self.base_url, "api.kimi.com"): + return False + + try: + from hermes_cli.auth import resolve_kimi_coding_runtime_credentials, kimi_coding_default_headers + + creds = resolve_kimi_coding_runtime_credentials( + force_refresh=force, + allow_api_key_fallback=False, + ) + except Exception as exc: + logger.debug("Kimi credential refresh failed: %s", exc) + return False + + api_key = creds.get("api_key") + base_url = creds.get("base_url") + source = str(creds.get("source") or "") + if source not in {"kimi-cli-oauth", "kimi-cli-oauth-refresh"}: + return False + if not isinstance(api_key, str) or not api_key.strip(): + return False + if not isinstance(base_url, str) or not base_url.strip(): + return False + + self.api_key = api_key.strip() + self.base_url = base_url.strip().rstrip("/") + self._client_kwargs["api_key"] = self.api_key + self._client_kwargs["base_url"] = self.base_url + self._client_kwargs["default_headers"] = kimi_coding_default_headers() + + if not self._replace_primary_openai_client(reason="kimi_credential_refresh"): + return False return True def _try_refresh_anthropic_client_credentials(self) -> bool: diff --git a/skills/autoresearch/a-evolve/SKILL.md b/skills/autoresearch/a-evolve/SKILL.md new file mode 100644 index 000000000000..7e986a33e038 --- /dev/null +++ b/skills/autoresearch/a-evolve/SKILL.md @@ -0,0 +1,202 @@ +--- +name: a-evolve +description: > + Apply A-Evolve's agentic evolution methodology to improve AI agent performance + across runs. Use when the user wants to diagnose agent failures, generate + targeted skills from error patterns, evolve system prompts, or accumulate + episodic knowledge. Works standalone or inside AutoResearchClaw pipelines. + Triggers on: "evolve", "self-improve", "diagnose failures", "generate skills + from errors", "what went wrong and how to fix it", or any mention of A-Evolve. +--- + +# A-Evolve: Agentic Evolution Skill + +Apply the **Solve → Observe → Evolve → Gate → Reload** methodology from +[A-Evolve](https://github.com/A-EVO-Lab/a-evolve) to iteratively improve +agent performance. This skill is prompt-based — no external dependencies, +no harness changes. You analyze failures, propose workspace mutations, and +generate durable artifacts (skills, prompt patches, knowledge entries) that +the agent can load in future runs. + +## Core Loop + +When asked to evolve or improve agent performance, follow this 5-step loop: + +### 1. Solve (Collect Evidence) + +Gather the agent's execution artifacts. Ask the user for or locate: +- Run logs, error traces, or experiment outputs +- Pass/fail results per task +- Metric values (accuracy, reward, success rate) +- Any existing session files from previous runs + +If inside Hermes AutoResearch, look at: +- `artifacts/hermes-research-*/` — experiment outputs per round +- Lattice task event history (`lattice show --events`) +- Lattice comments — each round posts KEPT/IMPROVED/DISCARDED + metric +- `ExperimentRunner.history.to_dict()` — full round history in memory + +### 2. Observe (Diagnose) + +Analyze the collected evidence to produce structured observations: + +For each failed or underperforming task, identify: +- **Error category**: code bug, timeout, wrong approach, missing knowledge, + API misuse, hallucinated reference, prompt ambiguity, etc. +- **Root cause**: What specifically went wrong and why +- **Frequency**: Is this a one-off or a recurring pattern across tasks? +- **Severity**: blocking (pipeline crash) / degrading (wrong result) / + cosmetic (formatting issue) + +Write observations as a structured list: + +``` +## Observations (Batch N) + +### OBS-1: [Category] Short description +- Tasks affected: task_001, task_005, task_012 +- Root cause: ... +- Frequency: 3/50 tasks (6%) +- Severity: degrading + +### OBS-2: ... +``` + +### 3. Evolve (Propose Mutations) + +Based on observations, propose one or more of these mutation types: + +**A. Generate a Skill** (for recurring patterns, frequency ≥ 3) + +Write a new `SKILL.md` file that teaches the agent how to handle this +pattern. A good evolved skill: +- Targets a specific failure category, not generic advice +- Contains concrete steps the agent should follow +- Includes a "when to apply" trigger condition +- Is short (under 100 lines) and self-contained + +Example — if the agent keeps failing at API pagination: + +```markdown +--- +name: api-pagination-handler +description: > + Handle paginated API responses correctly. Use when making API calls + that may return partial results, or when results seem truncated. +--- + +When calling any API that supports pagination: + +1. Check response for pagination indicators: `next_page`, `offset`, + `has_more`, `cursor`, or truncated result counts. +2. If paginated, loop until all pages are collected. +3. Concatenate results before processing. +4. Set a max-page safety limit (default: 20) to prevent infinite loops. +5. Log total items collected vs expected count if available. +``` + +**B. Patch the System Prompt** (for prompt ambiguity or missing guidance) + +Write a short addendum to the system prompt that addresses the gap. +Keep patches minimal — one paragraph per issue. Format: + +``` +## Prompt Patch: [Issue] +Append to system prompt: +> When [specific situation], always [specific action] because [reason]. +``` + +**C. Add a Knowledge Entry** (for factual gaps or learned heuristics) + +Record a reusable insight as a knowledge entry: + +```json +{ + "id": "know-001", + "category": "experiment_design", + "insight": "Synthetic benchmarks with <100 samples produce high-variance results. Always use ≥500 samples or report confidence intervals.", + "source": "observation OBS-3 from batch 2", + "confidence": 0.85 +} +``` + +**D. Do Nothing** (if observation is a one-off, severity is cosmetic, +or the fix would be too broad / risky) + +### 4. Gate (Validate) + +Before accepting any mutation, check: + +- **Specificity**: Does it target the observed failure without being so + broad it could cause regressions elsewhere? +- **Testability**: Could you verify this mutation helps by re-running the + failed tasks? +- **Blast radius**: How much of the agent's behavior does this change? + Prefer small, targeted mutations over large rewrites. +- **Consistency**: Does it contradict existing skills or prompt guidance? + +If a mutation fails the gate, either refine it or discard it. +Explain your reasoning to the user. + +### 5. Reload (Apply and Record) + +Present the accepted mutations to the user. For each: +- State what changed and why +- Show the artifact (skill file, prompt patch, knowledge entry) +- Suggest where to place it in the project + +For Hermes AutoResearch projects, recommended locations: + +| Artifact | Location | +|----------|----------| +| Evolved skill | `skills/autoresearch/evolved//SKILL.md` | +| Prompt patch | Edit the inline templates in `agent/research/supervisor.py:_build_task_brief` | +| Knowledge entry | `~/.hermes/evolution/lessons.jsonl` via `EvolutionStore.append_many()` | +| Observation log | `~/.hermes/research-workspace//observations/.md` | + +Keep a running version log so the user can track what evolved and when: + +``` +## Evolution Log +- evo-1 (2026-03-30): Generated `api-pagination-handler` skill from OBS-1 +- evo-2 (2026-03-30): Prompt patch for citation format from OBS-4 +``` + +## Usage with Hermes AutoResearch + +This skill maps to Hermes Karpathy loop steps: + +| Loop Step | Evolution Role | +|-----------|---------------| +| Step 3: DELEGATE | Source of Solve artifacts — delegate_task outputs | +| Step 4: METRIC | Main Observe trigger — parse what went wrong in metric extraction | +| Step 5: KEEP/DISCARD | Natural Gate — KEPT = accept, DISCARDED = evolve | +| EvolutionStore | Lessons persisted via `EvolutionStore.append_many()` | + +When the user says "evolve my research pipeline" or similar: + +1. Ask which run to analyze (or find the latest `artifacts/hermes-research-*/`) +2. Run the Observe step on Lattice round comments + experiment outputs +3. Propose mutations targeting the weakest loop steps +4. Generate skill files in `skills/autoresearch/evolved/` + +## Anti-Patterns + +Do NOT: +- Generate vague, generic skills ("always be careful", "check your work") +- Propose mutations for one-off errors that won't recur +- Rewrite the entire system prompt — patch it surgically +- Generate more than 3 skills per evolution cycle (quality over quantity) +- Mutate tool code unless the user explicitly asks for it + +## Relationship to EvolutionStore + +Hermes uses `EvolutionStore` (`agent/research/evolution.py`) as the lesson persistence layer. +Evolved skills from this process can be placed in `skills/autoresearch/evolved/` +so they are available in future research sessions. The two systems are complementary: + +- **A-Evolve skill**: Deep, targeted mutation from structured observation +- **EvolutionStore lesson**: Broad pattern captured with time-decay weighting (`LessonEntry`) + +Both can coexist. Skills generated here are higher-precision; EvolutionStore +lessons are higher-recall and decay naturally over time (30-day half-life). diff --git a/skills/autoresearch/domain/biology-biopython/SKILL.md b/skills/autoresearch/domain/biology-biopython/SKILL.md new file mode 100644 index 000000000000..d1f1a5732baf --- /dev/null +++ b/skills/autoresearch/domain/biology-biopython/SKILL.md @@ -0,0 +1,65 @@ +--- +name: biology-biopython +description: Bioinformatics with Biopython for sequence manipulation, file parsing, BLAST, and phylogenetics. Use when working with DNA/RNA/protein sequences or biological databases. +metadata: + category: domain + trigger-keywords: "sequence,FASTA,genome,protein,BLAST,phylogenetic,biopython,bioinformatics,gene,DNA,RNA" + applicable-stages: "9,10,12" + priority: "4" + version: "1.0" + author: researchclaw + references: "adapted from K-Dense-AI/claude-scientific-skills" +--- + +## Biopython Bioinformatics Best Practice + +### Sequence Manipulation +1. Create sequences: `from Bio.Seq import Seq; seq = Seq("ATGCGA")` +2. Complement: `seq.complement()`; Reverse complement: `seq.reverse_complement()` +3. Transcription: `seq.transcribe()` (DNA to RNA) +4. Translation: `seq.translate()` (DNA/RNA to protein) +5. GC content: `from Bio.SeqUtils import gc_fraction; gc_fraction(seq)` +6. Molecular weight: `from Bio.SeqUtils import molecular_weight` + +### File Parsing (SeqIO) +1. Read FASTA: `for rec in SeqIO.parse("file.fasta", "fasta"): ...` +2. Read GenBank: `for rec in SeqIO.parse("file.gb", "genbank"): ...` +3. Read single record: `rec = SeqIO.read("file.fasta", "fasta")` +4. Write sequences: `SeqIO.write(records, "output.fasta", "fasta")` +5. Convert formats: `SeqIO.convert("input.gb", "genbank", "output.fasta", "fasta")` +6. Index large files: `idx = SeqIO.index("large.fasta", "fasta")` for random access + +### BLAST Operations +1. Online BLAST: `from Bio.Blast import NCBIWWW; result = NCBIWWW.qblast("blastn", "nt", seq)` +2. Parse results: `from Bio.Blast import NCBIXML; records = NCBIXML.parse(result)` +3. Local BLAST: run via subprocess, parse XML output with NCBIXML +4. Always set `Entrez.email` before any NCBI access +5. Filter results by e-value (typically < 1e-5) and coverage + +### NCBI Database Access (Entrez) +1. Always set email: `Entrez.email = "your@email.com"` +2. Search: `handle = Entrez.esearch(db="pubmed", term="query")` +3. Fetch records: `handle = Entrez.efetch(db="nucleotide", id="ID", rettype="fasta")` +4. Use API key for higher rate limits (10 req/s vs 3 req/s) +5. Respect NCBI rate limits; add delays between batch requests + +### Phylogenetics (Bio.Phylo) +1. Read trees: `from Bio import Phylo; tree = Phylo.read("tree.nwk", "newick")` +2. Draw trees: `Phylo.draw(tree)` or `Phylo.draw_ascii(tree)` +3. Supported formats: newick, nexus, phyloxml +4. Traverse clades: `for clade in tree.find_clades(): ...` +5. Calculate distances: `tree.distance(clade1, clade2)` + +### Structure Analysis (Bio.PDB) +1. Parse PDB: `parser = PDBParser(); structure = parser.get_structure("id", "file.pdb")` +2. Hierarchy: Structure > Model > Chain > Residue > Atom +3. Get atoms: iterate through `structure.get_atoms()` +4. Calculate distances: use atom coordinate vectors +5. For mmCIF files: use `MMCIFParser()` instead of `PDBParser()` + +### Common Pitfalls +1. Always handle `SeqIO.parse` as an iterator — it exhausts after one pass +2. Check sequence alphabet compatibility before operations +3. Large files: use `SeqIO.index()` not `SeqIO.to_dict()` to avoid memory issues +4. Set proper timeout for remote BLAST queries (can take minutes) +5. Validate parsed data — missing annotations are common in public databases diff --git a/skills/autoresearch/domain/chemistry-rdkit/SKILL.md b/skills/autoresearch/domain/chemistry-rdkit/SKILL.md new file mode 100644 index 000000000000..b5f9cd9ea789 --- /dev/null +++ b/skills/autoresearch/domain/chemistry-rdkit/SKILL.md @@ -0,0 +1,59 @@ +--- +name: chemistry-rdkit +description: Computational chemistry with RDKit for molecular analysis, descriptors, fingerprints, and substructure search. Use when working with SMILES, drug discovery, or cheminformatics tasks. +metadata: + category: domain + trigger-keywords: "molecule,SMILES,chemical,drug,rdkit,fingerprint,molecular,compound,reaction,cheminformatics" + applicable-stages: "9,10,12" + priority: "4" + version: "1.0" + author: researchclaw + references: "adapted from K-Dense-AI/claude-scientific-skills" +--- + +## RDKit Cheminformatics Best Practice + +### Molecular I/O +1. Create molecules from SMILES: `mol = Chem.MolFromSmiles('CCO')` +2. Always check for None: `MolFromSmiles` returns None on invalid input +3. Convert to canonical SMILES: `Chem.MolToSmiles(mol)` +4. Read SDF files: `suppl = Chem.SDMolSupplier('file.sdf')` +5. Read SMILES files: `suppl = Chem.SmilesMolSupplier('file.smi')` +6. Write molecules: `writer = Chem.SDWriter('output.sdf')` + +### Molecular Descriptors +1. Molecular weight: `Descriptors.MolWt(mol)` +2. LogP (lipophilicity): `Descriptors.MolLogP(mol)` +3. TPSA (polar surface area): `Descriptors.TPSA(mol)` +4. H-bond donors/acceptors: `Descriptors.NumHDonors(mol)`, `Descriptors.NumHAcceptors(mol)` +5. Rotatable bonds: `Descriptors.NumRotatableBonds(mol)` +6. Lipinski Rule of 5: MW <= 500, LogP <= 5, HBD <= 5, HBA <= 10 + +### Fingerprints and Similarity +1. Morgan (circular) fingerprints: `AllChem.GetMorganFingerprintAsBitVect(mol, radius=2, nBits=2048)` +2. RDKit fingerprints: `Chem.RDKFingerprint(mol)` +3. MACCS keys: `MACCSkeys.GenMACCSKeys(mol)` +4. Tanimoto similarity: `DataStructs.TanimotoSimilarity(fp1, fp2)` +5. Use radius=2 (ECFP4 equivalent) as default for most applications +6. For virtual screening, Tanimoto > 0.7 suggests structural similarity + +### Substructure Search +1. SMARTS patterns: `pattern = Chem.MolFromSmarts('[OH]')` +2. Check match: `mol.HasSubstructMatch(pattern)` +3. Get all matches: `mol.GetSubstructMatches(pattern)` +4. Common SMARTS: `[#6](=O)[OH]` (carboxylic acid), `[NH2]` (primary amine) +5. Filter compound libraries by functional group presence + +### Property Calculation Patterns +1. Batch processing: iterate over SDMolSupplier, skip None entries +2. Use `Chem.Descriptors.descList` for all available descriptors +3. For ADMET filtering, calculate Lipinski, Veber, and PAINS filters +4. Generate 3D coordinates: `AllChem.EmbedMolecule(mol, AllChem.ETKDG())` +5. Minimize energy: `AllChem.MMFFOptimizeMolecule(mol)` + +### Common Pitfalls +1. Always sanitize molecules (default behavior) — disable only when needed +2. Add hydrogens explicitly for 3D work: `Chem.AddHs(mol)` +3. Handle stereochemistry: use `Chem.AssignStereochemistry(mol)` +4. Large SDF files: use `ForwardSDMolSupplier` for memory efficiency +5. Kekulization errors usually indicate invalid SMILES input diff --git a/skills/autoresearch/domain/cv-classification/SKILL.md b/skills/autoresearch/domain/cv-classification/SKILL.md new file mode 100644 index 000000000000..1622ba697cd7 --- /dev/null +++ b/skills/autoresearch/domain/cv-classification/SKILL.md @@ -0,0 +1,30 @@ +--- +name: cv-classification +description: Best practices for image classification tasks. Use when working on CIFAR, ImageNet, or other classification benchmarks. +metadata: + category: domain + trigger-keywords: "classification,image,cifar,imagenet,resnet,vision,cnn,vit" + applicable-stages: "9,10" + priority: "3" + version: "1.0" + author: researchclaw + references: "He et al., Deep Residual Learning, CVPR 2016; Dosovitskiy et al., An Image is Worth 16x16 Words, ICLR 2021" +--- + +## Image Classification Best Practice +Architecture selection: +- Small scale (CIFAR-10/100): ResNet-18/34, WideResNet, Simple ViT +- Medium scale: ResNet-50, EfficientNet-B0/B1, DeiT-Small +- Large scale: ViT-B/16, ConvNeXt, Swin Transformer + +Training recipe: +- Optimizer: AdamW (lr=1e-3 to 3e-4) or SGD (lr=0.1 with cosine decay) +- Weight decay: 0.01-0.1 for AdamW, 5e-4 for SGD +- Data augmentation: RandomCrop, RandomHorizontalFlip, Cutout/CutMix +- Warmup: 5-10 epochs linear warmup for transformers +- Batch size: 128-256 for CNNs, 512-1024 for ViTs (if memory allows) + +Standard benchmarks: +- CIFAR-10: ~96% (ResNet-18), ~97% (WideResNet) +- CIFAR-100: ~80% (ResNet-18), ~84% (WideResNet) +- ImageNet: ~76% (ResNet-50), ~81% (ViT-B/16) diff --git a/skills/autoresearch/domain/cv-detection/SKILL.md b/skills/autoresearch/domain/cv-detection/SKILL.md new file mode 100644 index 000000000000..653df211ad47 --- /dev/null +++ b/skills/autoresearch/domain/cv-detection/SKILL.md @@ -0,0 +1,29 @@ +--- +name: cv-detection +description: Best practices for object detection tasks. Use when working on COCO, VOC, or detection architectures like YOLO and DETR. +metadata: + category: domain + trigger-keywords: "detection,object,bbox,yolo,coco,anchor,faster rcnn" + applicable-stages: "9,10" + priority: "5" + version: "1.0" + author: researchclaw + references: "Ren et al., Faster R-CNN, NeurIPS 2015; Carion et al., End-to-End Object Detection with Transformers, ECCV 2020" +--- + +## Object Detection Best Practice +Architecture families: +- One-stage: YOLO (v5/v8), SSD, RetinaNet, FCOS +- Two-stage: Faster R-CNN, Cascade R-CNN +- Transformer: DETR, DINO, RT-DETR + +Training recipe: +- Use pre-trained backbone (ImageNet) +- Multi-scale training and testing +- IoU threshold: 0.5 for mAP50, 0.5:0.95 for mAP +- Use FPN for multi-scale feature extraction +- Focal loss for class imbalance in one-stage detectors + +Standard benchmarks: +- COCO val2017: ~37 mAP (Faster R-CNN R50), ~51 mAP (DINO Swin-L) +- Pascal VOC: ~80 mAP50 (Faster R-CNN) diff --git a/skills/autoresearch/domain/nlp-alignment/SKILL.md b/skills/autoresearch/domain/nlp-alignment/SKILL.md new file mode 100644 index 000000000000..33a2ba557d46 --- /dev/null +++ b/skills/autoresearch/domain/nlp-alignment/SKILL.md @@ -0,0 +1,31 @@ +--- +name: nlp-alignment +description: Best practices for LLM alignment techniques including RLHF, DPO, and instruction tuning. Use when working on alignment or safety. +metadata: + category: domain + trigger-keywords: "alignment,rlhf,dpo,reward model,preference,instruction tuning,safety" + applicable-stages: "9,10" + priority: "4" + version: "1.0" + author: researchclaw + references: "Ouyang et al., Training language models to follow instructions, NeurIPS 2022; Rafailov et al., DPO, NeurIPS 2023" +--- + +## LLM Alignment Best Practice +Methods: +- RLHF: Train reward model → PPO fine-tuning (complex but powerful) +- DPO: Direct preference optimization (simpler, no reward model needed) +- GRPO: Group relative policy optimization +- SFT: Supervised fine-tuning as alignment baseline + +Training recipe: +- Start with SFT on high-quality instruction data +- DPO: lr=5e-7, beta=0.1, batch_size=64 +- PPO: lr=1e-6, clip=0.2, KL coeff=0.02 +- Use reference model for KL penalty +- Evaluate on safety benchmarks (TruthfulQA, BBQ, etc.) + +Common pitfalls: +- Reward hacking: model finds shortcuts to high reward +- Mode collapse: model generates repetitive outputs +- Catastrophic forgetting: loses general capabilities diff --git a/skills/autoresearch/domain/nlp-pretraining/SKILL.md b/skills/autoresearch/domain/nlp-pretraining/SKILL.md new file mode 100644 index 000000000000..f5db9cd9229a --- /dev/null +++ b/skills/autoresearch/domain/nlp-pretraining/SKILL.md @@ -0,0 +1,31 @@ +--- +name: nlp-pretraining +description: Best practices for language model pretraining and fine-tuning. Use when generating or reviewing NLP training code. +metadata: + category: domain + trigger-keywords: "language model,pretraining,fine-tuning,bert,gpt,llm,transformer,nlp,text" + applicable-stages: "9,10" + priority: "3" + version: "1.0" + author: researchclaw + references: "Devlin et al., BERT, NAACL 2019; Hu et al., LoRA, ICLR 2022" +--- + +## NLP Pretraining/Fine-tuning Best Practice +Fine-tuning recipe: +- Use pre-trained checkpoints (HuggingFace hub) +- AdamW optimizer, lr=2e-5 to 5e-5 +- Linear warmup (6% of total steps) + linear decay +- Batch size: 16-32 (use gradient accumulation for larger effective batch) +- 3-5 epochs for classification, 1-2 for generation +- Weight decay: 0.01 + +Parameter-efficient methods: +- LoRA: r=8-64, alpha=16-128, apply to q/v projections +- Prefix tuning: 10-20 prefix tokens +- Adapters: bottleneck dimension 64-256 + +Evaluation: +- Classification: accuracy, F1 (macro for imbalanced) +- Generation: perplexity, BLEU/ROUGE, human evaluation +- Use multiple seeds and report mean +/- std diff --git a/skills/autoresearch/domain/rl-policy-optimization/SKILL.md b/skills/autoresearch/domain/rl-policy-optimization/SKILL.md new file mode 100644 index 000000000000..2ee972f9e600 --- /dev/null +++ b/skills/autoresearch/domain/rl-policy-optimization/SKILL.md @@ -0,0 +1,37 @@ +--- +name: rl-policy-optimization +description: Best practices for reinforcement learning policy optimization. Use when working on RL agents, PPO, SAC, or reward design. +metadata: + category: domain + trigger-keywords: "reinforcement learning,rl,policy,reward,agent,environment,ppo,sac" + applicable-stages: "9,10" + priority: "3" + version: "1.0" + author: researchclaw + references: "Schulman et al., Proximal Policy Optimization, 2017; Haarnoja et al., Soft Actor-Critic, ICML 2018" +--- + +## RL Policy Optimization Best Practice +Algorithm selection: +- Discrete actions: PPO, DQN, A2C +- Continuous actions: SAC, TD3, PPO +- Multi-agent: MAPPO, QMIX +- Offline: CQL, IQL, Decision Transformer + +Training recipe: +- PPO: clip=0.2, lr=3e-4, gamma=0.99, GAE lambda=0.95 +- SAC: lr=3e-4, tau=0.005, auto-tune alpha +- Use vectorized environments (e.g., gymnasium.vector) +- Normalize observations and rewards +- Log episode return, episode length, value loss, policy entropy + +Evaluation: +- Report mean +/- std over 10+ evaluation episodes +- Use deterministic policy for evaluation +- Compare against random policy and simple baselines +- Report sample efficiency (return vs. env steps) + +Common pitfalls: +- Reward shaping can introduce bias +- Seed sensitivity is HIGH — use 5+ seeds +- Hyperparameter sensitivity — do a small sweep diff --git a/skills/autoresearch/hypothesis-formulation/SKILL.md b/skills/autoresearch/hypothesis-formulation/SKILL.md new file mode 100644 index 000000000000..9be439fe0b01 --- /dev/null +++ b/skills/autoresearch/hypothesis-formulation/SKILL.md @@ -0,0 +1,48 @@ +--- +name: hypothesis-formulation +description: Structured scientific hypothesis generation from observations. Use when formulating testable hypotheses, competing explanations, or experimental predictions. +metadata: + category: experiment + trigger-keywords: "hypothesis,prediction,mechanism,falsifiable,null,alternative,testable" + applicable-stages: "7,8,9" + priority: "3" + version: "1.0" + author: hermes +--- + +## Hypothesis Formulation Best Practice + +### Structured Hypothesis Development +1. Start with a clear observation or pattern that requires explanation +2. Review existing literature for known mechanisms and prior explanations +3. Identify what is already established vs. what remains uncertain +4. Formulate the hypothesis as a specific, testable statement +5. Ensure the hypothesis is falsifiable — define what outcome would refute it + +### Hypothesis Format +1. **Null hypothesis (H0)**: There is no effect or no difference +2. **Alternative hypothesis (H1)**: There is a specific, directional effect +3. State both explicitly; design experiments to reject H0 +4. Use "If... then... because..." structure for mechanistic hypotheses: + - If [independent variable is manipulated], then [predicted outcome], because [proposed mechanism] + +### Generating Competing Hypotheses +1. Propose at least 2-3 plausible explanations for the same observation +2. For each, identify unique predictions that distinguish it from alternatives +3. Rank hypotheses by parsimony, consistency with prior evidence, and testability +4. Design experiments that can discriminate between competing hypotheses +5. Consider confounding variables that could produce the same observation + +### Testable Predictions +1. Derive specific, measurable predictions from each hypothesis +2. Define expected effect direction AND approximate magnitude +3. Specify what experimental conditions would confirm vs. refute the prediction +4. Identify potential confounds and plan controls to address them +5. Ensure predictions are achievable with available methods and resources + +### Aligning with Experimental Design +1. Map each hypothesis to a concrete experimental condition or comparison +2. Ensure sample size is adequate to detect the predicted effect (power analysis) +3. Pre-register hypotheses and analysis plans when possible +4. Distinguish confirmatory (hypothesis-testing) from exploratory analyses +5. Plan for both positive and null results — what will you conclude in each case? diff --git a/skills/autoresearch/karpathy-guidelines/SKILL.md b/skills/autoresearch/karpathy-guidelines/SKILL.md new file mode 100644 index 000000000000..962e81c385ea --- /dev/null +++ b/skills/autoresearch/karpathy-guidelines/SKILL.md @@ -0,0 +1,115 @@ +--- +name: karpathy-guidelines +description: > + Behavioral guidelines to reduce common LLM coding mistakes in research + experiments. Use when writing, reviewing, or iterating on experiment code + to avoid overcomplication, make surgical changes, surface assumptions, and + define verifiable metric-based success criteria. Auto-applied inside the + Hermes AutoResearch loop. Triggers on: "simplify", "refactor experiment", + "why isn't the metric improving", "code review", or any iteration step. +metadata: + author: hermes + source: https://x.com/karpathy/status/2015883857489522876 + category: experiment + priority: "1" +--- + +# Karpathy Research Guidelines + +Behavioral guidelines for LLM-driven experiment code, derived from Andrej +Karpathy's observations on common LLM coding pitfalls. Applied to every +iteration of the Hermes AutoResearch Karpathy loop. + +**Tradeoff:** These guidelines bias toward caution over speed. For trivial +one-shot experiments, use judgment. + +--- + +## 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before writing or modifying any experiment code: + +- State your **assumptions** explicitly: what do you expect `main.py` to do? + What is the binding bottleneck causing the metric to be where it is? +- If multiple interpretations exist, present them — don't pick silently. +- If a simpler approach exists, say so. Prefer it. +- If something is unclear, name what is confusing in your NOTES output. + Do NOT guess silently and move on. + +In the research loop, this means: before touching `main.py`, write a brief +mental model of why the last metric was what it was. If you can't explain +it, don't change it yet. + +## 2. Simplicity First + +**Minimum code that moves the metric. Nothing speculative.** + +- No features beyond what the hypothesis requires. +- No abstractions for single-use experiment code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, write 50. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" +If yes, simplify. In experiment code, complexity is an enemy — it hides +the signal you're trying to measure. + +## 3. Surgical Changes + +**Touch only what your hypothesis requires. Clean up only your own mess.** + +When iterating on experiment code: + +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code or issues, mention them in NOTES — + don't fix them silently. + +When your changes create orphans: + +- Remove imports/variables that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +**The test:** Every changed line should trace directly to the hypothesis +that motivated this iteration. + +## 4. Goal-Driven Execution + +**Define success in metric terms. Loop until verified.** + +Transform vague improvement goals into verifiable metric movements: + +- "Make it faster" → "`accuracy` should increase from 0.72 toward 0.80" +- "Fix convergence" → "`loss` should decrease by at least 10% vs. baseline" +- "Try a different optimizer" → "Adam should yield higher `accuracy` than SGD at iter N" + +For each iteration, state a brief plan before coding: + +``` +Hypothesis: [what I think will improve the metric and why] +Change: [the ONE thing I will modify] +Verify: [metric moves from X toward Y] +``` + +Strong success criteria let the loop self-correct. Weak criteria +("make it better") waste iterations and lose signal. + +--- + +## Application to Hermes AutoResearch Loop + +| Loop Step | Karpathy Principle | +|-----------|-------------------| +| Step 0: Think | Principle 1 — state assumptions + bottleneck before any code | +| Step 1: Run | Principle 3 — touch only what the hypothesis requires | +| Step 2: Measure | Principle 4 — compare against your stated success criterion | +| Improve iteration | All 4 — think → minimal change → verify → repeat | +| Early stop (3 non-improving) | Principle 1 — re-examine assumptions before giving up | + +If 3 consecutive iterations fail to improve the metric, stop and ask: +"Is my mental model of the bottleneck correct?" before iterating further. +This is the Karpathy diagnostic: the loop failing usually means the +hypothesis was wrong, not that you need more iterations. diff --git a/skills/autoresearch/literature-search/SKILL.md b/skills/autoresearch/literature-search/SKILL.md new file mode 100644 index 000000000000..cd4faeeca8b7 --- /dev/null +++ b/skills/autoresearch/literature-search/SKILL.md @@ -0,0 +1,56 @@ +--- +name: literature-search +description: Systematic literature review methodology including search strategy, screening, and synthesis. Use when conducting literature reviews or writing background sections. +metadata: + category: experiment + trigger-keywords: "literature,review,systematic,PRISMA,search,database,PubMed,arXiv,citation" + applicable-stages: "3,4,5,6" + priority: "2" + version: "1.0" + author: hermes +--- + +## Literature Search Best Practice + +### Search Strategy Design +1. Define research question using PICO framework (Population, Intervention, Comparison, Outcome) +2. Identify 2-4 core concepts from the research question +3. List synonyms, abbreviations, and related terms for each concept +4. Combine terms with Boolean operators: AND (between concepts), OR (within synonyms) +5. Select at least 3 complementary databases relevant to the domain: + - Biomedical: PubMed, Scopus, Web of Science + - Computer science: arXiv, Semantic Scholar, DBLP, ACL Anthology + - Interdisciplinary: Google Scholar, OpenAlex +6. Document exact search strings for reproducibility + +### Inclusion and Exclusion Criteria +1. Define date range (e.g., last 5-10 years for rapidly evolving fields) +2. Specify language restrictions (typically English) +3. Specify publication types (peer-reviewed, preprints, conference papers) +4. Define study design requirements (RCTs, observational, computational) +5. Set domain-specific filters (species, methodology, sample size) +6. Document all criteria BEFORE screening begins + +### PRISMA Methodology +1. Record total hits from each database before deduplication +2. Remove duplicates and record count +3. Screen titles and abstracts against inclusion criteria (record excluded count) +4. Full-text review of remaining papers (record excluded with reasons) +5. Report final included studies with PRISMA flow diagram +6. For scoping reviews, use PRISMA-ScR extension + +### Screening and Quality Assessment +1. Use two-pass screening: title/abstract first, then full text +2. Apply quality assessment tools appropriate to study type: + - RCTs: Cochrane Risk of Bias tool + - Observational: Newcastle-Ottawa Scale + - ML papers: check reproducibility, dataset validity, statistical rigor +3. Extract data systematically using a predefined extraction form + +### Synthesis Approaches +1. **Narrative synthesis**: Organize findings thematically, identify patterns and contradictions +2. **Meta-analysis**: Pool quantitative results when studies are sufficiently homogeneous +3. **Gap analysis**: Explicitly identify what is NOT covered in the literature +4. Summarize key findings per theme with supporting citation counts +5. Highlight conflicting results and possible explanations +6. End with clear statement of research gaps that motivate your study diff --git a/skills/autoresearch/scientific-visualization/SKILL.md b/skills/autoresearch/scientific-visualization/SKILL.md new file mode 100644 index 000000000000..c2911ef3943a --- /dev/null +++ b/skills/autoresearch/scientific-visualization/SKILL.md @@ -0,0 +1,56 @@ +--- +name: scientific-visualization +description: Publication-ready scientific figure design with matplotlib and seaborn. Use when creating journal submission figures with proper formatting, accessibility, and statistical annotations. +metadata: + category: writing + trigger-keywords: "figure,plot,chart,visualization,matplotlib,seaborn,colorblind,publication" + applicable-stages: "14,17,22" + priority: "3" + version: "1.0" + author: researchclaw + references: "adapted from K-Dense-AI/claude-scientific-skills" +--- + +## Scientific Visualization Best Practice + +### Figure Design Principles +1. Every figure must have a clear, self-contained message +2. Minimize chartjunk: remove gridlines, background shading, and 3D effects +3. Use direct labeling instead of legends when possible +4. Remove top and right spines for cleaner appearance +5. Ensure all text is readable at final print size (minimum 6pt font) + +### Journal Figure Sizing +1. **Single column**: 3.3-3.5 inches (85-89 mm) wide +2. **1.5 column**: 4.5-5.5 inches (114-140 mm) wide +3. **Double column / full width**: 6.5-7.1 inches (165-180 mm) wide +4. Resolution: 300 DPI minimum for raster; prefer vector formats (PDF, EPS, SVG) +5. Check target journal author guidelines for exact specifications + +### Colorblind-Safe Design +1. Use colorblind-friendly palettes: seaborn "colorblind", Okabe-Ito, viridis, cividis +2. NEVER rely on color alone — combine with shape, pattern, or line style +3. Avoid red-green combinations; prefer blue-orange or blue-yellow contrasts +4. Test figures with a colorblind simulator before submission +5. Ensure figures work in grayscale for print journals + +### Multi-Panel Layouts +1. Label panels with uppercase letters: (A), (B), (C) in bold, top-left corner +2. Use consistent axis scales across panels when comparing related data +3. Share axes where appropriate to reduce redundancy +4. Maintain consistent font sizes and line widths across all panels +5. Use `plt.subplots()` with `constrained_layout=True` for automatic spacing + +### Statistical Annotations on Figures +1. Show individual data points alongside summary statistics (box + strip plots) +2. Always include error bars; specify type in caption (SEM, SD, 95% CI) +3. Use significance brackets with stars: * p<.05, ** p<.01, *** p<.001 +4. Annotate effect sizes or key statistics directly on the figure when helpful +5. Never use bar charts for small-n data — use dot plots or box plots instead + +### Export and Quality Checklist +1. Save in vector format (PDF/SVG) for line art; TIFF/PNG for photographs +2. Embed fonts or convert text to outlines for cross-platform consistency +3. Verify axis labels include units in parentheses: "Time (s)", "Force (N)" +4. Ensure figure caption fully explains all symbols, abbreviations, and panels +5. Check that color-coded elements match between figure and caption diff --git a/skills/autoresearch/scientific-writing/SKILL.md b/skills/autoresearch/scientific-writing/SKILL.md new file mode 100644 index 000000000000..6a4d00c81176 --- /dev/null +++ b/skills/autoresearch/scientific-writing/SKILL.md @@ -0,0 +1,56 @@ +--- +name: scientific-writing +description: Academic manuscript writing with IMRAD structure, citation formatting, and reporting guidelines. Use when drafting or revising research papers. +metadata: + category: writing + trigger-keywords: "paper,manuscript,writing,IMRAD,citation,abstract,introduction,methods,results,discussion" + applicable-stages: "16,17,19" + priority: "2" + version: "1.0" + author: researchclaw + references: "adapted from K-Dense-AI/claude-scientific-skills" +--- + +## Scientific Writing Best Practice + +### IMRAD Structure +1. **Abstract**: State objective, methods, key results, and conclusion in 150-300 words +2. **Introduction**: Move from broad context to specific gap to your contribution (funnel structure) +3. **Methods**: Sufficient detail for replication; use past tense, passive voice +4. **Results**: Present findings without interpretation; pair text with figures/tables +5. **Discussion**: Interpret results, compare with literature, acknowledge limitations, state implications + +### Paragraph-Level Guidance +1. Each paragraph should convey ONE main idea +2. Open with a topic sentence; close with a transition to the next paragraph +3. Write in full flowing prose — never submit bullet points as final manuscript text +4. Use active voice for clarity: "We measured..." not "Measurements were taken..." +5. Vary sentence length; aim for average 15-25 words per sentence + +### Citation Best Practices +1. Cite primary sources over reviews when making specific claims +2. Use citation styles consistently (APA, Vancouver, IEEE) per target journal +3. Every factual claim needs a citation unless it is common knowledge in the field +4. Avoid citation strings of 5+ references — select the most relevant 2-3 +5. Self-citations should be limited to genuinely relevant prior work + +### Common Writing Pitfalls +1. Avoid hedge-stacking: "It might possibly suggest..." — choose one hedge +2. Do not start sentences with "It is well known that" — cite or remove +3. Distinguish "significant" (statistical) from "substantial" (practical) +4. Ensure figures/tables are referenced in text BEFORE they appear +5. Keep abbreviations to a minimum; define each on first use + +### Reporting Guidelines +1. Randomized trials: follow CONSORT checklist +2. Observational studies: follow STROBE checklist +3. Systematic reviews: follow PRISMA checklist +4. Diagnostic accuracy: follow STARD checklist +5. Always check target journal's author guidelines for specific requirements + +### Revision Checklist +1. Verify all figures/tables are cited in text and numbered sequentially +2. Confirm reference list matches in-text citations exactly +3. Check that abstract accurately reflects the final manuscript content +4. Ensure methods section enables independent replication +5. Read aloud to catch awkward phrasing and run-on sentences diff --git a/skills/autoresearch/statistical-reporting/SKILL.md b/skills/autoresearch/statistical-reporting/SKILL.md new file mode 100644 index 000000000000..5a3fa15d7296 --- /dev/null +++ b/skills/autoresearch/statistical-reporting/SKILL.md @@ -0,0 +1,58 @@ +--- +name: statistical-reporting +description: Statistical test selection, assumption checking, and APA-formatted reporting. Use when analyzing experimental results or writing results sections. +metadata: + category: writing + trigger-keywords: "statistic,hypothesis test,p-value,regression,ANOVA,t-test,effect size,confidence interval" + applicable-stages: "14,17" + priority: "3" + version: "1.0" + author: researchclaw + references: "adapted from K-Dense-AI/claude-scientific-skills" +--- + +## Statistical Reporting Best Practice + +### Test Selection Quick Reference +1. **Comparing two groups (independent, normal)**: Independent t-test +2. **Comparing two groups (independent, non-normal)**: Mann-Whitney U test +3. **Comparing two groups (paired, normal)**: Paired t-test +4. **Comparing two groups (paired, non-normal)**: Wilcoxon signed-rank test +5. **Comparing 3+ groups (independent, normal)**: One-way ANOVA + post-hoc +6. **Comparing 3+ groups (non-normal)**: Kruskal-Wallis test +7. **Relationship between continuous variables**: Pearson or Spearman correlation +8. **Categorical outcomes**: Chi-square or Fisher's exact test +9. **Predicting continuous outcome**: Linear regression +10. **Predicting binary outcome**: Logistic regression + +### Assumption Checking +1. **Normality**: Shapiro-Wilk test (n < 50) or visual Q-Q plots +2. **Homogeneity of variance**: Levene's test before t-tests and ANOVA +3. **Independence**: Verify study design ensures independent observations +4. **Linearity**: Scatter plots and residual plots for regression +5. **Multicollinearity**: VIF < 5 for multiple regression predictors +6. When assumptions are violated, use non-parametric alternatives or robust methods + +### APA Reporting Format +1. **t-test**: t(df) = X.XX, p = .XXX, d = X.XX +2. **ANOVA**: F(df_between, df_within) = X.XX, p = .XXX, eta-squared = .XX +3. **Correlation**: r(df) = .XX, p = .XXX [95% CI: .XX, .XX] +4. **Chi-square**: chi-square(df, N = XXX) = X.XX, p = .XXX +5. **Regression**: beta = X.XX, SE = X.XX, t = X.XX, p = .XXX +6. Always report exact p-values (not "p < .05") unless p < .001 +7. Use leading zero for values that can exceed 1 (e.g., t = 0.50) but not for those bounded by 1 (e.g., p = .032, r = .45) + +### Effect Sizes +1. ALWAYS report effect sizes alongside p-values +2. Cohen's d for group comparisons: small = 0.2, medium = 0.5, large = 0.8 +3. Eta-squared for ANOVA: small = .01, medium = .06, large = .14 +4. R-squared for regression: report adjusted R-squared for multiple predictors +5. Odds ratios for logistic regression with 95% confidence intervals +6. Distinguish statistical significance from practical significance + +### Common Mistakes to Avoid +1. Never say "the results were not significant, therefore there is no effect" +2. Do not confuse correlation with causation in observational data +3. Apply multiple comparison corrections (Bonferroni, FDR) when running many tests +4. Report confidence intervals, not just point estimates +5. State whether tests are one-tailed or two-tailed and justify the choice diff --git a/tests/agent/research/DESIGN-HRM94-95.md b/tests/agent/research/DESIGN-HRM94-95.md new file mode 100644 index 000000000000..678c6a7a28b8 --- /dev/null +++ b/tests/agent/research/DESIGN-HRM94-95.md @@ -0,0 +1,115 @@ +# Design Document: HRM-94 & HRM-95 + +## Scope +- **Do NOT modify** `agent/research/supervisor.py` (owned by another worker). +- Touch only: + - `agent/research/job_runner.py` + - `agent/research/runner.py` (optional, for per-iteration timeout) + - `tools/research_tool.py` (stale checker) + - `tools/research_job_tool.py` (status action wiring) + +--- + +## HRM-94 — Real Timeout with SIGTERM/SIGKILL + +### Problem +`job_runner.py` currently runs the full `run_research` loop in the same Python +process. If a sub-agent or API call hangs, the job never terminates and +`state.json` never reflects a timeout. + +### Proposed Solution +1. `job_runner.py` reads `timeout_sec` from the job spec (default 0 = unlimited). +2. When `timeout_sec > 0`, wrap the call to `run_research` in a + `multiprocessing.Process` so the parent can monitor wall-clock time. +3. The parent waits `timeout_sec` for the child to finish. +4. On expiry: + - `os.kill(child_pid, signal.SIGTERM)` + - Wait 5 s (`child.join(timeout=5)`) + - If still alive: `os.kill(child_pid, signal.SIGKILL)` + `child.join(timeout=1)` + - Write `state.json` with `status="timeout"` and `error="Timed out after Xs"`. +5. If the child finishes normally, the parent reads `result.json` / `state.json` + already written by the child and returns its exit code. + +### Why multiprocessing? +- Python threads cannot be forcefully killed. +- `delegate_task` hangs inside API calls or subprocess tools; only a real OS + signal can break a stuck syscall. +- Using a child process keeps the parent alive long enough to write the + "timeout" state. + +### Where to change +- `agent/research/job_runner.py` + - Extract the body of `main()` (from agent build onward) into a + `_run_research_child(spec_path)` helper that can be the `target` of + `multiprocessing.Process`. + - Add signal-safe cleanup in the child (`signal.SIGTERM` handler that writes + state and exits gracefully). + - Parent loop: `proc.start()` → `proc.join(timeout=timeout_sec)` → kill + escalation if still alive. +- `tools/research_tool.py` + - Add `timeout_sec: int = 0` to `run_research()` signature so the spec field + can flow through without breaking existing callers. + +### Tests (RED stubs) +- `tests/agent/research/test_job_runner_timeout.py` + - `test_timeout_sec_forwarded_to_run_research` + - `test_timeout_writes_state_timeout_on_expiry` + - `test_timeout_sends_sigterm_then_sigkill` + +--- + +## HRM-95 — Heartbeat / Stale Detection + +### Problem +Detached jobs have no liveness probe. If the `job_runner` process dies +(OOM, `kill -9`, host reboot), `state.json` stays `"running"` forever. + +### Proposed Solution +1. **Heartbeat writer** (`agent/research/job_runner.py`) + - After writing `status="running"`, start a daemon thread that writes + `/heartbeat` every 30 s. + - File format: `{"ts": , "pid": }`. + - Stop the thread in the `finally` block before exiting. + +2. **Stale checker** (`tools/research_tool.py`) + - `check_research_stale(checkpoint_dir: str, stale_threshold_sec=90.0) -> bool` + - Reads `heartbeat`, returns `True` if missing or `now - ts > threshold`. + +3. **Status wiring** (`tools/research_job_tool.py`) + - `_action_status()` calls `check_research_stale(str(job_dir))` when the + current status is `"queued"` or `"running"`. + - If stale, overwrite `state.json` with `status="stale"` and + `stale_reason="no heartbeat for >90s"`. + +### Why 30 s / 90 s? +- Same heartbeat cadence already used by `tools/delegate_tool.py` + (`_HEARTBEAT_INTERVAL = 30`). +- 90 s = 3 missed heartbeats — tolerates one GC pause or slow disk write. + +### Where to change +- `agent/research/job_runner.py` + - Add `_write_heartbeat(job_dir)` and `_heartbeat_loop(job_dir, stop_event)`. + - Start/stop the daemon thread around the research loop. +- `tools/research_tool.py` + - Add `check_research_stale()` function. +- `tools/research_job_tool.py` + - Import and call `check_research_stale` inside `_action_status`. + +### Tests (RED stubs) +- `tests/agent/research/test_heartbeat_stale.py` + - `test_heartbeat_file_created` + - `test_heartbeat_updated_during_run` + - `test_stale_when_no_heartbeat` + - `test_stale_after_90s` + - `test_not_stale_within_90s` + - `test_status_marks_stale_when_heartbeat_missing` + +--- + +## Open Questions +1. Should `runner.py` (`ExperimentRunner`) also enforce a per-iteration + `time_budget_sec` timeout on `_delegate_fn`? This is separate from the + global HRM-94 timeout but could be added later without conflicting. +2. Should stale jobs be auto-resumable? Out of scope for these tickets — + resume logic lives in `research_job_tool.py` and already handles + `"interrupted"` / `"failed"`; we may need to add `"stale"` to the allow-list. diff --git a/tests/agent/research/__init__.py b/tests/agent/research/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tests/agent/research/test_ab_testing.py b/tests/agent/research/test_ab_testing.py new file mode 100644 index 000000000000..7a3c94be7b78 --- /dev/null +++ b/tests/agent/research/test_ab_testing.py @@ -0,0 +1,310 @@ +"""Unit tests for agent.research.ab_testing (HRM-110). + +No integration mark needed — these tests use mocked ExperimentHistory +objects and verify aggregation / reporting logic only. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import pytest + +from agent.research.ab_testing import ( + ResearchABTester, + StrategyConfig, + StrategyRun, + StrategySummary, +) +from agent.research.runner import ExperimentHistory, ExperimentResult + + +# --------------------------------------------------------------------------- +# Helpers — build minimal ExperimentHistory without running real workers +# --------------------------------------------------------------------------- + +def _make_result( + iteration: int, + primary_metric: float | None, + cost_usd: float = 0.0, + tokens_in: int = 0, + tokens_out: int = 0, + improved: bool = False, + code: str = "", +) -> ExperimentResult: + return ExperimentResult( + run_id="run-1", + iteration=iteration, + code=code, + metrics={"accuracy": primary_metric} if primary_metric is not None else {}, + primary_metric=primary_metric, + improved=improved, + kept=improved, + elapsed_sec=1.0, + stdout="", + stderr="", + cost_usd=cost_usd, + tokens_in=tokens_in, + tokens_out=tokens_out, + ) + + +def _make_history( + baseline_metric: float | None, + results: list[ExperimentResult], +) -> ExperimentHistory: + hist = ExperimentHistory(baseline_metric=baseline_metric) + for r in results: + hist.add(r) + # best_result is not auto-updated by add(), so set it manually for tests + best = max( + (r for r in results if r.primary_metric is not None), + key=lambda r: r.primary_metric or float("-inf"), + default=None, + ) + hist.best_result = best + return hist + + +# --------------------------------------------------------------------------- +# StrategyConfig +# --------------------------------------------------------------------------- + +class TestStrategyConfig: + def test_to_dict(self): + cfg = StrategyConfig(name="foo", fan_out=3, use_moa=False, max_iterations=5) + d = cfg.to_dict() + assert d == { + "name": "foo", + "fan_out": 3, + "use_moa": False, + "max_iterations": 5, + "time_budget_sec": 0, + "keep_threshold": 0.0, + } + + +# --------------------------------------------------------------------------- +# StrategyRun +# --------------------------------------------------------------------------- + +class TestStrategyRun: + def test_properties(self): + hist = _make_history( + baseline_metric=0.5, + results=[ + _make_result(0, 0.5, cost_usd=0.1, tokens_in=100, tokens_out=50), + _make_result(1, 0.7, cost_usd=0.2, tokens_in=200, tokens_out=100, improved=True), + ], + ) + run = StrategyRun( + strategy_name="seq", + repeat=0, + history=hist, + elapsed_sec=10.0, + workspace=Path("/tmp"), + ) + assert run.best_metric == 0.7 + assert run.baseline_metric == 0.5 + assert run.total_cost_usd == pytest.approx(0.3) + assert run.total_tokens_in == 300 + assert run.total_tokens_out == 150 + assert run.iterations_to_converge == 2 + assert run.improvement_rate == pytest.approx((0.7 - 0.5) / 0.5) + + def test_improvement_rate_zero_baseline(self): + hist = _make_history( + baseline_metric=0.0, + results=[ + _make_result(0, 0.0), + _make_result(1, 0.0), + ], + ) + run = StrategyRun( + strategy_name="seq", repeat=0, history=hist, elapsed_sec=1.0, workspace=Path("/tmp") + ) + assert run.improvement_rate == 0.0 + + def test_improvement_rate_inf(self): + hist = _make_history( + baseline_metric=0.0, + results=[ + _make_result(0, 0.0), + _make_result(1, 0.5, improved=True), + ], + ) + run = StrategyRun( + strategy_name="seq", repeat=0, history=hist, elapsed_sec=1.0, workspace=Path("/tmp") + ) + assert run.improvement_rate == float("inf") + + +# --------------------------------------------------------------------------- +# StrategySummary +# --------------------------------------------------------------------------- + +class TestStrategySummary: + def test_mean_and_std(self): + runs = [ + StrategyRun( + strategy_name="s", + repeat=i, + history=_make_history( + baseline_metric=0.5, + results=[_make_result(0, 0.5 + i * 0.1)], + ), + elapsed_sec=10.0 + i, + workspace=Path("/tmp"), + ) + for i in range(3) + ] + summary = StrategySummary(strategy_name="s", runs=runs) + assert summary.mean_best_metric == pytest.approx((0.5 + 0.6 + 0.7) / 3) + assert summary.std_best_metric is not None + assert summary.std_best_metric > 0 + assert summary.mean_elapsed_sec == pytest.approx((10 + 11 + 12) / 3) + assert summary.mean_iterations == pytest.approx(1.0) + + def test_empty_runs(self): + summary = StrategySummary(strategy_name="empty") + assert summary.mean_best_metric is None + assert summary.std_best_metric is None + + +# --------------------------------------------------------------------------- +# ResearchABTester formatting +# --------------------------------------------------------------------------- + +class TestResearchABTesterFormatting: + def test_format_report(self): + summaries = [ + StrategySummary( + strategy_name="sequential", + runs=[ + StrategyRun( + strategy_name="sequential", + repeat=0, + history=_make_history( + baseline_metric=0.5, + results=[ + _make_result(0, 0.5, cost_usd=0.1), + _make_result(1, 0.6, cost_usd=0.1, improved=True), + ], + ), + elapsed_sec=10.0, + workspace=Path("/tmp"), + ), + ], + ), + StrategySummary( + strategy_name="fanout3", + runs=[ + StrategyRun( + strategy_name="fanout3", + repeat=0, + history=_make_history( + baseline_metric=0.5, + results=[ + _make_result(0, 0.5, cost_usd=0.1), + _make_result(1, 0.65, cost_usd=0.2, improved=True), + ], + ), + elapsed_sec=15.0, + workspace=Path("/tmp"), + ), + ], + ), + ] + report = ResearchABTester.format_report(summaries) + assert "A/B Test Report" in report + assert "sequential" in report + assert "fanout3" in report + assert "Winner by metric" in report + assert "Winner by cost" in report + + def test_to_json(self): + summaries = [ + StrategySummary( + strategy_name="seq", + runs=[ + StrategyRun( + strategy_name="seq", + repeat=0, + history=_make_history( + baseline_metric=0.5, + results=[_make_result(0, 0.5)], + ), + elapsed_sec=5.0, + workspace=Path("/tmp"), + ), + ], + ), + ] + raw = ResearchABTester.to_json(summaries) + data = json.loads(raw) + assert len(data) == 1 + assert data[0]["strategy"] == "seq" + assert data[0]["mean_best_metric"] == 0.5 + assert data[0]["repeats"] == 1 + + +# --------------------------------------------------------------------------- +# ResearchABTester.compare mocking +# --------------------------------------------------------------------------- + +class TestResearchABTesterCompare: + def test_compare_runs_each_strategy(self, tmp_path: Path, monkeypatch: Any): + """Mock supervisor.run so compare() executes without real workers.""" + from agent.research.supervisor import ResearchSupervisor + + call_log: list[tuple[str, int, bool]] = [] + + def _fake_run( + self: Any, + spec: Any, + initial_attempt: str, + *, + run_id: str, + max_iterations: int = 5, + time_budget_sec: int = 0, + keep_threshold: float = 0.0, + llm: Any = None, + worker_toolsets: Any = None, + checkpoint_dir: Any = None, + fan_out: int = 1, + use_moa: bool = True, + ) -> ExperimentHistory: + call_log.append((run_id, fan_out, use_moa)) + return _make_history( + baseline_metric=0.5, + results=[_make_result(0, 0.5 + fan_out * 0.05)], + ) + + monkeypatch.setattr(ResearchSupervisor, "run", _fake_run) + + tester = ResearchABTester( + parent_agent=object(), + workspace=tmp_path, + ) + from agent.research.supervisor import TaskSpec + + spec = TaskSpec( + topic="test", + deliverable="test", + metric_key="accuracy", + ) + strategies = [ + StrategyConfig(name="seq", fan_out=1), + StrategyConfig(name="fan3", fan_out=3, use_moa=False), + ] + summaries = tester.compare(spec, strategies, repeats=2) + + assert len(summaries) == 2 + assert len(summaries[0].runs) == 2 + assert len(summaries[1].runs) == 2 + # Each run should have been called with correct fan_out / use_moa + assert any(fan_out == 1 for _, fan_out, _ in call_log) + assert any(fan_out == 3 for _, fan_out, _ in call_log) diff --git a/tests/agent/research/test_ab_testing_integration.py b/tests/agent/research/test_ab_testing_integration.py new file mode 100644 index 000000000000..b589584a5c34 --- /dev/null +++ b/tests/agent/research/test_ab_testing_integration.py @@ -0,0 +1,135 @@ +"""Integration test for ResearchABTester end-to-end with mocked workers. + +Runs a synthetic A/B test comparing sequential vs fan-out strategies +without real LLM calls or subagents. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from agent.research.ab_testing import ResearchABTester, StrategyConfig +from agent.research.runner import ExperimentHistory, ExperimentResult +from agent.research.supervisor import ResearchSupervisor, TaskSpec + + +def _make_result( + iteration: int, + primary_metric: float | None, + cost_usd: float = 0.05, + tokens_in: int = 100, + tokens_out: int = 50, + improved: bool = False, + code: str = "", +) -> ExperimentResult: + return ExperimentResult( + run_id="run-1", + iteration=iteration, + code=code, + metrics={"accuracy": primary_metric} if primary_metric is not None else {}, + primary_metric=primary_metric, + improved=improved, + kept=improved, + elapsed_sec=1.0, + stdout="", + stderr="", + cost_usd=cost_usd, + tokens_in=tokens_in, + tokens_out=tokens_out, + ) + + +def _make_history(baseline: float, best: float, iterations: int = 2) -> ExperimentHistory: + results = [_make_result(0, baseline, cost_usd=0.05, tokens_in=100, tokens_out=50)] + for i in range(1, iterations): + results.append( + _make_result( + i, + baseline + (best - baseline) * (i / (iterations - 1)), + cost_usd=0.05, + tokens_in=100, + tokens_out=50, + improved=True, + ) + ) + hist = ExperimentHistory(baseline_metric=baseline) + for r in results: + hist.add(r) + hist.best_result = max( + (r for r in results if r.primary_metric is not None), + key=lambda r: r.primary_metric or float("-inf"), + ) + return hist + + +@pytest.mark.integration +def test_ab_test_end_to_end(tmp_path: Path, monkeypatch: Any) -> None: + """Run a synthetic A/B test: sequential vs fan-out.""" + + def _fake_run( + self: Any, + spec: Any, + initial_attempt: str, + *, + run_id: str, + max_iterations: int = 5, + time_budget_sec: int = 0, + keep_threshold: float = 0.0, + llm: Any = None, + worker_toolsets: Any = None, + checkpoint_dir: Any = None, + fan_out: int = 1, + use_moa: bool = True, + ) -> ExperimentHistory: + # Simulate that fan-out achieves slightly better metric + baseline = 0.5 + best = 0.6 if fan_out == 1 else 0.7 + return _make_history(baseline, best, iterations=max_iterations + 1) + + monkeypatch.setattr(ResearchSupervisor, "run", _fake_run) + + tester = ResearchABTester( + parent_agent=MagicMock(), + workspace=tmp_path, + ) + + spec = TaskSpec( + topic="Synthetic benchmark", + deliverable="Dummy deliverable", + metric_key="accuracy", + ) + + strategies = [ + StrategyConfig(name="sequential", fan_out=1, max_iterations=2), + StrategyConfig(name="fanout2", fan_out=2, use_moa=False, max_iterations=2), + ] + + summaries = tester.compare(spec, strategies, initial_attempt="", repeats=1) + + assert len(summaries) == 2 + seq, fan = summaries + assert seq.strategy_name == "sequential" + assert fan.strategy_name == "fanout2" + + # Fan-out should show better metric in this synthetic scenario + assert fan.mean_best_metric == pytest.approx(0.7) + assert seq.mean_best_metric == pytest.approx(0.6) + + # Report should contain both strategies and winner lines + report = tester.format_report(summaries) + assert "sequential" in report + assert "fanout2" in report + assert "Winner by metric" in report + assert "Winner by cost" in report + + # JSON should round-trip + raw = tester.to_json(summaries) + data = json.loads(raw) + assert len(data) == 2 + assert data[1]["strategy"] == "fanout2" + assert data[1]["mean_best_metric"] == pytest.approx(0.7) diff --git a/tests/agent/research/test_auto_specify.py b/tests/agent/research/test_auto_specify.py new file mode 100644 index 000000000000..771456f7e174 --- /dev/null +++ b/tests/agent/research/test_auto_specify.py @@ -0,0 +1,103 @@ +"""Tests for agent.research.auto_specify.""" +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.auto_specify import auto_specify_topic, _extract_json_blob + + +def _fake_aux_response(content: str): + resp = MagicMock() + resp.choices = [MagicMock()] + resp.choices[0].message.content = content + return resp + + +class TestExtractJsonBlob: + def test_clean_json(self): + out = _extract_json_blob('{"a": 1}') + assert out == {"a": 1} + + def test_fenced_json(self): + out = _extract_json_blob('```json\n{"a": 1}\n```') + assert out == {"a": 1} + + def test_prose_around(self): + out = _extract_json_blob('Sure, here is: {"a": 1} done.') + assert out == {"a": 1} + + def test_empty_returns_none(self): + assert _extract_json_blob("") is None + + def test_unparseable_returns_none(self): + assert _extract_json_blob("not json at all") is None + + def test_array_returns_none(self): + # We only accept top-level objects. + assert _extract_json_blob("[1, 2, 3]") is None + + +class TestAutoSpecifyTopic: + def test_returns_structured_spec(self): + fake_client = MagicMock() + fake_client.chat.completions.create.return_value = _fake_aux_response( + '```json\n{"deliverable": "Python function classify(payload)",' + '"metric_key": "pass_rate", "metric_direction": "maximize",' + '"task_type": "code", "evaluation_mode": "self_report"}\n```' + ) + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + return_value=(fake_client, "test-model"), + ): + out = auto_specify_topic("classify daemoncraft heartbeat events") + assert out is not None + assert out["deliverable"] == "Python function classify(payload)" + assert out["metric_key"] == "pass_rate" + assert out["task_type"] == "code" + + def test_returns_none_on_aux_error(self): + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + side_effect=RuntimeError("no aux configured"), + ): + assert auto_specify_topic("vague topic") is None + + def test_returns_none_when_aux_returns_none_client(self): + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + return_value=(None, None), + ): + assert auto_specify_topic("vague topic") is None + + def test_returns_none_on_unparseable_output(self): + fake_client = MagicMock() + fake_client.chat.completions.create.return_value = _fake_aux_response( + "I think you should make a thing that scores high" + ) + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + return_value=(fake_client, "test-model"), + ): + assert auto_specify_topic("vague") is None + + def test_returns_none_for_empty_topic(self): + # Even with a working aux, empty input is rejected before the call. + fake_client = MagicMock() + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + return_value=(fake_client, "test-model"), + ): + assert auto_specify_topic("") is None + assert auto_specify_topic(" ") is None + fake_client.chat.completions.create.assert_not_called() + + def test_swallows_api_exception(self): + fake_client = MagicMock() + fake_client.chat.completions.create.side_effect = RuntimeError("rate limit") + with patch( + "agent.research.auto_specify.get_text_auxiliary_client", + return_value=(fake_client, "test-model"), + ): + assert auto_specify_topic("topic") is None diff --git a/tests/agent/research/test_checkpoint_resume.py b/tests/agent/research/test_checkpoint_resume.py new file mode 100644 index 000000000000..df6a0d5ca7e8 --- /dev/null +++ b/tests/agent/research/test_checkpoint_resume.py @@ -0,0 +1,387 @@ +"""HRM-93: Durable checkpoint + resume for the research loop. + +The job_runner-driven research loop must: + 1. Write checkpoint.json atomically at the end of each iteration. + 2. Read checkpoint.json (and history.json) on startup and resume from the + last completed iteration instead of restarting from baseline. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.runner import ( + ExperimentHistory, + ExperimentResult, +) +from agent.research.job_runner import _detect_resume +from agent.research.supervisor import ( + ResearchSupervisor, + TaskSpec, + _atomic_write_text, + _load_checkpoint, +) + + +# --------------------------------------------------------------------------- +# Helpers (mirrors test_research_supervisor.py fixtures) +# --------------------------------------------------------------------------- + +def _mock_delegate_json(metric_value: float, metric_key: str = "accuracy") -> str: + summary = ( + f"Done.\n" + f"METRIC: {metric_key}={metric_value} STATUS: improved NOTES: mock\n" + ) + return json.dumps({ + "results": [{ + "task_index": 0, + "status": "completed", + "summary": summary, + "api_calls": 1, + "duration_seconds": 0.1, + "exit_reason": "completed", + "tokens": {"input": 1, "output": 1}, + "tool_trace": [], + }], + "total_duration_seconds": 0.1, + }) + + +@pytest.fixture() +def parent_agent() -> MagicMock: + a = MagicMock() + a.model = "claude-sonnet-4-6" + a.base_url = "https://example" + a.api_key = "k" + a.provider = "anthropic" + a.api_mode = "anthropic_messages" + a.providers_allowed = None + a.providers_ignored = None + a.providers_order = None + a.provider_sort = None + a.enabled_toolsets = ["terminal", "file"] + a._delegate_depth = 0 + a._active_children = [] + a._active_children_lock = None + return a + + +@pytest.fixture() +def code_spec() -> TaskSpec: + return TaskSpec( + topic="t", + deliverable="d", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + ) + + +def _make_result(iteration: int, metric: float, run_id: str = "rid") -> ExperimentResult: + return ExperimentResult( + run_id=run_id, + iteration=iteration, + code=f"# iter {iteration}", + metrics={"accuracy": metric}, + primary_metric=metric, + improved=True, + kept=True, + elapsed_sec=0.1, + stdout=f"METRIC: accuracy={metric}", + stderr="", + error=None, + ) + + +def _write_snapshot_stub(checkpoint_dir: Path, iteration: int) -> None: + """Write a minimal snapshots/iter-{N}.json so _load_checkpoint's + consistency check accepts the resume.""" + snap_dir = checkpoint_dir / "snapshots" + snap_dir.mkdir(parents=True, exist_ok=True) + (snap_dir / f"iter-{iteration}.json").write_text(json.dumps({ + "iteration": iteration, + "messages": [], + "metrics": {}, + "files": [], + })) + + +# --------------------------------------------------------------------------- +# Atomic write +# --------------------------------------------------------------------------- + +class TestAtomicWrite: + def test_writes_file_with_final_content(self, tmp_path: Path): + target = tmp_path / "x.json" + _atomic_write_text(target, '{"a": 1}') + assert target.read_text() == '{"a": 1}' + + def test_no_tmp_left_behind(self, tmp_path: Path): + target = tmp_path / "x.json" + _atomic_write_text(target, '{"a": 1}') + leftovers = [p.name for p in tmp_path.iterdir() if p.name.endswith(".tmp")] + assert leftovers == [], f"unexpected .tmp leftovers: {leftovers}" + + def test_overwrite_existing(self, tmp_path: Path): + target = tmp_path / "x.json" + target.write_text("OLD") + _atomic_write_text(target, "NEW") + assert target.read_text() == "NEW" + + +# --------------------------------------------------------------------------- +# Checkpoint loading +# --------------------------------------------------------------------------- + +class TestLoadCheckpoint: + def test_returns_none_when_missing(self, tmp_path: Path): + assert _load_checkpoint(tmp_path) is None + + def test_returns_none_when_only_checkpoint_present(self, tmp_path: Path): + # checkpoint.json without history.json is unusable for resume + (tmp_path / "checkpoint.json").write_text(json.dumps({"round": 1})) + assert _load_checkpoint(tmp_path) is None + + def test_loads_history_and_iteration(self, tmp_path: Path): + history = ExperimentHistory() + history.add(_make_result(0, 0.5)) + history.add(_make_result(1, 0.7)) + history.best_result = history.results[1] + + (tmp_path / "history.json").write_text(json.dumps(history.to_dict())) + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 1, + "total_rounds": 2, + "best_metric": 0.7, + "updated_at": 0, + })) + _write_snapshot_stub(tmp_path, 1) + + loaded = _load_checkpoint(tmp_path) + assert loaded is not None + loaded_history, current_iteration = loaded + assert current_iteration == 1 + assert len(loaded_history.results) == 2 + assert loaded_history.best_result is not None + assert loaded_history.best_result.primary_metric == pytest.approx(0.7) + + +class TestLoadCheckpointConsistency: + """Fix-2: cross-file consistency between checkpoint.json, history.json, + and snapshots/iter-{N}.json. A crash mid-write must not produce a + silent resume that skips a round.""" + + def test_returns_none_when_history_shorter_than_round(self, tmp_path: Path): + # checkpoint claims round=2 (3 iterations completed) but history + # only contains 1 result. Resuming would silently skip rounds. + history = ExperimentHistory() + history.add(_make_result(0, 0.5)) + (tmp_path / "history.json").write_text(json.dumps(history.to_dict())) + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 2, "total_rounds": 3, "best_metric": 0.5, "updated_at": 0, + })) + _write_snapshot_stub(tmp_path, 2) + + assert _load_checkpoint(tmp_path) is None + + def test_returns_none_when_history_longer_than_round(self, tmp_path: Path): + # Symmetric mismatch: history has 3 results but checkpoint says + # round=1. Either checkpoint.json or history.json was clobbered. + history = ExperimentHistory() + for i in range(3): + history.add(_make_result(i, 0.1 * (i + 1))) + (tmp_path / "history.json").write_text(json.dumps(history.to_dict())) + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 1, "total_rounds": 3, "best_metric": 0.3, "updated_at": 0, + })) + _write_snapshot_stub(tmp_path, 1) + + assert _load_checkpoint(tmp_path) is None + + def test_returns_none_when_snapshot_missing(self, tmp_path: Path): + history = ExperimentHistory() + history.add(_make_result(0, 0.4)) + history.add(_make_result(1, 0.6)) + history.add(_make_result(2, 0.8)) + (tmp_path / "history.json").write_text(json.dumps(history.to_dict())) + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 2, "total_rounds": 3, "best_metric": 0.8, "updated_at": 0, + })) + # Note: no _write_snapshot_stub call → iter-2.json is absent. + assert _load_checkpoint(tmp_path) is None + + def test_returns_state_when_all_consistent(self, tmp_path: Path): + history = ExperimentHistory() + history.add(_make_result(0, 0.4)) + history.add(_make_result(1, 0.6)) + history.add(_make_result(2, 0.8)) + history.best_result = history.results[-1] + (tmp_path / "history.json").write_text(json.dumps(history.to_dict())) + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 2, "total_rounds": 3, "best_metric": 0.8, "updated_at": 0, + })) + _write_snapshot_stub(tmp_path, 2) + + loaded = _load_checkpoint(tmp_path) + assert loaded is not None + loaded_history, current_iteration = loaded + assert current_iteration == 2 + assert len(loaded_history.results) == 3 + + +# --------------------------------------------------------------------------- +# End-to-end resume (mocked delegate) +# --------------------------------------------------------------------------- + +@pytest.mark.integration +class TestResumeFromCheckpoint: + def test_baseline_skipped_when_checkpoint_exists( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + """If checkpoint says iteration 0 (baseline) is done, supervisor must + not call delegate_task again for the baseline.""" + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + # Pre-seed checkpoint at iteration=0 (baseline complete) + history = ExperimentHistory() + history.add(_make_result(0, 0.42, run_id="resume-001")) + history.best_result = history.results[0] + (checkpoint_dir / "history.json").write_text(json.dumps(history.to_dict())) + (checkpoint_dir / "checkpoint.json").write_text(json.dumps({ + "round": 0, "total_rounds": 1, "best_metric": 0.42, "updated_at": 0, + })) + _write_snapshot_stub(checkpoint_dir, 0) + + with patch( + "tools.delegate_tool.delegate_task", + return_value=_mock_delegate_json(0.99), + ) as mock_delegate: + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + new_history = supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="resume-001", + max_iterations=0, # baseline only + llm=None, + checkpoint_dir=checkpoint_dir, + ) + + # Baseline was skipped → delegate_task should NOT have been invoked + assert mock_delegate.call_count == 0 + # Pre-existing baseline preserved + assert len(new_history.results) == 1 + assert new_history.results[0].primary_metric == pytest.approx(0.42) + + def test_iteration_loop_resumes_after_completed_round( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + """If checkpoint says round=2 is done, the next call should run only + the remaining iterations (3..max).""" + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + history = ExperimentHistory() + for i in range(3): # iterations 0, 1, 2 + history.add(_make_result(i, 0.1 * (i + 1), run_id="resume-002")) + history.best_result = history.results[-1] + (checkpoint_dir / "history.json").write_text(json.dumps(history.to_dict())) + (checkpoint_dir / "checkpoint.json").write_text(json.dumps({ + "round": 2, "total_rounds": 3, "best_metric": 0.3, "updated_at": 0, + })) + _write_snapshot_stub(checkpoint_dir, 2) + + # Stub the LLM so iterations 3..5 run + llm = MagicMock() + llm.chat.return_value = type("R", (), {"content": "x = 2"})() + + with patch( + "tools.delegate_tool.delegate_task", + return_value=_mock_delegate_json(0.5), + ) as mock_delegate: + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + new_history = supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="resume-002", + max_iterations=5, + llm=llm, + checkpoint_dir=checkpoint_dir, + ) + + # Only iterations 3, 4, 5 should run → at most 3 delegate calls. + # Early-stop may cut this short; assert it didn't redo earlier rounds. + assert mock_delegate.call_count <= 3 + # The new history must contain the original 3 results plus the new ones. + assert len(new_history.results) >= 3 + # The first three iterations are exactly the pre-seeded ones. + for i in range(3): + assert new_history.results[i].iteration == i + assert new_history.results[i].primary_metric == pytest.approx(0.1 * (i + 1)) + + def test_checkpoint_written_atomically_each_iteration( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + """After each iteration checkpoint.json must contain the current + round number and history.json must round-trip via from_dict.""" + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + with patch( + "tools.delegate_tool.delegate_task", + return_value=_mock_delegate_json(0.7), + ): + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="cp-001", + max_iterations=0, + llm=None, + checkpoint_dir=checkpoint_dir, + ) + + cp = json.loads((checkpoint_dir / "checkpoint.json").read_text()) + assert cp["round"] == 0 + history_data = json.loads((checkpoint_dir / "history.json").read_text()) + # Full-fidelity history → round-trips through from_dict + rebuilt = ExperimentHistory.from_dict(history_data) + assert len(rebuilt.results) == 1 + assert rebuilt.best_result is not None + + +# --------------------------------------------------------------------------- +# job_runner resume detection +# --------------------------------------------------------------------------- + +class TestDetectResume: + def test_returns_none_when_no_checkpoint(self, tmp_path: Path): + assert _detect_resume(tmp_path) is None + + def test_surfaces_round_and_best_metric(self, tmp_path: Path): + (tmp_path / "checkpoint.json").write_text(json.dumps({ + "round": 4, "total_rounds": 5, "best_metric": 0.92, "updated_at": 0, + })) + info = _detect_resume(tmp_path) + assert info is not None + assert info["resumed_from_round"] == 4 + assert info["resumed_total_rounds"] == 5 + assert info["resumed_best_metric"] == pytest.approx(0.92) + + def test_returns_none_on_corrupt_checkpoint(self, tmp_path: Path): + (tmp_path / "checkpoint.json").write_text("{not json") + assert _detect_resume(tmp_path) is None diff --git a/tests/agent/research/test_cost_accounting.py b/tests/agent/research/test_cost_accounting.py new file mode 100644 index 000000000000..c6a301435659 --- /dev/null +++ b/tests/agent/research/test_cost_accounting.py @@ -0,0 +1,468 @@ +"""HRM-100 — Cost accounting per iteration and per job. + +Covers: +- DelegateSandboxResult and ExperimentResult accept tokens_in/tokens_out/cost_usd +- ExperimentHistory round-trips through dict with cost fields +- Backward compat: missing cost fields default to 0 / 0.0 +- _run_worker extracts token/cost data from delegate_task JSON +- run_research returns iteration_costs + totals in result JSON +- job_runner writes totals into result.json +""" + +from __future__ import annotations + +import json +import os +import tempfile +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.runner import ( + DelegateSandboxResult, + ExperimentHistory, + ExperimentResult, + _result_from_dict, +) +from agent.research.supervisor import _call_delegate_task, ResearchSupervisor, TaskSpec + + +# --------------------------------------------------------------------------- +# DelegateSandboxResult + ExperimentResult fields +# --------------------------------------------------------------------------- + +def test_delegate_sandbox_result_defaults(): + r = DelegateSandboxResult(metrics={}, stdout="", stderr="", elapsed_sec=1.0) + assert r.tokens_in == 0 + assert r.tokens_out == 0 + assert r.cost_usd == 0.0 + + +def test_experiment_result_defaults(): + r = ExperimentResult( + run_id="r1", + iteration=0, + code="code", + metrics={}, + primary_metric=0.5, + improved=True, + kept=True, + elapsed_sec=1.0, + stdout="", + stderr="", + ) + assert r.tokens_in == 0 + assert r.tokens_out == 0 + assert r.cost_usd == 0.0 + + +def test_experiment_result_with_costs(): + r = ExperimentResult( + run_id="r1", + iteration=1, + code="code", + metrics={"m": 0.9}, + primary_metric=0.9, + improved=True, + kept=True, + elapsed_sec=2.0, + stdout="ok", + stderr="", + tokens_in=100, + tokens_out=50, + cost_usd=0.0015, + ) + assert r.tokens_in == 100 + assert r.tokens_out == 50 + assert r.cost_usd == 0.0015 + + +# --------------------------------------------------------------------------- +# ExperimentHistory round-trip +# --------------------------------------------------------------------------- + +def test_history_to_dict_includes_costs(): + hist = ExperimentHistory() + hist.add( + ExperimentResult( + run_id="r1", iteration=0, code="c", metrics={}, primary_metric=0.5, + improved=True, kept=True, elapsed_sec=1.0, stdout="", stderr="", + tokens_in=10, tokens_out=5, cost_usd=0.0001, + ) + ) + hist.add( + ExperimentResult( + run_id="r1", iteration=1, code="c2", metrics={}, primary_metric=0.6, + improved=True, kept=True, elapsed_sec=1.0, stdout="", stderr="", + tokens_in=20, tokens_out=10, cost_usd=0.0002, + ) + ) + d = hist.to_dict() + results = d["results"] + assert len(results) == 2 + assert results[0]["tokens_in"] == 10 + assert results[1]["cost_usd"] == 0.0002 + + +def test_history_from_dict_backward_compat(): + """Old checkpoints without cost fields load with zeros.""" + old = { + "results": [ + { + "run_id": "r1", + "iteration": 0, + "code": "c", + "metrics": {}, + "primary_metric": 0.5, + "improved": True, + "kept": True, + "elapsed_sec": 1.0, + "stdout": "", + "stderr": "", + "error": None, + } + ], + "best_result": None, + "baseline_metric": 0.5, + } + hist = ExperimentHistory.from_dict(old) + assert len(hist.results) == 1 + assert hist.results[0].tokens_in == 0 + assert hist.results[0].tokens_out == 0 + assert hist.results[0].cost_usd == 0.0 + + +def test_history_from_dict_with_costs(): + data = { + "results": [ + { + "run_id": "r1", + "iteration": 0, + "code": "c", + "metrics": {}, + "primary_metric": 0.5, + "improved": True, + "kept": True, + "elapsed_sec": 1.0, + "stdout": "", + "stderr": "", + "error": None, + "tokens_in": 42, + "tokens_out": 7, + "cost_usd": 0.003, + } + ], + "best_result": None, + "baseline_metric": 0.5, + } + hist = ExperimentHistory.from_dict(data) + assert hist.results[0].tokens_in == 42 + assert hist.results[0].tokens_out == 7 + assert hist.results[0].cost_usd == 0.003 + + +# --------------------------------------------------------------------------- +# _result_from_dict backward compat +# --------------------------------------------------------------------------- + +def test_result_from_dict_missing_cost_fields(): + data = { + "run_id": "r1", + "iteration": 0, + "code": "c", + "metrics": {}, + "primary_metric": 0.5, + "improved": True, + "kept": True, + "elapsed_sec": 1.0, + "stdout": "", + "stderr": "", + "error": None, + } + r = _result_from_dict(data) + assert r is not None + assert r.tokens_in == 0 + assert r.tokens_out == 0 + assert r.cost_usd == 0.0 + + +# --------------------------------------------------------------------------- +# _run_worker extracts tokens / cost from delegate_task JSON +# --------------------------------------------------------------------------- + +def test_run_worker_extracts_cost_data(monkeypatch): + """_run_worker should pluck tokens and _child_cost_usd from delegate result.""" + fake_result_json = { + "results": [ + { + "status": "completed", + "summary": "METRIC: pass_rate=0.8 STATUS: improved NOTES: ok", + "tokens": {"input": 123, "output": 45}, + "_child_cost_usd": 0.005, + } + ] + } + + def fake_call_delegate(goal, context, *, parent_agent, toolsets): + return fake_result_json + + monkeypatch.setattr( + "agent.research.supervisor._call_delegate_task", fake_call_delegate + ) + + parent = MagicMock() + parent.model = "gpt-4o" + supervisor = ResearchSupervisor(parent_agent=parent) + + spec = TaskSpec( + topic="test", + deliverable="test", + metric_key="pass_rate", + task_type="generic", + ) + + with tempfile.TemporaryDirectory() as tmpdir: + result = supervisor._run_worker( + goal="do it", + working_dir=tmpdir, + attempt="code", + spec=spec, + time_budget_sec=0, + iteration=0, + worker_toolsets=["terminal"], + llm=None, + ) + + assert result.tokens_in == 123 + assert result.tokens_out == 45 + assert result.cost_usd == 0.005 + + +def test_run_worker_degrades_when_no_cost_data(monkeypatch): + """If delegate_task omits tokens/cost, defaults must be zero.""" + fake_result_json = { + "results": [ + { + "status": "completed", + "summary": "METRIC: pass_rate=0.8 STATUS: improved NOTES: ok", + } + ] + } + + def fake_call_delegate(goal, context, *, parent_agent, toolsets): + return fake_result_json + + monkeypatch.setattr( + "agent.research.supervisor._call_delegate_task", fake_call_delegate + ) + + parent = MagicMock() + supervisor = ResearchSupervisor(parent_agent=parent) + + spec = TaskSpec( + topic="test", + deliverable="test", + metric_key="pass_rate", + task_type="generic", + ) + + with tempfile.TemporaryDirectory() as tmpdir: + result = supervisor._run_worker( + goal="do it", + working_dir=tmpdir, + attempt="code", + spec=spec, + time_budget_sec=0, + iteration=0, + worker_toolsets=["terminal"], + llm=None, + ) + + assert result.tokens_in == 0 + assert result.tokens_out == 0 + assert result.cost_usd == 0.0 + + +# --------------------------------------------------------------------------- +# run_research returns totals +# --------------------------------------------------------------------------- + +def test_run_research_returns_cost_totals(monkeypatch): + """run_research JSON must contain iteration_costs and totals.""" + from tools.research_tool import run_research + + hist = ExperimentHistory() + hist.add( + ExperimentResult( + run_id="r1", iteration=0, code="c", metrics={"m": 0.5}, + primary_metric=0.5, improved=True, kept=True, elapsed_sec=1.0, + stdout="", stderr="", tokens_in=100, tokens_out=50, cost_usd=0.001, + ) + ) + hist.add( + ExperimentResult( + run_id="r1", iteration=1, code="c2", metrics={"m": 0.7}, + primary_metric=0.7, improved=True, kept=True, elapsed_sec=1.0, + stdout="", stderr="", tokens_in=200, tokens_out=100, cost_usd=0.002, + ) + ) + + fake_supervisor = MagicMock() + fake_supervisor.run.return_value = hist + + monkeypatch.setattr( + "tools.research_tool.ResearchSupervisor", lambda **kw: fake_supervisor + ) + monkeypatch.setattr( + "hermes_constants.get_hermes_home", lambda: Path(tempfile.gettempdir()) + ) + + parent = MagicMock() + raw = run_research( + topic="t", + deliverable="d", + metric_key="m", + parent_agent=parent, + max_iterations=2, + ) + result = json.loads(raw) + + assert result["total_tokens_in"] == 300 + assert result["total_tokens_out"] == 150 + assert result["total_cost_usd"] == 0.003 + assert result["total_iterations"] == 2 + assert "iteration_costs" in result + assert len(result["iteration_costs"]) == 2 + assert result["iteration_costs"][0]["tokens_in"] == 100 + assert result["iteration_costs"][1]["cost_usd"] == 0.002 + + +def test_run_research_backward_compat_no_cost_data(monkeypatch): + """If history results have zero costs, totals must still be present.""" + from tools.research_tool import run_research + + hist = ExperimentHistory() + hist.add( + ExperimentResult( + run_id="r1", iteration=0, code="c", metrics={"m": 0.5}, + primary_metric=0.5, improved=True, kept=True, elapsed_sec=1.0, + stdout="", stderr="", + ) + ) + + fake_supervisor = MagicMock() + fake_supervisor.run.return_value = hist + + monkeypatch.setattr( + "tools.research_tool.ResearchSupervisor", lambda **kw: fake_supervisor + ) + monkeypatch.setattr( + "hermes_constants.get_hermes_home", lambda: Path(tempfile.gettempdir()) + ) + + parent = MagicMock() + raw = run_research( + topic="t", + deliverable="d", + metric_key="m", + parent_agent=parent, + max_iterations=1, + ) + result = json.loads(raw) + + assert result["total_tokens_in"] == 0 + assert result["total_tokens_out"] == 0 + assert result["total_cost_usd"] == 0.0 + assert result["total_iterations"] == 1 + assert len(result["iteration_costs"]) == 1 + assert result["iteration_costs"][0]["cost_usd"] == 0.0 + + +# --------------------------------------------------------------------------- +# job_runner wires totals into result.json +# --------------------------------------------------------------------------- + +def test_job_runner_writes_cost_totals(monkeypatch, tmp_path: Path): + """job_runner main must persist cost totals in result.json.""" + from agent.research import job_runner + + job_dir = tmp_path / "job" + job_dir.mkdir() + + spec = { + "job_id": "j1", + "job_dir": str(job_dir), + "topic": "t", + "deliverable": "d", + "metric_key": "m", + "max_iterations": 2, + } + spec_path = job_dir / "job.json" + spec_path.write_text(json.dumps(spec)) + + fake_result = { + "run_id": "r1", + "iterations": 2, + "best_metric": 0.9, + "metric_key": "m", + "metric_direction": "maximize", + "best_notes": "nice", + "workspace": "/tmp/ws", + "learnings_file": "/tmp/ws/l.jsonl", + "iteration_costs": [ + {"iteration": 0, "tokens_in": 10, "tokens_out": 5, "cost_usd": 0.0001}, + {"iteration": 1, "tokens_in": 20, "tokens_out": 10, "cost_usd": 0.0002}, + ], + "total_tokens_in": 30, + "total_tokens_out": 15, + "total_cost_usd": 0.0003, + "total_iterations": 2, + } + + monkeypatch.setattr( + "agent.research.job_runner._build_agent", lambda spec: MagicMock() + ) + + # With the subprocess model (Fix-3) we can't monkeypatch run_research in the + # child process. Instead mock _spawn_child to return a completed process + # and have _child_main write the fake result directly. + def _fake_child_main(spec_path: str) -> int: + spec = json.loads(Path(spec_path).read_text()) + job_dir = Path(spec["job_dir"]) + (job_dir / "result.json").write_text(json.dumps(fake_result)) + state_path = job_dir / "state.json" + state = json.loads(state_path.read_text()) if state_path.exists() else {} + state.update({"status": "completed", **fake_result}) + state_path.write_text(json.dumps(state, indent=2)) + return 0 + + monkeypatch.setattr( + "agent.research.job_runner._child_main", _fake_child_main + ) + + # Also mock _spawn_child so it calls our fake _child_main inline + # and returns a mock Popen that looks finished. + def _fake_spawn_child(spec_path: str): + rc = _fake_child_main(spec_path) + mock_proc = MagicMock() + mock_proc.poll.return_value = rc + mock_proc.pid = 12345 + return mock_proc + + monkeypatch.setattr( + "agent.research.job_runner._spawn_child", _fake_spawn_child + ) + + rc = job_runner.main(str(spec_path)) + assert rc == 0 + + result_path = job_dir / "result.json" + assert result_path.exists() + written = json.loads(result_path.read_text()) + assert written["total_tokens_in"] == 30 + assert written["total_tokens_out"] == 15 + assert written["total_cost_usd"] == 0.0003 + assert written["total_iterations"] == 2 + assert len(written["iteration_costs"]) == 2 diff --git a/tests/agent/research/test_events.py b/tests/agent/research/test_events.py new file mode 100644 index 000000000000..3751abb2f4f5 --- /dev/null +++ b/tests/agent/research/test_events.py @@ -0,0 +1,46 @@ +"""Tests for agent.research.events (HRM-101).""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from agent.research.events import ResearchEvent, emit_event + + +def test_event_enum_members(): + assert ResearchEvent.JOB_STARTED.name == "JOB_STARTED" + assert ResearchEvent.ITERATION_COMPLETED.name == "ITERATION_COMPLETED" + + +def test_emit_event_appends_jsonl(tmp_path: Path): + job_dir = tmp_path / "job" + job_dir.mkdir() + + emit_event(job_dir, ResearchEvent.JOB_STARTED, {"job_id": "j1"}) + emit_event(job_dir, ResearchEvent.ITERATION_COMPLETED, {"iteration": 1}) + + events_file = job_dir / "events.jsonl" + assert events_file.exists() + + lines = events_file.read_text().strip().split("\n") + assert len(lines) == 2 + + e0 = json.loads(lines[0]) + assert e0["event"] == "JOB_STARTED" + assert e0["data"]["job_id"] == "j1" + assert "ts" in e0 + + e1 = json.loads(lines[1]) + assert e1["event"] == "ITERATION_COMPLETED" + assert e1["data"]["iteration"] == 1 + + +def test_emit_event_with_none_data(tmp_path: Path): + job_dir = tmp_path / "job" + job_dir.mkdir() + + emit_event(job_dir, ResearchEvent.CHECKPOINT_SAVED) + lines = (job_dir / "events.jsonl").read_text().strip().split("\n") + assert json.loads(lines[0])["data"] == {} diff --git a/tests/agent/research/test_heartbeat_stale.py b/tests/agent/research/test_heartbeat_stale.py new file mode 100644 index 000000000000..82e4435643e5 --- /dev/null +++ b/tests/agent/research/test_heartbeat_stale.py @@ -0,0 +1,168 @@ +"""HRM-95 — heartbeat / stale detection. + +The child subprocess refreshes ``/heartbeat.json`` every +``HERMES_JOB_HEARTBEAT_INTERVAL`` seconds (default 30). The parent +watches that file; if it goes older than ``HERMES_JOB_STALE_THRESHOLD`` +(default 90) the parent kills the child and writes +``status="stale"``. External callers can probe staleness via +``tools.research_tool.check_research_stale``. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from unittest.mock import patch + +import pytest + +pytestmark = pytest.mark.live_system_guard_bypass + +from agent.research.job_runner import main as job_runner_main +from tools.research_tool import check_research_stale + + +@pytest.fixture +def fast_runner_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("HERMES_JOB_HEARTBEAT_INTERVAL", "0.3") + monkeypatch.setenv("HERMES_JOB_STALE_THRESHOLD", "1.5") + monkeypatch.setenv("HERMES_JOB_POLL_INTERVAL", "0.1") + monkeypatch.setenv("HERMES_JOB_SIGTERM_GRACE", "1") + import agent.research.job_runner as jr + jr._HEARTBEAT_INTERVAL = 0.3 + jr._STALE_THRESHOLD = 1.5 + jr._POLL_INTERVAL = 0.1 + jr._SIGTERM_GRACE = 1 + + +@pytest.fixture +def fake_spec(tmp_path: Path): + def _make(**overrides): + base = { + "job_id": "test-job", + "job_dir": str(tmp_path), + "topic": "t", + "deliverable": "d", + "metric_key": "m", + "timeout_sec": 0, + } + base.update(overrides) + spec_path = tmp_path / "job.json" + spec_path.write_text(json.dumps(base)) + return spec_path + return _make + + +class TestHeartbeatWriter: + def test_heartbeat_file_created(self, tmp_path: Path, fake_spec, fast_runner_env): + """The child writes heartbeat.json before exiting.""" + spec_path = fake_spec( + _test_mode="sleep", + _test_sleep_sec=0.5, + ) + rc = job_runner_main(str(spec_path)) + assert rc == 0 + hb_path = tmp_path / "heartbeat.json" + assert hb_path.exists() + data = json.loads(hb_path.read_text()) + assert "ts" in data + assert "pid" in data + + def test_heartbeat_pid_is_child(self, tmp_path: Path, fake_spec, fast_runner_env): + """heartbeat.json's pid is the child's, not the parent's.""" + spec_path = fake_spec( + _test_mode="sleep", + _test_sleep_sec=0.5, + ) + job_runner_main(str(spec_path)) + state = json.loads((tmp_path / "state.json").read_text()) + hb = json.loads((tmp_path / "heartbeat.json").read_text()) + assert hb["pid"] == state["child_pid"] + + +class TestParentStaleDetection: + def test_stale_child_killed_with_status_stale( + self, tmp_path: Path, fake_spec, fast_runner_env + ): + """If the child stops refreshing heartbeat, parent kills it.""" + spec_path = fake_spec( + timeout_sec=0, # no wall-clock timeout + _test_mode="freeze_heartbeat", + _test_sleep_sec=30, # would otherwise sleep forever + ) + t0 = time.monotonic() + rc = job_runner_main(str(spec_path)) + elapsed = time.monotonic() - t0 + + assert rc != 0 + state = json.loads((tmp_path / "state.json").read_text()) + assert state["status"] == "stale", f"got status={state.get('status')!r}" + # With threshold=1.5 and grace=1, kill should happen well before 30s. + assert elapsed < 10, f"stale kill took {elapsed:.1f}s" + + +class TestStaleChecker: + def test_stale_when_no_heartbeat(self, tmp_path: Path): + assert check_research_stale(str(tmp_path)) is True + + def test_stale_after_threshold(self, tmp_path: Path): + hb = tmp_path / "heartbeat.json" + hb.write_text(json.dumps({"ts": time.time() - 120, "pid": 1})) + assert check_research_stale(str(tmp_path)) is True + + def test_not_stale_within_threshold(self, tmp_path: Path): + hb = tmp_path / "heartbeat.json" + hb.write_text(json.dumps({"ts": time.time() - 30, "pid": 1})) + assert check_research_stale(str(tmp_path)) is False + + def test_stale_when_corrupt(self, tmp_path: Path): + hb = tmp_path / "heartbeat.json" + hb.write_text("{not json") + assert check_research_stale(str(tmp_path)) is True + + def test_custom_threshold_honored(self, tmp_path: Path): + hb = tmp_path / "heartbeat.json" + hb.write_text(json.dumps({"ts": time.time() - 10, "pid": 1})) + assert check_research_stale(str(tmp_path), stale_threshold_sec=5.0) is True + assert check_research_stale(str(tmp_path), stale_threshold_sec=60.0) is False + + +class TestResearchJobToolStale: + def test_status_marks_stale_when_heartbeat_missing(self, tmp_path: Path): + """_action_status flips status to 'stale' when heartbeat is gone.""" + from tools.research_job_tool import _action_status + + job_id = "stale-job" + job_dir = tmp_path / "research-jobs" / job_id + job_dir.mkdir(parents=True) + (job_dir / "state.json").write_text(json.dumps({ + "job_id": job_id, + "status": "running", + "process_session_id": "sess-1", + })) + + with patch("tools.research_job_tool._job_dir", return_value=job_dir): + result = _action_status({"job_id": job_id}) + parsed = json.loads(result) + assert parsed.get("status") == "stale" + assert "no heartbeat" in parsed.get("stale_reason", "").lower() + + def test_status_does_not_mark_stale_when_heartbeat_fresh(self, tmp_path: Path): + from tools.research_job_tool import _action_status + + job_id = "live-job" + job_dir = tmp_path / "research-jobs" / job_id + job_dir.mkdir(parents=True) + (job_dir / "state.json").write_text(json.dumps({ + "job_id": job_id, + "status": "running", + })) + (job_dir / "heartbeat.json").write_text(json.dumps({ + "ts": time.time(), "pid": 42, + })) + + with patch("tools.research_job_tool._job_dir", return_value=job_dir): + result = _action_status({"job_id": job_id}) + parsed = json.loads(result) + assert parsed.get("status") == "running" diff --git a/tests/agent/research/test_job_runner_timeout.py b/tests/agent/research/test_job_runner_timeout.py new file mode 100644 index 000000000000..12e4d28ecb80 --- /dev/null +++ b/tests/agent/research/test_job_runner_timeout.py @@ -0,0 +1,120 @@ +"""HRM-94 — parent-side timeout enforcement. + +The job_runner now spawns a fresh Python subprocess for the actual +research loop. The parent watches wall-clock time and signals the child +on expiry: SIGTERM, then SIGKILL after a 5 s grace. On timeout it +overwrites ``state.json`` with ``status="timeout"``. + +Tests use the ``_test_mode`` hook in the spec so we don't need to mock +``run_research`` across a process boundary — the child interprets the +hook and just sleeps. Tunables (poll interval, SIGTERM grace) are +shrunk via env vars to keep the suite fast. +""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.live_system_guard_bypass + +from agent.research.job_runner import main as job_runner_main + + +@pytest.fixture +def fast_runner_env(monkeypatch: pytest.MonkeyPatch) -> None: + """Shrink job_runner timeouts so the suite finishes in seconds. + + These env vars are read at import time by job_runner — the child + subprocess is a fresh interpreter that will read them from os.environ + we propagate via subprocess.Popen's default env inheritance. + """ + monkeypatch.setenv("HERMES_JOB_HEARTBEAT_INTERVAL", "0.3") + monkeypatch.setenv("HERMES_JOB_STALE_THRESHOLD", "60") # large — we want timeout, not stale + monkeypatch.setenv("HERMES_JOB_POLL_INTERVAL", "0.1") + monkeypatch.setenv("HERMES_JOB_SIGTERM_GRACE", "1") + # Module-level constants in the parent process were captured at + # import time. Rebind them so the running parent uses the test + # values too. + import agent.research.job_runner as jr + jr._HEARTBEAT_INTERVAL = 0.3 + jr._STALE_THRESHOLD = 60 + jr._POLL_INTERVAL = 0.1 + jr._SIGTERM_GRACE = 1 + + +@pytest.fixture +def fake_spec(tmp_path: Path): + def _make(**overrides): + base = { + "job_id": "test-job", + "job_dir": str(tmp_path), + "topic": "t", + "deliverable": "d", + "metric_key": "m", + "timeout_sec": 0, + } + base.update(overrides) + spec_path = tmp_path / "job.json" + spec_path.write_text(json.dumps(base)) + return spec_path + return _make + + +class TestJobRunnerTimeout: + def test_completes_within_timeout(self, tmp_path: Path, fake_spec, fast_runner_env): + """Child finishes before timeout → parent returns 0, status=completed.""" + spec_path = fake_spec( + timeout_sec=10, + _test_mode="sleep", + _test_sleep_sec=0.5, + ) + rc = job_runner_main(str(spec_path)) + assert rc == 0 + state = json.loads((tmp_path / "state.json").read_text()) + assert state["status"] == "completed" + + def test_timeout_writes_state_timeout(self, tmp_path: Path, fake_spec, fast_runner_env): + """Child runs longer than timeout → parent kills it, status=timeout.""" + spec_path = fake_spec( + timeout_sec=2, + _test_mode="sleep", + _test_sleep_sec=30, + ) + t0 = time.monotonic() + rc = job_runner_main(str(spec_path)) + elapsed = time.monotonic() - t0 + + assert rc != 0 + state = json.loads((tmp_path / "state.json").read_text()) + assert state.get("status") == "timeout", f"got status={state.get('status')!r}" + assert "Timed out" in state.get("error", "") + # Should kill within ~timeout + grace, not run the full 30 s sleep. + assert elapsed < 10, f"timeout took {elapsed:.1f}s — kill escalation broken?" + + def test_child_pid_recorded_in_state(self, tmp_path: Path, fake_spec, fast_runner_env): + spec_path = fake_spec( + timeout_sec=10, + _test_mode="sleep", + _test_sleep_sec=0.5, + ) + job_runner_main(str(spec_path)) + state = json.loads((tmp_path / "state.json").read_text()) + assert isinstance(state.get("child_pid"), int) + assert state["child_pid"] != os.getpid() + + def test_no_timeout_when_zero(self, tmp_path: Path, fake_spec, fast_runner_env): + """timeout_sec=0 disables the wall-clock guard.""" + spec_path = fake_spec( + timeout_sec=0, + _test_mode="sleep", + _test_sleep_sec=0.3, + ) + rc = job_runner_main(str(spec_path)) + assert rc == 0 + state = json.loads((tmp_path / "state.json").read_text()) + assert state["status"] == "completed" diff --git a/tests/agent/research/test_kanban_rename.py b/tests/agent/research/test_kanban_rename.py new file mode 100644 index 000000000000..37f1eeb300bc --- /dev/null +++ b/tests/agent/research/test_kanban_rename.py @@ -0,0 +1,40 @@ +"""Pin the May 9 Lattice → Kanban rename. + +The public surface for tracked research runs is `kanban_task_id`. The +old `lattice_task_id` parameter was removed when ProgressSink/KanbanSink +replaced the inline Lattice CLI shell-out. Tests in this file regress +if the rename gets undone or partially reverted. +""" + +import inspect + + +def _params(callable_obj): + return set(inspect.signature(callable_obj).parameters.keys()) + + +def test_run_research_signature_uses_kanban_task_id(): + from tools.research_tool import run_research + params = _params(run_research) + assert "kanban_task_id" in params, ( + "run_research must accept kanban_task_id (the post-Lattice public name)" + ) + assert "lattice_task_id" not in params, ( + "lattice_task_id was removed in the ProgressSink/KanbanSink refactor; " + "its return indicates an incomplete or reverted rename" + ) + + +def test_research_job_signature_uses_kanban_task_id(): + from tools.research_job_tool import RESEARCH_JOB_SCHEMA + props = RESEARCH_JOB_SCHEMA["parameters"]["properties"] + assert "kanban_task_id" in props + assert "lattice_task_id" not in props + + +def test_research_job_handler_signature_uses_kanban_task_id(): + """The _action_start handler must thread kanban_task_id through to spec.""" + import tools.research_job_tool as mod + src = inspect.getsource(mod._action_start) + assert "kanban_task_id" in src + assert "lattice_task_id" not in src diff --git a/tests/agent/research/test_polish_fixes.py b/tests/agent/research/test_polish_fixes.py new file mode 100644 index 000000000000..deef85dec40d --- /dev/null +++ b/tests/agent/research/test_polish_fixes.py @@ -0,0 +1,256 @@ +"""Tests locking in the three polish fixes for ResearchSupervisor mechanics: + +1. acceptance_criterion is parsed and short-circuits the loop +2. _observe distinguishes plateau (neutral) from regression +3. disable_evolution_overlay suppresses the overlay loader +""" +from __future__ import annotations + +import json +import tempfile +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.runner import ExperimentResult +from agent.research.supervisor import ( + ResearchSupervisor, + TaskSpec, + _parse_acceptance_criterion, +) + + +# --------------------------------------------------------------------------- +# 1. Acceptance criterion parser +# --------------------------------------------------------------------------- + +class TestAcceptanceCriterionParser: + def test_parses_geq(self): + test = _parse_acceptance_criterion("pass_rate >= 0.9") + assert test is not None + assert test(0.95) is True + assert test(0.9) is True + assert test(0.89) is False + + def test_parses_lt(self): + test = _parse_acceptance_criterion("latency_ms < 200") + assert test is not None + assert test(199.9) is True + assert test(200) is False + assert test(250) is False + + def test_parses_op_only(self): + test = _parse_acceptance_criterion(">= 0.5") + assert test is not None + assert test(0.5) is True + assert test(0.4) is False + + def test_parses_negative_threshold(self): + test = _parse_acceptance_criterion("delta > -0.01") + assert test is not None + assert test(0.0) is True + assert test(-0.02) is False + + def test_qualitative_returns_none(self): + assert _parse_acceptance_criterion("looks good to a human reviewer") is None + + def test_empty_returns_none(self): + assert _parse_acceptance_criterion("") is None + + +# --------------------------------------------------------------------------- +# 2. _observe plateau / regression / neutral +# --------------------------------------------------------------------------- + +def _make_result(iteration, primary_metric, improved): + return ExperimentResult( + run_id="test", + iteration=iteration, + code="", + metrics={"pass_rate": str(primary_metric)} if primary_metric is not None else {}, + primary_metric=primary_metric, + improved=improved, + kept=improved, + elapsed_sec=0.0, + stdout="METRIC: pass_rate=%s NOTES: t" % primary_metric, + stderr="", + error=None, + ) + + +class TestObserveClassification: + def setup_method(self): + self.tmp = Path(tempfile.mkdtemp(prefix="observe-test-")) + self.spec = TaskSpec( + topic="t", deliverable="d", + metric_key="pass_rate", metric_direction="maximize", + ) + self.sup = ResearchSupervisor( + parent_agent=MagicMock(), workspace=self.tmp, + ) + + def _last_type(self): + line = (self.tmp / "learnings.jsonl").read_text().strip().splitlines()[-1] + return json.loads(line)["type"] + + def test_improvement_when_strictly_better(self): + r = _make_result(1, 0.9, improved=True) + self.sup._observe(r, self.spec, self.tmp, previous_best=0.7) + assert self._last_type() == "improvement" + + def test_neutral_when_equal_to_best(self): + r = _make_result(2, 0.8, improved=False) + self.sup._observe(r, self.spec, self.tmp, previous_best=0.8) + assert self._last_type() == "neutral" + + def test_regression_when_strictly_worse_maximize(self): + r = _make_result(3, 0.6, improved=False) + self.sup._observe(r, self.spec, self.tmp, previous_best=0.8) + assert self._last_type() == "regression" + + def test_regression_when_strictly_worse_minimize(self): + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="latency_ms", metric_direction="minimize", + ) + r = _make_result(2, 250.0, improved=False) + self.sup._observe(r, spec, self.tmp, previous_best=200.0) + assert self._last_type() == "regression" + + def test_failure_when_metric_none(self): + r = _make_result(1, None, improved=False) + self.sup._observe(r, self.spec, self.tmp, previous_best=0.5) + assert self._last_type() == "failure" + + def test_neutral_when_no_prior_best_and_not_improved(self): + # Edge case: result.improved=False with previous_best=None + # (e.g. first round had no metric). Should be neutral, not regression. + r = _make_result(0, 0.5, improved=False) + self.sup._observe(r, self.spec, self.tmp, previous_best=None) + assert self._last_type() == "neutral" + + +# --------------------------------------------------------------------------- +# 3. disable_evolution_overlay +# --------------------------------------------------------------------------- + +class TestDisableEvolutionOverlay: + def test_disabled_skips_loader(self): + tmp = Path(tempfile.mkdtemp(prefix="overlay-test-")) + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="m", metric_direction="maximize", + ) + sup = ResearchSupervisor( + parent_agent=MagicMock(), workspace=tmp, + ) + # Sentinel: if this gets called when disabled, the test fails. + with patch.object(sup, "_load_evolution_overlay") as mock_loader: + mock_loader.side_effect = AssertionError("should not be called when disabled") + with patch("agent.research.supervisor._call_delegate_task") as mock_dt: + mock_dt.return_value = {"results": [{"status": "completed", "summary": "METRIC: m=1.0"}]} + sup.run( + spec, initial_attempt="x", run_id="t", + max_iterations=0, # baseline only + disable_evolution_overlay=True, + ) + mock_loader.assert_not_called() + + def test_enabled_calls_loader(self): + tmp = Path(tempfile.mkdtemp(prefix="overlay-test-")) + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="m", metric_direction="maximize", + ) + sup = ResearchSupervisor( + parent_agent=MagicMock(), workspace=tmp, + ) + with patch.object(sup, "_load_evolution_overlay", return_value="") as mock_loader: + with patch("agent.research.supervisor._call_delegate_task") as mock_dt: + mock_dt.return_value = {"results": [{"status": "completed", "summary": "METRIC: m=1.0"}]} + sup.run( + spec, initial_attempt="x", run_id="t", + max_iterations=0, + disable_evolution_overlay=False, + ) + mock_loader.assert_called_once() + + +# --------------------------------------------------------------------------- +# 4. Acceptance criterion early termination (integration) +# --------------------------------------------------------------------------- + +@pytest.mark.integration +class TestAcceptanceTerminationE2E: + """Full supervisor.run() with scripted scores. Loop must short-circuit + when the metric crosses the criterion threshold.""" + + def test_loop_terminates_when_acceptance_met(self): + scores = [0.50, 0.70, 0.95, 0.99] # iter 2 should terminate + n = {"i": 0} + + def fake_delegate(*args, **kwargs): + idx = min(n["i"], len(scores) - 1) + n["i"] += 1 + return {"results": [{ + "status": "completed", + "summary": f"METRIC: pass_rate={scores[idx]} NOTES: iter-{idx}", + }]} + + tmp = Path(tempfile.mkdtemp(prefix="accept-e2e-")) + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="pass_rate", metric_direction="maximize", + acceptance_criterion="pass_rate >= 0.9", + ) + sup = ResearchSupervisor( + parent_agent=MagicMock(), workspace=tmp, + ) + stub_llm = MagicMock() + stub_llm.chat.return_value = MagicMock(content="```python\nx\n```") + + with patch("agent.research.supervisor._call_delegate_task", side_effect=fake_delegate): + history = sup.run( + spec, initial_attempt="x", run_id="t", + max_iterations=10, + llm=stub_llm, + disable_evolution_overlay=True, + ) + # Must have stopped at iter 2 (first score >= 0.9), not run all 10. + assert len(history.results) == 3, f"expected 3 iters (0,1,2), got {len(history.results)}" + assert history.best_result.primary_metric == 0.95 + + def test_loop_runs_to_max_when_acceptance_not_met(self): + scores = [0.50, 0.60, 0.70] # never crosses 0.9 + n = {"i": 0} + + def fake_delegate(*args, **kwargs): + idx = min(n["i"], len(scores) - 1) + n["i"] += 1 + return {"results": [{ + "status": "completed", + "summary": f"METRIC: pass_rate={scores[idx]} NOTES: iter-{idx}", + }]} + + tmp = Path(tempfile.mkdtemp(prefix="accept-fail-")) + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="pass_rate", metric_direction="maximize", + acceptance_criterion="pass_rate >= 0.9", + ) + sup = ResearchSupervisor( + parent_agent=MagicMock(), workspace=tmp, + ) + stub_llm = MagicMock() + stub_llm.chat.return_value = MagicMock(content="```python\nx\n```") + + with patch("agent.research.supervisor._call_delegate_task", side_effect=fake_delegate): + history = sup.run( + spec, initial_attempt="x", run_id="t", + max_iterations=2, + llm=stub_llm, + disable_evolution_overlay=True, + ) + # All 3 iterations executed (0, 1, 2) since acceptance never met. + assert len(history.results) == 3 diff --git a/tests/agent/research/test_research_job_oauth.py b/tests/agent/research/test_research_job_oauth.py new file mode 100644 index 000000000000..6edb8a8be9b7 --- /dev/null +++ b/tests/agent/research/test_research_job_oauth.py @@ -0,0 +1,104 @@ +"""Verify provider-neutral config handling for detached research jobs. + +The spec generator inherits the user's configured delegation runtime first, +then the main runtime. It must not pin provider-specific API keys from env: +credential resolution belongs to the normal Hermes provider/auth machinery in +AIAgent, not to the research-job tool. +""" +from __future__ import annotations + +import json + + +def test_spec_omits_api_key_when_env_unset(monkeypatch, tmp_path): + """_load_config_for_job must not pin provider-specific API keys.""" + monkeypatch.delenv("KIMI_API_KEY", raising=False) + + config_dir = tmp_path / "hermes-home" + config_dir.mkdir() + (config_dir / "config.yaml").write_text( + "model:\n default: kimi-k2.6\n provider: kimi-coding\n" + ) + monkeypatch.setattr( + "tools.research_job_tool.get_hermes_home", + lambda: config_dir, + ) + + from tools.research_job_tool import _load_config_for_job + spec = _load_config_for_job() + assert "api_key" not in spec, ( + f"Spec must not pin provider-specific API keys (got {spec!r}). " + "Credential resolution belongs to the standard Hermes provider/auth path." + ) + assert spec["provider"] == "kimi-coding" + assert spec["model"] == "kimi-k2.6" + + +def test_spec_prefers_delegation_runtime(monkeypatch, tmp_path): + """Detached research jobs inherit delegation runtime when configured.""" + config_dir = tmp_path / "hermes-home" + config_dir.mkdir() + (config_dir / "config.yaml").write_text( + "model:\n" + " default: main-model\n" + " provider: main-provider\n" + "delegation:\n" + " model: delegate-model\n" + " provider: delegate-provider\n" + " base_url: https://example.invalid/v1\n" + " api_mode: chat_completions\n" + ) + monkeypatch.setattr( + "tools.research_job_tool.get_hermes_home", + lambda: config_dir, + ) + + from tools.research_job_tool import _load_config_for_job + spec = _load_config_for_job() + assert spec == { + "model": "delegate-model", + "provider": "delegate-provider", + "base_url": "https://example.invalid/v1", + "api_mode": "chat_completions", + } + + +def test_action_start_does_not_write_null_api_key(monkeypatch, tmp_path): + """When resolved cfg has no api_key, persisted spec.json has no null api_key.""" + monkeypatch.setattr( + "tools.research_job_tool._load_config_for_job", + lambda: { + "model": "test-model", + "provider": "test-provider", + "base_url": "https://example.invalid/v1", + }, + ) + monkeypatch.setattr( + "tools.research_job_tool._job_dir", + lambda job_id: tmp_path / "jobs" / job_id, + ) + + monkeypatch.setattr( + "tools.terminal_tool.terminal_tool", + lambda **kw: json.dumps({"session_id": "stub-session", "pid": 12345}), + ) + + from tools.research_job_tool import _action_start + + out = _action_start({ + "topic": "trivial sort fn", + "deliverable": "a Python sort function", + "metric_key": "pass_rate", + "acceptance_criterion": "pass_rate >= 1.0", + "max_iterations": 1, + }) + + result = json.loads(out) + job_id = result["job_id"] + spec_path = tmp_path / "jobs" / job_id / "job.json" + spec = json.loads(spec_path.read_text()) + + assert "api_key" not in spec, ( + f"Spec must not write null api_key to job.json (got {spec!r})" + ) + assert spec["provider"] == "test-provider" diff --git a/tests/agent/research/test_research_job_tool_config.py b/tests/agent/research/test_research_job_tool_config.py new file mode 100644 index 000000000000..7512ba6ba702 --- /dev/null +++ b/tests/agent/research/test_research_job_tool_config.py @@ -0,0 +1,21 @@ +"""Tests for config consistency between research_tool and research_job_tool (HRM-102).""" +from __future__ import annotations + +import pytest + +from tools.research_job_tool import research_job + + +def test_research_job_accepts_acceptance_criterion(): + """research_job must accept acceptance_criterion without error.""" + # Only verify the function signature accepts the param + import inspect + sig = inspect.signature(research_job) + assert "acceptance_criterion" in sig.parameters + + +def test_research_job_accepts_timeout_sec(): + """research_job must accept timeout_sec without error.""" + import inspect + sig = inspect.signature(research_job) + assert "timeout_sec" in sig.parameters diff --git a/tests/agent/research/test_run_research_auto_specify.py b/tests/agent/research/test_run_research_auto_specify.py new file mode 100644 index 000000000000..a260aa4ea339 --- /dev/null +++ b/tests/agent/research/test_run_research_auto_specify.py @@ -0,0 +1,160 @@ +"""run_research(auto_specify=True) fills missing fields from a vague topic +without ever overriding explicit caller values.""" +from __future__ import annotations + +import json +from unittest.mock import MagicMock, patch + +import pytest + + +SCAFFOLD = { + "deliverable": "Python classify(payload) function", + "metric_key": "pass_rate", + "metric_direction": "minimize", # deliberately not the default + "task_type": "code", + "evaluation_mode": "llm_judge", # deliberately not the default + "evaluation_prompt": "Score 0-1: does the function classify correctly?", +} + + +def _captured_supervisor_factory(captured: dict): + def factory(**kwargs): + sup = MagicMock() + sup.run.return_value = MagicMock(results=[], best_result=None) + captured["sup"] = sup + return sup + return factory + + +class TestAutoSpecifyFillsMissingFields: + def test_fills_when_deliverable_and_metric_empty(self): + captured: dict = {} + with patch( + "agent.research.auto_specify.auto_specify_topic", + return_value=SCAFFOLD, + ), patch( + "tools.research_tool.ResearchSupervisor", + side_effect=_captured_supervisor_factory(captured), + ): + from tools.research_tool import run_research + out = run_research( + topic="classify daemoncraft heartbeat events", + parent_agent=MagicMock(), + auto_specify=True, + disable_evolution_overlay=True, + ) + json.loads(out) # smoke: must be valid JSON + assert captured["sup"].run.called + spec = captured["sup"].run.call_args.args[0] + # Phase C must adopt scaffold values that differ from the run_research defaults. + assert spec.deliverable == SCAFFOLD["deliverable"] + assert spec.metric_key == SCAFFOLD["metric_key"] + assert spec.metric_direction == "minimize" + assert spec.task_type == "code" + assert spec.evaluation_mode == "llm_judge" + assert spec.evaluation_prompt == SCAFFOLD["evaluation_prompt"] + + def test_does_not_override_explicit_caller_values(self): + """Caller passed deliverable + metric_key + task_type explicitly → + scaffold's competing values must be ignored.""" + captured: dict = {} + with patch( + "agent.research.auto_specify.auto_specify_topic", + return_value=SCAFFOLD, + ), patch( + "tools.research_tool.ResearchSupervisor", + side_effect=_captured_supervisor_factory(captured), + ): + from tools.research_tool import run_research + run_research( + topic="some topic", + deliverable="EXPLICIT deliverable", + metric_key="EXPLICIT_metric", + task_type="research", # explicit, must stick + parent_agent=MagicMock(), + auto_specify=True, + disable_evolution_overlay=True, + ) + spec = captured["sup"].run.call_args.args[0] + assert spec.deliverable == "EXPLICIT deliverable" + assert spec.metric_key == "EXPLICIT_metric" + assert spec.task_type == "research" + + def test_falls_back_when_aux_returns_none(self): + """auto_specify failure must not crash run_research.""" + captured: dict = {} + with patch( + "agent.research.auto_specify.auto_specify_topic", + return_value=None, + ), patch( + "tools.research_tool.ResearchSupervisor", + side_effect=_captured_supervisor_factory(captured), + ): + from tools.research_tool import run_research + run_research( + topic="vague", + parent_agent=MagicMock(), + auto_specify=True, + disable_evolution_overlay=True, + ) + spec = captured["sup"].run.call_args.args[0] + # Defaults for missing fields stick. + assert spec.deliverable == "" + assert spec.metric_key == "" + assert spec.metric_direction == "maximize" + assert spec.task_type == "generic" + assert spec.evaluation_mode == "self_report" + + def test_disabled_does_not_call_aux(self): + """auto_specify=False (default) must not invoke the aux LLM.""" + captured: dict = {} + called = {"aux": False} + + def fail_aux(*a, **kw): + called["aux"] = True + return SCAFFOLD + + with patch( + "agent.research.auto_specify.auto_specify_topic", + side_effect=fail_aux, + ), patch( + "tools.research_tool.ResearchSupervisor", + side_effect=_captured_supervisor_factory(captured), + ): + from tools.research_tool import run_research + run_research( + topic="t", + deliverable="d", + metric_key="m", + parent_agent=MagicMock(), + disable_evolution_overlay=True, + ) + assert called["aux"] is False + + def test_skipped_when_caller_supplied_both_required_fields(self): + """If caller provided deliverable AND metric_key, auto_specify is a no-op.""" + called = {"aux": False} + + def fail_aux(*a, **kw): + called["aux"] = True + return SCAFFOLD + + captured: dict = {} + with patch( + "agent.research.auto_specify.auto_specify_topic", + side_effect=fail_aux, + ), patch( + "tools.research_tool.ResearchSupervisor", + side_effect=_captured_supervisor_factory(captured), + ): + from tools.research_tool import run_research + run_research( + topic="t", + deliverable="explicit", + metric_key="explicit_metric", + parent_agent=MagicMock(), + auto_specify=True, + disable_evolution_overlay=True, + ) + assert called["aux"] is False diff --git a/tests/agent/research/test_sinks.py b/tests/agent/research/test_sinks.py new file mode 100644 index 000000000000..9a4b37b7f665 --- /dev/null +++ b/tests/agent/research/test_sinks.py @@ -0,0 +1,151 @@ +"""Tests for ProgressSink Protocol and concrete sink implementations.""" +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from agent.research.runner import ExperimentResult +from agent.research.sinks import StubSink +from agent.research.supervisor import TaskSpec + + +def _spec() -> TaskSpec: + return TaskSpec( + topic="t", deliverable="d", + metric_key="pass_rate", metric_direction="maximize", + ) + + +def _result(iteration: int = 0, primary_metric: float = 0.5) -> ExperimentResult: + return ExperimentResult( + run_id="rid", iteration=iteration, code="", + metrics={"pass_rate": str(primary_metric)}, + primary_metric=primary_metric, + improved=True, kept=True, + elapsed_sec=0.1, stdout="", stderr="", error=None, + ) + + +class TestStubSink: + def test_run_started_does_not_raise(self): + StubSink().run_started(_spec(), "rid") + + def test_iteration_observed_does_not_raise(self, tmp_path: Path): + StubSink().iteration_observed(0, _result(), tmp_path) + + def test_run_completed_does_not_raise(self): + history = MagicMock() + history.results = [_result()] + history.best_result = _result() + StubSink().run_completed(history) + + def test_comment_does_not_raise(self): + StubSink().comment("hello") + + +import sqlite3 + +from hermes_cli import kanban_db + +from agent.research.sinks import KanbanSink + + +@pytest.fixture +def kanban_db_path(tmp_path, monkeypatch): + """Pin a clean kanban.db path. Use HERMES_KANBAN_DB so kanban_db_path() + resolves through the env override (highest-precedence) and we skip + the board / current-board state machine entirely. connect() auto- + initializes the schema, so we don't call init_db(). + """ + db_path = tmp_path / "kanban.db" + monkeypatch.setenv("HERMES_KANBAN_DB", str(db_path)) + return db_path + + +@pytest.fixture +def kanban_conn(kanban_db_path): + """Helper: an OPEN connection for tests that want to inspect/insert + directly. Production code (KanbanSink) opens its own short-lived + connection per call; this fixture is for test setup + assertion only.""" + conn = kanban_db.connect(kanban_db_path) + yield conn + conn.close() + + +class TestKanbanSink: + def test_existing_task_id_appends_comments(self, kanban_db_path, kanban_conn): + task_id = kanban_db.create_task( + kanban_conn, title="parent run", body="research run wrapper", + created_by="test", + ) + sink = KanbanSink(task_id=task_id, db_path=kanban_db_path) + sink.run_started(_spec(), "rid-001") + sink.iteration_observed(0, _result(0, 0.5), Path("/tmp")) + sink.iteration_observed(1, _result(1, 0.7), Path("/tmp")) + + comments = kanban_db.list_comments(kanban_conn, task_id) + assert len(comments) == 3 + assert "rid-001" in comments[0].body + assert "0.5" in comments[1].body + assert "0.7" in comments[2].body + + def test_run_completed_completes_task(self, kanban_db_path, kanban_conn): + task_id = kanban_db.create_task( + kanban_conn, title="r", body="b", created_by="test", + ) + sink = KanbanSink(task_id=task_id, db_path=kanban_db_path) + history = MagicMock() + history.results = [_result(0, 0.5), _result(1, 0.9)] + history.best_result = _result(1, 0.9) + sink.run_completed(history) + + task = kanban_db.get_task(kanban_conn, task_id) + assert task.status == "done" + + def test_complete_on_run_completed_false_keeps_task_open( + self, kanban_db_path, kanban_conn, + ): + """A/B testing case: per-strategy sub-sinks must NOT close the task.""" + task_id = kanban_db.create_task( + kanban_conn, title="r", body="b", created_by="test", + ) + sink = KanbanSink( + task_id=task_id, + db_path=kanban_db_path, + complete_on_run_completed=False, + ) + history = MagicMock() + history.results = [_result(0, 0.9)] + history.best_result = _result(0, 0.9) + sink.run_completed(history) + + task = kanban_db.get_task(kanban_conn, task_id) + assert task.status != "done" + + def test_no_task_id_is_log_only(self, kanban_db_path, kanban_conn): + sink = KanbanSink(task_id=None, db_path=kanban_db_path) + sink.run_started(_spec(), "rid") + sink.iteration_observed(0, _result(), Path("/tmp")) + sink.run_completed(MagicMock(results=[], best_result=None)) + sink.comment("hi") + # No task should have been created. + all_tasks = kanban_db.list_tasks(kanban_conn) + assert len(all_tasks) == 0 + + def test_db_error_does_not_raise( + self, kanban_db_path, kanban_conn, monkeypatch, + ): + task_id = kanban_db.create_task( + kanban_conn, title="r", body="b", created_by="test", + ) + + def boom(*a, **kw): + raise sqlite3.OperationalError("forced") + + monkeypatch.setattr(kanban_db, "add_comment", boom) + sink = KanbanSink(task_id=task_id, db_path=kanban_db_path) + # Must not raise. + sink.run_started(_spec(), "rid") + sink.iteration_observed(0, _result(), Path("/tmp")) diff --git a/tests/agent/research/test_snapshots.py b/tests/agent/research/test_snapshots.py new file mode 100644 index 000000000000..a6e61cab18ba --- /dev/null +++ b/tests/agent/research/test_snapshots.py @@ -0,0 +1,318 @@ +"""HRM-96: Atomic workspace snapshots per iteration. + +After every completed iteration the supervisor must write +/snapshots/iter-{N}.json containing: + - iteration + - messages (full ExperimentHistory results up to iteration N) + - metrics (last result's metrics dict) + - files (list of {path, content} for the round directory) + +Rollback is a single operation: restore_snapshot(snapshot_path, target_dir) +must rewrite every captured file at its captured relative path inside +target_dir. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.supervisor import ( + ResearchSupervisor, + TaskSpec, + restore_snapshot, +) + + +def _mock_delegate_json(metric_value: float, metric_key: str = "accuracy") -> str: + summary = ( + f"Done.\n" + f"METRIC: {metric_key}={metric_value} STATUS: improved NOTES: mock\n" + ) + return json.dumps({ + "results": [{ + "task_index": 0, + "status": "completed", + "summary": summary, + "api_calls": 1, + "duration_seconds": 0.1, + "exit_reason": "completed", + "tokens": {"input": 1, "output": 1}, + "tool_trace": [], + }], + "total_duration_seconds": 0.1, + }) + + +@pytest.fixture() +def parent_agent() -> MagicMock: + a = MagicMock() + a.model = "claude-sonnet-4-6" + a.base_url = "https://example" + a.api_key = "k" + a.provider = "anthropic" + a.api_mode = "anthropic_messages" + a.providers_allowed = None + a.providers_ignored = None + a.providers_order = None + a.provider_sort = None + a.enabled_toolsets = ["terminal", "file"] + a._delegate_depth = 0 + a._active_children = [] + a._active_children_lock = None + return a + + +@pytest.fixture() +def code_spec() -> TaskSpec: + return TaskSpec( + topic="t", + deliverable="d", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + ) + + +# --------------------------------------------------------------------------- +# Snapshot capture +# --------------------------------------------------------------------------- + +@pytest.mark.integration +class TestSnapshotCapture: + def test_snapshots_dir_created( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + with patch( + "tools.delegate_tool.delegate_task", + return_value=_mock_delegate_json(0.5), + ): + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="snap-001", + max_iterations=0, + llm=None, + checkpoint_dir=checkpoint_dir, + ) + + snapshots = checkpoint_dir / "snapshots" + assert snapshots.is_dir(), "snapshots/ must exist" + files = sorted(p.name for p in snapshots.iterdir()) + assert files == ["iter-0.json"] + + def test_snapshot_contains_required_fields( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + with patch( + "tools.delegate_tool.delegate_task", + return_value=_mock_delegate_json(0.6), + ): + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="snap-002", + max_iterations=0, + llm=None, + checkpoint_dir=checkpoint_dir, + ) + + snap = json.loads( + (checkpoint_dir / "snapshots" / "iter-0.json").read_text() + ) + assert snap["iteration"] == 0 + assert isinstance(snap["messages"], list) + assert len(snap["messages"]) == 1 # baseline only + assert isinstance(snap["metrics"], dict) + assert snap["metrics"].get("accuracy") == pytest.approx(0.6) + assert isinstance(snap["files"], list) + # The supervisor wrote attempt.py + task_brief.md → at least 2 entries + captured = {entry["path"] for entry in snap["files"]} + assert any(p.endswith("attempt.py") for p in captured) + assert any(p.endswith("task_brief.md") for p in captured) + + def test_one_snapshot_per_iteration( + self, tmp_path: Path, parent_agent: MagicMock, code_spec: TaskSpec + ): + workspace = tmp_path / "ws" + checkpoint_dir = tmp_path / "job" + checkpoint_dir.mkdir() + + # LLM stub so the loop actually iterates + llm = MagicMock() + llm.chat.return_value = type("R", (), {"content": "x = 2"})() + + # Returns ascending metrics so the loop keeps "improving" + delegates = iter([ + _mock_delegate_json(0.1), + _mock_delegate_json(0.2), + _mock_delegate_json(0.3), + ]) + + with patch( + "tools.delegate_tool.delegate_task", + side_effect=lambda *a, **kw: next(delegates), + ): + supervisor = ResearchSupervisor( + parent_agent=parent_agent, workspace=workspace + ) + supervisor.run( + code_spec, + initial_attempt="x = 1", + run_id="snap-003", + max_iterations=2, + llm=llm, + checkpoint_dir=checkpoint_dir, + ) + + snapshots = sorted( + (checkpoint_dir / "snapshots").iterdir(), key=lambda p: p.name + ) + names = [p.name for p in snapshots] + assert names == ["iter-0.json", "iter-1.json", "iter-2.json"] + + +# --------------------------------------------------------------------------- +# Rollback +# --------------------------------------------------------------------------- + +class TestRestoreSnapshot: + def test_restore_recreates_files(self, tmp_path: Path): + snapshot = { + "iteration": 1, + "messages": [], + "metrics": {"accuracy": 0.7}, + "files": [ + {"path": "round-x-iter1/attempt.py", "content": "print('hi')\n"}, + {"path": "round-x-iter1/results.json", "content": '{"accuracy": 0.7}'}, + ], + } + snap_path = tmp_path / "iter-1.json" + snap_path.write_text(json.dumps(snapshot)) + + target = tmp_path / "restored" + restore_snapshot(snap_path, target) + + assert (target / "round-x-iter1" / "attempt.py").read_text() == "print('hi')\n" + assert (target / "round-x-iter1" / "results.json").read_text() == '{"accuracy": 0.7}' + + def test_restore_overwrites_existing(self, tmp_path: Path): + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [{"path": "f.txt", "content": "NEW"}], + })) + target = tmp_path / "out" + target.mkdir() + (target / "f.txt").write_text("OLD") + + restore_snapshot(snap_path, target) + + assert (target / "f.txt").read_text() == "NEW" + + def test_restore_rejects_path_traversal(self, tmp_path: Path): + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [{"path": "../../escape.txt", "content": "X"}], + })) + target = tmp_path / "out" + + with pytest.raises(ValueError): + restore_snapshot(snap_path, target) + + def test_restore_rejects_single_dotdot(self, tmp_path: Path): + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [{"path": "../escape.txt", "content": "X"}], + })) + target = tmp_path / "out" + + with pytest.raises(ValueError): + restore_snapshot(snap_path, target) + + def test_restore_rejects_symlink_in_parent_path(self, tmp_path: Path): + target = tmp_path / "out" + target.mkdir() + outside = tmp_path / "outside" + outside.mkdir() + # A subdirectory of target is replaced with a symlink to outside. + # Even though the resolved path of "round-x/payload" lands inside + # /tmp/.../outside (escaping target's tree on resolution), and is + # therefore caught by layer 1, the symlink check must reject it + # explicitly with a "traverses a symlink" message — defense in + # depth in case the symlink target is itself inside target_dir. + symlinked = target / "round-x" + symlinked.symlink_to(outside, target_is_directory=True) + + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [{"path": "round-x/payload.txt", "content": "X"}], + })) + + with pytest.raises(ValueError): + restore_snapshot(snap_path, target) + + def test_restore_rejects_symlink_target_inside(self, tmp_path: Path): + """Symlink that points back inside target_dir is still refused — + we don't traverse symlinks at all.""" + target = tmp_path / "out" + target.mkdir() + real_sub = target / "real" + real_sub.mkdir() + symlinked = target / "via-link" + symlinked.symlink_to(real_sub, target_is_directory=True) + + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [{"path": "via-link/file.txt", "content": "X"}], + })) + + with pytest.raises(ValueError): + restore_snapshot(snap_path, target) + + def test_restore_normal_relative_path_succeeds(self, tmp_path: Path): + snap_path = tmp_path / "iter-0.json" + snap_path.write_text(json.dumps({ + "iteration": 0, + "messages": [], + "metrics": {}, + "files": [ + {"path": "round-1/sub/dir/attempt.py", "content": "ok\n"}, + ], + })) + target = tmp_path / "out" + + restore_snapshot(snap_path, target) + + assert (target / "round-1/sub/dir/attempt.py").read_text() == "ok\n" diff --git a/tests/agent/research/test_supervisor_fan_out.py b/tests/agent/research/test_supervisor_fan_out.py new file mode 100644 index 000000000000..43a86570f694 --- /dev/null +++ b/tests/agent/research/test_supervisor_fan_out.py @@ -0,0 +1,531 @@ +"""Tests for HRM-108: Hypothesis Fan-Out parallelism in ResearchSupervisor.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.supervisor import ResearchSupervisor, TaskSpec, _call_delegate_task_batch +from agent.research.runner import ExperimentHistory, ExperimentResult, DelegateSandboxResult + + +@pytest.fixture +def mock_llm(): + """LLM mock that returns content from a configurable response.""" + m = MagicMock() + m.chat.return_value = MagicMock(content="") + return m + + +@pytest.fixture +def mock_parent_agent(): + """Minimal parent agent mock for delegate_task.""" + m = MagicMock() + m._delegate_depth = 0 + m.session_id = "test-session" + m._interrupt_requested = False + return m + + +@pytest.fixture +def tmp_workspace(tmp_path: Path) -> Path: + return tmp_path / "workspace" + + +@pytest.fixture +def code_spec() -> TaskSpec: + return TaskSpec( + topic="sort an array", + deliverable="python function sort_array(arr)", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + ) + + +class TestFanOutAttempts: + """_fan_out_attempts generates N diverse revision hypotheses.""" + + def test_fan_out_generates_n_attempts( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + mock_llm.chat.return_value = MagicMock( + content=( + "=== VARIANT 1 ===\n" + "[hypothesis: use quicksort]\n" + "def sort_array(arr): return sorted(arr)\n" + "=== VARIANT 2 ===\n" + "[hypothesis: use mergesort]\n" + "def sort_array(arr): return merge_sort(arr)\n" + "=== VARIANT 3 ===\n" + "[hypothesis: use heapsort]\n" + "def sort_array(arr): return heap_sort(arr)\n" + ) + ) + + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + history.add( + ExperimentResult( + run_id="r1", + iteration=0, + code="def sort_array(arr): pass", + metrics={"accuracy": 0.5}, + primary_metric=0.5, + improved=True, + kept=True, + elapsed_sec=1.0, + stdout="baseline", + stderr="", + ) + ) + + attempts = supervisor._fan_out_attempts( + mock_llm, code_spec, "def sort_array(arr): pass", history, n=3 + ) + + assert len(attempts) == 3 + assert "sorted(arr)" in attempts[0] + assert "merge_sort" in attempts[1] + assert "heap_sort" in attempts[2] + + def test_fan_out_pads_with_current_attempt_if_parsing_yields_fewer( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + mock_llm.chat.return_value = MagicMock( + content="=== VARIANT 1 ===\n[hypothesis: only one]\ndef f(): pass\n" + ) + + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + + attempts = supervisor._fan_out_attempts( + mock_llm, code_spec, "original", history, n=3 + ) + + assert len(attempts) == 3 + assert "f(): pass" in attempts[0] + assert attempts[1] == "original" + assert attempts[2] == "original" + + def test_fan_out_fallback_on_llm_error( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + mock_llm.chat.side_effect = RuntimeError("API failure") + + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + + attempts = supervisor._fan_out_attempts( + mock_llm, code_spec, "original", history, n=2 + ) + + assert len(attempts) == 1 + assert attempts[0] == "original" + + +class TestRunFanOutIteration: + """_run_fan_out_iteration executes N workers in parallel and returns sorted results.""" + + def test_fan_out_runs_n_workers(self, mock_parent_agent, tmp_workspace, code_spec): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + + # Seed a baseline so current_best is known + history.add( + ExperimentResult( + run_id="r1", + iteration=0, + code="baseline", + metrics={"accuracy": 0.5}, + primary_metric=0.5, + improved=True, + kept=True, + elapsed_sec=1.0, + stdout="ok", + stderr="", + ) + ) + + def fake_batch(tasks, **kwargs): + # Return one result per task + results = [] + for i, _ in enumerate(tasks): + results.append({ + "task_index": i, + "status": "completed", + "summary": f"METRIC: accuracy={0.6 + i * 0.1} STATUS: improved NOTES: ok", + "error": None, + "duration_seconds": 1.0, + "api_calls": 1, + }) + return {"results": results} + + with patch( + "agent.research.supervisor._call_delegate_task_batch", + side_effect=fake_batch, + ): + results = supervisor._run_fan_out_iteration( + spec=code_spec, + attempts=["attempt0", "attempt1", "attempt2"], + run_id="test-run", + iteration=1, + time_budget_sec=0, + worker_toolsets=None, + llm=None, + history=history, + ) + + assert len(results) == 3 + # All should be in history + assert len(history.results) == 4 # baseline + 3 fan-out + + def test_fan_out_selects_best_first(self, mock_parent_agent, tmp_workspace, code_spec): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + history.add( + ExperimentResult( + run_id="r1", + iteration=0, + code="baseline", + metrics={"accuracy": 0.5}, + primary_metric=0.5, + improved=True, + kept=True, + elapsed_sec=1.0, + stdout="ok", + stderr="", + ) + ) + + def fake_batch(tasks, **kwargs): + results = [] + for i, _ in enumerate(tasks): + # accuracy values: 0.55, 0.75, 0.65 + acc = 0.55 if i == 0 else (0.75 if i == 1 else 0.65) + results.append({ + "task_index": i, + "status": "completed", + "summary": f"METRIC: accuracy={acc} STATUS: improved NOTES: ok", + "error": None, + "duration_seconds": 1.0, + "api_calls": 1, + }) + return {"results": results} + + with patch( + "agent.research.supervisor._call_delegate_task_batch", + side_effect=fake_batch, + ): + results = supervisor._run_fan_out_iteration( + spec=code_spec, + attempts=["a0", "a1", "a2"], + run_id="test-run", + iteration=1, + time_budget_sec=0, + worker_toolsets=None, + llm=None, + history=history, + ) + + # Best first: 0.75, 0.65, 0.55 + assert results[0].primary_metric == pytest.approx(0.75) + assert results[1].primary_metric == pytest.approx(0.65) + assert results[2].primary_metric == pytest.approx(0.55) + + def test_fan_out_fallback_on_batch_error( + self, mock_parent_agent, tmp_workspace, code_spec + ): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = ExperimentHistory() + + with patch( + "agent.research.supervisor._call_delegate_task_batch", + side_effect=RuntimeError("batch failed"), + ): + results = supervisor._run_fan_out_iteration( + spec=code_spec, + attempts=["a0", "a1"], + run_id="test-run", + iteration=1, + time_budget_sec=0, + worker_toolsets=None, + llm=None, + history=history, + ) + + assert len(results) == 2 + assert all(r.error is not None for r in results) + assert all(r.primary_metric is None for r in results) + + +class TestFanOutIntegration: + """Integration tests for the fan-out loop via ResearchSupervisor.run().""" + + def test_run_with_fan_out_parameter(self, mock_llm, mock_parent_agent, tmp_workspace, code_spec): + """run() accepts fan_out parameter and executes parallel iterations.""" + mock_llm.chat.return_value = MagicMock( + content=( + "=== VARIANT 1 ===\n" + "[hypothesis: A]\n" + "code A\n" + "=== VARIANT 2 ===\n" + "[hypothesis: B]\n" + "code B\n" + ) + ) + + call_count = {"batch": 0} + + def fake_batch(tasks, **kwargs): + call_count["batch"] += 1 + results = [] + for i, _ in enumerate(tasks): + acc = 0.7 if i == 0 else 0.8 + results.append({ + "task_index": i, + "status": "completed", + "summary": f"METRIC: accuracy={acc} STATUS: improved NOTES: ok", + "error": None, + "duration_seconds": 1.0, + }) + return {"results": results} + + with patch( + "agent.research.supervisor._call_delegate_task_batch", + side_effect=fake_batch, + ), patch.object(ResearchSupervisor, "_observe"), patch.object( + ResearchSupervisor, "_checkpoint" + ), patch.object(ResearchSupervisor, "_snapshot"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + # Patch delegate_fn for baseline ( ExperimentRunner uses it ) + with patch( + "agent.research.supervisor._call_delegate_task", + return_value={ + "results": [{ + "status": "completed", + "summary": "METRIC: accuracy=0.5 STATUS: neutral NOTES: baseline", + }] + }, + ): + history = supervisor.run( + code_spec, + initial_attempt="def sort_array(arr): pass", + run_id="fan-out-test", + max_iterations=1, + llm=mock_llm, + fan_out=2, + ) + + assert call_count["batch"] == 1 + assert len(history.results) == 3 # baseline + 2 fan-out branches + assert history.best_result is not None + assert history.best_result.primary_metric == pytest.approx(0.8) + + def test_run_fan_out_1_is_sequential(self, mock_llm, mock_parent_agent, tmp_workspace, code_spec): + """fan_out=1 uses the original sequential path (no batch calls).""" + mock_llm.chat.return_value = MagicMock(content="improved code") + + with patch( + "agent.research.supervisor._call_delegate_task_batch" + ) as mock_batch, patch.object(ResearchSupervisor, "_observe"), patch.object( + ResearchSupervisor, "_checkpoint" + ), patch.object(ResearchSupervisor, "_snapshot"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + with patch( + "agent.research.supervisor._call_delegate_task", + return_value={ + "results": [{ + "status": "completed", + "summary": "METRIC: accuracy=0.5 STATUS: neutral NOTES: baseline", + }] + }, + ): + supervisor.run( + code_spec, + initial_attempt="def sort_array(arr): pass", + run_id="seq-test", + max_iterations=1, + llm=mock_llm, + fan_out=1, + ) + + mock_batch.assert_not_called() + + +class TestAggregateAttempts: + """_aggregate_attempts synthesizes N fan-out results into a super-attempt.""" + + def test_aggregate_combines_branches(self, mock_llm, mock_parent_agent, tmp_workspace, code_spec): + mock_llm.chat.return_value = MagicMock( + content="# Super-attempt combining quicksort + mergesort insights\n" + "def sort_array(arr): return sorted(arr, key=len)\n" + ) + + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + results = [ + ExperimentResult( + run_id="r1", iteration=1, code="code A", + metrics={"accuracy": 0.8}, primary_metric=0.8, + improved=True, kept=True, elapsed_sec=1.0, + stdout="worker A output", stderr="", + ), + ExperimentResult( + run_id="r1", iteration=1, code="code B", + metrics={"accuracy": 0.7}, primary_metric=0.7, + improved=False, kept=False, elapsed_sec=1.0, + stdout="worker B output", stderr="", + ), + ] + + aggregated = supervisor._aggregate_attempts( + mock_llm, code_spec, results, "original_best" + ) + + assert "Super-attempt" in aggregated or "sorted" in aggregated + # LLM should have been called with branch summaries + prompt = mock_llm.chat.call_args[0][0][0]["content"] + assert "BRANCH 1" in prompt + assert "BRANCH 2" in prompt + assert "worker A output" in prompt + assert "worker B output" in prompt + + def test_aggregate_fallback_on_llm_error( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + mock_llm.chat.side_effect = RuntimeError("API failure") + + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + results = [ + ExperimentResult( + run_id="r1", iteration=1, code="best_code", + metrics={"accuracy": 0.8}, primary_metric=0.8, + improved=True, kept=True, elapsed_sec=1.0, + stdout="ok", stderr="", + ), + ] + + aggregated = supervisor._aggregate_attempts( + mock_llm, code_spec, results, "original_best" + ) + + # Fallback to best branch code + assert aggregated == "best_code" + + def test_aggregate_fallback_on_empty_results( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + + aggregated = supervisor._aggregate_attempts( + mock_llm, code_spec, [], "original_best" + ) + + assert aggregated == "original_best" + + +class TestMoaIntegration: + """Integration tests for MOA aggregation in the fan-out loop.""" + + def test_run_with_fan_out_uses_aggregated_attempt( + self, mock_llm, mock_parent_agent, tmp_workspace, code_spec + ): + """When fan_out > 1, the next iteration starts from the aggregated attempt.""" + # First call: fan_out generation, Second call: MOA aggregation + mock_llm.chat.side_effect = [ + MagicMock( + content=( + "=== VARIANT 1 ===\n[hypothesis: A]\ncode A\n" + "=== VARIANT 2 ===\n[hypothesis: B]\ncode B\n" + ) + ), + MagicMock(content="aggregated super code"), + ] + + def fake_batch(tasks, **kwargs): + results = [] + for i, _ in enumerate(tasks): + acc = 0.7 if i == 0 else 0.8 + results.append({ + "task_index": i, + "status": "completed", + "summary": f"METRIC: accuracy={acc} STATUS: improved NOTES: ok", + "error": None, + "duration_seconds": 1.0, + }) + return {"results": results} + + with patch( + "agent.research.supervisor._call_delegate_task_batch", + side_effect=fake_batch, + ), patch.object(ResearchSupervisor, "_observe"), patch.object( + ResearchSupervisor, "_checkpoint" + ), patch.object(ResearchSupervisor, "_snapshot"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + with patch( + "agent.research.supervisor._call_delegate_task", + return_value={ + "results": [{ + "status": "completed", + "summary": "METRIC: accuracy=0.5 STATUS: neutral NOTES: baseline", + }] + }, + ): + history = supervisor.run( + code_spec, + initial_attempt="def sort_array(arr): pass", + run_id="moa-test", + max_iterations=1, + llm=mock_llm, + fan_out=2, + ) + + # The aggregated attempt should be written to the workspace for the next + # iteration (even though there is no next iteration in this test). + # We verify the LLM was called twice: once for fan-out, once for MOA. + assert mock_llm.chat.call_count == 2 + # Second call should be the aggregation prompt + second_prompt = mock_llm.chat.call_args_list[1][0][0][0]["content"] + assert "Synthesis Instructions (MOA)" in second_prompt + assert "BRANCH 1" in second_prompt + assert "BRANCH 2" in second_prompt diff --git a/tests/agent/test_factory.py b/tests/agent/test_factory.py new file mode 100644 index 000000000000..8e283344fa57 --- /dev/null +++ b/tests/agent/test_factory.py @@ -0,0 +1,84 @@ +"""Tests for agent.factory — centralized AIAgent construction for +detached entrypoints (HRM-57). + +Run with: + pytest tests/agent/test_factory.py -q --override-ini="addopts=" +""" +from __future__ import annotations + +import os +from unittest.mock import patch, MagicMock + +from agent.factory import build_agent_for_research_job + + +# A minimal spec the factory should accept. +_SPEC = { + "job_id": "test-001", + "model": "kimi-k2.6", + "provider": "kimi-coding", + "base_url": "https://api.kimi.com/coding/v1", + "api_key": "", + "toolsets": ["research", "terminal", "file"], +} + + +class TestBuildAgentForResearchJob: + def test_default_inherits_profile_context(self): + """Without skip_* in spec, defaults flip to load profile context — + consistent with HRM-58: research workers want the curated profile.""" + captured: dict = {} + with patch("run_agent.AIAgent") as MockAgent: + MockAgent.side_effect = lambda *a, **kw: captured.update(kw) or MagicMock() + build_agent_for_research_job(_SPEC) + assert captured["skip_context_files"] is False + assert captured["skip_memory"] is False + assert captured["model"] == "kimi-k2.6" + assert captured["session_id"] == "research-job:test-001" + + def test_spec_can_opt_out_of_profile_context(self): + """spec.skip_context_files=True still wins for callers that want + a blank-slate detached run (e.g. provider benchmarking).""" + captured: dict = {} + spec = {**_SPEC, "skip_context_files": True, "skip_memory": True} + with patch("run_agent.AIAgent") as MockAgent: + MockAgent.side_effect = lambda *a, **kw: captured.update(kw) or MagicMock() + build_agent_for_research_job(spec) + assert captured["skip_context_files"] is True + assert captured["skip_memory"] is True + + def test_runtime_invariants_patched_post_init(self): + """The HRM-57-full kwargs (delegate_depth / terminal_cwd / cwd / + subdirectory_hints) never landed in upstream AIAgent.__init__, so + the factory patches them post-construction. This test pins that + contract so it doesn't silently regress to the pre-fix behavior + where AIAgent would crash on unexpected kwargs.""" + with patch("run_agent.AIAgent") as MockAgent: + # Real AIAgent doesn't accept these kwargs — simulate that. + mock_agent = MagicMock( + spec=[ + "tool_progress_callback", + ], + ) + mock_agent.tool_progress_callback = None + MockAgent.return_value = mock_agent + agent = build_agent_for_research_job(_SPEC) + + # Patched-on attributes the delegate_task code path reads. + assert agent._delegate_depth == 0 + assert agent.terminal_cwd # non-empty string + assert agent.cwd + assert agent._subdirectory_hints is None + + def test_progress_callback_is_no_op_when_none(self): + """The factory still sets a no-op tool_progress_callback when AIAgent + leaves it as None — callers can dispatch without nil-checking.""" + with patch("run_agent.AIAgent") as MockAgent: + mock_agent = MagicMock() + mock_agent.tool_progress_callback = None + MockAgent.return_value = mock_agent + agent = build_agent_for_research_job(_SPEC) + + assert agent.tool_progress_callback is not None + # Calling it should not raise + agent.tool_progress_callback("event", "name", "preview") diff --git a/tests/agent/test_research_supervisor.py b/tests/agent/test_research_supervisor.py new file mode 100644 index 000000000000..a2d534629591 --- /dev/null +++ b/tests/agent/test_research_supervisor.py @@ -0,0 +1,690 @@ +"""Integration tests for ResearchSupervisor — Karpathy inner loop. + +Run with: + pytest tests/agent/test_research_supervisor.py -m integration --override-ini="addopts=" + +The 'integration' mark is required because these tests write to a real tmpdir, +run the full ExperimentRunner loop, and verify end-to-end metric parsing. +Tests tagged 'unit' run without the mark. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any +from unittest.mock import MagicMock, patch + +import pytest + +from agent.research.runner import ( + DelegateSandboxResult, + ExperimentHistory, + ExperimentRunner, + HermesExperimentConfig, +) +from agent.research.metrics import UniversalMetricParser +from agent.research.supervisor import ( + ResearchSupervisor, + TaskSpec, + _build_task_brief, + _extract_iteration, +) + + +# --------------------------------------------------------------------------- +# Unit tests (no integration mark needed) +# --------------------------------------------------------------------------- + +class TestBuildTaskBrief: + def _code_spec(self, **kwargs) -> TaskSpec: + defaults = dict( + topic="optimizer comparison", + deliverable="Python comparison of Adam vs SGD on MNIST", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + hypothesis="Adam converges faster than SGD", + ) + defaults.update(kwargs) + return TaskSpec(**defaults) + + def test_contains_topic_and_metric(self): + spec = self._code_spec() + md = _build_task_brief( + spec, + iteration=1, + round_dir="/tmp/round-001-iter1", + time_budget_sec=120, + ) + assert "optimizer comparison" in md + assert "accuracy" in md + assert "higher" in md # metric_direction="maximize" renders as "higher" + assert "120" in md + assert "METRIC: accuracy=" in md + + def test_contains_time_guard_instructions(self): + spec = TaskSpec( + topic="t", deliverable="d", metric_key="loss", + metric_direction="minimize", task_type="code", + ) + md = _build_task_brief(spec, iteration=0, round_dir="/tmp/rd", time_budget_sec=60) + assert "TIME_ESTIMATE" in md + assert "80%" in md + + def test_iteration_zero_is_baseline(self): + spec = self._code_spec() + md = _build_task_brief(spec, iteration=0, round_dir="/tmp/rd", time_budget_sec=300) + assert "Establish a baseline" in md + + def test_iteration_positive_is_improve(self): + spec = self._code_spec() + md = _build_task_brief(spec, iteration=2, round_dir="/tmp/rd", time_budget_sec=300) + assert "Improve" in md + + def test_search_task_brief(self): + spec = TaskSpec( + topic="Find attention mechanism papers", + deliverable="Ranked list of papers", + metric_key="relevance_score", + task_type="search", + ) + md = _build_task_brief(spec, iteration=0, round_dir="/tmp/rd", time_budget_sec=120) + assert "Search" in md + assert "relevance_score" in md + assert "attempt.md" in md + + def test_research_task_brief(self): + spec = TaskSpec( + topic="State of diffusion models", + deliverable="Technical synthesis", + metric_key="completeness_score", + task_type="research", + ) + md = _build_task_brief(spec, iteration=0, round_dir="/tmp/rd", time_budget_sec=300) + assert "Research" in md + assert "completeness_score" in md + + def test_generic_task_brief(self): + spec = TaskSpec( + topic="Optimize search latency", + deliverable="Modified implementation", + metric_key="latency_ms", + metric_direction="minimize", + task_type="generic", + ) + md = _build_task_brief(spec, iteration=0, round_dir="/tmp/rd", time_budget_sec=300) + assert "latency_ms" in md + + +class TestExtractIteration: + def test_round_dir_with_iter(self): + assert _extract_iteration("/tmp/round-abc-iter3") == 3 + assert _extract_iteration("/tmp/round-xyz-iter0") == 0 + assert _extract_iteration("/tmp/round-foo-iter12") == 12 + + def test_malformed_returns_zero(self): + assert _extract_iteration("/tmp/no-iter-here") == 0 + assert _extract_iteration("") == 0 + + +# --------------------------------------------------------------------------- +# Integration tests — full loop with mocked delegate_task +# --------------------------------------------------------------------------- + +pytestmark_integration = pytest.mark.integration + + +def _make_delegate_result(metric_value: float, metric_key: str = "accuracy") -> str: + """Build a fake delegate_task JSON result with a metric in stdout.""" + summary = ( + f"Experiment complete.\n" + f"METRIC: {metric_key}={metric_value} STATUS: improved NOTES: mock result\n" + f"All done." + ) + return json.dumps({ + "results": [ + { + "task_index": 0, + "status": "completed", + "summary": summary, + "api_calls": 5, + "duration_seconds": 1.2, + "exit_reason": "completed", + "tokens": {"input": 100, "output": 50}, + "tool_trace": [], + } + ], + "total_duration_seconds": 1.2, + }) + + +def _make_failed_delegate_result(error: str = "Worker timed out") -> str: + return json.dumps({ + "results": [ + { + "task_index": 0, + "status": "failed", + "summary": "", + "error": error, + "api_calls": 1, + "duration_seconds": 5.0, + "exit_reason": "max_iterations", + "tokens": {"input": 20, "output": 0}, + "tool_trace": [], + } + ], + "total_duration_seconds": 5.0, + }) + + +@pytest.fixture() +def tmp_workspace(tmp_path: Path) -> Path: + return tmp_path / "research-workspace" + + +@pytest.fixture() +def mock_parent_agent() -> MagicMock: + agent = MagicMock() + agent.model = "claude-sonnet-4-6" + agent.base_url = "https://api.anthropic.com" + agent.api_key = "test-key" + agent.provider = "anthropic" + agent.api_mode = "anthropic_messages" + agent.providers_allowed = None + agent.providers_ignored = None + agent.providers_order = None + agent.provider_sort = None + agent.enabled_toolsets = ["terminal", "file"] + agent._delegate_depth = 0 + agent._active_children = [] + agent._active_children_lock = None + return agent + + +@pytest.fixture() +def code_spec() -> TaskSpec: + return TaskSpec( + topic="Optimizer comparison on MNIST", + deliverable="Python script comparing Adam vs SGD with accuracy metric", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + hypothesis="Adam converges faster than SGD", + ) + + +@pytest.mark.integration +class TestResearchSupervisorBaseline: + """Full loop with a mocked delegate_task — no real subagent spawned.""" + + def test_baseline_only_no_llm(self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec): + """Supervisor runs baseline experiment, returns history with 1 result.""" + metric_value = 0.85 + + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(metric_value)): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + code_spec, + initial_attempt="print('accuracy: 0.85')", + run_id="test-baseline-001", + max_iterations=3, + time_budget_sec=60, + llm=None, # baseline only + ) + + assert len(history.results) == 1 + assert history.baseline_metric == pytest.approx(metric_value, abs=0.001) + assert history.results[0].iteration == 0 + assert history.results[0].primary_metric == pytest.approx(metric_value, abs=0.001) + assert history.results[0].kept is True # first result always kept + + def test_task_brief_written_to_round_dir(self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec): + """Supervisor must write task_brief.md and attempt.py before calling delegate_task.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.75)): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + code_spec, + initial_attempt="# baseline code\nprint('accuracy: 0.75')", + run_id="test-files-001", + llm=None, + ) + + run_dir = tmp_workspace / "test-files-001" + assert run_dir.exists(), "run dir must be created" + round_dirs = [p for p in run_dir.iterdir() if p.is_dir()] + assert len(round_dirs) >= 1 + round_dir = round_dirs[0] + assert (round_dir / "attempt.py").exists(), "attempt.py must be written for code tasks" + assert (round_dir / "task_brief.md").exists(), "task_brief.md must be written by supervisor" + brief = (round_dir / "task_brief.md").read_text() + assert "Optimizer comparison on MNIST" in brief + assert "accuracy" in brief + + def test_failed_worker_records_error(self, tmp_workspace: Path, mock_parent_agent: MagicMock): + """When delegate_task returns failed status, result has error and is not kept.""" + spec = TaskSpec( + topic="Crash test", + deliverable="code that fails", + metric_key="accuracy", + task_type="code", + ) + with patch("tools.delegate_tool.delegate_task", return_value=_make_failed_delegate_result("Worker crashed")): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + spec, + initial_attempt="raise RuntimeError('oops')", + run_id="test-fail-001", + llm=None, + ) + + assert len(history.results) == 1 + result = history.results[0] + assert result.error is not None + assert result.kept is False + assert result.primary_metric is None + + def test_progress_sink_default_is_log_only(self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec): + """When no progress_sink is passed, the default StubSink is used and + the loop runs to completion without raising.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.9)): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + code_spec, + initial_attempt="pass", + run_id="test-comment-001", + llm=None, + ) + + assert len(history.results) == 1 + + def test_learnings_jsonl_written(self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec): + """Autogenesis Observe step: learnings.jsonl is written with HeartbeatMemorySystem schema.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.88)): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + code_spec, + initial_attempt="print('accuracy: 0.88')", + run_id="test-learnings-001", + llm=None, + ) + + learnings_file = tmp_workspace / "test-learnings-001" / "learnings.jsonl" + assert learnings_file.exists(), "learnings.jsonl must be written by _observe()" + lines = [json.loads(l) for l in learnings_file.read_text().splitlines() if l.strip()] + assert len(lines) == 1 # one entry per iteration + entry = lines[0] + # Verify HeartbeatMemorySystem schema + assert "type" in entry + assert "key" in entry + assert "insight" in entry + assert "confidence" in entry + assert "source" in entry + assert entry["key"] == "accuracy" + assert entry["type"] in ("improvement", "regression", "failure") + assert entry["source"] == "iter-0" + + def test_search_task_writes_attempt_md(self, tmp_workspace: Path, mock_parent_agent: MagicMock): + """Search tasks write attempt.md, not attempt.py.""" + spec = TaskSpec( + topic="Find papers on transformers", + deliverable="Ranked list of papers", + metric_key="relevance_score", + task_type="search", + ) + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.8, "relevance_score")): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + spec, + initial_attempt="search query: transformer papers after 2022", + run_id="test-search-001", + llm=None, + ) + + run_dir = tmp_workspace / "test-search-001" + round_dirs = [p for p in run_dir.iterdir() if p.is_dir()] + assert len(round_dirs) >= 1 + round_dir = round_dirs[0] + assert (round_dir / "attempt.md").exists(), "attempt.md must be written for search tasks" + assert not (round_dir / "attempt.py").exists(), "attempt.py must NOT be written for search tasks" + + + def test_rollback_uses_on_disk_artifact(self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec): + """Fix #1 (audit): rollback restores on-disk artifact, not the seed string.""" + call_count = 0 + BEST_ARTIFACT = "# best on-disk version\nprint('accuracy: 0.90')" + + def capturing_delegate(goal, context, toolsets, parent_agent, inherit_profile=False): + nonlocal call_count + call_count += 1 + # Find the round dir from goal string and write a modified attempt.py + import re as _re + m = _re.search(r"round-[^\s]+", goal) + if m: + rd = tmp_workspace / "test-rollback-001" / m.group(0) + rd.mkdir(parents=True, exist_ok=True) + if call_count == 1: + # baseline — write a specific artifact to disk + (rd / "attempt.py").write_text(BEST_ARTIFACT, encoding="utf-8") + return _make_delegate_result(0.90) + else: + # iter1 — write a worse artifact but report regression + (rd / "attempt.py").write_text("# worse attempt", encoding="utf-8") + return _make_delegate_result(0.70) + return _make_delegate_result(0.0) + + mock_llm = MagicMock() + mock_llm.chat.return_value = MagicMock(content="```python\nprint('iter attempt')\n```") + + with patch("tools.delegate_tool.delegate_task", side_effect=capturing_delegate): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + code_spec, + initial_attempt="# initial seed", + run_id="test-rollback-001", + max_iterations=1, + llm=mock_llm, + ) + + # After regression, rollback should restore the best on-disk artifact + assert history.best_result is not None + assert history.best_result.primary_metric == pytest.approx(0.90, abs=0.001) + + +@pytest.mark.integration +class TestResearchSupervisorIterations: + """Multi-iteration loop with a mock LLM client.""" + + def _make_mock_llm(self) -> MagicMock: + class MockResponse: + content = "```python\nprint('updated code')\n```" + + llm = MagicMock() + llm.chat.return_value = MockResponse() + return llm + + def test_two_iteration_improvement(self, tmp_workspace: Path, mock_parent_agent: MagicMock): + """Loop improves once then plateaus — verifies history and best_result.""" + metric_sequence = iter([0.70, 0.82, 0.81, 0.80]) # baseline, iter1 improves, iter2/3 regress + + def side_effect(goal, context, toolsets, parent_agent, inherit_profile=False): + val = next(metric_sequence, 0.80) + return _make_delegate_result(val) + + spec = TaskSpec( + topic="Improvement test", + deliverable="Adam should converge better", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + ) + mock_llm = self._make_mock_llm() + + with patch("tools.delegate_tool.delegate_task", side_effect=side_effect): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + spec, + initial_attempt="# initial", + run_id="test-iter-001", + max_iterations=5, + llm=mock_llm, + ) + + # Should have stopped after 3 non-improving iterations past the best + assert len(history.results) >= 2 + best = history.best_result + assert best is not None + assert best.primary_metric == pytest.approx(0.82, abs=0.001) + + def test_early_stop_on_no_improvement(self, tmp_workspace: Path, mock_parent_agent: MagicMock): + """Loop stops early after 3 consecutive non-improving iterations.""" + call_count = 0 + + def side_effect(goal, context, toolsets, parent_agent, inherit_profile=False): + nonlocal call_count + call_count += 1 + if call_count == 1: + return _make_delegate_result(0.5) # baseline + return _make_delegate_result(0.4) # always regress + + spec = TaskSpec( + topic="Early stop test", + deliverable="This will not improve", + metric_key="accuracy", + metric_direction="maximize", + task_type="code", + ) + mock_llm = MagicMock() + mock_llm.chat.return_value = MagicMock(content="```python\npass\n```") + + with patch("tools.delegate_tool.delegate_task", side_effect=side_effect): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + spec, + initial_attempt="# bad code", + run_id="test-early-001", + max_iterations=10, + llm=mock_llm, + ) + + # baseline + 3 failing iterations = 4 total + assert len(history.results) == 4 + assert call_count == 4 + + +# --------------------------------------------------------------------------- +# HRM-59: EvolutionStore wiring v1 — persist lessons after run() +# --------------------------------------------------------------------------- + +class TestEvolutionPersistence: + def test_evolve_writes_one_lesson_per_iteration( + self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec, tmp_path: Path + ): + """After run() returns, the EvolutionStore JSONL must have one entry + per ExperimentResult — covering improved/discarded/error severities.""" + from agent.research.evolution import EvolutionStore + + evolution_dir = tmp_path / "evolution-home" / "evolution" + + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.85)), \ + patch("agent.research.supervisor.get_hermes_home", return_value=tmp_path / "evolution-home"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + code_spec, + initial_attempt="print('accuracy: 0.85')", + run_id="evo-test-001", + llm=None, # baseline only -> 1 result + ) + + lessons = EvolutionStore(evolution_dir).load_all() + assert len(lessons) == 1, "baseline-only run produces one lesson" + assert lessons[0].run_id == "evo-test-001" + assert lessons[0].stage_name == "iter_0" + assert lessons[0].severity in {"info", "warning"} + + def test_evolve_failure_does_not_break_run( + self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec, tmp_path: Path + ): + """If _evolve raises, run() must still return the history cleanly.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.9)), \ + patch.object(ResearchSupervisor, "_evolve", side_effect=RuntimeError("disk full")): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + history = supervisor.run( + code_spec, + initial_attempt="print('ok')", + run_id="evo-fail-001", + llm=None, + ) + + assert history is not None + assert len(history.results) == 1 + + +# --------------------------------------------------------------------------- +# HRM-62: EvolutionStore overlay wiring v2 — lessons surface in worker briefs +# --------------------------------------------------------------------------- + +class TestEvolutionOverlay: + def test_seeded_lesson_appears_in_worker_brief( + self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec, tmp_path: Path + ): + """Seed a lesson into the EvolutionStore, run a baseline-only experiment, + and assert the worker's task_brief.md contains the seeded text.""" + from agent.research.evolution import EvolutionStore, LessonEntry, LessonCategory + from datetime import datetime, timezone + + evolution_dir = tmp_path / "evolution-home" / "evolution" + evolution_dir.mkdir(parents=True) + EvolutionStore(evolution_dir).append(LessonEntry( + stage_name="research_loop", + stage_num=0, + category=LessonCategory.PIPELINE, + severity="error", + description="ALPHA-MARKER-XYZ: workers must validate metric before reporting", + timestamp=datetime.now(timezone.utc).isoformat(), + run_id="seed-001", + )) + + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.85)), \ + patch("agent.research.supervisor.get_hermes_home", return_value=tmp_path / "evolution-home"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + code_spec, + initial_attempt="print('accuracy: 0.85')", + run_id="overlay-test-001", + llm=None, + ) + + run_dir = tmp_workspace / "overlay-test-001" + round_dirs = [p for p in run_dir.iterdir() if p.is_dir()] + brief_text = (round_dirs[0] / "task_brief.md").read_text() + assert "ALPHA-MARKER-XYZ" in brief_text, "seeded lesson must surface in worker brief" + assert "Lessons from Prior Runs" in brief_text, "overlay header expected" + + def test_no_overlay_when_store_empty( + self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec, tmp_path: Path + ): + """With no past lessons, the brief is unchanged — no empty overlay header.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.8)), \ + patch("agent.research.supervisor.get_hermes_home", return_value=tmp_path / "fresh-home"): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + supervisor.run( + code_spec, + initial_attempt="print('ok')", + run_id="empty-overlay-001", + llm=None, + ) + + run_dir = tmp_workspace / "empty-overlay-001" + round_dirs = [p for p in run_dir.iterdir() if p.is_dir()] + brief_text = (round_dirs[0] / "task_brief.md").read_text() + assert "Lessons from Prior Runs" not in brief_text + # And the brief still starts with the normal builder output + assert brief_text.lstrip().startswith("# Task Brief") + + def test_overlay_load_failure_does_not_break_run( + self, tmp_workspace: Path, mock_parent_agent: MagicMock, code_spec: TaskSpec + ): + """If _load_evolution_overlay raises, run() must still complete cleanly.""" + with patch("tools.delegate_tool.delegate_task", return_value=_make_delegate_result(0.9)), \ + patch.object(ResearchSupervisor, "_load_evolution_overlay", side_effect=RuntimeError("disk read failed")): + supervisor = ResearchSupervisor( + parent_agent=mock_parent_agent, + workspace=tmp_workspace, + ) + # run() invokes the helper directly; without protection it would propagate. + # The helper itself catches; the test asserts the user-facing contract: no propagation. + try: + history = supervisor.run( + code_spec, + initial_attempt="x", + run_id="overlay-fail-001", + llm=None, + ) + except RuntimeError: + # If propagation happens, mark as failure explicitly. + pytest.fail("_load_evolution_overlay failure must be swallowed") + assert history is not None + + +# --------------------------------------------------------------------------- +# ProgressSink wiring (Task 3 in the prune plan) +# --------------------------------------------------------------------------- + +class TestSupervisorCallsProgressSink: + @pytest.mark.integration + def test_sink_receives_run_started_iteration_run_completed(self, tmp_path): + """Even on the baseline-only path (llm is None), the sink must see + run_started, exactly one iteration_observed (iter=0), and run_completed. + Audit fix #4: the early-return path was missing run_completed.""" + from agent.research.sinks import StubSink + + sink = StubSink() + sink.run_started = MagicMock(side_effect=sink.run_started) + sink.iteration_observed = MagicMock(side_effect=sink.iteration_observed) + sink.run_completed = MagicMock(side_effect=sink.run_completed) + + spec = TaskSpec( + topic="t", deliverable="d", + metric_key="m", metric_direction="maximize", + ) + + def fake_delegate(*a, **kw): + return {"results": [{"status": "completed", "summary": "METRIC: m=0.5"}]} + + with patch("agent.research.supervisor._call_delegate_task", side_effect=fake_delegate): + sup = ResearchSupervisor( + parent_agent=MagicMock(), + workspace=tmp_path, + progress_sink=sink, + ) + sup.run( + spec, initial_attempt="x", run_id="rid", + max_iterations=0, # baseline only + llm=None, + disable_evolution_overlay=True, + ) + + sink.run_started.assert_called_once() + assert sink.iteration_observed.call_count == 1 + sink.run_completed.assert_called_once() diff --git a/tests/gateway/test_daemoncraft_cycle_detector.py b/tests/gateway/test_daemoncraft_cycle_detector.py new file mode 100644 index 000000000000..102de43745d7 --- /dev/null +++ b/tests/gateway/test_daemoncraft_cycle_detector.py @@ -0,0 +1,171 @@ +"""Unit tests for CycleDetector ported into gateway/platforms/daemoncraft.py.""" +from __future__ import annotations + +import os +import sys +import types +from typing import Any +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +# gateway/platforms/__init__.py eagerly imports yuanbao (httpx) and daemoncraft +# itself needs aiohttp. Stub missing optional deps before import. +def _stub_module(name: str, **attrs): + if name not in sys.modules: + mod = types.ModuleType(name) + for k, v in attrs.items(): + setattr(mod, k, v) + sys.modules[name] = mod + +_stub_module("httpx") +_stub_module("aiohttp", WSMsgType=MagicMock(), ClientSession=MagicMock) + +# Import the standalone class directly — no server needed +from gateway.platforms.daemoncraft import CycleDetector + + +# --------------------------------------------------------------------------- +# CycleDetector unit tests +# --------------------------------------------------------------------------- + +class TestCycleDetectorUnit: + def test_no_cycle_below_threshold(self): + cd = CycleDetector(n=3, window=10, action="warn") + for _ in range(2): + r = cd.record("tool_a", {}) + assert r.triggered is False + + def test_cycle_on_nth_identical_call(self): + cd = CycleDetector(n=3, window=10, action="warn") + r = None + for _ in range(3): + r = cd.record("tool_a", {}) + assert r.triggered is True + assert r.count >= 3 + + def test_no_double_trigger_on_n_plus_one(self): + cd = CycleDetector(n=3, window=10, action="warn") + for _ in range(3): + cd.record("tool_a", {}) + # 4th call: same sig, already triggered — should suppress + r = cd.record("tool_a", {}) + assert r.triggered is False + + def test_different_tool_names_no_cycle(self): + cd = CycleDetector(n=3, window=10, action="warn") + results = [] + for i in range(6): + results.append(cd.record(f"tool_{i}", {})) + assert not any(r.triggered for r in results) + + def test_different_args_no_cycle(self): + cd = CycleDetector(n=3, window=10, action="warn") + results = [] + for i in range(6): + results.append(cd.record("tool_a", {"x": i})) + assert not any(r.triggered for r in results) + + def test_cycle_clears_after_different_sig_dominates(self): + """After suppression, a NEW dominant sig should trigger fresh.""" + # Use small window=3 so tool_b can fully dominate and evict tool_a + cd = CycleDetector(n=3, window=3, action="warn") + # Trigger first cycle for tool_a + for _ in range(3): + cd.record("tool_a", {}) + # Flood with tool_b — fills the window, clears _last_triggered_sig + for _ in range(3): + cd.record("tool_b", {}) + # Now tool_a again — should trigger fresh (suppression was cleared) + r = None + for _ in range(3): + r = cd.record("tool_a", {}) + assert r.triggered is True + + +# --------------------------------------------------------------------------- +# DaemonCraftAdapter._check_cycle integration tests +# --------------------------------------------------------------------------- + +def _make_adapter(): + """Build a minimal DaemonCraftAdapter with all external deps mocked.""" + from gateway.platforms.daemoncraft import DaemonCraftAdapter + from gateway.config import PlatformConfig + + cfg = PlatformConfig( + enabled=True, + extra={ + "bot_api_url": "http://localhost:9999", + "bot_username": "TestBot", + "profile": "test", + }, + ) + adapter = DaemonCraftAdapter(cfg) + # Patch _interrupt_agent so tests don't need a real HTTP session + adapter._interrupt_agent = AsyncMock() + return adapter + + +class TestCheckCycleMethod: + @pytest.mark.anyio + async def test_returns_false_when_no_detector(self): + adapter = _make_adapter() + assert adapter._cycle_detector is None + result = await adapter._check_cycle("mc_perceive", {}) + assert result is False + + @pytest.mark.anyio + async def test_returns_false_for_non_cycling_calls(self): + adapter = _make_adapter() + adapter._cycle_detector = CycleDetector(n=3, window=10, action="warn") + result = await adapter._check_cycle("mc_perceive", {}) + assert result is False + + @pytest.mark.anyio + async def test_warn_action_returns_false_on_cycle(self): + """Cycle detected with action='warn' should log but NOT interrupt.""" + adapter = _make_adapter() + adapter._cycle_detector = CycleDetector(n=3, window=10, action="warn") + for _ in range(2): + await adapter._check_cycle("mc_perceive", {}) + result = await adapter._check_cycle("mc_perceive", {}) + # warn = no interrupt + assert result is False + adapter._interrupt_agent.assert_not_called() + + @pytest.mark.anyio + async def test_interrupt_action_returns_true_and_calls_interrupt(self): + """Cycle with action='interrupt' should call _interrupt_agent and return True.""" + adapter = _make_adapter() + adapter._cycle_detector = CycleDetector(n=3, window=10, action="interrupt") + for _ in range(2): + await adapter._check_cycle("mc_perceive", {}) + result = await adapter._check_cycle("mc_perceive", {}) + assert result is True + adapter._interrupt_agent.assert_called_once_with("cycle_detected") + + +class TestAdapterCycleDetectorInit: + @pytest.mark.anyio + async def test_no_detector_when_mc_cycle_n_zero(self, monkeypatch): + monkeypatch.delenv("MC_CYCLE_N", raising=False) + adapter = _make_adapter() + # Patch connect internals so no actual socket is opened + with patch("gateway.platforms.daemoncraft.aiohttp.ClientSession", return_value=MagicMock()), \ + patch("asyncio.create_task", return_value=MagicMock()): + await adapter.connect() + assert adapter._cycle_detector is None + + @pytest.mark.anyio + async def test_detector_created_when_mc_cycle_n_set(self, monkeypatch): + monkeypatch.setenv("MC_CYCLE_N", "3") + monkeypatch.setenv("MC_CYCLE_WINDOW", "10") + monkeypatch.setenv("MC_CYCLE_ACTION", "warn") + adapter = _make_adapter() + with patch("gateway.platforms.daemoncraft.aiohttp.ClientSession", return_value=MagicMock()), \ + patch("asyncio.create_task", return_value=MagicMock()): + await adapter.connect() + assert adapter._cycle_detector is not None + assert adapter._cycle_detector.n == 3 + assert adapter._cycle_detector.window == 10 + assert adapter._cycle_detector.action == "warn" diff --git a/tests/gateway/test_daemoncraft_patches.py b/tests/gateway/test_daemoncraft_patches.py new file mode 100644 index 000000000000..7efae1eedd2d --- /dev/null +++ b/tests/gateway/test_daemoncraft_patches.py @@ -0,0 +1,197 @@ +"""Tests for CycleDetector and synthetic world-state injection in daemoncraft.py.""" +from __future__ import annotations + +import sys +import types +from unittest.mock import AsyncMock, MagicMock, call, patch + +import pytest + +# --------------------------------------------------------------------------- +# Stub heavy optional deps before importing daemoncraft +# --------------------------------------------------------------------------- + +def _stub_module(name: str, **attrs): + if name not in sys.modules: + mod = types.ModuleType(name) + for k, v in attrs.items(): + setattr(mod, k, v) + sys.modules[name] = mod + +_stub_module("httpx") +_stub_module("aiohttp", WSMsgType=MagicMock(), ClientSession=MagicMock) + +from gateway.platforms.daemoncraft import CycleDetector # noqa: E402 + + +# =========================================================================== +# CycleDetector tests +# =========================================================================== + +class TestCycleDetector: + """5 focused tests for CycleDetector behaviour.""" + + def test_no_trigger_below_threshold(self): + """N-1 identical calls must NOT trigger.""" + cd = CycleDetector(n=4, window=20, action="warn") + results = [cd.record("tool_x", {"k": "v"}) for _ in range(3)] + assert not any(r.triggered for r in results) + + def test_trigger_at_nth_identical_call(self): + """The Nth identical call must trigger.""" + cd = CycleDetector(n=3, window=10, action="warn") + r = None + for _ in range(3): + r = cd.record("loop_tool", {}) + assert r.triggered is True + assert r.count >= 3 + + def test_reset_after_action_no_double_trigger(self): + """After triggering, subsequent calls with the same sig should NOT re-trigger.""" + cd = CycleDetector(n=3, window=10, action="warn") + for _ in range(3): + cd.record("loop_tool", {}) + # 4th and 5th same-sig calls — suppressed + r4 = cd.record("loop_tool", {}) + r5 = cd.record("loop_tool", {}) + assert r4.triggered is False + assert r5.triggered is False + + def test_window_size_evicts_old_entries(self): + """Once the ring buffer (size=window) is filled with other sigs, old counts are gone.""" + # window=3: buffer holds at most 3 entries + cd = CycleDetector(n=3, window=3, action="warn") + # Two calls of "old_tool" — not yet triggering + cd.record("old_tool", {}) + cd.record("old_tool", {}) + # Fill buffer with 3 different sigs, evicting "old_tool" entries + cd.record("tool_b", {}) + cd.record("tool_c", {}) + cd.record("tool_d", {}) + # Now one more "old_tool" — only 1 in window, should not trigger + r = cd.record("old_tool", {}) + assert r.triggered is False + + def test_different_sigs_do_not_trigger(self): + """Calls with different args must not be counted together.""" + cd = CycleDetector(n=3, window=10, action="warn") + results = [cd.record("tool_a", {"n": i}) for i in range(6)] + assert not any(r.triggered for r in results) + + +# =========================================================================== +# synthetic world-state injection tests +# =========================================================================== + +def _make_adapter(): + from gateway.platforms.daemoncraft import DaemonCraftAdapter + from gateway.config import PlatformConfig + + cfg = PlatformConfig( + enabled=True, + extra={ + "bot_api_url": "http://localhost:9999", + "bot_username": "TestBot", + "profile": "test", + }, + ) + adapter = DaemonCraftAdapter(cfg) + return adapter + + +def _wire_adapter(adapter, *, session_id="world-session-1", hook_results=()): + """Attach a mock session_store and stub invoke_hook.""" + store = MagicMock() + store.append_to_transcript = MagicMock() + adapter._session_store = store + + # Stub _get_world_session_id + adapter._get_world_session_id = MagicMock(return_value=session_id) + return store + + +class TestSyntheticPerceiveHook: + """3 tests covering the transform_tool_result hook path.""" + + @pytest.mark.anyio + async def test_hook_called_before_transcript_append(self): + """invoke_hook must be called; tool_msg append comes after it.""" + adapter = _make_adapter() + store = _wire_adapter(adapter) + call_order = [] + + def fake_invoke_hook(event, **kwargs): + call_order.append("hook") + return iter([]) # no replacement + + # Capture append_to_transcript calls in order + original_append = store.append_to_transcript + def recording_append(sid, msg): + call_order.append(("append", msg["role"])) + store.append_to_transcript.side_effect = recording_append + + with patch("gateway.platforms.daemoncraft.invoke_hook", fake_invoke_hook, create=True), \ + patch.dict(sys.modules, {"hermes_cli.plugins": types.SimpleNamespace(invoke_hook=fake_invoke_hook)}): + # Patch the local import inside the synthetic injection path + import importlib + import gateway.platforms.daemoncraft as dc_mod + with patch.object(dc_mod, "_inject_synthetic_world_state_hook_module", None, create=True): + # We patch the from-import by monkeypatching the module namespace + pass + + # Direct patch: replace hermes_cli.plugins in sys.modules + fake_plugins = types.ModuleType("hermes_cli.plugins") + fake_plugins.invoke_hook = fake_invoke_hook + sys.modules["hermes_cli.plugins"] = fake_plugins + sys.modules.setdefault("hermes_cli", types.ModuleType("hermes_cli")) + + await adapter._inject_synthetic_world_state({"x": 1}) + + # assistant append should come first, then hook, then tool append + assert ("append", "assistant") in call_order + assert ("append", "tool") in call_order + assert call_order.index(("append", "assistant")) < call_order.index("hook") + assert call_order.index("hook") < call_order.index(("append", "tool")) + + @pytest.mark.anyio + async def test_hook_receives_mc_perceive_tool_name(self): + """Downstream consumers still see mc_perceive-shaped synthetic results.""" + adapter = _make_adapter() + _wire_adapter(adapter) + + received_kwargs: dict = {} + + def fake_invoke_hook(event, **kwargs): + received_kwargs.update({"event": event, **kwargs}) + return iter([]) + + fake_plugins = types.ModuleType("hermes_cli.plugins") + fake_plugins.invoke_hook = fake_invoke_hook + sys.modules["hermes_cli.plugins"] = fake_plugins + sys.modules.setdefault("hermes_cli", types.ModuleType("hermes_cli")) + + await adapter._inject_synthetic_world_state({"obs": "block"}) + + assert received_kwargs.get("event") == "transform_tool_result" + assert received_kwargs.get("tool_name") == "mc_perceive" + + @pytest.mark.anyio + async def test_transcript_appended_even_if_hook_raises(self): + """If invoke_hook raises, transcript append must still happen.""" + adapter = _make_adapter() + store = _wire_adapter(adapter) + + def exploding_hook(event, **kwargs): + raise RuntimeError("hook boom") + + fake_plugins = types.ModuleType("hermes_cli.plugins") + fake_plugins.invoke_hook = exploding_hook + sys.modules["hermes_cli.plugins"] = fake_plugins + sys.modules.setdefault("hermes_cli", types.ModuleType("hermes_cli")) + + await adapter._inject_synthetic_world_state({"obs": "fire"}) + + # Both assistant_msg and tool_msg must have been appended + assert store.append_to_transcript.call_count == 2 + roles = [c.args[1]["role"] for c in store.append_to_transcript.call_args_list] + assert roles == ["assistant", "tool"] diff --git a/tests/hermes_cli/test_kanban_review.py b/tests/hermes_cli/test_kanban_review.py new file mode 100644 index 000000000000..d3765c5ddc91 --- /dev/null +++ b/tests/hermes_cli/test_kanban_review.py @@ -0,0 +1,672 @@ +"""Tests for the ship-review graph creation helper and CLI.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from hermes_cli import kanban_db as kb +from hermes_cli import kanban_review as kr + + +@pytest.fixture +def kanban_home(tmp_path, monkeypatch): + home = tmp_path / ".hermes" + home.mkdir() + monkeypatch.setenv("HERMES_HOME", str(home)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + kb.init_db() + return home + + +@pytest.fixture +def fake_repo(tmp_path): + repo = tmp_path / "repo" + repo.mkdir() + (repo / ".git").mkdir() + return str(repo) + + +# --------------------------------------------------------------------------- +# ReviewGraphSpec +# --------------------------------------------------------------------------- + +def test_review_graph_spec_fields(): + spec = kr.ReviewGraphSpec( + repo_path="/tmp/repo", + base="nousmain", + head="feat/auth", + title="Review PR #42", + assignee="miki", + ready=True, + idempotency_prefix="prefix", + skills=["github-code-review"], + body="Extra context", + ) + assert spec.repo_path == "/tmp/repo" + assert spec.base == "nousmain" + assert spec.head == "feat/auth" + assert spec.title == "Review PR #42" + assert spec.assignee == "miki" + assert spec.ready is True + assert spec.idempotency_prefix == "prefix" + assert spec.skills == ["github-code-review"] + assert spec.body == "Extra context" + + +def test_review_graph_spec_defaults(): + spec = kr.ReviewGraphSpec(repo_path="/tmp/repo", base="main", head="feat/x", title="T") + assert spec.assignee is None + assert spec.ready is False + assert spec.idempotency_prefix is None + assert spec.skills == [] + assert spec.body is None + + +# --------------------------------------------------------------------------- +# Core helper tests +# --------------------------------------------------------------------------- + +def test_create_review_graph_smoke(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + assert result["created"] is True + assert result["parent_id"].startswith("t_") + assert len(result["reviewer_ids"]) == 3 + assert result["synthesis_id"].startswith("t_") + + # Verify synthesis is gated on reviewers + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + # Reviewers are parallel — they have no parents + assert kb.parent_ids(conn, rid) == [] + synth_parents = kb.parent_ids(conn, result["synthesis_id"]) + assert set(synth_parents) == set(result["reviewer_ids"]) + + +def test_create_review_graph_idempotent(kanban_home, fake_repo): + r1 = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + r2 = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + assert r1["parent_id"] == r2["parent_id"] + assert r1["reviewer_ids"] == r2["reviewer_ids"] + assert r1["synthesis_id"] == r2["synthesis_id"] + assert r2["created"] is False + + +def test_create_review_graph_idempotent_different_base_or_head(kanban_home, fake_repo): + """Changing base or head creates a new graph.""" + r1 = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + r2 = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/other", + repo_path=fake_repo, + ) + r3 = kr.create_review_graph( + title="Review PR #42", + base="main", + head="feat/auth", + repo_path=fake_repo, + ) + assert r1["parent_id"] != r2["parent_id"] + assert r1["parent_id"] != r3["parent_id"] + assert r2["parent_id"] != r3["parent_id"] + + +def test_create_review_graph_triage_by_default(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for tid in [result["parent_id"], *result["reviewer_ids"], result["synthesis_id"]]: + task = kb.get_task(conn, tid) + assert task.status == "triage" + + +def test_create_review_graph_ready_mode(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ready=True, + ) + with kb.connect() as conn: + parent = kb.get_task(conn, result["parent_id"]) + assert parent.status == "ready" + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + assert task.status == "ready" + # Synthesis starts as todo because its parents (reviewers) are not done. + synth = kb.get_task(conn, result["synthesis_id"]) + assert synth.status == "todo" + + +def test_create_review_graph_assignee_and_skills(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + assignee="miki", + skills=["github-code-review"], + ) + with kb.connect() as conn: + for tid in [result["parent_id"], *result["reviewer_ids"], result["synthesis_id"]]: + task = kb.get_task(conn, tid) + assert task.assignee == "miki" + assert task.skills == ["github-code-review"] + + +def test_create_review_graph_workspace_is_dir(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for tid in [result["parent_id"], *result["reviewer_ids"], result["synthesis_id"]]: + task = kb.get_task(conn, tid) + assert task.workspace_kind == "dir" + assert task.workspace_path == str(Path(fake_repo).resolve()) + + +def test_create_review_graph_body_appended(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + body="Extra context here", + ) + with kb.connect() as conn: + parent = kb.get_task(conn, result["parent_id"]) + assert "Extra context here" in parent.body + assert "nousmain" in parent.body + assert "feat/auth" in parent.body + + +def test_create_review_graph_parent_body_has_review_only_contract(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + parent = kb.get_task(conn, result["parent_id"]) + assert "REVIEW-ONLY v1" in parent.body + assert "Do NOT modify source code" in parent.body + + +def test_create_review_graph_reviewer_bodies_have_review_only_contract(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + assert "REVIEW-ONLY v1" in task.body + assert "Do NOT modify source code" in task.body + + +def test_create_review_graph_base_head_in_synthesis_body(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert "nousmain" in synth.body + assert "feat/auth" in synth.body + + +# --------------------------------------------------------------------------- +# CLI integration tests +# --------------------------------------------------------------------------- + +def test_cli_review_create_json(kanban_home, fake_repo): + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} --json" + ) + payload = json.loads(out) + assert payload["parent_id"].startswith("t_") + assert len(payload["reviewer_ids"]) == 3 + assert payload["synthesis_id"].startswith("t_") + assert payload["created"] is True + + +def test_cli_review_create_human_output(kanban_home, fake_repo): + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo}" + ) + assert "Created review graph" in out + assert "parent:" in out + assert "reviewer 1:" in out + assert "reviewer 2:" in out + assert "reviewer 3:" in out + assert "synthesis:" in out + + +def test_cli_review_create_idempotent_human_output(kanban_home, fake_repo): + from hermes_cli import kanban as kc + + kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo}" + ) + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo}" + ) + assert "Found existing review graph" in out + assert "all cards already existed" in out + + +def test_cli_review_create_missing_repo(kanban_home): + from hermes_cli import kanban as kc + + out = kc.run_slash( + "review create 'Review PR #42' --base nousmain --head feat/auth --repo /nonexistent/path" + ) + assert "is not a directory" in out + + +def test_cli_review_create_ready_flag(kanban_home, fake_repo): + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} --ready --json" + ) + payload = json.loads(out) + with kb.connect() as conn: + parent = kb.get_task(conn, payload["parent_id"]) + assert parent.status == "ready" + + +def test_cli_review_create_with_skills(kanban_home, fake_repo): + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} " + f"--skill github-code-review --skill security-scan --json" + ) + payload = json.loads(out) + with kb.connect() as conn: + task = kb.get_task(conn, payload["parent_id"]) + assert "github-code-review" in task.skills + assert "security-scan" in task.skills + + +def test_cli_review_create_base_head_required(kanban_home, fake_repo): + """Missing --base or --head should produce a usage error.""" + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --repo {fake_repo}" + ) + assert "usage error" in out.lower() + + +# --------------------------------------------------------------------------- +# Hardened template contract tests +# --------------------------------------------------------------------------- + +def test_reviewer_bodies_contain_exact_diff_command(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + assert "git diff nousmain...feat/auth --stat" in task.body + assert "git diff nousmain...feat/auth\n" in task.body + + +def test_reviewer_bodies_contain_severity_labels(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + assert "**Critical**" in task.body + assert "**Important**" in task.body + assert "**Optional/Nit**" in task.body + + +def test_reviewer_bodies_contain_kanban_contract(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + assert "kanban_complete" in task.body + assert "kanban_block" in task.body + assert '"findings":' in task.body + + +def test_reviewer_bodies_contain_role_specific_checklists(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + expected = { + "code-quality": [ + "Readability and naming conventions", + "DRY violations", + "Complexity and function length", + "Architectural consistency", + "Type safety", + "Documentation completeness", + ], + "security": [ + "Injection vectors", + "Hardcoded secrets", + "Input validation", + "Authentication/authorization gaps", + "Unsafe deserialization", + "Dependency risks", + "Privilege escalation", + ], + "test-coverage": [ + "New logic has accompanying tests", + "Edge cases are covered", + "Regression tests", + "Test readability", + "CI pass status", + "Integration / E2E coverage", + ], + } + with kb.connect() as conn: + roles = ["code-quality", "security", "test-coverage"] + for rid, role in zip(result["reviewer_ids"], roles): + task = kb.get_task(conn, rid) + for snippet in expected[role]: + assert snippet in task.body, f"{role} body missing: {snippet}" + + +def test_synthesis_body_contains_go_no_go(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert "GO/NO-GO decision" in synth.body + + +def test_synthesis_body_contains_all_required_sections(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert "Blockers" in synth.body + assert "Recommended fixes" in synth.body + assert "Acknowledged risks" in synth.body + assert "Rollback plan" in synth.body + assert "Evidence reviewed" in synth.body + + +def test_synthesis_body_contains_default_no_go_on_critical(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert "NO-GO" in synth.body + assert "Critical finding exists" in synth.body + + +def test_synthesis_body_contains_kanban_contract(kanban_home, fake_repo): + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert "kanban_complete" in synth.body + assert "kanban_block" in synth.body + assert '"ship_decision":' in synth.body + assert '"blockers":' in synth.body + assert '"recommended_fixes":' in synth.body + assert '"acknowledged_risks":' in synth.body + assert '"rollback_plan":' in synth.body + assert '"evidence_reviewed":' in synth.body + + +def test_generated_bodies_do_not_reference_skills(kanban_home, fake_repo): + """Template bodies must not instruct workers to load skills that may be + missing from the Miki profile. + """ + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + with kb.connect() as conn: + for tid in result["reviewer_ids"] + [result["synthesis_id"]]: + task = kb.get_task(conn, tid) + # Reject explicit skill-loading instructions (case-insensitive) + lower = task.body.lower() + assert "load the `" not in lower + assert "use the `" not in lower + assert "skill `" not in lower + + +# --------------------------------------------------------------------------- +# Synthesis promotion tests +# --------------------------------------------------------------------------- + +def test_reviewer_completion_promotes_synthesis(kanban_home, fake_repo): + """When all three reviewers are marked done, complete_task's internal + recompute_ready promotes the synthesis card from todo to ready.""" + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ready=True, + ) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + assert synth.status == "todo" + + for rid in result["reviewer_ids"]: + kb.complete_task(conn, rid, summary="review done") + + # complete_task calls recompute_ready internally; the third completion + # promotes the synthesis automatically. + synth = kb.get_task(conn, result["synthesis_id"]) + assert synth.status == "ready" + + +def test_synthesis_stays_todo_until_all_reviewers_done(kanban_home, fake_repo): + """If only two of three reviewers are done, synthesis stays in todo.""" + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ready=True, + ) + with kb.connect() as conn: + for rid in result["reviewer_ids"][:2]: + kb.complete_task(conn, rid, summary="review done") + + synth = kb.get_task(conn, result["synthesis_id"]) + assert synth.status == "todo" + + +# --------------------------------------------------------------------------- +# Self-contained body tests +# --------------------------------------------------------------------------- + +def test_self_contained_reviewer_bodies(kanban_home, fake_repo): + """Reviewer bodies must contain everything the worker needs without + relying on conversation history or external context. + """ + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + ws_path = str(Path(fake_repo).resolve()) + with kb.connect() as conn: + for rid in result["reviewer_ids"]: + task = kb.get_task(conn, rid) + body = task.body + assert "nousmain" in body + assert "feat/auth" in body + assert ws_path in body + assert "git diff" in body + assert "REVIEW-ONLY" in body + assert "kanban_complete" in body + assert "kanban_block" in body + assert "**Critical**" in body + assert "Checklist:" in body + + +def test_self_contained_synthesis_body(kanban_home, fake_repo): + """Synthesis body must contain everything the worker needs without + relying on conversation history or external context. + """ + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + ws_path = str(Path(fake_repo).resolve()) + with kb.connect() as conn: + synth = kb.get_task(conn, result["synthesis_id"]) + body = synth.body + assert "nousmain" in body + assert "feat/auth" in body + assert ws_path in body + assert "code-quality" in body + assert "security" in body + assert "test-coverage" in body + assert "GO/NO-GO" in body + assert "kanban_complete" in body + assert "kanban_block" in body + + +def test_self_contained_parent_body(kanban_home, fake_repo): + """Parent body must contain everything the worker needs without + relying on conversation history or external context. + """ + result = kr.create_review_graph( + title="Review PR #42", + base="nousmain", + head="feat/auth", + repo_path=fake_repo, + ) + ws_path = str(Path(fake_repo).resolve()) + with kb.connect() as conn: + parent = kb.get_task(conn, result["parent_id"]) + body = parent.body + assert "nousmain" in body + assert "feat/auth" in body + assert ws_path in body + assert "REVIEW-ONLY" in body + assert "code-quality" in body + assert "security" in body + assert "test-coverage" in body + assert "Synthesis card will aggregate" in body + + +# --------------------------------------------------------------------------- +# JSON CLI output structure +# --------------------------------------------------------------------------- + +def test_cli_review_create_json_structure(kanban_home, fake_repo): + """JSON output must contain exact keys with correct types.""" + from hermes_cli import kanban as kc + + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} --json" + ) + payload = json.loads(out) + assert set(payload.keys()) == {"parent_id", "reviewer_ids", "synthesis_id", "created"} + assert isinstance(payload["parent_id"], str) + assert isinstance(payload["reviewer_ids"], list) + assert len(payload["reviewer_ids"]) == 3 + for rid in payload["reviewer_ids"]: + assert isinstance(rid, str) + assert rid.startswith("t_") + assert isinstance(payload["synthesis_id"], str) + assert payload["synthesis_id"].startswith("t_") + assert isinstance(payload["created"], bool) + + +def test_cli_review_create_json_idempotent_returns_false(kanban_home, fake_repo): + """Second invocation with same params must return created=False.""" + from hermes_cli import kanban as kc + + kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} --json" + ) + out = kc.run_slash( + f"review create 'Review PR #42' --base nousmain --head feat/auth --repo {fake_repo} --json" + ) + payload = json.loads(out) + assert payload["created"] is False + diff --git a/tests/hermes_cli/test_runtime_provider_resolution.py b/tests/hermes_cli/test_runtime_provider_resolution.py index 3e788fe3d538..533314eaf2f6 100644 --- a/tests/hermes_cli/test_runtime_provider_resolution.py +++ b/tests/hermes_cli/test_runtime_provider_resolution.py @@ -977,6 +977,46 @@ def test_named_custom_provider_does_not_shadow_builtin_provider(monkeypatch): assert resolved["requested_provider"] == "nous" +def test_kimi_runtime_uses_cli_oauth_when_api_key_missing(monkeypatch): + """Kimi Coding must use ~/.kimi OAuth credentials when available. + + Regression guard for the fork patch: resolve_api_key_provider_credentials() + now centralizes OAuth resolution, so the runtime provider receives the + OAuth token directly instead of falling through to no-key-required (which + drops Kimi's required X-Msh headers and causes 404s). + """ + monkeypatch.setattr(rp, "resolve_provider", lambda *a, **k: "kimi-coding") + monkeypatch.setattr(rp, "load_pool", lambda provider: None) + monkeypatch.setattr( + rp, + "_get_model_config", + lambda: { + "provider": "kimi-coding", + # Existing fork configs may still store the OpenAI-wire URL. + "base_url": "https://api.kimi.com/coding/v1", + "default": "kimi-k2.6", + }, + ) + monkeypatch.setattr( + rp, + "resolve_api_key_provider_credentials", + lambda provider: { + "provider": provider, + "api_key": "oauth-token", + "base_url": "https://api.kimi.com/coding/v1", + "source": "kimi-cli-oauth", + }, + ) + + resolved = rp.resolve_runtime_provider(requested="kimi-coding") + + assert resolved["provider"] == "kimi-coding" + assert resolved["api_key"] == "oauth-token" + assert resolved["source"] == "kimi-cli-oauth" + assert resolved["base_url"] == "https://api.kimi.com/coding/v1" + assert resolved["api_mode"] == "chat_completions" + + def test_named_custom_provider_wins_over_builtin_alias(monkeypatch): """A custom_providers entry named after a built-in *alias* (not a canonical provider name) must win over the built-in. Regression guard for #15743: diff --git a/tests/tools/test_embodied_plan_tool.py b/tests/tools/test_embodied_plan_tool.py new file mode 100644 index 000000000000..be644afa6adc --- /dev/null +++ b/tests/tools/test_embodied_plan_tool.py @@ -0,0 +1,320 @@ +"""Tests for tools.embodied_plan_tool.""" +from __future__ import annotations + +import json +from unittest.mock import MagicMock, patch + +import pytest + +import httpx + + +def test_tool_registered(): + """Importing the module should register the tool in the global registry.""" + from tools.registry import registry + import tools.embodied_plan_tool # noqa: F401 + + tool = registry.get_entry("embodied_plan") + assert tool is not None + assert tool.toolset == "embodiment" + assert tool.schema["function"]["name"] == "embodied_plan" + + +def test_handler_rejects_missing_intent(): + from tools.embodied_plan_tool import _handler + + out = _handler({}) + payload = json.loads(out) + assert payload["ok"] is False + assert payload["error"]["error_type"] == "missing_intent" + + +def test_handler_rejects_non_string_intent(): + from tools.embodied_plan_tool import _handler + + out = _handler({"intent": 42}) + payload = json.loads(out) + assert payload["ok"] is False + assert payload["error"]["error_type"] == "missing_intent" + + +def test_handler_posts_intent_to_service(): + """Standard happy path — handler posts to /intent and returns the + service's response verbatim.""" + from tools.embodied_plan_tool import _handler + + fake_response = MagicMock() + fake_response.json.return_value = { + "ok": True, + "context_id": "abc-123", + "plan": { + "body_plan": ["scan", "mine"], + "checks": ["time=day"], + "tool_calls": [{"name": "scan_nearby", "arguments": {"radius": 16}}], + "failure_policy": "ask the player", + "operational_risk": "low", + }, + "execution_results": [{"tool": "scan_nearby", "ok": True, "data": {}}], + "elapsed_seconds": 1.2, + } + fake_response.status_code = 200 + + captured = {} + def fake_post(url, json=None, timeout=None): + captured["url"] = url + captured["body"] = json + captured["timeout"] = timeout + return fake_response + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({ + "intent": "Help the player gather wood before night.", + "autonomy_level": 2, + "allowed_tools": ["scan_nearby", "mine_block"], + }) + + assert captured["url"].endswith("/intent") + assert captured["body"]["intent"] == "Help the player gather wood before night." + assert captured["body"]["autonomy_level"] == 2 + assert captured["body"]["allowed_tools"] == ["scan_nearby", "mine_block"] + payload = json.loads(out) + assert payload["ok"] is True + assert payload["plan"]["operational_risk"] == "low" + + +def test_handler_omits_none_optional_fields(): + """Optional fields that are None must NOT be in the request body — the + service treats absence as 'use default', not as 'use None'.""" + from tools.embodied_plan_tool import _handler + + captured = {} + def fake_post(url, json=None, timeout=None): + captured["body"] = json + resp = MagicMock() + resp.json.return_value = {"ok": True} + resp.status_code = 200 + return resp + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + _handler({ + "intent": "Do a thing.", + "previous_error": None, # explicitly None — should not be forwarded + }) + + assert "intent" in captured["body"] + assert "previous_error" not in captured["body"] + + +def test_handler_handles_timeout(): + from tools.embodied_plan_tool import _handler + + with patch("tools.embodied_plan_tool.httpx.post", + side_effect=httpx.TimeoutException("request timed out")): + out = _handler({"intent": "test"}) + payload = json.loads(out) + assert payload["ok"] is False + assert payload["error"]["error_type"] == "embodied_service_timeout" + + +def test_handler_handles_connection_error(): + from tools.embodied_plan_tool import _handler + + with patch("tools.embodied_plan_tool.httpx.post", + side_effect=httpx.ConnectError("connection refused")): + out = _handler({"intent": "test"}) + payload = json.loads(out) + assert payload["ok"] is False + assert payload["error"]["error_type"] == "embodied_service_unreachable" + + +def test_handler_handles_non_json_response(): + from tools.embodied_plan_tool import _handler + + fake_response = MagicMock() + fake_response.json.side_effect = json.JSONDecodeError("bad", "", 0) + fake_response.status_code = 502 + fake_response.text = "bad gateway" + + with patch("tools.embodied_plan_tool.httpx.post", return_value=fake_response): + out = _handler({"intent": "test"}) + payload = json.loads(out) + assert payload["ok"] is False + assert payload["error"]["error_type"] == "embodied_service_bad_response" + + +def test_check_service_available_validates_url(): + from tools.embodied_plan_tool import _check_service_available + + assert _check_service_available() is True # default http://localhost:7790 + + +def test_service_url_respects_env(monkeypatch): + monkeypatch.setenv("EMBODIED_SERVICE_URL", "http://10.10.20.5:7790") + from tools.embodied_plan_tool import _service_url + + assert _service_url() == "http://10.10.20.5:7790" + + +# ─── Policy-mode tests ──────────────────────────────────────────────────────── + +def test_policy_mode_raw_explicit(): + """Explicit 'raw' should bypass the policy layer entirely.""" + from tools.embodied_plan_tool import _handler + + captured = {} + def fake_post(url, json=None, timeout=None): + captured["body"] = json + resp = MagicMock() + resp.json.return_value = {"ok": True, "plan": {}, "execution_results": []} + resp.status_code = 200 + return resp + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({"intent": "Mine 1 oak_log", "policy_mode": "raw"}) + + assert captured["body"]["intent"] == "Mine 1 oak_log" + payload = json.loads(out) + assert payload["ok"] is True + + +def test_policy_scope_filter_blocks_joke(monkeypatch): + """Out-of-scope chat (joke) must be handled upstream — no HTTP call.""" + monkeypatch.setenv("BOT_API_URL", "http://bot:3000") + from tools.embodied_plan_tool import _handler + + calls = [] + def fake_post(url, json=None, timeout=None): + calls.append(json) + return MagicMock() + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({"intent": "Contame un chiste corto", "policy_mode": "auto"}) + + assert len(calls) == 0 + payload = json.loads(out) + assert payload["ok"] is True + assert payload["outcome"] == "policy_handled_upstream" + assert payload["policy_layer"] == "scope" + assert payload["mitigation"]["intent_original"] == "Contame un chiste corto" + + +def test_policy_ambiguity_blocks_vague(monkeypatch): + """Vague intents must be handled upstream — no HTTP call.""" + monkeypatch.setenv("BOT_API_URL", "http://bot:3000") + from tools.embodied_plan_tool import _handler + + calls = [] + def fake_post(url, json=None, timeout=None): + calls.append(json) + return MagicMock() + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({"intent": "Hacé algo entretenido", "policy_mode": "auto"}) + + assert len(calls) == 0 + payload = json.loads(out) + assert payload["ok"] is True + assert payload["outcome"] == "policy_handled_upstream" + assert payload["policy_layer"] == "ambiguity" + + +def test_policy_decomposes_and_normalizes(monkeypatch): + """Multi-step intents are split, normalized, and posted sequentially.""" + monkeypatch.setenv("BOT_API_URL", "http://bot:3000") + from tools.embodied_plan_tool import _handler + + calls = [] + def fake_post(url, json=None, timeout=None): + calls.append(json) + resp = MagicMock() + resp.json.return_value = { + "ok": True, + "plan": {"body_plan": []}, + "execution_results": [{"tool": "scan_nearby", "ok": True}], + } + resp.status_code = 200 + return resp + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({ + "intent": "Mine 3 oak_log and craft a crafting table", + "policy_mode": "auto", + }) + + assert len(calls) == 2 + # First sub-intent should be normalized English imperative + assert calls[0]["intent"].startswith("Mine") + # Second sub-intent should also be normalized + assert calls[1]["intent"].startswith("Craft") + + payload = json.loads(out) + assert payload["ok"] is True + assert payload["outcome"] == "embodied_ready" + assert len(payload["mitigation"]["normalized_chain"]) == 2 + assert payload["mitigation"]["category_chain"][0] == "mining" + assert len(payload["execution_results"]) == 2 + + +def test_policy_narrows_tools(monkeypatch): + """Policy should narrow allowed_tools by intent category.""" + monkeypatch.setenv("BOT_API_URL", "http://bot:3000") + from tools.embodied_plan_tool import _handler + + calls = [] + def fake_post(url, json=None, timeout=None): + calls.append(json) + resp = MagicMock() + resp.json.return_value = {"ok": True, "plan": {}, "execution_results": []} + resp.status_code = 200 + return resp + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({"intent": "Equip torch", "policy_mode": "auto"}) + + assert len(calls) == 1 + narrowed = calls[0].get("allowed_tools", []) + assert "equip_item" in narrowed + assert "mine_block" not in narrowed + + payload = json.loads(out) + assert payload["mitigation"]["allowed_tools_chain"][0] == narrowed + + +def test_policy_threads_previous_error(monkeypatch): + """If a sub-intent fails, previous_error is threaded into the next call.""" + monkeypatch.setenv("BOT_API_URL", "http://bot:3000") + from tools.embodied_plan_tool import _handler + + call_idx = 0 + def fake_post(url, json=None, timeout=None): + nonlocal call_idx + call_idx += 1 + resp = MagicMock() + if call_idx == 1: + resp.json.return_value = { + "ok": False, + "error": { + "error_type": "missing_material", + "details": "no oak_log in inventory", + }, + } + else: + resp.json.return_value = { + "ok": True, + "plan": {}, + "execution_results": [{"tool": "scan_nearby", "ok": True}], + } + resp.status_code = 200 + return resp + + with patch("tools.embodied_plan_tool.httpx.post", side_effect=fake_post): + out = _handler({ + "intent": "Mine 3 oak_log and craft a crafting table", + "policy_mode": "auto", + }) + + payload = json.loads(out) + assert payload["ok"] is True + assert len(payload["execution_results"]) == 2 + assert payload["execution_results"][0]["ok"] is False + assert payload["execution_results"][1]["ok"] is True diff --git a/tests/tools/test_kimi_webbridge.py b/tests/tools/test_kimi_webbridge.py new file mode 100644 index 000000000000..041719e95a59 --- /dev/null +++ b/tests/tools/test_kimi_webbridge.py @@ -0,0 +1,215 @@ +"""Tests for the Kimi WebBridge toolset.""" + +import json +from unittest.mock import MagicMock, patch + +import pytest + +from tools.kimi_webbridge import ( + _check_bridge, + _get_daemon_url, + _validate_screenshot_path, + kimi_webbridge_click, + kimi_webbridge_close_session, + kimi_webbridge_close_tab, + kimi_webbridge_evaluate, + kimi_webbridge_fill, + kimi_webbridge_find_tab, + kimi_webbridge_list_tabs, + kimi_webbridge_navigate, + kimi_webbridge_save_pdf, + kimi_webbridge_save_screenshot, + kimi_webbridge_screenshot, + kimi_webbridge_snapshot, +) + + +def _mock_response(data: dict, status_code: int = 200) -> MagicMock: + """Build a mock requests.Response.""" + resp = MagicMock() + resp.status_code = status_code + resp.json.return_value = data + resp.raise_for_status.return_value = None + return resp + + +class TestGetDaemonUrl: + def test_default_fallback(self): + with patch("hermes_cli.config.load_config", side_effect=Exception("no config")): + assert _get_daemon_url() == "http://127.0.0.1:10086" + + def test_from_config(self): + fake_cfg = {"providers": {"kimi_webbridge": {"base_url": "http://localhost:9999"}}} + with patch("hermes_cli.config.load_config", return_value=fake_cfg): + assert _get_daemon_url() == "http://localhost:9999" + + def test_trailing_slash_stripped(self): + fake_cfg = {"providers": {"kimi_webbridge": {"base_url": "http://localhost:9999/"}}} + with patch("hermes_cli.config.load_config", return_value=fake_cfg): + assert _get_daemon_url() == "http://localhost:9999" + + +class TestCheckBridge: + def test_true_when_daemon_responds(self): + with patch("tools.kimi_webbridge.requests.post", return_value=_mock_response({})): + assert _check_bridge() is True + + def test_false_when_request_fails(self): + with patch("tools.kimi_webbridge.requests.post", side_effect=Exception("connection refused")): + assert _check_bridge() is False + + def test_false_when_non_200(self): + resp = _mock_response({}, status_code=500) + with patch("tools.kimi_webbridge.requests.post", return_value=resp): + assert _check_bridge() is False + + +class TestValidateScreenshotPath: + def test_default_path(self): + path = _validate_screenshot_path(None) + assert str(path).startswith("/tmp/kimi-webbridge-screenshots/") + assert path.suffix == ".png" + + def test_valid_tmp_path(self): + path = _validate_screenshot_path("/tmp/foo.png") + assert str(path) == "/tmp/foo.png" + + def test_valid_home_path(self): + path = _validate_screenshot_path("~/foo.png") + assert "foo.png" in str(path) + + def test_invalid_path_rejected(self): + with pytest.raises(ValueError, match="must be under /tmp or home directory"): + _validate_screenshot_path("/etc/passwd") + + +class TestNavigate: + def test_navigate_success(self): + mock = _mock_response({"ok": True, "data": {"success": True, "url": "https://example.com", "tabId": 42}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_navigate("https://example.com")) + assert result["ok"] is True + assert result["data"]["url"] == "https://example.com" + + def test_navigate_with_group_title(self): + mock = _mock_response({"ok": True, "data": {"success": True}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock) as post: + kimi_webbridge_navigate("https://example.com", group_title="my-group") + call_args = post.call_args[1]["json"] + assert call_args["args"]["group_title"] == "my-group" + + +class TestSnapshot: + def test_snapshot_returns_tree(self): + mock = _mock_response({"ok": True, "data": {"url": "https://example.com", "title": "Example", "tree": []}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_snapshot()) + assert result["data"]["title"] == "Example" + + +class TestClick: + def test_click_by_selector(self): + mock = _mock_response({"ok": True, "data": {"success": True, "tag": "button", "text": "Submit"}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_click("@e5")) + assert result["data"]["tag"] == "button" + + +class TestFill: + def test_fill_input(self): + mock = _mock_response({"ok": True, "data": {"success": True, "tag": "input", "mode": "value"}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_fill("@e3", "hello")) + assert result["data"]["mode"] == "value" + + +class TestEvaluate: + def test_evaluate_js(self): + mock = _mock_response({"ok": True, "data": {"type": "string", "value": "42"}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_evaluate("document.title")) + assert result["data"]["value"] == "42" + + +class TestListTabs: + def test_list_tabs(self): + mock = _mock_response({"ok": True, "data": {"success": True, "tabs": [{"tabId": 1, "url": "https://a.com"}]}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_list_tabs()) + assert len(result["data"]["tabs"]) == 1 + + +class TestCloseTab: + def test_close_tab(self): + mock = _mock_response({"ok": True, "data": {"success": True, "closed": True}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_close_tab()) + assert result["data"]["closed"] is True + + +class TestCloseSession: + def test_close_session(self): + mock = _mock_response({"ok": True, "data": {"success": True, "closed": 3}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_close_session()) + assert result["data"]["closed"] == 3 + + +class TestFindTab: + def test_find_tab(self): + mock = _mock_response({"ok": True, "data": {"success": True, "url": "https://kimi.com", "tabId": 7}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_find_tab("https://kimi.com", active=True)) + assert result["data"]["tabId"] == 7 + + +class TestSavePdf: + def test_save_pdf(self): + mock = _mock_response({"ok": True, "data": {"path": "/tmp/test.pdf", "sizeBytes": 1234}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_save_pdf(file_name="test.pdf")) + assert result["data"]["path"] == "/tmp/test.pdf" + + +class TestScreenshot: + def test_screenshot_strips_base64(self): + mock = _mock_response({"ok": True, "data": {"format": "png", "dataLength": 50000, "data": "iVBORw0KGgo" * 1000}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_screenshot()) + inner = result["data"] + assert "base64 image data" in inner["data"] + + +class TestSaveScreenshot: + def test_save_screenshot_success(self, tmp_path): + fake_b64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==" + mock = _mock_response({"ok": True, "data": {"format": "png", "dataLength": len(fake_b64), "data": fake_b64}}) + output = tmp_path / "shot.png" + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_save_screenshot(str(output))) + assert result["success"] is True + assert result["path"] == str(output) + assert output.exists() + + def test_save_screenshot_failure_no_data(self): + mock = _mock_response({"ok": True, "data": {"format": "png"}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_save_screenshot()) + assert "error" in result + + def test_save_screenshot_invalid_path(self): + fake_b64 = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==" + mock = _mock_response({"ok": True, "data": {"format": "png", "dataLength": len(fake_b64), "data": fake_b64}}) + with patch("tools.kimi_webbridge.requests.post", return_value=mock): + result = json.loads(kimi_webbridge_save_screenshot("/etc/evil.png")) + assert "error" in result + assert "must be under /tmp or home directory" in result.get("message", "") + + +class TestErrorHandling: + def test_request_exception_returns_error_dict(self): + import requests + with patch("tools.kimi_webbridge.requests.post", side_effect=requests.ConnectionError("boom")): + result = json.loads(kimi_webbridge_navigate("https://example.com")) + assert result["error"] is True + assert "boom" in result["message"] diff --git a/tests/tools/test_mc_bit_tool.py b/tests/tools/test_mc_bit_tool.py new file mode 100644 index 000000000000..d623114bdcc8 --- /dev/null +++ b/tests/tools/test_mc_bit_tool.py @@ -0,0 +1,79 @@ +"""Tests for the mc_bit Hermes tool wrapper.""" + +from __future__ import annotations + +import importlib + + +def test_mc_bit_handler_is_sync_and_formats_response(monkeypatch): + tool = importlib.import_module("tools.mc_bit_tool") + + captured = {} + + class FakeResponse: + def raise_for_status(self): + return None + + def json(self): + return { + "ok": True, + "data": { + "format": "surface", + "text": "GG\nTT\n", + "count": 4, + "elapsed_ms": 2, + }, + } + + def fake_get(url, params, timeout): + captured["url"] = url + captured["params"] = params + captured["timeout"] = timeout + return FakeResponse() + + monkeypatch.setenv("MC_API_URL", "http://bot.test:3003") + monkeypatch.setattr(tool.httpx, "get", fake_get) + + result = tool._handler({ + "x1": 1, + "y1": 2, + "z1": 3, + "x2": 4, + "y2": 5, + "z2": 6, + "format": "surface", + }) + + assert isinstance(result, str) + assert result == "mBit surface (4 blocks, 2ms):\nGG\nTT\n" + assert captured == { + "url": "http://bot.test:3003/blocks", + "params": {"x1": 1, "y1": 2, "z1": 3, "x2": 4, "y2": 5, "z2": 6, "format": "surface"}, + "timeout": 10.0, + } + + +def test_mc_bit_missing_coordinates_returns_error(): + tool = importlib.import_module("tools.mc_bit_tool") + + result = tool._handler({"x1": 1}) + + assert "requires x1, y1, z1, x2, y2, z2" in result + assert "missing:" in result + + +def test_mc_bit_invalid_format_returns_error(): + tool = importlib.import_module("tools.mc_bit_tool") + + result = tool._handler({ + "x1": 1, + "y1": 2, + "z1": 3, + "x2": 4, + "y2": 5, + "z2": 6, + "format": "bad", + }) + + assert "unsupported format" in result + assert "binary" in result diff --git a/tools/bot_api_url_ctx.py b/tools/bot_api_url_ctx.py new file mode 100644 index 000000000000..df66038adf8f --- /dev/null +++ b/tools/bot_api_url_ctx.py @@ -0,0 +1,53 @@ +"""Session-scoped bot API URL routing. + +The DaemonCraft (and previously AlterCraft) gateway adapter receives chat +messages from a Minecraft world over WebSocket, and needs the tool layer +to dispatch HTTP back to the *same* bot that sent the message — not just +to whatever the process-wide env var says. This contextvar is the +mechanism: the gateway sets it inside `handle_message`, tools read it +through `get_bot_api_url`, and the gateway resets it on exit. + +This module replaces the same-named contextvar that lived inside +`tools/minecraft_tools.py` (retired 2026-05-09 along with the rest of +the mc_*/altercraft_* toolset stack — see legacy/altercraft-toolsets +branch). Keeping the contextvar in a neutral module decouples the +gateway from any specific toolset implementation. + +Today only the embodied service path (POST → embodied service → bot) +needs this. Future tools that hit a Mineflayer bot directly should +import from here rather than reintroducing a per-toolset contextvar. +""" +from __future__ import annotations + +import contextvars +import os +from typing import Optional + + +_bot_api_url_ctx: contextvars.ContextVar[Optional[str]] = contextvars.ContextVar( + "bot_api_url", default=None +) + + +def get_bot_api_url() -> str: + """Resolve the active bot HTTP API URL for the current call context. + + Priority: + 1. Context variable (set by the gateway adapter for the lifetime + of one inbound message) + 2. ``MC_API_URL`` environment variable (CLI / legacy fallback) + 3. Default ``http://localhost:3001`` + """ + url = _bot_api_url_ctx.get() + if url: + return url + return os.getenv("MC_API_URL", "http://localhost:3001") + + +def set_bot_api_url(url: str) -> contextvars.Token: + """Set the contextvar and return the token. Caller MUST `reset` it.""" + return _bot_api_url_ctx.set(url) + + +def reset_bot_api_url(token: contextvars.Token) -> None: + _bot_api_url_ctx.reset(token) diff --git a/tools/embodied_plan_tool.py b/tools/embodied_plan_tool.py new file mode 100644 index 000000000000..5622918bedf8 --- /dev/null +++ b/tools/embodied_plan_tool.py @@ -0,0 +1,528 @@ +#!/usr/bin/env python3 +"""embodied_plan — single-tool body orchestration delegate. + +The Hermes-side counterpart to the DaemonCraft embodied service v1. + +Hermes' cloud LLM (Kimi/MiniMax/etc.) calls this **one tool** when it +needs the body to do something. The embodied service handles: + + 1. Reading world_state from bot/server.js + 2. Filtering allowed_tools by executor_supported + 3. Composing a canonical Gemma-Andy v2 payload + 4. Calling Ollama (gemma-andy:e4b-v2-2-3-q8_0) + 5. Parsing the response (with strip + bracket fallback) + 6. Dispatching each tool_call to bot/server.js + 7. Returning the assembled {plan, execution_results} + +Hermes never has to know about the granular Mineflayer mc_* tools — that +is Gemma-Andy's job. Path B canonical per team architectural decision +2026-05-08 (see vault/concepts/gemma-andy-embodied-service.md and +vault/epics/E002-body-protocol-wireup.md). + +Environment: + EMBODIED_SERVICE_URL Base URL of the embodied service + (default: http://localhost:7790) + EMBODIED_PLAN_TIMEOUT Per-request timeout in seconds + (default: 60 — Ollama + dispatch can be slow) +""" + +from __future__ import annotations + +import json +import logging +import os +from typing import Any + +import httpx + +from tools.registry import registry +from tools.bot_api_url_ctx import get_bot_api_url + +logger = logging.getLogger(__name__) + + +def _service_url() -> str: + return os.environ.get("EMBODIED_SERVICE_URL", "http://localhost:7790").rstrip("/") + + +def _bot_api_url(args: dict[str, Any]) -> str | None: + """Resolve bot API URL from args, env, or gateway context.""" + return args.get("bot_api_url") or os.environ.get("BOT_API_URL") or get_bot_api_url() or None + + +def _timeout() -> float: + try: + return float(os.environ.get("EMBODIED_PLAN_TIMEOUT", "60")) + except ValueError: + return 60.0 + + +# --------------------------------------------------------------------------- +# Policy integration (optional — graceful degradation if gemma_policy missing) +# --------------------------------------------------------------------------- + +try: + from tools.gemma_policy import GemmaPolicy + + _GemmaPolicy = GemmaPolicy +except Exception as _import_err: # pragma: no cover + logger.debug("gemma_policy not available: %s", _import_err) + _GemmaPolicy = None + + +def _policy_mode_default() -> str: + """Platform-aware default: 'auto' when DaemonCraft bot context is detected, + 'raw' for CLI and other platforms (backward compatibility).""" + return "auto" if os.environ.get("BOT_API_URL") else "raw" + + +# --------------------------------------------------------------------------- +# HTTP helpers +# --------------------------------------------------------------------------- + +def _cancel_bot_task(bot_api_url: str | None = None) -> None: + """Cancel any active task on the bot server before sending a new intent. + + Fire-and-forget with a 2-second timeout — we never block a new + embodied_plan call waiting for the old task to finish. If the bot + is unreachable or the cancel fails, we proceed anyway; the new + intent will surface the conflict as a tool-level 409 if needed. + + This prevents the #1 cause of embodied_plan timeouts: calling a + new intent while the bot is still executing tool_calls from the + previous plan.""" + + url = bot_api_url or os.environ.get("BOT_API_URL") + if not url: + return + try: + httpx.post(f"{url.rstrip('/')}/task/cancel", json={}, timeout=2.0) + except Exception: + pass # Fire-and-forget — bot might be down, we proceed either way + + +def _post_intent(body: dict[str, Any]) -> dict[str, Any]: + """POST *body* to the embodied-service ``/intent`` endpoint and return the + parsed JSON response as a Python dict. + + All network and decode errors are caught and returned as + ``{"ok": False, "error": {...}}`` so callers never raise.""" + url = f"{_service_url()}/intent" + timeout = _timeout() + try: + resp = httpx.post(url, json=body, timeout=timeout) + except httpx.TimeoutException: + return { + "ok": False, + "error": { + "error_type": "embodied_service_timeout", + "details": f"timed out after {timeout}s waiting for {url}", + }, + } + except httpx.RequestError as exc: + return { + "ok": False, + "error": { + "error_type": "embodied_service_unreachable", + "details": f"{type(exc).__name__}: {exc}", + }, + } + try: + return resp.json() + except json.JSONDecodeError: + return { + "ok": False, + "error": { + "error_type": "embodied_service_bad_response", + "details": f"non-JSON body (status {resp.status_code}): {resp.text[:200]}", + }, + } + + +# --------------------------------------------------------------------------- +# Result summarizer — gives the LLM a one-line verdict instead of raw JSON +# --------------------------------------------------------------------------- + +def _summarize_result(result: dict, intent: str) -> str: + """Return a one-line human-readable summary of what gAndy did.""" + if not isinstance(result, dict): + return f"gAndy returned unexpected: {str(result)[:100]}" + + ok = result.get("ok") + plan = result.get("plan") or {} + exec_results = result.get("execution_results") or [] + outcome = result.get("outcome", "") + + # Policy handled upstream + if result.get("policy_handled"): + return f"gAndy: handled by {result.get('policy_layer','?')} — {result.get('policy_reason','')[:120]}" + + # Success + if ok: + body_plan = plan.get("body_plan") or [] + tool_names = [s.get("tool", "?") for s in body_plan[:3]] if isinstance(body_plan, list) else [] + tools_str = ", ".join(tool_names) if tool_names else "acted" + if exec_results: + ok_count = sum(1 for r in exec_results if r.get("ok")) + return f"gAndy: OK — {ok_count}/{len(exec_results)} steps succeeded ({tools_str}). Intent: {intent[:80]}" + return f"gAndy: OK — {tools_str}. Intent: {intent[:80]}" + + # Failure + error = result.get("error") or {} + error_type = error.get("error_type", "unknown") if isinstance(error, dict) else str(error)[:80] + details = error.get("details", "") if isinstance(error, dict) else "" + if outcome == "policy_handled_upstream": + return f"gAndy: handled upstream ({error_type})" + if exec_results: + failed_steps = [r for r in exec_results if not r.get("ok")] + if failed_steps: + first = failed_steps[0] + return f"gAndy: FAILED — {first.get('error_type','?')}: {first.get('details','')[:100]}" + return f"gAndy: FAILED — {error_type}: {details[:100]}" + +# --------------------------------------------------------------------------- +# Raw passthrough handler (debugging escape hatch) +# --------------------------------------------------------------------------- + +def _raw_handler(args: dict[str, Any]) -> str: + """Forward the intent verbatim to the embodied service.""" + intent = args.get("intent", "") + if not intent or not isinstance(intent, str): + return json.dumps({ + "ok": False, + "error": { + "error_type": "missing_intent", + "details": "embodied_plan requires a non-empty 'intent' string", + }, + }) + + body: dict[str, Any] = {"intent": intent} + for k in ( + "autonomy_level", + "allowed_tools", + "guardian_constraints", + "previous_error", + "deadline_seconds", + ): + if k in args and args[k] is not None: + body[k] = args[k] + + bot_api_url = _bot_api_url(args) + if bot_api_url: + body["bot_api_url"] = bot_api_url + + result = _post_intent(body) + + # Tier 2a recovery: deterministic synthesis for spatial failures. + failed = [r for r in (result.get("execution_results") or []) if not r.get("ok")] + if failed and not body.get("previous_error"): + first_failure = failed[0] + error_type = first_failure.get("error_type", "") + retry_body = dict(body) + if error_type in ("target_occupied", "bot_in_target", "no_solid_neighbor"): + retry_body["previous_error"] = { + "tool": first_failure.get("tool"), + "error_type": error_type, + "details": first_failure.get("details", ""), + } + retry_body["_recovery_hint"] = "spatial_retry_adjacent" + result = _post_intent(retry_body) + + # Build a human-readable summary so the LLM doesn't have to parse the full JSON + summary = _summarize_result(result, body.get("intent", "")) + result["_summary"] = summary + + return json.dumps(result) + + +# --------------------------------------------------------------------------- +# Policy-wrapped handler +# --------------------------------------------------------------------------- + +def _policy_handler(args: dict[str, Any]) -> str: + """Run the GemmaPolicy L2→L3→L5→(L1+L4) pipeline before calling the + embodied service. Sub-intents are POSTed sequentially to ``/intent``; + ``previous_error`` is threaded between calls.""" + intent = args.get("intent", "") + if not intent or not isinstance(intent, str): + return json.dumps({ + "ok": False, + "error": { + "error_type": "missing_intent", + "details": "embodied_plan requires a non-empty 'intent' string", + }, + }) + + if _GemmaPolicy is None: + logger.warning("policy_mode=auto but gemma_policy unavailable; falling back to raw") + return _raw_handler(args) + + policy = _GemmaPolicy() + policy_result = policy.execute(intent) + + # L2 / L3 cut — handled upstream, do NOT call /intent + if policy_result["policy_handled"]: + return json.dumps({ + "ok": True, + "outcome": "policy_handled_upstream", + "policy_handled": True, + "policy_layer": policy_result["policy_layer"], + "policy_reason": policy_result["policy_reason"], + "plan": None, + "execution_results": [], + "mitigation": { + "policy_layer": policy_result["policy_layer"], + "policy_reason": policy_result["policy_reason"], + "intent_original": intent, + "sub_intents_count": 0, + "normalized_chain": [], + "category_chain": [], + "allowed_tools_chain": [], + "sub_intent_outcomes": [], + }, + }) + + sub_intents = policy_result["sub_intents"] + categories = policy_result["categories"] + allowed_tools_chain = policy_result["allowed_tools"] + + all_execution_results: list[dict] = [] + aggregated_plan = None + previous_error = None + sub_intent_outcomes: list[str] = [] + + for idx, (sub_intent, category, policy_tools) in enumerate(zip( + sub_intents, categories, allowed_tools_chain + )): + body: dict[str, Any] = {"intent": sub_intent} + for k in ("autonomy_level", "guardian_constraints", "deadline_seconds"): + if k in args and args[k] is not None: + body[k] = args[k] + + # Use caller's allowed_tools if explicitly provided, else policy-narrowed + if "allowed_tools" in args and args["allowed_tools"] is not None: + body["allowed_tools"] = args["allowed_tools"] + else: + body["allowed_tools"] = policy_tools + + if previous_error is not None: + body["previous_error"] = previous_error + + bot_api_url = _bot_api_url(args) + if bot_api_url: + body["bot_api_url"] = bot_api_url + + # Pass verification metadata so the embodied service can log the full pipeline + body["_verification_meta"] = { + "intent_original": intent, + "policy_layer": "decomposition" if len(sub_intents) > 1 else (policy_result.get("policy_layer") or "normalization"), + "category": category, + "sub_intent_index": idx, + "sub_intents_total": len(sub_intents), + } + + result = _post_intent(body) + + if result.get("ok"): + if result.get("execution_results"): + all_execution_results.extend(result["execution_results"]) + if result.get("plan") and aggregated_plan is None: + aggregated_plan = result["plan"] + previous_error = None + sub_intent_outcomes.append("embodied_succeeded") + else: + previous_error = { + "tool": result.get("error", {}).get("error_type", "unknown"), + "error_type": result.get("error", {}).get("error_type", "other"), + "details": result.get("error", {}).get("details", "unknown error"), + } + all_execution_results.append({ + "ok": False, + "tool": "embodied_plan", + "error_type": previous_error["error_type"], + "details": previous_error["details"], + }) + sub_intent_outcomes.append("embodied_failed") + + return json.dumps({ + "ok": True, + "outcome": "embodied_ready", + "policy_handled": False, + "plan": aggregated_plan, + "execution_results": all_execution_results, + "mitigation": { + "policy_layer": "decomposition" if len(sub_intents) > 1 else "normalization", + "policy_reason": f"processed into {len(sub_intents)} sub-intent(s)", + "intent_original": intent, + "sub_intents_count": len(sub_intents), + "normalized_chain": sub_intents, + "category_chain": categories, + "allowed_tools_chain": allowed_tools_chain, + "sub_intent_outcomes": sub_intent_outcomes, + "strategy": policy_result.get("strategy", "embodied_plan"), + "needs_setup": policy_result.get("needs_setup", False), + }, + }) + + +# --------------------------------------------------------------------------- +# Tool schema +# --------------------------------------------------------------------------- + +EMBODIED_PLAN_SCHEMA = { + "type": "function", + "function": { + "name": "embodied_plan", + "description": ( + "Delegate a body task in Minecraft to the embodied service " + "(Gemma-Andy via Ollama). Use this when the user wants the " + "agent's Minecraft character to DO something — gather, build, " + "fight, navigate, craft, etc. The service handles world-state " + "perception, tool selection, and execution against bot/server.js. " + "You only describe the high-level intent in natural language. " + "DO NOT use granular mc_* tools when this tool is available — " + "this one collapses what would be 5-15 LLM rounds into a single " + "delegation backed by a fine-tuned local model.\n\n" + "USE WHEN:\n" + "- The user asks the bot to do something physical in Minecraft\n" + "- A multi-step body task (gather → craft → place)\n" + "- A movement / navigation request\n" + "- A combat / defensive action\n\n" + "NOT FOR:\n" + "- Conversation, narrative, education (handle yourself)\n" + "- Reading/explaining game state to the user (handle yourself)\n" + "- Tasks outside body orchestration (writing code, web research, etc.)" + ), + "parameters": { + "type": "object", + "properties": { + "intent": { + "type": "string", + "description": ( + "Natural-language description of what the bot should do. " + "Be CONCRETE. Include 'what', 'where', and 'why' when " + "relevant. Examples: 'Help the player gather 12 oak logs " + "before night.' / 'Go to coordinates [120, 64, -33] but " + "avoid the ravine.' / 'Build a small shelter using planks " + "from the inventory.' Ambiguous intents are okay — the " + "embodied service will respond with an ask_clarification " + "tool_call which surfaces a question to ask the user." + ), + }, + "policy_mode": { + "type": "string", + "enum": ["auto", "raw"], + "description": ( + "Policy wrapping mode. 'auto' enables the GemmaPolicy " + "layer (scope filter, ambiguity detection, decomposition, " + "normalization, tool narrowing). 'raw' forwards the intent " + "verbatim for debugging. Default is platform-aware: 'auto' " + "when BOT_API_URL is set (DaemonCraft gateway), 'raw' " + "otherwise (CLI backward compatibility)." + ), + }, + "autonomy_level": { + "type": "integer", + "description": ( + "Guardian autonomy. 0=observer / 1=assistant / " + "2=supervised builder (DEFAULT, safe for kids+adults) / " + "3=autonomous companion / 4=advanced operator (risky)." + ), + "default": 2, + }, + "allowed_tools": { + "type": "array", + "items": {"type": "string"}, + "description": ( + "Optional override of the tool subset Gemma-Andy may use. " + "Names must be canonical v2 tool names. When omitted, the " + "service uses its default safe set. The service further " + "filters by executor_supported, so passing tools the bot " + "server doesn't implement is harmless — they're dropped." + ), + }, + "guardian_constraints": { + "type": "object", + "description": ( + "Optional override of the safety constraints. Recognized " + "fields include no_tnt, no_protected_zone_edit, " + "protected_zone_owner, plus any no_ bool flags. " + "Defaults are sane (no_tnt=true, no_protected_zone_edit=true)." + ), + }, + "previous_error": { + "type": "object", + "description": ( + "Optional. Pass when the previous embodied_plan call's " + "execution_results contained a failure and you want " + "Gemma-Andy to compose a recovery plan. Shape: " + "{tool: , error_type: 'stuck'|'no_path'|'tool_timeout'|" + "'hazard_detected'|'missing_material'|'other', " + "details: }." + ), + }, + "deadline_seconds": { + "type": "integer", + "description": ( + "Wall-clock budget for the WHOLE call (compose + Ollama + " + "dispatch). Default 30. Set higher for long execution " + "sequences." + ), + "default": 30, + }, + }, + "required": ["intent"], + }, + }, +} + + +# --------------------------------------------------------------------------- +# Main handler +# --------------------------------------------------------------------------- + +def _handler(args: dict[str, Any] | None = None, **_kw: Any) -> str: + args = args or {} + + # Auto-cancel any active bot task before sending a new intent. + # Without this, a second embodied_plan call times out because the + # bot still has currentTask.status='running' from the previous + # plan's fire-and-forget tool_calls. See card t_c518b077. + _cancel_bot_task(_bot_api_url(args)) + + policy_mode = args.get("policy_mode") + if policy_mode is None: + policy_mode = _policy_mode_default() + + if policy_mode == "raw": + return _raw_handler(args) + return _policy_handler(args) + + +def _check_service_available() -> bool: + """Light availability check — does NOT call /health (would block tool + discovery on a slow service). The check_fn is invoked at toolset + enumeration time; an unreachable service still lets the tool register + and produce a clean error at call time. We just verify the URL parses.""" + try: + url = _service_url() + return url.startswith("http://") or url.startswith("https://") + except Exception: + return False + + +# --------------------------------------------------------------------------- +# Registration +# --------------------------------------------------------------------------- + +# AST check in tools/registry.py only recognizes `registry.register(...)` +# at module scope, not inside loops or conditionals. +registry.register( + name="embodied_plan", + toolset="embodiment", + schema=EMBODIED_PLAN_SCHEMA, + handler=_handler, + check_fn=_check_service_available, + emoji="🤖", + description=EMBODIED_PLAN_SCHEMA["function"]["description"], +) diff --git a/tools/file_tools.py b/tools/file_tools.py index c0b2fd066287..a37efb193743 100644 --- a/tools/file_tools.py +++ b/tools/file_tools.py @@ -982,18 +982,7 @@ def read_file_tool(path: str, offset: int = 1, limit: int = 500, task_id: str = except Exception: logger.debug("file_state.record_read failed", exc_info=True) - if count >= 4: - # Hard block: stop returning content to break the loop - return json.dumps({ - "error": ( - f"BLOCKED: You have read this exact file region {count} times in a row. " - "The content has NOT changed. You already have this information. " - "STOP re-reading and proceed with your task." - ), - "path": path, - "already_read": count, - }, ensure_ascii=False) - elif count >= 3: + if count >= 3: result_dict["_warning"] = ( f"You have read this exact file region {count} times consecutively. " "The content has not changed since your last read. Use the information you already have. " diff --git a/tools/gemma_policy.py b/tools/gemma_policy.py new file mode 100644 index 000000000000..199d6568f829 --- /dev/null +++ b/tools/gemma_policy.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +"""gemma_policy.py — 5-layer Hermes mitigation policy for Gemma-Andy upstream. + +Pure policy filter: no HTTP calls, no Ollama, no embodied-service dependencies. +Consumes a raw user intent string and returns either: + - policy_handled_upstream (L2 scope cut or L3 ambiguity cut) + - embodied_ready (L5→L1→L4 pipeline result with sub-intents) + +Layers (canonical order): + L2 — scope filter : non-body intents → handled upstream + L3 — ambiguity : vague intents → handled upstream + L5 — decompose : multi-step intents → atomic sub-intents + L1 — normalize : per sub-intent, ES→EN imperative, canonical names + L4 — narrow tools : per sub-intent, classify category, narrow allowed_tools + +Flow: L2 → L3 → L5 → (for each sub) → L1 → L4 + +Reference: Mar-IA-no/deamoncraft-gemma4-andy mitigation/hermes_policy.py +""" +from __future__ import annotations + +import os +import re +from typing import Optional + + +class GemmaPolicy: + # ── L2: Out-of-scope regex ──────────────────────────────── + OUT_OF_SCOPE_REGEX = re.compile( + r"\b(chiste|joke|cantam|hola|chau|buenas|gracias|de nada|" + r"qué pensás|que pensas|qué te parece|que te parece|opinión|opinion|" + r"explicame|explícame|definí|defini|definition|tell me about|" + r"por qué|por que|why does|how does|qué es|que es|" + r"sumar|restar|multiplicar|dividir|cuánto es|cuanto es|2\s*\+\s*2)\b", + re.IGNORECASE, + ) + + # ── L3: Ambiguity tokens ────────────────────────────────── + AMBIGUITY_TOKENS = re.compile( + r"\b(algo entretenido|algo bueno|algo divertido|cualquier cosa|" + r"something good|something fun|whatever|por ahí|por ahi|" + r"hacé algo|hace algo|do something|haz algo|alrededor sin más|" + r"andá por|anda por)\b", + re.IGNORECASE, + ) + + # ── L4: Category keywords (most specific first) ─────────── + CATEGORY_KEYWORDS = [ + ("navigation", ["andá", "anda ", "vení", "veni ", "venite", "follow", "go to", "goto", "alejate", "alejame", "flee", "seguime", "acercate", "come to", "come here", "ven aca", "ven acá", "find and approach", "approach", "find the player", "stop within", "stay within", "move away from", "move away", "move_away", "flee from", "flee_from", "get away from"]), + ("equip", ["equipá", "equipa", "equip", "ponete", "pongate"]), + ("toss", ["tirá", "tira", "toss", "drop", "dejá caer", "deja caer"]), + ("pickup", ["recogé", "recoge", "pickup", "agarrá", "agarra", "levantá", "levanta", "pick up"]), + ("food", ["comé", "comer", "comelo", "eat ", "drink", "bebé", "bebe", "morder", "ingerir"]), + ("memory", ["acordate", "marcá", "marca ", "recordá", "remember", "volvé a", "return to", "olvidá", "forget"]), + ("mining", ["minar", "mine ", "conseguí", "consegui", "gather", "dig "]), + ("build", ["construí", "construye", "construí ", "pongá", "place ", "build ", "make a "]), + ("combat", ["atacá", "ataca", "attack", "defendé", "defend", "raise_shield"]), + ("inventory_query", ["inventario", "inventory", "decime qué tenés", "decime que tenes", "mostrame el inventario", "what do you have", "show inventory"]), + ] + + CATEGORY_TOOLS = { + "navigation": ["scan_nearby", "goto", "follow", "stop_movement", "move_away"], + "mining": ["scan_nearby", "goto", "mine_block", "mine_blocks", "collect_drops", "get_inventory"], + "equip": ["get_inventory", "equip_item"], + "toss": ["get_inventory", "toss_item"], + "pickup": ["scan_nearby", "pickup_item", "get_inventory"], + "inventory_query": ["get_inventory"], + "memory": ["remember_here", "goto_remembered_place", "forget_place", "get_inventory"], + "food": ["consume_food", "get_inventory"], + "build": ["scan_nearby", "goto", "place_block", "equip_item", "get_inventory"], + "combat": ["scan_nearby", "attack_entity", "flee_from", "raise_shield", "consume_food"], + } + COMMON_SAFE = ["ask_clarification", "report_execution_error"] + GUARDIAN_AWARE_CATEGORIES = {"navigation", "combat", "default"} + + # ── Strategy map: category → execution method (benchmark-validated 2026-05-16) ─ + STRATEGY_MAP = { + "navigation": "embodied_plan", + "mining": "embodied_plan", + "equip": "embodied_plan", + "toss": "embodied_plan", + "pickup": "embodied_plan", + "inventory_query": "embodied_plan", + "memory": "embodied_plan", + "food": "embodied_plan", + "build": "embodied_plan", + "combat": "embodied_plan", + "default": "embodied_plan", + } + # Categories that benefit from setup (clear floor, give items) before execution + NEEDS_SETUP = {"build"} + # When Gemma-Andy fails (timeout, clutter, tool_not_implemented), fall back to: + FALLBACK_METHOD = "mc_direct" + + # ── L5: Decomposition ───────────────────────────────────── + DECOMPOSE_CONNECTORS = re.compile( + r"(\s+después\s+|\s+despues\s+|\s+luego\s+|\s+y después\s+|" + r"\s+y luego\s+|\s+y después de\s+|\bthen\b|" + r"(?<=[a-záéíóú])\.\s+(?=[A-ZÁÉÍÓÚ])|^\s*\d+[\.\)])", + re.IGNORECASE, + ) + CONSTRAINT_LEAD = re.compile( + r"^(stop within|stay within|stay near|stay close|stay\s|" + r"avoid|do not|don't|never|while|during|" + r"without|keep|be careful|carefully|" + r"sin (?:hacer|tocar|salir|hurt)|mantente|manténte|" + r"evitando|cuidando|cuidado con)\b", + re.IGNORECASE, + ) + + # ── L1: Verb map ES → EN imperative ─────────────────────── + VERB_MAP = [ + ("acordate de", "Remember"), + ("acordate", "Remember"), + ("recordá", "Remember"), + ("marcá", "Mark"), + ("marca ", "Mark "), + ("volvé a", "Return to"), + ("volvé", "Return"), + ("vuelve a", "Return to"), + ("alejate de", "Move away from"), + ("alejate", "Move away from"), + ("alejame", "Move away from"), + ("andá a", "Go to"), + ("andá", "Go to"), + ("anda a", "Go to"), + ("caminá", "Walk"), + ("caminar", "Walk"), + ("camina ", "Walk "), + ("vení a", "Come to"), + ("vení", "Come to"), + ("venite a", "Come to"), + ("venite", "Come to"), + ("veni a", "Come to"), + ("acercate", "Approach"), + ("comé", "Eat"), + ("comer ", "Eat "), + ("minar", "Mine"), + ("minas", "Mine"), + ("conseguí", "Get"), + ("consegui", "Get"), + ("tirá", "Toss"), + ("tira ", "Toss "), + ("equipá", "Equip"), + ("equipa", "Equip"), + ("recogé", "Pick up"), + ("recoge", "Pick up"), + ("agarrá", "Pick up"), + ("agarra ", "Pick up "), + ("construí", "Build"), + ("construye", "Build"), + ("pongá", "Place"), + ("atacá", "Attack"), + ("ataca ", "Attack "), + ("defendé", "Defend"), + ("seguime", "Follow"), + ("decime qué tenés", "Tell me what you have"), + ("decime que tenes", "Tell me what you have"), + ("mostrame el inventario", "Show your inventory"), + ("hacé", "Do"), + ("hace ", "Do "), + ] + + def __init__(self, player_name: str | None = None, bot_name: str | None = None): + self.player_name = player_name or os.getenv("HERMES_PLAYER_NAME", "player") + self.bot_name = bot_name or os.getenv("HERMES_BOT_NAME", "minecraft_bot") + + # ─── Layer 2 ────────────────────────────────────────────── + def is_out_of_scope(self, intent: str) -> tuple[bool, str | None]: + if not intent: + return False, None + m = self.OUT_OF_SCOPE_REGEX.search(intent) + return (bool(m), m.group(0) if m else None) + + # ─── Layer 3 ────────────────────────────────────────────── + def is_ambiguous(self, intent: str) -> tuple[bool, str | None]: + if not intent: + return False, None + m = self.AMBIGUITY_TOKENS.search(intent) + return (bool(m), m.group(0) if m else None) + + # ─── Layer 4 ────────────────────────────────────────────── + def classify_category(self, intent: str) -> str: + low = (intent or "").lower() + for cat, kws in self.CATEGORY_KEYWORDS: + for kw in kws: + if re.search(rf"\b{re.escape(kw)}", low): + return cat + return "default" + + def get_allowed_tools(self, category: str) -> list[str] | None: + if category == "default": + return None + base = list(self.CATEGORY_TOOLS[category]) + list(self.COMMON_SAFE) + if category in self.GUARDIAN_AWARE_CATEGORIES: + base.append("raise_guardian_event") + return base + + # ─── Layer 5 ────────────────────────────────────────────── + def decompose(self, intent: str) -> list[str]: + """Split intent into atomic sub-intents. + + Conservative: only split when there are sequential PHYSICAL ACTIONS + connected by temporal markers and the following clause starts with + a new action verb (not a constraint/modifier). + + Constraint sub-intents (starting with "stop within", "avoid", etc.) + are merged back into the previous sub-intent. + """ + if not intent: + return [] + if not self.DECOMPOSE_CONNECTORS.search(intent): + return self._try_split_and(intent.strip()) + parts = self.DECOMPOSE_CONNECTORS.split(intent) + CONNECTOR_TOKENS = {"después", "despues", "luego", "then", "y después", "y luego", "y", "y después de"} + atomic_raw = [] + for p in parts: + if not p: + continue + stripped = p.strip(" ,.;").strip() + if not stripped: + continue + low = stripped.lower() + if low in CONNECTOR_TOKENS: + continue + if re.match(r"^\d+[\.\)]?$", stripped): + continue + atomic_raw.append(stripped) + if len(atomic_raw) <= 1: + return self._try_split_and(intent.strip()) + # Constraint detection: merge constraint sub-intents back into the previous one + merged: list[str] = [atomic_raw[0]] + for sub in atomic_raw[1:]: + if self.CONSTRAINT_LEAD.match(sub): + merged[-1] = merged[-1].rstrip(" ,.;") + "; " + sub + else: + merged.append(sub) + if len(merged) > 1: + return merged + return self._try_split_and(intent.strip()) + + def _try_split_and(self, intent: str) -> list[str]: + """Fallback split on ' and ' / ' y ' when both sides look like complete clauses.""" + for connector in (r"\s+and\s+", r"\s+y\s+"): + match = re.search(connector, intent, re.IGNORECASE) + if match: + left = intent[: match.start()].strip() + right = intent[match.end() :].strip() + if len(left.split()) >= 2 and len(right.split()) >= 2: + return [left, right] + return [intent] + + # ─── Layer 1 ────────────────────────────────────────────── + def normalize_surface(self, intent: str) -> str: + n = intent + # Special case: bare "ven/vení/come here" without specific target + bare_come = re.compile( + r"^\s*(ven[ií]?|venite|come)\s*" + r"(aca|acá|aqui|aquí|here|por aqui|por aca)?" + r"[\s,\.!?]*$", + re.IGNORECASE, + ) + bare_approach = re.compile( + r"^\s*acerc[aá]te[\s,\.!?]*$", + re.IGNORECASE, + ) + s = n.strip() + if bare_come.match(s) or bare_approach.match(s): + return f"Follow the player named {self.player_name} and stay within 3 blocks." + # Verb mapping + for es, en in self.VERB_MAP: + n = re.sub(rf"\b{re.escape(es)}\b", en, n, flags=re.IGNORECASE) + # Compact whitespace + n = re.sub(r"\s+", " ", n).strip() + # Pronoun replacements + player_re = re.escape(self.player_name) + n = re.sub(rf"\bdel jugador(?:\s+llamado)?\s+{player_re}\b", f"of the player named {self.player_name}", n, flags=re.IGNORECASE) + n = re.sub(rf"\bal jugador(?:\s+llamado)?\s+{player_re}\b", f"to the player named {self.player_name}", n, flags=re.IGNORECASE) + n = re.sub(rf"\bel jugador(?:\s+llamado)?\s+{player_re}\b", f"the player named {self.player_name}", n, flags=re.IGNORECASE) + n = re.sub(r"\btu posición\b", "your current position", n, flags=re.IGNORECASE) + n = re.sub(r"\bal jugador\b", f"to the player named {self.player_name}", n, flags=re.IGNORECASE) + n = re.sub(r"\bel jugador\b", f"the player named {self.player_name}", n, flags=re.IGNORECASE) + n = re.sub(r"\bdel jugador\b", f"of the player named {self.player_name}", n, flags=re.IGNORECASE) + # Ensure terminal period + if not n.endswith("."): + n = n + "." + # Capitalize first letter + if n and n[0].islower(): + n = n[0].upper() + n[1:] + return n + + # ─── Orchestrator ───────────────────────────────────────── + def execute(self, user_intent: str) -> dict: + """Run the full L2→L3→L5→(L1+L4) pipeline. + + Returns: + dict with keys: + - ok (bool) + - outcome (str): "policy_handled_upstream" or "embodied_ready" + - policy_handled (bool) + - policy_layer (str|None): "scope" or "ambiguity" if cut + - policy_reason (str|None) + - sub_intents (list[str]): normalized atomic intents + - categories (list[str]): per sub-intent + - allowed_tools (list[list[str]|None]): per sub-intent + - execution_results (list): always [] (reserved for downstream) + - plan (None): reserved for downstream + """ + # L2 — scope + oos, reason = self.is_out_of_scope(user_intent) + if oos: + return { + "ok": True, + "outcome": "policy_handled_upstream", + "policy_handled": True, + "policy_layer": "scope", + "policy_reason": f"out_of_scope: matched '{reason}'", + "sub_intents": [], + "categories": [], + "allowed_tools": [], + "execution_results": [], + "plan": None, + } + + # L3 — ambiguity + amb, token = self.is_ambiguous(user_intent) + if amb: + return { + "ok": True, + "outcome": "policy_handled_upstream", + "policy_handled": True, + "policy_layer": "ambiguity", + "policy_reason": f"ambiguous: matched '{token}'; ask user for clarification", + "sub_intents": [], + "categories": [], + "allowed_tools": [], + "execution_results": [], + "plan": None, + } + + # L5 — decompose + raw_subs = self.decompose(user_intent) + + # L1 + L4 per sub-intent + normalized_chain: list[str] = [] + category_chain: list[str] = [] + allowed_chain: list[list[str] | None] = [] + + for sub in raw_subs: + normalized = self.normalize_surface(sub) + normalized_chain.append(normalized) + category = self.classify_category(normalized) + category_chain.append(category) + allowed = self.get_allowed_tools(category) + allowed_chain.append(allowed) + + return { + "ok": True, + "outcome": "embodied_ready", + "policy_handled": False, + "policy_layer": None, + "policy_reason": None, + "sub_intents": normalized_chain, + "categories": category_chain, + "allowed_tools": allowed_chain, + "strategy": self.STRATEGY_MAP.get(category_chain[0] if category_chain else "default", "embodied_plan"), + "needs_setup": any(c in self.NEEDS_SETUP for c in category_chain), + "execution_results": [], + "plan": None, + } diff --git a/tools/kimi_webbridge.py b/tools/kimi_webbridge.py new file mode 100644 index 000000000000..d6dd3768a680 --- /dev/null +++ b/tools/kimi_webbridge.py @@ -0,0 +1,470 @@ +"""Kimi WebBridge integration for Hermes Agent. + +Provides browser automation via the local Kimi WebBridge daemon. Unlike the +built-in ``browser`` toolset (which uses Playwright/Browserbase/Camofox), +this controls the user's REAL browser with their actual login sessions. + +Configuration +------------- +Add to ``~/.hermes/config.yaml``:: + + providers: + kimi_webbridge: + base_url: http://127.0.0.1:10086 + +If omitted, ``base_url`` defaults to ``http://127.0.0.1:10086``. + +The toolset is off by default (``_DEFAULT_OFF_TOOLSETS``) because it requires +the Kimi WebBridge browser extension + daemon to be installed separately. +See https://www.kimi.com/features/webbridge for setup instructions. +""" + +import base64 +import json +import os +from pathlib import Path +from typing import Optional + +import requests + +from tools.registry import registry + +_DEFAULT_DAEMON_URL = "http://127.0.0.1:10086" +_DEFAULT_SESSION = "hermes" +_COMMAND_ENDPOINT = "/command" + + +def _get_daemon_url() -> str: + """Resolve daemon URL from config or fallback.""" + try: + from hermes_cli.config import load_config + cfg = load_config() + url = cfg.get("providers", {}).get("kimi_webbridge", {}).get("base_url") + if url: + return url.rstrip("/") + except Exception: + pass + return _DEFAULT_DAEMON_URL + + +def _bridge_call(action: str, args: Optional[dict] = None, session: Optional[str] = None) -> dict: + """POST a command to the Kimi WebBridge daemon.""" + url = _get_daemon_url() + _COMMAND_ENDPOINT + payload = { + "action": action, + "args": args or {}, + "session": session or _DEFAULT_SESSION, + } + try: + resp = requests.post(url, json=payload, timeout=30) + resp.raise_for_status() + return resp.json() + except requests.RequestException as exc: + return {"error": True, "message": str(exc)} + + +def _check_bridge() -> bool: + """Return True if the Kimi WebBridge daemon is reachable.""" + url = _get_daemon_url() + _COMMAND_ENDPOINT + try: + resp = requests.post( + url, + json={"action": "list_tabs", "args": {}, "session": _DEFAULT_SESSION}, + timeout=3, + ) + return resp.status_code == 200 + except Exception: + return False + + +def _validate_screenshot_path(output_path: Optional[str]) -> Path: + """Ensure screenshot path is safe and within allowed directories.""" + if output_path is None: + return Path(f"/tmp/kimi-webbridge-screenshots/{_DEFAULT_SESSION}_{os.getpid()}.png") + + path = Path(output_path).resolve() + allowed_roots = [ + Path("/tmp").resolve(), + Path.home().resolve(), + ] + if not any(str(path).startswith(str(root)) for root in allowed_roots): + raise ValueError(f"Screenshot path must be under /tmp or home directory, got: {output_path}") + return path + + +# ═══════════════════════════════════════════════════════════════════════════════ +# Tool handlers +# ═══════════════════════════════════════════════════════════════════════════════ + +def kimi_webbridge_navigate( + url: str, + new_tab: bool = True, + group_title: Optional[str] = None, + session: Optional[str] = None, +) -> str: + """Navigate to a URL in the user's real browser.""" + args: dict = {"url": url, "newTab": new_tab} + if group_title: + args["group_title"] = group_title + return json.dumps(_bridge_call("navigate", args, session)) + + +def kimi_webbridge_find_tab( + url: str, + active: bool = False, + session: Optional[str] = None, +) -> str: + """Find and reuse an already-open tab by URL or domain.""" + return json.dumps(_bridge_call("find_tab", {"url": url, "active": active}, session)) + + +def kimi_webbridge_snapshot(session: Optional[str] = None) -> str: + """Get an accessibility tree snapshot of the current page with @e refs.""" + return json.dumps(_bridge_call("snapshot", {}, session)) + + +def kimi_webbridge_click(selector: str, session: Optional[str] = None) -> str: + """Click an element by @e ref or CSS selector.""" + return json.dumps(_bridge_call("click", {"selector": selector}, session)) + + +def kimi_webbridge_fill( + selector: str, + value: str, + session: Optional[str] = None, +) -> str: + """Fill an input, textarea, or contenteditable element. Clears existing content.""" + return json.dumps(_bridge_call("fill", {"selector": selector, "value": value}, session)) + + +def kimi_webbridge_evaluate(code: str, session: Optional[str] = None) -> str: + """Evaluate JavaScript in the current page. Supports async/await.""" + return json.dumps(_bridge_call("evaluate", {"code": code}, session)) + + +def kimi_webbridge_screenshot( + format: str = "png", + quality: int = 90, + selector: Optional[str] = None, + session: Optional[str] = None, +) -> str: + """Take a screenshot. + + Returns a lightweight result. The actual image data is stripped from context + to avoid flooding the token window; use ``kimi_webbridge_save_screenshot`` + when you need the file on disk. + """ + args: dict = {"format": format, "quality": quality} + if selector: + args["selector"] = selector + result = _bridge_call("screenshot", args, session) + inner = result.get("data", {}) if isinstance(result.get("data"), dict) else result + b64 = inner.get("data") if isinstance(inner, dict) else None + if b64 and isinstance(b64, str) and len(b64) > 1000: + inner["data"] = f"" + return json.dumps(result) + + +def kimi_webbridge_save_screenshot( + output_path: Optional[str] = None, + format: str = "png", + quality: int = 90, + session: Optional[str] = None, +) -> str: + """Take a screenshot and save it to disk, returning only the file path.""" + args: dict = {"format": format, "quality": quality} + result = _bridge_call("screenshot", args, session) + inner = result.get("data", {}) if isinstance(result.get("data"), dict) else result + b64 = inner.get("data") if isinstance(inner, dict) else None + if not b64 or not isinstance(b64, str): + return json.dumps({"error": "screenshot failed", "details": result}) + + try: + path = _validate_screenshot_path(output_path) + except ValueError as exc: + return json.dumps({"error": "invalid path", "message": str(exc)}) + + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(base64.b64decode(b64)) + return json.dumps({"success": True, "path": str(path), "size_bytes": path.stat().st_size}) + + +def kimi_webbridge_list_tabs(session: Optional[str] = None) -> str: + """List all tabs in the current session.""" + return json.dumps(_bridge_call("list_tabs", {}, session)) + + +def kimi_webbridge_close_tab(session: Optional[str] = None) -> str: + """Close the current tab.""" + return json.dumps(_bridge_call("close_tab", {}, session)) + + +def kimi_webbridge_close_session(session: Optional[str] = None) -> str: + """Close all tabs in the session. Call at the end of a task.""" + return json.dumps(_bridge_call("close_session", {}, session)) + + +def kimi_webbridge_save_pdf( + paper_format: str = "letter", + landscape: bool = False, + scale: float = 1.0, + print_background: bool = True, + file_name: Optional[str] = None, + session: Optional[str] = None, +) -> str: + """Save the current page as a PDF.""" + args: dict = { + "paper_format": paper_format, + "landscape": landscape, + "scale": scale, + "print_background": print_background, + } + if file_name: + args["file_name"] = file_name + return json.dumps(_bridge_call("save_as_pdf", args, session)) + + +# ═══════════════════════════════════════════════════════════════════════════════ +# Registry +# ═══════════════════════════════════════════════════════════════════════════════ + +_SCHEMA_BASE = { + "type": "object", + "properties": { + "session": {"type": "string", "default": "hermes", "description": "Session name for tab isolation"}, + }, +} + + +def _schema(required: list, extra_props: dict) -> dict: + props = {**_SCHEMA_BASE["properties"], **extra_props} + return {"type": "object", "properties": props, "required": required} + + +registry.register( + name="kimi_webbridge_navigate", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_navigate", + "description": "Navigate the user's real browser to a URL via Kimi WebBridge. Uses the user's actual login sessions.", + "parameters": _schema( + ["url"], + { + "url": {"type": "string", "description": "URL to navigate to"}, + "new_tab": {"type": "boolean", "description": "Open in a new tab", "default": True}, + "group_title": {"type": "string", "description": "Visible label for the tab group"}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_navigate( + url=args["url"], + new_tab=args.get("new_tab", True), + group_title=args.get("group_title"), + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_find_tab", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_find_tab", + "description": "Find and reuse an already-open tab by URL or domain. Use when the user refers to an existing page.", + "parameters": _schema( + ["url"], + { + "url": {"type": "string", "description": "URL or domain to match"}, + "active": {"type": "boolean", "description": "Pick the currently-viewed tab", "default": False}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_find_tab( + url=args["url"], + active=args.get("active", False), + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_snapshot", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_snapshot", + "description": "Get an accessibility tree snapshot of the current page. Returns interactive elements with @e refs for clicking/filling.", + "parameters": _schema([], {}), + }, + handler=lambda args, **kw: kimi_webbridge_snapshot(session=args.get("session")), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_click", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_click", + "description": "Click an element by @e ref or CSS selector. Use @e refs from kimi_webbridge_snapshot when available.", + "parameters": _schema( + ["selector"], + {"selector": {"type": "string", "description": "@e ref (e.g. @e5) or CSS selector"}}, + ), + }, + handler=lambda args, **kw: kimi_webbridge_click( + selector=args["selector"], + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_fill", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_fill", + "description": "Fill an input, textarea, or contenteditable element. Clears existing content before inserting.", + "parameters": _schema( + ["selector", "value"], + { + "selector": {"type": "string", "description": "@e ref or CSS selector"}, + "value": {"type": "string", "description": "Text to insert"}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_fill( + selector=args["selector"], + value=args["value"], + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_evaluate", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_evaluate", + "description": "Evaluate JavaScript in the current page. Supports async/await. Use for scrolling, extracting data, or complex interactions.", + "parameters": _schema( + ["code"], + {"code": {"type": "string", "description": "JavaScript code to run"}}, + ), + }, + handler=lambda args, **kw: kimi_webbridge_evaluate( + code=args["code"], + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_screenshot", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_screenshot", + "description": "Take a screenshot. Returns a lightweight result — base64 data is stripped to avoid context flooding.", + "parameters": _schema( + [], + { + "format": {"type": "string", "enum": ["png", "jpeg"], "default": "png"}, + "quality": {"type": "integer", "default": 90}, + "selector": {"type": "string", "description": "Optional CSS selector or @e ref to capture only that element"}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_screenshot( + format=args.get("format", "png"), + quality=args.get("quality", 90), + selector=args.get("selector"), + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_save_screenshot", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_save_screenshot", + "description": "Take a screenshot and save it to disk. Returns only the file path — safe for context windows.", + "parameters": _schema( + [], + { + "output_path": {"type": "string", "description": "Where to save the image (default: auto-generated in /tmp)"}, + "format": {"type": "string", "enum": ["png", "jpeg"], "default": "png"}, + "quality": {"type": "integer", "default": 90}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_save_screenshot( + output_path=args.get("output_path"), + format=args.get("format", "png"), + quality=args.get("quality", 90), + session=args.get("session"), + ), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_list_tabs", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_list_tabs", + "description": "List all tabs in the current session.", + "parameters": _schema([], {}), + }, + handler=lambda args, **kw: kimi_webbridge_list_tabs(session=args.get("session")), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_close_tab", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_close_tab", + "description": "Close the current tab.", + "parameters": _schema([], {}), + }, + handler=lambda args, **kw: kimi_webbridge_close_tab(session=args.get("session")), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_close_session", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_close_session", + "description": "Close all tabs in the session. Call at the end of a task.", + "parameters": _schema([], {}), + }, + handler=lambda args, **kw: kimi_webbridge_close_session(session=args.get("session")), + check_fn=_check_bridge, +) + +registry.register( + name="kimi_webbridge_save_pdf", + toolset="kimi_webbridge", + schema={ + "name": "kimi_webbridge_save_pdf", + "description": "Save the current page as a PDF.", + "parameters": _schema( + [], + { + "paper_format": {"type": "string", "default": "letter"}, + "landscape": {"type": "boolean", "default": False}, + "scale": {"type": "number", "default": 1.0}, + "print_background": {"type": "boolean", "default": True}, + "file_name": {"type": "string", "description": "Custom filename; defaults to page title"}, + }, + ), + }, + handler=lambda args, **kw: kimi_webbridge_save_pdf( + paper_format=args.get("paper_format", "letter"), + landscape=args.get("landscape", False), + scale=args.get("scale", 1.0), + print_background=args.get("print_background", True), + file_name=args.get("file_name"), + session=args.get("session"), + ), + check_fn=_check_bridge, +) diff --git a/tools/mc_bit_tool.py b/tools/mc_bit_tool.py new file mode 100644 index 000000000000..ec1c4c708578 --- /dev/null +++ b/tools/mc_bit_tool.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 +"""mc_bit — mBit chunk perception for Hermes. + +Queries the bot server's GET /blocks endpoint with mBit visual encoding. +Returns a text-native spatial representation of a Minecraft volume with +1 unique character per block, no symbol collisions, and a legend showing +which block each character represents. + +The visual format distinguishes all 1166 vanilla Minecraft 1.21 blocks: +- yellow_terracotta, brown_terracotta, orange_terracotta, red_terracotta + each get their own character (the old 'full' format collapsed all 16 + terracotta colors to 'T'). +- door types (oak, iron, spruce, etc.) all share '◫' (door = door). +- chest / trapped_chest / ender_chest all share '◰' (chest = chest). +- furnace / blast_furnace / smoker all share '⊡' (furnace = furnace). +- crafting_table / cartography_table / smithing_table / fletching_table / loom + all share '⊞' (crafting = crafting). +- beds (16 colors) all share '⊏' (bed = bed). +- glass types (18) all share '▢' (glass = glass). +- Mnemonic overrides for super-common blocks: air→' ', water→'~', lava→'!', + redstone_wire→'R', torch→'†', lantern→'◊'. +- The remaining ~1090 block names get unique CJK Unified Ideographs + (U+4E00+) assigned alphabetically and deterministically. + +For pathfinding ground truth, use `format='binary'` which gives a 0/1 +walkability grid (0=walkable, 1=solid, Y-major). The server supports +this as a separate endpoint for performance; it does not use the visual +char mapping. + +For quick cardinal clearances, use mc_perceive(type='scene') instead. +For bot state / inventory, use mc_perceive(type='status') or 'nearby'. +""" + +from __future__ import annotations + +from typing import Any + +import httpx + +from tools.bot_api_url_ctx import get_bot_api_url +from tools.registry import registry + + +def _bot_url() -> str: + return get_bot_api_url().rstrip("/") + + +def _missing_required(args: dict[str, Any]) -> list[str]: + return [name for name in ("x1", "y1", "z1", "x2", "y2", "z2") if args.get(name) is None] + + +def _handler(args: dict[str, Any] | None = None, **_kw: Any) -> str: + args = args or {} + missing = _missing_required(args) + if missing: + return ( + "Error: mc_bit requires x1, y1, z1, x2, y2, z2 " + f"(missing: {', '.join(missing)}). Use mc_perceive(type='status') " + "to get bot position first." + ) + + fmt = str(args.get("format", "visual")) + if fmt not in ("visual",): + return ( + f"mc_bit error: unsupported format {fmt!r}. " + "The only supported format is 'visual' (1 unique char per block, no collisions, with legend). " + "For walkability ground truth, parse the visual output — walkable blocks are: ' ' (air), " + "'~' (water), '!' (lava), ',' (short_grass), ';' (tall_grass), '†' (torch), '◊' (lantern), " + "and the CJK chars mapped to other plants/leaves. Or use mc_perceive(type='scene') for cardinal clearances." + ) + + try: + params: dict[str, int | str] = { + "x1": int(args["x1"]), + "y1": int(args["y1"]), + "z1": int(args["z1"]), + "x2": int(args["x2"]), + "y2": int(args["y2"]), + "z2": int(args["z2"]), + "format": fmt, + } + if args.get("cx") is not None: + params["cx"] = int(args["cx"]) + if args.get("cz") is not None: + params["cz"] = int(args["cz"]) + except (TypeError, ValueError) as exc: + return f"mc_bit error: coordinates must be integers ({exc})" + + try: + resp = httpx.get(f"{_bot_url()}/blocks", params=params, timeout=10.0) + resp.raise_for_status() + data = resp.json() + except Exception as exc: + return f"mc_bit error: {exc}" + + if not data.get("ok"): + return f"mc_bit error: {data.get('error', 'unknown')}" + + d = data["data"] + text = d.get("text", "") + count = d.get("count", 0) + elapsed = d.get("elapsed_ms", 0) + return f"mBit {fmt} ({count} blocks, {elapsed}ms):\n{text}" + + +registry.register( + name="mc_bit", + toolset="embodiment", + schema={ + "type": "function", + "function": { + "name": "mc_bit", + "description": ( + "Perceive a 3D chunk of the Minecraft world as text using the mBit 'visual' format. " + "Returns a spatial text representation of blocks in the given volume with 1 unique character per block, " + "no symbol collisions, and a legend at the bottom showing which block each character represents.\n\n" + "The visual format distinguishes all 1166 vanilla Minecraft 1.21 blocks (yellow_terracotta ≠ brown_terracotta ≠ " + "orange_terracotta etc.). Door types share '◫', chest types share '◰', furnace types share '⊡', " + "crafting tables share '⊞', beds share '⊏', glass types share '▢'. Mnemonic chars for super-common blocks: " + "air→' ', water→'~', lava→'!', redstone_wire→'R', torch→'†', lantern→'◊'. The remaining ~1090 block names " + "get unique CJK Unified Ideographs.\n\n" + "Use this ONLY when you need a raw block grid (spatial awareness over a volume) — to understand terrain layout, " + "plan builds, or verify exact block placement before/after acting. For bot state, inventory, nearby entities, " + "chat, or quick status checks, use mc_perceive instead.\n\n" + "Formats:\n" + "- visual: 1 unique char per block with a legend. The only supported format. " + "Best for distinguishing block types (yellow_terracotta ≠ brown_terracotta ≠ orange_terracotta etc.).\n\n" + "Walkable blocks in the visual output: ' ' (air, cave_air, void_air), '~' (water), " + "'!' (lava), ',' (short_grass), ';' (tall_grass), '†' (torch/wall_torch/soul_torch), " + "'◊' (lantern/soul_lantern), and the CJK chars mapped to other plants/leaves. " + "For quick cardinal clearances, use mc_perceive(type='scene').\n\n" + "TIP: scan a small volume (≤8x8x8 = 512 blocks) when you need exact block-level awareness — " + "larger volumes return lots of chars to read." + ), + "parameters": { + "type": "object", + "properties": { + "x1": {"type": "integer", "description": "Min X coordinate"}, + "y1": {"type": "integer", "description": "Min Y coordinate"}, + "z1": {"type": "integer", "description": "Min Z coordinate"}, + "x2": {"type": "integer", "description": "Max X coordinate"}, + "y2": {"type": "integer", "description": "Max Y coordinate"}, + "z2": {"type": "integer", "description": "Max Z coordinate"}, + "format": { + "type": "string", + "enum": ["visual"], + "description": "mBit format. 'visual' is the only supported format (1 unique char per block, no collisions, with legend).", + "default": "visual", + }, + "cx": {"type": "integer", "description": "Center X for context (optional)"}, + "cz": {"type": "integer", "description": "Center Z for context (optional)"}, + }, + "required": ["x1", "y1", "z1", "x2", "y2", "z2"], + }, + }, + }, + handler=_handler, + emoji="🧊", + description="Perceive a 3D Minecraft chunk as text (mBit visual format, no collisions)", +) diff --git a/tools/mc_navigate_tool.py b/tools/mc_navigate_tool.py new file mode 100755 index 000000000000..78048750709f --- /dev/null +++ b/tools/mc_navigate_tool.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +"""mc_navigate — semantic + geometric perception macros for Hermes. + +Calls the bot server's GET /navigate?action=... endpoint. Returns +structured JSON for the LLM to consume (no grid parsing required). + +Actions (11 total): + + SEMANTIC (5) — answer high-level questions: + identify_cave — am I in a cave? (escape direction, sky access) + identify_interior — am I inside a structure? (enriched: doors, safety, furni) + find_doors — list all doors in radius + verify_door — check a specific door's state + scan_structure — full structure context + + GEOMETRIC (5) — answer spatial questions: + walkable — list of cells the bot can stand on + path_to — run pathfinder with timeout + corners — 4 corner cells of walkable area + escape_routes — cardinal directions with distance + blocker + structure_outline — bounding boxes of distinct structures + + EXACT (1) — companion to type-based CJK visual: + verify_block — exact block name at a position (the 10% case) + + LEGEND (1): + visual_legend — canonical block→char mapping (server source of truth) + +Type-based CJK mapping: see SOUL_daemoncraft §Perception Macros. +""" +from __future__ import annotations + +import json +import sys +from typing import Any + +import httpx + + +def _bot_url() -> str: + import os + return ( + os.environ.get("BOT_API_URL") + or os.environ.get("MC_API_URL") + or "http://localhost:3003" + ) + + +def _handler(args: dict[str, Any] | None = None, **_kw: Any) -> str: + args = args or {} + action = args.get("action") + if not action: + return ( + "Error: mc_navigate requires an 'action' parameter. " + "Valid actions: identify_cave, identify_interior, find_doors, " + "verify_door, scan_structure, walkable, path_to, corners, " + "escape_routes, structure_outline, verify_block, visual_legend." + ) + + # Build query params from args (skip action since it goes in path) + params: dict[str, Any] = {"action": action} + for k, v in args.items(): + if k == "action": + continue + params[k] = v + + try: + resp = httpx.get(f"{_bot_url()}/navigate", params=params, timeout=10.0) + resp.raise_for_status() + data = resp.json() + except Exception as exc: + return f"mc_navigate error: {exc}" + + if not data.get("ok"): + return f"mc_navigate error: {data.get('error', 'unknown')}" + + d = data.get("data", {}) + + # Compact summaries for the most common actions + if action == "identify_cave": + lines = ["identify_cave:"] + lines.append(f" is_cave: {d.get('is_cave')}") + if d.get("is_cave"): + lines.append(f" ceiling_height: {d.get('ceiling_height')}") + lines.append(f" has_sky_access: {d.get('has_sky_access')}") + lines.append(f" sky_light: {d.get('sky_light')}") + lines.append(f" exit_direction: {d.get('exit_direction')}") + lines.append(f" escape_tools: {d.get('escape_tools')}") + lines.append(f" depth_blocks: {d.get('depth_blocks')}") + return "\n".join(lines) + + if action == "identify_interior": + lines = ["identify_interior:"] + lines.append(f" is_interior: {d.get('is_interior')}") + lines.append(f" structure_type: {d.get('structure_type')}") + if d.get("is_interior"): + lines.append(f" ceiling_height: {d.get('ceiling_height')}") + lines.append(f" wall_count: {d.get('wall_count')}") + lines.append(f" volume_blocks: {d.get('volume_blocks')}") + ap = d.get("access_points", []) + if ap: + lines.append(f" access_points: {len(ap)} (open={sum(1 for x in ap if x.get('is_open'))})") + else: + lines.append(f" access_points: 0") + mb = d.get("missing_blocks", []) + lines.append(f" missing_blocks: {len(mb)}") + furni = d.get("furni", {}) + if furni: + lines.append(f" furni: {furni}") + lines.append(f" hostile_presence: {d.get('hostile_presence')}") + lines.append(f" is_safe: {d.get('is_safe')}") + issues = d.get("safety_issues", []) + if issues: + lines.append(f" safety_issues: {issues[:5]}") + return "\n".join(lines) + + if action == "find_doors": + doors = d.get("doors", []) + lines = [f"find_doors: {len(doors)} door(s)"] + for door in doors[:5]: + lines.append(f" {door.get('position')}: {door.get('type')} " + f"is_open={door.get('is_open')} blocking={door.get('is_blocking')}") + return "\n".join(lines) + + if action == "verify_door": + return (f"verify_door: {d.get('type')} is_open={d.get('is_open')} " + f"hinge={d.get('hinge_side')} has_top={d.get('has_door_top')}") + + if action == "scan_structure": + return json.dumps(d, indent=2)[:2000] + + if action == "walkable": + cells = d.get("cells", []) + return f"walkable: {len(cells)} cells (capped at 256)" + + if action == "path_to": + lines = ["path_to:"] + lines.append(f" reachable: {d.get('reachable')}") + if d.get("reachable"): + wp = d.get("waypoints", []) + lines.append(f" waypoints: {len(wp)}") + lines.append(f" distance: {d.get('distance', '?')}") + else: + lines.append(f" reason: {d.get('reason', '?')}") + return "\n".join(lines) + + if action == "corners": + corners = d.get("corners", []) + lines = [f"corners: {len(corners)} corner(s)"] + for c in corners[:4]: + lines.append(f" {c}") + return "\n".join(lines) + + if action == "escape_routes": + lines = ["escape_routes:"] + lines.append(f" best_escape: {d.get('best_escape')}") + cards = d.get("cardinals", {}) + for k in ("north", "south", "east", "west", "up", "down"): + v = cards.get(k, {}) + if v: + lines.append(f" {k}: free={v.get('free')} distance={v.get('distance', '?')}") + return "\n".join(lines) + + if action == "structure_outline": + bboxes = d.get("bounding_boxes", []) + return f"structure_outline: {len(bboxes)} structure(s)" + + if action == "verify_block": + return (f"verify_block: {d.get('position')} = {d.get('block')} " + f"(category={d.get('category')}, walkable={d.get('is_walkable')})") + + if action == "visual_legend": + # Compact: char → first block name + mapping = d.get("mapping", {}) + char_to_name = d.get("char_to_names", {}) + lines = [f"visual_legend: {d.get('block_count')} blocks → " + f"{d.get('char_count')} distinct chars"] + for ch, name in list(char_to_name.items())[:8]: + extras = d.get("mapping", {}).get(name, {}).get("+more", 0) + if extras: + lines.append(f" {ch} = {name} (+{extras} more)") + else: + lines.append(f" {ch} = {name}") + if len(char_to_name) > 8: + lines.append(f" ... and {len(char_to_name) - 8} more") + return "\n".join(lines) + + # Default: dump JSON + return json.dumps(d, indent=2)[:3000] + + +# Registry import +try: + from tools.registry import registry +except ImportError: + # Fallback for direct execution / tests + import os + sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) + from hermes_cli.tools import registry + + +registry.register( + name="mc_navigate", + toolset="embodiment", + schema={ + "type": "function", + "function": { + "name": "mc_navigate", + "description": ( + "Semantic + geometric perception macros for the Minecraft world. " + "Returns structured JSON instead of text grids. Use this INSTEAD " + "of parsing mc_bit output for any of these questions:\n\n" + "Semantic (5):\n" + "- identify_cave: am I in a cave?\n" + "- identify_interior: am I inside a structure? (returns access_points, missing_blocks, furni, hostile_presence, is_safe, safety_issues)\n" + "- find_doors: list all doors in radius with is_open state\n" + "- verify_door: check a specific door\n" + "- scan_structure: full structure context (interior + doors + furni + safety)\n\n" + "Geometric (5):\n" + "- walkable: list of cells the bot can stand on\n" + "- path_to: run pathfinder with timeout, returns reachable + waypoints\n" + "- corners: 4 corner cells of walkable area\n" + "- escape_routes: cardinal directions with distance + blocker + best_escape\n" + "- structure_outline: bounding boxes of distinct structures\n\n" + "Exact (1):\n" + "- verify_block: exact block name at a position (10% case where the type-based CJK in mbit isn't enough)\n\n" + "Legend (1):\n" + "- visual_legend: canonical block→char mapping (server source of truth)\n\n" + "Required: action. Optional: x, y, z (anchor position), radius, target_x, target_y, target_z.\n\n" + "See SOUL_daemoncraft §Perception Macros for the decision rule." + ), + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "identify_cave", "identify_interior", "find_doors", + "verify_door", "scan_structure", "walkable", "path_to", + "corners", "escape_routes", "structure_outline", + "verify_block", "visual_legend", + ], + "description": "Which perception macro to run", + }, + "x": {"type": "number", "description": "Anchor X (default: bot position)"}, + "y": {"type": "number", "description": "Anchor Y (default: bot position)"}, + "z": {"type": "number", "description": "Anchor Z (default: bot position)"}, + "radius": {"type": "number", "description": "Scan radius in blocks (default varies by action)"}, + "target_x": {"type": "number", "description": "Pathfinder target X (path_to)"}, + "target_y": {"type": "number", "description": "Pathfinder target Y (path_to)"}, + "target_z": {"type": "number", "description": "Pathfinder target Z (path_to)"}, + }, + "required": ["action"], + }, + }, + }, + handler=_handler, + emoji="🧭", + description="Semantic + geometric perception macros (identify_cave, find_doors, walkable, path_to, etc.)", +) diff --git a/tools/minecraft_tools.py b/tools/minecraft_tools.py new file mode 100644 index 000000000000..2f124043d328 --- /dev/null +++ b/tools/minecraft_tools.py @@ -0,0 +1,2202 @@ +#!/usr/bin/env python3 + +""" +HermesCraft — Embodied Hermes agents for Minecraft + +Copyright (c) 2026 bigph00t + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +""" + +""" +HermesCraft Minecraft Tools — Consolidated Toolset + +Native Hermes toolset that wraps the Mineflayer bot HTTP API. + +Instead of 77 individual mc_* tools (which bloat context window and cause +decision paralysis), this consolidated set exposes 8 high-level tools. +Each tool uses an 'action' or 'type' parameter to route to the correct +bot API endpoint. + +Environment: + MC_API_URL - Bot server URL (default: http://localhost:3001) +""" + +import json +import os +import re +import threading +import urllib.request +import urllib.error +from typing import Any, Dict, Optional + +from tools.registry import registry, tool_error +from tools.bot_api_url_ctx import get_bot_api_url + + +# ═══════════════════════════════════════════════════════════════════════════════ +# Session-scoped endpoint resolution (delegates to neutral bot_api_url_ctx) +# ═══════════════════════════════════════════════════════════════════════════════ + +def _get_bot_api_url(_session_id: Optional[str] = None) -> str: + """Resolve the bot API URL for the current execution context. + + _session_id is ignored — kept for backward compat with legacy callers. + Delegates to bot_api_url_ctx which reads the contextvar set by the + gateway adapter, then falls back to MC_API_URL env var. + """ + return get_bot_api_url() + +# Global cancel event — set by agent_loop.py when chat arrives during a turn +_cancel_event: Optional[threading.Event] = None + + +def set_cancel_event(event: Optional[threading.Event]): + """Wire the cancel event from agent_loop so tool calls can be interrupted mid-flight.""" + global _cancel_event + _cancel_event = event + + +def _api_get(path: str, timeout: int = 15, session_id: Optional[str] = None) -> dict: + url = f"{_get_bot_api_url()}{path}" + try: + with urllib.request.urlopen(url, timeout=timeout) as resp: + return json.loads(resp.read().decode("utf-8")) + except urllib.error.HTTPError as e: + try: + body = json.loads(e.read().decode("utf-8")) + return body + except Exception: + return {"ok": False, "error": f"Bot server error: {e.code} {e.reason}"} + except urllib.error.URLError as e: + return {"ok": False, "error": f"Bot server not responding at {_get_bot_api_url(session_id)}: {e}"} + except Exception as e: + return {"ok": False, "error": str(e)} + + +def _cancel_bot_action(session_id: Optional[str] = None): + """Tell the bot server to stop whatever it's doing (mining, moving, etc.).""" + try: + req = urllib.request.Request( + f"{_get_bot_api_url(session_id)}/task/cancel", + data=b"{}", + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(req, timeout=5) as resp: + pass + except Exception: + pass + + +def _api_post(path: str, data: Optional[dict] = None, timeout: int = 300, session_id: Optional[str] = None) -> dict: + """POST to the bot server. Runs in a thread so it can be cancelled mid-flight.""" + url = f"{_get_bot_api_url()}{path}" + payload = json.dumps(data or {}).encode("utf-8") + req = urllib.request.Request(url, data=payload, headers={"Content-Type": "application/json"}, method="POST") + + result_container: dict = {} + exception_container: dict = {} + + def do_request(): + try: + with urllib.request.urlopen(req, timeout=timeout) as resp: + result_container["result"] = json.loads(resp.read().decode("utf-8")) + except Exception as e: + exception_container["error"] = e + + t = threading.Thread(target=do_request) + t.start() + + # Poll every 0.5s — if cancel_event fires, abort the server action and return + poll_interval = 0.5 + elapsed = 0.0 + while t.is_alive() and elapsed < timeout: + t.join(timeout=poll_interval) + elapsed += poll_interval + if _cancel_event is not None and _cancel_event.is_set(): + _cancel_bot_action(session_id=session_id) + return {"ok": False, "error": "Interrupted by new chat message — action cancelled."} + + if t.is_alive(): + # Still running after timeout — abandon it + return {"ok": False, "error": f"Request timed out after {timeout}s"} + + if "error" in exception_container: + e = exception_container["error"] + if isinstance(e, urllib.error.HTTPError): + try: + body = json.loads(e.read().decode("utf-8")) + return body + except Exception: + return {"ok": False, "error": f"Bot server error: {e.code} {e.reason}"} + elif isinstance(e, urllib.error.URLError): + return {"ok": False, "error": f"Bot server not responding at {_get_bot_api_url(session_id)}: {e}"} + else: + return {"ok": False, "error": str(e)} + + return result_container.get("result", {}) + + +def _fmt(resp: dict) -> str: + if not resp.get("ok", True): + return f"Error: {resp.get('error', 'Unknown error')}" + parts = [] + if "result" in resp: + parts.append(f"Result: {resp['result']}") + # Include judge if present (before state — agent needs to see outcome first) + if "_judge" in resp and isinstance(resp["_judge"], dict): + j = resp["_judge"] + j_parts = [f"Judge: {j.get('outcome', '?')} ({j.get('confidence', '?')})"] + if j.get("reason_code"): + j_parts.append(f"[{j['reason_code']}]") + delta = j.get("position_delta") + if delta: + j_parts.append(f"Delta: dx={delta.get('dx',0):+.1f}, dy={delta.get('dy',0):+.1f}, dz={delta.get('dz',0):+.1f}") + if j.get("error"): + j_parts.append(f"Error: {j['error']}") + parts.append(" ".join(j_parts)) + if "task_id" in resp: + parts.append(f"Task {resp['task_id']} started ({resp.get('status', 'running')})") + if "task" in resp and isinstance(resp.get("task"), dict): + t = resp["task"] + parts.append(f"Task: {t.get('action')} | status: {t.get('status')} | elapsed: {t.get('elapsed_s', '?')}s") + if t.get("error"): + parts.append(f"Task error: {t['error']}") + state = resp.get("state") + if state: + for k, v in state.items(): + if k not in ("new_chat", "task"): + parts.append(f"{k}: {v}") + data = resp.get("data") + if data and isinstance(data, dict): + if "summary" in data: + parts.append(data["summary"]) + elif "messages" in data: + for m in data["messages"][-10:]: + w = " [whisper]" if m.get("whisper") else "" + parts.append(f"<{m['from']}> {m['message']}{w}") + elif "map" in data: + parts.append(data["map"]) + parts.append(f"Center: {data.get('center', '?')} Scale: {data.get('scale', '?')}") + else: + for k, v in list(data.items())[:15]: + parts.append(f"{k}: {v}") + if "locations" in resp: + for loc in resp["locations"][:10]: + parts.append(f" ({loc.get('x', '?')}, {loc.get('y', '?')}, {loc.get('z', '?')}) — {loc.get('distance', '?')}m") + return "\n".join(parts) if parts else json.dumps(resp, indent=2) + + +def check_minecraft_available() -> bool: + try: + result = _api_get("/health", timeout=3) + return result.get("ok", False) or result.get("status") == "ok" + except Exception: + return False + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 1. mc_perceive — Observation and state gathering +# ═══════════════════════════════════════════════════════════════════════════════ + +_PERCEIVE_GET_ENDPOINTS = { + "status": "/status", + "inventory": "/inventory", + "nearby": "/nearby", + "look": "/look", + "scene": "/scene", + "screenshot": "/screenshot", + "map": "/map", + "read_chat": "/chat", + "overhear": "/overhear", + "sounds": "/sounds", + "stats": "/stats", + "health": "/health", + "deaths": "/deaths", + "commands": "/commands", + "furnaces": "/furnaces", + "task_status": "/task", + "social": "/social", +} + +_PERCEIVE_POST_ENDPOINTS = { + "team_status": "/action/team_status", + "report": "/action/report", + "fair_play": "/action/set_fair_play", +} + + +def _handle_mc_perceive(args: dict, **kwargs) -> str: + """Observe the Minecraft world: status, inventory, surroundings, chat, etc.""" + ptype = args.get("type", "status") + + if ptype in _PERCEIVE_GET_ENDPOINTS: + path = _PERCEIVE_GET_ENDPOINTS[ptype] + if ptype == "nearby": + path += f'?radius={args.get("radius", 32)}' + elif ptype == "scene": + path += f'?range={args.get("range", 16)}' + elif ptype == "map": + path += f'?radius={args.get("radius", 16)}' + elif ptype in ("read_chat", "overhear"): + path += f'?count={args.get("count", 20)}' + elif ptype == "screenshot": + w = args.get("width", 1280) + h = args.get("height", 720) + path += f'?width={w}&height={h}' + return _fmt(_api_get(path)) + + if ptype in _PERCEIVE_POST_ENDPOINTS: + endpoint = _PERCEIVE_POST_ENDPOINTS[ptype] + payload = {} + if ptype == "report": + if "message" not in args: + return "Error: message is required for report" + payload["message"] = args["message"] + elif ptype == "fair_play": + payload["enabled"] = args.get("enabled", True) + return _fmt(_api_post(endpoint, payload)) + + return f"Error: unknown perceive type '{ptype}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 2. mc_move — Navigation and locomotion +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_move(args: dict, **kwargs) -> str: + """Move the bot: goto coordinates, follow a player, stop, etc.""" + action = args.get("action", "stop") + payload: Dict[str, Any] = {} + + if action == "goto": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for goto" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/goto", payload)) + + if action == "goto_near": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for goto_near" + payload = {"x": args["x"], "y": args["y"], "z": args["z"], "range": args.get("range", 2)} + return _fmt(_api_post("/action/goto_near", payload)) + + if action == "follow": + if "player" not in args: + return "Error: player is required for follow" + return _fmt(_api_post("/action/follow", {"player": args["player"]})) + + if action == "stop": + return _fmt(_api_post("/action/stop")) + + if action == "deathpoint": + return _fmt(_api_post("/action/deathpoint")) + + return f"Error: unknown move action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 3. mc_mine — Resource gathering and block interaction +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_mine(args: dict, **kwargs) -> str: + """Mine, dig, collect, and find resources in the world.""" + action = args.get("action", "pickup") + payload: Dict[str, Any] = {} + + if action == "collect": + if "block" not in args: + return "Error: block is required for collect" + payload = {"block": args["block"], "count": args.get("count", 1)} + return _fmt(_api_post("/action/collect", payload)) + + if action == "dig": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for dig" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/dig", payload)) + + if action == "pickup": + return _fmt(_api_post("/action/pickup")) + + if action == "find_blocks": + if "block" not in args: + return "Error: block is required for find_blocks" + payload = {"block": args["block"], "radius": args.get("radius", 32), "count": args.get("count", 10)} + return _fmt(_api_post("/action/find_blocks", payload)) + + if action == "find_entities": + payload = {"radius": args.get("radius", 32)} + if args.get("type"): + payload["type"] = args["type"] + return _fmt(_api_post("/action/find_entities", payload)) + + return f"Error: unknown mine action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 4. mc_build — Construction, placement, and block interaction +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_build(args: dict, **kwargs) -> str: + """Build, place blocks, fill areas, interact with blocks, and utility actions.""" + action = args.get("action", "use") + payload: Dict[str, Any] = {} + + if action == "place": + if "block" not in args: + return "Error: block is required for place" + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for place" + payload = {"block": args["block"], "x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/place", payload)) + + if action == "fill": + if "block" not in args: + return "Error: block is required for fill" + for coord in ("x1", "y1", "z1", "x2", "y2", "z2"): + if coord not in args: + return f"Error: {coord} is required for fill" + payload = { + "block": args["block"], + "x1": args["x1"], "y1": args["y1"], "z1": args["z1"], + "x2": args["x2"], "y2": args["y2"], "z2": args["z2"], + "hollow": args.get("hollow", False), + } + return _fmt(_api_post("/action/place_fill", payload)) + + if action == "interact": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for interact" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/interact", payload)) + + if action == "till": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for till" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/till", payload)) + + if action == "bonemeal": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for bonemeal" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/bonemeal", payload)) + + if action == "flatten": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for flatten" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/flatten", payload)) + + if action == "ignite": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for ignite" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/ignite", payload)) + + if action == "fish": + return _fmt(_api_post("/action/fish")) + + if action == "close": + return _fmt(_api_post("/action/close_screen")) + + if action == "use": + return _fmt(_api_post("/action/use")) + + if action == "toss": + if "item" not in args: + return "Error: item is required for toss" + payload = {"item": args["item"]} + if args.get("count") is not None: + payload["count"] = args["count"] + return _fmt(_api_post("/action/toss", payload)) + + if action == "sleep": + return _fmt(_api_post("/action/sleep_bed")) + + if action == "wait": + payload = {"seconds": args.get("seconds", 5)} + return _fmt(_api_post("/action/wait", payload)) + + if action == "connect": + return _fmt(_api_post("/connect")) + + return f"Error: unknown build action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 5. mc_craft — Crafting, smelting, and recipes +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_craft(args: dict, **kwargs) -> str: + """Craft items, look up recipes, and manage furnaces.""" + action = args.get("action", "craft") + payload: Dict[str, Any] = {} + + if action == "craft": + if "item" not in args: + return "Error: item is required for craft" + payload = {"item": args["item"], "count": args.get("count", 1)} + return _fmt(_api_post("/action/craft", payload)) + + if action == "recipes": + if "item" not in args: + return "Error: item is required for recipes" + payload = {"item": args["item"]} + return _fmt(_api_post("/action/recipes", payload)) + + if action == "smelt": + if "input" not in args: + return "Error: input is required for smelt" + payload = {"input": args["input"], "count": args.get("count", 1)} + if args.get("fuel"): + payload["fuel"] = args["fuel"] + return _fmt(_api_post("/action/smelt", payload)) + + if action == "smelt_start": + if "input" not in args: + return "Error: input is required for smelt_start" + payload = {"input": args["input"], "count": args.get("count", 1)} + if args.get("fuel"): + payload["fuel"] = args["fuel"] + return _fmt(_api_post("/action/smelt_start", payload)) + + if action in ("furnace_check", "furnace_take"): + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for {action}" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + endpoint = "/action/furnace_check" if action == "furnace_check" else "/action/furnace_take" + return _fmt(_api_post(endpoint, payload)) + + return f"Error: unknown craft action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 6. mc_combat — Combat, equipment, and survival actions +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_combat(args: dict, **kwargs) -> str: + """Fight, flee, equip gear, eat, and execute combat maneuvers.""" + action = args.get("action", "eat") + payload: Dict[str, Any] = {} + + if action == "attack": + payload = {} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/attack", payload)) + + if action == "fight": + payload = {"retreat_health": args.get("retreat_health", 6), "duration": args.get("duration", 30)} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/fight", payload)) + + if action == "flee": + payload = {"distance": args.get("distance", 16)} + return _fmt(_api_post("/action/flee", payload)) + + if action == "eat": + return _fmt(_api_post("/action/eat")) + + if action == "equip": + if "item" not in args: + return "Error: item is required for equip" + payload = {"item": args["item"], "slot": args.get("slot", "hand")} + return _fmt(_api_post("/action/equip", payload)) + + if action == "sneak": + payload = {"enable": args.get("enable", True)} + return _fmt(_api_post("/action/sneak", payload)) + + if action == "shield": + payload = {"duration": args.get("duration", 3)} + return _fmt(_api_post("/action/shield_block", payload)) + + if action == "shoot": + payload = {"predict": args.get("predict", True)} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/shoot", payload)) + + if action == "sprint_attack": + payload = {} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/sprint_attack", payload)) + + if action == "crit": + payload = {} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/critical_hit", payload)) + + if action == "strafe": + payload = {"direction": args.get("direction", "random"), "duration": args.get("duration", 5)} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/strafe", payload)) + + if action == "combo": + payload = {"style": args.get("style", "aggressive")} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/action/combo", payload)) + + return f"Error: unknown combat action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 7. mc_chat — Communication and team coordination +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_chat(args: dict, **kwargs) -> str: + """Send messages: public chat, whispers, team chat, rally points, etc.""" + action = args.get("action", "chat") + payload: Dict[str, Any] = {} + + if action == "chat": + if "message" not in args: + return "Error: message is required for chat" + return _fmt(_api_post("/action/chat", {"message": args["message"]})) + + if action == "whisper": + if "player" not in args or "message" not in args: + return "Error: player and message are required for whisper" + return _fmt(_api_post("/action/whisper", {"player": args["player"], "message": args["message"]})) + + if action == "chat_to": + if "player" not in args or "message" not in args: + return "Error: player and message are required for chat_to" + return _fmt(_api_post("/action/chat_to", {"player": args["player"], "message": args["message"]})) + + if action == "team_chat": + if "message" not in args: + return "Error: message is required for team_chat" + return _fmt(_api_post("/action/team_chat", {"message": args["message"]})) + + if action == "rally": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for rally" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + if args.get("message"): + payload["message"] = args["message"] + return _fmt(_api_post("/action/rally", payload)) + + if action == "set_team": + if "team" not in args: + return "Error: team is required for set_team" + payload = {"team": args["team"], "role": args.get("role", "warrior")} + if args.get("teammates"): + payload["teammates"] = args["teammates"].split(",") + return _fmt(_api_post("/action/set_team", payload)) + + if action == "complete_command": + payload = {"index": args.get("index", 0)} + return _fmt(_api_post("/action/complete_command", payload)) + + return f"Error: unknown chat action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# 8. mc_manage — Containers, waypoints, and background tasks +# ═══════════════════════════════════════════════════════════════════════════════ + +def _handle_mc_manage(args: dict, **kwargs) -> str: + """Manage containers, saved locations, and background tasks.""" + action = args.get("action", "marks") + payload: Dict[str, Any] = {} + + if action == "chest": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for chest" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/action/list_container", payload)) + + if action == "deposit": + if "item" not in args: + return "Error: item is required for deposit" + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for deposit" + payload = {"item": args["item"], "x": args["x"], "y": args["y"], "z": args["z"], "count": args.get("count", 0)} + return _fmt(_api_post("/action/deposit", payload)) + + if action == "withdraw": + if "item" not in args: + return "Error: item is required for withdraw" + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for withdraw" + payload = {"item": args["item"], "x": args["x"], "y": args["y"], "z": args["z"], "count": args.get("count", 0)} + return _fmt(_api_post("/action/withdraw", payload)) + + if action == "mark": + if "name" not in args: + return "Error: name is required for mark" + payload = {"name": args["name"], "note": args.get("note", "")} + return _fmt(_api_post("/action/mark", payload)) + + if action == "marks": + return _fmt(_api_post("/action/marks")) + + if action == "go_mark": + if "name" not in args: + return "Error: name is required for go_mark" + return _fmt(_api_post("/action/go_mark", {"name": args["name"]})) + + if action == "unmark": + if "name" not in args: + return "Error: name is required for unmark" + return _fmt(_api_post("/action/unmark", {"name": args["name"]})) + + if action == "bg_goto": + for coord in ("x", "y", "z"): + if coord not in args: + return f"Error: {coord} is required for bg_goto" + payload = {"x": args["x"], "y": args["y"], "z": args["z"]} + return _fmt(_api_post("/task/goto", payload)) + + if action == "bg_collect": + if "block" not in args: + return "Error: block is required for bg_collect" + payload = {"block": args["block"], "count": args.get("count", 1)} + return _fmt(_api_post("/task/collect", payload)) + + if action == "bg_fight": + payload = {"retreat_health": args.get("retreat_health", 6), "duration": args.get("duration", 30)} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/task/fight", payload)) + + if action == "bg_combo": + payload = {"style": args.get("style", "aggressive")} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/task/combo", payload)) + + if action == "bg_strafe": + payload = {"direction": args.get("direction", "random"), "duration": args.get("duration", 10)} + if args.get("target"): + payload["target"] = args["target"] + return _fmt(_api_post("/task/strafe", payload)) + + if action == "cancel": + return _fmt(_api_post("/task/cancel")) + + if action == "task_status": + return _fmt(_api_get("/task")) + + return f"Error: unknown manage action '{action}'" + + +# ═══════════════════════════════════════════════════════════════════════════════ +# Tool Schemas +# ═══════════════════════════════════════════════════════════════════════════════ + +MC_PERCEIVE_SCHEMA = { + "name": "mc_perceive", + "description": "Observe the Minecraft world. Use 'status' for full state, 'inventory' for items, 'nearby' for blocks/entities, 'look' for a narrative description, 'scene' for fair-play view, 'map' for ASCII top-down, 'read_chat' for recent messages, 'social' for interaction summary, 'sounds' for audio events, 'health' for quick vitals, 'deaths' for death log, 'commands' for pending orders, 'furnaces' for active furnaces, 'task_status' for background tasks, 'team_status' for teammates, 'report' to send intel, 'fair_play' to toggle fairness mode.", + "parameters": { + "type": "object", + "properties": { + "type": { + "type": "string", + "enum": ["status", "inventory", "nearby", "look", "scene", "map", "read_chat", "overhear", "sounds", "stats", "health", "deaths", "commands", "furnaces", "task_status", "social", "team_status", "report", "fair_play"], + "description": "What to observe", + }, + "radius": {"type": "number", "description": "Scan radius for nearby/map"}, + "range": {"type": "number", "description": "View range for scene"}, + "count": {"type": "number", "description": "Message count for read_chat/overhear"}, + "message": {"type": "string", "description": "Intel message for report action"}, + "enabled": {"type": "boolean", "description": "Toggle fair play mode on/off"}, + }, + "required": ["type"], + }, +} + +MC_MOVE_SCHEMA = { + "name": "mc_move", + "description": "Navigate the bot. 'goto' walks to exact coordinates. 'goto_near' stops within a range. 'follow' trails a player. 'stop' halts all movement. 'deathpoint' returns to last death location.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["goto", "goto_near", "follow", "stop", "deathpoint"], + "description": "Movement action", + }, + "x": {"type": "number", "description": "X coordinate"}, + "y": {"type": "number", "description": "Y coordinate"}, + "z": {"type": "number", "description": "Z coordinate"}, + "player": {"type": "string", "description": "Player name to follow"}, + "range": {"type": "number", "description": "Acceptable distance for goto_near"}, + }, + "required": ["action"], + }, +} + +MC_MINE_SCHEMA = { + "name": "mc_mine", + "description": "Gather resources. 'collect' mines N blocks of a type. 'dig' breaks a specific block. 'pickup' grabs nearby drops. 'find_blocks' locates block positions. 'find_entities' scans for mobs/players.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["collect", "dig", "pickup", "find_blocks", "find_entities"], + "description": "Mining action", + }, + "block": {"type": "string", "description": "Block type (e.g. oak_log, iron_ore)"}, + "x": {"type": "number", "description": "X coordinate for dig"}, + "y": {"type": "number", "description": "Y coordinate for dig"}, + "z": {"type": "number", "description": "Z coordinate for dig"}, + "count": {"type": "number", "description": "How many blocks to mine or max results"}, + "radius": {"type": "number", "description": "Search radius"}, + "entity_type": {"type": "string", "description": "Entity filter for find_entities"}, + }, + "required": ["action"], + }, +} + +MC_BUILD_SCHEMA = { + "name": "mc_build", + "description": "Build and interact with the world. 'place' a single block. 'fill' a volume. 'interact' right-clicks a block (chests, doors, furnaces). 'till' hoes grass_block/dirt into farmland. 'bonemeal' grows crops/saplings. 'flatten' shovels grass/dirt into dirt_path. 'ignite' lights netherrack/TNT/campfires with flint_and_steel. 'fish' casts a fishing rod. 'close' any open screen. 'use' activates held item. 'toss' drops items. 'sleep' finds a bed. 'wait' pauses. 'connect' reconnects the bot.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["place", "fill", "interact", "till", "bonemeal", "flatten", "ignite", "fish", "close", "use", "toss", "sleep", "wait", "connect"], + "description": "Build/interaction action", + }, + "block": {"type": "string", "description": "Block type for place/fill"}, + "x": {"type": "number"}, "y": {"type": "number"}, "z": {"type": "number"}, + "x1": {"type": "number"}, "y1": {"type": "number"}, "z1": {"type": "number"}, + "x2": {"type": "number"}, "y2": {"type": "number"}, "z2": {"type": "number"}, + "hollow": {"type": "boolean", "description": "Fill hollow for fill action"}, + "item": {"type": "string", "description": "Item for toss"}, + "count": {"type": "number", "description": "Item count for toss"}, + "seconds": {"type": "number", "description": "Seconds to wait"}, + }, + "required": ["action"], + }, +} + +MC_CRAFT_SCHEMA = { + "name": "mc_craft", + "description": "Craft items and manage furnaces. 'craft' creates an item. 'recipes' looks up requirements. 'smelt' cooks in furnace and waits. 'smelt_start' loads furnace and leaves. 'furnace_check' inspects a furnace. 'furnace_take' collects output.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["craft", "recipes", "smelt", "smelt_start", "furnace_check", "furnace_take"], + "description": "Crafting action", + }, + "item": {"type": "string", "description": "Item name for craft/recipes"}, + "input": {"type": "string", "description": "Input material for smelting"}, + "fuel": {"type": "string", "description": "Fuel for smelting (optional)"}, + "count": {"type": "number", "description": "Quantity"}, + "x": {"type": "number"}, "y": {"type": "number"}, "z": {"type": "number"}, + }, + "required": ["action"], + }, +} + +MC_COMBAT_SCHEMA = { + "name": "mc_combat", + "description": "Combat and survival. 'attack' a target. 'fight' sustained combat with retreat threshold. 'flee' from hostiles. 'eat' best food. 'equip' an item. 'sneak' toggle. 'shield' block. 'shoot' bow. 'sprint_attack' for knockback. 'crit' for jump-attack. 'strafe' while fighting. 'combo' executes a style sequence.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["attack", "fight", "flee", "eat", "equip", "sneak", "shield", "shoot", "sprint_attack", "crit", "strafe", "combo"], + "description": "Combat action", + }, + "target": {"type": "string", "description": "Target mob or player"}, + "retreat_health": {"type": "number", "description": "HP threshold to retreat during fight"}, + "duration": {"type": "number", "description": "Duration in seconds for fight/strafe"}, + "distance": {"type": "number", "description": "Flee distance"}, + "item": {"type": "string", "description": "Item to equip"}, + "slot": {"type": "string", "description": "Equipment slot (hand, head, chest, legs, feet, off-hand)"}, + "enable": {"type": "boolean", "description": "Enable/disable sneak"}, + "predict": {"type": "boolean", "description": "Predict target movement for shoot"}, + "direction": {"type": "string", "description": "Strafe direction: left, right, random"}, + "style": {"type": "string", "description": "Combo style: aggressive, defensive, balanced"}, + }, + "required": ["action"], + }, +} + +MC_CHAT_SCHEMA = { + "name": "mc_chat", + "description": "Communication. 'chat' public message. 'whisper' private to one player. 'chat_to' alternative private message. 'team_chat' to teammates. 'rally' sets a team rally point. 'set_team' assigns team/role. 'complete_command' marks a pending order done.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["chat", "whisper", "chat_to", "team_chat", "rally", "set_team", "complete_command"], + "description": "Chat action", + }, + "message": {"type": "string", "description": "Message content"}, + "player": {"type": "string", "description": "Target player for whisper/chat_to"}, + "x": {"type": "number"}, "y": {"type": "number"}, "z": {"type": "number"}, + "team": {"type": "string", "description": "Team name for set_team"}, + "role": {"type": "string", "description": "Role for set_team (default: warrior)"}, + "teammates": {"type": "string", "description": "Comma-separated teammate names for set_team"}, + "index": {"type": "number", "description": "Command index to complete"}, + }, + "required": ["action"], + }, +} + +MC_MANAGE_SCHEMA = { + "name": "mc_manage", + "description": "Manage containers, waypoints, and background tasks. 'chest' lists contents. 'deposit'/'withdraw' items. 'mark' saves current location. 'marks' lists waypoints. 'go_mark' navigates to one. 'unmark' deletes. 'bg_goto'/'bg_collect'/'bg_fight' background tasks. 'bg_combo'/'bg_strafe' background combat. 'cancel' stops background task. 'task_status' checks progress.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["chest", "deposit", "withdraw", "mark", "marks", "go_mark", "unmark", "bg_goto", "bg_collect", "bg_fight", "bg_combo", "bg_strafe", "cancel", "task_status"], + "description": "Management action", + }, + "item": {"type": "string", "description": "Item name for deposit/withdraw"}, + "x": {"type": "number"}, "y": {"type": "number"}, "z": {"type": "number"}, + "count": {"type": "number", "description": "Item count for deposit/withdraw or block count for bg_collect"}, + "name": {"type": "string", "description": "Waypoint name for mark/go_mark/unmark"}, + "note": {"type": "string", "description": "Optional note for mark"}, + "block": {"type": "string", "description": "Block type for bg_collect"}, + "target": {"type": "string", "description": "Target for bg_fight/bg_combo/bg_strafe"}, + "retreat_health": {"type": "number"}, + "duration": {"type": "number"}, + "style": {"type": "string", "description": "Combo style for bg_combo"}, + "direction": {"type": "string", "description": "Strafe direction for bg_strafe"}, + }, + "required": ["action"], + }, +} + + +# ═══════════════════════════════════════════════════════════════════ +# 9. mc_plan — Persistent goal & task planning +# ═══════════════════════════════════════════════════════════════════ + +def _handle_mc_plan(args: dict, **kwargs) -> str: + """Manage persistent goals and tasks. Bots use this to remember multi-step projects across turns.""" + action = args.get("action", "get_plan") + payload: Dict[str, Any] = {} + + if action == "set_goal": + if "goal" not in args: + return "Error: goal is required for set_goal" + payload = { + "action": "set_goal", + "goal": args["goal"], + "tasks": args.get("tasks", []), + } + return _fmt(_api_post("/action/plan", payload)) + + if action == "get_plan": + return _fmt(_api_post("/action/plan", {"action": "get_plan"})) + + if action == "update_task": + if "task_id" not in args: + return "Error: task_id is required for update_task" + payload = { + "action": "update_task", + "task_id": args["task_id"], + "status": args.get("status"), + "result": args.get("result"), + "attempt": args.get("attempt"), + } + return _fmt(_api_post("/action/plan", payload)) + + if action == "add_task": + if "goal" not in args: + return "Error: goal (task description) is required for add_task" + payload = { + "action": "add_task", + "goal": args["goal"], + "status": args.get("status", "pending"), + } + return _fmt(_api_post("/action/plan", payload)) + + if action == "remove_task": + if "task_id" not in args: + return "Error: task_id is required for remove_task" + payload = { + "action": "remove_task", + "task_id": args["task_id"], + } + return _fmt(_api_post("/action/plan", payload)) + + if action == "clear_goal": + return _fmt(_api_post("/action/plan", {"action": "clear_goal"})) + + return f"Error: unknown plan action '{action}'" + + +MC_PLAN_SCHEMA = { + "name": "mc_plan", + "description": "Persistent goal and task management. Use this to plan multi-step projects that survive across turns. 'set_goal' creates a goal with tasks. 'get_plan' reads current progress. 'update_task' marks tasks done/in_progress/blocked. 'add_task' appends a task. 'remove_task' deletes one. 'clear_goal' resets everything.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["set_goal", "get_plan", "update_task", "add_task", "remove_task", "clear_goal"], + "description": "Planning action", + }, + "goal": {"type": "string", "description": "Goal description (for set_goal) or task description (for add_task)"}, + "tasks": { + "type": "array", + "description": "List of tasks for set_goal", + "items": { + "type": "object", + "properties": { + "description": {"type": "string"}, + "status": {"type": "string", "enum": ["pending", "in_progress", "done", "blocked"]}, + "attempts": {"type": "number"}, + }, + }, + }, + "task_id": {"type": "number", "description": "Zero-based task index for update/remove"}, + "status": {"type": "string", "enum": ["pending", "in_progress", "done", "blocked"], "description": "New status for update_task"}, + "result": {"type": "string", "description": "Optional result note for update_task"}, + "attempt": {"type": "boolean", "description": "If true, increments attempt counter for update_task"}, + }, + "required": ["action"], + }, +} + + +# ═══════════════════════════════════════════════════════════════════ +# 9b. mc_plan_decompose — Fase 3 strategic plan decomposition (Hermes LLM) +# ═══════════════════════════════════════════════════════════════════ +# +# Hermes (Steve / cloud LLM) calls this tool to break a high-level goal into +# an ordered PlanManifest of SubPlans. Every SubPlan **MUST** include a +# machine-checkable "verify" spec — this is the non-negotiable anti-hallucination +# guard from the GePeTo contract. The Python handler + PlanOrchestrator both +# reject manifests lacking verify. +# +# The returned JSON can be consumed by the body (agent_loop PlanOrchestrator) +# or re-injected into future heartbeats for cross-layer visibility. +# +# Intent strings are embodied-level (understood by Gemma-Andy via /intent). +# Verify types: INVENTORY_HAS | AREA_CLEAR | POSITION_REACHED | BLOCK_PLACED | ENTITY_NEARBY + +def _handle_mc_plan_decompose(args: dict, **kwargs) -> str: + """Decompose a complex goal into verifiable sub-plans. + + The LLM must supply a 'verify' object for **every** entry in sub_plans. + Missing verify → immediate error (forces correct decomposition). + """ + goal = (args.get("goal") or "").strip() + if not goal: + return "Error: 'goal' is required for mc_plan_decompose" + + raw_subs = args.get("sub_plans") or [] + if not isinstance(raw_subs, list) or len(raw_subs) == 0: + return "Error: 'sub_plans' must be a non-empty array" + + cleaned: list[dict[str, Any]] = [] + for idx, sp in enumerate(raw_subs): + if not isinstance(sp, dict): + return f"Error: sub_plans[{idx}] must be an object" + intent = (sp.get("intent") or "").strip() + if not intent: + return f"Error: sub_plans[{idx}] requires non-empty 'intent'" + + verify = sp.get("verify") + if not isinstance(verify, dict) or not verify.get("type"): + return ( + f"Error: sub_plans[{idx}] (intent={intent[:50]}) is MISSING REQUIRED 'verify' dict " + "with 'type' (INVENTORY_HAS / AREA_CLEAR / POSITION_REACHED / BLOCK_PLACED / ENTITY_NEARBY). " + "This is the anti-hallucination guard — every step must be machine-verifiable." + ) + + try: + order = int(sp.get("order", idx)) + except Exception: + order = idx + depends = sp.get("depends_on") or [] + if not isinstance(depends, list): + depends = [] + + cleaned.append({ + "intent": intent, + "verify": verify, + "order": order, + "depends_on": depends, + }) + + manifest: dict[str, Any] = { + "goal": goal, + "sub_plans": cleaned, + "estimated_time_s": int(args.get("estimated_time_s", 300)), + "abort_on_failure": bool(args.get("abort_on_failure", True)), + } + + # Best-effort forward to bot server (future /plan/manifest endpoint). + # If the endpoint does not exist yet the manifest is still returned validated. + forwarded = False + try: + resp = _api_post("/plan/manifest", {"action": "set_manifest", "manifest": manifest}, timeout=8) + if isinstance(resp, dict) and resp.get("ok"): + forwarded = True + except Exception: + # Non-fatal — the validated manifest is still useful to the caller / body layer. + pass + + payload = { + "ok": True, + "manifest": manifest, + "forwarded_to_body": forwarded, + "note": "Every sub-plan carries a VerifySpec. Use PlanOrchestrator.execute_plan() in the body to run.", + } + return json.dumps(payload, indent=2) + + +MC_PLAN_DECOMPOSE_SCHEMA = { + "name": "mc_plan_decompose", + "description": ( + "Fase 3: Decompose a high-level, multi-step Minecraft goal into an ordered PlanManifest. " + "Hermes (the strategic LLM) is responsible for producing the decomposition. " + "CRITICAL: every object in 'sub_plans' MUST contain a 'verify' field (object) with 'type' " + "and the parameters matching that type. Supported verify types: INVENTORY_HAS (item+count), " + "AREA_CLEAR (x1,z1,x2,z2,y,max_blocks_above), POSITION_REACHED (target_x/y/z + max_distance), " + "BLOCK_PLACED (block_x/y/z + block_material), ENTITY_NEARBY (entity_type + entity_distance). " + "The handler and PlanOrchestrator will reject any sub-plan lacking a valid verify — this prevents hallucinated steps. " + "depends_on is a list of 'order' values from other sub-plans. order values should be unique and ascending for sequential steps." + ), + "parameters": { + "type": "object", + "properties": { + "goal": { + "type": "string", + "description": "The high-level natural language goal to decompose (e.g. 'build a small wooden house with door and roof')", + }, + "sub_plans": { + "type": "array", + "description": "Ordered list of atomic, verifiable sub-plans. Execution order is determined by 'order' + 'depends_on'.", + "items": { + "type": "object", + "properties": { + "intent": { + "type": "string", + "description": "Embodied intent string passed to Gemma/embodied service (e.g. 'mine 64 oak_log', 'goto 128 64 -80', 'craft 4 oak_planks')", + }, + "verify": { + "type": "object", + "description": "MANDATORY machine-checkable verification predicate. Must include 'type' + params for that type.", + "properties": { + "type": {"type": "string", "enum": ["INVENTORY_HAS", "AREA_CLEAR", "POSITION_REACHED", "BLOCK_PLACED", "ENTITY_NEARBY"]}, + # inventory + "item": {"type": "string"}, + "count": {"type": "integer"}, + # area_clear + "x1": {"type": "integer"}, "z1": {"type": "integer"}, + "x2": {"type": "integer"}, "z2": {"type": "integer"}, + "y": {"type": "integer"}, "max_blocks_above": {"type": "integer"}, + # position + "target_x": {"type": "integer"}, "target_y": {"type": "integer"}, "target_z": {"type": "integer"}, + "max_distance": {"type": "number"}, + # block_placed + "block_x": {"type": "integer"}, "block_y": {"type": "integer"}, "block_z": {"type": "integer"}, + "block_material": {"type": "string"}, + # entity + "entity_type": {"type": "string"}, + "entity_distance": {"type": "number"}, + }, + "required": ["type"], + }, + "order": { + "type": "integer", + "description": "Execution priority / sequence number. Lower runs earlier. Must be unique within manifest.", + }, + "depends_on": { + "type": "array", + "items": {"type": "integer"}, + "description": "List of 'order' values that must complete before this sub-plan may start.", + }, + }, + "required": ["intent", "verify", "order"], + }, + }, + "estimated_time_s": { + "type": "integer", + "description": "Rough total time budget for the whole manifest (seconds).", + }, + "abort_on_failure": { + "type": "boolean", + "description": "If true (default), first VerifySpec failure escalates to Hermes with previous_error for replanning.", + }, + }, + "required": ["goal", "sub_plans"], + }, +} + + +# ═══════════════════════════════════════════════════════════════════ +# 10. mc_screenshot — Ray-traced world capture +# ═══════════════════════════════════════════════════════════════════ + +def _handle_mc_screenshot(args: dict, **kwargs) -> str: + """Take a screenshot of the Minecraft world from the bot's first-person perspective. + + Uses prismarine-viewer (Three.js WebGL renderer) + puppeteer headless Chrome. + The image is saved as PNG to the bot server and the path is returned. + """ + payload: Dict[str, Any] = {} + if "width" in args: + payload["width"] = args["width"] + if "height" in args: + payload["height"] = args["height"] + if "file_name" in args: + fname = args["file_name"] + if not fname.endswith(".png"): + fname += ".png" + payload["file_name"] = fname + + resp = _api_post("/action/screenshot", payload, timeout=300) + if not resp.get("ok", True): + return f"Error: {resp.get('error', 'Screenshot failed')}" + + path = resp.get("path", "unknown") + width = resp.get("width", "?") + height = resp.get("height", "?") + return f"Screenshot saved to {path} ({width}x{height})" + + +MC_SCREENSHOT_SCHEMA = { + "name": "mc_screenshot", + "description": "Take a screenshot of the Minecraft world from the bot's eyes. Uses a WebGL renderer (prismarine-viewer) served on a local port and captured via headless Chrome. Produces a PNG image. Specify width/height (default 1280x720, max 1920x1080) and optionally a custom file_name. The returned path is an absolute PNG file path. If you need to SEE what is in the image, call vision_analyze with the returned path.", + "parameters": { + "type": "object", + "properties": { + "width": {"type": "number", "description": "Image width in pixels (default: 1280, max: 1920)"}, + "height": {"type": "number", "description": "Image height in pixels (default: 720, max: 1080)"}, + "file_name": {"type": "string", "description": "Custom filename for the screenshot (optional). Will be saved as a .png file."}, + }, + }, +} + + +# ═══════════════════════════════════════════════════════════════════ +# 11. mc_command — Execute Minecraft server commands +# ═══════════════════════════════════════════════════════════════════ + +def _handle_mc_command(args: dict, **kwargs) -> str: + """Execute a Minecraft server command via the bot's chat interface. + + The bot must have operator privileges for most commands. + Commands are sent as chat messages starting with '/' and are executed + by the server without appearing in public chat. + """ + command = args.get("command", "") + if not command: + return "Error: command is required" + if not command.startswith("/"): + command = "/" + command + + # ═─ Intercept /godmode toggle ─══════════════════════════════════════ + stripped = command.strip().lower() + if stripped == "/godmode on" or stripped == "/godmode": + _gm_path = Path.home() / ".local" / "share" / "daemoncraft" / "rolemaster" / "godmode" + _gm_path.parent.mkdir(parents=True, exist_ok=True) + _gm_path.write_text("on") + return "Godmode ENABLED. The Daemon Guardian will keep you in creative mode with invulnerability effects." + if stripped == "/godmode off": + _gm_path = Path.home() / ".local" / "share" / "daemoncraft" / "rolemaster" / "godmode" + _gm_path.parent.mkdir(parents=True, exist_ok=True) + _gm_path.write_text("off") + return "Godmode DISABLED. The Daemon Guardian is paused. You can now take damage, drown, or switch gamemodes. Say '/godmode on' to restore protection." + + return _fmt(_api_post("/chat/send", {"message": command})) + + +MC_COMMAND_SCHEMA = { + "name": "mc_command", + "description": "Execute any Minecraft server command. The bot must have operator privileges. Examples: /weather thunder, /time set midnight, /summon zombie ~ ~ ~, /give @p diamond 1, /effect give @p blindness 10, /playsound ambient.cave ambient @p, /tellraw @p {\"text\":\"Hello\"}, /setblock ~ ~ ~ stone, /fill x1 y1 z1 x2 y2 z2 water. This is the primary tool for world manipulation in Role Master mode.", + "parameters": { + "type": "object", + "properties": { + "command": { + "type": "string", + "description": "Minecraft command to execute. Must start with / or it will be added automatically.", + }, + }, + "required": ["command"], + }, +} + + +# ═══════════════════════════════════════════════════════════════════ +# 12. mc_story — Narrative state tracker for Role Master mode +# ═══════════════════════════════════════════════════════════════════ + +import contextvars +import os +from pathlib import Path + +_STORY_PATH = Path(os.getenv("DAEMONCRAFT_STORY_PATH", Path.home() / ".local" / "share" / "daemoncraft" / "story.json")) +_BLUEPRINT_PATH = Path(os.getenv("DAEMONCRAFT_BLUEPRINT_PATH", Path.home() / ".local" / "share" / "daemoncraft" / "blueprint.json")) +# Shared blueprints directory used by the dashboard and mc_story +_BLUEPRINTS_DIR = Path(__file__).parent.parent / "blueprints" + + +def _load_story() -> dict: + if _STORY_PATH.exists(): + try: + return json.loads(_STORY_PATH.read_text()) + except Exception: + pass + return { + "title": None, + "phase": None, + "phase_started_at": None, + "phase_timeout_minutes": None, + "last_player_activity": None, + "day": 1, + "flags": {}, + "objectives": [], + "events": [], + "player_choices": {}, + "active_sensors": [], + "active_blueprint": None, + "active_blueprint_tag": None, + } + + +def _save_story(story: dict) -> None: + _STORY_PATH.parent.mkdir(parents=True, exist_ok=True) + _STORY_PATH.write_text(json.dumps(story, indent=2)) + + +def _handle_mc_story(args: dict, **kwargs) -> str: + """Track narrative state for Role Master adventures. Pure Python — no bot server needed.""" + action = args.get("action", "get_state") + story = _load_story() + + if action == "get_state": + import datetime as _dt + lines = [ + f"Story: {story.get('title') or 'Untitled'}", + f"Phase: {story.get('phase') or 'none'}", + f"Day: {story.get('day', 1)}", + f"Active blueprint: {story.get('active_blueprint', 'none')}", + f"Active blueprint tag: {story.get('active_blueprint_tag', 'none')}", + f"Flags: {json.dumps(story.get('flags', {}))}", + f"Objectives ({len(story.get('objectives', []))}):", + ] + for obj in story.get("objectives", []): + status = obj.get("status", "pending") + lines.append(f" [{status}] {obj.get('title', 'Untitled')}: {obj.get('description', '')}") + # Timeout info + timeout = story.get("phase_timeout_minutes") + started = story.get("phase_started_at") + last_act = story.get("last_player_activity") + if timeout and started: + elapsed = (_dt.datetime.now(_dt.timezone.utc) - _dt.datetime.fromisoformat(started)).total_seconds() / 60 + remaining = timeout - elapsed + lines.append(f"Phase timeout: {max(0, remaining):.1f} minutes remaining") + if last_act: + ago = (_dt.datetime.now(_dt.timezone.utc) - _dt.datetime.fromisoformat(last_act)).total_seconds() / 60 + lines.append(f"Last player activity: {ago:.1f} minutes ago") + lines.append(f"Events ({len(story.get('events', []))}): {story.get('events', [])[-5:]}") + return "\n".join(lines) + + if action == "set_flag": + key = args.get("key") + value = args.get("value") + if key is None: + return "Error: key is required for set_flag" + story["flags"][key] = value + _save_story(story) + return f"Flag set: {key} = {value}" + + if action == "advance_phase": + phase = args.get("phase") + if not phase: + return "Error: phase is required for advance_phase" + import datetime as _dt + story["phase"] = phase + story["phase_started_at"] = _dt.datetime.now(_dt.timezone.utc).isoformat() + timeout = args.get("timeout_minutes") + if timeout is not None: + story["phase_timeout_minutes"] = timeout + story["events"].append(f"Advanced to phase: {phase}") + _save_story(story) + return f"Phase advanced to: {phase}" + + if action == "record_activity": + import datetime as _dt + story["last_player_activity"] = _dt.datetime.now(_dt.timezone.utc).isoformat() + _save_story(story) + return "Player activity recorded" + + if action == "check_timeout": + import datetime as _dt + phase = story.get("phase") + timeout = story.get("phase_timeout_minutes") + started = story.get("phase_started_at") + last_act = story.get("last_player_activity") + if not phase or not timeout: + return "No active phase with timeout" + # Use last_player_activity if available, otherwise phase_started_at + ref_time = last_act or started + if not ref_time: + return "No reference time for timeout check" + elapsed = (_dt.datetime.now(_dt.timezone.utc) - _dt.datetime.fromisoformat(ref_time)).total_seconds() / 60 + if elapsed > timeout: + story["phase"] = None + story["phase_started_at"] = None + story["phase_timeout_minutes"] = None + # Reset objectives of abandoned phase + for obj in story.get("objectives", []): + if obj.get("status") == "pending": + obj["status"] = "abandoned" + _save_story(story) + return f"Phase '{phase}' ABANDONED after {elapsed:.1f} minutes of inactivity. Objectives reset." + return f"Phase '{phase}' still active. {timeout - elapsed:.1f} minutes remaining." + + if action == "reset_phase": + phase = args.get("phase") + if phase: + story["events"].append(f"Phase reset: {phase}") + story["phase"] = None + story["phase_started_at"] = None + story["phase_timeout_minutes"] = None + for obj in story.get("objectives", []): + if obj.get("status") in ("pending", "abandoned"): + obj["status"] = "pending" + _save_story(story) + return f"Phase reset. Current phase: none. Pending objectives restored." + + if action == "advance_day": + story["day"] = story.get("day", 1) + 1 + story["events"].append(f"Day advanced to {story['day']}") + _save_story(story) + return f"Day advanced to {story['day']}" + + if action == "add_objective": + title = args.get("title") + if not title: + return "Error: title is required for add_objective" + obj = { + "id": len(story.get("objectives", [])), + "title": title, + "description": args.get("description", ""), + "status": "pending", + "optional": args.get("optional", False), + } + story.setdefault("objectives", []).append(obj) + story["events"].append(f"Added objective: {title}") + _save_story(story) + return f"Objective added: {title}" + + if action == "complete_objective": + obj_id = args.get("objective_id") + if obj_id is None: + return "Error: objective_id is required for complete_objective" + objectives = story.get("objectives", []) + if obj_id < 0 or obj_id >= len(objectives): + return f"Error: objective_id {obj_id} not found" + objectives[obj_id]["status"] = "done" + story["events"].append(f"Completed objective: {objectives[obj_id]['title']}") + _save_story(story) + return f"Objective completed: {objectives[obj_id]['title']}" + + if action == "log_event": + event = args.get("event") + if not event: + return "Error: event is required for log_event" + story.setdefault("events", []).append(event) + _save_story(story) + return f"Event logged: {event}" + + if action == "get_events": + count = args.get("count", 10) + events = story.get("events", []) + recent = events[-count:] if events else [] + return "Recent events:\n" + "\n".join(f" {i+1}. {e}" for i, e in enumerate(recent)) if recent else "No events recorded yet." + + if action == "set_title": + title = args.get("title") + if not title: + return "Error: title is required for set_title" + story["title"] = title + _save_story(story) + return f"Story title set: {title}" + + if action == "record_choice": + player = args.get("player", "unknown") + choice = args.get("choice") + if not choice: + return "Error: choice is required for record_choice" + story.setdefault("player_choices", {})[player] = choice + story["events"].append(f"{player} chose: {choice}") + _save_story(story) + return f"Choice recorded for {player}: {choice}" + + if action == "reset": + _save_story({ + "title": None, + "phase": None, + "day": 1, + "flags": {}, + "objectives": [], + "events": [], + "player_choices": {}, + }) + return "Story state reset" + + if action == "save_blueprint": + blueprint = args.get("blueprint") + name = args.get("name") + if not blueprint: + return "Error: blueprint JSON is required for save_blueprint" + if not isinstance(blueprint, dict): + return "Error: blueprint must be a JSON object" + if name: + target = _BLUEPRINTS_DIR / f"{name}.json" + _BLUEPRINTS_DIR.mkdir(parents=True, exist_ok=True) + else: + target = _BLUEPRINT_PATH + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(blueprint, indent=2)) + return f"Blueprint saved: {blueprint.get('metadata', {}).get('title', 'Untitled')}" + + if action == "load_blueprint": + name = args.get("name") + if name: + target = _BLUEPRINTS_DIR / f"{name}.json" + else: + target = _BLUEPRINT_PATH + if not target.exists(): + return f"No blueprint found: {target.name}" + try: + bp = json.loads(target.read_text()) + title = bp.get("metadata", {}).get("title", "Untitled") + phases = len(bp.get("phases", [])) + entities = len(bp.get("entities", [])) + # Store blueprint tag in story state for cleanup reference + tag = re.sub(r'[^a-z0-9_]', '_', title.lower()) + story["active_blueprint"] = str(target.name) + story["active_blueprint_tag"] = f"dc_blueprint_{tag}" + _save_story(story) + return f"Blueprint: {title}\nTag: dc_blueprint_{tag}\nPhases: {phases}\nEntities: {entities}\nFlags: {json.dumps(bp.get('flags', {}))}" + except Exception as e: + return f"Error loading blueprint: {e}" + + if action == "check_score": + player = args.get("player") + objective = args.get("objective") + if not player or not objective: + return "Error: player and objective are required for check_score" + result = _api_get(f"/scoreboard?objective={objective}&player={player}") + if not result.get("ok"): + return _fmt(result) + data = result.get("data", {}) + score = data.get("score", 0) + note = data.get("note", "") + return f"Score for {player} on {objective}: {score}" + (f" ({note})" if note else "") + + if action == "set_score": + player = args.get("player") + objective = args.get("objective") + value = args.get("value", 0) + if not player or not objective: + return "Error: player and objective are required for set_score" + result = _api_post("/chat/send", {"message": f"/scoreboard players set {player} {objective} {value}"}) + return _fmt(result) + + if action == "run_function": + function = args.get("function") + if not function: + return "Error: function path is required for run_function" + result = _api_post("/chat/send", {"message": f"/function {function}"}) + return _fmt(result) + + if action == "setup_sensors": + sensors = args.get("sensors", []) + if not sensors: + return "Error: sensors list required for setup_sensors" + created = [] + for s in sensors: + name = s.get("name") + criterion = s.get("criterion", "dummy") + poll_command = s.get("poll_command") + if not name: + continue + # Create scoreboard in Minecraft + _api_post("/chat/send", {"message": f"/scoreboard objectives add {name} {criterion}"}) + # Register/update in story state + existing = story.get("active_sensors", []) + existing = [x for x in existing if x.get("name") != name] + existing.append({"name": name, "criterion": criterion, "poll_command": poll_command}) + story["active_sensors"] = existing + created.append(name) + _save_story(story) + return f"Sensors created and registered: {created}" + + if action == "poll_sensors": + player = args.get("player", "@a") + reset = args.get("reset", True) + sensors = story.get("active_sensors", []) + if not sensors: + return "No active sensors" + results = [] + for s in sensors: + name = s.get("name") + poll_command = s.get("poll_command") + # Execute poll command for dummy sensors (proximity, zone, etc.) + if poll_command: + _api_post("/chat/send", {"message": poll_command}) + # Read score via native API + result = _api_get(f"/scoreboard?objective={name}&player={player}") + if result.get("ok"): + score = result.get("data", {}).get("score", 0) + fired = score > 0 + if fired and reset: + _api_post("/chat/send", {"message": f"/scoreboard players set {player} {name} 0"}) + results.append(f"{name}: {score}" + (" (fired)" if fired else "")) + else: + results.append(f"{name}: error") + return "Sensor poll results:\n" + "\n".join(results) + + if action == "cleanup_sensors": + targets = args.get("sensors", []) + sensors = story.get("active_sensors", []) + if not targets: + # Default: cleanup all + targets = [s.get("name") for s in sensors] + removed = [] + for name in targets: + _api_post("/chat/send", {"message": f"/scoreboard objectives remove {name}"}) + removed.append(name) + story["active_sensors"] = [s for s in sensors if s.get("name") not in targets] + _save_story(story) + return f"Sensors removed: {removed}. Remaining: {[s['name'] for s in story['active_sensors']]}" + + return f"Error: unknown story action '{action}'" + + +MC_STORY_SCHEMA = { + "name": "mc_story", + "description": "Narrative state tracker for Role Master mode. Tracks story phase, day counter, flags, objectives, events, player choices, and active scoreboard sensors across sessions. Supports phase timeouts, activity tracking, and sensor restoration for quest-like progression. All data persists in a JSON file. No bot connection required.", + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "get_state", "set_flag", "advance_phase", "advance_day", + "add_objective", "complete_objective", "log_event", "get_events", + "set_title", "record_choice", "reset", + "save_blueprint", "load_blueprint", + "record_activity", "check_timeout", "reset_phase", + "check_score", "set_score", "run_function", + "setup_sensors", "poll_sensors", "cleanup_sensors", + ], + "description": "Story management action", + }, + "key": {"type": "string", "description": "Flag key (for set_flag)"}, + "value": {"type": ["string", "number", "boolean"], "description": "Flag value (for set_flag)"}, + "phase": {"type": "string", "description": "Phase name (for advance_phase or reset_phase)"}, + "timeout_minutes": {"type": "number", "description": "Minutes before phase is abandoned if no player activity (for advance_phase)"}, + "title": {"type": "string", "description": "Objective or story title"}, + "description": {"type": "string", "description": "Objective description"}, + "objective_id": {"type": "number", "description": "Objective index to complete"}, + "event": {"type": "string", "description": "Event description to log"}, + "count": {"type": "number", "description": "Number of recent events to retrieve (for get_events; default: 10)"}, + "player": {"type": "string", "description": "Player name (for record_choice or check_score/set_score)"}, + "choice": {"type": "string", "description": "Choice description (for record_choice)"}, + "optional": {"type": "boolean", "description": "Whether objective is optional"}, + "blueprint": {"type": "object", "description": "Full adventure blueprint JSON (for save_blueprint)"}, + "objective": {"type": "string", "description": "Scoreboard objective name (for check_score/set_score)"}, + "sensors": { + "type": "array", + "description": "List of sensor objects for setup_sensors or cleanup_sensors. Each object: {name, criterion, poll_command?}", + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "criterion": {"type": "string"}, + "poll_command": {"type": "string", "description": "Optional /execute command for dummy sensors"}, + }, + }, + }, + "reset": {"type": "boolean", "description": "Whether to reset fired sensor scores to 0 after polling (for poll_sensors; default: true)"}, + "function": {"type": "string", "description": "Datapack function path (for run_function)"}, + }, + "required": ["action"], + }, +} + + +MC_REGISTRY_SCHEMA = { + "name": "mc_registry", + "description": "Query the shared Minecraft validation registry for canonical lists of biomes, entities, items, blocks, effects, and scoreboard criteria. Use this when you need to know valid values for adventure blueprints (e.g., 'what flying passive mobs exist?', 'what biomes are in the overworld?', 'is crow a valid entity?'). Results are sourced from minecraft-data for the configured server version.", + "parameters": { + "type": "object", + "properties": { + "category": { + "type": "string", + "enum": ["biomes", "entities", "items", "blocks", "effects", "scoreboard_criteria"], + "description": "Registry category to query", + }, + "filter": {"type": "string", "description": "Optional substring filter on name or displayName (case-insensitive)"}, + "limit": {"type": "number", "description": "Max results to return (default 20, max 100)"}, + "type_filter": {"type": "string", "description": "For entities: filter by type (e.g. mob, animal, hostile, passive, ambient)"}, + "dimension": {"type": "string", "description": "For biomes: filter by dimension (overworld, nether, end)"}, + }, + "required": ["category"], + }, +} + +def _handle_mc_registry(args: dict, **kwargs) -> str: + category = args.get("category") + filt = (args.get("filter") or "").lower() + limit = min(int(args.get("limit") or 20), 100) + type_filter = (args.get("type_filter") or "").lower() + dimension = (args.get("dimension") or "").lower() + + registry_path = Path(__file__).parent.parent / "data" / "minecraft-registry.json" + if not registry_path.exists(): + return "Error: minecraft-registry.json not found. Run scripts/generate-minecraft-registry.js to create it." + + try: + registry = json.loads(registry_path.read_text()) + except Exception as e: + return f"Error reading registry: {e}" + + items = registry.get(category) + if items is None: + return f"Error: unknown category '{category}'. Valid: biomes, entities, items, blocks, effects, scoreboard_criteria" + + results = [] + for item in items: + name = item.get("name", "") + display = item.get("displayName", "") + if filt and filt not in name.lower() and filt not in display.lower(): + continue + if category == "entities" and type_filter: + if type_filter not in (item.get("type") or "").lower(): + continue + if category == "biomes" and dimension: + if dimension not in (item.get("dimension") or "").lower(): + continue + results.append(item) + + if not results: + return f"No {category} matched the filters." + + lines = [f"{category} ({len(results)} matches, showing first {min(limit, len(results))}):"] + for item in results[:limit]: + if category == "entities": + lines.append(f" - {item['name']} ({item.get('displayName','')}) type={item.get('type','')}, category={item.get('category','')}") + elif category == "biomes": + lines.append(f" - {item['name']} ({item.get('displayName','')}) dimension={item.get('dimension','')}") + elif category == "scoreboard_criteria": + lines.append(f" - {item['name']} — {item.get('description','')}") + else: + lines.append(f" - {item['name']} ({item.get('displayName','')})") + + if len(results) > limit: + lines.append(f" ... and {len(results) - limit} more") + + return "\n".join(lines) + + +# ══════════════════════════════════════════════════════════════════════════════════════════ +# Registry +# ══════════════════════════════════════════════════════════════════════════════════════ + +registry.register( + name="mc_perceive", + toolset="minecraft", + schema=MC_PERCEIVE_SCHEMA, + handler=lambda args, **kw: _handle_mc_perceive(args, **kw), +) +registry.register( + name="mc_move", + toolset="minecraft", + schema=MC_MOVE_SCHEMA, + handler=lambda args, **kw: _handle_mc_move(args, **kw), +) +registry.register( + name="mc_mine", + toolset="minecraft", + schema=MC_MINE_SCHEMA, + handler=lambda args, **kw: _handle_mc_mine(args, **kw), +) +registry.register( + name="mc_build", + toolset="minecraft", + schema=MC_BUILD_SCHEMA, + handler=lambda args, **kw: _handle_mc_build(args, **kw), +) +registry.register( + name="mc_craft", + toolset="minecraft", + schema=MC_CRAFT_SCHEMA, + handler=lambda args, **kw: _handle_mc_craft(args, **kw), +) +registry.register( + name="mc_combat", + toolset="minecraft", + schema=MC_COMBAT_SCHEMA, + handler=lambda args, **kw: _handle_mc_combat(args, **kw), +) +# ── Environment flag: loop mode suppresses mc_chat registration ── +# The gateway (social layer) needs mc_chat. The loop (body layer) does not. +if not os.getenv("DC_LOOP_MODE"): + registry.register( + name="mc_chat", + toolset="minecraft", + schema=MC_CHAT_SCHEMA, + handler=lambda args, **kw: _handle_mc_chat(args, **kw), + ) +else: + print("[minecraft_tools] DC_LOOP_MODE=1 — mc_chat tool suppressed for body-only mode", flush=True) +registry.register( + name="mc_manage", + toolset="minecraft", + schema=MC_MANAGE_SCHEMA, + handler=lambda args, **kw: _handle_mc_manage(args, **kw), +) +registry.register( + name="mc_plan", + toolset="minecraft", + schema=MC_PLAN_SCHEMA, + handler=lambda args, **kw: _handle_mc_plan(args, **kw), +) +registry.register( + name="mc_screenshot", + toolset="minecraft", + schema=MC_SCREENSHOT_SCHEMA, + handler=lambda args, **kw: _handle_mc_screenshot(args, **kw), +) +registry.register( + name="mc_command", + toolset="minecraft", + schema=MC_COMMAND_SCHEMA, + handler=lambda args, **kw: _handle_mc_command(args, **kw), +) +MC_NOOP_SCHEMA = { + "type": "object", + "properties": { + "reason": { + "type": "string", + "description": "Optional reason for choosing no action.", + }, + }, +} + +def _handle_mc_noop(args: Dict[str, Any], **kw) -> str: + """No-op tool for wake-up events where the agent chooses not to react.""" + return "No action taken." + + +registry.register( + name="mc_story", + toolset="minecraft", + schema=MC_STORY_SCHEMA, + handler=lambda args, **kw: _handle_mc_story(args, **kw), +) + +registry.register( + name="mc_registry", + toolset="minecraft", + schema=MC_REGISTRY_SCHEMA, + handler=lambda args, **kw: _handle_mc_registry(args, **kw), +) + +# ═══════════════════════════════════════════════════════════════════ +# mc_interoception — Body-internal state (health, hunger, runner activity) +# ═══════════════════════════════════════════════════════════════════ + +MC_INTEROCEPTION_SCHEMA = { + "type": "object", + "properties": { + "detail": { + "type": "boolean", + "description": "If true, return full reflex history (capped at 10). Default: summary view (last 3 with timestamps + aggregated count of older).", + }, + }, +} + +def _handle_mc_interoception(args: dict, **kwargs) -> str: + """Query body-internal state: health, food, holding, position, runner activity. + + Returns a delta of what the body (L2 runner) has been doing since the last + query. Each call updates the `since` timestamp — the next call only returns + new activity. + + Args: + detail: If True, return full reflex history. Default: summary. + """ + detail = args.get("detail", False) + params = f"?detail={'true' if detail else 'false'}" + resp = _api_get(f"/interoception{params}") + if not resp.get("ok", True): + return f"Error: {resp.get('error', 'interoception unavailable')}" + return json.dumps(resp.get("data", {}), indent=2) + +registry.register( + name="mc_interoception", + toolset="minecraft", + schema=MC_INTEROCEPTION_SCHEMA, + handler=lambda args, **kw: _handle_mc_interoception(args, **kw), +) + +registry.register( + name="mc_no_op", + toolset="minecraft", + schema=MC_NOOP_SCHEMA, + handler=lambda args, **kw: _handle_mc_noop(args, **kw), +) + +# ═══════════════════════════════════════════════════════════════════ +# mc_plan_decompose — Hermes decomposes multi-step goals into PlanManifest +# ═══════════════════════════════════════════════════════════════════ + +MC_PLAN_DECOMPOSE_SCHEMA = { + "type": "object", + "properties": { + "goal": { + "type": "string", + "description": "Natural language description of the multi-step goal to decompose.", + }, + "context": { + "type": "object", + "description": "Optional: current world state (position, inventory summary, nearby entities). Helps Hermes produce grounded sub-plans.", + }, + }, + "required": ["goal"], +} + + +def _handle_mc_plan_decompose(args: dict, **kwargs) -> str: + """Decompose a multi-step goal into a PlanManifest with verified SubPlans. + + Hermes (LLM) is responsible for generating the sub-plan structure. This tool: + 1. Receives the goal + context + 2. Returns a PlanManifest-ready JSON schema for Hermes to fill in + 3. Validates that every SubPlan has a VerifySpec (anti-hallucination guard) + + The actual decomposition happens in the LLM's response — this tool provides + the contract and validation. The PlanOrchestrator (DaemonCraft Fase 3) executes + the manifest. + + Returns a JSON schema template for PlanManifest that Hermes should populate, + plus the validation rules. + """ + goal = args.get("goal", "") + context = args.get("context", {}) + + if not goal or not goal.strip(): + return json.dumps({"error": "mc_plan_decompose requires non-empty 'goal'"}) + + return json.dumps({ + "instruction": ( + "Decompose the following goal into a PlanManifest. " + "Each SubPlan MUST have a 'verify' block. " + "Supported verify types: INVENTORY_HAS, POSITION_REACHED, AREA_CLEAR, " + "BLOCK_PLACED, ENTITY_NEARBY. " + "Return the manifest as a JSON object matching this schema." + ), + "goal": goal, + "context": context, + "schema": { + "goal": "", + "estimated_time_s": 300, + "abort_on_failure": True, + "sub_plans": [ + { + "intent": "", + "order": 0, + "depends_on": [], + "verify": { + "type": "INVENTORY_HAS", + "item": "oak_log", + "count": 64, + }, + }, + ], + }, + "validation_rules": [ + "Every sub_plan MUST have a 'verify' block with a valid 'type'.", + "Verify type INVENTORY_HAS requires 'item' and 'count'.", + "Verify type POSITION_REACHED requires 'target_x', 'target_y', 'target_z'.", + "depends_on must reference valid 'order' values (not self, not future if order < dep).", + "No sub-plan without verify will be executed (anti-hallucination guard).", + ], + }) + + +registry.register( + name="mc_plan_decompose", + toolset="minecraft", + schema=MC_PLAN_DECOMPOSE_SCHEMA, + handler=lambda args, **kw: _handle_mc_plan_decompose(args, **kw), +) + +# ═══════════════════════════════════════════════════════════════════ +# mc_start_quantified_intent — Hermes starts tracking a quantified intent +# ═══════════════════════════════════════════════════════════════════ + +MC_START_QUANTIFIED_INTENT_SCHEMA = { + "type": "object", + "properties": { + "intent_type": { + "type": "string", + "description": "Intent type: 'mine', 'gather', 'collect', etc.", + }, + "target_count": { + "type": "integer", + "description": "How many to mine/gather (e.g. 64).", + }, + "verify_spec": { + "type": "object", + "description": "Optional verify spec. If omitted, executor uses best-effort tracking.", + "properties": { + "type": {"type": "string", "description": "VerifyType: INVENTORY_HAS, etc."}, + "item": {"type": "string"}, + "count": {"type": "integer"}, + }, + }, + }, + "required": ["intent_type", "target_count"], +} + + +def _handle_mc_start_quantified_intent(args: dict, **kwargs) -> str: + """Start tracking a quantified intent via the bot server's shared state. + + Writes to executor_intent.json, which agent_loop.py polls on heartbeat ticks. + The QuantifiedIntentExecutor snapshots inventory baseline and tracks progress + across L2 reflex preemption. + + Args: + intent_type: 'mine', 'gather', 'collect', etc. + target_count: How many units to track (e.g. 64) + verify_spec: Optional dict with {type, item, count} + """ + intent_type = args.get("intent_type", "") + target_count = int(args.get("target_count", 0)) + verify_spec = args.get("verify_spec") + + if not intent_type or target_count <= 0: + return json.dumps({"error": "intent_type and target_count > 0 required"}) + + payload = { + "intent_type": intent_type, + "target_count": target_count, + "verify_spec": verify_spec, + } + resp = _api_post("/executor/start-intent", payload) + if not resp.get("ok"): + return f"Error: {resp.get('error', 'start-intent failed')}" + return json.dumps(resp.get("data", {})) + + +registry.register( + name="mc_start_quantified_intent", + toolset="minecraft", + schema=MC_START_QUANTIFIED_INTENT_SCHEMA, + handler=lambda args, **kw: _handle_mc_start_quantified_intent(args, **kw), +) + +# ═══════════════════════════════════════════════════════════════════ +# mc_submit_plan — Hermes submits a PlanManifest for orchestration +# ═══════════════════════════════════════════════════════════════════ + +MC_SUBMIT_PLAN_SCHEMA = { + "type": "object", + "properties": { + "manifest": { + "type": "object", + "description": "Full PlanManifest dict matching the schema from mc_plan_decompose.", + }, + }, + "required": ["manifest"], +} + + +def _handle_mc_submit_plan(args: dict, **kwargs) -> str: + """Submit a PlanManifest for execution by the PlanOrchestrator. + + Writes to plan_manifest.json, which agent_loop.py polls on heartbeat ticks. + The orchestrator validates (anti-hallucination guard: every SubPlan must + have a VerifySpec) and executes sub-plans respecting order and depends_on. + + Use mc_plan_decompose first to get the schema + validation rules, + then call mc_submit_plan with the completed manifest. + """ + manifest = args.get("manifest") + if not manifest: + return json.dumps({"error": "manifest is required"}) + + payload = {"manifest": manifest} + resp = _api_post("/plan/submit", payload) + if not resp.get("ok"): + return f"Error: {resp.get('error', 'plan submission failed')}" + return json.dumps(resp.get("data", {"received": True})) + + +registry.register( + name="mc_submit_plan", + toolset="minecraft", + schema=MC_SUBMIT_PLAN_SCHEMA, + handler=lambda args, **kw: _handle_mc_submit_plan(args, **kw), +) + +# ═══════════════════════════════════════════════════════════════════ +# mc_macro — Pre-canned multi-step skills (staircase, spiral, etc.) +# ═══════════════════════════════════════════════════════════════════ + +MC_MACRO_SCHEMA = { + "type": "object", + "properties": { + "macro": { + "type": "string", + "enum": ["staircase", "spiral", "tunnel"], + "description": "Which macro skill to execute. 'staircase' mines a 1-wide diagonal staircase upward in a cardinal direction. 'spiral' rotates direction every N steps to create a caracol staircase. 'tunnel' mines a 2-high 1-wide horizontal tunnel in a cardinal direction." + }, + "direction": { + "type": "string", + "enum": ["west", "east", "north", "south"], + "description": "Cardinal direction. Required for 'staircase' and 'tunnel'." + }, + "target_y": { + "type": "number", + "description": "Target Y level. Required for 'staircase' and 'spiral'." + }, + "steps_per_side": { + "type": "number", + "description": "For 'spiral': steps before rotating 90°. 2 = tight spiral with 1-block center pillar. Default 3." + }, + "distance": { + "type": "number", + "description": "For 'tunnel': how many blocks to tunnel. Default 10." + }, + }, + "required": ["macro"], +} + + +def _handle_mc_macro(args: dict, **kwargs) -> str: + """Execute a pre-canned macro skill via POST /macro.""" + macro = args.get("macro") + if not macro: + return "Error: 'macro' is required. Available: staircase, spiral" + + if macro == "staircase": + direction = args.get("direction") + target_y = args.get("target_y") + if not direction: + return "Error: 'direction' is required for staircase (west/east/north/south)" + if target_y is None: + return "Error: 'target_y' is required for staircase" + + resp = _api_post("/macro", { + "macro": "staircase", + "direction": direction, + "target_y": target_y, + }, timeout=600) + + if not resp.get("ok"): + return f"Error: {resp.get('error', 'staircase failed')}" + return ( + f"{resp.get('message', 'Done.')}\n" + f"steps: {resp.get('steps', '?')}, " + f"finalY: {resp.get('finalY', '?')}" + ) + + if macro == "spiral": + target_y = args.get("target_y") + steps_per_side = args.get("steps_per_side") + if target_y is None: + return "Error: 'target_y' is required for spiral" + + body = {"macro": "spiral", "target_y": target_y} + if steps_per_side is not None: + body["steps_per_side"] = steps_per_side + + resp = _api_post("/macro", body, timeout=600) + + if not resp.get("ok"): + return f"Error: {resp.get('error', 'spiral failed')}" + return ( + f"{resp.get('message', 'Done.')}\n" + f"steps: {resp.get('steps', '?')}, " + f"finalY: {resp.get('finalY', '?')}" + ) + + if macro == "tunnel": + direction = args.get("direction") + distance = args.get("distance", 10) + if not direction: + return "Error: 'direction' is required for tunnel (west/east/north/south)" + + resp = _api_post("/macro", { + "macro": "tunnel", + "direction": direction, + "distance": distance, + }, timeout=600) + + if not resp.get("ok"): + return f"Error: {resp.get('error', 'tunnel failed')}" + return ( + f"{resp.get('message', 'Done.')}\n" + f"steps: {resp.get('steps', '?')}, " + f"from: ({resp.get('startPos', {}).get('x', '?')},{resp.get('startPos', {}).get('y', '?')},{resp.get('startPos', {}).get('z', '?')}), " + f"to: ({resp.get('endPos', {}).get('x', '?')},{resp.get('endPos', {}).get('y', '?')},{resp.get('endPos', {}).get('z', '?')})" + ) + + return f"Error: unknown macro '{macro}'. Available: staircase, spiral, tunnel" + + +registry.register( + name="mc_macro", + toolset="minecraft", + schema=MC_MACRO_SCHEMA, + handler=lambda args, **kw: _handle_mc_macro(args, **kw), +) diff --git a/tools/research_job_tool.py b/tools/research_job_tool.py new file mode 100644 index 000000000000..1259f0aa84c8 --- /dev/null +++ b/tools/research_job_tool.py @@ -0,0 +1,413 @@ +"""research_job_tool — orchestrate long-running research jobs as detached OS processes. + +Provides start, status, collect, and resume operations for research loops +that outlive a single agent turn. +""" + +from __future__ import annotations + +import json +import logging +import secrets +import shlex +import sys +import time +from pathlib import Path +from typing import Any, Optional + +from hermes_constants import get_hermes_home +from tools.registry import registry, tool_error + +logger = logging.getLogger(__name__) + + +def _job_dir(job_id: str) -> Path: + return get_hermes_home() / "research-jobs" / job_id + + +def _write_job_spec(job_id: str, spec: dict[str, Any]) -> Path: + jd = _job_dir(job_id) + jd.mkdir(parents=True, exist_ok=True) + spec_path = jd / "job.json" + spec_path.write_text(json.dumps(spec, indent=2)) + return spec_path + + +def _load_config_for_job() -> dict[str, Any]: + """Read Hermes config to extract model/provider/base_url for the runner. + + Detached jobs inherit the user's configured delegation runtime first, then + the main model runtime. Do not hardcode a provider here: providers imply + cost/privacy, and this tool should preserve the user's existing choice. + """ + import yaml + config_path = get_hermes_home() / "config.yaml" + if not config_path.exists(): + return {} + cfg = yaml.safe_load(config_path.read_text()) or {} + model_cfg = cfg.get("model", {}) + if not isinstance(model_cfg, dict): + model_cfg = {"default": model_cfg} if isinstance(model_cfg, str) else {} + delegation_cfg = cfg.get("delegation", {}) + if not isinstance(delegation_cfg, dict): + delegation_cfg = {} + spec = { + "model": delegation_cfg.get("model") or model_cfg.get("default"), + "provider": delegation_cfg.get("provider") or model_cfg.get("provider"), + "base_url": delegation_cfg.get("base_url") or model_cfg.get("base_url"), + "api_mode": delegation_cfg.get("api_mode") or model_cfg.get("api_mode"), + } + return {k: v for k, v in spec.items() if v} + + +# --------------------------------------------------------------------------- +# Tool schema +# --------------------------------------------------------------------------- + +RESEARCH_JOB_SCHEMA = { + "name": "research_job", + "description": ( + "Start, monitor, or resume a long-running research job as a detached OS process. " + "Use this instead of run_research when the loop may take longer than a single " + "agent turn (e.g. >5 minutes). Jobs are durable: state is checkpointed to disk " + "after every round, and can be resumed if the process crashes.\n\n" + "USE WHEN:\n" + "- A research task needs multiple iterations and may take 10+ minutes\n" + "- You cannot afford to keep a foreground agent alive as a watcher\n\n" + "NOT FOR:\n" + "- One-shot tasks (use delegate_task directly)\n" + "- Tasks that fit in a single agent turn (use run_research)\n\n" + "IMPORTANT: This tool spawns a background process. Poll progress with " + "research_job(action='status', job_id=), collect the final result " + "with research_job(action='collect', job_id=), or wait for the " + "process completion notification." + ), + "parameters": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": ["start", "status", "collect", "resume"], + "description": "Operation to perform on the research job.", + }, + "job_id": { + "type": "string", + "description": "Job identifier. Required for status, collect, resume. Generated on start if omitted.", + }, + "topic": { + "type": "string", + "description": "What to research. Required for start.", + }, + "deliverable": { + "type": "string", + "description": "Concrete output the worker must produce. Required for start.", + }, + "metric_key": { + "type": "string", + "description": "Name of the metric to optimize. Required for start.", + }, + "metric_direction": { + "type": "string", + "enum": ["maximize", "minimize"], + "description": "Whether higher or lower metric values are better. Default: maximize.", + }, + "task_type": { + "type": "string", + "enum": ["code", "search", "research", "generic"], + "description": "Task domain. Default: generic.", + }, + "evaluation_mode": { + "type": "string", + "enum": ["self_report", "llm_judge"], + "description": "How to score worker output. Default: self_report.", + }, + "evaluation_prompt": { + "type": "string", + "description": "For llm_judge mode: scoring rubric.", + }, + "max_iterations": { + "type": "integer", + "description": "Max improvement iterations after baseline. Default: 3.", + }, + "time_budget_sec": { + "type": "integer", + "description": "Time budget per worker invocation in seconds. Default: 0 (unlimited).", + }, + "kanban_task_id": { + "type": "string", + "description": ( + "Optional kanban task id (existing task) for round-by-round " + "progress comments. Caller must create the task; the job " + "does not auto-create." + ), + }, + "initial_attempt": { + "type": "string", + "description": "Optional starting scaffold for the worker.", + }, + "acceptance_criterion": { + "type": "string", + "description": "Optional acceptance criterion (e.g. pass_rate >= 0.95).", + }, + "timeout_sec": { + "type": "integer", + "description": "Wall-clock timeout for the entire job in seconds. Default: 0 (unlimited).", + }, + }, + "required": ["action"], + }, +} + + +# --------------------------------------------------------------------------- +# Actions +# --------------------------------------------------------------------------- + +def _action_start(args: dict[str, Any]) -> str: + job_id = args.get("job_id") or secrets.token_hex(8) + cfg = _load_config_for_job() + + spec = { + "job_id": job_id, + "topic": args.get("topic", ""), + "deliverable": args.get("deliverable", ""), + "metric_key": args.get("metric_key", ""), + "metric_direction": args.get("metric_direction", "maximize"), + "task_type": args.get("task_type", "generic"), + "evaluation_mode": args.get("evaluation_mode", "self_report"), + "evaluation_prompt": args.get("evaluation_prompt", ""), + "max_iterations": args.get("max_iterations", 3), + "time_budget_sec": args.get("time_budget_sec", 0), + "kanban_task_id": args.get("kanban_task_id"), + "initial_attempt": args.get("initial_attempt", ""), + "acceptance_criterion": args.get("acceptance_criterion", ""), + "timeout_sec": args.get("timeout_sec", 0), + "toolsets": ["research", "terminal", "file", "web"], + } + spec.update(cfg) + spec_path = _write_job_spec(job_id, spec) + job_dir = _job_dir(job_id) + + hermes_root = get_hermes_home() / "hermes-agent" + python_bin = hermes_root / "venv" / "bin" / "python" + if not python_bin.exists(): + python_bin = hermes_root / ".venv" / "bin" / "python" + if not python_bin.exists(): + python_bin = Path(sys.executable) + cmd = ( + f"HERMES_YOLO_MODE=1 {shlex.quote(str(python_bin))} " + f"-m agent.research.job_runner {shlex.quote(str(spec_path))}" + ) + + # Spawn via terminal_tool in background + from tools.terminal_tool import terminal_tool + raw = terminal_tool( + command=cmd, + background=True, + notify_on_complete=True, + workdir=str(hermes_root), + ) + proc = json.loads(raw) if isinstance(raw, str) else raw + + state = { + "job_id": job_id, + "status": "queued", + "process_session_id": proc.get("session_id"), + "pid": proc.get("pid"), + "job_dir": str(job_dir), + "spec_path": str(spec_path), + } + (job_dir / "state.json").write_text(json.dumps(state, indent=2)) + + return json.dumps({ + "ok": True, + "job_id": job_id, + "status": "queued", + "message": f"Research job {job_id} queued. Poll with research_job_status or wait for completion notification.", + "job_dir": str(job_dir), + "process_session_id": proc.get("session_id"), + }, indent=2) + + +def _action_status(args: dict[str, Any]) -> str: + job_id = args.get("job_id", "") + if not job_id: + return tool_error("job_id is required for status") + + job_dir = _job_dir(job_id) + state_path = job_dir / "state.json" + if not state_path.exists(): + return json.dumps({"ok": False, "error": f"Job {job_id} not found"}, indent=2) + + state = json.loads(state_path.read_text()) + + # If still running, also poll the background process + if state.get("status") in ("queued", "running"): + session_id = state.get("process_session_id") + if session_id: + try: + from tools.process_registry import process + proc_info = process(action="poll", session_id=session_id) + state["process_alive"] = proc_info.get("status") == "running" + state["process_uptime_seconds"] = proc_info.get("uptime_seconds") + except Exception: + state["process_alive"] = False + + # HRM-95: heartbeat-based liveness. If the parent job_runner died + # without writing a terminal status (OOM, kill -9, host reboot), + # the heartbeat file goes cold within 90 s. Surface that as + # status="stale" so callers stop waiting for a job that won't + # progress. We only mark — we don't rewrite state.json from here + # because _action_status is read-only by contract. + try: + from tools.research_tool import check_research_stale + if check_research_stale(str(job_dir)): + state["status"] = "stale" + state["stale_reason"] = "no heartbeat for >90s" + except Exception: + pass + + # Include latest metric if available + history_path = job_dir / "history.json" + if history_path.exists(): + try: + history = json.loads(history_path.read_text()) + best = history.get("best") + if best: + state["best_metric"] = best.get("primary_metric") + state["best_iteration"] = best.get("iteration") + except Exception: + pass + + return json.dumps({"ok": True, **state}, indent=2) + + +def _action_collect(args: dict[str, Any]) -> str: + job_id = args.get("job_id", "") + if not job_id: + return tool_error("job_id is required for collect") + + job_dir = _job_dir(job_id) + result_path = job_dir / "result.json" + state_path = job_dir / "state.json" + + if not result_path.exists(): + status = "unknown" + if state_path.exists(): + status = json.loads(state_path.read_text()).get("status", "unknown") + return json.dumps({ + "ok": False, + "error": f"Result not ready. Job status: {status}", + "job_id": job_id, + }, indent=2) + + result = json.loads(result_path.read_text()) + return json.dumps({"ok": True, "job_id": job_id, **result}, indent=2) + + +def _action_resume(args: dict[str, Any]) -> str: + job_id = args.get("job_id", "") + if not job_id: + return tool_error("job_id is required for resume") + + job_dir = _job_dir(job_id) + state_path = job_dir / "state.json" + spec_path = job_dir / "job.json" + history_path = job_dir / "history.json" + + if not state_path.exists() or not spec_path.exists(): + return json.dumps({"ok": False, "error": f"Job {job_id} not found"}, indent=2) + + state = json.loads(state_path.read_text()) + if state.get("status") not in ("interrupted", "failed"): + return json.dumps({ + "ok": False, + "error": f"Cannot resume job in status '{state.get('status')}'. Only interrupted or failed jobs can be resumed." + }, indent=2) + + # Mark as resuming and re-launch + state["status"] = "resuming" + state_path.write_text(json.dumps(state, indent=2)) + + hermes_root = get_hermes_home() / "hermes-agent" + python_bin = hermes_root / "venv" / "bin" / "python" + if not python_bin.exists(): + python_bin = hermes_root / ".venv" / "bin" / "python" + if not python_bin.exists(): + python_bin = Path(sys.executable) + cmd = ( + f"HERMES_YOLO_MODE=1 {shlex.quote(str(python_bin))} " + f"-m agent.research.job_runner {shlex.quote(str(spec_path))}" + ) + + from tools.terminal_tool import terminal_tool + raw = terminal_tool( + command=cmd, + background=True, + notify_on_complete=True, + workdir=str(hermes_root), + ) + proc = json.loads(raw) if isinstance(raw, str) else raw + + state["status"] = "queued" + state["process_session_id"] = proc.get("session_id") + state["pid"] = proc.get("pid") + state["resumed_at"] = time.time() + state_path.write_text(json.dumps(state, indent=2)) + + return json.dumps({ + "ok": True, + "job_id": job_id, + "status": "queued", + "message": f"Research job {job_id} resumed.", + "process_session_id": proc.get("session_id"), + }, indent=2) + + +# --------------------------------------------------------------------------- +# Tool handler +# --------------------------------------------------------------------------- + +def research_job( + action: str, + job_id: str = "", + topic: str = "", + deliverable: str = "", + metric_key: str = "", + metric_direction: str = "maximize", + task_type: str = "generic", + evaluation_mode: str = "self_report", + evaluation_prompt: str = "", + max_iterations: int = 3, + time_budget_sec: int = 0, + kanban_task_id: str = "", + initial_attempt: str = "", + acceptance_criterion: str = "", + timeout_sec: int = 0, + **_: Any, +) -> str: + if action == "start": + if not topic or not deliverable or not metric_key: + return tool_error("topic, deliverable, and metric_key are required for start") + return _action_start(locals()) + elif action == "status": + return _action_status(locals()) + elif action == "collect": + return _action_collect(locals()) + elif action == "resume": + return _action_resume(locals()) + else: + return tool_error(f"Unknown action: {action}") + + +# --------------------------------------------------------------------------- +# Registry +# --------------------------------------------------------------------------- + +registry.register( + name="research_job", + toolset="research", + schema=RESEARCH_JOB_SCHEMA, + handler=lambda args, **kw: research_job(**args), + emoji="📋", +) diff --git a/tools/research_tool.py b/tools/research_tool.py new file mode 100644 index 000000000000..9bc2129585c2 --- /dev/null +++ b/tools/research_tool.py @@ -0,0 +1,482 @@ +"""run_research — iterative self-improving research loop tool. + +Exposes ResearchSupervisor as a tool callable by the LLM, following the +same pattern as delegate_task. The LLM calls run_research when a task +benefits from multiple iterations scored against a measurable metric. + +Autogenesis AOOR loop: Act → Observe → Optimize → Remember. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import time +from pathlib import Path +from types import SimpleNamespace +from typing import Any, Optional + +# Import at module scope so unittest.mock.patch("tools.research_tool.ResearchSupervisor") +# resolves correctly. Audit fix #5: previously imported inside run_research, +# which broke patch-based tests. +from agent.research.supervisor import ResearchSupervisor, TaskSpec + +logger = logging.getLogger(__name__) + +# --------------------------------------------------------------------------- +# Tool schema +# --------------------------------------------------------------------------- + +RESEARCH_TOOL_SCHEMA = { + "name": "run_research", + "description": ( + "Run a self-improving research loop on any task with a measurable deliverable. " + "Spawns worker subagents iteratively, scores their output against a metric, " + "and applies LLM-guided hypothesis revision to improve the metric across rounds.\n\n" + "USE WHEN:\n" + "- A task requires iterative improvement toward a measurable quality criterion\n" + "- You need web research, code optimization, or synthesis with self-evaluation\n" + "- Single-shot delegate_task is not enough — the task benefits from multiple rounds\n\n" + "NOT FOR:\n" + "- One-shot tasks (use delegate_task directly)\n" + "- Tasks with no measurable metric (use delegate_task)\n\n" + "IMPORTANT: This tool spawns multiple subagents and can run for several minutes. " + "Inform the user before calling it." + ), + "parameters": { + "type": "object", + "properties": { + "topic": { + "type": "string", + "description": "What to research or accomplish. Be specific.", + }, + "deliverable": { + "type": "string", + "description": ( + "Concrete output the worker must produce. " + "E.g. 'Python class with insert/search/delete', " + "'ranked list of papers with abstracts and relevance scores'." + ), + }, + "metric_key": { + "type": "string", + "description": ( + "Name of the metric to optimize. " + "E.g. 'pass_rate', 'relevance_score', 'completeness_score', 'latency_ms'." + ), + }, + "metric_direction": { + "type": "string", + "enum": ["maximize", "minimize"], + "description": "Whether higher or lower metric values are better. Default: maximize.", + }, + "task_type": { + "type": "string", + "enum": ["code", "search", "research", "generic"], + "description": ( + "Task domain. Controls worker brief template and default toolsets. " + "code=terminal+file, search/research=web+file, generic=terminal+file." + ), + }, + "acceptance_criterion": { + "type": "string", + "description": ( + "Optional stopping criterion. When parseable as " + "' ' (op: >=, <=, >, <, ==), the loop " + "exits as soon as the latest iteration's metric satisfies it. " + "Qualitative criteria (free-form text) are passed to the worker " + "via the brief but do not auto-terminate the loop. " + "E.g. 'pass_rate >= 0.95', 'latency_ms < 200'." + ), + }, + "evaluation_mode": { + "type": "string", + "enum": ["self_report", "llm_judge"], + "description": ( + "How to score worker output. " + "self_report: worker emits METRIC line. " + "llm_judge: supervisor scores the deliverable externally using evaluation_prompt." + ), + }, + "evaluation_prompt": { + "type": "string", + "description": ( + "For llm_judge mode: scoring rubric. " + "E.g. 'Score 0-1: does this paper list cover attention mechanisms published after 2022?'" + ), + }, + "initial_attempt": { + "type": "string", + "description": ( + "Optional starting deliverable or scaffold. " + "For code tasks: skeleton code. For research: initial outline. " + "Leave empty to let the worker start from scratch." + ), + }, + "max_iterations": { + "type": "integer", + "description": "Max improvement iterations after baseline (default: 3). Each spawns a worker.", + }, + "time_budget_sec": { + "type": "integer", + "description": "Time budget per worker invocation in seconds (default: 300).", + }, + "kanban_task_id": { + "type": "string", + "description": ( + "Optional kanban task id (existing task). When set, run " + "progress posts as comments and the task is transitioned " + "to 'done' on completion. Caller must create the task; " + "the tool does not auto-create." + ), + }, + "strategies": { + "type": "array", + "description": ( + "Optional A/B test strategies. If provided, runs each strategy " + "and returns a comparison table instead of a single run. " + "Each item is an object with: name, fan_out (int), use_moa (bool), max_iterations (int)." + ), + "items": { + "type": "object", + "properties": { + "name": {"type": "string"}, + "fan_out": {"type": "integer", "default": 1}, + "use_moa": {"type": "boolean", "default": True}, + "max_iterations": {"type": "integer", "default": 3}, + }, + "required": ["name"], + }, + }, + "repeats": { + "type": "integer", + "description": "Number of repeats per strategy when running A/B tests (default: 1).", + "default": 1, + }, + "disable_evolution_overlay": { + "type": "boolean", + "description": ( + "If true, do not prepend cross-run lessons from " + "$HERMES_HOME/evolution to the worker brief. Useful for " + "isolated tests, CI runs, or first-time tasks where the " + "global lesson store would only add noise. Default: false." + ), + "default": False, + }, + "auto_specify": { + "type": "boolean", + "description": ( + "When true and deliverable/metric_key are missing, call " + "the kanban triage_specifier auxiliary LLM to flesh out " + "the TaskSpec from the topic alone. Empty fields only — " + "explicit caller values are never overridden. Falls back " + "to the original args when the aux LLM is unavailable. " + "Default: false." + ), + "default": False, + }, + }, + "required": ["topic"], + }, +} + + +# --------------------------------------------------------------------------- +# LLM bridge — wraps auxiliary_client.call_llm to match supervisor's Protocol +# --------------------------------------------------------------------------- + +class _LLMBridge: + """Adapter: auxiliary_client.call_llm → _ChatClient Protocol expected by ResearchSupervisor.""" + + def chat(self, messages: list[dict[str, str]], *, system: str | None = None) -> Any: + from agent.auxiliary_client import call_llm + + full_messages: list[dict[str, str]] = [] + if system: + full_messages.append({"role": "system", "content": system}) + full_messages.extend(messages) + + try: + resp = call_llm(messages=full_messages, max_tokens=4096) + text = resp.choices[0].message.content or "" + except Exception as exc: + logger.warning("_LLMBridge.chat failed: %s", exc) + text = "" + + return SimpleNamespace(content=text) + + +# --------------------------------------------------------------------------- +# Tool handler +# --------------------------------------------------------------------------- + +def run_research( + topic: str, + deliverable: str = "", + metric_key: str = "", + metric_direction: Optional[str] = None, + task_type: Optional[str] = None, + acceptance_criterion: str = "", + evaluation_mode: Optional[str] = None, + evaluation_prompt: str = "", + initial_attempt: str = "", + max_iterations: int = 3, + time_budget_sec: int = 0, + kanban_task_id: Optional[str] = None, + parent_agent: Any = None, + checkpoint_dir: Optional[str] = None, + timeout_sec: int = 0, + strategies: Optional[list[dict[str, Any]]] = None, + repeats: int = 1, + disable_evolution_overlay: bool = False, + auto_specify: bool = False, +) -> str: + if parent_agent is None: + return json.dumps({"error": "run_research requires a parent_agent context."}) + + from hermes_constants import get_hermes_home + + # Phase C — auto-specify a vague topic when caller left the + # scaffolding fields empty/None. Only fills empty fields; never + # overrides explicit caller values. Falls back to the original args + # when the aux LLM is unavailable or returns unparseable output. + if auto_specify and (not deliverable or not metric_key): + from agent.research.auto_specify import auto_specify_topic + scaffold = auto_specify_topic(topic) + if scaffold: + if not deliverable: + deliverable = str(scaffold.get("deliverable") or "") + if not metric_key: + metric_key = str(scaffold.get("metric_key") or "") + if metric_direction is None: + metric_direction = scaffold.get("metric_direction") or None + if task_type is None: + task_type = scaffold.get("task_type") or None + if evaluation_mode is None: + evaluation_mode = scaffold.get("evaluation_mode") or None + if not evaluation_prompt: + evaluation_prompt = str(scaffold.get("evaluation_prompt") or "") + else: + logger.warning( + "auto_specify: topic %r could not be fleshed out; running with " + "the original (possibly empty) args.", topic, + ) + + # Normalize defaults AFTER auto_specify so we don't conflate an + # auto-filled value with a caller-supplied one above. + if metric_direction is None: + metric_direction = "maximize" + if task_type is None: + task_type = "generic" + if evaluation_mode is None: + evaluation_mode = "self_report" + + spec = TaskSpec( + topic=topic, + deliverable=deliverable, + metric_key=metric_key, + metric_direction=metric_direction, + task_type=task_type, + acceptance_criterion=acceptance_criterion, + evaluation_mode=evaluation_mode, + evaluation_prompt=evaluation_prompt, + ) + + run_id = hashlib.sha1(f"{topic}:{time.time()}".encode()).hexdigest()[:12] + workspace = get_hermes_home() / "research-workspace" + + # Build the progress sink. Kanban (when task_id set) > Stub (default). + # db_path is captured NOW so subsequent KanbanSink calls don't re-resolve + # the active board mid-run. + if kanban_task_id: + from hermes_cli import kanban_db + from agent.research.sinks import KanbanSink + try: + db_path = kanban_db.kanban_db_path() + sink = KanbanSink(task_id=kanban_task_id, db_path=db_path) + except Exception as exc: + logger.warning( + "Failed to resolve kanban db_path for task %s: %s. " + "Falling back to log-only sink.", kanban_task_id, exc, + ) + from agent.research.sinks import StubSink + sink = StubSink() + else: + from agent.research.sinks import StubSink + sink = StubSink() + + # A/B testing path + if strategies: + from agent.research.ab_testing import ResearchABTester, StrategyConfig + + strategy_configs = [ + StrategyConfig( + name=s.get("name", f"strategy-{i}"), + fan_out=s.get("fan_out", 1), + use_moa=s.get("use_moa", True), + max_iterations=s.get("max_iterations", max_iterations), + time_budget_sec=time_budget_sec, + ) + for i, s in enumerate(strategies) + ] + + tester = ResearchABTester( + parent_agent=parent_agent, + workspace=workspace, + progress_sink=sink, + llm=_LLMBridge(), + ) + try: + summaries = tester.compare( + spec, + strategy_configs, + initial_attempt=initial_attempt, + repeats=repeats, + run_prefix=run_id, + ) + except Exception as exc: + logger.exception("A/B test failed for run_id=%s: %s", run_id, exc) + return json.dumps({"error": str(exc), "run_id": run_id}) + + return json.dumps({ + "run_id": run_id, + "ab_test": True, + "report": tester.format_report(summaries), + "json": json.loads(tester.to_json(summaries)), + }, indent=2) + + # Single-run path + supervisor = ResearchSupervisor( + parent_agent=parent_agent, + workspace=workspace, + progress_sink=sink, + ) + + try: + history = supervisor.run( + spec, + initial_attempt=initial_attempt, + run_id=run_id, + max_iterations=max_iterations, + time_budget_sec=time_budget_sec, + llm=_LLMBridge(), + checkpoint_dir=Path(checkpoint_dir) if checkpoint_dir else None, + disable_evolution_overlay=disable_evolution_overlay, + ) + except Exception as exc: + logger.exception("run_research failed for run_id=%s: %s", run_id, exc) + return json.dumps({"error": str(exc), "run_id": run_id}) + + best = history.best_result + best_notes = "" + if best and best.stdout: + import re + m = re.search(r"NOTES:\s*(.+)", best.stdout) + best_notes = m.group(1).strip() if m else "" + + # Aggregate cost accounting across iterations + iteration_costs = [] + total_tokens_in = 0 + total_tokens_out = 0 + total_cost_usd = 0.0 + for r in history.results: + iteration_costs.append({ + "iteration": r.iteration, + "tokens_in": r.tokens_in, + "tokens_out": r.tokens_out, + "cost_usd": round(r.cost_usd, 6), + "primary_metric": r.primary_metric, + "improved": r.improved, + "kept": r.kept, + }) + total_tokens_in += r.tokens_in + total_tokens_out += r.tokens_out + total_cost_usd += r.cost_usd + + return json.dumps({ + "run_id": run_id, + "iterations": len(history.results), + "best_metric": best.primary_metric if best else None, + "metric_key": metric_key, + "metric_direction": metric_direction, + "best_notes": best_notes, + "workspace": str(workspace / run_id), + "learnings_file": str(workspace / run_id / "learnings.jsonl"), + "iteration_costs": iteration_costs, + "total_tokens_in": total_tokens_in, + "total_tokens_out": total_tokens_out, + "total_cost_usd": round(total_cost_usd, 6), + "total_iterations": len(history.results), + }, indent=2) + + +def check_research_stale(checkpoint_dir: str, stale_threshold_sec: float = 90.0) -> bool: + """Return True if the research job at checkpoint_dir has no recent heartbeat. + + The heartbeat file is ``/heartbeat.json``, written by + ``agent.research.job_runner._child_main`` every 30 s with the schema + ``{"ts": , "pid": }``. A job is stale when: + + * the file is missing, or + * the file is unreadable / malformed, or + * ``now - ts`` exceeds ``stale_threshold_sec`` (default 90 s = 3 + missed beats, tolerating one GC pause / slow disk). + + Used by ``tools/research_job_tool._action_status`` to mark dead + detached jobs and by the parent in job_runner to decide when to kill + a stuck child. + """ + hb = Path(checkpoint_dir) / "heartbeat.json" + if not hb.exists(): + return True + try: + data = json.loads(hb.read_text(encoding="utf-8")) + ts = float(data.get("ts", 0)) + except (OSError, json.JSONDecodeError, TypeError, ValueError): + return True + return (time.time() - ts) > stale_threshold_sec + + +# --------------------------------------------------------------------------- +# Registry +# --------------------------------------------------------------------------- + +from tools.registry import registry, tool_error # noqa: E402 + + +def _check_research_requirements() -> bool: + try: + from agent.research.supervisor import ResearchSupervisor # noqa: F401 + return True + except ImportError: + return False + + +registry.register( + name="run_research", + toolset="research", + schema=RESEARCH_TOOL_SCHEMA, + handler=lambda args, **kw: run_research( + topic=args.get("topic", ""), + deliverable=args.get("deliverable", ""), + metric_key=args.get("metric_key", ""), + metric_direction=args.get("metric_direction", "maximize"), + task_type=args.get("task_type", "generic"), + acceptance_criterion=args.get("acceptance_criterion", ""), + evaluation_mode=args.get("evaluation_mode", "self_report"), + evaluation_prompt=args.get("evaluation_prompt", ""), + initial_attempt=args.get("initial_attempt", ""), + max_iterations=args.get("max_iterations", 3), + time_budget_sec=args.get("time_budget_sec", 0), + kanban_task_id=args.get("kanban_task_id"), + parent_agent=kw.get("parent_agent"), + checkpoint_dir=args.get("checkpoint_dir"), + strategies=args.get("strategies"), + repeats=args.get("repeats", 1), + disable_evolution_overlay=args.get("disable_evolution_overlay", False), + auto_specify=args.get("auto_specify", False), + ), + check_fn=_check_research_requirements, + emoji="🔬", +) diff --git a/tools/terminal_tool.py b/tools/terminal_tool.py index f20f2abcbb50..5d9caa53ce6d 100644 --- a/tools/terminal_tool.py +++ b/tools/terminal_tool.py @@ -2606,7 +2606,7 @@ def check_terminal_requirements() -> bool: print(f" TERMINAL_CWD: {os.getenv('TERMINAL_CWD', _safe_getcwd())}") from hermes_constants import display_hermes_home as _dhh print(f" TERMINAL_SANDBOX_DIR: {os.getenv('TERMINAL_SANDBOX_DIR', f'{_dhh()}/sandboxes')}") - print(f" TERMINAL_TIMEOUT: {os.getenv('TERMINAL_TIMEOUT', '60')}") + print(f" TERMINAL_TIMEOUT: {os.getenv('TERMINAL_TIMEOUT', '180')}") print(f" TERMINAL_LIFETIME_SECONDS: {os.getenv('TERMINAL_LIFETIME_SECONDS', '300')}") diff --git a/toolsets.py b/toolsets.py index 5c67bfb21148..f0edc098273d 100644 --- a/toolsets.py +++ b/toolsets.py @@ -55,8 +55,8 @@ "session_search", # Clarifying questions "clarify", - # Code execution + delegation - "execute_code", "delegate_task", + # Code execution + delegation + research loop + "execute_code", "delegate_task", "run_research", "research_job", # Cronjob management "cronjob", # Cross-platform messaging (gated on gateway running via check_fn) @@ -170,6 +170,12 @@ "includes": [] }, + "research": { + "description": "Iterative self-improving research loop: run_research spawns worker subagents, scores output against a metric, and applies LLM-guided hypothesis revision across iterations. research_job is the detached, resumable variant for long-running loops (>5min).", + "tools": ["run_research", "research_job"], + "includes": [] + }, + "browser": { "description": "Browser automation for web interaction (navigate, click, type, scroll, iframes, hold-click) with web search for finding URLs", "tools": [ @@ -277,6 +283,17 @@ "includes": [], }, + "minecraft": { + "description": "Minecraft embodied agent tools — perceive, navigate, build, craft, combat, manage, screenshot, command, story, registry", + "tools": [ + "mc_perceive", "mc_move", "mc_mine", "mc_build", + "mc_craft", "mc_combat", "mc_manage", "mc_plan", + "mc_screenshot", "mc_command", "mc_story", "mc_registry", + "mc_chat", "mc_no_op", + ], + "includes": [], + }, + "discord": { "description": "Discord read and participate tools (fetch messages, search members, create threads)", "tools": ["discord"], diff --git a/ui-tui/src/app/interfaces.ts b/ui-tui/src/app/interfaces.ts index f7297c151da3..7ae6c5124cd6 100644 --- a/ui-tui/src/app/interfaces.ts +++ b/ui-tui/src/app/interfaces.ts @@ -119,6 +119,7 @@ export interface UiState { compact: boolean detailsMode: DetailsMode detailsModeCommandOverride: boolean + historyNavRequiresEmptyInput: boolean info: null | SessionInfo liveSessionCount: number inlineDiffs: boolean diff --git a/ui-tui/src/app/uiStore.ts b/ui-tui/src/app/uiStore.ts index 470f4264b941..817558341958 100644 --- a/ui-tui/src/app/uiStore.ts +++ b/ui-tui/src/app/uiStore.ts @@ -13,6 +13,7 @@ const buildUiState = (): UiState => ({ compact: false, detailsMode: 'collapsed', detailsModeCommandOverride: false, + historyNavRequiresEmptyInput: false, indicatorStyle: DEFAULT_INDICATOR_STYLE, info: null, liveSessionCount: 0, diff --git a/ui-tui/src/app/useConfigSync.ts b/ui-tui/src/app/useConfigSync.ts index f159bbbd17bc..af7420058014 100644 --- a/ui-tui/src/app/useConfigSync.ts +++ b/ui-tui/src/app/useConfigSync.ts @@ -188,6 +188,7 @@ export const applyDisplay = ( setVoiceRecordKey?: (v: ParsedVoiceRecordKey) => void ) => { const d = cfg?.config?.display ?? {} + const t = cfg?.config?.tui ?? {} setBell(!!d.bell_on_complete) @@ -207,6 +208,7 @@ export const applyDisplay = ( compact: !!d.tui_compact, detailsMode: resolveDetailsMode(d), detailsModeCommandOverride: false, + historyNavRequiresEmptyInput: !!t.history_nav_requires_empty_input, indicatorStyle: normalizeIndicatorStyle(d.tui_status_indicator), inlineDiffs: d.inline_diffs !== false, mouseTracking: normalizeMouseTracking(d), diff --git a/ui-tui/src/app/useInputHandlers.ts b/ui-tui/src/app/useInputHandlers.ts index 4e8dac7e3c23..88cda7148720 100644 --- a/ui-tui/src/app/useInputHandlers.ts +++ b/ui-tui/src/app/useInputHandlers.ts @@ -439,6 +439,10 @@ export function useInputHandlers(ctx: InputHandlerContext): InputHandlerResult { !cState.input || (cursor !== null && cState.input.lastIndexOf('\n', Math.max(0, cursor - 1)) < 0) if (noLineAbove) { + if (getUiState().historyNavRequiresEmptyInput && cState.input) { + return + } + cycleQueue(1) || cycleHistory(-1) return @@ -450,7 +454,17 @@ export function useInputHandlers(ctx: InputHandlerContext): InputHandlerResult { const cursor = inputSel && inputSel.start === inputSel.end ? inputSel.start : null const noLineBelow = !cState.input || (cursor !== null && cState.input.indexOf('\n', cursor) < 0) - if (noLineBelow || cState.historyIdx !== null) { + if (cState.historyIdx !== null) { + cycleQueue(-1) || cycleHistory(1) + + return + } + + if (noLineBelow) { + if (getUiState().historyNavRequiresEmptyInput && cState.input) { + return + } + cycleQueue(-1) || cycleHistory(1) return diff --git a/ui-tui/src/gatewayTypes.ts b/ui-tui/src/gatewayTypes.ts index 00a3b458911f..5aa2b8b7ebf8 100644 --- a/ui-tui/src/gatewayTypes.ts +++ b/ui-tui/src/gatewayTypes.ts @@ -98,8 +98,12 @@ export interface ConfigVoiceConfig { record_key?: unknown } +export interface ConfigTuiConfig { + history_nav_requires_empty_input?: boolean +} + export interface ConfigFullResponse { - config?: { display?: ConfigDisplayConfig; voice?: ConfigVoiceConfig; paste_collapse_threshold?: number; paste_collapse_char_threshold?: number } + config?: { display?: ConfigDisplayConfig; voice?: ConfigVoiceConfig; tui?: ConfigTuiConfig; paste_collapse_threshold?: number; paste_collapse_char_threshold?: number } } export interface ConfigMtimeResponse { diff --git a/website/docs/user-guide/features/kanban-ship-review.md b/website/docs/user-guide/features/kanban-ship-review.md new file mode 100644 index 000000000000..5890b259bd8a --- /dev/null +++ b/website/docs/user-guide/features/kanban-ship-review.md @@ -0,0 +1,145 @@ +--- +sidebar_position: 13 +title: "Ship Review (kanban review)" +description: "Create durable review graphs for code changes with safe triage, ready dispatch, and REVIEW-ONLY contracts" +--- + +# Ship Review — Kanban Review Graphs + +`hermes kanban review create` builds a durable 5-card review graph for any git change. It replaces ad-hoc "hey can someone review this?" messages with a structured, tracked, and replayable workflow. + +## The graph shape + +``` +Parent review card (organisational umbrella) +├─ [REVIEW] Code quality ─┐ +├─ [REVIEW] Security │ parallel reviewers +└─ [REVIEW] Test coverage ─┘ + │ + ▼ +[SYNTHESIS] GO/NO-GO decision ← gated on all three reviewers +``` + +1. **Parent card** — holds the base..head context and the REVIEW-ONLY contract. +2. **Three reviewers** — run in parallel, each with a role-specific checklist. +3. **Synthesis** — auto-promotes to `ready` once all reviewers finish. It reads their handoffs and produces a GO/NO-GO decision. + +## Safe triage by default + +By default every card is created in `triage`: + +```bash +hermes kanban review create "Review PR #42" \ + --base nousmain \ + --head feat/auth \ + --repo /home/me/Projects/myapp \ + --assignee miki +``` + +Nothing dispatches until a human explicitly promotes cards. This is the safe pattern for reviews that need scheduling or human triage. + +## Ready dispatch + +If you want the reviewers to start immediately: + +```bash +hermes kanban review create "Review PR #42" \ + --base nousmain \ + --head feat/auth \ + --repo /home/me/Projects/myapp \ + --assignee miki \ + --ready +``` + +With `--ready`: +- Parent + reviewer cards start in `ready` (dispatcher picks them up on next tick). +- Synthesis card starts in `todo` because its parents (the reviewers) are not yet `done`. +- As each reviewer completes, `kanban_db` auto-runs `recompute_ready`. +- When the third reviewer finishes, the synthesis auto-promotes from `todo` → `ready`. + +## Local Miki example + +A concrete invocation on the Hermes repo itself, using `--json` for scripting: + +```bash +hermes kanban review create "Ship kanban review orchestration" \ + --base nousmain \ + --head feat/kanban-ship-review-orchestration \ + --repo /home/nicolas/Projects/hermes-agent \ + --assignee miki \ + --triage \ + --json +``` + +Output: +```json +{ + "parent_id": "t_a1b2c3d4", + "reviewer_ids": ["t_e5f6g7h8", "t_i9j0k1l2", "t_m3n4o5p6"], + "synthesis_id": "t_q7r8s9t0", + "created": true +} +``` + +Rerun the same command and you get the **same IDs** — the graph is idempotent by `sha256(repo realpath) + base + head + role`. + +## Review-only limitation + +Every generated body contains a **REVIEW-ONLY v1** contract: + +> Do NOT modify source code. Report findings as structured metadata only. + +This is intentional. Reviewer workers are scoped to read, analyse, and report. They do not patch, commit, or push. If a reviewer finds a bug, it records the finding in `kanban_complete(metadata={"findings": [...]})` and the synthesis task decides whether to spawn a separate remediation task. + +The contract exists because: +- **Auditability** — a review that silently fixes its own findings is indistinguishable from a no-op. +- **Separation of concerns** — reviewers judge; other agents (or humans) remediate. +- **Safety** — a reviewer with write access could introduce new issues while fixing old ones, especially when running autonomously. + +## JSON CLI output + +Pass `--json` to get machine-readable output: + +```bash +hermes kanban review create "Review PR #42" \ + --base main --head feat/x --repo . --json +``` + +Keys: +- `parent_id` — the organisational umbrella card +- `reviewer_ids` — list of 3 reviewer task ids +- `synthesis_id` — the synthesis task id +- `created` — `true` if new cards were created, `false` if all existed already + +## Idempotency + +The graph is keyed by the **resolved repo path** + **base** + **head** + **role**. Changing any of `base`, `head`, or the absolute repo path creates a new graph. Moving the repo directory (e.g., symlinks that resolve differently) also creates a new graph — use stable absolute paths in automation. + +## Skills + +Attach skills to every card with `--skill` (repeatable): + +```bash +hermes kanban review create "Review auth PR" \ + --base main --head feat/auth --repo . \ + --assignee reviewer \ + --skill github-code-review \ + --skill security-pr-audit +``` + +These are force-loaded into the worker alongside the built-in `kanban-worker` skill. + +## Body templates + +Each role gets a hardened body with: +- Exact `git diff` commands to run +- Severity labels (**Critical**, **Important**, **Optional/Nit**) +- Role-specific checklist (code-quality, security, test-coverage) +- `kanban_complete` / `kanban_block` contract with expected metadata shape + +The synthesis body expects: +- GO/NO-GO decision with rationale +- Blockers, recommended fixes, acknowledged risks +- Rollback plan and evidence reviewed + +Bodies are self-contained — a worker can execute the review without conversation history or external context.