Port from superagent-ai/grok-cli: verify subsystem (run-recipe detection + environment manifest + hermes verify smoke runner)

[teknium1]

hermes verify answers one question: does this project actually run? One command: detect the framework → install → build → test → boot the server → poll readiness → structured pass/fail. It ports the verify subsystem from superagent-ai/grok-cli, rescoped to fill exactly the gap Hermes has — runtime smoke verification — and to plug into the verification stack Hermes already ships instead of standing beside it.

Following the AGENTS.md footprint ladder, this lands as a CLI command — zero model-tool footprint.

How it fits what Hermes already has

Hermes already has three verification pieces this PR integrates with (and deliberately does not duplicate):

1. Project facts (agent/coding_context.py — detect_project_facts() / project_facts_for()): cheap, byte-stable detection of manifests, package managers, and test/lint/build verify commands. Feeds the system-prompt workspace snapshot, the verify-on-stop nudge, and the desktop verify UI. Unchanged — it stays the prompt-time source of truth.
2. Evidence ledger (agent/verification_evidence.py): passive record of what the agent actually proved — passing scripts/run_tests.sh or canonical test commands mark a workspace passed.
3. Verify-on-stop (agent/verification_stop.py + conversation_loop.py): when the model ends a turn after editing code without fresh passing evidence, a bounded synthetic nudge lists the changed paths and the detected verify commands (max 2 attempts, surface-aware agent.verify_on_stop: auto).

What was missing: all of that is test/lint/build only. Nothing could infer a start command, a port, or a readiness path — no way to prove "the server boots and serves HTTP". That runtime layer is what grok-cli's verify subsystem does well, and it's the only thing this PR adds as new machinery.

The closed loop this PR creates:

edit code → verify-on-stop guard fires → nudge suggests `hermes verify --json`
        → hermes verify runs build+test+boot+readiness → records into the evidence ledger
        → workspace marked passed → guard satisfied → turn may end

Concretely:

• hermes verify records into the evidence ledger. New record_verify_run() in agent/verification_evidence.py (explicit CLI-side write — nothing to classify, the caller knows the run's semantics). A passing run marks the workspace passed exactly like a passing scripts/run_tests.sh; a failing run records the failure. Partial runs (--phase subset / --skip-start) are recorded as targeted scope so a partial pass is never presented as full workspace green. Fail-silent: a ledger problem never changes the CLI's exit code or output.
• The verify-on-stop nudge is recipe-aware. When the edited workspace has a runnable recipe (a start command detected, or a saved .hermes/environment.json), the nudge names hermes verify --json as the preferred full check alongside today's verifyCommands wording; otherwise the wording is unchanged. Detection is cheap and wrapped in try/except — a broken detector can never break the nudge path (tested). No new injection points, no prompt changes: the nudge is the same synthetic turn-time message that already existed.
• Detection is unified, not forked. detect_project_facts stays the cheap prompt layer (untouched — byte-stable prompt requirement). Where grok's detection is more comprehensive (framework-specific commands, lockfile-based PM choice across pnpm/bun/yarn/npm, uv/poetry/pipenv, Makefile target parsing), the recipe layer uses it; where the recipe misses commands project facts already promised the model (scripts/run_tests.sh, configured pytest), the CLI's detect path merges them into the recipe's test list. A saved manifest is never merged into — the user-edited file is the source of truth. Layer ownership is documented in the agent/verify/recipes.py module docstring.

Not duplicated: no second project-facts detector, no new evidence store, no new stop-guard mechanism, no new model tool, no system-prompt changes.

Changes

• agent/verify/recipes.py — static detectors ported nearly 1:1 from grok's recipes.ts, mirroring its detection order and command choices: Node frameworks (Next.js, SvelteKit, Astro, Remix, CRA, Vite, generic) with lockfile-based package-manager selection (pnpm-lock.yaml → pnpm, bun.lock/bun.lockb → bun, yarn.lock → yarn, else npm), Python (manage.py/deps → Django, fastapi/uvicorn → FastAPI, flask → Flask, generic w/ uv/poetry/pipenv/pyproject awareness), Go (go.mod), Rust (Cargo.toml), Java (pom.xml → Maven, build.gradle(.kts) → Gradle w/ wrapper detection), Makefile target parsing, docker-compose. Produces a Recipe(name, kind, bootstrap[], build[], test[], start, port, readiness_path) dataclass with a tolerant from_dict normalizer. Module docstring documents the layer split vs agent/coding_context.py.
• agent/verify/environment.py — versioned manifest at <project>/.hermes/environment.json ({version: 1, recipe: {...}, updatedAt}). Loader tolerates missing/malformed files (degrades to fresh detection); a saved manifest wins over detection.
• agent/verify/runner.py — run_verify() executes bootstrap → build → test sequentially (captured output, per-phase timeout), then launches recipe.start in its own process group, polls http://127.0.0.1:<port><readiness_path> (default 60s), and tears down with SIGTERM → SIGKILL escalation (Windows-safe fallback to terminate()/kill()).
• agent/verification_evidence.py — new record_verify_run() (explicit ledger write for hermes verify results) plus a shared _insert_evidence() factored out of record_terminal_result — same insert/state-repoint/prune path for both.
• agent/verification_stop.py — recipe-aware nudge: _workspace_has_runnable_recipe() (manifest exists, or detected recipe has a start command; fail-silent) extends the command instruction to suggest hermes verify --json; the no-canonical-command branch prefers hermes verify over an ad-hoc temp script when a runnable recipe exists.
• hermes_cli/subcommands/verify.py + hermes_cli/verify_cmd.py + hermes_cli/main.py — hermes verify registered like hermes doctor. Flags: --detect-only, --save, --skip-start, --phase bootstrap|build|test|start (repeatable), --port, --timeout, --ready-timeout, --json. On the detect path (not manifest), detect_project_facts verify commands the recipe missed are merged into its test list; on completion the result is recorded into the evidence ledger under HERMES_SESSION_ID when set.
• tests/verify/ — 78 tests: the original 61 (detection fixtures, manifest roundtrip/precedence, runner phases/timeouts, readiness polling against real http.server, CLI paths) plus 17 integration tests in test_ledger_and_nudge_integration.py: ledger recorded on pass/fail, targeted scope for partial runs, HERMES_SESSION_ID attribution, the full closed loop (edit → nudge fires → hermes verify passes → nudge gone), nudge mentions hermes verify --json with a start command or manifest and keeps plain wording without one, detector exceptions stay silent, project-facts merge (added/deduped/never applied to manifests).

Scoped per plan: no browser QA, no sandbox checkpoints, no LLM recipe refinement.

Validation

Check	Result
scripts/run_tests.sh tests/verify/ tests/agent/test_verification_stop.py tests/agent/test_verification_evidence.py	✅ 6 files, 94 passed, 0 failed (4.6s)
ruff check on all changed files	✅ All checks passed
scripts/check-windows-footguns.py on all changed files	✅ No footguns (5 files)
E2E closed loop (tested): edit → nudge → passing hermes verify → verification_status = passed → nudge suppressed	✅ test_passing_verify_run_satisfies_stop_guard
scripts/audit_pr_attribution.py --fix	✅ all contributor emails mapped

Sources

• https://github.com/superagent-ai/grok-cli/blob/main/src/verify/recipes.ts
• https://github.com/superagent-ai/grok-cli/blob/main/src/verify/environment.ts

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on bff34f7}

⚠️ Warnings

OSV vulnerability scan · View job

52 known vulnerabilities found in pinned dependencies.

• CVE-2026-71436 in package-lock.json
• CVE-2026-71436 in website/package-lock.json
• CVE-2026-71437 in package-lock.json
• CVE-2026-71437 in website/package-lock.json
• CVE-2026-13697 in package-lock.json
• CVE-2026-13697 in plugins/platforms/photon/sidecar/package-lock.json
• CVE-2026-13697 in website/package-lock.json
• GHSA-55q2-fjhq-7xh7 in package-lock.json
• GHSA-55q2-fjhq-7xh7 in website/package-lock.json
• GHSA-5p4m-2wfm-xmqj in package-lock.json
• GHSA-5p4m-2wfm-xmqj in website/package-lock.json
• CVE-2026-71554 in uv.lock
• CVE-2026-50159 in package-lock.json
• CVE-2026-50159 in website/package-lock.json
• CVE-2026-18446 in website/package-lock.json
• CVE-2026-54285 in plugins/platforms/photon/sidecar/package-lock.json
• CVE-2026-16728 in package-lock.json
• CVE-2026-16728 in package-lock.json
• CVE-2026-16728 in plugins/platforms/photon/sidecar/package-lock.json
• CVE-2026-16728 in website/package-lock.json

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

---

debug info

CI timings

CI timings · View report · View job

Wall time 4m5s vs 5m26s (-24.8%). 13 job(s) slower, 10 faster,

• Python tests / Run tests slice 8/12: +38.0s
• Python tests / Run tests slice 5/12: +28.0s
• Python tests / Run tests slice 12/12: +15.0s
• Python tests / Run tests slice 6/12: -11.0s
• Python tests / Run tests slice 7/12: -10.0s

[spfcraze]

This was generated by AI during triage.

Summary:
The new agent/verify/runner.py uses os.killpg and signal.SIGKILL — APIs scripts/check-windows-footguns.py bans as Windows-incompatible — so the blocking Windows-footguns gate is red on this head and the verify start phase crashes on Windows.

Problems:

• The PR head (d4b4b9a) fails "Python lints / Windows footguns (blocking)"; the checker (scripts/check-windows-footguns.py:209-247) flags bare os.killpg and bare signal.SIGKILL with Windows-incompatibility messages.
• _terminate_process_group calls os.killpg(pgid, signal.SIGTERM) and os.killpg(pgid, signal.SIGKILL); _run_start_phase also uses os.getpgid(proc.pid) and Popen(start_new_session=True) — all POSIX-only.
• On Windows those calls raise at runtime: os.getpgid raises AttributeError, which the except (ProcessLookupError, PermissionError) clause does not cover, and start_new_session=True raises ValueError inside Popen — so hermes verify's start phase crashes there instead of smoke-testing the app.

Solution:
Gate the process-group teardown behind a platform check — hasattr(os, "killpg") with getattr(signal, "SIGKILL", signal.SIGTERM), the fix the footgun script suggests — or declare the start phase POSIX-only and skip it on Windows.

---

Checked against d4b4b9a — the PR head when this was written — and aacd319, main at the same moment.