feat(cli): add runtime smoke and context audit commands

[psionic73]

Summary

• add read-only hermes context audit diagnostics for prompt/context surface inspection
• add read-only hermes smoke diagnostics for lightweight runtime checks
• wire both commands through the CLI subcommand registry with tests

Tests

• /root/.hermes/hermes-agent/venv/bin/python -m pytest tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py -q
• Result: 37 passed, 3 warnings
• Also covered by combined targeted suite: 189 passed, 3 warnings

[psionic73]

Refreshed this PR onto current origin/main and force-pushed with lease.

Current status:

• New head: 49957b8fc670ce95bf270c863b64b40cdb44a6e1
• Base: aaa2e2cb882060b3c97d91452155c2363a5e2d30
• Branch divergence after refresh: 1 ahead / 0 behind origin/main
• GitHub reports: MERGEABLE / CLEAN
• GitHub checks after refresh: 19 success, 7 skipped, 0 pending/failing

Local verification after refresh:

• git diff --check origin/main..HEAD — OK
• python -m pytest -o 'addopts=' tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py -q → 37 passed, 3 warnings
• python -m ruff check hermes_cli/context_audit.py hermes_cli/main.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py — OK

Backup branch kept locally before the final refresh: backup/pr49757-refresh-20260623_203622.

[psionic73]

Refreshed this PR onto current origin/main and force-pushed with lease.

Current status:

• New head: 0d21aab45ebf85e5a208e86f6fe51b9d38a7aa5c
• Base: 77d2b50751f3
• Branch divergence after refresh: 1 ahead / 0 behind origin/main
• GitHub reports: MERGEABLE / CLEAN
• GitHub checks after refresh: 19 success, 7 skipped, 0 pending/failing

Local verification after refresh:

• python -m pytest -o 'addopts=' tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py -q → 37 passed, 3 warnings
• python -m ruff check hermes_cli/context_audit.py hermes_cli/main.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py — OK
• python -m hermes_cli.main context audit --cwd /tmp --platform cli --json smoke — OK (CONTEXT_AUDIT_SMOKE_OK)

Backup branch kept locally before the final refresh: backup/pr49757-repeat-20260624_202910.

[psionic73]

Refreshed this PR onto current origin/main.

• New head: cf5bbaf8c046
• Base: 2c0820c9ff55
• Mergeability: MERGEABLE / CLEAN
• GitHub checks after refresh: 23 success, 7 skipped, 1 neutral, 0 pending, 0 failing
• Local verification: PYTHONPATH="$WT" venv/bin/python -m pytest -q tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py → 37 passed; ruff touched files → passed

No failing or pending checks observed after the refresh.

[teknium1]

Thanks for the diagnostics work. The current PR needs correctness and side-effect changes before it is ready for salvage.

Problems

• hermes_cli/context_audit.py:76 reads the cap below agent, but current main resolves top-level context_file_max_chars (agent/prompt_builder.py:1201-1219; hermes_cli/config.py:1346-1352).
• context_audit.py:61-68 lists every cwd candidate, while the real builder chooses one source by priority and may load a parent .hermes.md (agent/prompt_builder.py:1850-1982), so the audit can report the wrong source/cap status.
• smoke.py:146-187 writes files and launches persistent chat sessions by default; this conflicts with its read-only framing. It also hardcodes /tmp, which is not native-Windows portable.
• The new commands are absent from _coalesce_session_name_args()'s command set (hermes_cli/main.py:11070-11135), so resume/continue preprocessing can misparse them.

Suggested changes

• Reuse the prompt-builder discovery/effective-cap contract.
• Make stateful/network smoke probes opt-in, use a portable temporary path, and add resume/continue coverage.
• Re-scope the new HERMES_SMOKE_CLI override to argparse/config rather than a non-secret HERMES_* setting.

Automated hermes-sweeper review.

[teknium1]

The live resolver reads top-level context_file_max_chars (agent/prompt_builder.py:1211-1219; hermes_cli/config.py:1346-1352), not agent.*`. This audit can report a 20K cap when a user has configured a different effective cap; reuse the prompt-builder resolver or expose a shared helper.

[teknium1]

This inventory does not match what is injected: build_context_files_prompt() uses first-match priority and searches .hermes.md/HERMES.md up to the git root (agent/prompt_builder.py:1850-1982). Derive the displayed source from that discovery path rather than listing every cwd candidate.

[teknium1]

This hardcodes a POSIX /tmp path and creates files despite the command being presented as read-only. Use a platform-neutral temporary-directory API, and make persistent artifact creation explicit.

[teknium1]

Please also add context and smoke to _coalesce_session_name_args()'s _SUBCOMMANDS set (hermes_cli/main.py:11070-11135), otherwise --continue/--resume preprocessing can consume either new command as part of an unquoted session name.

[psionic73]

Addressed the review feedback and refreshed this PR onto current origin/main.

Changes made in this refresh:

• context audit now uses the prompt builder's effective context_file_max_chars resolution instead of reading agent.* config keys.
• context audit now mirrors the real project-context discovery priority (nearest .hermes.md/HERMES.md up to git root, then AGENTS.md, then CLAUDE.md, then cursor rules) instead of listing every cwd candidate.
• smoke defaults are now side-effect-light: no artifact writes and no persistent chat sessions unless explicitly requested.
• Added --chat, --write-artifacts, --output-dir, and --cli; removed the non-secret HERMES_SMOKE_CLI env override path in favor of CLI configuration.
• Artifact default uses the platform temp directory via Python APIs when --write-artifacts is requested, not hardcoded /tmp.
• Added context and smoke to resume/continue subcommand boundary handling and covered them with tests.

Local verification after rebase:

• python -m pytest -o 'addopts=' tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py tests/hermes_cli/test_coalesce_session_args.py -q → 60 passed
• python -m ruff check hermes_cli/context_audit.py hermes_cli/main.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_subcommands_batch.py tests/hermes_cli/test_coalesce_session_args.py → passed
• git diff --check → passed for the worktree delta
• python -m hermes_cli.main context audit --cwd /tmp --platform cli --json → OK
• python -m hermes_cli.main smoke --json → OK, artifact_dir: null, 7 results
• python -m hermes_cli.main smoke --write-artifacts --json → OK, wrote summary under platform temp dir

CI note:

• The pushed head is 5cc461ce9a6398881d1aaebb653f281276faf5a3, 1 commit ahead / 0 behind origin/main.
• GitHub reports the PR as mergeable.
• Current CI has one failure in Python tests / Run tests slice 8/8: tests/agent/test_codex_responses_adapter.py::test_normalize_codex_response_salvage_is_xai_scoped.
• That file and agent/codex_runtime.py are byte-identical between this PR head and origin/main, and the same test fails locally on this refreshed branch, so this appears unrelated to the smoke/context-audit changes.

[psionic73]

Refreshed this PR onto current main after the Photon blocker was closed as already implemented on main.

Current verified state:

• head: 8a1dcec8123471fc047ca13395cc4424726b5498
• base: 8defb9fd60bebe2802eaab7c57fa2ee6a4ff6281
• ahead/behind vs current origin/main: 0 / 1
• GitHub mergeability: MERGEABLE / CLEAN
• checks: 27 success, 10 skipped, 1 neutral, 0 pending, 0 failing

Local verification before pushing:

• ruff check hermes_cli/context_audit.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py → passed
• pytest -o addopts= tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py -q → 16 passed
• git diff --check origin/main...HEAD → passed
• py_compile for the changed CLI modules → passed
• CLI parser smoke for hermes context --help, hermes context audit --help, and hermes smoke --help → passed

This should be ready for maintainer review/merge from my side.

[psionic73]

Refreshed this PR onto current origin/main and force-pushed with lease.

Current verified state:

• New head: d775564c286801a66bb56b5f4a6da2f6df562b3d
• Base: 03fa32c92dd445eb64c7f67434dd91b32c40701d
• Branch divergence after refresh: 0 behind / 1 ahead origin/main
• GitHub reports: MERGEABLE / CLEAN
• GitHub checks after refresh: 33 success, 11 skipped, 1 neutral, 0 pending, 0 failing

Local verification:

• PYTHONPATH="$WT" /root/.hermes/hermes-agent/.venv/bin/python -m pytest -q tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py -o 'addopts=' → 16 passed
• /root/.hermes/hermes-agent/.venv/bin/ruff check hermes_cli/context_audit.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py → passed

Backup branch: backup/pr49757-refresh-20260810_111946.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

feat(cli): add runtime smoke and context audit commands

1. Process-wide cwd mutation in a "read-only, offline" diagnostic. measure_context_budget calls os.chdir(root) (restored in finally) while running compute_prompt_breakdown and the prompt builder. os.chdir is a global side effect — if the audit is ever invoked from a server/gateway context, or runs concurrently with other threads that depend on cwd (e.g. a background review fork), it can race. Prefer passing an explicit cwd into the breakdown computation if the API supports it, or at least document the side effect in the docstring.
2. Coupling to private prompt-builder internals. _context_file_limit, _find_project_context_files, and drain_truncation_warnings reach into _get_context_file_max_chars, _find_hermes_md, etc. A rename in the prompt builder silently changes the audit's behavior (fallback to the 20k default / different discovery order) without any test failing. A stable public accessor or a contract test pinning the coupling would make the audit robust.
3. Exact-match assertion against an LLM reply. smoke_profile (opt-in via --chat) asserts " ".join(out.split()) == expected where expected is a sentinel the model must reproduce verbatim. Models routinely add punctuation, preamble, or trailing whitespace; a single extra token fails the check. Consider substring/prefix matching (or a machine-parseable sentinel on its own line) and document the strictness in --help.
4. Default profile list assumes host profiles exist. run_smoke defaults to ("default", "cheap", "lab"); on a host without cheap/lab, --chat reports FAIL for profiles that were never created. Consider skipping missing profiles with a note rather than failing the run.

[psionic73]

Follow-up after the automated review and a fresh rebase onto current origin/main.

Changes pushed in 4a2232bc1efc7260d462c3c3b48ee3cfa0eba12e:

• Documented the one remaining cwd side effect in measure_context_budget and surfaced it in the human-readable context audit notes.
• Kept project-context discovery aligned with prompt-builder first-match-wins behavior and documented that coupling in the output notes.
• Relaxed smoke_profile from exact full-stdout equality to requiring the sentinel token in the output, so harmless model preamble/formatting no longer false-fails.
• Skips missing named profiles during opt-in --chat smokes instead of failing hosts that do not have cheap/lab configured.
• Added regression coverage for the sentinel-in-output and missing-profile cases.

Local verification on the pushed head:

• pytest -q tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py -o 'addopts=' → 18 passed
• ruff check hermes_cli/context_audit.py hermes_cli/smoke.py hermes_cli/subcommands/context.py hermes_cli/subcommands/smoke.py tests/hermes_cli/test_context_audit.py tests/hermes_cli/test_smoke.py tests/hermes_cli/test_coalesce_session_args.py tests/hermes_cli/test_subcommands_batch.py → passed
• pytest -q tests/gateway/test_goal_max_turns_config.py::test_gateway_goal_uses_goals_max_turns_from_full_config tests/gateway/test_session_api.py::test_session_chat_stream_treats_pre_existing_poisoned_row_as_no_model -o 'addopts=' on this branch → 2 passed

Current GitHub CI is still blocked by gateway tests outside this PR's diff:

• tests/gateway/test_goal_max_turns_config.py::test_gateway_goal_uses_goals_max_turns_from_full_config failed in CI, but passed locally on this branch.
• tests/gateway/test_session_api.py::test_session_chat_stream_treats_pre_existing_poisoned_row_as_no_model failed in CI; I also reproduced that same test failure on clean current origin/main, so it is not introduced by this PR.

I am not mixing a gateway/session-api fix into this CLI diagnostics PR unless maintainers want that scope broadened.