Skip to content

fix(agent): gate verify-on-stop nudge off for messaging surfaces - #52412

Merged
OutThisLife merged 1 commit into
NousResearch:mainfrom
GodsBoy:fix/verify-on-stop-messaging-surface-leak
Jun 26, 2026
Merged

fix(agent): gate verify-on-stop nudge off for messaging surfaces#52412
OutThisLife merged 1 commit into
NousResearch:mainfrom
GodsBoy:fix/verify-on-stop-messaging-surface-leak

Conversation

@GodsBoy

@GodsBoy GodsBoy commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Makes the verify-on-stop guard (added in #52296 / #52297) surface-aware so it stops leaking internal verification machinery into user-facing chat replies on messaging surfaces.

On a gateway messaging session (Telegram, Discord, etc.), after an agent edits local files in a workspace with no canonical test command, the harness injects the edit -> verify nudge, which tells the model to write a /tmp/hermes-verify-*.py script, run it, and "summarize it explicitly as ad-hoc verification." The model complies, and that ad-hoc verification receipt becomes the final turn response the gateway delivers to the human user as chat noise:

Ad-hoc verification rerun for the changed files.
Temporary script: /tmp/hermes-verify-4hupyoi1.py
Result: PASS ... cleanup=removed
This is ad-hoc verification, not a canonical full suite green.

The guard is valuable in a coding CLI session; on a chat surface it is internal machinery the user should never see.

This PR resolves a surface-aware default instead of a hardcoded one. agent.verify_on_stop now defaults to the sentinel "auto", which verify_on_stop_enabled() resolves to:

  • ON for interactive coding surfaces (CLI, TUI, desktop) and programmatic callers (API server, webhooks).
  • OFF for conversational messaging surfaces (Telegram, Discord, Slack, Signal, WhatsApp, etc.).

The surface is read from HERMES_SESSION_PLATFORM (what the gateway actually binds), with HERMES_SESSION_SOURCE and HERMES_PLATFORM as fallbacks, mirroring the sibling platform resolution in agent/skill_commands.py and agent/prompt_builder.py. An explicit HERMES_VERIFY_ON_STOP env var or a boolean agent.verify_on_stop config still overrides in either direction. The passive verification ledger and the call site are untouched.

Related Issue

Fixes #52411

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • agent/verification_stop.py: verify_on_stop_enabled() resolves a surface-aware default when no explicit env/config is set; honors an explicit boolean config or recognized string tokens; new _session_is_messaging_surface() reads HERMES_SESSION_PLATFORM + HERMES_SESSION_SOURCE + HERMES_PLATFORM and classifies against _NON_MESSAGING_SESSION_SURFACES (the local/programmatic surfaces, mirroring apps/desktop/src/lib/session-source.ts LOCAL_SESSION_SOURCE_IDS).
  • hermes_cli/config.py: DEFAULT_CONFIG agent.verify_on_stop changes from True to the sentinel "auto" so the surface-aware default is actually reachable through load_config().
  • cli-config.yaml.example: documents agent.verify_on_stop and the "auto" default.
  • tests/agent/test_verification_stop.py: surface-aware coverage including messaging-platform OFF, interactive/programmatic ON (CLI, TUI, desktop, codex, api_server, webhook), env/config override both directions, the "auto" sentinel, and an end-to-end test that drives verify_on_stop_enabled() through the real load_config() to prove the sentinel reaches the surface branch.

How to Test

  1. pytest tests/agent/test_verification_stop.py -q (32 passing).
  2. Reproduction of the original bug and proof of the fix:
import os
from agent.verification_stop import verify_on_stop_enabled, _session_is_messaging_surface

os.environ["HERMES_SESSION_PLATFORM"] = "telegram"   # what the gateway binds for a Telegram turn
assert _session_is_messaging_surface() is True
assert verify_on_stop_enabled() is False             # messaging surface -> nudge OFF, no leak

os.environ["HERMES_SESSION_PLATFORM"] = ""
os.environ["HERMES_SESSION_SOURCE"] = "cli"
assert verify_on_stop_enabled() is True              # CLI coding surface -> nudge ON (unchanged)

os.environ["HERMES_SESSION_SOURCE"] = "tui"
assert verify_on_stop_enabled() is True              # TUI/desktop coding surface -> nudge ON
  1. Override still works: HERMES_VERIFY_ON_STOP=1 forces ON on a messaging surface; agent.verify_on_stop: false forces OFF on a coding surface.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(agent):)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix (no unrelated commits)
  • I've run pytest tests/ -q on the affected suites and all tests pass (verification + full config suite: 213 passing)
  • I've added tests for my changes (required for bug fixes)
  • I've tested on my platform: Ubuntu 24.04

Documentation & Housekeeping

  • I've updated relevant documentation (cli-config.yaml.example, docstrings)
  • I've updated cli-config.yaml.example since I changed a config key default
  • N/A architecture/workflow docs
  • I've considered cross-platform impact (pure Python stdlib, env/contextvar reads; no platform-specific code)
  • N/A tool descriptions/schemas

Screenshots / Logs

image

The verify-on-stop guard (PRs NousResearch#52296, NousResearch#52297) defaulted ON for every
session, so on gateway messaging surfaces (Telegram, Discord, etc.) the
model complied with the nudge by writing a hermes-verify temp script and
emitting an ad-hoc verification summary, which the gateway delivered to
the end user as chat noise.

Resolve a surface-aware default instead. The DEFAULT_CONFIG value becomes
the sentinel "auto", which verify_on_stop_enabled() resolves to ON for
interactive coding surfaces (CLI, TUI, desktop) and programmatic callers,
and OFF for conversational messaging surfaces. The surface is read from
HERMES_SESSION_PLATFORM (what the gateway actually binds), with
HERMES_SESSION_SOURCE and HERMES_PLATFORM as fallbacks, matching the
sibling resolution in skill_commands.py and prompt_builder.py. An explicit
HERMES_VERIFY_ON_STOP env var or a boolean agent.verify_on_stop config
still overrides in either direction.

The passive evidence ledger and the call site are untouched.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists labels Jun 25, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Makes the verify-on-stop guard surface-aware so it stops leaking internal verification machinery into user-facing chat replies on messaging surfaces (Telegram, Discord, WhatsApp, etc.).

Looks Good

  • Non-messaging session surfaces frozenset defines known non-messaging surfaces (cli, tui, desktop, gateway, etc.)
  • Checks both HERMES_SESSION_PLATFORM and HERMES_SESSION_SOURCE with fallback to HERMES_PLATFORM
  • Default-deny design: unrecognized identities treated as messaging (OFF) so new platforms never leak verification receipts before the set is updated
  • Config default is auto which resolves to ON for interactive surfaces, OFF for messaging
  • Precedence: explicit env var, then config value, then surface-aware default
  • 4 files, 199 additions, well-scoped

Reviewed by Hermes Agent (cron)

@OutThisLife
OutThisLife merged commit a2b49e6 into NousResearch:main Jun 26, 2026
27 checks passed
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…-messaging-surface-leak

fix(agent): gate verify-on-stop nudge off for messaging surfaces
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…-messaging-surface-leak

fix(agent): gate verify-on-stop nudge off for messaging surfaces
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…-messaging-surface-leak

fix(agent): gate verify-on-stop nudge off for messaging surfaces
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…-messaging-surface-leak

fix(agent): gate verify-on-stop nudge off for messaging surfaces
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…-messaging-surface-leak

fix(agent): gate verify-on-stop nudge off for messaging surfaces
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: verify-on-stop nudge leaks ad-hoc verification receipt into messaging (Telegram) replies

4 participants