Skip to content

feat(goals): gate continuation on fresh artifact smoke test - #66838

Closed
QuixThe2nd wants to merge 3 commits into
NousResearch:mainfrom
QuixThe2nd:feat/goal-previous-deliverable-smoke-gate
Closed

feat(goals): gate continuation on fresh artifact smoke test#66838
QuixThe2nd wants to merge 3 commits into
NousResearch:mainfrom
QuixThe2nd:feat/goal-previous-deliverable-smoke-gate

Conversation

@QuixThe2nd

Copy link
Copy Markdown

Description

Structured /goal contracts can name objective verification criteria, but the autonomous loop can still advance or finish using evidence from a previous artifact generation.

This change adds a mandatory previous-deliverable smoke gate for coding goals with structured verification evidence:

  • verification events are bound to a SHA-256 identity covering Git HEAD, tracked contents, hidden tracked modifications, untracked/ignored/explicit paths, and initialized submodule state;
  • missing or unstable Git inventory, unsafe paths, read failures, and configured path/byte/output bounds fail closed;
  • cached PASS is bound to the exact artifact, completion contract/subgoals, and the canonical verification-event payload digest;
  • the recorded verification command is replayed with shell=False, a 30-second timeout, a 1 MiB output bound, artifact-root containment, and rejection of shell/env wrappers, external executables/scripts/tests/configs, and symlink escapes;
  • timeout, output overflow, mutation, nonzero exit, invalid verifier output, and INCONCLUSIVE remain non-passing;
  • descendants are cleaned up after normal exit or timeout, including detached children on supported Linux cgroup v2 hosts;
  • a fresh isolated goal_judge reviews bounded evidence for the exact unchanged artifact;
  • only exact PASS is cached; failure or inconclusive review replaces normal continuation or done with a repair/re-verification turn;
  • enforcement lives at the central GoalManager boundary used by CLI, TUI, and gateway callers;
  • the async gateway offloads the blocking gate so it does not stall the event loop.

The existing SQLite evidence ledger migrates schema v1 to v2 in place. Bare goals and non-coding goals without applicable evidence retain their current behavior.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings in the exercised test surface
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally in the exercised goals/verification surface
  • Any dependent changes have been merged and published in downstream modules

How has this been tested?

  • focused goals, evidence, TUI, and current gateway-overlap matrix: 170 passed
  • broader goals, verification, CLI, gateway, Kanban, and TUI matrix: 578 passed
  • repository-wide ruff check .: passed
  • python scripts/check-windows-footguns.py --all: 772 files scanned, no findings
  • git diff --check: passed
  • fresh-context adversarial review of exact commit 38717ba6513820d78fb085e28984f4d6c7016f52: PASS with 581 passed in its required matrix
  • independent reviewer process/path probe:
    • normal-exit parent returned 0 and its detached child was removed;
    • shell/env wrappers and an external test path were rejected before execution
  • synthetic merge with the live upstream tip completed without conflicts after upstream changed both gateway/run.py and tui_gateway/server.py

The complete repository suite is left to CI; this PR does not claim a local full-suite pass.

Reviewer focus

Please scrutinize:

  • exact artifact and verification-event identity semantics;
  • cache invalidation across persistence/reload and goal mutation;
  • command/path containment and process cleanup;
  • post-rerun and pre-cache mutation checks;
  • central done/continuation enforcement;
  • gateway event-loop offload and TUI structured-contract persistence.

@alt-glitch alt-glitch added type/feature New feature or request comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/tui Terminal UI (ui-tui/ + tui_gateway/) P3 Low — cosmetic, nice to have needs-decision Awaiting maintainer decision before any implementation sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 18, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to merged #50501: this adds an artifact-bound re-verification gate on top of completion contracts rather than duplicating the contract feature. Maintainer direction is needed on the mandatory continuation policy.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough artifact-identity and process-cleanup work. The underlying completion gap is real on current main: hermes_cli/goals.py:1428-1481 accepts a structured-goal done verdict without fresh artifact-bound verification.

Problems

  • The new applicability path at hermes_cli/goals.py:2133 accepts _latest_goal_verification_status(self.session_id), while the new latest_verification_status() chooses the latest session-wide workspace rather than evidence associated with this goal. A later structured non-coding or unrelated goal can therefore rerun and be blocked by an earlier coding check. This does not enforce the coding-goal scope documented in website/docs/user-guide/features/goals.md:70.
  • The new non-coding test covers only not_applicable evidence; it misses a session with prior unrelated passing evidence.

Suggested changes

  • Associate evidence with the active goal/contract or goal generation before enabling this mandatory gate.
  • Add the cross-goal regression case above.

This is an automated hermes-sweeper review.

Comment thread hermes_cli/goals.py Outdated
@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 19, 2026
QuixThe2nd added 2 commits July 19, 2026 17:08
@QuixThe2nd
QuixThe2nd force-pushed the feat/goal-previous-deliverable-smoke-gate branch from 38717ba to 375d20d Compare July 19, 2026 08:05
@QuixThe2nd

Copy link
Copy Markdown
Author

Addressed in commit 375d20ddc95da358d94bba728d39cd159c6b40f4 (rebased on current origin/main): GoalState.created_at now defines the active goal generation; verification evidence and edit activity from before that boundary are rejected for the smoke gate. The fix covers both the persisted-fallback lookup (latest_verification_status takes a not_before argument scoped against verification_state.last_edit_at and verification_events.created_at) and explicit verification_state inputs (_verification_for_goal_generation). Cross-goal regression case added (prior coding evidence does not block an unrelated non-coding goal, and a current-goal edit+event still does). 254 tests across goals/verification/gateway pass; ruff/windows-footguns clean; fresh-context verifier reports PASS. Review thread resolved.

@QuixThe2nd QuixThe2nd closed this Aug 16, 2026
@QuixThe2nd
QuixThe2nd deleted the feat/goal-previous-deliverable-smoke-gate branch August 16, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery comp/tui Terminal UI (ui-tui/ + tui_gateway/) needs-decision Awaiting maintainer decision before any implementation P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants