Skip to content

fix(kanban): require tool evidence before worker completion - #32940

Open
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/32746-kanban-tool-evidence
Open

fix(kanban): require tool evidence before worker completion#32940
LeonSGP43 wants to merge 1 commit into
NousResearch:mainfrom
LeonSGP43:fix/32746-kanban-tool-evidence

Conversation

@LeonSGP43

@LeonSGP43 LeonSGP43 commented May 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • auto-block dispatcher-scoped workers that call kanban_complete without any prior non-kanban_* tool calls in their recorded session
  • emit a first-class protocol_violation event for that path so operators can distinguish fabricated completion from an ordinary block
  • cover the new gate with targeted kanban tool tests and keep existing worker session stamping expectations explicit

Closes #32746

Local proof

  • uv run --frozen --extra dev pytest tests/tools/test_kanban_tools.py -q
  • uv run --frozen --extra dev ruff check hermes_cli/kanban_db.py tools/kanban_tools.py tests/tools/test_kanban_tools.py
  • git diff --check

Notes

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins labels May 27, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the fabricated-completion path. The underlying gap remains on current main: tools/kanban_tools.py:626-632 completes a dispatcher worker without a non-kanban tool-evidence check.

Problems

  • The new gate reads SessionDB.get_messages() in tools/kanban_tools.py:511-513. Those rows are durable history only (hermes_state.py:4016-4026); the agent flushes its in-memory messages through _persist_session at run_agent.py:1678-1695, whose current call sites are after the CLI run returns. Therefore a valid live sequence such as read_file followed by kanban_complete is not yet visible to this query and would be auto-blocked.
  • The new tests seed SessionDB before directly calling _handle_complete, so they do not cover that live ordering.

Suggested changes

  • Source evidence from active-run tool activity, or explicitly persist it before evaluating the completion gate.
  • Add an end-to-end same-run non-kanban-tool → completion test, alongside the no-tool-call rejection case.

Automated hermes-sweeper review.

Comment thread tools/kanban_tools.py
try:
kb, conn = _connect(board=board)
try:
if os.environ.get("HERMES_KANBAN_TASK") == tid:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This gate reads only persisted session rows, but the current agent flushes its in-memory tool-call messages via _persist_session after the CLI run returns (run_agent.py:1678-1695; call sites are in cli.py). A normal worker's read_file followed by kanban_complete in the same run therefore has no durable evidence yet and will be auto-blocked. Please source active-run activity or persist it before evaluating this gate.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
@benperry6

Copy link
Copy Markdown
Contributor

I prepared a tested follow-up directly on this PR's author branch rather than opening a competing upstream PR:

It changes Kanban completion proof from tool-call intent to successful material execution evidence, with a metadata-only current-run receipt in task_events, native Hermes/Codex/MCP coverage, bounded cross-process visibility wait, and fail-closed protocol_violation handling.

Verification: 19 targeted Kanban, 24 Codex projector, and 58 Codex app-server session tests passed; Ruff, py_compile, and git diff --check passed; independent review PASS.

The separate product decision about a reasoning-only/no-tool completion exception remains intentionally with maintainers.

@benperry6

Copy link
Copy Markdown
Contributor

Current-main refresh, kept separate from every other Kanban fix.

The bug still reproduced on that base: a worker could complete its card after emitting a tool-call intention without any successful material result. The refreshed patch records metadata-only evidence only after successful real Hermes/Codex tool completion, binds it to the current run, and leaves the card in flight on a protocol violation.

Verification on the exact commit: RED reproduction observed; 13 focused tests passed; 123 tests across the touched files passed; the complete Kanban matrix passed with 509 passed, 0 failed, 1 Windows-only skip. Ruff, compileall, and git diff --check passed.

This supersedes my older author-branch follow-up e19ff6c499f0b9832f05af127254c28cc333cf8d. I did not open a competing upstream PR; the commit is available for the existing PR author or maintainers to cherry-pick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kanban dispatcher accepts kanban_complete with zero tool calls (no tool-evidence gate)

4 participants