Skip to content

[codex] Add progress outcome canary - #40

Merged
OmarB97 merged 1 commit into
mainfrom
fix/progress-outcome-canary
May 31, 2026
Merged

[codex] Add progress outcome canary#40
OmarB97 merged 1 commit into
mainfrom
fix/progress-outcome-canary

Conversation

@OmarB97

@OmarB97 OmarB97 commented May 31, 2026

Copy link
Copy Markdown
Owner

Summary

  • add a turn-level progress outcome canary that watches tool rounds, not just individual tool families
  • inject one synthetic progress nudge after repeated tool rounds with no observable work outcome
  • record progress canary metadata in the run result and logs, while staying quiet after landed file mutations or state-changing commands

Root Cause

The search/read/terminal guardrails catch known local loop shapes, but an agent can still burn several tool rounds doing plausible-looking diagnostics without producing a concrete outcome. There was no higher-level assertion that a turn had made an edit, changed durable state, opened/updated workflow state, or deliberately changed strategy.

Validation

  • ./scripts/run_tests.sh tests/run_agent/test_tool_call_guardrail_runtime.py tests/agent/test_tool_guardrails.py tests/tools/test_read_loop_detection.py
  • python -m ruff check agent/progress_outcome.py agent/agent_init.py agent/conversation_loop.py run_agent.py tests/run_agent/test_tool_call_guardrail_runtime.py
  • git -c core.fsmonitor=false diff --check

Live Canary

From live Hermes at c49733f7a, the terminal usage canary blocked the repeated meshctl task ... guess and executed only the bad variants plus the help probe before this PR was started.

@OmarB97
OmarB97 marked this pull request as ready for review May 31, 2026 05:09
@OmarB97
OmarB97 merged commit 443e8b0 into main May 31, 2026
20 of 22 checks passed
@github-actions

Copy link
Copy Markdown

🔎 Lint report: fix/progress-outcome-canary vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9560 on HEAD, 9558 on base (🆕 +2)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 4964 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant