Skip to content

fix(acp): reset is_running when prompt() raises before the executor guard - #71028

Open
israellot wants to merge 1 commit into
NousResearch:mainfrom
israellot:fix/acp-prompt-brick
Open

fix(acp): reset is_running when prompt() raises before the executor guard#71028
israellot wants to merge 1 commit into
NousResearch:mainfrom
israellot:fix/acp-prompt-brick

Conversation

@israellot

Copy link
Copy Markdown
Contributor

What & why

prompt() in acp_adapter/server.py sets state.is_running = True under the runtime lock, but its exception protection only begins at the executor call much later. Everything between the flip and the executor try — callback-factory construction (make_tool_progress_cb / make_message_cb / …), the edit-approval requester wiring, agent callback assignment — is an unprotected raise window.

If anything in that setup region raises, the exception escapes with is_running still True and the session is permanently bricked: every subsequent user message hits the busy guard and is answered with Queued for the next turn. (N queued), and no turn ever drains the queue. The only recovery is restarting the editor / agent process.

Fix at the root cause: open one outer try: immediately after the runtime-lock block that sets is_running=True, enclosing the setup region, the executor call, and the existing post-turn tail. It closes with except BaseException: that resets is_running / current_prompt_text under the runtime lock and re-raises. The existing inner executor except (which also resets and shapes the error response) is kept unchanged — the outer reset is idempotent with it. The busy-branch early returns happen before is_running is set and stay outside the guard.

The diff is mostly re-indentation of the guarded block; git diff -w shows the real change is ~18 added lines.

How to test

bash scripts/run_tests.sh tests/acp/test_server.py -q
bash scripts/run_tests.sh tests/acp/ -q

New test: TestPrompt::test_prompt_setup_exception_still_releases_session — patches acp_adapter.server.make_message_cb (a setup-region callback factory) to raise, calls prompt(), and asserts:

  1. the exception propagates (behavior unchanged for the caller),
  2. state.is_running is False afterwards and current_prompt_text is cleared,
  3. a follow-up prompt() on the same session runs (run_conversation mocked) instead of queueing into a dead session, and queued_prompts stays empty.

Before the fix, step 2 fails: AssertionError: setup exception left is_running=True — session bricked.

Verification on this branch:

  • bash scripts/run_tests.sh tests/acp/test_server.py -q → 87 passed, 0 failed
  • bash scripts/run_tests.sh tests/acp/ -q → 311 passed, 0 failed (13 files)
  • scripts/check-windows-footguns.py acp_adapter/server.py tests/acp/test_server.py → no footguns

Reproduction sketch

Any transient failure in the setup region reproduces it, e.g. a raise inside a callback factory or the requester wiring:

# after session/new, make any setup-region call raise once:
with patch("acp_adapter.server.make_message_cb", side_effect=RuntimeError("boom")):
    await agent.prompt(prompt=[TextContentBlock(type="text", text="hi")], session_id=sid)
# session is now bricked without this fix:
await agent.prompt(...)  # -> "Queued for the next turn. (1 queued)" forever

Platforms tested

Linux fully verified; Windows/macOS not manually tested (change is pure Python control flow, no platform-specific I/O; footguns check clean).

Related — independent

Prior art: this fix has been running in a public fork — YallaPlay/hermes-agent@6942586

@alt-glitch alt-glitch added type/bug Something isn't working comp/acp Agent Communication Protocol adapter P2 Medium — degraded but workaround exists labels Jul 24, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks — the reported failure window is present on current main: prompt() sets state.is_running = True at acp_adapter/server.py:1652, calls callback factories at acp_adapter/server.py:1686-1696, and does not enter its existing executor guard until acp_adapter/server.py:1843. A setup exception can therefore bypass the resets at acp_adapter/server.py:1857-1859; later input follows the busy queue path at acp_adapter/server.py:1627-1668.

Problems

  • GitHub reports this branch as conflicting. Since its base (7cd48733) current main added the session-cwd binding at acp_adapter/server.py:1764-1773; resolving the broad reindent mechanically risks losing that newer behavior.

Suggested changes

  • Salvage the outer reset guard onto current prompt() while preserving the cwd binding and existing executor-specific error response.
  • Move the regression test into the current TestPrompt section at tests/acp/test_server.py:404, because the historical test context was pruned on main.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
@israellot
israellot force-pushed the fix/acp-prompt-brick branch from 551e898 to 92f53b6 Compare July 30, 2026 11:07
@israellot

Copy link
Copy Markdown
Contributor Author

Both points addressed in 92f53b665 (branch rebuilt directly on current main at 14db1a99e, so the conflict is gone — GitHub now reports mergeable: MERGEABLE).

1. Conflicting branch / risk of losing the newer session-cwd binding. Rather than resolving the old reindent mechanically, I reset the branch to current main and re-applied only the guard. The cwd binding is preserved verbatim — set_session_vars(session_key=session_id, cwd=state.cwd) and its explanatory comment are still there (now at acp_adapter/server.py:1770-1784, just one indent level deeper inside the guard). The executor-specific except that shapes the error response is likewise untouched; the new outer except BaseException only resets is_running / current_prompt_text under state.runtime_lock and re-raises, so it is idempotent with the inner resets. The busy-queue early returns still sit before the flip and stay outside the guard.

git diff -w upstream/main..HEADacp_adapter/server.py | 18 ++++++++++++++++++ — the whole functional change is the 11-line comment + try: at acp_adapter/server.py:1670-1680 and the 7-line except BaseException: at acp_adapter/server.py:1998-2004; everything else in the raw diff is re-indentation of the guarded block.

2. Test moved into the current TestPrompt section. test_prompt_setup_exception_still_releases_session now lives at tests/acp/test_server.py:413, immediately after test_prompt_returns_refusal_for_unknown_session in the surviving TestPrompt class at tests/acp/test_server.py:404. It uses only fixtures that exist on current main (agent, MagicMock(spec=acp.Client)), patches acp_adapter.server.make_message_cb to raise inside the setup region, and asserts (a) the exception still propagates, (b) is_running is False and current_prompt_text is cleared, (c) a follow-up prompt() actually runs instead of queueing, with queued_prompts == [].

Verification on 92f53b665:

  • bash scripts/run_tests.sh tests/acp/test_server.py -q → 30 tests passed, 0 failed
  • bash scripts/run_tests.sh tests/acp/ -q → 14 files, 126 tests passed, 0 failed
  • python3 scripts/check-windows-footguns.py acp_adapter/server.py tests/acp/test_server.py → no footguns
  • Red-before-green confirmed on current main: with the server.py change stashed and only the test applied, pytest tests/acp/test_server.py -k setup_exception fails with AssertionError: setup exception left is_running=True — session bricked.

@israellot

Copy link
Copy Markdown
Contributor Author

CI failure on this PR is unrelated to the change — flagging it rather than working around it, since the flake is in main and worth a proper fix by whoever owns #74487.

The failure

Python tests / Run tests slice 8/8 — one test:

tests/hermes_cli/test_update_eol_churn.py::test_churn_across_more_files_than_fit_in_one_argv
>       assert len(_dirty(repo)) == len(files)
E       AssertionError: assert 77 == 1200
tests/hermes_cli/test_update_eol_churn.py:188: AssertionError

This PR touches only acp_adapter/server.py and tests/acp/test_server.py (git diff --name-only upstream/main..HEAD) — nothing under hermes_cli/, and nothing that test imports. The file arrived on main ~an hour before this run in #74487 (5e807390f, 2026-07-30T04:53Z); its first green run on main at 14db1a99e (job 90852740902) shows tests/hermes_cli/test_update_eol_churn.py (9✓, 2.4s), and the same file in the same slice at the same commit failed on this PR's run (job 90853900700). Same code, same slice, different outcome — it's flaky, not caused by this branch.

Why it flakes (mechanism, not a guess)

The failing line is the fixture precondition, before _normalize_managed_eol is even called. _managed_repo builds the broken state, then asserts all 1200 files read dirty under -c core.autocrlf=false. Whether they do depends on git's racy-index heuristic:

  • _managed_repo commits LF content, sets autocrlf=true, deletes the files and runs git checkout -- ., so the worktree gets CRLF and the index records the CRLF stat (git ls-files --debug shows size: 11 for an on-disk b'VALUE = 0\r\n', also 11 bytes — the stat cache says "clean").
  • The only reason the CRLF churn is visible to the probe at all is that the checkout wrote those files in the same mtime second as the index write, so git considers each entry racy and re-reads content instead of trusting the cache.
  • The moment the index write lands in a later second than some of the files, git trusts the stat cache for those and reports them clean. Locally I can flip the assertion between 1200 and 0 purely by advancing the index mtime — no content change:
dirty right after fixture:                          1200
dirty with index mtime +5s (no racy re-check):         0
dirty after `autocrlf=true git update-index --refresh`: 0

A 1200-file checkout on a loaded 8-worker runner takes long enough to straddle a second boundary, so a subset of files falls outside the racy window — hence 77, and hence the 1200-file test being the one that fails first. The suite's own repro line is order-independent (python -m pytest tests/hermes_cli/test_update_eol_churn.py), so nothing else in the slice is involved.

Suggested fix for the fixture: make the precondition independent of mtime granularity — either drop the == len(files) count assertion (it isn't what the test is about; the argv-length behaviour is), or force the racy state explicitly before asserting, e.g. backdate .git/index (os.utime) or git update-index --really-refresh under autocrlf=true so the cache state is deterministic rather than clock-dependent.

One thing worth a second look beyond the test

The same stat-cache behaviour reaches the product code, so this may be more than a flaky assertion. _normalize_managed_eol computes eol_only = _dirty() - _dirty("--ignore-cr-at-eol"), and on a settled checkout — one where any later git command has refreshed the index, which is the normal state of a real long-lived Windows install — the probe reports the CRLF tree as clean, eol_only is empty, the restore is skipped, and core.autocrlf false is written anyway:

A) FRESH checkout (racy window open)     before: crlf=3/3 autocrlf=true  probe_dirty=3
                                          after: crlf=3/3 autocrlf=false probe_dirty=3   <- repaired
B) SETTLED checkout (index refreshed)    before: crlf=3/3 autocrlf=true  probe_dirty=0
                                          after: crlf=3/3 autocrlf=false probe_dirty=0   <- restore skipped, pin written

Case B is exactly the outcome the docstring says the coupling exists to prevent ("pinning alone would expose every text file as modified and hand the update an autostash of the whole tree"). If that's right, the repair wants a forced refresh (git update-index --really-refresh, or comparing against git diff-index output that ignores the stat cache) before deciding eol_only is empty. Flagging for #74487's author rather than changing it here — out of scope for this PR, and I can't reproduce the Windows side.

(Environment note: the CI runner is git 2.54.0 on runner image 20260720.247.2; the numbers above are from git 2.43.0 locally, where 5 of the 9 tests in this file fail for the same stat-cache reason. The mechanism is version-independent; how often it bites is not.)

Happy to open a separate issue or PR for the fixture fix if that's useful — just say which you'd prefer. Nothing on this branch needs to change for it, so this PR is ready as-is apart from the red check.

pierrenode added a commit to pierrenode/hermes-agent that referenced this pull request Aug 11, 2026
…d don't resurrect stale pre-reset history

fc05247 (NousResearch#80770) fixed tui_gateway/server.py replaying a turn's
pre-turn history snapshot on the next prompt after a mid-turn crash:
AIAgent persists its working transcript into agent._session_messages
independently of the gateway's own in-memory history, via
agent._persist_session() -- called per tool round/API call throughout
a turn (agent/conversation_loop.py), not just at clean completion. If
a turn raises, that working copy is ahead of whatever snapshot the
caller was holding before the turn started.

acp_adapter/server.py::_run_agent() has the identical shape: on the
exception path, prefer agent._session_messages (the crash-time working
copy) over state.history when it's a list; fall back to state.history
unchanged if the crash happened before any persist ran.

Review found the isinstance(agent_messages, list) guard never actually
discriminates: agent/agent_init.py initializes _session_messages to []
(never None), so it's ALWAYS a list, on a real agent as much as
mid-turn. Two consequences:

- The intended fallback path only ever ran in the original test's mock
  agent, whose unconfigured _session_messages auto-attribute (a
  MagicMock, not a list) is what made isinstance() false there -- not
  anything a real agent produces.
- More seriously: /reset (_cmd_reset) clears state.history but never
  touches agent._session_messages -- reset_session_state() (run_agent.py)
  only resets session-scoped token counters and the context-compressor
  engine, not this cache. A crash before the FIRST persist of a NEW
  turn right after /reset returned the _session_messages leftover from
  the PRE-reset conversation (a non-empty list, passes isinstance()),
  resurrecting the "cleared" conversation on the next prompt.

Fix: snapshot _session_messages' content before the turn starts, and
on the exception path only adopt it when it actually changed during
this turn (persist ran) -- otherwise it's stale, and state.history (the
correct pre-turn/post-reset value) is what belongs on the next prompt.

Testing:
- Updated the existing "no working history" test to set
  agent._session_messages = [] explicitly (what a real agent's
  pre-persist state actually is) instead of relying on the mock's
  unconfigured auto-attribute -- exercising the real invariant instead
  of an artifact of the mock.
- Added a new regression reproducing the reported /reset bug directly:
  pre-reset _session_messages present, state.history cleared, turn
  crashes before any persist -- asserts state.history stays empty
  instead of adopting the stale pre-reset transcript.
- Mutation-verified: reverting just the production fix reproduces the
  reported bug in the new /reset test (state.history resurrects the
  pre-reset conversation); the other two tests are unaffected either
  way, confirming they don't (by themselves) catch this specific bug.
- Full tests/acp/ (129) + tests/acp_adapter/ (15) pass.
- ruff check clean.

Note: NousResearch#71028 (open, unrelated fix for a session-bricking bug in a
different raise window earlier in prompt()) re-indents this same
except-block as part of wrapping the whole method in an outer try --
its diff shows the block's content is otherwise unchanged. Textual
(re-indentation) merge-conflict risk only, not a semantic one.
…uard

prompt() flips state.is_running=True under the runtime lock, but its
protective try blocks only cover the executor call and nothing before
it. The SETUP region in between — callback-factory construction
(make_tool_progress_cb / make_message_cb / ...), session-context and
edit-approval requester wiring, agent callback assignment — is an
unprotected raise window: any exception there escapes with is_running
still True, permanently bricking the session. Every subsequent user
message then hits the busy guard and gets "Queued for the next turn.
(N queued)" with no turn ever draining the queue; the only recovery is
restarting the editor/agent process.

Fix at the root cause: wrap everything from immediately after the
is_running=True flip through the end of the post-turn tail in one outer
try whose except BaseException resets is_running/current_prompt_text
under the runtime lock and re-raises. The existing inner executor
except (which also resets and shapes the error response) is kept
unchanged — the outer reset is idempotent with it. The busy-branch
early returns happen BEFORE is_running is set and stay outside the
guard.

Re-applied on current main rather than merge-resolving the previous
branch: its raw diff was ~570 lines of re-indentation over a stale
base, and mechanically resolving that risked dropping the session
binding main added since. The re-indent was done programmatically on
asserted block boundaries; git diff -w shows +18/-0, and the newer
set_session_vars() call is preserved inside the guard.

Test drives the real prompt() path with make_message_cb (a setup-region
callback factory) raising, asserts is_running is reset, then proves a
follow-up prompt on the same session runs instead of queueing into a
dead session.
@israellot
israellot force-pushed the fix/acp-prompt-brick branch from 92f53b6 to 2c36f0f Compare August 19, 2026 12:41
@israellot

Copy link
Copy Markdown
Contributor Author

Rebuilt on current main at 2c36f0fa476f4ef46698e54b413bda207c3cf296. Both review findings and the red CI slice are addressed.

1. "GitHub reports this branch as conflicting; resolving the reindent mechanically risks losing newer main behavior"

I did not resolve the conflict. I reset the branch to current main and re-applied the semantic change, because the previous diff was ~570 lines of re-indentation over a stale base, and that is exactly where a newer upstream line gets dropped silently.

The re-indent was applied programmatically on asserted block boundaries (first line of the guarded region and the final return PromptResponse(...)), not by hand. The receipt:

$ git diff -w --stat upstream/main..HEAD -- acp_adapter/server.py
 acp_adapter/server.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

18 lines added, 0 removed, with whitespace ignored. That is the whole functional change: the outer try at acp_adapter/server.py:1923 and the except BaseException: reset at acp_adapter/server.py:2231.

The specific newer behavior you flagged survives, one indent level deeper inside the guard:

$ grep -n 'set_session_vars(' acp_adapter/server.py
2040:                    session_tokens = set_session_vars(

mergeable is now MERGEABLE per gh pr view 71028 --json mergeable.

2. "Move the regression test into the current TestPrompt section; the historical test context was pruned on main"

Done. The test now lives at tests/acp/test_server.py:450, inside class TestPrompt (tests/acp/test_server.py:404), immediately after test_prompt_binds_session_id_into_subprocess_env. It uses only the agent and mock_manager fixtures that exist on current main; I re-added no pruned fixtures.

3. The red Python tests / Run tests slice 8/8

Not ours, and already fixed on main. The failing test was tests/hermes_cli/test_update_eol_churn.py::test_churn_across_more_files_than_fit_in_one_argv (1 failed, 8 passed) — a file this PR does not touch and does not import:

$ git diff --name-only upstream/main..HEAD
acp_adapter/server.py
tests/acp/test_server.py

It was a git stat-cache flake: the fixture's dirtiness precondition depended on worktree-vs-index mtime. main fixed it in fe497d8722 ("test(update): make EOL-churn dirtiness deterministic under racy-git stat caching", 2026-08-02) by bumping every worktree mtime past the index write — dated after the run that failed here. Rebuilding on current main picks that up, and the file passes on this branch:

$ bash scripts/run_tests.sh tests/hermes_cli/test_update_eol_churn.py -q
=== Summary: 1 files, 9 tests passed, 0 failed (100% complete) in 1.4s

Red-before-green

With only the source file stashed and the new test left in place, the test fails on the exact assertion it is meant to pin:

$ git stash push -- acp_adapter/server.py
$ bash scripts/run_tests.sh tests/acp/test_server.py -q
FAILED tests/acp/test_server.py::TestPrompt::test_prompt_setup_exception_still_releases_session
1 failed, 30 passed in 5.67s

E       AssertionError: setup exception left is_running=True — session bricked
E       assert True is False
tests/acp/test_server.py:480: AssertionError

The test drives the real prompt() path with make_message_cb (a setup-region callback factory, called after the is_running = True flip at acp_adapter/server.py:1908) raising, then asserts a follow-up prompt on the same session actually runs instead of queueing into a session no turn will drain.

Verification

$ bash scripts/run_tests.sh tests/acp/test_server.py -q
=== Summary: 1 files, 31 tests passed, 0 failed in 6.2s

$ bash scripts/run_tests.sh tests/acp/ -q
=== Summary: 14 files, 136 tests passed, 0 failed in 9.5s

$ python -m ruff check acp_adapter/server.py tests/acp/test_server.py
All checks passed!

$ python scripts/check-windows-footguns.py acp_adapter/server.py tests/acp/test_server.py
✓ No Windows footguns found (2 file(s) scanned).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/acp Agent Communication Protocol adapter P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants