fix(oneshot): shut down memory provider to prevent SIGABRT on exit (Honcho crash)

[koshaji]

Problem

Running hermes -z (oneshot mode) with a memory provider that spawns daemon worker threads — e.g. Honcho — crashes on exit with SIGABRT (exit 134) and no Python traceback:

$ hermes -z "hi"   # with memory.provider: honcho
Hi Hani! ...
Fatal Python error: Aborted   # (via PYTHONFAULTHANDLER=1)

Built-in memory (provider: '') is unaffected.

Root cause

The interactive CLI registers an atexit handler (_run_cleanup in cli.py) that calls agent.shutdown_memory_provider(), but hermes_cli/oneshot.py does not — _run_agent() builds its AIAgent directly and sys.exit(run_oneshot(...)) without registering that cleanup.

So when the provider has daemon threads still blocked in C-level I/O at interpreter exit (Honcho's httpx socket recv), CPython forcibly kills them during Py_FinalizeEx via PyThread_exit_thread → __pthread_unwind → abort(). Confirmed by gdb: the aborting stack is Py_FinalizeEx → PyGC_Collect → … → take_gil → PyThread_exit_thread → __pthread_unwind → abort(), with worker threads blocked in __GI___poll → sock_recv.

This is the well-known CPython daemon-thread-at-exit issue (gh-97940 / bpo-20526).

Fix

1. hermes_cli/oneshot.py (general fix): register an atexit hook in _run_agent() that calls agent.shutdown_memory_provider() before the interpreter finalizes — mirroring the interactive CLI. This protects any memory provider that uses background threads (Honcho today; potentially mem0/retaindb/etc.).

2. plugins/memory/honcho (defensive hardening, so shutdown() actually unblocks the workers):

   • Join all daemon threads (init / context-prefetch / sync) in HonchoMemoryProvider.shutdown().
   • Close the Honcho SDK's underlying httpx.Client (manager._honcho._http) — this interrupts any worker still blocked in sock_recv (httpx raises a connection-closed error the worker absorbs), so the subsequent join(timeout=...) calls actually complete instead of racing a 30s read poll.
   • Track fire-and-forget honcho-context-prefetch threads in HonchoSessionManager so they can be joined in shutdown().

Verification

• hermes -z with Honcho enabled: exit 0 (was 134), reproduced across 3 runs with different prompts.
• hermes -z with built-in memory: exit 0 (no regression).
• memory.provider left at '' (built-in) in the repro so the gateway stays safe; users can set it to honcho after this fix.

Notes

The Honcho server itself was healthy throughout (it built a correct user model via its LLM); the crash was entirely on the Hermes side at session teardown. The oneshot-vs-interactive cleanup asymmetry is the real bug.

[Copilot]

Pull request overview

Fixes a oneshot-mode shutdown gap where external memory providers (notably Honcho) could leave daemon threads blocked in C-level I/O during interpreter teardown, leading to SIGABRT on exit. The PR adds oneshot cleanup and strengthens Honcho provider shutdown/thread tracking to enable a clean exit.

Changes:

• Register an atexit cleanup in hermes_cli/oneshot.py to shut down the agent’s memory provider on process exit.
• Track Honcho “fire-and-forget” context-prefetch threads in HonchoSessionManager and join them during shutdown.
• Harden Honcho provider shutdown to join worker threads and close the underlying Honcho SDK HTTP client to unblock in-flight reads.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
plugins/memory/honcho/session.py	Adds tracking/joining of context-prefetch threads during shutdown.
plugins/memory/honcho/__init__.py	Expands Honcho provider shutdown logic to join worker threads and close the Honcho HTTP client.
hermes_cli/oneshot.py	Adds an atexit hook in oneshot mode to shut down the memory provider before interpreter finalization.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[koshaji]

Update: strengthened the oneshot fix to also cover the multi-provider case.

The graceful shutdown alone resolves Honcho-only, but I found a residual SIGABRT when Honcho + a cloud-browser provider (Browser Use) are both active — Honcho's daemon worker threads (honcho-prefetch, honcho-context-prefetch, honcho-async-writer) remain blocked in native recv at exit despite shutdown() closing the httpx client + joining (the close doesn't reliably interrupt them when another async provider is active).

hermes_cli/oneshot.py now also does, after best-effort shutdown_memory_provider():

for _stream in (sys.stdout, sys.stderr): _stream.flush()
os._exit(0)

i.e. once the oneshot has emitted its output and attempted graceful teardown, it exits directly to bypass Py_FinalizeEx entirely — which is the step that aborts on any surviving daemon thread, regardless of provider. The OS reclaims the threads. This is oneshot-only (hermes -z); the interactive CLI / gateway are long-running and unaffected.

Verified: hermes -z with Honcho + Browser Use + a browser task now exits 0 (was 134); built-in memory unaffected.

[koshaji]

Pushed a follow-up commit addressing all four review comments:

• oneshot transcript ([Bug]: Gateway restart drops session memory — shutdown_memory_provider receives empty messages #15165 parity): the atexit hook now forwards agent._session_messages to shutdown_memory_provider() (with the same isinstance(..., list) guard as _run_cleanup), so providers' on_session_end() see the real conversation. Extracted the registration into a module-level _register_oneshot_memory_cleanup(agent) helper.
• shutdown() dropping live threads: retain threads still alive after the timed join instead of clear()-ing unconditionally, so the post-httpx-close retry can re-join previously-blocked workers.
• unbounded prefetch list: prune finished threads before appending in prefetch_context().
• test coverage: added tests/cli/test_oneshot_memory_cleanup.py (hook registration, transcript forwarding, no-arg fallback, exit-on-raise).

Ready for another look.

[koshaji]

Superseded by #61875 — cleaner approach using atexit hook without the _os._exit(0) bypass. Closing this in favor of the rebased version.