fix(openviking): join runtime-autostart thread on shutdown (SIGABRT-at-exit) - #49832
fix(openviking): join runtime-autostart thread on shutdown (SIGABRT-at-exit)#49832koshaji wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Fixes a daemon-thread-at-interpreter-exit crash class in the OpenViking memory provider by ensuring the runtime-autostart waiter thread can be stopped promptly and is joined during shutdown.
Changes:
- Add a
should_stopcallback to_wait_for_openviking_health()and pass the provider shutdown flag from the runtime-autostart waiter. - Update
OpenVikingMemoryProvider.shutdown()to also join the runtime-autostart thread. - Add tests covering the new early-exit behavior and verifying
shutdown()waits for the runtime-start thread to complete.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
plugins/memory/openviking/__init__.py |
Adds shutdown-aware health waiting and joins the autostart waiter thread during provider teardown. |
tests/plugins/memory/test_openviking_shutdown.py |
New tests to prevent regressions around shutdown and daemon waiter behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if not _wait_for_openviking_health( | ||
| endpoint, | ||
| timeout_seconds=_LOCAL_OPENVIKING_AUTOSTART_TIMEOUT, | ||
| should_stop=lambda: self._shutting_down, | ||
| ): |
|
Related: #49498 (Honcho memory-provider SIGABRT-at-exit fix). Same daemon-thread-at-exit failure class, different provider: this joins OpenViking's tracked |
…t-exit) `OpenVikingMemoryProvider.shutdown()` joins in-flight writers, deferred-commit threads, and prefetch threads, but not `_runtime_start_thread` — the tracked `daemon=True` waiter that runs `_finish_runtime_openviking_start`, which blocks on network health probes (`_wait_for_openviking_health` polling + a `_VikingClient.health()` request). If the local OpenViking runtime is slow or unreachable, that waiter can still be blocked in network I/O at interpreter exit. CPython then forcibly kills it during `Py_FinalizeEx` (`PyThread_exit_thread` -> `__pthread_unwind` -> `abort()`), producing SIGABRT (exit 134) with no traceback — the same daemon- thread-at-exit failure class fixed for the Honcho provider. Fix: - `shutdown()` now joins `_runtime_start_thread` (timeout-bounded) alongside the other tracked threads. - `_wait_for_openviking_health()` gains a `should_stop` callback; the waiter passes `lambda: self._shutting_down` so the poll loop bails out promptly once `shutdown()` flips the flag, instead of lingering up to the 60s autostart timeout and timing out the join (which would leave the thread alive). - Add tests/plugins/memory/test_openviking_shutdown.py covering the short-circuit and the shutdown-joins-runtime-thread behaviour. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5471ec7 to
7b6edc2
Compare
…t comments - Suppress all 3 _emit_runtime_warning calls during clean shutdown (timeout, health-check, and exception paths) to avoid misleading logs - Add comment documenting residual health() call window in shutdown join - Correct abort mechanism comments: CPython abandons daemon threads at teardown, does not forcibly kill them (gh-97940)
Add the AUTHOR_MAP entry required for the salvaged NousResearch#49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for identifying the unjoined runtime-autostart thread. The underlying gap is present on current main: plugins/memory/openviking/__init__.py:2042 starts _runtime_start_thread as a daemon, and shutdown() at :3320-3341 does not join it.
Problems
plugins/memory/openviking/__init__.py:3349addsself._prefetch_lock/self._prefetch_threads, but neither attribute exists in the provider. The added block makes everyshutdown()raiseAttributeError.- The new
should_stopkwarg atplugins/memory/openviking/__init__.py:2069invalidates the exact kwargs assertion intests/plugins/memory/test_openviking_provider.py:915-918; this existing test must be updated.
Suggested changes
- Drop the unrelated prefetch block unless matching tracked prefetch state is added in this provider.
- Extend the existing runtime-waiter test to verify the callback is supplied and returns false before shutdown.
Automated hermes-sweeper review.
| @@ -3330,6 +3349,15 @@ def shutdown(self) -> None: | |||
| deferred_workers = list(self._deferred_commit_threads) | |||
There was a problem hiding this comment.
OpenVikingMemoryProvider has no _prefetch_lock or _prefetch_threads field (verified across the current provider), so this makes every shutdown() raise AttributeError. Please remove this unrelated block, or add the actual tracked prefetch state and its producer if prefetch is intended here.
| if not _wait_for_openviking_health( | ||
| endpoint, | ||
| timeout_seconds=_LOCAL_OPENVIKING_AUTOSTART_TIMEOUT, | ||
| should_stop=lambda: self._shutting_down, |
There was a problem hiding this comment.
This new keyword breaks the existing exact-kwargs assertion in tests/plugins/memory/test_openviking_provider.py:915-918, which currently expects only timeout_seconds. Update that test to expect and exercise the callback.
Add the AUTHOR_MAP entry required for the salvaged #49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.
Add the AUTHOR_MAP entry required for the salvaged NousResearch#49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.
Problem
OpenVikingMemoryProvider.shutdown()joins its in-flight writers, deferred-commit threads, and prefetch threads — but not_runtime_start_thread, the trackeddaemon=Truewaiter started by_start_runtime_openviking_waiter(). That thread runs_finish_runtime_openviking_start(), which blocks on network I/O:_wait_for_openviking_health()polls reachability for up to_LOCAL_OPENVIKING_AUTOSTART_TIMEOUT(60s) and then issues a_VikingClient.health()request.If the local OpenViking runtime is slow or unreachable, that waiter can still be blocked in network I/O at interpreter exit. CPython then forcibly kills it during
Py_FinalizeEx(PyThread_exit_thread→__pthread_unwind→abort()), producing SIGABRT (exit 134) with no traceback.This is the same daemon-thread-at-exit failure class that was fixed for the Honcho provider (gh-97940 / bpo-20526) — OpenViking has the same shape but one tracked thread was omitted from
shutdown().Fix
shutdown()now joins_runtime_start_thread(timeout-bounded) alongside the other tracked threads._wait_for_openviking_health()gains an optionalshould_stopcallback; the waiter passeslambda: self._shutting_downso the poll loop bails out promptly onceshutdown()sets the flag — otherwise the join would just time out against the 60s autostart wait and leave the thread alive (defeating the purpose). This mirrors how the Honcho fix closes the httpx client to unblock its worker so the subsequent join lands.Tests
tests/plugins/memory/test_openviking_shutdown.py:should_stopreturns true;shutdown()actually waits for the runtime-start thread (the fake waiter's post-stop work has completed by the timeshutdown()returns; without the join it would not).Both pass locally. Found while hardening memory-provider teardown after the Honcho oneshot SIGABRT fix (#49498).