Skip to content

fix(openviking): join runtime-autostart thread on shutdown (SIGABRT-at-exit) - #49832

Closed
koshaji wants to merge 2 commits into
NousResearch:mainfrom
koshaji:fix/openviking-runtime-start-thread-shutdown
Closed

fix(openviking): join runtime-autostart thread on shutdown (SIGABRT-at-exit)#49832
koshaji wants to merge 2 commits into
NousResearch:mainfrom
koshaji:fix/openviking-runtime-start-thread-shutdown

Conversation

@koshaji

@koshaji koshaji commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Problem

OpenVikingMemoryProvider.shutdown() joins its in-flight writers, deferred-commit threads, and prefetch threads — but not _runtime_start_thread, the tracked daemon=True waiter started by _start_runtime_openviking_waiter(). That thread runs _finish_runtime_openviking_start(), which blocks on network I/O: _wait_for_openviking_health() polls reachability for up to _LOCAL_OPENVIKING_AUTOSTART_TIMEOUT (60s) and then issues a _VikingClient.health() request.

If the local OpenViking runtime is slow or unreachable, that waiter can still be blocked in network I/O at interpreter exit. CPython then forcibly kills it during Py_FinalizeEx (PyThread_exit_thread__pthread_unwindabort()), producing SIGABRT (exit 134) with no traceback.

This is the same daemon-thread-at-exit failure class that was fixed for the Honcho provider (gh-97940 / bpo-20526) — OpenViking has the same shape but one tracked thread was omitted from shutdown().

Fix

  1. shutdown() now joins _runtime_start_thread (timeout-bounded) alongside the other tracked threads.
  2. _wait_for_openviking_health() gains an optional should_stop callback; the waiter passes lambda: self._shutting_down so the poll loop bails out promptly once shutdown() sets the flag — otherwise the join would just time out against the 60s autostart wait and leave the thread alive (defeating the purpose). This mirrors how the Honcho fix closes the httpx client to unblock its worker so the subsequent join lands.

Tests

tests/plugins/memory/test_openviking_shutdown.py:

  • the health waiter short-circuits (no network probe) when should_stop returns true;
  • shutdown() actually waits for the runtime-start thread (the fake waiter's post-stop work has completed by the time shutdown() returns; without the join it would not).

Both pass locally. Found while hardening memory-provider teardown after the Honcho oneshot SIGABRT fix (#49498).

Copilot AI review requested due to automatic review settings June 20, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a daemon-thread-at-interpreter-exit crash class in the OpenViking memory provider by ensuring the runtime-autostart waiter thread can be stopped promptly and is joined during shutdown.

Changes:

  • Add a should_stop callback to _wait_for_openviking_health() and pass the provider shutdown flag from the runtime-autostart waiter.
  • Update OpenVikingMemoryProvider.shutdown() to also join the runtime-autostart thread.
  • Add tests covering the new early-exit behavior and verifying shutdown() waits for the runtime-start thread to complete.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
plugins/memory/openviking/__init__.py Adds shutdown-aware health waiting and joins the autostart waiter thread during provider teardown.
tests/plugins/memory/test_openviking_shutdown.py New tests to prevent regressions around shutdown and daemon waiter behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 1908 to 1912
if not _wait_for_openviking_health(
endpoint,
timeout_seconds=_LOCAL_OPENVIKING_AUTOSTART_TIMEOUT,
should_stop=lambda: self._shutting_down,
):
@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins tool/memory Memory tool and memory providers P3 Low — cosmetic, nice to have labels Jun 20, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: #49498 (Honcho memory-provider SIGABRT-at-exit fix).

Same daemon-thread-at-exit failure class, different provider: this joins OpenViking's tracked _runtime_start_thread (the autostart health waiter) in shutdown() — the one thread the existing teardown omitted — and adds a should_stop short-circuit so the join lands instead of timing out against the 60s autostart wait. Sibling fix to #49498, not a duplicate.

…t-exit)

`OpenVikingMemoryProvider.shutdown()` joins in-flight writers, deferred-commit
threads, and prefetch threads, but not `_runtime_start_thread` — the tracked
`daemon=True` waiter that runs `_finish_runtime_openviking_start`, which blocks
on network health probes (`_wait_for_openviking_health` polling + a
`_VikingClient.health()` request).

If the local OpenViking runtime is slow or unreachable, that waiter can still
be blocked in network I/O at interpreter exit. CPython then forcibly kills it
during `Py_FinalizeEx` (`PyThread_exit_thread` -> `__pthread_unwind` ->
`abort()`), producing SIGABRT (exit 134) with no traceback — the same daemon-
thread-at-exit failure class fixed for the Honcho provider.

Fix:
- `shutdown()` now joins `_runtime_start_thread` (timeout-bounded) alongside the
  other tracked threads.
- `_wait_for_openviking_health()` gains a `should_stop` callback; the waiter
  passes `lambda: self._shutting_down` so the poll loop bails out promptly once
  `shutdown()` flips the flag, instead of lingering up to the 60s autostart
  timeout and timing out the join (which would leave the thread alive).
- Add tests/plugins/memory/test_openviking_shutdown.py covering the short-circuit
  and the shutdown-joins-runtime-thread behaviour.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@koshaji
koshaji force-pushed the fix/openviking-runtime-start-thread-shutdown branch from 5471ec7 to 7b6edc2 Compare July 11, 2026 12:29
…t comments

- Suppress all 3 _emit_runtime_warning calls during clean shutdown
  (timeout, health-check, and exception paths) to avoid misleading logs
- Add comment documenting residual health() call window in shutdown join
- Correct abort mechanism comments: CPython abandons daemon threads at
  teardown, does not forcibly kill them (gh-97940)
ehz0ah added a commit to ehz0ah/hermes-agent that referenced this pull request Jul 14, 2026
Add the AUTHOR_MAP entry required for the salvaged NousResearch#49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for identifying the unjoined runtime-autostart thread. The underlying gap is present on current main: plugins/memory/openviking/__init__.py:2042 starts _runtime_start_thread as a daemon, and shutdown() at :3320-3341 does not join it.

Problems

  • plugins/memory/openviking/__init__.py:3349 adds self._prefetch_lock / self._prefetch_threads, but neither attribute exists in the provider. The added block makes every shutdown() raise AttributeError.
  • The new should_stop kwarg at plugins/memory/openviking/__init__.py:2069 invalidates the exact kwargs assertion in tests/plugins/memory/test_openviking_provider.py:915-918; this existing test must be updated.

Suggested changes

  • Drop the unrelated prefetch block unless matching tracked prefetch state is added in this provider.
  • Extend the existing runtime-waiter test to verify the callback is supplied and returns false before shutdown.

Automated hermes-sweeper review.

@@ -3330,6 +3349,15 @@ def shutdown(self) -> None:
deferred_workers = list(self._deferred_commit_threads)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenVikingMemoryProvider has no _prefetch_lock or _prefetch_threads field (verified across the current provider), so this makes every shutdown() raise AttributeError. Please remove this unrelated block, or add the actual tracked prefetch state and its producer if prefetch is intended here.

if not _wait_for_openviking_health(
endpoint,
timeout_seconds=_LOCAL_OPENVIKING_AUTOSTART_TIMEOUT,
should_stop=lambda: self._shutting_down,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new keyword breaks the existing exact-kwargs assertion in tests/plugins/memory/test_openviking_provider.py:915-918, which currently expects only timeout_seconds. Update that test to expect and exercise the callback.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 14, 2026
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Superseded by #51952, which includes your commit cherry-picked with authorship preserved (joining the runtime-autostart thread on shutdown to fix the SIGABRT-at-exit, adapted to keep the existing memory-write drain). Thanks @koshaji — credit retained in git history and scripts/release.py.

@koshaji
koshaji deleted the fix/openviking-runtime-start-thread-shutdown branch July 22, 2026 08:53
kshitijk4poor pushed a commit that referenced this pull request Jul 24, 2026
Add the AUTHOR_MAP entry required for the salvaged #49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
Add the AUTHOR_MAP entry required for the salvaged NousResearch#49832 OpenViking shutdown fix so contributor attribution CI can resolve the original author.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/memory Memory tool and memory providers type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants