fix(gateway): defer cron scheduler and kanban init to prevent Windows event loop stall - #73605
Open
webtecnica wants to merge 5 commits into
Open
fix(gateway): defer cron scheduler and kanban init to prevent Windows event loop stall#73605webtecnica wants to merge 5 commits into
webtecnica wants to merge 5 commits into
Conversation
Closes NousResearch#73175 The existing only checked top-level key absence. Arguments that violated , , , nested , , or constraints were dispatched blind, producing opaque downstream failures that cheap models would loop on. Fix uses to catch all non-type constraint violations before dispatch. Type mismatches are intentionally tolerated — Hermes' downstream already handles arbitrary type coercion, and the validator should not be the only source of type repair for deferred tools. The existing required-only fast-path is preserved for backward compatibility and nicer error formatting when top-level required keys are missing. Safety: the entire validator is wrapped in try/except, so any unexpected schema format or exception falls through to blind dispatch (same as before). Type errors (validator == 'type') are silently passed through.
Some OpenAI-compatible TTS backends (e.g. Speaches/Kokoro, custom self-hosted servers) only support a subset of response_formats and reject opus with HTTP 400/422. When _generate_openai_tts() sends response_format=opus (derived from the .ogg output path), these backends fail before any bytes are produced. Fix: - Add _is_response_format_rejection() to identify 400/422 errors that mention response_format. - Add _create_speech_with_format_fallback() that retries once with mp3 on format rejection. - Add _repair_mp3_in_ogg_container() that detects mp3-in-ogg by peeking at the file header and transcodes to real Ogg/Opus via ffmpeg. - Wire the fallback into _generate_openai_tts() and always run the container repair for .ogg paths (it is a no-op on well-formed Ogg data). Closes NousResearch#73470
… event loop stall Defer heavy cron/kanban initialization to after gateway.ready signal, preventing ~45s event loop stall on Windows. Changes: - gateway/run.py: Move resolve_cron_scheduler() and profile discovery into the cron daemon thread target so synchronous YAML config I/O never blocks the event loop during start_gateway(). - hermes_cli/web_server.py: Defer the desktop cron scheduler start from the lifespan handler to the first WebSocket connection's on_ready callback (after gateway.ready is delivered). - tui_gateway/ws.py: Add optional on_ready callback to handle_ws(), fired after gateway.ready is sent, so callers can defer heavy init. Closes NousResearch#73435
alt-glitch
added
type/bug
Contributor
Author
|
@teknium1 This PR defers cron scheduler and kanban init to prevent Windows event loop stall. P2 bug affecting Windows users. Ready for review. |
teknium1
reviewed
Jul 30, 2026
teknium1
left a comment
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for isolating a plausible off-loop startup direction. Current main still resolves the gateway cron provider synchronously at gateway/run.py:25496-25500, but this branch needs correction before it can safely preserve scheduler lifecycle behavior.
Problems
gateway/run.py:24494makes the provider local to_start_cron_in_thread, while the unchanged shutdown path still callscron_provider.stop()atgateway/run.py:24582. That name is no longer bound, so shutdown does not call the provider's stop method.- The PR says kanban initialization is deferred, but the current dispatcher still performs config loading and
kanban_dbimport before its first off-loop call (gateway/kanban_watchers.py:965-995); this PR does not alter that path. - The branch bundles four unrelated commits and is currently conflicting with main; the Windows startup salvage should be separated from those changes.
Suggested changes
- Retain a shutdown-visible provider reference and test start/stop lifecycle behavior.
- Defer the kanban bootstrap too, or reduce the scope and claim to cron/Desktop only.
- Reconcile the focused change with the current web-server warmup path (
hermes_cli/web_server.py:198-204).
Automated hermes-sweeper review.
| target=cron_provider.start, | ||
| args=(cron_stop,), | ||
| kwargs=cron_start_kwargs, | ||
| target=_start_cron_in_thread, |
Contributor
There was a problem hiding this comment.
_provider is local to this thread target, but the unchanged shutdown code later calls cron_provider.stop() at line 24582. Preserve a synchronized outer provider reference (including the shutdown-before-resolution case) and add a start/stop regression test; otherwise provider shutdown is skipped after this change.
| import socket | ||
| import threading | ||
| from typing import Any | ||
| from typing import Any, Awaitable, Callable, Optional, Awaitable, Callable, Optional |
Contributor
There was a problem hiding this comment.
Remove the duplicated Awaitable, Callable, and Optional imports.
teknium1
added
sweeper:risk-compatibility
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #73435 (P1) — ~45s asyncio event loop stall on Windows during hermes serve startup.\n\nRoot cause: cron scheduler + kanban dispatcher initialization (YAML parsing, SQLite I/O) ran synchronously on the event loop thread, stalling WebSocket ready frame delivery.\n\nFix: Deferred heavy init to after gateway.ready signal via deferred-start pattern and on_ready callback.\n\nFiles: gateway/run.py, web_server.py, tui_gateway/ws.py