fix(dashboard): recycle console executor when all workers are permanently stuck

[sprmn24]

What does this PR do?

Fixes a dashboard console denial-of-service: after 4 stuck console command threads, every subsequent command for every session hangs until process restart.

Type of Change

• Bug fix (non-breaking change which fixes an issue)

Changes Made

The console command thread pool (_CONSOLE_EXECUTOR_MAX_WORKERS=4) could become permanently wedged. asyncio.wait_for abandons the await on timeout but cannot kill the Python thread, so a truly stuck worker is gone from the pool forever. After 4 such hangs from any session, every subsequent run_in_executor submission queues behind threads that will never return — hanging the entire dashboard console until process restart.

Fix:

• Add _new_console_executor() factory (moves atexit registration there)
• Track _console_executor_stuck_count under _console_executor_stuck_lock
• Add _note_console_submission_outcome(stuck=True/False): increments on timeout, decrements if the thread eventually returns (merely slow vs truly stuck), and atomically swaps in a fresh pool when all workers are wedged
• Switch run_command from loop.run_in_executor to executor.submit + asyncio.wrap_future so a done-callback fires even after wait_for gives up, letting slow-but-not-stuck submissions decrement the counter

How to Test

Send 4 console commands that each hang longer than 60s. Before: the 5th command from any session hangs forever. After: the pool is automatically replaced and the 5th command executes normally.

Checklist

• Single focused change
• No unrelated modifications
• No new dependencies

[teknium1]

Thanks for addressing a real dashboard-console availability failure: current main still submits console commands to a fixed four-worker pool and only times out the await (hermes_cli/web_server.py:15402-15413).

Problems

• The done callback at hermes_cli/web_server.py:13308 decrements the global count for every non-cancelled future, not just a future that previously timed out. A normal successful command can therefore erase the count for a different permanently blocked worker.
• _console_executor_stuck_count has no executor-generation ownership (hermes_cli/web_server.py:13017). A late completion from a retired executor can decrement timeout state accumulated by the replacement executor.
• The diff adds no timeout/replacement regression test; existing console WS tests cover normal execution and cancellation only (tests/hermes_cli/test_web_server_console_ws.py:73-134).

Suggested changes

• Associate timeout bookkeeping with each future and executor generation; only a timed-out future's completion may clear its own mark.
• Add deterministic tests for four blocked workers, replacement, and late completion from the old pool.

Automated hermes-sweeper review.

[teknium1]

This callback runs for every non-cancelled future, including commands that finished before the timeout. If one worker has timed out, a later normal command decrements its global count and can prevent the pool from recycling after all four workers are actually stuck. Track timeout state on this specific future (and its executor generation) before clearing it.