Skip to content

fix(tools): spawn daemon workers with the 3.14 worker context - #72955

Closed
mrwogu wants to merge 1 commit into
NousResearch:mainfrom
mrwogu:fix/python314-daemon-pool-worker-context
Closed

fix(tools): spawn daemon workers with the 3.14 worker context#72955
mrwogu wants to merge 1 commit into
NousResearch:mainfrom
mrwogu:fix/python314-daemon-pool-worker-context

Conversation

@mrwogu

@mrwogu mrwogu commented Jul 27, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes the DaemonThreadPoolExecutor crash on Python 3.14 and, unlike the open patch in #57459, keeps the pool actually working there.

CPython 3.14 changed concurrent.futures.thread._worker from (executor_ref, work_queue, initializer, initargs) to (executor_ref, ctx, work_queue), where ctx comes from self._create_worker_context(). ThreadPoolExecutor.__init__ no longer sets _initializer / _initargs at all. tools/daemon_pool.py still mirrored the 3.8-3.13 shape, so every worker spawn raised:

AttributeError: 'DaemonThreadPoolExecutor' object has no attribute '_initializer'

Live traceback from a Homebrew install running on 3.14.6 (hermes-agent/2026.6.5), triggered by an ordinary parallel tool batch:

File "run_agent.py", line 6309, in _execute_tool_calls_concurrent
  return execute_tool_calls_concurrent(self, assistant_message, messages, effective_task_id, api_call_count)
File "agent/tool_executor.py", line 691, in execute_tool_calls_concurrent
  f = executor.submit(
File ".../python3.14/concurrent/futures/thread.py", line 215, in submit
  self._adjust_thread_count()
File "tools/daemon_pool.py", line 58, in _adjust_thread_count
  self._initializer,
AttributeError: 'DaemonThreadPoolExecutor' object has no attribute '_initializer'

The agent surfaced it as Error during OpenAI-compatible API call #N, which sends the outer loop into retries for what is really a local spawn failure.

Why this approach rather than #57459

#57459 reads the two attributes through getattr(..., None) but still passes a 4-tuple to a 3-parameter _worker. That silences the AttributeError at submit() and moves the failure into the worker thread, where TypeError kills the thread and the future never resolves. Loud crash becomes a hang. Probe on this machine, same class shapes side by side:

python 3.14.6
_worker params: 3
has _create_worker_context: True
Exception in thread probe_0:
TypeError: _worker() takes 3 positional arguments but 4 were given
getattr patch (#57459): HUNG: future never completed (worker thread died)
worker-context patch:   result=42

python 3.11.15
_worker params: 4
has _create_worker_context: False
getattr patch (#57459): result=42
worker-context patch:   result=42

So the fix has to switch the whole argument shape, not just guard the attribute reads. Same code site as #57459, #58598, #59157, #63777 and #63780; this one is the version that survives a submit() on 3.14.

requires-python is >=3.11,<3.14, which is why #50077 was closed as unreproducible. That bound does not protect installs in practice: the Homebrew formula builds the app against python@3.14, so the interpreter that runs hermes is 3.14.6 regardless. The patch keeps the legacy branch intact, so nothing changes on 3.11-3.13 whether or not the supported range moves later.

Related Issue

Fixes #58596

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • tools/daemon_pool.py: _adjust_thread_count now takes its thread arguments from a new _worker_args() helper, which returns (executor_ref, create_context(), work_queue) when the interpreter exposes _create_worker_context and the legacy (executor_ref, work_queue, initializer, initargs) tuple otherwise. Daemon threads and the skipped _threads_queues registration are unchanged, so abandoned workers still cannot block interpreter exit.
  • tests/tools/test_daemon_pool.py: adds test_worker_args_match_stdlib_worker_signature, which asserts the argument count matches inspect.signature(_worker) on whatever interpreter runs the suite. The next CPython signature change then fails in CI instead of in a user's parallel tool batch.

How to Test

  1. Reproduce on 3.14 before the patch: python3.14 -c "import sys; sys.path.insert(0,'.'); from tools.daemon_pool import DaemonThreadPoolExecutor as P; p=P(max_workers=1); print(p.submit(lambda: 1+1).result(timeout=5))" raises AttributeError: ... has no attribute '_initializer'.
  2. Run the same command after the patch: prints 2.
  3. Confirm daemon semantics still hold on 3.14: python3.14 with max_workers=2, initializer/initargs, then assert the worker reports threading.current_thread().daemon is True, that it is absent from concurrent.futures.thread._threads_queues, that the initializer ran, and that a second submit reuses the same thread id. All four hold.
  4. Suite: scripts/run_tests.sh tests/tools/test_daemon_pool.py -q (5 passed) and the consumer slices scripts/run_tests.sh tests/tools/test_daemon_pool.py tests/tools/test_async_delegation.py tests/tools/test_delegate.py -q (205 passed).
  5. python3 scripts/check-windows-footguns.py tools/daemon_pool.py tests/tools/test_daemon_pool.py reports no findings.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate — fix(tools): Python 3.14 compatibility for DaemonThreadPoolExecutor #57459 is open at the same site; the probe above shows its patch hangs on 3.14, so this supersedes rather than repeats it
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — via scripts/run_tests.sh on the touched module and its consumers
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15 (Darwin 25.5.0), CPython 3.11.15 and 3.14.6

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — module docstring already documents the mirrored implementation; the version range comment is updated to 3.8-3.14
  • N/A — no config keys changed
  • N/A — no architecture or workflow change
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — the branch is interpreter-version based, not platform based, and touches no paths, signals, or subprocesses
  • N/A — no tool descriptions or schemas changed

Screenshots / Logs

Probe script used for the comparison above (throwaway, not part of the diff): it subclasses ThreadPoolExecutor twice, once with the #57459 argument shape and once with this PR's, then submits one task with result(timeout=5) on each. Output for 3.14.6 and 3.11.15 is quoted in full in the section above.

CPython 3.14 replaced the `(work_queue, initializer, initargs)` worker
arguments with a per-worker context object and dropped the executor's
`_initializer` / `_initargs` attributes. `_adjust_thread_count` mirrored
the 3.8-3.13 shape, so every parallel tool batch raised
`AttributeError: 'DaemonThreadPoolExecutor' object has no attribute
'_initializer'` in `submit()`.

Worker arguments are now built from `_create_worker_context()` when the
interpreter provides it and from the legacy attributes otherwise, so the
same source works on 3.11 through 3.14. A regression test ties the
argument count to the stdlib `_worker` signature so the next CPython
change fails loudly instead of at runtime.
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists labels Jul 27, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #65182: both adapt DaemonThreadPoolExecutor worker arguments for CPython 3.14 using _create_worker_context() at the same spawn site. #57459's obsolete four-argument worker call is not the canonical fix.

@mrwogu

mrwogu commented Jul 28, 2026

Copy link
Copy Markdown
Author

Agreed on the duplicate call: #65182 is older and uses the same _create_worker_context() contract, so it should be the one that lands. Closing this.

The parts of this PR that are not in #65182 are moved to #65182 (comment):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DaemonThreadPoolExecutor crashes on Python 3.14: _initializer attribute removed

2 participants