fix(streaming): attribute worker-thread failures to their session

[fabiosiqueira]

The gap

hermes_logging promises one thing: every log record in the process carries a
[session_id] tag, so lines from concurrent sessions can be told apart. The
tag comes from a threading.local (hermes_logging.py:80), set by
set_session_context().

interruptible_streaming_api_call runs the provider call on a worker
thread. Nothing binds that thread, so it starts unbound and every record it
emits is formatted without a session tag — the contract breaks silently, and it
breaks precisely where it costs the most: the worker is also where a stream
failure is logged. In a process serving concurrent sessions, the one line
that carries the cause of a failed stream is the one line that names no
session.

The second half of the same line: Streaming failed before delivery logs the
exception without exc_info. That branch is not transport-only — it also
catches client-side exceptions raised while accumulating the stream. So all
that survives a failure is

INFO agent.chat_completion_helpers: Streaming failed before delivery: 'dict' object has no attribute 'model_dump'

No session (it cannot be tied to a turn) and no traceback (the raising frame,
ours or the SDK's, is unknown). I hit this debugging a project of my own: with
more than one session live in the process, that line was unattributable after
the fact, and the failure had to be chased by other means.

The change

Two edits in agent/chat_completion_helpers.py:

• bind_worker_session_context(agent) — a small helper that binds
  agent.session_id to the current thread's log context, called as the first
  statement of the streaming worker's _call(). Best-effort by contract: an
  agent without session_id leaves the thread unbound rather than raising
  inside the worker.
• exc_info=True on the pre-delivery stream failure log.

This is the same class of gap as #41726 (sync logging session context on
compaction id rotation) — a code path that runs outside the thread the context
was bound on. No behavior changes beyond attribution and the traceback; the log
level is unchanged.

Why not the existing _context_thread_target?

The worker is already wrapped with it (chat_completion_helpers.py:4071), but
that helper propagates ContextVars, and the session context is a
threading.local (hermes_logging.py:80) — contextvars.copy_context() does
not carry it. Making _session_context a ContextVar would fix this class of
gap everywhere at once and is arguably the better end state, but it changes the
semantics of set_session_context / clear_session_context for every caller,
so I kept this PR to the one call site. Happy to take that route instead if
you'd prefer it.

Tests

tests/agent/test_stream_worker_session_context.py — 4 cases covering the
binding's contract:

• worker records carry the agent's session tag (this is the regression; it goes
  red if the helper is a no-op)
• the binding does not leak into the calling thread
• a session-less agent leaves the worker unbound
• a broken logging module does not raise inside the worker

The helper resolves hermes_logging through sys.modules and reinstalls the
record factory from the live module, so the tests are order-independent in the
full suite (several tests reload modules).

pytest tests/agent/test_stream_worker_session_context.py   4 passed
pytest tests/agent -k stream                               122 passed, 4 skipped
pytest tests/agent                                         no new failures vs. main

— 🤖 Claude Opus 5

[alt-glitch]

This was generated by AI during triage.

Related to #41726: that fix updates log context after compaction ID rotation; this patch addresses the separate streaming-worker thread boundary.

[teknium1]

Thanks for isolating the thread-local logging gap; the standard streaming-worker premise is real on current main. hermes_logging.py:80 uses threading.local, while agent/chat_completion_helpers.py:61-64 copies only ContextVars into worker threads.

Problems

• The added binding covers only the normal _call worker. interruptible_streaming_api_call also runs Bedrock streaming in _bedrock_call (agent/chat_completion_helpers.py:2553), launched at :2669-2671; that worker can emit the stream-denial log at :2587-2591 and remains untagged.
• tests/agent/test_stream_worker_session_context.py exercises the helper directly, not the production interruptible_streaming_api_call worker placement, and has no Bedrock-path coverage.

Suggested changes

• Bind the session context at the start of _bedrock_call too.
• Add production-path tests for standard and Bedrock streaming workers, asserting a worker-created record carries the agent session tag.

Automated hermes-sweeper review.

[teknium1]

This binds only the OpenAI/Anthropic worker. interruptible_streaming_api_call also starts the Bedrock _bedrock_call worker at current main agent/chat_completion_helpers.py:2553/:2669; its stream-denial branch logs at :2587, so please bind the same context there or Bedrock streaming remains unattributed.

[fabiosiqueira]

Both points addressed in ba83507cc, rebased onto current main (was 527 commits
behind; now mergeable, single commit). One thing the rebase turned up that changes
the diff, below.

1. Bedrock worker

You were right that the binding covered only _call. _bedrock_call now binds at
its own entry (agent/chat_completion_helpers.py:2583), so the IAM stream-denial log
carries the session tag too.

Worth naming why it's bound from agent.session_id rather than inherited from the
calling thread: _context_thread_target already copies the caller's ContextVars, so
the tempting fix is to propagate the log context there and cover every worker at
once. That silently does nothing for any caller that reached streaming without going
through turn_context — the calling thread is itself unbound in that case. The agent
is the ground truth; the caller's thread-local is a side effect.

2. Production-path tests

Both tests now drive the real interruptible_streaming_api_call and assert on a
record emitted from inside the worker (matched on record.thread, since handlers
run on the emitting thread). The calling thread is deliberately left unbound in the
fixture, so a tag leaking in from the test itself cannot make them pass:

• standard path — provider raises before any delta, worker takes the
  "Streaming failed before delivery" branch;
• Bedrock path — converse_stream denied, worker logs the IAM fallback
  (mirrors the fixture in tests/agent/test_bedrock_interrupt_post_worker.py).

Verified under mutation, not just green: dropping the bind from _bedrock_call fails
the Bedrock test alone, dropping it from _call fails the standard test alone.

3. The exc_info half is gone — superseded by 8e191af0f

The original commit also added exc_info=True to the pre-delivery failure log. On
2026-07-31 8e191af0f (@knocksen, fix(moa): stream completed aggregator responses safely) converted that same call to logger.exception, which is a strict superset —
traceback plus ERROR level. That was the rebase conflict, and keeping our side would
have quietly demoted the level back to INFO. Dropped, and the commit message no longer
claims it. This PR is now purely the session-attribution change.

Also renamed the helper to _bind_worker_session_context — it has no callers outside
this module, and everything private here is underscore-prefixed.

Known gap, deliberately not folded in

interruptible_api_call's worker (:897) has the same root cause and three log sites
inside it (:754, :859, :871). It's the same one-line fix, but this PR is scoped
to streaming; I'd rather send it separately than widen a PR you've already reviewed.
Say the word if you'd prefer it here instead.

No overlap with #11212 (@wantingNBi): that one makes the thread-local a per-thread
stack so nested run_conversation restores the parent tag. A freshly spawned worker
still starts with an empty stack, so the two are complementary — and our single
set_session_context call happens in a daemon thread that then exits, so it pushes
nothing that needs popping.

Verified: tests/agent/test_stream_worker_session_context.py 5 passed; the
neighbouring streaming/Bedrock/logging suites clean (188 passed across
test_bedrock_interrupt_post_worker, test_bedrock_integration, test_bedrock_adapter,
test_streaming, test_partial_stream_finish_reason, test_stream_interrupt_retry,
test_stream_stale_circuit_breaker, test_stream_single_writer_65991,
test_hermes_logging, test_compression_logging_session_context), plus py_compile
on the touched modules.

— 🤖 Claude Opus 5