refactor(codex): drop SDK responses.stream() helper; consume events directly by teknium1 · Pull Request #33042 · NousResearch/hermes-agent

teknium1 · 2026-05-27T05:10:19Z

Summary

The OpenAI SDK's high-level client.responses.stream(...) helper does post-hoc typed reconstruction from response.completed.response.output. The chatgpt.com Codex backend (gpt-5.5, today) returns that field as null, crashing the SDK with TypeError: 'NoneType' object is not iterable mid-iteration.

#32963 patched the symptom. This removes the helper from the call path entirely — responses.create(stream=True) raw event iteration + per-event reconstruction. Same strategy OpenClaw uses for the same backend. Bug class is now structurally impossible, not patched.

Closes the recurrence vector for the May 26 outage (issue #11179) and the (today, gpt-5.5) recurrence pattern.

Changes

agent/codex_runtime.py: new _consume_codex_event_stream() shared consumer. run_codex_stream() uses responses.create(stream=True) directly. run_codex_create_stream_fallback() is now a thin alias — the primary IS the fallback shape.
agent/auxiliary_client.py: _CodexCompletionsAdapter uses the same consumer. Old null-output recovery helpers (_responses_null_output_iterable_error, _responses_backfilled_response) removed as unreferenced.
Tests migrated: responses.stream mocks → responses.create iterables. New regression test (test_recovers_output_item_when_terminal_event_has_null_output) asserts the auxiliary path returns streamed items even when the terminal frame's output is literally null.

Validation

Live test against fresh OpenAI Codex OAuth, gpt-5.5 on chatgpt.com/backend-api/codex:

Raw probe confirmed response.completed.response.output = null is current steady-state, not transient
New consumer built full response correctly: status=completed, output_text="OK", usage with reasoning tokens propagated
All 11 SSE event types from the wire consumed (response.created, in_progress, output_item.added, output_item.done, content_part.added, output_text.delta, output_text.done, content_part.done, completed)

Tests: tests/run_agent/test_run_agent_codex_responses.py + tests/agent/test_auxiliary_client.py → 242 passed.

	Before	After
`responses.stream()` calls	2 (main + aux)	0
`get_final_response()` calls	2	0
Try/except for null-output `TypeError`	2 sites	0 (impossible)
Backfill recovery helpers	3 functions	0 (primary path)
Net LOC	—	`-91`

Infographic

…irectly The OpenAI Python SDK's high-level `client.responses.stream(...)` helper does post-hoc typed reconstruction from the terminal `response.completed.response.output` field. The chatgpt.com Codex backend has been observed (today, gpt-5.5) to ship `response.output = null` on terminal frames, which crashes the SDK with `TypeError: 'NoneType' object is not iterable` mid-iteration. Carlton's #32963 patched the symptom by wrapping the helper in try/except and recovering from the same per-event accumulator the SDK was supposed to populate. This PR removes the helper from the call path entirely: we now use `client.responses.create(stream=True)` (raw AsyncIterable of SSE events) and assemble the final response object ourselves from `response.output_item.done` events as they arrive. The terminal event's `output` field is never read for content. Same strategy OpenClaw uses for the same backend. This makes Hermes structurally immune to the bug class, not patched. The next time OpenAI ships a shape change to chatgpt.com's terminal frame, our consumer keeps working because it doesn't read that frame for content — only for usage/status/id. Changes - `agent/codex_runtime.py`: new `_consume_codex_event_stream()` shared consumer; `run_codex_stream()` uses `responses.create(stream=True)`; `run_codex_create_stream_fallback()` collapses into a thin alias since the primary path now does what the fallback used to do. - `agent/auxiliary_client.py`: `_CodexCompletionsAdapter` uses the same consumer; old null-output recovery helpers deleted as unreferenced. - Tests migrated: fixtures that mocked `responses.stream` now mock `responses.create` returning a raw iterable. New regression test asserts the auxiliary path returns streamed items even when the terminal event's `output` is literally `null`. Validation - Live: tested against fresh OAuth on `chatgpt.com/backend-api/codex` with `gpt-5.5` — response built correctly with `response.output=null` on the terminal frame, all events consumed, usage/reasoning tokens propagated. - `tests/run_agent/test_run_agent_codex_responses.py` + `tests/agent/test_auxiliary_client.py`: 242 passed.

github-actions · 2026-05-27T05:10:58Z

🔎 Lint report: `hermes/hermes-5bf34d29` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9398 on HEAD, 9400 on base (✅ -2)

🆕 New issues (5):

Rule	Count
`invalid-assignment`	2
`invalid-argument-type`	2
`unresolved-import`	1

First entries

agent/codex_runtime.py:279: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to `str`
agent/codex_runtime.py:482: [invalid-argument-type] invalid-argument-type: Argument to function `_consume_codex_event_stream` is incorrect: Expected `str`, found `Unknown | None`
agent/codex_runtime.py:358: [invalid-assignment] invalid-assignment: Object of type `Any | None` is not assignable to `str`
agent/auxiliary_client.py:818: [invalid-argument-type] invalid-argument-type: Argument to function `_consume_codex_event_stream` is incorrect: Expected `str`, found `Any | None`
tests/run_agent/test_codex_xai_oauth_recovery.py:112: [unresolved-import] unresolved-import: Cannot resolve imported module `httpx`

✅ Fixed issues (4):

Rule	Count
`invalid-argument-type`	2
`invalid-parameter-default`	1
`invalid-type-form`	1

First entries

agent/codex_runtime.py:185: [invalid-parameter-default] invalid-parameter-default: Default value of type `None` is not assignable to annotated parameter type `str`
agent/codex_runtime.py:323: [invalid-argument-type] invalid-argument-type: Argument to function `_codex_backfilled_response` is incorrect: Expected `str`, found `Unknown | None`
agent/auxiliary_client.py:871: [invalid-argument-type] invalid-argument-type: Argument to function `_responses_backfilled_response` is incorrect: Expected `str`, found `Any | None`
agent/codex_runtime.py:210: [invalid-type-form] invalid-type-form: Function `callable` is not valid in a parameter annotation: Did you mean `collections.abc.Callable`?

Unchanged: 4969 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

CI surfaced 10 test failures across tests/run_agent/test_streaming.py and tests/run_agent/test_codex_xai_oauth_recovery.py — both files had their own `responses.stream(...)` mocks I missed in the first sweep. agent/codex_runtime.py: _consume_codex_event_stream() now raises "Codex Responses stream did not emit a terminal response" when the stream ends without any terminal frame AND no usable content. This preserves the signal callers used to get from the SDK's high-level helper, which they distinguished from "completed with empty body" in error handling. Tests migrated: - test_streaming.py: text-delta callback, activity-touch, and remote-protocol-error tests all switch from mocking responses.stream to responses.create returning an iterable of events. - test_codex_xai_oauth_recovery.py: prelude-error tests are recast as wire-error-event tests (the new path raises _StreamErrorEvent directly when the wire emits type=error, which is strictly better than the old two-phase "SDK RuntimeError → retry → fallback"). The retry-on-transport-error test moves from responses.stream side-effect to responses.create side-effect. Verified live against chatgpt.com Codex with gpt-5.5 — AIAgent.chat() through the full codex_responses path returns correctly, 319/319 targeted tests passing.

@xxxigm

…registered Salvages the transport-side fix from #32911 (@xxxigm). Closes #32892. The openai SDK's responses.stream() / responses.parse() eagerly call _make_tools(tools), which iterates tools without a None guard. Passing tools=None raises TypeError: 'NoneType' object is not iterable before any HTTP request is issued (openai==2.24.0). PR #33042 already removed responses.stream() from our own Codex call paths, so the specific iteration crash inside _make_tools is no longer on the hot path. But the right API contract is to omit tools entirely when there are no functions to expose — passing tools=None to the backend is semantically wrong regardless of the SDK's iteration behavior, and we'd hit it again on any future code path that hasn't migrated off responses.stream(). This applies the transport-level part of @xxxigm's fix: move 'tools': response_tools into the if response_tools: branch so the key is omitted when there are no tools, just like tool_choice and parallel_tool_calls already are. Skips the run_agent.py-side _strip_sdk_none_iterables helper from their PR — that path is now obsolete because the SDK helper that needed defending is gone. Tests - tests/run_agent/test_codex_no_tools_nonetype.py: 6 tests trimmed from @xxxigm's original 13-test file. Drops the obsolete tests for _strip_sdk_none_iterables and _RecordingResponsesStream (helpers that don't exist on main anymore), keeps the transport behavior tests + the SDK contract sanity check that ensures we notice if upstream ever fixes _make_tools(None). - 6/6 passing locally. Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>

…irectly (NousResearch#33042) * refactor(codex): drop SDK responses.stream() helper; consume events directly The OpenAI Python SDK's high-level `client.responses.stream(...)` helper does post-hoc typed reconstruction from the terminal `response.completed.response.output` field. The chatgpt.com Codex backend has been observed (today, gpt-5.5) to ship `response.output = null` on terminal frames, which crashes the SDK with `TypeError: 'NoneType' object is not iterable` mid-iteration. Carlton's NousResearch#32963 patched the symptom by wrapping the helper in try/except and recovering from the same per-event accumulator the SDK was supposed to populate. This PR removes the helper from the call path entirely: we now use `client.responses.create(stream=True)` (raw AsyncIterable of SSE events) and assemble the final response object ourselves from `response.output_item.done` events as they arrive. The terminal event's `output` field is never read for content. Same strategy OpenClaw uses for the same backend. This makes Hermes structurally immune to the bug class, not patched. The next time OpenAI ships a shape change to chatgpt.com's terminal frame, our consumer keeps working because it doesn't read that frame for content — only for usage/status/id. Changes - `agent/codex_runtime.py`: new `_consume_codex_event_stream()` shared consumer; `run_codex_stream()` uses `responses.create(stream=True)`; `run_codex_create_stream_fallback()` collapses into a thin alias since the primary path now does what the fallback used to do. - `agent/auxiliary_client.py`: `_CodexCompletionsAdapter` uses the same consumer; old null-output recovery helpers deleted as unreferenced. - Tests migrated: fixtures that mocked `responses.stream` now mock `responses.create` returning a raw iterable. New regression test asserts the auxiliary path returns streamed items even when the terminal event's `output` is literally `null`. Validation - Live: tested against fresh OAuth on `chatgpt.com/backend-api/codex` with `gpt-5.5` — response built correctly with `response.output=null` on the terminal frame, all events consumed, usage/reasoning tokens propagated. - `tests/run_agent/test_run_agent_codex_responses.py` + `tests/agent/test_auxiliary_client.py`: 242 passed. * test+fix(codex): migrate streaming tests, raise on truncated streams CI surfaced 10 test failures across tests/run_agent/test_streaming.py and tests/run_agent/test_codex_xai_oauth_recovery.py — both files had their own `responses.stream(...)` mocks I missed in the first sweep. agent/codex_runtime.py: _consume_codex_event_stream() now raises "Codex Responses stream did not emit a terminal response" when the stream ends without any terminal frame AND no usable content. This preserves the signal callers used to get from the SDK's high-level helper, which they distinguished from "completed with empty body" in error handling. Tests migrated: - test_streaming.py: text-delta callback, activity-touch, and remote-protocol-error tests all switch from mocking responses.stream to responses.create returning an iterable of events. - test_codex_xai_oauth_recovery.py: prelude-error tests are recast as wire-error-event tests (the new path raises _StreamErrorEvent directly when the wire emits type=error, which is strictly better than the old two-phase "SDK RuntimeError → retry → fallback"). The retry-on-transport-error test moves from responses.stream side-effect to responses.create side-effect. Verified live against chatgpt.com Codex with gpt-5.5 — AIAgent.chat() through the full codex_responses path returns correctly, 319/319 targeted tests passing.

@pavegrid-1

…dapter Closes NousResearch#33368. `_CodexCompletionsAdapter.create()` iterates `final.output` from the Codex Responses stream. The event-driven consumer (introduced in NousResearch#33042) always sets `final.output` to a list, so this shape can't come from our own code path. But: - Mocked clients in tests can return a typed Response with `output=None` - Third-party shims / compatibility layers that bypass the consumer can do the same - A future code path that wraps a different consumer could regress The old code `getattr(final, "output", [])` returns `None` (not the default `[]`) when the attribute EXISTS but is `None`. Iterating `None` then raises `TypeError: 'NoneType' object is not iterable` — the exact error logged by title-generation when this fires. Fix: `getattr(final, "output", None) or []` — single-line defensive coerce. Cheap; zero risk. Regression test asserts the auxiliary path handles a final whose `.output` is `None` (via monkey-patched consumer) without raising and returns the expected chat.completions-shaped response. Reporter: @pavegrid-1 (issue NousResearch#33368).

@Brixyy

…ble provider shape error Salvages the intent of NousResearch#33136 (@Brixyy) onto current main. The original PR was written against the pre-refactor monolithic run_agent.py and added a top-level _is_nonretryable_local_validation_error() helper. Both target functions have since been extracted to agent/conversation_loop.py:2869, so the salvage applies the equivalent guard inline at that canonical location rather than reintroducing the helper. ## Why After NousResearch#33042 made our own Codex consumer structurally immune to NoneType crashes, third-party shims, mocked clients, and any future code path that hasn't migrated could still surface TypeError: 'NoneType' object is not iterable as a wire-shape mismatch. The agent loop's classifier currently treats ALL TypeError as a local programming bug and aborts non-retryable — users on stale Telegram/gateway turns saw bare "Non-retryable error (HTTP None)" with no recovery. This is a provider/SDK shape mismatch, not a local programming bug. The retry/fallback path should run, not be short-circuited. ## What agent/conversation_loop.py: extend is_local_validation_error to exclude TypeErrors whose message matches the NoneType-not-iterable shape (case- insensitive, both "NoneType" and "not iterable" must appear). tests/run_agent/test_jsondecodeerror_retryable.py: - update the mirror predicate to match the production check - add TestNoneTypeNotIterableIsRetryable class with 3 tests (the basic shape, message variants, unrelated TypeErrors still abort) - add TestAgentLoopSourceHasNoneTypeCarveOut to enforce the source-level invariant matches the test mirror ## Validation tests/run_agent/test_jsondecodeerror_retryable.py + tests/run_agent/test_31273_402_not_retried.py → 14/14 passing Co-authored-by: Brixyy <subrtt@gmail.com>

@xxxigm

…registered Salvages the transport-side fix from NousResearch#32911 (@xxxigm). Closes NousResearch#32892. The openai SDK's responses.stream() / responses.parse() eagerly call _make_tools(tools), which iterates tools without a None guard. Passing tools=None raises TypeError: 'NoneType' object is not iterable before any HTTP request is issued (openai==2.24.0). PR NousResearch#33042 already removed responses.stream() from our own Codex call paths, so the specific iteration crash inside _make_tools is no longer on the hot path. But the right API contract is to omit tools entirely when there are no functions to expose — passing tools=None to the backend is semantically wrong regardless of the SDK's iteration behavior, and we'd hit it again on any future code path that hasn't migrated off responses.stream(). This applies the transport-level part of @xxxigm's fix: move 'tools': response_tools into the if response_tools: branch so the key is omitted when there are no tools, just like tool_choice and parallel_tool_calls already are. Skips the run_agent.py-side _strip_sdk_none_iterables helper from their PR — that path is now obsolete because the SDK helper that needed defending is gone. Tests - tests/run_agent/test_codex_no_tools_nonetype.py: 6 tests trimmed from @xxxigm's original 13-test file. Drops the obsolete tests for _strip_sdk_none_iterables and _RecordingResponsesStream (helpers that don't exist on main anymore), keeps the transport behavior tests + the SDK contract sanity check that ensures we notice if upstream ever fixes _make_tools(None). - 6/6 passing locally. Co-authored-by: xxxigm <tuancanhnguyen706@gmail.com>

@pavegrid-1

…dapter Closes NousResearch#33368. `_CodexCompletionsAdapter.create()` iterates `final.output` from the Codex Responses stream. The event-driven consumer (introduced in NousResearch#33042) always sets `final.output` to a list, so this shape can't come from our own code path. But: - Mocked clients in tests can return a typed Response with `output=None` - Third-party shims / compatibility layers that bypass the consumer can do the same - A future code path that wraps a different consumer could regress The old code `getattr(final, "output", [])` returns `None` (not the default `[]`) when the attribute EXISTS but is `None`. Iterating `None` then raises `TypeError: 'NoneType' object is not iterable` — the exact error logged by title-generation when this fires. Fix: `getattr(final, "output", None) or []` — single-line defensive coerce. Cheap; zero risk. Regression test asserts the auxiliary path handles a final whose `.output` is `None` (via monkey-patched consumer) without raising and returns the expected chat.completions-shaped response. Reporter: @pavegrid-1 (issue NousResearch#33368).

github-advanced-security AI found potential problems May 27, 2026

View reviewed changes

Comment thread agent/codex_runtime.py Dismissed

Comment thread agent/codex_runtime.py Dismissed

alt-glitch added type/refactor Code restructuring, no behavior change P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/openai OpenAI / Codex Responses API codex labels May 27, 2026

alt-glitch mentioned this pull request May 27, 2026

fix(codex): urgently recover when streamed terminal output is null #33040

Closed

alt-glitch mentioned this pull request May 27, 2026

ChatGPT/Codex backend crashes openai-python stream parser (output=null on response.completed) #33059

Closed

HenryQW mentioned this pull request May 27, 2026

Hermes openai-codex fails on same machine/network where official Codex CLI still works #13834

Open

yangguangjin mentioned this pull request May 27, 2026

openai-codex/gpt-5.5 still unstable in Hermes v0.14.0: subagents almost always hit APIConnectionError/TTFB timeout while Codex CLI works #33075

Closed

teknium1 merged commit cb38ce2 into main May 27, 2026
25 of 26 checks passed

teknium1 deleted the hermes/hermes-5bf34d29 branch May 27, 2026 07:30

bautrey mentioned this pull request Jun 24, 2026

fix(gateway): reliability hardening — stall watchdog, Telegram poll heartbeat, supervised tasks, launchd respawn throttle #51876

Closed

teknium1 mentioned this pull request Jun 28, 2026

[Bug]: Telegram gateway mixes/queues independent operational requests in shared group chats; business workflows become unsafe under active sessions and runtime failures #33293

Closed

1 task

alt-glitch mentioned this pull request Jun 30, 2026

fix(codex): monkey-patch openai SDK parse_response to handle output=None #55797

Closed

alt-glitch mentioned this pull request Jul 1, 2026

openai-codex provider always returns response.output is empty — DEFAULT_CODEX_BASE_URL endpoint broken #5879

Closed

This was referenced Jul 13, 2026

bug: openai-codex provider — TypeError: 'NoneType' object is not iterable on every request (gpt-5.5) #33991

Closed

codex-oauth: 'NoneType' object is not iterable in _run_codex_stream (gpt-5.5) — every turn fails non-retryably #34124

Closed

Navigation Menu

Uh oh!

refactor(codex): drop SDK responses.stream() helper; consume events directly - #33042

refactor(codex): drop SDK responses.stream() helper; consume events directly#33042
teknium1 merged 2 commits into
mainfrom
hermes/hermes-5bf34d29

teknium1 commented May 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

teknium1 commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Validation

Infographic

Uh oh!

github-actions Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔎 Lint report: hermes/hermes-5bf34d29 vs origin/main

ruff

ty (type checker)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

teknium1 commented May 27, 2026 •

edited

Loading

github-actions Bot commented May 27, 2026 •

edited

Loading

🔎 Lint report: `hermes/hermes-5bf34d29` vs `origin/main`