Skip to content

[codex] Fix silent partial-stream recovery and bounded gateway restart - #41498

Closed
0-CYBERDYNE-SYSTEMS-0 wants to merge 1 commit into
NousResearch:mainfrom
0-CYBERDYNE-SYSTEMS-0:codex/fix-partial-stream-restart
Closed

[codex] Fix silent partial-stream recovery and bounded gateway restart#41498
0-CYBERDYNE-SYSTEMS-0 wants to merge 1 commit into
NousResearch:mainfrom
0-CYBERDYNE-SYSTEMS-0:codex/fix-partial-stream-restart

Conversation

@0-CYBERDYNE-SYSTEMS-0

Copy link
Copy Markdown

Summary

Fixes two related gateway failure modes observed during long Telegram agent runs:

  • partial stream recovery now surfaces a visible abnormal-turn explanation instead of treating the recovered fragment as a normal final preview
  • detached in-chat /restart now launches its external restart helper before graceful drain and bounds the helper wait by restart_drain_timeout + 5s

Root Cause

When streaming failed after partial delivery, partial_stream_recovery reused the streamed fragment as the final response and marked response_previewed=True. The gateway could then suppress normal final delivery even though the user only saw a partial stream, leaving the failure silent.

For /restart, the detached restart helper was previously launched from the shutdown path after drain work. If the gateway accepted /restart but then wedged during graceful drain, the helper could be delayed or wait indefinitely for the stuck PID to exit.

Changes

  • Keep response_previewed=False for partial stream recovery so gateway fallback delivery can send the recovered text plus the explanation.
  • Always append the turn-completion explainer for partial_stream_recovery, not only for empty or very short fragments.
  • Launch the detached restart helper as soon as a detached restart is requested.
  • Add an idempotency guard for the helper and bound its PID wait by the configured drain timeout plus a small grace window.
  • Add regression coverage for partial stream recovery visibility and detached restart helper behavior.

Validation

.venv/bin/python -m py_compile agent/conversation_loop.py run_agent.py tests/run_agent/test_run_agent.py tests/run_agent/test_turn_completion_explainer.py gateway/run.py tests/gateway/test_restart_drain.py
.venv/bin/pytest tests/run_agent/test_turn_completion_explainer.py tests/run_agent/test_run_agent.py::TestRunConversation::test_partial_stream_recovery_on_empty_stub tests/run_agent/test_run_agent.py::TestRunConversation::test_partial_stream_recovery_preempts_prior_turn_fallback tests/gateway/test_restart_drain.py -q
scripts/run_tests.sh tests/run_agent/test_turn_completion_explainer.py tests/gateway/test_restart_drain.py tests/run_agent/test_run_agent.py -- -q

32 passed for the focused pytest run, and 397 tests passed through scripts/run_tests.sh for the affected files.

Scope Note

This fixes silent partial-stream presentation and the accepted /restart drain path. A gateway process that is already unable to process Telegram updates still needs a separate external watchdog or persistence-timeout follow-up.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery provider/openai OpenAI / Codex Responses API labels Jun 7, 2026
teknium1 pushed a commit that referenced this pull request Jun 28, 2026
…d restart

Salvage of #41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via #53987. Your partial-stream-recovery fix was reworked onto current main with authorship preserved in git log (commit e860a40). The recovered fragment now stays unpreviewed so the gateway delivers it plus a visible 'send continue' footer. Thanks for the original fix!

pai-scaffolde pushed a commit to pai-scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
pai-scaffolde pushed a commit to pai-scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…d restart

Salvage of NousResearch#41498 (0-CYBERDYNE-SYSTEMS-0).

- Leave response_previewed false on partial_stream_recovery so gateway
  fallback delivery can send the recovered fragment plus explanation.
- Always append the turn-completion explainer for partial_stream_recovery,
  not only for empty or very short fragments (NousResearch#34452 gap).
- Launch the detached /restart helper before drain, idempotently, with a
  bounded wait of restart_drain_timeout + 5s.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists provider/openai OpenAI / Codex Responses API type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants