fix(providers): retry streaming requests dropped before first body bytes - #10779
fix(providers): retry streaming requests dropped before first body bytes#10779vincenzopalazzo wants to merge 6 commits into
Conversation
3cb2cf0 to
200ca24
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 200ca2404f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
200ca24 to
c0bdfbb
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c9d632a8d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Thank you for the work on this — it addresses a real reliability gap. We've captured the issue as #10926: the HTTP request can successfully establish a stream, ending the existing request retry scope, but reading or decoding that stream can still fail before the first response item reaches the agent. There are two open approaches to the same problem here and in #10534. Before choosing an implementation, we should agree in #10926 whether the retry boundary belongs around the first raw body chunk or the first parsed provider-stream item, along with replay-safety constraints and a verification plan. I've moved the issue to Accepted / design for that discussion. Per CONTRIBUTING.md, implementation should follow once the issue reaches Ready and the PR should link it. Please bring the design trade-offs and regression-test approach from this PR to the issue; they are useful input to that decision. |
1c9d632 to
91e8ed6
Compare
|
Verification after rebase
|
The with_retry scope in the OpenAI and OpenAI-compatible providers ended the moment response headers arrived, so an SSE connection cut before delivering any body (reqwest's "error decoding response body") surfaced as a mid-stream item error where retry is structurally unreachable, aborting the whole turn. Await the first body chunk inside the with_retry closure and prepend it to the reconstructed body stream, so pre-first-token failures — the common case with long-lived SSE connections — are transparently retried with the existing backoff. should_retry(NetworkError) is no longer dead code for this failure mode. Adds a regression test with a raw TCP server that drops the first connection after 200 headers and serves a full SSE response on retry.
91e8ed6 to
146a780
Compare
Apply the same first-body-chunk-inside-retry pattern to Databricks, OpenRouter, GitHub Copilot, NanoGPT, Tetrate, and Bedrock mantle so a 200 OK that dies before any body bytes is still retried. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0b78310eac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
rmcp 3 renamed Meta to MetaObject; update the sanitization test so the crate builds under --all-targets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Apply first_body_chunk inside with_retry for Databricks v2 Anthropic messages and the Anthropic provider stream path, using the shared body_stream_with_prefix helper. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: def841041f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Apply first_body_chunk inside with_retry for Google, ChatGPT Codex, Gemini OAuth, and Kimi Code stream paths using body_stream_with_prefix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
3d07798 to
a08ce85
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a08ce855dd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Apply first_body_chunk inside the retry path for Ollama streaming and GCP Vertex AI send_request_with_retry, replaying via body_stream_with_prefix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
closing since #11107 fixes thsis |
3 participants
Issue: #10887 (#10926, which @DOsinga opened for this PR, was closed as a duplicate of it and #10887 now carries the agreed plan).
Rebased onto
main.cargo test -p goose-providersis green (132 tests), including the newstream_retries_when_connection_drops_before_first_body_bytes. The rebase was not mechanical:mainhas since extractedstream_payload/stream_responses_payloadhelpers, so the first-chunk-inside-retry change now lives in those helpers instead of being inlined at each call site.Note
This PR does not match the plan agreed on #10887, and I would like a steer before reworking it.
The plan there is to add first-item retry in
stream_response_from_providerand leave stream-creation retries at the provider level. This PR instead stays entirely at the provider level: it pulls the first body chunk inside the provider's existingwith_retryscope, so a connection that dies before delivering any bytes is retried as a failed request rather than surfacing as a mid-stream item error where retry is unreachable.I think these are complementary rather than competing — this one closes the window between "headers arrived" and "first byte arrived", which is below the level
stream_response_from_providercan see, while the agreed plan covers a transient first item after a well-formed stream opens. But if the intent is for the agent-level handling to subsume this case, say so and I will close this in favour of an implementation of the #10887 plan.Summary
error decoding response body) bypassed goose's provider retry entirely:with_retrywrapped only the HTTP request andhandle_status(resp), so the retry scope ended the moment response headers arrived. A cut SSE connection surfaced as a per-itemErr(NetworkError)inside the body stream, long after any retry scope, and the agent's reply loop aborted the whole turn on the first stream error.should_retry(NetworkError) = truewas dead code for the one network error that actually occurs with SSE providers — it could only fire for pre-headers failures.first_body_chunk) inside thewith_retryclosure inopenai.rs(chat completions + responses API paths) andopenai_compatible.rs, then prepend it to the reconstructed body stream (stream_openai_compat_with_prefix/stream_responses_compat_with_prefix). Any failure before the first token — the common case — is now transparently retried with the existing backoff.Test plan
openai::tests::stream_retries_when_connection_drops_before_first_body_bytes: raw TCP server returns 200 headers with a promised 1024-byte body then drops the connection; retry serves a full SSE response. Verified the test fails without the fix and passes with it.cargo test -p goose-providers(87 passed)cargo clippy -p goose-providers -p goose --all-targets -- -D warnings