fix(relay): set Request.GetBody so the HTTP/2 transport can transparently retry after an upstream stream reset

[BlockchainHe]

📝 Description

Note

Maintainer follow-up (2026-08-06): The original BodyStorage.NewReader / Request.GetBody implementation and its two commits remain authored by @BlockchainHe. The branch is now rebased onto the latest main, with separate maintainer commits that:

• migrate the new Alpha Search path and bind replay metadata for all converted bodies;
• bind both ContentLength and GetBody on pass-through bodies, the Sora opaque fallback, and the Jimeng direct request path;
• clear and rebind body metadata at every channel attempt so cross-channel retries cannot retain a callback to closed storage;
• return upstream 301/302/303/307/308 responses without following them, using a shallow copy of the cached HTTP client so shared client state and HTTP/2 transport retries remain unchanged;
• add pass-through REFUSED_STREAM, graceful GOAWAY, cross-channel lifecycle, Sora, and redirect regression coverage.

Explicit Content-Length on type-erased pass-through bodies is a wire-visible improvement; those HTTP/1.1 requests previously used chunked transfer encoding. Current maintainer verification passed go test ./relay/... ./common/... -count=1, focused -race, HTTP/2 tests with -count=20, scoped build/vet, and independent relaykit test/build. A root go build ./... in this checkout is blocked only because generated web/dist assets are absent.

The detailed original analysis and proof below are preserved as submitted; the current scope and validation are recorded above.

While relaying to an HTTP/2 upstream, requests started failing with:

http2: Transport: cannot retry err [...] after Request.Body was written; define Request.GetBody to avoid this error

Go's transport can transparently retry these resets (REFUSED_STREAM / graceful GOAWAY), but only when Request.GetBody is set. The relay never sets it — the outbound body is a type-erased reader over BodyStorage — so every such reset surfaces to the caller even though a retry would have succeeded. While fixing this I also noticed DoTaskApiRequest installs a GetBody that re-reads an already-consumed reader: a retry there would silently send an empty body, which is worse than failing.

The fix hands out independent replay readers from BodyStorage (a new NewReader() on both the memory and disk implementations — zero-copy in both modes), carries the replay hook on RelayInfo next to UpstreamRequestBodySize, and injects it only when net/http didn't already derive a correct GetBody. The broken override is removed. Unit tests plus a raw-frame HTTP/2 end-to-end test cover both the transparent-retry path and the empty-body regression.

Full analysis: problem, fix design, and test coverage

Problem

1. Requests on the main relay paths can never be transparently retried by Go's HTTP/2 transport

Per net/http semantics, the HTTP/2 transport can transparently retry a request that fails with a retry-safe error — a stream-level RST_STREAM(REFUSED_STREAM) (RFC 9113 §8.7: the upstream guarantees the stream was not processed; common with proxy/CDN-fronted upstreams under load or during graceful restarts) or a graceful connection-level GOAWAY. However, once the request body has been written, retrying requires Request.GetBody to be non-nil; otherwise the request fails outright:

http2: Transport: cannot retry err [stream error: stream ID 1; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error

The outbound body built by DoApiRequest is a type-erased io.Reader (common.ReaderOnly(BodyStorage)), and net/http only auto-derives GetBody for *bytes.Reader / *bytes.Buffer / *strings.Reader. So GetBody stays nil on every path going through DoApiRequest — chat / claude / gemini / responses / embedding / image / rerank. This is the other half of the same type-erasure issue that applyUpstreamContentLength already handles for ContentLength.

2. The hand-rolled GetBody in DoTaskApiRequest silently replays an empty body (worse than failing)

req.GetBody = func() (io.ReadCloser, error) {
    return io.NopCloser(requestBody), nil
}

This returns the same, already-consumed reader: if the transport ever retries (or a redirect needs to replay the body), the second attempt sends an empty body. Moreover, most task adaptors pass *bytes.Reader bodies, for which net/http had already derived a correct snapshot-based GetBody — this code overwrote a correct implementation with a broken one.

Fix

BodyStorage retains the full payload for the lifetime of the request (in memory, or in a disk-backed temp file above the configured threshold), so replay support only needed wiring:

• BodyStorage gains a NewReader() (io.ReadCloser, error) method returning an independent reader per call, as required by the Request.GetBody contract ("a new copy of Body"): the memory implementation returns a fresh bytes.Reader over the same immutable backing array (zero-copy), and the disk implementation opens a fresh file descriptor (zero-copy, kernel page-cache backed). Independent cursors mean replays can never race an in-flight read.
• NewOutboundJSONBody now also returns getBody func() (io.ReadCloser, error) backed by NewReader(); closing a replayed body never affects the underlying storage (its lifecycle stays owned by the handler's defer closer.Close()), and once the storage is released, GetBody fails with ErrStorageClosed instead of replaying stale data.
• RelayInfo gains an UpstreamRequestGetBody field, set at the same call sites (8 handlers, 9 call sites) and with the same lifecycle as the existing UpstreamRequestBodySize.
• A new applyUpstreamGetBody helper (symmetric with applyUpstreamContentLength) wires it into DoApiRequest / DoFormRequest / DoTaskApiRequest, and only injects when req.GetBody == nil — it never overrides a correct auto-derived implementation.
• The broken override in DoTaskApiRequest is removed: *bytes.Reader / *bytes.Buffer-style bodies fall back to net/http's snapshot-based GetBody, and non-replayable type-erased bodies keep GetBody == nil, so a retry fails loudly instead of silently sending a corrupted (empty) request.

Tests

• End-to-end HTTP/2 retry test (relay/channel/api_request_getbody_test.go), against a raw-frame HTTP/2 test server that reads the full request body, answers the first attempt with RST_STREAM(REFUSED_STREAM), and serves the retried stream normally with 200:
  • TestUpstreamGetBody_HTTP2RetryAfterUpstreamStreamReset: builds the request the same way DoApiRequest does (type-erased body + both applyUpstream* helpers) and asserts the transport retries transparently and the server receives a byte-identical body on both attempts;
  • TestUpstreamGetBody_HTTP2CannotRetryWithoutGetBody: negative case reproducing the pre-fix cannot retry err ... after Request.Body was written failure.
• TestNewOutboundJSONBody_GetBodyReadersAreIndependent (+ _DiskStorage): two readers obtained from GetBody are read interleaved and each still yields the complete, correct payload — independent cursors verified for both the memory- and disk-backed storage modes.
• TestApplyUpstreamGetBody_*: injected GetBody is non-nil and two consecutive calls both return the full body; an existing GetBody is never overridden; without a replay source it stays nil.
• TestNewOutboundJSONBody_* (relay/common/outbound_body_test.go): after the main body is consumed (or partially read), getBody replays the full body repeatedly; closing a replayed body does not affect the storage lifecycle.
• TestDoTaskApiRequest_KeepsReplayableGetBody: regression test for problem 2 — after the request completes, req.GetBody still returns the full body repeatedly (it returned an empty body before the fix).
• go build ./... passes; go vet reports no new findings on the touched packages (identical to the main baseline); go test ./relay/... ./common/... all pass; the two HTTP/2 end-to-end tests run flake-free with -count=20.

🚀 Type of change

• 🐛 Bug fix - please link the corresponding issue; do not classify design trade-offs, expectation mismatches, or misunderstandings as bugs
• ✨ New feature - major features should be discussed in an issue first
• ⚡ Performance / refactor
• 📝 Documentation

🔗 Related Issue

• No corresponding issue (I searched existing issues and PRs and found no duplicate report). This is a deterministic defect rather than a design trade-off: the Go transport's own error message says "define Request.GetBody to avoid this error", and this PR ships a reproducible negative test capturing the pre-fix behavior. Happy to open a tracking issue and link it here if the maintainers prefer.

✅ 提交前检查项 / Checklist

• Human confirmation: I have personally reviewed, organized, and written this description; it is not unprocessed AI output.
• Not a duplicate: I have searched existing issues and PRs and confirmed this is not a duplicate submission.
• Bug fix note: if this PR is labeled Bug fix, I have filed or linked a corresponding issue, and I am not classifying design trade-offs, expectation mismatches, or misunderstandings as bugs.
• Understanding: I understand how these changes work and their potential impact.
• Focused scope: this PR contains no code changes unrelated to the task at hand.
• Local verification: I have run and passed the tests (or verified manually) locally, so maintainers can use the results for review.
• Security & compliance: the code contains no sensitive credentials and follows the project's coding conventions.

📸 Proof of Work

$ go build ./...
(ok, no output)

$ gofmt -l <all files touched by this PR>
(no output — clean)

$ go test ./relay/channel/ -run 'ApplyUpstreamGetBody|DoTaskApiRequest|UpstreamGetBody_HTTP2' -v -count=1
--- PASS: TestDoTaskApiRequest_KeepsReplayableGetBody (0.00s)
--- PASS: TestUpstreamGetBody_HTTP2RetryAfterUpstreamStreamReset (0.00s)
--- PASS: TestUpstreamGetBody_HTTP2CannotRetryWithoutGetBody (0.00s)
--- PASS: TestApplyUpstreamGetBody_SetsReplayableGetBody (0.00s)
--- PASS: TestApplyUpstreamGetBody_NoopWithoutReplaySource (0.00s)
--- PASS: TestApplyUpstreamGetBody_KeepsExistingGetBody (0.00s)
PASS
ok      github.com/QuantumNous/new-api/relay/channel    0.813s

$ go test ./relay/common/ -run 'OutboundJSONBody' -v -count=1
--- PASS: TestNewOutboundJSONBody_GetBodyAfterPartialRead (0.00s)
--- PASS: TestNewOutboundJSONBody_GetBodyReplaysFullBody (0.00s)
PASS
ok      github.com/QuantumNous/new-api/relay/common     1.215s

$ go test ./relay/channel/ -run 'UpstreamGetBody_HTTP2' -count=20   # flake check
ok      github.com/QuantumNous/new-api/relay/channel    0.412s

$ go test ./relay/... ./common/... -count=1
(all 14 packages with tests: ok, no FAIL)

Genuine pre-fix reproduction, captured by running the same e2e scenario without the applyUpstreamGetBody wiring:

Error: Post "http://upstream.test/v1/chat/completions": http2: Transport: cannot retry err
[stream error: stream ID 1; REFUSED_STREAM; received from peer] after Request.Body was written;
define Request.GetBody to avoid this error

Summary by CodeRabbit

• Bug Fixes

  • Improved upstream request retries by safely replaying request bodies when supported.
  • Prevented retries from sending empty or corrupted payloads after a request body has been consumed.
  • Ensured HTTP/2 stream-reset retries resend complete request bodies.
  • Preserved upstream redirect responses without following redirect targets unexpectedly.
  • Improved request-body handling across API, task, and pass-through requests.

• Tests

  • Added coverage for replay behavior, partial reads, redirects, and HTTP/2 retry scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f13afe51-cd63-40d9-89cd-1d70e0858c7d

📥 Commits

Reviewing files that changed from the base of the PR and between 0ab0202 and dabee60.

📒 Files selected for processing (20)

• common/body_storage.go
• relay/alpha_search_handler.go
• relay/channel/api_request.go
• relay/channel/api_request_getbody_test.go
• relay/channel/api_request_redirect_test.go
• relay/channel/jimeng/adaptor.go
• relay/channel/task/sora/adaptor.go
• relay/channel/task/sora/adaptor_test.go
• relay/chat_completions_via_responses.go
• relay/claude_handler.go
• relay/common/outbound_body.go
• relay/common/outbound_body_test.go
• relay/common/relay_info.go
• relay/common/relay_info_test.go
• relay/compatible_handler.go
• relay/embedding_handler.go
• relay/gemini_handler.go
• relay/image_handler.go
• relay/rerank_handler.go
• relay/responses_handler.go

🚧 Files skipped from review as they are similar to previous changes (7)

• relay/rerank_handler.go
• relay/compatible_handler.go
• relay/gemini_handler.go
• relay/chat_completions_via_responses.go
• relay/common/outbound_body.go
• common/body_storage.go
• relay/embedding_handler.go

---

Walkthrough

Changes

Upstream body replay

Layer / File(s)	Summary
Replayable body contract relay/common/outbound_body.go, common/body_storage.go, relay/common/relay_info.go, relay/common/outbound_body_test.go	NewOutboundJSONBody returns a callback that creates independent readers. RelayInfo stores the callback and body size. Memory- and disk-backed replay behavior is tested.
Handler replay metadata propagation relay/*_handler.go, relay/chat_completions_via_responses.go	Pass-through and converted request handlers store UpstreamRequestGetBody with the upstream body size.
Channel request retry integration relay/channel/api_request.go, relay/channel/jimeng/adaptor.go, relay/channel/task/sora/adaptor.go, relay/channel/api_request_getbody_test.go, relay/common/relay_info_test.go	Channel requests apply body metadata. Task requests no longer replay consumed readers. HTTP/2 tests cover reset, GOAWAY, and missing GetBody cases.
Redirect response handling relay/channel/api_request.go, relay/channel/api_request_redirect_test.go	Requests use a shallow client copy with redirects disabled. Tests verify that upstream redirect responses and request bodies remain unchanged.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• QuantumNous/new-api#6648: Uses the body replay metadata in a Volc Native task adaptor flow.

Suggested reviewers: calcium-ion

Poem

A rabbit stored each body with care,
So every retry had bytes to spare.
A reset stream could not undo
The complete payload sent anew.
Redirects stayed where the upstream route drew.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 28.13% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding Request.GetBody support for HTTP/2 retries after upstream stream resets.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@relay/common/outbound_body.go`:
- Around line 26-43: Update NewOutboundJSONBody’s getBody closure to create and
return a fresh reader backed by an independent copy of the original body data,
rather than rewinding and reusing storage. Preserve the existing error
propagation and io.ReadCloser return contract so retries and redirects each
receive an independently positioned reader.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6be0eeda-9a45-45a6-8380-8b75e651d239

📥 Commits

Reviewing files that changed from the base of the PR and between a63364d and 04488b0.

📒 Files selected for processing (13)

• relay/channel/api_request.go
• relay/channel/api_request_getbody_test.go
• relay/chat_completions_via_responses.go
• relay/claude_handler.go
• relay/common/outbound_body.go
• relay/common/outbound_body_test.go
• relay/common/relay_info.go
• relay/compatible_handler.go
• relay/embedding_handler.go
• relay/gemini_handler.go
• relay/image_handler.go
• relay/rerank_handler.go
• relay/responses_handler.go

[coderabbitai]

🧹 Nitpick comments (1)

relay/common/outbound_body_test.go (1)

20-26: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Use testify/assert for non-fatal value checks.

As per coding guidelines, Go backend tests must use testify/assert for non-fatal value checks, reserving testify/require strictly for setup and fatal assertions. The require.Equal and require.EqualValues calls throughout this test file act as non-fatal checks for the replay payloads and should be updated. Ensure "github.com/stretchr/testify/assert" is imported if not already present.

• relay/common/outbound_body_test.go#L20-L26: Change require.EqualValues(t, len(payload), size) and require.Equal(t, payload, first) to use assert.
• relay/common/outbound_body_test.go#L36-L36: Change require.Equal to assert.Equal.
• relay/common/outbound_body_test.go#L58-L68: Change require.Equal to assert.Equal on lines 58 and 67.
• relay/common/outbound_body_test.go#L84-L84: Change require.Equal to assert.Equal.
• relay/common/outbound_body_test.go#L96-L111: Change require.Equal to assert.Equal on lines 96, 101, 106, and 111.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@relay/common/outbound_body_test.go` around lines 20 - 26, In
relay/common/outbound_body_test.go, replace the non-fatal require.EqualValues
and require.Equal checks throughout the outbound body tests with
assert.EqualValues and assert.Equal, including the checks in lines 20-26, 36,
58, 67, 84, 96, 101, 106, and 111; add the testify/assert import if needed,
while retaining require for setup and fatal assertions.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@relay/common/outbound_body_test.go`:
- Around line 20-26: In relay/common/outbound_body_test.go, replace the
non-fatal require.EqualValues and require.Equal checks throughout the outbound
body tests with assert.EqualValues and assert.Equal, including the checks in
lines 20-26, 36, 58, 67, 84, 96, 101, 106, and 111; add the testify/assert
import if needed, while retaining require for setup and fatal assertions.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8e0a4352-193b-41c0-b2c7-e1b8eacf3796

📥 Commits

Reviewing files that changed from the base of the PR and between 04488b0 and c5952ee.

📒 Files selected for processing (3)

• common/body_storage.go
• relay/common/outbound_body.go
• relay/common/outbound_body_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• relay/common/outbound_body.go

[BlockchainHe]

@Calcium-Ion Could you please review this fix?

Recommended for merge: it restores safe HTTP/2 transparent retries by providing independent replayable request bodies through Request.GetBody. This prevents retry-safe REFUSED_STREAM / GOAWAY failures from surfacing to callers, while avoiding the prior risk of replaying an empty body. The shared-cursor concern has been addressed with independent memory- and disk-backed readers, covered by targeted regression and end-to-end retry tests. Thank you!

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[Calcium-Ion]

感谢 @BlockchainHe 提供核心的 BodyStorage.NewReader / Request.GetBody 实现；

本次补全包括：rebase 到最新 main（含 Alpha Search 迁移）、所有 pass-through/Sora/Jimeng 路径的 replay metadata、跨渠道尝试的字段清理与重新绑定，以及禁止 relay 自动跟随上游 301/302/303/307/308（3xx 原样交回错误处理链，HTTP/2 transport retry 不受影响）。