fix(middleware): support zstd Content-Encoding in DecompressRequestMiddleware - #6348
fix(middleware): support zstd Content-Encoding in DecompressRequestMiddleware#6348xiaoyuyu6420 wants to merge 1 commit into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds Zstandard request-body decompression, compressed-body auto-detection, explicit error handling, request metadata cleanup, and comprehensive middleware tests covering encoded, invalid, uncompressed, identity, and GET request paths. ChangesRequest decompression
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant DecompressRequestMiddleware
participant Decoder
participant JSONHandler
Client->>DecompressRequestMiddleware: Send encoded or auto-detectable request body
DecompressRequestMiddleware->>Decoder: Create gzip, Brotli, or Zstandard reader
Decoder-->>DecompressRequestMiddleware: Return decompressed stream
DecompressRequestMiddleware->>JSONHandler: Provide limited body without encoding metadata
JSONHandler-->>Client: Return request result
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
middleware/gzip.go (1)
74-74: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winSet zstd decoder concurrency to 1
zstd.NewReaderdefaults to 4 decoders orGOMAXPROCS, so this path can still use extra goroutines. If request bodies are always processed synchronously, passzstd.WithDecoderConcurrency(1)here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@middleware/gzip.go` at line 74, Update the zstd.NewReader call in the request-body decompression path to pass zstd.WithDecoderConcurrency(1), ensuring synchronous processing uses only one decoder goroutine while preserving the existing reader and error-handling behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@middleware/gzip.go`:
- Line 74: Update the zstd.NewReader call in the request-body decompression path
to pass zstd.WithDecoderConcurrency(1), ensuring synchronous processing uses
only one decoder goroutine while preserving the existing reader and
error-handling behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 913f835d-7785-41c8-901a-e2008bb6dfd3
📒 Files selected for processing (3)
go.modmiddleware/gzip.gomiddleware/gzip_test.go
Follow-up to QuantumNous#6348 (self-review before reviewer picks it up). AGENTS.md:121 mandates testify for new backend tests; the initial gzip_test.go used bare t.Fatalf/t.Errorf. Migrated to require.* for setup + fatal assertions and assert.* for non-fatal value checks, matching middleware/auth_test.go and rate_limit_test.go. Also added TestDecompressRequestMiddleware_ZstdProducesValidJSON, which feeds a zstd-compressed body through the middleware and confirms the result is unmarshallable via common.UnmarshalJsonStr — i.e. it guards the regression at the layer where QuantumNous#6313 actually surfaces (downstream JSON parsing), not just at byte equality. All 6 tests pass: go test ./middleware/ -run DecompressRequestMiddleware -count=1
|
Follow-up self-review before this gets reviewed:
All 6 tests pass ( |
315dee6 to
0417e95
Compare
|
Pre-empting a question that came up while reviewing this change against the older sibling PR #4936 (whose CodeRabbit flagged it): Why After the body is replaced with the decompressed reader,
Net: a stale Calling it out here so reviewers don't have to re-derive it. |
|
非常需要这个PR，现在我Codex只能开本地压缩来用，把端点改成OpenAI就JSON格式错误，太难受了 PS：我自己Fork了你的PR打包了一份，很好用，感谢老哥，终于可以远程压缩了，不用再忍受上下文一长压缩了之后，Codex就和弱智一样忘事或者顾头不顾腚的痛苦了👍 |
0417e95 to
6fc4844
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
middleware/gzip.go (2)
107-135: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winConsider synchronous decoding for per-request zstd decoders.
By default
zstd.NewReaderdecodes streams concurrently across multiple internal goroutines. For per-request JSON bodies (typically small), spawning that concurrency machinery on every request adds avoidable goroutine/channel overhead; the library docs note thatWithDecoderConcurrency(1)decompresses synchronously without spawning goroutines.- zstdReader, err := zstd.NewReader(src) + zstdReader, err := zstd.NewReader(src, zstd.WithDecoderConcurrency(1))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@middleware/gzip.go` around lines 107 - 135, Configure the per-request decoder created in the zstd branch of the request-body decompression flow to use synchronous decoding by passing the library’s single-concurrency decoder option to zstd.NewReader. Preserve the existing error handling, wrapped read-closer behavior, and header cleanup.
68-149: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the repeated wrap/cleanup logic shared by gzip, br, and zstd branches.
The
wrapMaxBytes(&readCloser{...}) → clearContentLength() → Header.Del("Content-Encoding")sequence is duplicated near-verbatim across all three compressed-encoding branches, differing only in the underlying reader/closeFn. A small helper would remove the repetition and reduce the chance of one branch drifting (e.g., forgettingclearContentLength()).♻️ Proposed helper
+ finalizeDecoded := func(r io.Reader, closeFn func() error) { + c.Request.Body = wrapMaxBytes(&readCloser{Reader: r, closeFn: closeFn}) + clearContentLength() + c.Request.Header.Del("Content-Encoding") + } switch encoding { case "gzip": ... - c.Request.Body = wrapMaxBytes(&readCloser{ - Reader: gzipReader, - closeFn: func() error { - _ = gzipReader.Close() - return origBody.Close() - }, - }) - clearContentLength() - c.Request.Header.Del("Content-Encoding") + finalizeDecoded(gzipReader, func() error { + _ = gzipReader.Close() + return origBody.Close() + })Apply similarly to the
brandzstdbranches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@middleware/gzip.go` around lines 68 - 149, Extract the shared request-body setup into a helper near the middleware logic, accepting the decoded reader and cleanup function, and have it perform wrapMaxBytes, clearContentLength, and Content-Encoding removal. Replace the duplicated setup in the gzip, br, and zstd branches with calls to this helper while preserving each decoder’s existing reader and close behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@middleware/gzip_test.go`:
- Around line 145-182: Update
TestDecompressRequestMiddleware_InvalidZstdDoesNotLeak to assert the recorder
returns HTTP 400 for malformed zstd input, matching
TestDecompressRequestMiddleware_InvalidGzipReturns400. Revise the test comment
and function description to state that zstd decoding errors are surfaced as a
400 response, and remove the outdated conditional assertion focused only on
preventing raw-byte leakage.
In `@middleware/gzip.go`:
- Around line 48-65: Restrict the auto-detection block in
DecompressRequestMiddleware() to requests with a supporting JSON Content-Type,
while preserving explicit Content-Encoding handling. Only peek and infer gzip or
zstd when Content-Encoding is absent or identity and the request is JSON; leave
audio and multipart image uploads unchanged.
- Around line 107-135: Update the zstd branch around zstdReader initialization
to immediately read one decoded byte before assigning it to c.Request.Body,
treating any probe error as invalid zstd input through the existing 400 response
path. When the probe succeeds, preserve that byte by prepending it to the reader
before wrapping, so JSON binding receives the complete decompressed body.
---
Nitpick comments:
In `@middleware/gzip.go`:
- Around line 107-135: Configure the per-request decoder created in the zstd
branch of the request-body decompression flow to use synchronous decoding by
passing the library’s single-concurrency decoder option to zstd.NewReader.
Preserve the existing error handling, wrapped read-closer behavior, and header
cleanup.
- Around line 68-149: Extract the shared request-body setup into a helper near
the middleware logic, accepting the decoded reader and cleanup function, and
have it perform wrapMaxBytes, clearContentLength, and Content-Encoding removal.
Replace the duplicated setup in the gzip, br, and zstd branches with calls to
this helper while preserving each decoder’s existing reader and close behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 315326e6-4750-4c5c-88ad-f3af0365b6dd
📒 Files selected for processing (3)
go.modmiddleware/gzip.gomiddleware/gzip_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
- go.mod
|
|
||
| encoding := strings.ToLower(c.GetHeader("Content-Encoding")) | ||
| encoding = strings.TrimSpace(encoding) | ||
|
|
||
| switch c.GetHeader("Content-Encoding") { | ||
| // When Content-Encoding is absent or "identity", peek at the first few | ||
| // bytes to auto-detect gzip/zstd compression. JSON's first byte is | ||
| // never 0x1f (gzip) or 0x28 (zstd), so false-positive detection is | ||
| // impossible for valid JSON payloads. | ||
| var br *bufio.Reader | ||
| if encoding == "" || encoding == "identity" { | ||
| br = bufio.NewReader(origBody) | ||
| peek, _ := br.Peek(4) | ||
| if len(peek) >= 2 && peek[0] == 0x1f && peek[1] == 0x8b { | ||
| encoding = "gzip" | ||
| } else if len(peek) >= 4 && peek[0] == 0x28 && peek[1] == 0xb5 && peek[2] == 0x2f && peek[3] == 0xfd { | ||
| encoding = "zstd" | ||
| } | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check where DecompressRequestMiddleware is registered and whether binary-upload
# routes (audio/image) are excluded from it.
rg -n 'DecompressRequestMiddleware' --type=go -C3
rg -n --type=go -C2 -i 'audio|image|transcription|whisper' routerRepository: QuantumNous/new-api
Length of output: 157
🏁 Script executed:
#!/bin/bash
set -u
printf 'Tracked Go files matching gzip/compress/router names:\n'
git ls-files '*.go' | rg -i 'middleware|gzip|compress|route|router|upload|audio|image|whisper|transcription' | head -200 || true
printf '\nSearch DecompressRequestMiddleware/ gzip middleware symbols across repo:\n'
rg -n -i 'DecompressRequestMiddleware|RequestCompression|Compression|gzip|zstd|Content-Encoding' --glob '*.go' --glob '*.ts' --glob '*.tsx' . | head -300 || trueRepository: QuantumNous/new-api
Length of output: 15381
🏁 Script executed:
#!/bin/bash
set -u
printf 'middleware/gzip.go outline:\n'
ast-grep outline middleware/gzip.go --view expanded || true
printf '\nSlices of middleware/gzip.go:\n'
sed -n '1,150p' middleware/gzip.go | cat -n
printf '\nrouter/relay-router.go relevant lines:\n'
sed -n '1,220p' router/relay-router.go | cat -n
printf '\nSearch for body-limit/read body middlewares near relay routes:\n'
rg -n -i -C3 'RequestBodyLimit|LimitRequestBody|GinLimitRequestBody|C.PostForm|ReadString|Request.Body|BodyBytes|multipart|form-data' --glob '*.go' --glob '*.ts' router middleware relay relaykit service common | head -400Repository: QuantumNous/new-api
Length of output: 39723
🏁 Script executed:
#!/bin/bash
set -u
printf 'Relay audio handler outline:\n'
ast-grep outline relay/audio_handler.go --view expanded || true
printf '\nRelay audio handler relevant lines:\n'
sed -n '1,260p' relay/audio_handler.go | cat -n
printf '\nRelay image handler outline / relevant lines:\n'
ast-grep outline relay/image_handler.go --view expanded || true
sed -n '1,220p' relay/image_handler.go | cat -n
printf '\nBehavioral probe: multipart/form-data requests reach middleware decompression before distributor?\n'
# Read-only structural check: list middleware ordering and distributor's request-body parsing behavior.
rg -n -C4 'func SetRelayRouter|Use\\(middleware\\.|Distributor\\(|ReadForm|PostForm|ParseMultipartForm|c\\.Request\\.Body' --glob '*.go' router relay middleware service | head -250Repository: QuantumNous/new-api
Length of output: 9788
Scope auto-detected gzip/zstd decompression to JSON relay input.
DecompressRequestMiddleware() is applied to the relay router before /v1/audio/transcriptions, /v1/audio/translations, and multipart image uploads. Since the middleware peeks the raw body when Content-Encoding is missing, image/audio uploads whose first bytes coincidentally match 0x1f 0x8b or 0x28 0xb5 0x2f 0xfd can be routed through a decompressor before the distributor/handler sees them. Only apply this fallback on JSON relay requests, or gate it behind both missing Content-Encoding and a supporting Content-Type.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@middleware/gzip.go` around lines 48 - 65, Restrict the auto-detection block
in DecompressRequestMiddleware() to requests with a supporting JSON
Content-Type, while preserving explicit Content-Encoding handling. Only peek and
infer gzip or zstd when Content-Encoding is absent or identity and the request
is JSON; leave audio and multipart image uploads unchanged.
| case "zstd": | ||
| src := io.Reader(origBody) | ||
| if br != nil { | ||
| src = br | ||
| } | ||
| // OpenAI Codex CLI/Desktop default to zstd request-body compression | ||
| // (client feature `enable_request_compression`). Without this branch | ||
| // the raw zstd frame (magic 0x28 0xb5 0x2f 0xfd) is handed to the | ||
| // JSON parser and fails with `invalid character '('`. | ||
| zstdReader, err := zstd.NewReader(src) | ||
| if err != nil { | ||
| _ = origBody.Close() | ||
| c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{ | ||
| "error": gin.H{ | ||
| "message": "invalid zstd body", | ||
| "type": "invalid_request_error", | ||
| }, | ||
| }) | ||
| return | ||
| } | ||
| c.Request.Body = wrapMaxBytes(&readCloser{ | ||
| Reader: zstdReader, | ||
| closeFn: func() error { | ||
| zstdReader.Close() | ||
| return origBody.Close() | ||
| }, | ||
| }) | ||
| clearContentLength() | ||
| c.Request.Header.Del("Content-Encoding") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Does klauspost/compress zstd.NewReader validate the frame header eagerly or only on first Read?
💡 Result:
Yes, klauspost/compress zstd.NewReader validates the Zstandard frame header eagerly upon initialization, specifically during the Reset process [1][2]. When you call NewReader(r), it internally calls Reset(r) [1][2]. The Reset method initiates the decoding process for the provided io.Reader [2]. If a valid io.Reader is provided, the decoder immediately begins parsing the input to establish the Zstandard frame state [1][2]. If the frame header cannot be parsed (e.g., due to an invalid magic number or insufficient input), the error is returned immediately by NewReader [1][2]. Because the decoder is designed to handle streaming data, it must validate the frame header to understand the stream's parameters (such as window size and dictionary usage) before any Read operation can successfully yield decompressed data. While NewReader is the constructor, the underlying Reset logic ensures the stream is ready for reading by performing this initial validation [1][2].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate middleware/gzip.go and module =="
git ls-files | rg '(^|/)middleware/gzip\.go$|(^|/)go\.mod$|go\.sum$|middleware/gzip.*' || true
if [ -f go.mod ]; then
rg -n 'klauspost/compress|zstd' go.mod go.sum || true
fi
echo "== inspect middleware/gzip.go outline and relevant section =="
if [ -f middleware/gzip.go ]; then
wc -l middleware/gzip.go
sed -n '1,180p' middleware/gzip.go | cat -n
fi
echo "== search zstd-related handling comments/calls in middleware/gzip.go =="
rg -n 'zstd\.NewReader|klauspost/compress|NewReader|invalid zstd|Content-Encoding|gzip' middleware/gzip.go || true
echo "== inspect module dependency metadata if available =="
if [ -f go.sum ]; then
rg -n 'klauspost/compress' go.sum | head -20 || true
fiRepository: QuantumNous/new-api
Length of output: 8502
🌐 Web query:
klauspost compress zstd v1.18.0 NewReader Reset magic number header validation Read behavior
💡 Result:
In the klauspost/compress library (including version 1.18.0), the zstd package handles magic number validation and stream resetting as follows: NewReader and Reset Behavior The NewReader function initializes a decoder, which is designed to be stored and reused to avoid memory allocations [1][2][3]. To reuse a decoder for a new Zstandard stream, the Reset(r io.Reader) method is used [4][1][3]. When Reset(r) is called: - If r is a valid io.Reader, the decoder clears the previous stream state, drains the output, and prepares to read from the new stream [4][3]. - If r is nil, the decoder enters a state where it expects no further operations other than another Reset, DecodeAll, or Close [4][3]. - The decoder can be safely reused even if the previous stream failed [1][2][3]. - Reset cannot be used after the Close method has been called, as calling Close terminates the internal goroutines and invalidates the decoder [1][2][3]. Magic Number Header Validation The zstd decoder performs magic number validation as part of its stream processing. If the input does not begin with the expected Zstandard magic number, the decoder will return an error, typically ErrMagicMismatch [5]. For users who need to validate the magic number explicitly without initiating a full decoding process, the library provides the zstd.Header type. The Header.DecodeAndStrip(in []byte) method can be used to inspect and validate the beginning of a byte slice, checking for the standard Zstandard frame magic number or skippable frame magic numbers [6]. It returns an error if the magic number is invalid [6]. Version 1.18.0 Specifics The v1.18.0 release (dated February 19, 2025) included various internal refactorings and bug fixes, such as improving small-block compression speed, adding unsafe little-endian loaders, and fixing minor issues related to error handling and buffer sizes [7][8][9][10]. It did not introduce breaking changes to the fundamental behavior of NewReader, Reset, or the core validation logic used in previous versions [7][10].
Citations:
- 1: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 2: https://github.com/klauspost/compress/tree/master/zstd
- 3: https://ref.gotd.dev/pkg/github.com/klauspost/compress/zstd.html
- 4: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 5: https://github.com/klauspost/compress/blob/v1.18.5/zstd/zstd.go
- 6: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decodeheader.go
- 7: https://github.com/klauspost/compress/releases/tag/v1.18.0
- 8: https://chromium.googlesource.com/external/github.com/klauspost/compress/+/refs/tags/v1.18.0
- 9: https://newreleases.io/project/github/klauspost/compress/release/v1.18.0
- 10: klauspost/compress@v1.17.11...v1.18.0
🌐 Web query:
github.com/klauspost/compress v1.18 zstd decoder.go NewReader Reset reads bytes magic number
💡 Result:
In the klauspost/compress library (v1.18), the zstd Decoder does not explicitly "read" the magic number at the exact moment Reset(r) is called [1][2]. Instead, the Reset(r) method reinitializes the decoder's state to process a new stream, preparing it for subsequent operations [1][2]. The actual reading of the input stream, which includes verifying the Zstandard magic number, occurs later when the decoder begins performing read operations (such as calling Read or DecodeAll) on the new reader [1][2]. Key details regarding this behavior include: 1. Reset(r) Mechanism: The Reset method accepts an io.Reader [1][2]. If a valid (non-nil) reader is provided, the decoder clears internal buffers and prepares the internal state to begin decoding from that reader [1][2]. If a nil reader is provided, the decoder releases references to the previous reader and enters a state where it expects either a future Reset or a call to DecodeAll/Close [1][2][3]. 2. Magic Number Verification: The Zstandard format specifies a magic number at the start of a valid frame. The decoder checks for this magic number during the decoding process, specifically when it starts consuming the data provided to the Reader [4]. If the provided stream does not start with the correct magic number, the decoder will return ErrMagicMismatch [4]. 3. Resource Management: Using Reset is a recommended practice to avoid unnecessary memory allocations by reusing the existing decoder's resources [5][6]. If you are encountering issues, ensure that the io.Reader passed to Reset is valid and that it correctly supplies the Zstandard-encoded stream, as the decoder only validates the data structure once it begins processing bytes [1][4].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 3: zstd: free Decoder resources when Reset is called with a nil io.Reader klauspost/compress#305
- 4: https://github.com/klauspost/compress/blob/master/zstd/zstd.go
- 5: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 6: https://github.com/klauspost/compress/tree/master/zstd
🌐 Web query:
github klauspost compress v1.18 zstd decoder.go NewReader zstd.NewReader Reset calls Read magic
💡 Result:
In the klauspost/compress library (v1.18+), the zstd.Decoder handles Zstandard stream initialization and magic number validation as follows: 1. Initialization and Reset: When you create a decoder using zstd.NewReader(r,...), it internally calls the Reset(r) method to initialize the stream [1][2]. The Reset(r) method allows you to reuse an existing decoder instance with a new io.Reader, which helps avoid unnecessary allocations [1][3][4]. 2. Magic Number Validation: The decoder expects the input stream to begin with a valid Zstandard frame header, which includes a specific "magic number" [5]. If the provided io.Reader does not start with this expected magic number, the decoder returns an ErrMagicMismatch error [5]. 3. Common Pitfalls: A common cause for receiving an "invalid input: magic number mismatch" error is passing data that is not a raw Zstandard stream, such as data that has not been properly compressed or data that has been modified [5][6]. Additionally, users must ensure the provided io.Reader is correctly positioned at the start of the compressed data [1][4]. 4. Proper Resource Management: - To reuse a decoder, call Reset(r). If you are finished with the decoder, you must call Close to stop any running internal goroutines and release resources [3][4]. - After calling Close, the decoder instance can no longer be used [3][4][7]. - If you call Reset(nil), it releases references to the previous reader, but the decoder instance remains in a state where only Reset, DecodeAll, or Close can be called [1][2][7].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 3: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 4: https://github.com/klauspost/compress/tree/master/zstd
- 5: https://github.com/klauspost/compress/blob/master/zstd/zstd.go
- 6: zstd: invalid input: magic number mismatch on simplest possible example. klauspost/compress#1039
- 7: https://ref.gotd.dev/pkg/github.com/klauspost/compress/zstd.html
🌐 Web query:
klauspost compress v1.18.0 zstd decoder.go NewReader code Read decoder Reset
💡 Result:
In the klauspost/compress library (zstd package), the Decoder is designed for high performance by minimizing allocations, which is achieved by reusing the Decoder instance across multiple streams [1][2]. Decoder Operations NewReader: Used to create a new decoder instance [3][4]. You can pass a nil reader to NewReader if you intend to initialize the decoder later using Reset [3][4]. Reset: This method allows you to reuse an existing Decoder for a new stream, which avoids the overhead of creating a new object and its associated allocations [3][1][2]. After the current stream is processed, call Reset(newReader) to begin decoding a different stream [3][5]. Handling Resources: - To release resources held by the previous reader without starting a new stream, you can call Reset(nil) [3][4][5]. - After calling Reset(nil), the decoder should only be used for subsequent Reset, DecodeAll, or Close operations [3][4][5]. - You must call Close when you are finished with the Decoder entirely to stop internal goroutines and release resources [1][2]. Once Close is called, the Decoder cannot be reused [1][4][5]. Read: The Decoder implements the io.Reader interface, so you use the standard Read(p []byte) method to pull decompressed data from the stream [5]. Example Pattern // Create a decoder for reuse decoder, _:= zstd.NewReader(nil) defer decoder.Close // Process multiple streams for _, stream:= range streams { decoder.Reset(stream) io.Copy(destination, decoder) } For more advanced requirements, ResetWithOptions(r io.Reader, opts...DOption) is also available, allowing you to reconfigure the decoder while resetting it [5][6].
Citations:
- 1: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 2: https://github.com/klauspost/compress/tree/master/zstd
- 3: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 4: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 5: https://ref.gotd.dev/pkg/github.com/klauspost/compress/zstd.html
- 6: https://pkg.go.dev/github.com/klauspost/compress/zstd
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== download klauspost compress zstd source at v1.18.0 if tool is available =="
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
cd "$tmp"
go get github.com/klauspost/compress/zstd@v1.18.0 2>/dev/null || true
echo "$GOMODCACHE"
if [ -n "$GOMODCACHE" ]; then
find "$GOMODCACHE/github.com/klauspost/compress" -type f -path '*zstd*' 2>/dev/null | head -50 || true
echo "== relevant decoder.go snippets =="
find "$GOMODCACHE/github.com/klauspost/compress" -type f -path '*decoder.go' 2>/dev/null | head -20 | while read -r f; do
echo "-- $f"
rg -n 'func NewReader|func \(d \*Decoder\) Reset|ErrMagicMismatch|magic|Read\(' "$f" -C 3 || true
done
fi
echo "== local test-like byte sequence check =="
python3 - <<'PY'
magic = bytes.fromhex('28b52ffd')
print("zstd magic bytes in code branch:", " ".join(f"0x{b:02x}" for b in magic))
print("JSON starts with '{':", ord('{'))
PYRepository: QuantumNous/new-api
Length of output: 296
🌐 Web query:
github.com klauspost compress v1.18.0 zstd decoder.go NewReader Reset calls d.decodedRead ResetBytes Read header magic
💡 Result:
In the klauspost/compress library (v1.18.0 and later), the zstd decoder's Reset method is designed to re-initialize an existing decoder instance to process a new stream, which helps minimize allocations [1][2]. When Reset(r io.Reader) is called: 1. It verifies the decoder state to ensure it has not been closed (returning ErrDecoderClosed if it has) [3][4]. 2. If a non-nil reader is provided, it resets the internal state and stream pointers to prepare for the new data source [3][4]. 3. For stream-based decoding, it either restarts the sync decoder or manages the stream decoder goroutines, waiting for existing work to complete via d.streamWg.Wait before switching to the new reader [3][4]. The specific handling of header magic and bytes is managed by the lower-level decoding logic during the read process, not directly as a monolithic step within the Reset function itself [3][4]. The Reset method primarily acts to clear the previous stream's buffers and set up the decoder for the next sequence of operations [4][1]. If Reset is called with a nil reader, the decoder releases references to the previous reader and enters a state where it expects no further operations other than another Reset, DecodeAll, or Close [3][4].
Citations:
- 1: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 2: https://chromium.googlesource.com/external/github.com/klauspost/compress/+/refs/tags/v1.10.6/zstd/decoder.go
- 3: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 4: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
🌐 Web query:
github.com/klauspost/compress v1.18.0 zstd/decoder.go NewReader Reset calls Read d.decodedRead ResetBytes
💡 Result:
In the klauspost/compress library (specifically the zstd package), the Decoder.Reset(r io.Reader) method is designed to allow the reuse of an existing Decoder instance for a new input stream, which significantly reduces allocations compared to creating a new Decoder via NewReader [1][2][3]. When Decoder.Reset(r io.Reader) is called: 1. It first ensures the current decoder state is handled by calling d.drainOutput [1][2]. 2. If a nil reader is provided, it effectively resets the decoder to a state where it expects no input, allowing the user to release references to the previous reader [1][2][4]. 3. It initializes the internal state to process the new stream provided by the io.Reader [1][2]. Regarding your query about Read, d.decodedRead, and ResetBytes: - The Read(p []byte) method is the standard io.Reader interface implementation for the Decoder, which provides decompressed data to the caller [5][4]. - The term d.decodedRead does not appear as a standard public field or method in the current decoder implementation; internal state management typically handles data buffers directly (e.g., d.current.b) [1][5]. - There is no method named ResetBytes in the public API of the zstd Decoder. The Reset(r io.Reader) method is the primary mechanism for resetting the decoder [1][2][3]. If you are performing stateless decoding of in-memory buffers, the recommended approach is to use DecodeAll, which handles buffer management automatically and does not require manual reader resetting [3][6]. In summary, to reuse a Decoder, you should call Reset(r io.Reader) with your new stream. There is no ResetBytes method, and internal read states are managed automatically by the Read method [1][2][6].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 3: https://eve-os.readthedocs.io/pkg/pillar/vendor/github.com/klauspost/compress/zstd/
- 4: zstd: Read doc typo klauspost/compress#1023
- 5: https://chromium.googlesource.com/external/github.com/klauspost/compress/+/refs/tags/v1.10.6/zstd/decoder.go
- 6: https://github.com/klauspost/compress/blob/master/zstd/README.md
🌐 Web query:
@github.com/klauspost/compress v1.18.0 decRead magic mismatch NewReader
💡 Result:
The error "invalid input: magic number mismatch" in the github.com/klauspost/compress library is a generic diagnostic message indicating that the data being read does not conform to the expected format (e.g., Zstd, Snappy/S2, or Zlib) [1][2]. This error is typically raised when the decompressor attempts to parse the beginning of a stream or block and fails to find the correct magic number [1][3]. Common causes for this error include: 1. Data Corruption: The input data has been corrupted or truncated [4]. 2. Incorrect Compression Format: Attempting to decompress data using the wrong algorithm (e.g., trying to read non-Zstd data with a Zstd reader) [1][2]. 3. Improper Buffer Usage: If using EncodeAll or DecodeAll with working buffers, providing a buffer that is not sized correctly (e.g., providing a slice with length instead of capacity, or failing to pass a zero-length slice with sufficient capacity) can lead to unexpected behavior and framing errors [1]. 4. Concurrent or Unsafe Access: Using Reset on a reader while it is still actively reading or being used by another goroutine can result in race conditions, leading to unexpected data states and magic number mismatches [2]. If you encounter this, verify that: - The data being passed to the decoder is valid and matches the format expected by that specific decoder [1][2]. - Buffers passed to methods like DecodeAll are properly managed (often requiring a zero-length slice with adequate capacity) [1]. - No concurrent, unsynchronized access to the same reader instance is occurring [2]. There were no specific breaking changes in version v1.18.0 known to trigger this error as a regression; it remains a standard error for malformed input [5][6]. If the issue persists with valid data, ensure that no external factors (like network interruption or file system issues) are corrupting the stream before it reaches the decoder [4].
Citations:
- 1: zstd: invalid input: magic number mismatch on simplest possible example. klauspost/compress#1039
- 2: Test got "invalid input: magic number mismatch" klauspost/compress#377
- 3: https://github.com/klauspost/compress/blob/v1.18.5/s2/reader.go
- 4: [Kobo Libra 2] Failed to allocate ZSTD decompression buffer koreader/koreader#9771
- 5: https://github.com/klauspost/compress/releases/tag/v1.18.0
- 6: klauspost/compress@v1.17.11...v1.18.0
🌐 Web query:
github.com/klauspost/compress @v1.18.0 zstd decoder.go "func NewReader" "Reset(r" "d.current.b"
💡 Result:
In the klauspost/compress library (v1.18.0 and later), the Reset method for the zstd decoder is used to re-initialize an existing decoder instance to process a new io.Reader stream [1][2]. This pattern allows for the reuse of resources, significantly reducing allocations compared to creating a new decoder via NewReader [1][3]. When you call Decoder.Reset(r io.Reader), the following happens regarding d.current.b: 1. Handling Nil Input: If the provided reader r is nil, the decoder effectively resets its internal state without starting a new stream [1][4]. Specifically, the code performs the following cleanup: if r == nil { d.current.err = ErrDecoderNilInput if len(d.current.b) > 0 { d.current.b = d.current.b[:0] // Resets the slice length to 0 while keeping capacity } d.current.flushed = true return nil } 2. Memory Reuse: By setting d.current.b = d.current.b[:0], the decoder clears the slice of buffered bytes but retains its underlying capacity [1][4]. This allows the decoder to reuse the existing buffer for future operations, consistent with the library's design goal of operating with minimal allocations after a "warmup" period [5][3]. The Reset method should not be called after the Close function has been invoked, as the decoder can no longer be reused once closed [1][3]. If you need to release resources without closing the decoder permanently, you can call Reset with a nil reader [1][3].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 3: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 4: https://github.com/Lelu-ai/lelu/blob/main/engine/vendor/github.com/klauspost/compress/zstd/decoder.go
- 5: https://chromium.googlesource.com/external/github.com/klauspost/compress/+/refs/tags/v1.16.0/zstd/decoder.go
🌐 Web query:
github.com/klauspost/compress v1.18.0 "decRead" "magic" "zstd"
💡 Result:
In the klauspost/compress library (including version 1.18.0), there is no public function named decRead for checking magic numbers in Zstandard data [1][2][3]. The package identifies Zstandard frames by verifying the magic number (0xFD2FB528) during decoding operations, such as within the DecodeAndStrip method located in zstd/decodeheader.go [4]. When the magic number does not match the expected value, the library returns the error ErrMagicMismatch [4][5]. For applications that need to check for the presence of the Zstandard magic number in a byte slice without initializing a full decoder, the library does not provide a dedicated public HasMagic helper function [6]. Users typically implement this check manually by verifying the first four bytes against the standard Zstd magic sequence [6]: // Example of manually checking for Zstandard magic bytes func HasMagic(src []byte) bool { return len(src) >= 4 && src == 0x28 && src[1] == 0xB5 && src[2] == 0x2F && src[3] == 0xFD }
Citations:
- 1: https://github.com/klauspost/compress/releases/tag/v1.18.0
- 2: https://pkg.go.dev/github.com/klauspost/compress@v1.18.0
- 3: https://chromium.googlesource.com/external/github.com/klauspost/compress/+/refs/tags/v1.18.0
- 4: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decodeheader.go
- 5: https://github.com/klauspost/compress/blob/v1.18.5/zstd/zstd.go
- 6: zstd: provide a "contains magic number" func klauspost/compress#253
🌐 Web query:
klauspost compress v1.18.0 zstd decoder.go "d.current.err" "Reset(r" "zstd.NewReader"
💡 Result:
In the klauspost/compress library (specifically the zstd package), the expression d.current.err within the Reset(r io.Reader) method is used to manage the decoder's state during stream transitions [1][2]. When you call zstd.NewReader(nil), the decoder is initialized with a nil input, and d.current.err is set to ErrDecoderNilInput [1][2]. The Reset(r) method is then used to provide a new io.Reader to the decoder, allowing it to reuse existing resources and avoid the overhead of reallocating buffers [1][3][2]. Key points regarding the implementation: 1. State Management: The Reset function checks if the decoder has been closed (d.current.err == ErrDecoderClosed) before proceeding [1][2]. If a nil reader is passed to Reset, the decoder updates its state by setting d.current.err to ErrDecoderNilInput and clearing internal buffers, effectively pausing or resetting the input stream without shutting down the decoder [1][2]. 2. Resource Reuse: By using NewReader(nil) followed by one or more Reset(r) calls, you can significantly reduce memory allocations [3][2]. This is the recommended approach for high-performance applications where multiple streams are decompressed sequentially [3][4]. 3. Version 1.18.0: This version maintained established patterns for decoder reuse [5][6]. It included improvements such as freeing decoder resources when Reset is called with a nil io.Reader, which helps manage memory more efficiently across stream transitions [5]. If you are encountering this code while debugging, it is part of the internal state machine that ensures the decoder correctly transitions between different input streams while maintaining high performance [1][2]. Always ensure that you call Close on the decoder when it is no longer needed to release all allocated goroutines and resources [3][4].
Citations:
- 1: https://github.com/klauspost/compress/blob/v1.18.5/zstd/decoder.go
- 2: https://github.com/klauspost/compress/blob/7633d627f0ca03e4e9e3f718b44f3328720a0167/zstd/decoder.go
- 3: https://github.com/klauspost/compress/blob/master/zstd/README.md
- 4: https://github.com/klauspost/compress/tree/master/zstd
- 5: https://pkg.go.dev/github.com/klauspost/compress/zstd
- 6: klauspost/compress@v1.17.9...v1.18.0
Probe the zstd stream before wrapping the request body.
zstd.NewReader only initializes/reset state before c.Next(); stream/header validation happens during the first Read(), so malformed zstd payloads still bypass this invalid zstd body 400 path and fail later during JSON binding as raw compressed bytes. Add an immediate probe read after NewReader and prepend the decoded byte if the probe succeeds.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@middleware/gzip.go` around lines 107 - 135, Update the zstd branch around
zstdReader initialization to immediately read one decoded byte before assigning
it to c.Request.Body, treating any probe error as invalid zstd input through the
existing 400 response path. When the probe succeeds, preserve that byte by
prepending it to the reader before wrapping, so JSON binding receives the
complete decompressed body.
6fc4844 to
d4730fe
Compare
…ddleware Closes QuantumNous#6313 Adds a zstd decompression branch to DecompressRequestMiddleware, mirroring the existing gzip/br branches. Also incorporates improvements from PR QuantumNous#4936 by @nerimoe (with permission — see PR comments): - Magic-byte peek auto-detection when Content-Encoding is missing or 'identity' (no false-positives: JSON first byte ≠ 0x1f/0x28) - Content-Length/Content-Length-header cleanup after decompression - OpenAI-style JSON error responses for invalid gzip/zstd and unsupported encodings (415) - bufio.Reader only created when peek is needed (not on explicit headers) 15 testify tests covering: zstd regression, encoding parity, peek detection, identity passthrough, 415 unsupported, 400 invalid gzip, Content-Length cleanup, GET skip, empty body, invalid zstd no-leak, end-to-end JSON unmarshal. Co-authored-by: nerimoe <i@neri.moe>
d4730fe to
4a23215
Compare
# Conflicts: # middleware/gzip.go # middleware/gzip_test.go
2 participants
Important
📝 变更描述 / Description
DecompressRequestMiddleware已挂在/v1relay 路由上，但它的switch只处理了gzip和br，没有zstd分支。带Content-Encoding: zstd的请求体被原封不动交给下游 JSON 解析器，zstd magic28 b5 2f fd的首字节0x28（ASCII(）立刻触发解析失败，客户端收到 HTTP 400：影响：近期的 OpenAI Codex CLI 与 Desktop（
wire_api = responses）默认开启请求体压缩（client featureenable_request_compression，默认 on）。由于完整请求体（system prompt + 工具 schema）即便只发一句hi也有几十 KB，必然超过压缩阈值 → 每一轮对话都必现此 400，错误信息还误导（说"invalid request"而非"请求体被压缩"）。这不是边缘场景。修复（位置：
middleware/gzip.go）：新增 zstd 解压分支
case "zstd":分支，镜像现有gzip/br分支：用zstd.NewReader解码 → 包入MaxBytesReader（与 gzip/br 一致的 post-decompression 限制）→Del("Content-Encoding")+clearContentLength()。zstd.NewReader是惰性的，无效帧在首 Read 时报错而非构造时（与 gzip 不同）。已添加注释说明。合并 PR #4936 的改进（by @nerimoe）
Content-Encoding缺失或为identity时，peek 前 4 字节自动检测 gzip（0x1f 0x8b）和 zstd（0x28 0xb5 0x2f 0xfd）压缩。JSON 首字节不可能是这两个值，无误判风险。ContentLength设为 -1、删除Content-Lengthheader，避免下游使用过期的压缩前大小。{"error":{"message":"...","type":"invalid_request_error"}}。deflate等不支持的编码不再 passthrough（会导致 JSON 解析失败），而是返回 415 + JSON 错误体。Content-Encoding: deflate的请求会透传到下游（大概率 JSON 解析失败），现在会被明确拒绝。bufio.Reader仅在需要 peek 时创建（编码明确为 gzip/br/zstd 时不创建），避免不必要的 4KB 缓冲开销。为什么这是 new-api 自身的程序缺陷而非上游/透传问题
请求在到达转发环节之前就已经在 new-api 自己的中间件里被错误处理了；同一个中间件已为
gzip/br做了相同适配，只差一个zstd分支，是明确的覆盖遗漏。🚀 变更类型 / Type of change
🔗 关联任务 / Related Issue
Content-Encoding为不支持的编码（如deflate）时，之前会 passthrough（下游大概率 JSON 解析失败），现在返回 415 Unsupported Media Type。这是更正确的错误处理，但如果有客户端依赖 passthrough 行为，需要调整。✅ 提交前检查项 / Checklist
middleware/gzip.go、middleware/gzip_test.go）+go.mod把已存在的klauspost/compress从 indirect 提升为直接依赖。http.MaxBytesReader限流，不存在解压炸弹放大；415 错误消息中的 encoding 值通过encoding/json自动转义，无注入风险。