feat: support for /audio/transcription and /audio/translation

[PatilHrushikesh]

Description
support for /audio/transcription and /audio/translation

Related Issues/PRs (if applicable)

Special notes for reviewers (if applicable)

Summary by CodeRabbit

• New Features

  • Added OpenAI-compatible audio endpoints (POST /v1/audio/transcriptions, /v1/audio/translations) with multipart upload handling and model rewriting.
  • Improved JSON-RPC parsing to accept UTF-8 BOM-prefixed payloads.

• Bug Fixes

  • Anthropic translators no longer require max_tokens — missing value defaults to 0.

• Documentation

  • Added docs for audio endpoints and a new "Scaling the AI Gateway Controller" page.

• Chores

  • Bumped Go toolchain and Docker build images to 1.26.x; updated dependencies and TypeScript dev dependency.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds OpenAI audio endpoints (transcriptions & translations) with multipart parsing, translators, tracing, metrics, and tests; upgrades Go toolchain/dependencies to 1.26; improves MCP proxy JSON-RPC/SSE handling; and updates docs, examples, and CI/workflows.

Changes

Cohort / File(s)	Summary
Go Toolchain & Dependency Config Dockerfile, go.mod, tools/go.mod, cmd/aigw/docker-compose.yaml, tests/data-plane/vcr/docker-compose*.yaml, site/package.json	Bumped Go toolchain to 1.26 across Docker and go.mod files; upgraded selected dependencies (OpenAI SDK, OpenTelemetry, google libs, typescript dev dep).
Audio Endpoint Schema & Endpoint Specs internal/apischema/openai/openai.go, internal/endpointspec/endpointspec.go, internal/endpointspec/endpointspec_test.go	Added OpenAI audio request/response types; extended Spec interface with ParseMultipartBody; implemented Transcription/Translation EndpointSpecs with multipart parsing, validation, translators, and tests.
Multipart Utilities internal/translator/multipart_helper.go, internal/translator/multipart_helper_test.go	New helpers to parse multipart boundaries and rewrite multipart payloads (replace model field) with tests and error handling.
Request Processing & ExtProc Registration cmd/extproc/mainlib/main.go, internal/extproc/processor_impl.go, internal/extproc/processor_impl_test.go	Registered audio endpoints; router now dispatches multipart requests via ParseMultipartBody; translators can receive original Content-Type via ContentTypeSetter; metrics/tracing wired for audio ops.
Audio Translators internal/translator/translator.go, internal/translator/openai_transcription.go, internal/translator/openai_transcription_test.go, internal/translator/openai_translation.go, internal/translator/openai_translation_test.go	Added ContentTypeSetter interface and OpenAI→OpenAI transcription/translation translators handling multipart rewrite, request header/path setting, response parsing, span recording, and error conversion.
Tracing (OpenInference) for Audio internal/tracing/tracingapi/api.go, internal/tracing/span.go, internal/tracing/tracer.go, internal/tracing/tracing.go, internal/tracing/openinference/openai/*.go, internal/tracing/*_test.go	Extended tracing API with transcription/translation tracers and spans; implemented OpenInference recorders for audio with input/output redaction, audio attributes (duration, language), constructors, and tests.
Metrics internal/metrics/genai.go	Added GenAIOperation constants for transcription and translation.
MCP Proxy / SSE / JSON-RPC internal/mcpproxy/sse.go, internal/mcpproxy/handlers.go, internal/mcpproxy/mcpproxy.go, internal/mcpproxy/session.go, internal/mcpproxy/*_test.go	Try JSON-RPC decode first (strip BOM), fallback to SSE parsing on decode failure, adjust readers, tighten JWT extraction, ensure tools list JSON encodes to empty slice; added tests for BOM cases.
Anthropic token handling & tests internal/translator/anthropic_helper.go, internal/translator/*_test.go, tests/data-plane/testupstream_test.go	Removed hard requirement for max_tokens; default to 0 when unset and updated tests/expectations (422→400 in some flows).
Test infra, VCR & Fixtures for Audio tests/internal/testopenai/*, tests/internal/testopeninference/*, tests/internal/testopenai/cassettes/*.yaml	Added audio request helpers, cassettes, multipart matching and sanitization, FastAPI audio test endpoints, OpenInference span fixtures, and associated tests.
Examples, Autoconfig & Docs examples/basic/aws-*.yaml, examples/access-log/*.yaml, examples/aigw/ollama.yaml, internal/autoconfig/testdata/*, site/docs/*, site/src/data/adopters/adopters.json	Moved EnvoyProxy namespace/linking to parametersRef; switched access-log dynamic metadata key to response_model; added docs for audio endpoints and scaling; added adopter entry.
Misc CI / Workflow .github/workflows/*	Pinned/updated workflow action revisions and docker/login-action to newer versions.
Other Tests & Internal Changes internal/json/json.go, internal/extensionserver/*, internal/mcpproxy/*, tests/e2e/token_ratelimit_test.go	Changed sonicjson config to case-sensitive, propagated context in extensionserver calls, added multipart-aware request matching/sanitization in tests, and fixed Prometheus response struct in e2e test.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Router as Request Processor
    participant EndpointSpec as Endpoint Spec
    participant Translator
    participant Backend

    Client->>Router: POST /v1/audio/transcriptions\n(multipart/form-data)
    Router->>Router: detect content-type == multipart/form-data
    Router->>EndpointSpec: ParseMultipartBody(body, contentType)
    EndpointSpec->>EndpointSpec: parse boundary\nextract form fields (model, file,...)
    EndpointSpec-->>Router: TranscriptionRequest + stream flag
    Router->>Translator: RequestBody(req)
    Translator->>Translator: optionally rewrite multipart (replace model)\nset headers (path, content-length)
    Translator-->>Backend: forward mutated headers + body
    Backend-->>Translator: 200 OK + body (JSON or text)
    Translator->>Translator: parse response into TranscriptionResponse\nrecord span (if enabled)
    Translator-->>Router: ResponseHeaders + ResponseBody
    Router-->>Client: 200 OK + response

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 I hopped through bytes, found music in form,
Models and multipart now dance in a swarm;
Traces and cassettes stitched neat and bright,
New audio routes hum into the night,
A tiny rabbit applauds this coded storm!

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 29.36% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: support for /audio/transcription and /audio/translation' clearly and concisely summarizes the primary change—adding support for two new OpenAI-compatible audio endpoints.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hrushikesh-audio-support

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 19

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@internal/apischema/openai/openai.go`:
- Around line 8388-8394: Replace the raw string types for response_format and
timestamp_granularities with constrained custom types and constants: define a
ResponseFormat type (e.g., type ResponseFormat string) and a
TimestampGranularity type (e.g., type TimestampGranularity string) with
enumerated constants for allowed values, then update the struct fields
ResponseFormat and TimestampGranularities to use ResponseFormat and
[]TimestampGranularity respectively (preserve json tags). Also update any other
references noted around the same area (the other occurrences at lines
~8401-8404) to use the new types and, if needed, add simple validation helpers
or UnmarshalText/MarshalText methods to enforce/convert allowed values.
- Around line 8409-8446: In the ResponseBody implementations in
internal/translator/openai_transcription.go (ResponseBody) and
internal/translator/openai_translation.go (ResponseBody) guard JSON
unmarshalling by checking the original request's ResponseFormat and Stream flags
and/or the HTTP Content-Type header: only attempt json.Unmarshal into
TranscriptionResponse or TranslationResponse when ResponseFormat indicates JSON
(or Content-Type is application/json) and Stream is false; otherwise skip
unmarshalling and record the raw response as Text (string(body)). Update both
ResponseBody handlers to use these checks so plain-text or streaming responses
are traced as raw text rather than attempting JSON unmarshal.

In `@internal/endpointspec/endpointspec.go`:
- Around line 737-745: The handler that reads the "model" form part incorrectly
marks hasModel true whenever the part exists even if its value is empty or
whitespace; modify the logic in the switch case that calls readFormField (and
the other similar "model" handling sites) to trim whitespace from the returned
val and only set req.Model and hasModel = true when the trimmed value is
non-empty—if trimmed val is empty, treat it as if the part was missing (do not
set hasModel or req.Model).
- Around line 753-784: The multipart parsing currently ignores readFormField and
parse errors, silently dropping malformed optional fields; update the parsing
branches in the handler that call readFormField (for "language", "prompt",
"response_format", "temperature", "stream", "timestamp_granularities[]") so that
if readFormField returns an error you propagate/return a 400 error instead of
ignoring it, and for "temperature" return a 400 when strconv.ParseFloat(val, 64)
fails rather than skipping setting req.Temperature, for "stream" accept only
explicit "true" or "false" (case-insensitive using strings.EqualFold) and return
a 400 for any other value, and for "timestamp_granularities[]" only append to
req.TimestampGranularities on successful read; keep using the existing
readFormField, req.* fields and strconv/strings helpers to locate and fix the
branches accordingly.
- Around line 775-795: The function parsing multipart transcription currently
parses the "stream" form field into req.Stream but the final return always uses
a literal false; update the return so the third return value is req.Stream
(instead of false) so callers receive the parsed streaming flag; locate the
return at the end of the parser (the line returning req.Model, &req, false, nil,
nil) and replace the false with req.Stream.

In `@internal/extproc/processor_impl_test.go`:
- Around line 1433-1436: The test currently assumes the model header is at index
0 which is flaky; instead iterate the slice returned by
re.RequestBody.GetResponse().GetHeaderMutation().SetHeaders (setHeaders) and
locate the entry whose Header.Key equals internalapi.ModelNameHeaderKeyDefault,
assert that such an entry exists, and then assert its Header.RawValue equals
"whisper-1"; keep the existing require.GreaterOrEqual check for len(setHeaders)
but replace the index-based require.Equal on setHeaders[0] with the described
search-and-assert logic.

In `@internal/mcpproxy/mcpproxy.go`:
- Around line 304-313: The code currently ignores errors from io.ReadAll when
reading resp.Body (inside the block handling Content-Type !=
"text/event-stream"), which lets transport/read failures be misclassified as
non-JSON-RPC and fall through to SSE parsing; update the logic in the block
around sseReader, resp.Body, tryDecodeJSONRPCMessage and rawMsg to check the
error returned by io.ReadAll, and if non-nil propagate or handle it immediately
(e.g., return the error up the call stack or set an explicit transport error),
instead of treating the read as successful and attempting SSE parsing with
partial data.

In `@internal/mcpproxy/session.go`:
- Around line 441-465: The Content-Type check and full ReadAll are unsafe for
streaming and miss params; parse the header with mime.ParseMediaType (or
equivalent) and treat types with base "application/json" or suffix "+json" as
JSON candidates, then wrap bodyReader in a buffered reader (bufio.Reader), Peek
or ReadUpTo a small prefix (e.g. 4KB) instead of io.ReadAll, run
tryDecodeJSONRPCMessage on that prefix, and if it decodes push the backendEvent
(same sseEvent/backend/startAt) and return; if it doesn't, reconstruct the
original stream by composing the buffered unread bytes with the remainder (e.g.
io.MultiReader(bufio.NewReader that returns the peeked bytes then the underlying
reader)) so the SSE parser can stream the rest without blocking.

In `@internal/translator/anthropic_helper.go`:
- Around line 627-633: The code currently falls back to max_tokens = 0 by
reading openAIReq.MaxCompletionTokens/openAIReq.MaxTokens into maxTokensVal,
which causes requests with missing token limits to be sent upstream; instead,
validate locally and reject requests that omit both MaxCompletionTokens and
MaxTokens before constructing the Anthropic payload. Update the logic around
maxTokensVal (and the analogous block at the later occurrence) to check if
cmp.Or(openAIReq.MaxCompletionTokens, openAIReq.MaxTokens) is nil and return a
clear validation error (rather than using 0), so the gateway returns a
consistent 4xx to the client without making an upstream call.

In `@internal/translator/openai_transcription_test.go`:
- Line 54: The test currently casts tr to the concrete type
*openAIToOpenAITranslatorV1Transcription to call SetContentType, which is
brittle; change the cast to an interface assertion instead (e.g. tr.(interface{
SetContentType(string) })) and call SetContentType on that interface so the test
relies only on the SetContentType behavior rather than the concrete type,
keeping references to tr and the SetContentType method.

In `@internal/translator/openai_translation.go`:
- Around line 75-93: The ResponseBody implementation for
openAIToOpenAITranslatorV1Translation silently ignores io.ReadAll errors; update
ResponseBody to handle readErr by calling the translation span error recorder
(e.g., call span.RecordError(readErr) when readErr != nil) so read failures are
observable, and likewise record json.Unmarshal errors (e.g.,
span.RecordError(jsonErr)) before falling back to recording raw data via
span.RecordResponse; use the existing tracingapi.TranslationSpan (span) and the
function openAIToOpenAITranslatorV1Translation.ResponseBody to locate where to
add these calls.

In `@site/docs/capabilities/llm-integrations/supported-endpoints.md`:
- Around line 195-253: The provider-endpoint compatibility table is missing
entries for the new "Audio Transcriptions" (POST /v1/audio/transcriptions) and
"Audio Translations" (POST /v1/audio/translations) sections; update the
compatibility table to add rows/columns for these two endpoints and mark support
for each listed provider (e.g., OpenAI and "Any OpenAI-compatible provider that
supports audio transcriptions/translations"), ensuring the table reflects
multipart/form-data upload and model selection via `model` form field /
`x-ai-eg-model` header as noted in the sections so docs remain consistent.

In `@site/package.json`:
- Line 35: The project pins "typescript" to "~6.0.2" in package.json which
triggers TS5101 errors because `@docusaurus/tsconfig`@3.9.2 still uses deprecated
"baseUrl": "."; either add "ignoreDeprecations": "6.0" to your tsconfig.json
(the file that extends "@docusaurus/tsconfig") under compilerOptions, or upgrade
Docusaurus to 3.10+ so `@docusaurus/tsconfig` is fixed — update package.json
dependency and run a fresh install and npm run typecheck to verify the CI issue
is resolved.

In `@tests/internal/testopenai/cassettes/audio-transcription-basic.yaml`:
- Around line 53-55: Redact environment-specific identifiers in the cassette
YAML by replacing the Openai-Project and X-Request-Id header values with stable
placeholders (e.g. "<REDACTED_PROJECT>", "<REDACTED_REQUEST_ID>") or remove
those keys entirely so no real project or request IDs are committed; update
every occurrence (including the other instance around lines 73-74) in the
fixture to use the placeholders and commit the sanitized cassette.

In `@tests/internal/testopenai/cassettes/audio-translation-basic.yaml`:
- Around line 47-50: The cassette contains a leaked project identifier in the
response header "Openai-Project"; update the test fixtures redaction rules to
include "Openai-Project" (add the header name to the response redaction list
used by the cassette/recorder) and then regenerate the audio-translation-basic
cassette so the recorded YAML no longer contains the proj_* value.

In `@tests/internal/testopenai/handler.go`:
- Around line 371-374: The current multipart branch in the request-matching
logic (the isMultipart(...) check that immediately returns true) makes multipart
requests indistinguishable; instead, in that branch parse and normalize the
multipart form for both the incoming http.Request and the stored interaction
headers/body (use getHeaderValue and the same parser used elsewhere), then
compare a whitelist of non-file form fields (e.g., "model", "language",
"response_format") while explicitly ignoring file parts and boundary bytes;
update the code around the isMultipart check to return true only if those
canonical form fields match, otherwise fall through to a non-match.

In `@tests/internal/testopenai/vcr.go`:
- Around line 109-112: The header deep-equal is still failing for multipart
requests because parameters (boundary) on Content-Type and rewritten
Content-Length differ; before the header comparison in
tests/internal/testopenai/vcr.go, detect multipart via isMultipart on
httpReq.Header["Content-Type"] and then normalize or remove those fields: strip
parameters from Content-Type (leave only "multipart/form-data") and remove or
set Content-Length to the placeholder value used in the cassette (the
"[multipart form data omitted]" length) so the header comparison uses normalized
values; update any header-normalization helper (or add one) used prior to the
header deep-equal so recorded cassettes and live multipart requests match.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 4283460f-5539-489c-935b-6df16f3bc0e6

📥 Commits

Reviewing files that changed from the base of the PR and between 5976c1f and 9d3c115.

⛔ Files ignored due to path filters (5)

• go.sum is excluded by !**/*.sum
• site/package-lock.json is excluded by !**/package-lock.json
• site/static/img/adopters/Simplifai.png is excluded by !**/*.png
• tests/internal/testopenai/testdata/sunrise_fact.wav is excluded by !**/*.wav
• tools/go.sum is excluded by !**/*.sum

📒 Files selected for processing (60)

• Dockerfile
• cmd/aigw/docker-compose.yaml
• cmd/extproc/mainlib/main.go
• examples/basic/aws-irsa.yaml
• examples/basic/aws-pod-identity.yaml
• go.mod
• internal/apischema/openai/openai.go
• internal/endpointspec/endpointspec.go
• internal/endpointspec/endpointspec_test.go
• internal/extproc/processor_impl.go
• internal/extproc/processor_impl_test.go
• internal/mcpproxy/handlers.go
• internal/mcpproxy/handlers_test.go
• internal/mcpproxy/mcpproxy.go
• internal/mcpproxy/session.go
• internal/mcpproxy/session_test.go
• internal/mcpproxy/sse.go
• internal/mcpproxy/sse_test.go
• internal/metrics/genai.go
• internal/tracing/openinference/openai/transcription.go
• internal/tracing/openinference/openai/transcription_test.go
• internal/tracing/openinference/openai/translation.go
• internal/tracing/openinference/openai/translation_test.go
• internal/tracing/openinference/openinference.go
• internal/tracing/span.go
• internal/tracing/tracer.go
• internal/tracing/tracer_test.go
• internal/tracing/tracing.go
• internal/tracing/tracing_test.go
• internal/tracing/tracingapi/api.go
• internal/translator/anthropic_helper.go
• internal/translator/multipart_helper.go
• internal/translator/multipart_helper_test.go
• internal/translator/openai_awsanthropic_test.go
• internal/translator/openai_gcpanthropic_test.go
• internal/translator/openai_transcription.go
• internal/translator/openai_transcription_test.go
• internal/translator/openai_translation.go
• internal/translator/openai_translation_test.go
• internal/translator/translator.go
• site/docs/capabilities/llm-integrations/supported-endpoints.md
• site/docs/getting-started/connect-providers/aws-bedrock.md
• site/package.json
• site/src/data/adopters/adopters.json
• tests/data-plane/testupstream_test.go
• tests/data-plane/vcr/docker-compose-otel.yaml
• tests/data-plane/vcr/docker-compose.yaml
• tests/internal/testopenai/audio_requests.go
• tests/internal/testopenai/audio_requests_test.go
• tests/internal/testopenai/cassettes.go
• tests/internal/testopenai/cassettes/audio-transcription-basic.yaml
• tests/internal/testopenai/cassettes/audio-translation-basic.yaml
• tests/internal/testopenai/handler.go
• tests/internal/testopenai/vcr.go
• tests/internal/testopeninference/openai_proxy.py
• tests/internal/testopeninference/requirements.txt
• tests/internal/testopeninference/spans/audio-transcription-basic.json
• tests/internal/testopeninference/spans/audio-translation-basic.json
• tests/internal/testopeninference/spans_test.go
• tools/go.mod

[coderabbitai]

🧹 Nitpick | 🔵 Trivial

Use typed enums for response_format and timestamp granularities.

Line [8391], Line [8393], and Line [8403] use raw string/[]string, which allows invalid values to flow through unchecked. Prefer constrained custom types/constants to reduce downstream validation errors.

♻️ Suggested refactor

+type AudioResponseFormat string
+
+const (
+	AudioResponseFormatJSON        AudioResponseFormat = "json"
+	AudioResponseFormatText        AudioResponseFormat = "text"
+	AudioResponseFormatSRT         AudioResponseFormat = "srt"
+	AudioResponseFormatVerboseJSON AudioResponseFormat = "verbose_json"
+	AudioResponseFormatVTT         AudioResponseFormat = "vtt"
+)
+
+type TimestampGranularity string
+
+const (
+	TimestampGranularityWord    TimestampGranularity = "word"
+	TimestampGranularitySegment TimestampGranularity = "segment"
+)
+
 type TranscriptionRequest struct {
 	Model                  string   `json:"model"`
 	Language               string   `json:"language,omitempty"`
 	Prompt                 string   `json:"prompt,omitempty"`
-	ResponseFormat         string   `json:"response_format,omitempty"`
+	ResponseFormat         AudioResponseFormat `json:"response_format,omitempty"`
 	Temperature            *float64 `json:"temperature,omitempty"`
-	TimestampGranularities []string `json:"timestamp_granularities,omitempty"`
+	TimestampGranularities []TimestampGranularity `json:"timestamp_granularities,omitempty"`
 	Stream                 bool     `json:"stream,omitempty"`
 	FileName               string   `json:"file_name,omitempty"`
 	FileSize               int64    `json:"file_size,omitempty"`
 }
 
 type TranslationRequest struct {
 	Model          string   `json:"model"`
 	Prompt         string   `json:"prompt,omitempty"`
-	ResponseFormat string   `json:"response_format,omitempty"`
+	ResponseFormat AudioResponseFormat `json:"response_format,omitempty"`
 	Temperature    *float64 `json:"temperature,omitempty"`
 	FileName       string   `json:"file_name,omitempty"`
 	FileSize       int64    `json:"file_size,omitempty"`
 }

Also applies to: 8401-8404

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/apischema/openai/openai.go` around lines 8388 - 8394, Replace the
raw string types for response_format and timestamp_granularities with
constrained custom types and constants: define a ResponseFormat type (e.g., type
ResponseFormat string) and a TimestampGranularity type (e.g., type
TimestampGranularity string) with enumerated constants for allowed values, then
update the struct fields ResponseFormat and TimestampGranularities to use
ResponseFormat and []TimestampGranularity respectively (preserve json tags).
Also update any other references noted around the same area (the other
occurrences at lines ~8401-8404) to use the new types and, if needed, add simple
validation helpers or UnmarshalText/MarshalText methods to enforce/convert
allowed values.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify whether audio transcription/translation responses are conditionally decoded
# by Content-Type/response_format before JSON unmarshalling.

echo "== Usages of response structs =="
rg -nP --type=go -C3 '\bTranscriptionResponse\b|\bTranslationResponse\b'

echo "== Audio endpoint handlers / translators and decode logic =="
rg -nP --type=go -C4 'audio/(transcriptions|translations)|response_format|ResponseFormat|Content-Type|application/json|text/plain|text/event-stream|json\.Unmarshal'

Repository: PatilHrushikesh/ai-gateway

Length of output: 50383

---

🏁 Script executed:

#!/bin/bash
# Focus on finding where audio responses are actually decoded
# Look for audio transcription/translation request/response handlers

echo "=== Audio translator response handling ==="
rg -nA20 'func.*ResponseBody.*TranscriptionRequest|func.*ResponseBody.*TranslationRequest' --type=go

echo "=== Response format / content-type checks in audio handlers ==="
rg -nB3 -A3 'ResponseFormat|response_format|text/plain|streaming' internal/translator/ --type=go

Repository: PatilHrushikesh/ai-gateway

Length of output: 50383

---

🏁 Script executed:

#!/bin/bash
# Look for all json.Unmarshal calls involving TranscriptionResponse or TranslationResponse
echo "=== All json.Unmarshal with audio response types ==="
rg -nB2 -A2 'json\.Unmarshal.*resp|json\.Unmarshal.*[Tt]ranscription|json\.Unmarshal.*[Tt]ranslation' --type=go | head -80

Repository: PatilHrushikesh/ai-gateway

Length of output: 6305

---

🏁 Script executed:

#!/bin/bash
# Check the full translator implementation for response body decoding
echo "=== openai_transcription.go full ResponseBody method ==="
sed -n '50,120p' internal/translator/openai_transcription.go

echo ""
echo "=== openai_translation.go full ResponseBody method ==="
sed -n '50,120p' internal/translator/openai_translation.go

Repository: PatilHrushikesh/ai-gateway

Length of output: 4262

---

🏁 Script executed:

#!/bin/bash
# Find where ResponseBody is called and how results are used
echo "=== Where ResponseBody result is used ==="
rg -nB5 -A10 'ResponseBody\(' --type=go internal/ | grep -A10 'newBody\|body\|result' | head -100

echo ""
echo "=== Check if audio responses pass through unchanged ==="
rg -nC5 'audio.*transcription|audio.*translation' internal/handler --type=go | head -80

Repository: PatilHrushikesh/ai-gateway

Length of output: 8661

---

🏁 Script executed:

#!/bin/bash
# Check OpenAI structs for response_format, streaming options
echo "=== Audio request structs with format/streaming options ==="
rg -nA15 'type TranscriptionRequest struct|type TranslationRequest struct' internal/apischema --type=go

echo ""
echo "=== Check if there are other response decode implementations ==="
fd -e go -path '*/internal/*' -type f | xargs rg -l 'TranscriptionResponse|TranslationResponse' | head -20

Repository: PatilHrushikesh/ai-gateway

Length of output: 3680

---

🏁 Script executed:

#!/bin/bash
# Search for ResponseFormat usage in audio/transcription/translation handling
echo "=== ResponseFormat handling in audio code ==="
rg -nC5 'ResponseFormat|response_format' internal/translator/openai_transcription.go internal/translator/openai_translation.go

echo ""
echo "=== Check if response is passed through without modification ==="
rg -nB10 'ResponseBody.*Transcription|ResponseBody.*Translation' internal/handler --type go 2>/dev/null || rg -nB10 'ResponseBody.*Transcription|ResponseBody.*Translation' internal/ --type go | head -100

Repository: PatilHrushikesh/ai-gateway

Length of output: 2442

---

🏁 Script executed:

#!/bin/bash
# Look for where audio ResponseBody result is actually used
echo "=== Audio response handling in request processors ==="
rg -nC10 'transcription.*ResponseBody|translation.*ResponseBody' --type go | head -150

echo ""
echo "=== Search for response streaming or plain-text handling ==="
rg -nC3 'text/plain|application/octet|streaming.*audio|Stream.*true' internal/translator/openai_transcription.go internal/translator/openai_translation.go

Repository: PatilHrushikesh/ai-gateway

Length of output: 185

---

🏁 Script executed:

#!/bin/bash
# Check if newBody is expected to be empty and response passes through elsewhere
echo "=== Audio endpoint handlers ==="
rg -n 'transcription|translation' internal/handler --type go 2>/dev/null | head -30

echo ""
echo "=== Check how translator results are used in proxying/response ==="
rg -nB3 -A8 'translator\.ResponseBody' internal/ --type go | grep -A8 'transcription\|translation' | head -80

Repository: PatilHrushikesh/ai-gateway

Length of output: 169

---

🏁 Script executed:

#!/bin/bash
# Find the main request handlers and proxying logic
echo "=== Find handler packages ==="
fd -type d -name handler internal/

echo ""
echo "=== Search for audio transcription/translation endpoint setup ==="
rg -rn 'audio/transcriptions|audio/translations' internal/ --type go | head -40

echo ""
echo "=== Find where body is proxied/written to client after translation ==="
rg -nC8 'newBody.*audio|body.*Transcription|body.*Translation|Transcription.*body|Translation.*body' internal/ --type go | head -120

Repository: PatilHrushikesh/ai-gateway

Length of output: 11812

---

🏁 Script executed:

#!/bin/bash
# Find proxy/routing logic that would use translator results
echo "=== Looking for translator instantiation and usage patterns ==="
rg -nC6 'NewTranscription.*Translator|NewTranslation.*Translator' internal/ --type go | head -100

echo ""
echo "=== Check if there's response modification or pass-through logic ==="
rg -n 'copyResponse|io\.Copy|writeResponse' internal/ --type go | grep -i 'audio\|transcription\|translation'

echo ""
echo "=== Find where responses are written after translation ==="
rg -nB5 -A5 'w\.Write.*newBody|w\.Write.*body' internal/ --type go | head -80

Repository: PatilHrushikesh/ai-gateway

Length of output: 10180

---

🏁 Script executed:

#!/bin/bash
# Critical finding: body is read but not returned in newBody
# Need to find where the original response body goes

echo "=== Check if response body handling is elsewhere ==="
rg -n 'io\.ReadAll.*body|body.*io\.ReadAll' internal/extproc internal/proxy --type go 2>/dev/null | head -40

echo ""
echo "=== Search for where audio responses bypass translation ==="
rg -n 'endpointspec.*Audio|TranscriptionEndpointSpec|TranslationEndpointSpec' internal/ --type go | head -30

Repository: PatilHrushikesh/ai-gateway

Length of output: 3549

---

🏁 Script executed:

#!/bin/bash
# The body is consumed in ResponseBody but not returned
# Check if there's a separate response pass-through mechanism

echo "=== Check endpointspec for audio handling ==="
sed -n '1,100p' internal/endpointspec/endpointspec.go | grep -A50 'TranscriptionEndpointSpec\|TranslationEndpointSpec'

echo ""
echo "=== Look for proxy/passthrough logic that doesn't use newBody ==="
rg -n 'upstream.*body|body.*client|client.*response' internal/extproc/processor.go --type go 2>/dev/null | head -50

Repository: PatilHrushikesh/ai-gateway

Length of output: 182

---

🏁 Script executed:

#!/bin/bash
# Verify the critical bug: body consumed in span recording but not returned
echo "=== Confirm: body is consumed in span recording section ==="
cat -n internal/translator/openai_transcription.go | sed -n '76,95p'

echo ""
echo "=== Confirm: body is also not used elsewhere in ResponseBody ==="
cat -n internal/translator/openai_translation.go | sed -n '74,93p'

Repository: PatilHrushikesh/ai-gateway

Length of output: 1913

---

🏁 Script executed:

#!/bin/bash
# Find how ResponseBody is used in actual proxying
echo "=== Search for ResponseBody call sites in processor/proxy ==="
rg -n 'translator\.ResponseBody' internal/ --type go | head -40

echo ""
echo "=== Check if body is expected to pass through when newBody is empty ==="
rg -nB5 -A10 '\.ResponseBody\(' internal/ --type go | grep -A10 'newBody' | head -80

Repository: PatilHrushikesh/ai-gateway

Length of output: 13171

---

🏁 Script executed:

#!/bin/bash
# Look at the actual processor that uses translator results
echo "=== Processor using translator ResponseBody ==="
fd -name 'processor*.go' internal/extproc/ --type f | xargs cat | head -300

Repository: PatilHrushikesh/ai-gateway

Length of output: 365

---

🏁 Script executed:

#!/bin/bash
# Find the actual proxy/processing logic that calls translator.ResponseBody
echo "=== Find where translator is instantiated and results used ==="
rg -n 'GetTranslator\|NewTranscription\|NewTranslation' internal/ --type go | head -20

echo ""
echo "=== Check if there's a default pass-through when newBody is empty/nil ==="
rg -nB3 -A8 'if.*newBody\|len(newBody)' internal/ --type go | grep -B3 -A8 'audio\|transcription\|translation' | head -60

Repository: PatilHrushikesh/ai-gateway

Length of output: 207

---

🏁 Script executed:

#!/bin/bash
# Critical: Find where audio responses are actually written to client
echo "=== Find response writing logic in extproc/processor ==="
ls -la internal/extproc/
cat internal/extproc/processor.go 2>/dev/null | head -200

Repository: PatilHrushikesh/ai-gateway

Length of output: 4627

---

🏁 Script executed:

#!/bin/bash
# Check the extproc processor implementation
echo "=== Looking for where newBody is used after ResponseBody call ==="
rg -n 'newBody' internal/extproc/processor.go --type go 2>/dev/null | head -40

Repository: PatilHrushikesh/ai-gateway

Length of output: 135

---

🏁 Script executed:

#!/bin/bash
# Check processor_impl.go for ResponseBody usage
echo "=== Search for ResponseBody calls in processor_impl.go ==="
rg -n 'ResponseBody' internal/extproc/processor_impl.go --type go

echo ""
echo "=== Get full context around ResponseBody calls ==="
rg -nB5 -A15 '\.ResponseBody\(' internal/extproc/processor_impl.go --type go | head -150

Repository: PatilHrushikesh/ai-gateway

Length of output: 2206

---

🏁 Script executed:

#!/bin/bash
# Check if there's a pass-through mechanism for empty newBody
echo "=== Check how newBody is handled after ResponseBody ==="
cat internal/extproc/processor_impl.go | grep -A30 'ResponseBody' | head -100

Repository: PatilHrushikesh/ai-gateway

Length of output: 4496

---

🏁 Script executed:

#!/bin/bash
# Critical: check what mutationsFromTranslationResult does with empty newBody
echo "=== Check mutationsFromTranslationResult implementation ==="
rg -n 'mutationsFromTranslationResult' internal/extproc/ --type go

echo ""
echo "=== Get the full implementation ==="
rg -nB3 -A20 'func mutationsFromTranslationResult' internal/extproc/ --type go

Repository: PatilHrushikesh/ai-gateway

Length of output: 2517

---

🏁 Script executed:

#!/bin/bash
# This is critical - if newBody is empty, what happens?
echo "=== Looking at what happens when newBody is nil/empty ==="
cat internal/extproc/processor_impl.go | sed -n '1,100p' | tail -50
grep -n 'mutationsFromTranslationResult\|BodyMutation' internal/extproc/processor_impl.go | head -20

Repository: PatilHrushikesh/ai-gateway

Length of output: 3987

---

🏁 Script executed:

#!/bin/bash
# Verify: when newBody is nil, the original body passes through
echo "=== Confirm the behavior: nil newBody means pass-through ==="
cat internal/extproc/util.go | sed -n '81,103p'

echo ""
echo "=== Verify audio transcription/translation returns nil newBody ==="
cat internal/translator/openai_transcription.go | sed -n '76,95p'
cat internal/translator/openai_translation.go | sed -n '74,93p'

Repository: PatilHrushikesh/ai-gateway

Length of output: 2582

---

Guard non-JSON formats before unmarshalling in audio response tracing.

The ResponseBody implementations in internal/translator/openai_transcription.go (lines 76–94) and internal/translator/openai_translation.go (lines 74–92) attempt JSON unmarshalling without checking response format or content-type before recording to the tracing span. While a fallback exists (wrapping raw data as Text), this approach produces inaccurate tracing data:

• If the response is in plain-text format (when response_format is "text"), the unmarshal will fail and the entire response will be recorded as a single Text field rather than being properly recognized as already-text content.
• If the response is streaming, the same issue applies.
• The request structs support ResponseFormat and Stream parameters, but the response handler ignores them.

Although the original response body passes through unchanged to clients (since newBody is nil), tracing observability is compromised for any non-JSON format responses.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/apischema/openai/openai.go` around lines 8409 - 8446, In the
ResponseBody implementations in internal/translator/openai_transcription.go
(ResponseBody) and internal/translator/openai_translation.go (ResponseBody)
guard JSON unmarshalling by checking the original request's ResponseFormat and
Stream flags and/or the HTTP Content-Type header: only attempt json.Unmarshal
into TranscriptionResponse or TranslationResponse when ResponseFormat indicates
JSON (or Content-Type is application/json) and Stream is false; otherwise skip
unmarshalling and record the raw response as Text (string(body)). Update both
ResponseBody handlers to use these checks so plain-text or streaming responses
are traced as raw text rather than attempting JSON unmarshal.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Treat an empty model value as missing.

hasModel flips to true as soon as the model part exists, so model= or whitespace-only values pass validation. The parser then returns an empty original model, which pushes a malformed request downstream instead of failing fast.

Suggested fix

 		case "model":
 			val, err := readFormField(part)
 			if err != nil {
 				return "", nil, false, nil, fmt.Errorf("%w: failed to read model field: %w", internalapi.ErrMalformedRequest, err)
 			}
-			req.Model = val
-			hasModel = true
+			req.Model = strings.TrimSpace(val)
+			hasModel = req.Model != ""

Also applies to: 788-790, 853-859, 888-890

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/endpointspec/endpointspec.go` around lines 737 - 745, The handler
that reads the "model" form part incorrectly marks hasModel true whenever the
part exists even if its value is empty or whitespace; modify the logic in the
switch case that calls readFormField (and the other similar "model" handling
sites) to trim whitespace from the returned val and only set req.Model and
hasModel = true when the trimmed value is non-empty—if trimmed val is empty,
treat it as if the part was missing (do not set hasModel or req.Model).

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Reject malformed optional multipart fields instead of silently dropping them.

These branches ignore readFormField failures, ignore strconv.ParseFloat failures, and coerce any non-"true" stream value to false. That turns malformed requests into subtly different valid ones, e.g. temperature=abc or a truncated prompt part no longer yields a 400.

Also applies to: 868-884

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/endpointspec/endpointspec.go` around lines 753 - 784, The multipart
parsing currently ignores readFormField and parse errors, silently dropping
malformed optional fields; update the parsing branches in the handler that call
readFormField (for "language", "prompt", "response_format", "temperature",
"stream", "timestamp_granularities[]") so that if readFormField returns an error
you propagate/return a 400 error instead of ignoring it, and for "temperature"
return a 400 when strconv.ParseFloat(val, 64) fails rather than skipping setting
req.Temperature, for "stream" accept only explicit "true" or "false"
(case-insensitive using strings.EqualFold) and return a 400 for any other value,
and for "timestamp_granularities[]" only append to req.TimestampGranularities on
successful read; keep using the existing readFormField, req.* fields and
strconv/strings helpers to locate and fix the branches accordingly.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Return req.Stream from the transcription parser.

Line 795 always returns false, even though Lines 775-779 already parse the multipart stream field into req.Stream. That means stream=true requests will still be handled as unary by the rest of the pipeline.

Suggested fix

-	return req.Model, &req, false, nil, nil
+	return req.Model, &req, req.Stream, nil, nil

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	case "stream":
	val, err := readFormField(part)
	if err == nil {
	req.Stream = strings.EqualFold(val, "true")
	}
	case "timestamp_granularities[]":
	val, err := readFormField(part)
	if err == nil {
	req.TimestampGranularities = append(req.TimestampGranularities, val)
	}
	}
	}
	if !hasModel {
	return "", nil, false, nil, fmt.Errorf("%w: missing required field 'model'", internalapi.ErrMalformedRequest)
	}
	if !hasFile {
	return "", nil, false, nil, fmt.Errorf("%w: missing required field 'file'", internalapi.ErrMalformedRequest)
	}
	return req.Model, &req, false, nil, nil
	case "stream":
	val, err := readFormField(part)
	if err == nil {
	req.Stream = strings.EqualFold(val, "true")
	}
	case "timestamp_granularities[]":
	val, err := readFormField(part)
	if err == nil {
	req.TimestampGranularities = append(req.TimestampGranularities, val)
	}
	}
	}
	if !hasModel {
	return "", nil, false, nil, fmt.Errorf("%w: missing required field 'model'", internalapi.ErrMalformedRequest)
	}
	if !hasFile {
	return "", nil, false, nil, fmt.Errorf("%w: missing required field 'file'", internalapi.ErrMalformedRequest)
	}
	return req.Model, &req, req.Stream, nil, nil

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/endpointspec/endpointspec.go` around lines 775 - 795, The function
parsing multipart transcription currently parses the "stream" form field into
req.Stream but the final return always uses a literal false; update the return
so the third return value is req.Stream (instead of false) so callers receive
the parsed streaming flag; locate the return at the end of the parser (the line
returning req.Model, &req, false, nil, nil) and replace the false with
req.Stream.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Current pinned versions:"
jq -r '.devDependencies.typescript, .dependencies["@docusaurus/core"], .devDependencies["@docusaurus/types"], .devDependencies["@docusaurus/tsconfig"]' site/package.json

echo
echo "Check availability of typescript@6.0.2:"
npm view typescript@6.0.2 version

echo
echo "Check Docusaurus peerDependencies:"
npm view `@docusaurus/core`@3.9.2 peerDependencies --json
npm view `@docusaurus/types`@3.9.1 peerDependencies --json
npm view `@docusaurus/tsconfig`@3.9.2 peerDependencies --json

Repository: PatilHrushikesh/ai-gateway

Length of output: 380

---

🏁 Script executed:

npm view `@docusaurus/tsconfig`@3.9.2 peerDependencies --json

Repository: PatilHrushikesh/ai-gateway

Length of output: 52

---

🌐 Web query:

TypeScript 6.0 Docusaurus 3.9 compatibility issues

💡 Result:

Yes, there is a known compatibility issue between TypeScript 6.0 and Docusaurus 3.9. The @docusaurus/tsconfig package includes "baseUrl": "." which is deprecated in TypeScript 6.0, causing error TS5101 when running tsc --noEmit. This affects any Docusaurus 3.9 project using TS 6.0. Docusaurus docs for 3.9.2 still recommend extending @docusaurus/tsconfig and setting "baseUrl": ".". Workaround: Add "ignoreDeprecations": "6.0" to your tsconfig.json: { "extends": "@docusaurus/tsconfig", "compilerOptions": { "ignoreDeprecations": "6.0" } } New sites created with create-docusaurus in Docusaurus 3.10+ include this by default and use TS 6.0. Full removal of baseUrl is planned for Docusaurus v4 to avoid breaking changes.

Citations:

• 1: @docusaurus/tsconfig: baseUrl is deprecated in TypeScript 6.0 (TS5101) facebook/docusaurus#11893
• 2: fix(create-docusaurus): fix support for TypeScript 6.0 + fix our CI facebook/docusaurus#11843
• 3: https://docusaurus.io/docs/next/typescript-support
• 4: facebook/docusaurus@4a0273f
• 5: https://www.docusaurus.io/blog/releases/3.10

---

Verify TypeScript 6 compatibility with Docusaurus 3.9.x before merge.

TypeScript 6.0 deprecated the "baseUrl": "." syntax that @docusaurus/tsconfig@3.9.2 still includes, causing TS5101 errors during npm run typecheck. This issue is documented in facebook/docusaurus#11893.

Workaround: Add "ignoreDeprecations": "6.0" to tsconfig.json:

{
  "extends": "@docusaurus/tsconfig",
  "compilerOptions": {
    "ignoreDeprecations": "6.0"
  }
}

Alternatively, upgrade to Docusaurus 3.10+ which includes this fix by default. Without one of these steps, docs CI will fail.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@site/package.json` at line 35, The project pins "typescript" to "~6.0.2" in
package.json which triggers TS5101 errors because `@docusaurus/tsconfig`@3.9.2
still uses deprecated "baseUrl": "."; either add "ignoreDeprecations": "6.0" to
your tsconfig.json (the file that extends "@docusaurus/tsconfig") under
compilerOptions, or upgrade Docusaurus to 3.10+ so `@docusaurus/tsconfig` is fixed
— update package.json dependency and run a fresh install and npm run typecheck
to verify the CI issue is resolved.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Redact provider/project identifiers from committed cassette fixtures.

Openai-Project and X-Request-Id values should not be committed as-is; they are environment-specific identifiers and increase metadata leakage risk.

🔐 Suggested fixture redaction

             Openai-Project:
-                - proj_iDVN2qKUWgjCn5caMdHLmSHH
+                - "[REDACTED]"
...
             X-Request-Id:
-                - req_32e2312be05c406c8f0d5ee4f2ce9353
+                - "[REDACTED]"

Also applies to: 73-74

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/internal/testopenai/cassettes/audio-transcription-basic.yaml` around
lines 53 - 55, Redact environment-specific identifiers in the cassette YAML by
replacing the Openai-Project and X-Request-Id header values with stable
placeholders (e.g. "<REDACTED_PROJECT>", "<REDACTED_REQUEST_ID>") or remove
those keys entirely so no real project or request IDs are committed; update
every occurrence (including the other instance around lines 73-74) in the
fixture to use the placeholders and commit the sanitized cassette.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Redact the OpenAI project identifier from the cassette.

Openai-Project is not needed for replay and should not be checked into fixtures. This leaks project-scoped metadata into the repo; add it to the response redaction list and regenerate the cassette.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/internal/testopenai/cassettes/audio-translation-basic.yaml` around
lines 47 - 50, The cassette contains a leaked project identifier in the response
header "Openai-Project"; update the test fixtures redaction rules to include
"Openai-Project" (add the header name to the response redaction list used by the
cassette/recorder) and then regenerate the audio-translation-basic cassette so
the recorded YAML no longer contains the proj_* value.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Multipart return true can replay the wrong interaction.

Line 371–374 skips all body/form matching, so multipart requests to the same endpoint become indistinguishable and can match stale/incorrect interactions.

A safer approach is to compare a normalized subset of form fields (e.g., model, language, response_format) while still ignoring file bytes/boundary.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/internal/testopenai/handler.go` around lines 371 - 374, The current
multipart branch in the request-matching logic (the isMultipart(...) check that
immediately returns true) makes multipart requests indistinguishable; instead,
in that branch parse and normalize the multipart form for both the incoming
http.Request and the stored interaction headers/body (use getHeaderValue and the
same parser used elsewhere), then compare a whitelist of non-file form fields
(e.g., "model", "language", "response_format") while explicitly ignoring file
parts and boundary bytes; update the code around the isMultipart check to return
true only if those canonical form fields match, otherwise fall through to a
non-match.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Multipart replay still won't match recorded cassettes.

Skipping body comparison here is not enough because the header deep-equal has already happened. With the current capture hook, recorded multipart requests keep a boundary-bearing Content-Type and a rewritten Content-Length for the "[multipart form data omitted]" placeholder, so a live multipart request will miss the existing cassette and keep recording new episodes. Normalize or ignore multipart Content-Type parameters and Content-Length before header comparison.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tests/internal/testopenai/vcr.go` around lines 109 - 112, The header
deep-equal is still failing for multipart requests because parameters (boundary)
on Content-Type and rewritten Content-Length differ; before the header
comparison in tests/internal/testopenai/vcr.go, detect multipart via isMultipart
on httpReq.Header["Content-Type"] and then normalize or remove those fields:
strip parameters from Content-Type (leave only "multipart/form-data") and remove
or set Content-Length to the placeholder value used in the cassette (the
"[multipart form data omitted]" length) so the header comparison uses normalized
values; update any header-normalization helper (or add one) used prior to the
header deep-equal so recorded cassettes and live multipart requests match.

[coderabbitai]

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

internal/extproc/processor_impl.go (1)

503-545: ⚠️ Potential issue | 🟠 Major

Don't derive response_model only from the terminal ResponseBody call.

Line 544 emits dynamic metadata only at end-of-stream, but it passes the responseModel from the current ResponseBody invocation. Lines 777-780 then omit response_model entirely when that value is empty. That means streamed translators that report the model on an earlier chunk—or backends that never expose a response-side model—will now produce an empty gen_ai.response.model even though the effective upstream model is known. Cache the last non-empty response model on upstreamProcessor and fall back to actualModel when none was observed.

🔧 Suggested fix

 type upstreamProcessor[ReqT, RespT, RespChunkT any, EndpointSpecT endpointspec.Spec[ReqT, RespT, RespChunkT]] struct {
 	parent *routerProcessor[ReqT, RespT, RespChunkT, EndpointSpecT]

 	logger             *slog.Logger
 	requestHeaders     map[string]string
 	responseHeaders    map[string]string
 	responseEncoding   string
 	compressedBuf      []byte // accumulates raw compressed bytes across streaming chunks
 	decompressedOffset int    // tracks decompressed bytes already returned
 	translator         translator.Translator[ReqT, tracingapi.Span[RespT, RespChunkT]]
 	modelNameOverride  internalapi.ModelNameOverride
+	responseModel      internalapi.ResponseModel
 	headerMutator      *headermutator.HeaderMutator
 	bodyMutator        *bodymutator.BodyMutator
 	backendName        string
 	handler            filterapi.BackendAuthHandler

 	newHeaders, newBody, tokenUsage, responseModel, err := u.translator.ResponseBody(u.responseHeaders, decodingResult.reader, body.EndOfStream, u.parent.span)
 	if err != nil {
 		return nil, fmt.Errorf("failed to transform response: %w", err)
 	}
+	if responseModel != "" {
+		u.responseModel = responseModel
+		u.metrics.SetResponseModel(responseModel)
+	}
 	headerMutation, bodyMutation := mutationsFromTranslationResult(newHeaders, newBody)
 ...
-		metadata, err := buildDynamicMetadata(u.parent.config, &u.costs, u.requestHeaders, u.backendName, responseModel)
+		metadata, err := buildDynamicMetadata(u.parent.config, &u.costs, u.requestHeaders, u.backendName, string(u.responseModel))

 	// Add the actual request model that was used (after any backend overrides were applied).
 	// At this point, the header contains the final model that was sent to the upstream.
 	actualModel := requestHeaders[internalapi.ModelNameHeaderKeyDefault]
 	metadata["model_name_override"] = &structpb.Value{Kind: &structpb.Value_StringValue{StringValue: actualModel}}

 	if backendName != "" {
 		metadata["backend_name"] = &structpb.Value{Kind: &structpb.Value_StringValue{StringValue: backendName}}
 	}

-	// responseModel is the actual model that served the request.
-	if responseModel != "" {
-		metadata["response_model"] = &structpb.Value{Kind: &structpb.Value_StringValue{StringValue: responseModel}}
-	}
+	effectiveResponseModel := actualModel
+	if responseModel != "" {
+		effectiveResponseModel = responseModel
+	}
+	metadata["response_model"] = &structpb.Value{Kind: &structpb.Value_StringValue{StringValue: effectiveResponseModel}}

Also applies to: 777-780

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/extproc/processor_impl.go` around lines 503 - 545, The code only
uses the immediate responseModel returned by translator.ResponseBody and can
miss models reported earlier in a stream; add a cached field on
upstreamProcessor (e.g., lastResponseModel string) and whenever
translator.ResponseBody returns a non-empty responseModel (in the block handling
newHeaders,newBody,tokenUsage,responseModel,err), set u.lastResponseModel =
responseModel; then change the metadata-building site (where
buildDynamicMetadata is called and in the later 777-780 branch) to derive the
model with fallback: use the current responseModel if non-empty, else
u.lastResponseModel, and if still empty fall back to u.actualModel before
passing into buildDynamicMetadata so gen_ai.response.model is populated from
earlier chunks or actualModel when appropriate.

♻️ Duplicate comments (2)

internal/mcpproxy/handlers.go (1)

772-825: ⚠️ Potential issue | 🟠 Major

Rewrite the downstream headers when the JSON path falls back to SSE.

Line 777 still reads the full body before the fallback decision, so a backend that streams SSE under a JSON content type will not flush anything until EOF. Then Lines 823-825 only swap the reader; they do not fix the headers that were already copied from the upstream JSON response, so clients can still see stale Content-Type/Content-Length values while receiving SSE payloads. That breaks streaming and can make the response unparsable on the client side.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/mcpproxy/handlers.go` around lines 772 - 825, The issue: after
detecting a fallback from JSON-RPC to SSE (in the block that calls
tryDecodeJSONRPCMessage and sets sseReader = bytes.NewReader(body)), the code
has already copied upstream headers (Content-Type/Content-Length) and drained
resp.Body, so clients see stale headers and streaming is broken; fix by
rewriting downstream headers when falling back to SSE: in the fallback branch
update w.Header() to set Content-Type to "text/event-stream" (or the SSE MIME
the backend uses), remove any Content-Length header (or set Transfer-Encoding:
chunked) and copy any other relevant upsteam headers, then call
w.WriteHeader(resp.StatusCode) before streaming from sseReader. Make changes in
the JSON path fallback where tryDecodeJSONRPCMessage returned false (the block
that sets sseReader = bytes.NewReader(body)), and ensure the SSE branch uses the
updated headers and sseReader for streaming instead of relying on the original
resp.Body.

internal/extproc/processor_impl_test.go (1)

1435-1438: ⚠️ Potential issue | 🟡 Minor

Avoid order-dependent header assertion in this test.

Line 1437 assumes the model header is always at index 0; header mutation order is not guaranteed and this can make the test flaky.

Proposed fix

 	setHeaders := re.RequestBody.GetResponse().GetHeaderMutation().SetHeaders
 	require.GreaterOrEqual(t, len(setHeaders), 1)
-	require.Equal(t, internalapi.ModelNameHeaderKeyDefault, setHeaders[0].Header.Key)
-	require.Equal(t, "whisper-1", string(setHeaders[0].Header.RawValue))
+	foundModel := false
+	for _, h := range setHeaders {
+		if h.Header.Key == internalapi.ModelNameHeaderKeyDefault {
+			require.Equal(t, "whisper-1", string(h.Header.RawValue))
+			foundModel = true
+			break
+		}
+	}
+	require.True(t, foundModel, "model header mutation should be present")

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/extproc/processor_impl_test.go` around lines 1435 - 1438, The test
currently assumes the model header is at index 0 (setHeaders[0]) which is
order-dependent; instead, iterate/search the slice returned by
re.RequestBody.GetResponse().GetHeaderMutation().SetHeaders (variable
setHeaders) for an entry whose Header.Key equals
internalapi.ModelNameHeaderKeyDefault and whose Header.RawValue equals
"whisper-1", and assert that such an entry exists (fail the test if not found)
so the assertion is order-independent.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@go.mod`:
- Line 4: Update all Docker image tags that currently use the unpinned patch tag
"golang:1.26" to the pinned patch version "golang:1.26.2" so they match go.mod;
specifically edit the Dockerfile (the FROM line that currently says golang:1.26)
and every docker-compose YAML entry that references golang:1.26 and replace it
with golang:1.26.2 to ensure the runtime image patch version is consistent with
go.mod.

In `@internal/extproc/processor_impl_test.go`:
- Around line 1441-1460: The test
Test_transcriptionProcessorUpstreamFilter_SetBackend_ContentTypeSetter should be
extended to actually drive the ContentTypeSetter path: create a
multipart/form-data content-type with a boundary and include it in headers, call
p.SetBackend(...) as before, then simulate a request flow that triggers the
setter by setting a ModelNameOverride on the backend/router and invoking
r.ProcessRequestHeaders (or the equivalent method that applies request header
transformations); assert that the request header "content-type" has been mutated
to the expected multipart value and that the original boundary string is
preserved, and keep the existing checks that p.translator and r.upstreamFilter
are non-nil to ensure the setter ran. Ensure you reference and call
transcriptionProcessorUpstreamFilter.SetBackend,
transcriptionProcessorRouterFilter.ProcessRequestHeaders (or the actual method
used to apply header transforms), and any ModelNameOverride field so the test
fails if SetContentType(...) is skipped.

In `@internal/json/json.go`:
- Around line 15-26: The wrapper currently sets sonicjson.Config{CaseSensitive:
true}.Froze(), which enforces strict case-sensitive decoding and breaks
equivalence with encoding/json used by Unmarshal, Marshal, NewEncoder,
NewDecoder; change the config to use the default (remove the explicit
CaseSensitive:true or set CaseSensitive:false) so sonic's fallback
case-insensitive matching is preserved, then keep Unmarshal = config.Unmarshal,
Marshal = config.Marshal, NewEncoder = config.NewEncoder, NewDecoder =
config.NewDecoder and ensure the inline comments remain accurate about
equivalence to encoding/json.

---

Outside diff comments:
In `@internal/extproc/processor_impl.go`:
- Around line 503-545: The code only uses the immediate responseModel returned
by translator.ResponseBody and can miss models reported earlier in a stream; add
a cached field on upstreamProcessor (e.g., lastResponseModel string) and
whenever translator.ResponseBody returns a non-empty responseModel (in the block
handling newHeaders,newBody,tokenUsage,responseModel,err), set
u.lastResponseModel = responseModel; then change the metadata-building site
(where buildDynamicMetadata is called and in the later 777-780 branch) to derive
the model with fallback: use the current responseModel if non-empty, else
u.lastResponseModel, and if still empty fall back to u.actualModel before
passing into buildDynamicMetadata so gen_ai.response.model is populated from
earlier chunks or actualModel when appropriate.

---

Duplicate comments:
In `@internal/extproc/processor_impl_test.go`:
- Around line 1435-1438: The test currently assumes the model header is at index
0 (setHeaders[0]) which is order-dependent; instead, iterate/search the slice
returned by re.RequestBody.GetResponse().GetHeaderMutation().SetHeaders
(variable setHeaders) for an entry whose Header.Key equals
internalapi.ModelNameHeaderKeyDefault and whose Header.RawValue equals
"whisper-1", and assert that such an entry exists (fail the test if not found)
so the assertion is order-independent.

In `@internal/mcpproxy/handlers.go`:
- Around line 772-825: The issue: after detecting a fallback from JSON-RPC to
SSE (in the block that calls tryDecodeJSONRPCMessage and sets sseReader =
bytes.NewReader(body)), the code has already copied upstream headers
(Content-Type/Content-Length) and drained resp.Body, so clients see stale
headers and streaming is broken; fix by rewriting downstream headers when
falling back to SSE: in the fallback branch update w.Header() to set
Content-Type to "text/event-stream" (or the SSE MIME the backend uses), remove
any Content-Length header (or set Transfer-Encoding: chunked) and copy any other
relevant upsteam headers, then call w.WriteHeader(resp.StatusCode) before
streaming from sseReader. Make changes in the JSON path fallback where
tryDecodeJSONRPCMessage returned false (the block that sets sseReader =
bytes.NewReader(body)), and ensure the SSE branch uses the updated headers and
sseReader for streaming instead of relying on the original resp.Body.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: f1f49dbc-c8a6-49a9-830b-d0a63cdd2b0e

📥 Commits

Reviewing files that changed from the base of the PR and between 9d3c115 and 8330f07.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (37)

• .github/workflows/build_and_test.yaml
• .github/workflows/codeql.yaml
• .github/workflows/docker_build_job.yaml
• .github/workflows/release.yaml
• examples/access-log/basic.yaml
• examples/aigw/ollama.yaml
• examples/mcp/openai-github.yaml
• go.mod
• internal/autoconfig/config.yaml.tmpl
• internal/autoconfig/testdata/anthropic.yaml
• internal/autoconfig/testdata/azure-openai-with-org-and-project.yaml
• internal/autoconfig/testdata/azure-openai.yaml
• internal/autoconfig/testdata/debug.yaml
• internal/autoconfig/testdata/kiwi.yaml
• internal/autoconfig/testdata/llamastack.yaml
• internal/autoconfig/testdata/openai-github.yaml
• internal/autoconfig/testdata/openai-ip.yaml
• internal/autoconfig/testdata/openai-otel-ip.yaml
• internal/autoconfig/testdata/openai-otel.yaml
• internal/autoconfig/testdata/openai-with-org-and-project.yaml
• internal/autoconfig/testdata/openai-with-org.yaml
• internal/autoconfig/testdata/openai-with-project.yaml
• internal/autoconfig/testdata/openai.yaml
• internal/autoconfig/testdata/openrouter.yaml
• internal/autoconfig/testdata/tars.yaml
• internal/extensionserver/extensionserver_test.go
• internal/extensionserver/post_translate_modify.go
• internal/extensionserver/post_translate_modify_test.go
• internal/extproc/processor_impl.go
• internal/extproc/processor_impl_test.go
• internal/json/json.go
• internal/mcpproxy/handlers.go
• internal/mcpproxy/handlers_test.go
• site/docs/capabilities/index.md
• site/docs/capabilities/observability/accesslogs.md
• site/docs/capabilities/scaling.md
• tests/e2e/token_ratelimit_test.go

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "go.mod toolchain declaration:"
rg -n '^go\s+[0-9]+\.[0-9]+(\.[0-9]+)?$' go.mod

echo
echo "CI workflow Go version references:"
if [ -d .github/workflows ]; then
  rg -n --iglob '*.yml' --iglob '*.yaml' 'setup-go|go-version|golang:' .github/workflows
else
  echo "No .github/workflows directory found."
fi

echo
echo "Docker/base-image Go version references:"
rg -n --iglob 'Dockerfile*' --iglob '*.yml' --iglob '*.yaml' 'golang:[0-9]+\.[0-9]+|GO_VERSION|go[[:space:]]+[0-9]+\.[0-9]+' .

Repository: PatilHrushikesh/ai-gateway

Length of output: 3142

---

🏁 Script executed:

cat -n Dockerfile | head -20

Repository: PatilHrushikesh/ai-gateway

Length of output: 977

---

🏁 Script executed:

rg -A 2 'GOLANG_BASE_IMAGE' Dockerfile

Repository: PatilHrushikesh/ai-gateway

Length of output: 239

---

🏁 Script executed:

rg -B 2 -A 2 'golang:1.26' . --iglob 'docker-compose*' --iglob 'Dockerfile*'

Repository: PatilHrushikesh/ai-gateway

Length of output: 2291

---

Pin Go version to 1.26.2 in Docker images.

CI workflows dynamically read 1.26.2 from go.mod via go-version-file, but Docker images reference golang:1.26 (unversioned patch tag), which may resolve to a different 1.26.x patch. Update the Dockerfile default and docker-compose files to use golang:1.26.2 to ensure consistency:

• Dockerfile line 12: golang:1.26 → golang:1.26.2
• docker-compose files: golang:1.26 → golang:1.26.2

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@go.mod` at line 4, Update all Docker image tags that currently use the
unpinned patch tag "golang:1.26" to the pinned patch version "golang:1.26.2" so
they match go.mod; specifically edit the Dockerfile (the FROM line that
currently says golang:1.26) and every docker-compose YAML entry that references
golang:1.26 and replace it with golang:1.26.2 to ensure the runtime image patch
version is consistent with go.mod.

[coderabbitai]

🧹 Nitpick | 🔵 Trivial

ContentTypeSetter test currently doesn’t verify setter effect.

This test only asserts non-nil translator/upstream filter. A regression where SetContentType(...) is skipped could still pass. Please drive an observable path (e.g., multipart request + ModelNameOverride + ProcessRequestHeaders) and assert multipart mutation succeeds with boundary-sensitive behavior.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/extproc/processor_impl_test.go` around lines 1441 - 1460, The test
Test_transcriptionProcessorUpstreamFilter_SetBackend_ContentTypeSetter should be
extended to actually drive the ContentTypeSetter path: create a
multipart/form-data content-type with a boundary and include it in headers, call
p.SetBackend(...) as before, then simulate a request flow that triggers the
setter by setting a ModelNameOverride on the backend/router and invoking
r.ProcessRequestHeaders (or the equivalent method that applies request header
transformations); assert that the request header "content-type" has been mutated
to the expected multipart value and that the original boundary string is
preserved, and keep the existing checks that p.translator and r.upstreamFilter
are non-nil to ensure the setter ran. Ensure you reference and call
transcriptionProcessorUpstreamFilter.SetBackend,
transcriptionProcessorRouterFilter.ProcessRequestHeaders (or the actual method
used to apply header transforms), and any ModelNameOverride field so the test
fails if SetContentType(...) is skipped.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🌐 Web query:

In github.com/bytedance/sonic, what exactly does Config.CaseSensitive change during Unmarshal (struct field names and json tags)? Also confirm how encoding/json matches keys by default (exact vs case-insensitive fallback).

💡 Result:

In github.com/bytedance/sonic, Config.CaseSensitive, when set to true, enables decoder.OptionCaseSensitive during Unmarshal operations. This changes key matching for struct fields from case-insensitive (default behavior, matching both struct field names and json tags with case-insensitive fallback) to strictly case-sensitive matching for both struct field names and json tags. By default (CaseSensitive=false), sonic behaves like encoding/json: it matches JSON object keys to struct fields/tags preferring exact case match but falling back to case-insensitive match if no exact match is found. When CaseSensitive=true, only exact case matches are used—no case-insensitive fallback—for both un-tagged struct field names and tagged field names. This feature was added via PR #709 (referenced in issue #690) and merged around Jan 2025, available in recent versions (e.g., v1.12.5+ commits, current main branch as of 2026). encoding/json by default matches keys preferring exact case but accepting case-insensitive fallback, as per official docs: "Unmarshal matches incoming object keys to the keys used by Marshal (either the struct field name or its tag), preferring an exact match but also accepting a case-insensitive match."

Citations:

• 1: [Feature Request] case sensitive unmarshal bytedance/sonic#690
• 2: https://github.com/bytedance/sonic/blob/main/sonic.go
• 3: https://github.com/bytedance/sonic/blob/main/api.go
• 4: https://stackoverflow.com/questions/49006073/json-unmarshal-struct-case-sensitively
• 5: https://github.com/bytedance/sonic/blob/5b4862b0/sonic.go
• 6: https://github.com/bytedance/sonic/blob/5b4862b0/api.go

---

🏁 Script executed:

#!/bin/bash
# Read-only scan: locate external JSON ingest paths that may be affected by stricter key matching.
rg -n -C2 'Claims\(|WellKnown|well-known|Unmarshal\(|NewDecoder\(' --type=go

Repository: PatilHrushikesh/ai-gateway

Length of output: 50383

---

Revert strict case-sensitive decoding in the shared JSON wrapper.

Setting CaseSensitive: true breaks compatibility with encoding/json. The sonic library with this flag removes case-insensitive fallback matching, but encoding/json (and sonic's default) accept case-insensitive matches when exact matches fail. The inline comments claiming equivalence to encoding/json become inaccurate. This will cause unmarshal failures on third-party payloads with mismatched key casing (OAuth/OIDC metadata, API responses, SSE events, etc.) across all downstream consumers of this wrapper.

Proposed fix

 var (
-	config = sonicjson.Config{
-		CaseSensitive: true,
-	}.Froze()
+	config = sonicjson.ConfigDefault.Froze()

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@internal/json/json.go` around lines 15 - 26, The wrapper currently sets
sonicjson.Config{CaseSensitive: true}.Froze(), which enforces strict
case-sensitive decoding and breaks equivalence with encoding/json used by
Unmarshal, Marshal, NewEncoder, NewDecoder; change the config to use the default
(remove the explicit CaseSensitive:true or set CaseSensitive:false) so sonic's
fallback case-insensitive matching is preserved, then keep Unmarshal =
config.Unmarshal, Marshal = config.Marshal, NewEncoder = config.NewEncoder,
NewDecoder = config.NewDecoder and ensure the inline comments remain accurate
about equivalence to encoding/json.