feat: support streaming for audio with raw audio chunk forwarding

[tovarsh]

PR说明

目前 new-api 的 /v1/audio/speech 接口不支持流式返回。
而根据 OpenAI 官方接口语义，audio/speech 支持在生成过程中以 HTTP 分块传输（chunked transfer）的方式持续返回音频字节流，该行为并非 SSE（Server-Sent Events）。

本 PR 在不破坏现有行为的前提下，为 /v1/audio/speech 增加了符合 OpenAI 语义的音频流式支持。

变更内容

1. 对齐 OpenAI 的 streaming 请求语义

在 AudioRequest 中新增字段：

Stream bool `json:"stream,omitempty"`

并调整流式判断逻辑：

func (r *AudioRequest) IsStream(c *gin.Context) bool {
	return r.Stream || r.StreamFormat == "sse"
}

说明：

• stream: true：用于对齐 OpenAI 官方 audio/speech 接口语义
• stream_format == "sse"：保留现有 new-api 的历史行为
• 两者并存，确保向后兼容

2. Relay 层实现音频流直通转发

当满足以下两个条件时：

• 响应 Content-Type 以 audio/ 开头
• 请求显式声明为流式（stream=true 或 stream_format=sse）

Relay 将：

• 跳过 DoResponse 解析逻辑
• 直接通过 io.Copy 将上游返回的音频字节流写入客户端响应
• 保持 HTTP chunked streaming 行为，不做任何封装或重编码

if strings.HasPrefix(contentType, "audio/") && request.IsStream(c) {
	// raw audio streaming passthrough
}

该实现严格遵循 audio/speech 的协议语义，未引入 SSE。

3. 完整、无损的响应 Header 转发

为避免多值 Header 丢失（如 Set-Cookie 等），响应头转发采用逐值 Add 的方式，并排除 hop-by-hop headers：

for k, vv := range httpResp.Header {
	if k == "Content-Length" || k == "Transfer-Encoding" || k == "Connection" {
		continue
	}
	for _, v := range vv {
		c.Writer.Header().Add(k, v)
	}
}

保证 Header 语义完整性。

4. 指标统计与配额处理

• 在首次 flush 时记录 FirstResponseTime，确保链路时延统计准确

• 由于 audio/speech 流式响应不返回 usage 信息：

  • 使用输入文本 rune 数作为保守的 token 估算
  • 通过现有 postConsumeQuota 路径记录消耗

• 不引入新的计费模型，避免行为不确定性

设计说明

• audio/speech 的“流式”本质是原始音频字节的分块传输
• 并非 text/event-stream，因此不应使用 SSE
• 本 PR 明确区分文本流与音频流，避免协议层混用

兼容性说明

• 非流式 /v1/audio/speech 行为完全不变
• 现有使用 stream_format == "sse" 的调用方不受影响
• 新增能力为可选增强，不引入破坏性变更

Summary by CodeRabbit

• New Features

  • Added support for HTTP chunked transfer streaming for audio responses as an alternative to existing streaming methods.
  • Introduced new streaming configuration option for audio requests.

• Documentation

  • Updated API schema to reflect new streaming capabilities.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

This PR introduces HTTP streaming support for audio requests by adding a Stream boolean field to AudioRequest in both the OpenAPI schema and DTO layer, updating streaming detection logic, and implementing streaming response handling in the audio handler with token-based quota consumption.

Changes

Cohort / File(s)	Summary
API Schema & DTO Definition docs/openapi/relay.json, dto/audio.go	Added Stream boolean field to SpeechRequest/AudioRequest schema (default: false) and updated IsStream logic to consider both new Stream flag and SSE format detection.
Streaming Handler Implementation relay/audio_handler.go	Implemented streaming response path for audio content with header forwarding, response body streaming to client, first-response tracking, upstream body cleanup with error logging, and quota consumption via token counting (utf8.RuneCountInString). Added nil-check and strict type-assertion for adaptor response with structured error handling.

Sequence Diagram

sequenceDiagram
    participant Client
    participant Handler as Audio Handler
    participant Upstream as Upstream Service
    participant Quota as Quota System

    Client->>Handler: Audio Request (Stream=true)
    Handler->>Handler: Validate request & check Stream flag
    Handler->>Upstream: Forward audio request
    Upstream-->>Handler: Response (audio/*, chunked)
    
    Handler->>Handler: Setup streaming response
    Handler->>Client: Forward headers (excluding Content-Length, etc.)
    
    loop Stream audio chunks
        Upstream-->>Handler: Chunk data
        Handler->>Client: Write chunk
        Handler->>Client: Flush (if supported)
    end
    
    Handler->>Handler: Token count input (utf8.RuneCountInString)
    Handler->>Quota: Consume quota based on tokens
    Quota-->>Handler: Quota updated
    Handler->>Upstream: Close upstream body

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• fix: openai 音频模型流模式未正确计费 #2160: Addresses audio streaming billing by extracting usage from OpenAI stream events; this PR implements the streaming path and quota consumption mechanism that complements correct billing accounting.

Suggested reviewers

• creamlike1024

Poem

🐰 Streams flow like water, chunks by chunk,
Audio now dances through the net,
With tokens counted, quotas spent,
No SSE needed—chunked transfer's best!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding streaming support for audio with raw chunk forwarding, which is the primary feature introduced across all modified files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

📜 Recent review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9301263 and 973998c.

📒 Files selected for processing (3)

• docs/openapi/relay.json
• dto/audio.go
• relay/audio_handler.go

🔇 Additional comments (8)

dto/audio.go (2)

19-19: LGTM! Clean addition of the Stream field.

The new Stream bool field is well-placed and uses omitempty to maintain backward compatibility with existing callers.

---

36-36: LGTM! Backward-compatible streaming detection.

The updated IsStream() method correctly supports both the new Stream flag and the legacy StreamFormat == "sse" pattern, maintaining full backward compatibility.

docs/openapi/relay.json (1)

3736-3741: LGTM! Well-documented OpenAPI schema addition.

The new stream field is clearly defined with:

• Appropriate type (boolean) and default (false)
• Clear description distinguishing HTTP chunked transfer from SSE

This aligns well with the DTO changes and handler implementation.

relay/audio_handler.go (5)

6-10: LGTM! Imports are well-justified.

All new imports (io, strings, time, unicode/utf8) are appropriately used in the streaming implementation below.

---

57-63: LGTM! Excellent defensive programming.

The nil check and type assertion prevent potential runtime panics and provide clear error messages. This is a solid improvement to error handling.

---

65-69: LGTM! Proper pre-streaming error check.

The status code validation is correctly positioned before the streaming path begins, ensuring errors are handled before any response headers are sent.

---

71-122: LGTM! Well-implemented streaming path with proper safeguards.

The streaming implementation demonstrates several strong practices:

• Correct content-type and streaming-mode detection (line 72)
• Proper resource cleanup with deferred body close and error logging (lines 73-77)
• Header forwarding that correctly excludes hop-by-hop headers using Add() to preserve multi-value headers like Set-Cookie (lines 79-86)
• Graceful handling of missing Flusher support (lines 89-94)
• First response time tracking for latency metrics (line 96)
• Conservative usage estimation via input tokenization when upstream doesn't provide usage info (lines 108-117)

Note on error handling (lines 98-102): Returning nil after io.Copy failure is acceptable since response headers have already been sent and partial data may have been streamed. The error is logged for debugging purposes, which is appropriate in this context.

---

124-136: LGTM! Existing non-streaming path preserved.

The non-streaming behavior remains unchanged, maintaining full backward compatibility with existing callers.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[tovarsh]

[tovarsh]

Could you let me know why this was closed? Is there an alternative implementation being worked on?

[Calcium-Ion]

Sorry, we just did a force push, which caused your PR to be closed.

[tovarsh]

lol, all right.