Feature/range price - #4459
Conversation
WalkthroughThis PR introduces video generation support in the playground, TTS audio handling with disk-based caching and duration-based billing, context-tier graded pricing, enhanced debug request/response logging, and corresponding frontend UI updates for the new pricing controls. Changes
Sequence DiagramssequenceDiagram
participant Client
participant Controller
participant Relay
participant Cache
participant OpenAI as Upstream OpenAI
participant FileSystem as Disk
Client->>Controller: POST /pg/chat/completions (TTS model)
activate Controller
Controller->>Relay: PlaygroundTTSHelper
deactivate Controller
activate Relay
Relay->>Relay: Extract voice, speed, response_format from body
Relay->>Relay: Model mapping & request conversion
Relay->>OpenAI: POST /v1/audio/speech (non-streaming)
deactivate Relay
activate OpenAI
OpenAI-->>Relay: Audio bytes + Content-Type
deactivate OpenAI
activate Relay
Relay->>FileSystem: Write UUID-keyed audio file to temp dir
deactivate Relay
activate FileSystem
FileSystem-->>Relay: File persisted, path generated
deactivate FileSystem
activate Relay
Relay->>Relay: Consume quota based on duration/pricing
Relay->>Relay: Emit SSE chunk with markdown link to /pg/audio/{id}
Relay-->>Client: SSE chunk (markdown + link)
deactivate Relay
Client->>Controller: GET /pg/audio/{id} (fetch audio)
activate Controller
Controller->>Cache: GetCachedAudio(id)
activate Cache
Cache->>FileSystem: Read cached audio bytes
FileSystem-->>Cache: Audio bytes
Cache-->>Controller: Audio bytes + MIME type
deactivate Cache
Controller-->>Client: 200 + audio bytes (with Cache-Control: no-store)
deactivate Controller
sequenceDiagram
participant Client
participant Controller
participant Relay
participant OpenAI as Upstream OpenAI
participant VideoAPI as Video Status API
Client->>Controller: POST /pg/chat/completions (video model)
activate Controller
Controller->>Relay: Handler routes to video relay
deactivate Controller
activate Relay
Relay->>OpenAI: POST /v1/videos (generate video)
deactivate Relay
activate OpenAI
OpenAI-->>Relay: SSE chunk with video object {id, status}
deactivate OpenAI
activate Relay
Note over Relay: Detect video object in SSE payload
Relay->>Relay: Extract video {id}
loop Poll until completed/failed (timeout 5min)
Relay->>VideoAPI: GET /videos/{id} (check status)
activate VideoAPI
VideoAPI-->>Relay: {status: 'processing'|'completed'|'failed'}
deactivate VideoAPI
Relay->>Relay: On status change: emit SSE progress chunk
Relay-->>Client: SSE progress chunk
end
Relay->>Relay: Generate markdown link to /pg/video/{channelId}/{id}/content
Relay->>Relay: Emit final SSE chunk with video link
Relay-->>Client: SSE chunk (markdown + video link)
deactivate Relay
Client->>Controller: GET /pg/video/{channelId}/{id}/content (fetch video)
activate Controller
Controller->>Controller: Lookup channel, build upstream URL
Controller->>OpenAI: GET /videos/{id}/content (proxy)
activate OpenAI
OpenAI-->>Controller: Video binary stream + Content-Type
deactivate OpenAI
Controller-->>Client: Stream video bytes
deactivate Controller
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
⚔️ Resolve merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 19
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
relay/channel/api_request.go (1)
304-345:⚠️ Potential issue | 🟠 MajorRedact upstream debug logs before dumping bodies and headers.
This path logs the upstream body verbatim and only masks
Authorization. Requests here can still carryx-api-key,x-goog-api-key, prompt content, and other sensitive fields, so enabling debug currently leaks far more than intended.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/api_request.go` around lines 304 - 345, The debug logging currently prints raw bodyBytes and req.Header; instead create redacted copies before calling logger.LogDebug: for headers (in the block using req.Header and logger.LogDebug) build a sanitized header map that masks or removes sensitive keys like "Authorization", "X-Api-Key", "X-Goog-Api-Key", "Api-Key" and any other custom secret headers (use strings.EqualFold to match), and for the body (where bodyBytes is logged) produce a redactedBody string that strips or replaces likely sensitive fields such as "prompt", "password", "secret", "token" or any JSON keys from the request body (or remove large/unknown bodies) and log the sanitized versions; ensure these redactions are applied both in the first debug block that logs bodyBytes and the second block that logs headers, referencing bodyBytes, req.Header, processHeaderOverride, applyHeaderOverrideToRequest and logger.LogDebug so you modify those logging sites to use the sanitized copies.service/quota.go (2)
269-291:⚠️ Potential issue | 🟡 MinorDuration-based path leaves token-ratio variables unused, but still passes them to
GenerateAudioOtherInfofor the consume log.Two related observations on the new minute-based branch:
completionRatio,audioRatio, andaudioCompletionRatio(lines 269-271) are computed unconditionally, but in theaudioMinutePrice > 0 && AudioDurationSeconds > 0branch they're never applied toquotaorlogContent. They're harmless in computation, but on line 337-338 they're forwarded intoGenerateAudioOtherInfotogether withmodelRatio/modelPrice. The resulting consume-log "other" payload will then surface ratios that weren't actually used to bill the request, which makes log/quota auditing for duration-billed STT requests misleading.Consider moving the token-ratio decimals into the
elsebranch (where they're actually consumed) and either (a) emitting a separateotherpayload for duration-billed requests that recordsaudioMinutePrice/durationMinutesinstead of the token ratios, or (b) zeroing those fields inGenerateAudioOtherInfowhenaudioMinutePrice > 0 && AudioDurationSeconds > 0.Also applies to: 337-338
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@service/quota.go` around lines 269 - 291, The code computes completionRatio, audioRatio, and audioCompletionRatio unconditionally but then uses a duration-based billing path when audioMinutePrice > 0 && relayInfo.AudioDurationSeconds > 0, causing GenerateAudioOtherInfo to receive token-ratio values that were not used; fix by moving the decimal calculations (completionRatio, audioRatio, audioCompletionRatio) into the token-billing else branch where they are actually applied, and in the audio-minute billing branch either call GenerateAudioOtherInfo with zeroed token-ratio fields or create a separate "other" payload that records audioMinutePrice and durationMinutes (use relayInfo.AudioDurationSeconds and the computed durationMinutes) so logs reflect the true billing factors; update calls to GenerateAudioOtherInfo accordingly.
316-323:⚠️ Potential issue | 🟡 Minor
AudioDurationSeconds > 0withTotalTokens == 0andaudioMinutePrice == 0will silently record a zero-quota success.The new condition
usage.TotalTokens == 0 && relayInfo.AudioDurationSeconds == 0is correct for the minute-billed path. However, whenaudioMinutePrice <= 0(no per-minute price configured) but the upstream still returned a non-zerodurationand zero tokens (e.g., a whisper response inverbose_jsonwithout usage), control falls into theelsebranch at line 292,calculateAudioQuotareturns0(all token counts are zero), and then this guard does not fire (becauseAudioDurationSeconds > 0). The result:model.UpdateUserUsedQuotaAndRequestCount(..., 0)and a successful consume log withquota = 0and no error log line, which masks the misconfiguration / loss of revenue.Consider also requiring
audioMinutePrice > 0(orquota > 0) before short-circuiting the error-log path:🛠 Suggested guard
- if usage.TotalTokens == 0 && relayInfo.AudioDurationSeconds == 0 { + durationBilled := audioMinutePrice > 0 && relayInfo.AudioDurationSeconds > 0 + if usage.TotalTokens == 0 && !durationBilled { // in this case, must be some error happened quota = 0 logContent += fmt.Sprintf("（可能是上游超时）") logger.LogError(ctx, ...) } else { model.UpdateUserUsedQuotaAndRequestCount(relayInfo.UserId, quota) model.UpdateChannelUsedQuota(relayInfo.ChannelId, quota) }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@service/quota.go` around lines 316 - 323, The current guard only treats TotalTokens==0 && AudioDurationSeconds==0 as an error, but when AudioDurationSeconds>0 and audioMinutePrice<=0 (or calculated quota==0) we currently record a successful zero-quota consume; update the logic around calculateAudioQuota/quota so that if usage.TotalTokens==0 and (relayInfo.AudioDurationSeconds==0 OR audioMinutePrice<=0 OR quota==0) we enter the error/log branch (set quota=0, log with logger.LogError including relayInfo and FinalPreConsumedQuota) instead of allowing model.UpdateUserUsedQuotaAndRequestCount to record a silent zero consume; reference usage.TotalTokens, relayInfo.AudioDurationSeconds, audioMinutePrice, calculateAudioQuota, quota, and model.UpdateUserUsedQuotaAndRequestCount when making the change.
🧹 Nitpick comments (13)
relay/helper/stream_scanner.go (1)
248-261: Optional: deduplicate the forward block.Lines 252–260 repeat the same
SetFirstResponseTime/ReceivedResponseCount++/select { case dataChan <- ...; case <-ctx.Done(); case <-stopChan }pattern as lines 270–279. A small local helper (e.g.forward(payload string) bool) returning whether to keep scanning would avoid the duplication and keep both paths in sync if cancellation semantics evolve later.♻️ Sketch
forward := func(payload string) bool { info.SetFirstResponseTime() info.ReceivedResponseCount++ select { case dataChan <- payload: return true case <-ctx.Done(), <-stopChan: return false } }(Note: Go
selectdoesn't allow combining cases on one line; expand as needed.)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/helper/stream_scanner.go` around lines 248 - 261, Duplicate forwarding logic in stream_scanner.go should be extracted into a small local helper to avoid repetition and keep cancellation semantics in sync: add a closure like forward := func(payload string) bool that calls info.SetFirstResponseTime(), increments info.ReceivedResponseCount, then performs the select to send payload on dataChan or return false if ctx.Done() or stopChan fires; replace both places that currently duplicate this block (the bare-JSON branch and the other SSE-data branch) with calls to forward(payload) and stop scanning when it returns false.relay/channel/openai/audio.go (1)
124-161: Branch condition treats unspecifiedresponse_formatas verbose, but upstream default is"json"(nodurationfield).OpenAI Whisper's default
response_formatisjson, notverbose_json— onlyverbose_jsonpopulatesdurationand segment-level fields. The conditionresponseFormat == "verbose_json" || responseFormat == ""will therefore enter the "verbose" branch when the client omits the field, but the upstream payload won't actually containduration, soinfo.AudioDurationSecondssimply stays unset (the> 0guard saves correctness). It's harmless, but the comment is misleading and you'll silently not get per-minute billing whenever a client uses the default format. Consider treating onlyresponseFormat == "verbose_json"as the verbose path, or document thatverbose_jsonis required to enable duration billing (it's already noted inModelRatioSettings.jsxextraText, so that's consistent — but the runtime behavior is easy to misread here).Also, the token normalization (PromptTokens← InputTokens, CompletionTokens← OutputTokens) is duplicated between both branches; extracting a small helper would tighten this up.
♻️ Suggested refactor
+func normalizeSTTUsage(u *dto.Usage) *dto.Usage { + if u == nil { + return nil + } + if u.PromptTokens == 0 { + u.PromptTokens = u.InputTokens + } + if u.CompletionTokens == 0 { + u.CompletionTokens = u.OutputTokens + } + return u +} @@ - // 尝试从 verbose_json 响应中提取音频时长，用于按分钟计费 - if responseFormat == "verbose_json" || responseFormat == "" { + // verbose_json 响应包含 duration 字段，用于按分钟计费 + if responseFormat == "verbose_json" { var verboseResp struct { Duration float64 `json:"duration"` Usage *dto.Usage `json:"usage"` } if err := common.Unmarshal(responseBody, &verboseResp); err == nil { if verboseResp.Duration > 0 { info.AudioDurationSeconds = verboseResp.Duration } if verboseResp.Usage != nil && verboseResp.Usage.TotalTokens > 0 { - usage := verboseResp.Usage - if usage.PromptTokens == 0 { - usage.PromptTokens = usage.InputTokens - } - if usage.CompletionTokens == 0 { - usage.CompletionTokens = usage.OutputTokens - } - return nil, usage + return nil, normalizeSTTUsage(verboseResp.Usage) } } } else { var responseData struct { Usage *dto.Usage `json:"usage"` } if err := common.Unmarshal(responseBody, &responseData); err == nil && responseData.Usage != nil { if responseData.Usage.TotalTokens > 0 { - usage := responseData.Usage - if usage.PromptTokens == 0 { - usage.PromptTokens = usage.InputTokens - } - if usage.CompletionTokens == 0 { - usage.CompletionTokens = usage.OutputTokens - } - return nil, usage + return nil, normalizeSTTUsage(responseData.Usage) } } }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/openai/audio.go` around lines 124 - 161, The branch wrongly treats an empty responseFormat as verbose_json; change the condition to only treat responseFormat == "verbose_json" as the branch that looks for duration and segment info (so info.AudioDurationSeconds is only attempted when verbose_json is explicitly requested), and factor out the duplicated token-normalization logic (currently duplicated in the verboseResp and responseData handling) into a small helper (e.g., normalizeUsage(usage *dto.Usage)) that sets PromptTokens from InputTokens and CompletionTokens from OutputTokens when zero and returns the normalized *dto.Usage; update the code paths that return usage (the blocks referencing verboseResp, responseData and their Usage) to call normalizeUsage before returning.controller/channel-test.go (1)
49-54:fluxsubstring is broad and may misclassify unrelated models.
strings.Contains(lower, "flux")is a 4-character substring with no boundary; any model whose name happens to embedflux(e.g., a future text/embedding model containing that substring) will be force-routed to/v1/images/generationsand built as anImageRequest. Consider tightening with a prefix/word-boundary match consistent with howveo-andsora-are handled (HasPrefix(lower, "flux")or check for"flux-"/known vendor prefixes), or maintaining an explicit allowlist.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@controller/channel-test.go` around lines 49 - 54, The current model detection uses strings.Contains(lower, "flux") which is too broad; update the condition that builds the image-model predicate (the boolean expression using variable lower and string checks like "stable-diffusion", "veo-", "sora-") to tighten the flux match — replace strings.Contains(lower, "flux") with a bounded check such as strings.HasPrefix(lower, "flux") or strings.HasPrefix(lower, "flux-") (or consult an explicit allowlist of known flux vendor prefixes) so model names containing "flux" elsewhere are not misrouted to image generation.types/context_tier.go (1)
12-24: Doc comment is incomplete: function also returns the last tier as a fallback when nothing matches.The godoc says only "Returns nil if the slice is empty," but lines 20-22 also fall back to
&tiers[len(tiers)-1]when no tier matches (i.e., user forgot the-1catch-all andpromptTokensexceeds everyMaxTokens). That fallback is reasonable graceful-degradation behavior, but it deserves to be in the doc so callers don't assume "no match → nil." Also worth noting: the contract requires tiers to be ordered ascending byMaxTokens; consider documenting that requirement (or sorting/validating in the loader atsetting/ratio_setting/model_ratio.gowhen JSON is parsed).📝 Suggested doc clarification
-// SelectContextTier returns the tier that matches the given prompt token count. -// Returns nil if the slice is empty. +// SelectContextTier returns the tier that matches the given prompt token count. +// Tiers MUST be ordered ascending by MaxTokens; the first tier whose MaxTokens +// is -1 (catch-all) or >= promptTokens is returned. +// If no tier matches and the slice is non-empty, the last tier is returned as +// a best-effort fallback. Returns nil only when tiers is empty. func SelectContextTier(tiers []ContextTierRatio, promptTokens int) *ContextTierRatio {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@types/context_tier.go` around lines 12 - 24, Update the godoc for SelectContextTier to state that it returns the first tier whose MaxTokens is -1 or >= promptTokens, that if no tier matches it returns the last tier in the slice (not nil) as a fallback, and that it only returns nil when the input slice is empty; also document the precondition that tiers must be ordered ascending by MaxTokens. Additionally, add/mention validation of this ordering when loading/parsing tiers (e.g. in setting/ratio_setting/model_ratio.go) or sort/validate the slice before calling SelectContextTier to ensure the contract is enforced.relay/channel/openai/relay-openai.go (2)
765-788: Image SSE chunks are missing standardId/Object/Created/Modelfields.Existing chunks emitted from this file (e.g.,
pollVideoUntilComplete,handleVideoStreamData) populateId,Object: "chat.completion.chunk",Created, andModel. The two chunks here leave them empty, which can confuse strict SSE clients and is inconsistent with the rest of the codebase. Suggest filling them in for parity:contentChunk := dto.ChatCompletionsStreamResponse{ + Id: "img-" + fmt.Sprintf("%d", time.Now().UnixNano()), + Object: "chat.completion.chunk", + Created: time.Now().Unix(), Choices: []dto.ChatCompletionsStreamResponseChoice{{🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/openai/relay-openai.go` around lines 765 - 788, The SSE chunks constructed as contentChunk and finishChunk lack the standard metadata fields (Id, Object, Created, Model); update the ChatCompletionsStreamResponse objects in this block to populate Id, Object ("chat.completion.chunk"), Created (timestamp/unix), and Model (same model string used elsewhere) just like pollVideoUntilComplete and handleVideoStreamData do, then emit them via helper.StringData(c, ...) so strict SSE clients receive consistent chunk structure.
745-748: Silent drop on unmarshal error.If the upstream returns a non-JSON or unexpected error body,
common.Unmarshalfails and the function emits[DONE]with no diagnostic chunk — the playground will show an empty assistant message with no indication of what went wrong. Consider emitting an error content chunk (similar to the❌ 视频生成失败pattern inhandleVideoStreamData) beforehelper.Done(c).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/channel/openai/relay-openai.go` around lines 745 - 748, When common.Unmarshal(responseBody, &imageResp) fails or imageResp.Data is empty, don't silently call helper.Done(c); instead construct and send an error content chunk (similar to the "❌ 视频生成失败" pattern used in handleVideoStreamData) before calling helper.Done(c). Capture the unmarshalling error and/or include the raw responseBody as diagnostic details, create the appropriate helper.Send... (or helper.WriteChunk) call that emits a visible assistant error message, then call helper.Done(c) so the playground shows the failure instead of an empty assistant message.relay/compatible_handler.go (2)
26-35: RenameextractImagePromptFromMessages— it is now also used as TTS input.It is invoked at line 110 to extract the TTS input text, not an image prompt, so the name is misleading. A neutral name like
lastUserMessageTextwould describe both call sites accurately.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/compatible_handler.go` around lines 26 - 35, The function name extractImagePromptFromMessages is misleading because it's also used to pull TTS input; rename the function to lastUserMessageText and update its comment to reflect that it returns the text of the last user message for any consumer (e.g., image prompt or TTS), then update all call sites (including the place that extracts TTS input) to use lastUserMessageText instead of extractImagePromptFromMessages and ensure dto.Message.StringContent() behavior is unchanged.
91-121: Beware:UnmarshalBodyReusableis invoked, butchatReqwas already populated from the same body — duplicate decode and silent error.
_ = common.UnmarshalBodyReusable(c, &ttsParams)ignores all errors, so a malformed JSON body silently falls through to the"alloy"/"mp3"defaults and the originalchatReq.Model/chatReq.Messagesvalues keep being used. That's likely fine, but consider:
- Using
chatReqdirectly when possible (e.g. extractVoice/Speed/ResponseFormatvia a single struct that embedsGeneralOpenAIRequestplus the extra fields) so you read the body only once.- Logging at debug level when the re-decode fails, so unexpected client schemas don't go unnoticed in production.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/compatible_handler.go` around lines 91 - 121, The code currently re-decodes the request body with common.UnmarshalBodyReusable into ttsParams while chatReq was already created from the same body, and the call ignores errors; change this to a single decode or handle errors: either define a composite struct that embeds dto.GeneralOpenAIRequest and adds Voice, Speed, ResponseFormat, decode the body once into that (then build dto.AudioRequest from the embedded fields and Messages) or at minimum check the error returned by common.UnmarshalBodyReusable and call processLogger.Debug/Debugf (or the local logger) when it fails so malformed JSON is visible; update the branch that constructs dto.AudioRequest and the call to PlaygroundTTSHelper(c, info) to use the parsed composite struct (or the validated ttsParams) and remove the silent ignore of UnmarshalBodyReusable errors.setting/ratio_setting/model_ratio.go (1)
747-754: Document theMaxTokens: -1sentinel.The sentinel meaning "no upper bound / catch-all" only exists in this comment. Anyone editing JSON via the admin UI (
UpdateContextTierRatioByJSONString) will not know they must terminate the list with-1, and a tier list without a-1entry will silently fail to match longer prompts (falling back throughrelay/helper/price.goto a default). At minimum add a doc comment ontypes.ContextTierRatio.MaxTokensdescribing the sentinel, and consider validating inUpdateContextTierRatioByJSONStringthat each list ends withMaxTokens == -1and is sorted ascending.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@setting/ratio_setting/model_ratio.go` around lines 747 - 754, Add documentation on the sentinel value for MaxTokens and validate admin-provided lists: update the comment/docs for types.ContextTierRatio.MaxTokens to state that MaxTokens == -1 is a required sentinel meaning "no upper bound / catch-all" and must be the final entry; then in UpdateContextTierRatioByJSONString add validation that the parsed slice is non-empty, that its last element has MaxTokens == -1, and that the MaxTokens values are strictly increasing (ascending) before accepting/saving, returning a clear error if these checks fail (referencing types.ContextTierRatio, UpdateContextTierRatioByJSONString, and defaultContextTierRatio to locate the logic).relay/playground_audio.go (4)
30-54: Minor: the cleanuptickeris never stopped, and an unrecoverable panic in the loop kills the cleanup forever.Two small hygiene improvements:
- Wrap the loop body in
defer recover()so a single panic (e.g., from a futureos.ReadDirchange) doesn't permanently disable cache cleanup.- Although the goroutine lives for the process lifetime, calling
defer ticker.Stop()is cheap and avoids tooling false positives about goroutine leaks.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/playground_audio.go` around lines 30 - 54, In the init() goroutine that runs the periodic cleanup, ensure the ticker is stopped by adding defer ticker.Stop() immediately after creating ticker, and protect the loop body against panics by wrapping each iteration (the code that calls audioDir(), os.ReadDir(), e.Info(), and os.Remove()) in a recover: use a deferred func() { if r := recover(); r != nil { /* swallow/log */ } } inside the loop so a single panic doesn’t kill the cleanup goroutine; reference the anonymous goroutine, ticker, and the calls to audioDir/os.ReadDir/e.Info/os.Remove when making the changes.
149-157: Consider sanitizingextbefore using it as a filename suffix.
audioExtFromContentTypeonly ever returns one of a small allow-listed set (.opus|.aac|.flac|.wav|.mp3), so today this is safe. However, the function ispackage‑private and the next refactor could plumb provider-supplied content-type or format strings directly into the path. Adding an explicit allow-list assertion (or usingfilepath.Clean+strings.ContainsAny(ext, "/\\")rejection) makes the invariant local and prevents a future regression that would let an attacker produce arbitrary filenames inos.TempDir()/new-api-audio.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/playground_audio.go` around lines 149 - 157, Sanitize and validate the extension returned from audioExtFromContentType before building filePath: ensure ext is exactly one of the allowed values (e.g., ".opus", ".aac", ".flac", ".wav", ".mp3") or reject if it contains any path separators or suspicious characters; if validation fails, log an error via logger.LogError and return the existing types.NewError like the os.WriteFile error path. Locate the call site using audioExtFromContentType and the variables audioID, filePath, audioDir and perform the check immediately after ext is assigned and before joining it into filePath.
58-85: Replace theReadDirlinear scan withfilepath.Glob.Each
/pg/audio/:audioIdrequest runsos.ReadDirover the entire cache directory and string-compares every entry. As the cache grows, this becomes O(N) per fetch and contends with the cleanup goroutine. Since the on-disk filename is always<id>.<ext>, a directfilepath.Glob(filepath.Join(dir, id+".*"))is a single syscall and avoids the loop:♻️ Suggested change
func GetCachedAudio(id string) ([]byte, string, bool) { - dir := audioDir() - entries, err := os.ReadDir(dir) - if err != nil { - return nil, "", false - } - for _, e := range entries { - if e.IsDir() { - continue - } - name := e.Name() - // match "uuid.ext" pattern - dotIdx := strings.LastIndex(name, ".") - if dotIdx == -1 { - continue - } - if name[:dotIdx] != id { - continue - } - ext := name[dotIdx:] - data, err := os.ReadFile(filepath.Join(dir, name)) - if err != nil { - return nil, "", false - } - return data, inferAudioContentType(ext[1:]), true // ext[1:] strips the leading "." - } - return nil, "", false + matches, err := filepath.Glob(filepath.Join(audioDir(), id+".*")) + if err != nil || len(matches) == 0 { + return nil, "", false + } + path := matches[0] + data, err := os.ReadFile(path) + if err != nil { + return nil, "", false + } + ext := filepath.Ext(path) + if ext == "" { + return data, "application/octet-stream", true + } + return data, inferAudioContentType(ext[1:]), true }This also makes the function safer if a non-audio file accidentally lands in the directory.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/playground_audio.go` around lines 58 - 85, GetCachedAudio currently scans the whole cache with os.ReadDir which is O(N); replace the linear scan by using filepath.Glob to directly match the filename pattern filepath.Join(audioDir(), id+".*"), handle the glob error (and return nil,"",false on error or no matches), pick the first match, extract the extension with filepath.Ext, read the file with os.ReadFile, and return the bytes and content type via inferAudioContentType(ext[1:]) (or false on read error); keep function name GetCachedAudio and helper audioDir()/inferAudioContentType references intact.
30-54: Cache TTL of 1 hour may be too aggressive given file-based serving.If a user revisits a previous playground conversation (refresh, re-open tab, browser history) more than an hour after it was generated, the markdown audio link
[Generated Audio](/pg/audio/<id>.mp3)will 404. Consider either:
- Extending the TTL to something like 24h (configurable).
- Refreshing the file's
ModTimeon read (touch on access inGetCachedAudio) so frequently-played clips don't get evicted.Also note that the cleanup goroutine never terminates on graceful shutdown — fine for the current process model, but worth keeping in mind if the app ever moves to a context-aware lifecycle.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@relay/playground_audio.go` around lines 30 - 54, The cleanup goroutine in init() currently evicts files older than 1 hour which causes valid playground audio links to 404; change the hardcoded cutoff to a configurable TTL (e.g., default 24h) used in the ticker loop (replace time.Hour with cfg.PlaygroundAudioTTL or a package-level default like defaultAudioTTL = 24*time.Hour) and use that variable when computing cutoff; additionally, in the audio fetch path (GetCachedAudio) update the file's mod time on successful read (use os.Chtimes or equivalent) so frequently accessed files are “touched” and not removed; optionally make the background goroutine context-aware (accept a context or package cancel) so it can stop on graceful shutdown.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@controller/channel-test.go`:
- Around line 45-55: The isImageOrVideoGenerationModel function inconsistently
checks some tokens against the original model and others against lower; update
all substring and prefix checks in isImageOrVideoGenerationModel to use the
normalized lower variable (e.g., replace strings.Contains(model, "gpt-image")
and strings.Contains(model, "dall-e") with strings.Contains(lower, "gpt-image")
and strings.Contains(lower, "dall-e")) so all comparisons are case-insensitive
and models like "DALL-E-3" or "GPT-Image-1" are classified correctly.
In `@controller/playground.go`:
- Around line 81-85: The handler currently maps every error from
model.GetChannelById(channelId, true) to a 404; change it to check
errors.Is(err, gorm.ErrRecordNotFound) and only return
c.JSON(http.StatusNotFound, ...) in that case; for any other error (e.g.,
transient DB/connection errors) return a 5xx response
(http.StatusInternalServerError or http.StatusBadGateway) and log/propagate the
original err so outages aren’t masked; update the branch around GetChannelById,
channelId and the c.JSON calls accordingly.
- Around line 112-117: Replace the per-request HTTP client created as client :=
&http.Client{} in controller/playground.go with a shared package-level
*http.Client that has a Transport configured with sensible timeouts (e.g.,
ResponseHeaderTimeout and IdleConnTimeout, and MaxIdleConns/MaxIdleConnsPerHost)
and optionally a global Timeout; ensure the handler still uses
NewRequestWithContext but uses this shared httpClient for client.Do so that
io.Copy(c.Writer, resp.Body) cannot hang indefinitely on slow upstreams and
connections are reused rather than allocating a new client per request.
- Around line 66-138: PlaygroundVideoProxy currently serves video content
without any rate limiting or auth; add protection by applying
middleware.DownloadRateLimit() to the pgVideoRouter route that mounts
PlaygroundVideoProxy (and the /pg/audio/:audioId proxy route) or require
authentication for those endpoints (or switch to HMAC-signed short‑TTL tokens
for generated proxy URLs), update any router comment that says "IDs are
unguessable" to clarify only videoId is high-entropy while channelId is
enumerable, and verify the handler (PlaygroundVideoProxy) still reads
Authorization headers or token checks as needed after adding the chosen
protection.
In `@middleware/request_logger.go`:
- Around line 14-27: The middleware currently buffers entire responses in
responseBodyWriter.buf via Write and WriteString, which causes unbounded memory
growth for large endpoints (e.g., relay-router installs this globally); change
responseBodyWriter to cap buffering: add a maxBufSize constant and in
responseBodyWriter.Write and WriteString only append up to remaining bytes
(ignore the rest) so the response still streams to the underlying
gin.ResponseWriter but only the first N bytes are kept for logging; update the
type (e.g., responseBodyWriter, buf, maxBufSize) and both Write/WriteString
implementations to enforce the limit and avoid full-memory accumulation (also
apply the same change to the other buffered region referenced at lines 67-87).
- Around line 38-87: Current logging dumps raw request and response bodies;
update the request and response logging to be content-type aware, size-capped,
and redact sensitive fields instead of printing raw payloads. In the request
block around bodyStorage/BodyBytes and the response block using rbw.buf, only
attempt to read and log bodies for text/* or application/json (check
c.Request.Header.Get("Content-Type") and rbw.Header().Get("Content-Type")),
enforce a small max size (e.g., 4 KB) for both request and response, and if body
exceeds the cap log a truncated message; when content-type is application/json
parse the JSON and redact known sensitive keys (e.g.,
"password","token","authorization","api_key","secret","apiKey") before logging;
for non-text/binary types log a placeholder like "<binary content omitted>" and
retain existing header redaction for Authorization/x-api-key. Apply changes near
GetBodyStorage, bodyBytes usage, and where rbw.buf.Bytes() is consumed, and
factor redaction parsing into a helper (e.g., redactJSON) used by the
middleware.
In `@model/main.go`:
- Line 205: The migration calls were commented out so AutoMigrate isn't running;
restore the migration steps by re-enabling the migrateDB() and migrateLOGDB()
calls (or explicitly replace them by calling migrateDBFast() if that was the
intended fast-path) so DB.AutoMigrate(...) runs; ensure InitLogDB() still
triggers creation of the Log table when LOG_SQL_DSN is set and verify
model.CheckSetup()/GetSetup()/RootUserExists() and model.InitOptionMap() run
after the migrations to avoid missing-table/column errors.
In `@relay/channel/api_request.go`:
- Around line 296-313: The code eagerly reads requestBody into bodyBytes for
every request which breaks streaming/disk-backed handling and uses extra memory;
change it to only read into memory when common2.DebugEnabled is true: leave
requestBody as the io.ReadCloser (or io.Reader) and pass it directly to
http.NewRequest for the normal path, and when DebugEnabled is true, tee or
buffer the stream (e.g., io.TeeReader into a bytes.Buffer or copy into a
temporary buffer) to capture body for logger.LogDebug while preserving the
original reader for http.NewRequest; update the code around requestBody,
bodyBytes, logger.LogDebug and http.NewRequest (c.Request.Method,
fullRequestURL) so buffering happens conditionally only for debugging.
In `@relay/channel/openai/relay-openai.go`:
- Around line 156-183: The polling loop currently retries unbounded on
network/read/unmarshal errors and never checks pollResp.StatusCode; update the
loop in relay-openai.go that constructs req/pollResp and decodes into
videoStatusResponse to (1) inspect pollResp.StatusCode and treat non-2xx as
terminal errors (return "", fmt.Errorf(...)), (2) maintain a consecutive failure
counter (e.g., failCount incremented on network/read/unmarshal/non-2xx and reset
on success) and return a terminal error once failures exceed a threshold (e.g.,
>5), and (3) replace the fixed sleep with a mild exponential backoff capped
(start ~3s, double up to ~30s) between retries to avoid hammering the upstream.
- Around line 140-141: Change the timeout context to derive from the HTTP
request context instead of Background by using
context.WithTimeout(c.Request.Context(), 10*time.Minute) for the ctx/cancel used
in the polling goroutine; remove the duplicated c.Request.Context().Done()
select arm inside the loop and replace time.Sleep(pollInterval) with a select
that waits on either ctx.Done() or time.After(pollInterval) so cancellation
(from client disconnect or timeout) is honoured immediately; update references
to ctx, cancel, and pollInterval in the polling loop accordingly.
- Around line 185-209: Remove the unreachable "created" switch branch in the
status handling block: locate the switch over status.Status (watch for
prevStatus, sendProgress, and the surrounding logic in relay-openai.go) and
delete the case "created" that calls sendProgress; keep only the valid OpenAI
statuses "queued", "in_progress", "completed", and "failed" so the code matches
the API contract and no-op for unknown/unexpected statuses.
In `@relay/compatible_handler.go`:
- Around line 50-67: The current heuristic helpers isVideoGenerationModel and
isTTSModel are too broad and misclassify models; tighten them by removing global
substring checks like "generate" and bare "speech" and instead match explicit
prefixes/suffixes or an allow-list. Update isVideoGenerationModel to check
canonical families/patterns only (e.g., strings.HasPrefix(lower,"veo"),
strings.HasPrefix(lower,"sora"), strings.Contains(lower,"-video-") or
strings.HasSuffix(lower,"-video"), and include explicit image families like
"-image-") and remove the generic "generate" check; update isTTSModel to only
match tts-specific forms (e.g., strings.HasPrefix(lower,"tts"),
strings.HasSuffix(lower,"-tts"), or exact known names like "cosyvoice" and
"sambert") or use an explicit allow-list array you compare against, so you no
longer rely on broad substrings that collide with STT or other models.
In `@relay/helper/stream_scanner.go`:
- Around line 247-263: The scanner currently forwards bare JSON lines (starting
with "{") unconditionally via dataChan to all StreamScannerHandler consumers,
which causes non-video-aware handlers (e.g., those calling common.Unmarshal in
their Handle/HandleStream methods) to fail; modify StreamScanner (the code
around the bare-JSON branch in scan loop) to gate forwarding by provider or
explicit opt-in (e.g., check IsPlayground + provider/type flag) or add a new
boolean (e.g., allowBareJSON) on StreamScanner/StreamScannerHandler and only
send trimmed JSON when that flag/provider indicates video-capable handling;
alternatively update non-video handlers (those that call common.Unmarshal
directly) to detect object type before unmarshalling (similar to
handleVideoStreamData) and ignore/skip unmarshall errors for unrecognized object
types, ensuring they call info.SetFirstResponseTime/ReceivedResponseCount only
for accepted objects and avoid calling sr.Error() on benign bare-JSON lines.
In `@setting/ratio_setting/model_ratio.go`:
- Around line 749-753: The "qwen3-max" tier list has a mispriced middle tier:
update the second entry in the "qwen3-max" slice (currently {MaxTokens: 131072,
InputRatio: 2.0, CompletionRatio: 4}) so its InputRatio is greater than the
first tier's 2.5 and less than the third tier's 3.5 (e.g., set InputRatio to
3.0) to ensure monotonic pricing; leave other fields unchanged unless you also
want to align CompletionRatio with official tiers.
In `@types/price_data.go`:
- Line 23: The debug/output created by ToSetting() omits the AudioMinutePrice
field so minute-based STT billing can be invisible; update the ToSetting()
implementation to include AudioMinutePrice (e.g., include the float value when
>0 or always include it) alongside the other pricing fields so it appears in
diagnostics/logs—locate the ToSetting() method for the type that defines
AudioMinutePrice and add the field name and its value to the returned
representation (or map/string) consistent with how other price fields are
rendered.
In `@web/src/components/common/markdown/MarkdownRenderer.jsx`:
- Around line 509-516: The audio-detection regex in MarkdownRenderer (the block
using hrefPath, aProps, href and returning <AudioPlayer />) misses .flac links,
so update the regex to include "flac" (e.g., add flac to the group and keep
case-insensitive matching) so /pg/audio/<uuid>.flac is detected and rendered by
AudioPlayer instead of a plain anchor.
- Around line 517-528: The video renderer flags URLs via isVideoUrl (uses
hrefPath and href) but always sets <source type='video/mp4'>; change it to
derive the MIME from the file extension (e.g., .mp4 -> video/mp4, .webm ->
video/webm, .ogv -> video/ogg, .avi/.mpeg -> video/mpeg) or omit the type
attribute when unknown, and use that computed MIME in the <source> for the video
element returned by the MarkdownRenderer component so the declared MIME matches
the actual file.
- Around line 475-478: The ReactMarkdown usage in MarkdownRenderer.jsx currently
sets urlTransform={(url) => url}, which bypasses react-markdown's built-in
URL/protocol sanitization and can allow unsafe schemes; remove the urlTransform
prop from the <ReactMarkdown> invocation (or stop overriding it) so
react-markdown's default URL filtering is preserved and dangerous protocols
(javascript:, vbscript:, file:, etc.) are blocked before URLs reach custom
renderers like your image/link renderers.
In `@web/src/hooks/playground/useApiRequest.jsx`:
- Around line 35-63: formatVideoResponse currently embeds hard-coded Chinese and
English labels (e.g., `模型`, `分辨率`, `时长`, `状态`, `**视频生成完成**`, `[Generated
Video]`) which bypasses i18n; fix by making the helper use the translation
function: either move `formatVideoResponse` inside the hook where
`useTranslation()` is available or change its signature to accept a `t`
parameter, replace each literal label with `t('key')` calls (create descriptive
keys like video.model, video.resolution, video.duration, video.status,
video.generatedTitle, video.generatedLink), update the call site(s) to pass `t`
if needed, and run `bun run i18n:extract` / `bun run i18n:sync` to register the
new keys.
---
Outside diff comments:
In `@relay/channel/api_request.go`:
- Around line 304-345: The debug logging currently prints raw bodyBytes and
req.Header; instead create redacted copies before calling logger.LogDebug: for
headers (in the block using req.Header and logger.LogDebug) build a sanitized
header map that masks or removes sensitive keys like "Authorization",
"X-Api-Key", "X-Goog-Api-Key", "Api-Key" and any other custom secret headers
(use strings.EqualFold to match), and for the body (where bodyBytes is logged)
produce a redactedBody string that strips or replaces likely sensitive fields
such as "prompt", "password", "secret", "token" or any JSON keys from the
request body (or remove large/unknown bodies) and log the sanitized versions;
ensure these redactions are applied both in the first debug block that logs
bodyBytes and the second block that logs headers, referencing bodyBytes,
req.Header, processHeaderOverride, applyHeaderOverrideToRequest and
logger.LogDebug so you modify those logging sites to use the sanitized copies.
In `@service/quota.go`:
- Around line 269-291: The code computes completionRatio, audioRatio, and
audioCompletionRatio unconditionally but then uses a duration-based billing path
when audioMinutePrice > 0 && relayInfo.AudioDurationSeconds > 0, causing
GenerateAudioOtherInfo to receive token-ratio values that were not used; fix by
moving the decimal calculations (completionRatio, audioRatio,
audioCompletionRatio) into the token-billing else branch where they are actually
applied, and in the audio-minute billing branch either call
GenerateAudioOtherInfo with zeroed token-ratio fields or create a separate
"other" payload that records audioMinutePrice and durationMinutes (use
relayInfo.AudioDurationSeconds and the computed durationMinutes) so logs reflect
the true billing factors; update calls to GenerateAudioOtherInfo accordingly.
- Around line 316-323: The current guard only treats TotalTokens==0 &&
AudioDurationSeconds==0 as an error, but when AudioDurationSeconds>0 and
audioMinutePrice<=0 (or calculated quota==0) we currently record a successful
zero-quota consume; update the logic around calculateAudioQuota/quota so that if
usage.TotalTokens==0 and (relayInfo.AudioDurationSeconds==0 OR
audioMinutePrice<=0 OR quota==0) we enter the error/log branch (set quota=0, log
with logger.LogError including relayInfo and FinalPreConsumedQuota) instead of
allowing model.UpdateUserUsedQuotaAndRequestCount to record a silent zero
consume; reference usage.TotalTokens, relayInfo.AudioDurationSeconds,
audioMinutePrice, calculateAudioQuota, quota, and
model.UpdateUserUsedQuotaAndRequestCount when making the change.
---
Nitpick comments:
In `@controller/channel-test.go`:
- Around line 49-54: The current model detection uses strings.Contains(lower,
"flux") which is too broad; update the condition that builds the image-model
predicate (the boolean expression using variable lower and string checks like
"stable-diffusion", "veo-", "sora-") to tighten the flux match — replace
strings.Contains(lower, "flux") with a bounded check such as
strings.HasPrefix(lower, "flux") or strings.HasPrefix(lower, "flux-") (or
consult an explicit allowlist of known flux vendor prefixes) so model names
containing "flux" elsewhere are not misrouted to image generation.
In `@relay/channel/openai/audio.go`:
- Around line 124-161: The branch wrongly treats an empty responseFormat as
verbose_json; change the condition to only treat responseFormat ==
"verbose_json" as the branch that looks for duration and segment info (so
info.AudioDurationSeconds is only attempted when verbose_json is explicitly
requested), and factor out the duplicated token-normalization logic (currently
duplicated in the verboseResp and responseData handling) into a small helper
(e.g., normalizeUsage(usage *dto.Usage)) that sets PromptTokens from InputTokens
and CompletionTokens from OutputTokens when zero and returns the normalized
*dto.Usage; update the code paths that return usage (the blocks referencing
verboseResp, responseData and their Usage) to call normalizeUsage before
returning.
In `@relay/channel/openai/relay-openai.go`:
- Around line 765-788: The SSE chunks constructed as contentChunk and
finishChunk lack the standard metadata fields (Id, Object, Created, Model);
update the ChatCompletionsStreamResponse objects in this block to populate Id,
Object ("chat.completion.chunk"), Created (timestamp/unix), and Model (same
model string used elsewhere) just like pollVideoUntilComplete and
handleVideoStreamData do, then emit them via helper.StringData(c, ...) so strict
SSE clients receive consistent chunk structure.
- Around line 745-748: When common.Unmarshal(responseBody, &imageResp) fails or
imageResp.Data is empty, don't silently call helper.Done(c); instead construct
and send an error content chunk (similar to the "❌ 视频生成失败" pattern used in
handleVideoStreamData) before calling helper.Done(c). Capture the unmarshalling
error and/or include the raw responseBody as diagnostic details, create the
appropriate helper.Send... (or helper.WriteChunk) call that emits a visible
assistant error message, then call helper.Done(c) so the playground shows the
failure instead of an empty assistant message.
In `@relay/compatible_handler.go`:
- Around line 26-35: The function name extractImagePromptFromMessages is
misleading because it's also used to pull TTS input; rename the function to
lastUserMessageText and update its comment to reflect that it returns the text
of the last user message for any consumer (e.g., image prompt or TTS), then
update all call sites (including the place that extracts TTS input) to use
lastUserMessageText instead of extractImagePromptFromMessages and ensure
dto.Message.StringContent() behavior is unchanged.
- Around line 91-121: The code currently re-decodes the request body with
common.UnmarshalBodyReusable into ttsParams while chatReq was already created
from the same body, and the call ignores errors; change this to a single decode
or handle errors: either define a composite struct that embeds
dto.GeneralOpenAIRequest and adds Voice, Speed, ResponseFormat, decode the body
once into that (then build dto.AudioRequest from the embedded fields and
Messages) or at minimum check the error returned by common.UnmarshalBodyReusable
and call processLogger.Debug/Debugf (or the local logger) when it fails so
malformed JSON is visible; update the branch that constructs dto.AudioRequest
and the call to PlaygroundTTSHelper(c, info) to use the parsed composite struct
(or the validated ttsParams) and remove the silent ignore of
UnmarshalBodyReusable errors.
In `@relay/helper/stream_scanner.go`:
- Around line 248-261: Duplicate forwarding logic in stream_scanner.go should be
extracted into a small local helper to avoid repetition and keep cancellation
semantics in sync: add a closure like forward := func(payload string) bool that
calls info.SetFirstResponseTime(), increments info.ReceivedResponseCount, then
performs the select to send payload on dataChan or return false if ctx.Done() or
stopChan fires; replace both places that currently duplicate this block (the
bare-JSON branch and the other SSE-data branch) with calls to forward(payload)
and stop scanning when it returns false.
In `@relay/playground_audio.go`:
- Around line 30-54: In the init() goroutine that runs the periodic cleanup,
ensure the ticker is stopped by adding defer ticker.Stop() immediately after
creating ticker, and protect the loop body against panics by wrapping each
iteration (the code that calls audioDir(), os.ReadDir(), e.Info(), and
os.Remove()) in a recover: use a deferred func() { if r := recover(); r != nil {
/* swallow/log */ } } inside the loop so a single panic doesn’t kill the cleanup
goroutine; reference the anonymous goroutine, ticker, and the calls to
audioDir/os.ReadDir/e.Info/os.Remove when making the changes.
- Around line 149-157: Sanitize and validate the extension returned from
audioExtFromContentType before building filePath: ensure ext is exactly one of
the allowed values (e.g., ".opus", ".aac", ".flac", ".wav", ".mp3") or reject if
it contains any path separators or suspicious characters; if validation fails,
log an error via logger.LogError and return the existing types.NewError like the
os.WriteFile error path. Locate the call site using audioExtFromContentType and
the variables audioID, filePath, audioDir and perform the check immediately
after ext is assigned and before joining it into filePath.
- Around line 58-85: GetCachedAudio currently scans the whole cache with
os.ReadDir which is O(N); replace the linear scan by using filepath.Glob to
directly match the filename pattern filepath.Join(audioDir(), id+".*"), handle
the glob error (and return nil,"",false on error or no matches), pick the first
match, extract the extension with filepath.Ext, read the file with os.ReadFile,
and return the bytes and content type via inferAudioContentType(ext[1:]) (or
false on read error); keep function name GetCachedAudio and helper
audioDir()/inferAudioContentType references intact.
- Around line 30-54: The cleanup goroutine in init() currently evicts files
older than 1 hour which causes valid playground audio links to 404; change the
hardcoded cutoff to a configurable TTL (e.g., default 24h) used in the ticker
loop (replace time.Hour with cfg.PlaygroundAudioTTL or a package-level default
like defaultAudioTTL = 24*time.Hour) and use that variable when computing
cutoff; additionally, in the audio fetch path (GetCachedAudio) update the file's
mod time on successful read (use os.Chtimes or equivalent) so frequently
accessed files are “touched” and not removed; optionally make the background
goroutine context-aware (accept a context or package cancel) so it can stop on
graceful shutdown.
In `@setting/ratio_setting/model_ratio.go`:
- Around line 747-754: Add documentation on the sentinel value for MaxTokens and
validate admin-provided lists: update the comment/docs for
types.ContextTierRatio.MaxTokens to state that MaxTokens == -1 is a required
sentinel meaning "no upper bound / catch-all" and must be the final entry; then
in UpdateContextTierRatioByJSONString add validation that the parsed slice is
non-empty, that its last element has MaxTokens == -1, and that the MaxTokens
values are strictly increasing (ascending) before accepting/saving, returning a
clear error if these checks fail (referencing types.ContextTierRatio,
UpdateContextTierRatioByJSONString, and defaultContextTierRatio to locate the
logic).
In `@types/context_tier.go`:
- Around line 12-24: Update the godoc for SelectContextTier to state that it
returns the first tier whose MaxTokens is -1 or >= promptTokens, that if no tier
matches it returns the last tier in the slice (not nil) as a fallback, and that
it only returns nil when the input slice is empty; also document the
precondition that tiers must be ordered ascending by MaxTokens. Additionally,
add/mention validation of this ordering when loading/parsing tiers (e.g. in
setting/ratio_setting/model_ratio.go) or sort/validate the slice before calling
SelectContextTier to ensure the contract is enforced.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f7548255-785a-4379-8ea9-8df6d609cbf6
⛔ Files ignored due to path filters (1)
web/bun.lockis excluded by!**/*.lock
📒 Files selected for processing (24)
common/endpoint_defaults.gocontroller/channel-test.gocontroller/playground.gomiddleware/request_logger.gomodel/main.gomodel/option.gorelay/channel/api_request.gorelay/channel/openai/audio.gorelay/channel/openai/relay-openai.gorelay/common/relay_info.gorelay/compatible_handler.gorelay/helper/price.gorelay/helper/stream_scanner.gorelay/playground_audio.gorouter/relay-router.goservice/quota.gosetting/ratio_setting/model_ratio.gotypes/context_tier.gotypes/price_data.goweb/src/components/common/markdown/MarkdownRenderer.jsxweb/src/hooks/playground/useApiRequest.jsxweb/src/i18n/locales/en.jsonweb/src/i18n/locales/zh-CN.jsonweb/src/pages/Setting/Ratio/ModelRatioSettings.jsx
| func isImageOrVideoGenerationModel(model string) bool { | ||
| lower := strings.ToLower(model) | ||
| return strings.Contains(model, "gpt-image") || | ||
| strings.Contains(model, "dall-e") || | ||
| strings.Contains(lower, "stable-diffusion") || | ||
| strings.Contains(lower, "flux") || | ||
| strings.HasPrefix(lower, "veo-") || | ||
| strings.HasPrefix(lower, "sora-") || | ||
| strings.Contains(lower, "imagen") || | ||
| strings.Contains(lower, "seedream") | ||
| } |
There was a problem hiding this comment.
Inconsistent case-sensitivity in model classification.
gpt-image and dall-e are matched against the raw model argument, while every other token is matched against lower. This means "DALL-E-3", "DALL-E", or "GPT-Image-1" will not be classified as image-generation models and will be misrouted to /v1/chat/completions during channel tests. Use lower consistently:
🐛 Proposed fix
func isImageOrVideoGenerationModel(model string) bool {
lower := strings.ToLower(model)
- return strings.Contains(model, "gpt-image") ||
- strings.Contains(model, "dall-e") ||
+ return strings.Contains(lower, "gpt-image") ||
+ strings.Contains(lower, "dall-e") ||
strings.Contains(lower, "stable-diffusion") ||
strings.Contains(lower, "flux") ||
strings.HasPrefix(lower, "veo-") ||
strings.HasPrefix(lower, "sora-") ||
strings.Contains(lower, "imagen") ||
strings.Contains(lower, "seedream")
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@controller/channel-test.go` around lines 45 - 55, The
isImageOrVideoGenerationModel function inconsistently checks some tokens against
the original model and others against lower; update all substring and prefix
checks in isImageOrVideoGenerationModel to use the normalized lower variable
(e.g., replace strings.Contains(model, "gpt-image") and strings.Contains(model,
"dall-e") with strings.Contains(lower, "gpt-image") and strings.Contains(lower,
"dall-e")) so all comparisons are case-insensitive and models like "DALL-E-3" or
"GPT-Image-1" are classified correctly.
| func PlaygroundVideoProxy(c *gin.Context) { | ||
| channelIdStr := c.Param("channelId") | ||
| videoId := c.Param("videoId") | ||
|
|
||
| if channelIdStr == "" || videoId == "" { | ||
| c.JSON(http.StatusBadRequest, gin.H{"error": "missing channelId or videoId"}) | ||
| return | ||
| } | ||
|
|
||
| channelId, err := strconv.Atoi(channelIdStr) | ||
| if err != nil { | ||
| c.JSON(http.StatusBadRequest, gin.H{"error": "invalid channelId"}) | ||
| return | ||
| } | ||
|
|
||
| channel, err := model.GetChannelById(channelId, true) | ||
| if err != nil { | ||
| c.JSON(http.StatusNotFound, gin.H{"error": "channel not found"}) | ||
| return | ||
| } | ||
|
|
||
| baseURL := "" | ||
| if channel.BaseURL != nil { | ||
| baseURL = strings.TrimRight(*channel.BaseURL, "/") | ||
| } | ||
| if baseURL == "" { | ||
| c.JSON(http.StatusBadRequest, gin.H{"error": "channel has no base URL"}) | ||
| return | ||
| } | ||
|
|
||
| keys := channel.GetKeys() | ||
| if len(keys) == 0 { | ||
| c.JSON(http.StatusInternalServerError, gin.H{"error": "channel has no API key"}) | ||
| return | ||
| } | ||
| apiKey := keys[0] | ||
|
|
||
| upstreamURL := fmt.Sprintf("%s/videos/%s/content", baseURL, videoId) | ||
|
|
||
| req, err := http.NewRequestWithContext(c.Request.Context(), http.MethodGet, upstreamURL, nil) | ||
| if err != nil { | ||
| c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create upstream request"}) | ||
| return | ||
| } | ||
| req.Header.Set("Authorization", "Bearer "+apiKey) | ||
|
|
||
| client := &http.Client{} | ||
| resp, err := client.Do(req) | ||
| if err != nil { | ||
| c.JSON(http.StatusBadGateway, gin.H{"error": "upstream request failed: " + err.Error()}) | ||
| return | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| if resp.StatusCode != http.StatusOK { | ||
| body, _ := io.ReadAll(resp.Body) | ||
| c.Data(resp.StatusCode, resp.Header.Get("Content-Type"), body) | ||
| return | ||
| } | ||
|
|
||
| contentType := resp.Header.Get("Content-Type") | ||
| if contentType == "" { | ||
| contentType = "video/mp4" | ||
| } | ||
| c.Header("Content-Type", contentType) | ||
| if cl := resp.Header.Get("Content-Length"); cl != "" { | ||
| c.Header("Content-Length", cl) | ||
| } | ||
| c.Header("Cache-Control", "no-store") | ||
|
|
||
| c.Status(http.StatusOK) | ||
| io.Copy(c.Writer, resp.Body) //nolint:errcheck | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm router wiring and auth middleware (or absence thereof) on /pg routes.
rg -nP -C5 '"/pg"|/pg/video|/pg/audio|PlaygroundVideoProxy|PlaygroundAudioProxy' --type=goRepository: QuantumNous/new-api
Length of output: 5729
🏁 Script executed:
#!/bin/bash
# 1. Find where videoId is generated or comes from (especially for Sora)
rg -nP 'videoID|videoId|video_id' controller/playground.go relay/channel/openai/ --type=go -A 2 -B 2 | head -60
# 2. Check for rate limiting implementation
rg -nP 'rate|limit|RateLimit|Limit' router/relay-router.go middleware/ --type=go 2>/dev/null | head -30
# 3. Verify videoId format from OpenAI Sora task/status handling
rg -nP 'type.*Task|TaskResponse|Status' relay/channel/openai/ --type=go | grep -i video | head -20Repository: QuantumNous/new-api
Length of output: 5712
🏁 Script executed:
#!/bin/bash
# Check OpenAI Sora task structure to understand videoId format
fd -t f 'sora|task' relay/channel/openai/ --type=go | xargs rg -l 'video.*id|Video' --type=go | head -5Repository: QuantumNous/new-api
Length of output: 365
🏁 Script executed:
#!/bin/bash
# Look for any rate limiting or abuse protection middleware
rg -nP 'Limiter|RateLimit|abuse|throttle|quota' --type=go | grep -i 'middleware\|controller\|handler' | head -20Repository: QuantumNous/new-api
Length of output: 2126
🏁 Script executed:
#!/bin/bash
# Check main router setup and global middleware
rg -nP 'func.*[Rr]outer|engine.*Use|main.go|router.go' router/ --type=go -B 2 -A 5 | head -100Repository: QuantumNous/new-api
Length of output: 3378
🏁 Script executed:
#!/bin/bash
# Look for how the main Gin engine is initialized and what global middleware is applied
fd -t f 'main.go' . | xargs cat | head -150Repository: QuantumNous/new-api
Length of output: 3916
🏁 Script executed:
#!/bin/bash
# Check for any wrapper around pgVideoRouter that might apply middleware globally
rg -nP 'pgVideoRouter|playgroundRouter|Group.*"/pg"' router/relay-router.go -B 5 -A 10Repository: QuantumNous/new-api
Length of output: 1197
🌐 Web query:
OpenAI Sora video ID format uuid entropy
💡 Result:
OpenAI Sora video IDs follow the format "video_" followed by a 64-character hexadecimal string. Example: "video_68d7512d07848190b3e45da0ecbebcde004da08e1e0678d5". This appears to be a custom opaque identifier rather than a standard UUID (which is typically 36 characters with hyphens in canonical form). The hex string provides high entropy (approximately 251 bits, since 64 hex digits = 256 bits minus any potential prefix/checksum), suitable for uniquely identifying billions of videos with negligible collision risk. No official documentation specifies the exact generation method (e.g., SHA-256 hash, truncated UUID, or random hex), but official API examples and references consistently use this prefixed hex format.
Citations:
- 1: https://developers.openai.com/api/docs/guides/video-generation
- 2: https://developers.openai.com/api/reference/resources/videos
- 3: https://developers.openai.com/api/reference/go/resources/videos/methods/retrieve
- 4: https://github.com/promptadvisers/sora-studio/blob/main/docs/openai-video-api-guide.md
- 5: https://platform.openai.com/docs/api-reference/videos/retrieve
- 6: http://developers.openai.com/api/docs/guides/video-generation
🌐 Web query:
OpenAI Sora API video task response ID format
💡 Result:
The OpenAI Sora API (Videos API at /v1/videos) returns a video task (job) response containing a unique ID for the video generation job. This ID is used to poll status via GET /videos/{video_id} and download the video via GET /videos/{video_id}/content once completed. The format is a string prefixed with "video_", followed by a 64-character hexadecimal string (32 bytes in hex). Example from official documentation: "video_68d7512d07848190b3e45da0ecbebcde004da08e1e0678d5" Other examples: "video_123", "video_abc123". The full response object is a JSON with fields like: { "id": "video_...", "object": "video", "created_at": 1758941485, "status": "queued", "model": "sora-2-pro", "progress": 0, "seconds": "8", "size": "1280x720" }
Citations:
- 1: https://developers.openai.com/api/docs/guides/video-generation
- 2: https://developers.openai.com/api/reference/resources/videos
- 3: https://developers.openai.com/api/reference/ruby/resources/videos/methods/create/
- 4: http://developers.openai.com/api/docs/guides/video-generation
Add rate limiting to /pg/video/:channelId/:videoId/content and /pg/audio/:audioId routes; clarify the authentication assumption.
The route group pgVideoRouter (relay-router.go:72-76) lacks any middleware protection. While OpenAI Sora videoId is sufficiently high-entropy (~251 bits), channelId is numeric and fully enumerable. An attacker can iterate channel IDs and, once a videoId is observed (via chat logs, screenshots, or cached links), fetch the full video binary on the target's quota without authentication or rate limiting.
Current threat model assumptions are partially undermined:
videoIdentropy: ✓ Sufficient (~251 bits; OpenAI "video_" + 64 hex chars)- Unauthenticated access: ✓ Confirmed—route has no auth middleware
- Rate limiting: ✗ Missing—no rate limiting applied despite
DownloadRateLimit()existing in middleware
Recommended fix:
- Apply
middleware.DownloadRateLimit()to both video and audio proxy routes, or - Require authentication (token or session) for these endpoints, or
- Add HMAC-signed URL tokens with short TTL to the generated proxy URLs.
The router comment (line 71) "IDs are unguessable" should be corrected—only videoId is unguessable; channelId is enumerable.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@controller/playground.go` around lines 66 - 138, PlaygroundVideoProxy
currently serves video content without any rate limiting or auth; add protection
by applying middleware.DownloadRateLimit() to the pgVideoRouter route that
mounts PlaygroundVideoProxy (and the /pg/audio/:audioId proxy route) or require
authentication for those endpoints (or switch to HMAC-signed short‑TTL tokens
for generated proxy URLs), update any router comment that says "IDs are
unguessable" to clarify only videoId is high-entropy while channelId is
enumerable, and verify the handler (PlaygroundVideoProxy) still reads
Authorization headers or token checks as needed after adding the chosen
protection.
| channel, err := model.GetChannelById(channelId, true) | ||
| if err != nil { | ||
| c.JSON(http.StatusNotFound, gin.H{"error": "channel not found"}) | ||
| return | ||
| } |
There was a problem hiding this comment.
Distinguish "not found" from transient DB errors.
GetChannelById can return errors other than gorm.ErrRecordNotFound (e.g., transient connection issues), but here every error is mapped to 404 channel not found, which is misleading and can mask outages. Consider checking for errors.Is(err, gorm.ErrRecordNotFound) and returning 500 (or 502) for everything else.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@controller/playground.go` around lines 81 - 85, The handler currently maps
every error from model.GetChannelById(channelId, true) to a 404; change it to
check errors.Is(err, gorm.ErrRecordNotFound) and only return
c.JSON(http.StatusNotFound, ...) in that case; for any other error (e.g.,
transient DB/connection errors) return a 5xx response
(http.StatusInternalServerError or http.StatusBadGateway) and log/propagate the
original err so outages aren’t masked; update the branch around GetChannelById,
channelId and the c.JSON calls accordingly.
| client := &http.Client{} | ||
| resp, err := client.Do(req) | ||
| if err != nil { | ||
| c.JSON(http.StatusBadGateway, gin.H{"error": "upstream request failed: " + err.Error()}) | ||
| return | ||
| } |
There was a problem hiding this comment.
Add timeout / safer transport to upstream HTTP client.
client := &http.Client{} has no Timeout and uses the default transport. If the upstream stalls after the headers are read, io.Copy(c.Writer, resp.Body) at line 137 can hang indefinitely, holding the request goroutine and a backend connection. Although c.Request.Context() is propagated via NewRequestWithContext, that only aborts in‑flight reads when the client disconnects — a slow/idle upstream will still block a healthy client.
Consider sharing a package-level *http.Client with at minimum a Transport configured for ResponseHeaderTimeout and IdleConnTimeout, and creating the client once instead of on every request.
🛠️ Suggested change
- client := &http.Client{}
- resp, err := client.Do(req)
+ // videoProxyClient is a package-level client; consider declaring at file scope.
+ client := &http.Client{
+ Transport: &http.Transport{
+ ResponseHeaderTimeout: 30 * time.Second,
+ IdleConnTimeout: 90 * time.Second,
+ },
+ }
+ resp, err := client.Do(req)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| client := &http.Client{} | |
| resp, err := client.Do(req) | |
| if err != nil { | |
| c.JSON(http.StatusBadGateway, gin.H{"error": "upstream request failed: " + err.Error()}) | |
| return | |
| } | |
| // videoProxyClient is a package-level client; consider declaring at file scope. | |
| client := &http.Client{ | |
| Transport: &http.Transport{ | |
| ResponseHeaderTimeout: 30 * time.Second, | |
| IdleConnTimeout: 90 * time.Second, | |
| }, | |
| } | |
| resp, err := client.Do(req) | |
| if err != nil { | |
| c.JSON(http.StatusBadGateway, gin.H{"error": "upstream request failed: " + err.Error()}) | |
| return | |
| } |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@controller/playground.go` around lines 112 - 117, Replace the per-request
HTTP client created as client := &http.Client{} in controller/playground.go with
a shared package-level *http.Client that has a Transport configured with
sensible timeouts (e.g., ResponseHeaderTimeout and IdleConnTimeout, and
MaxIdleConns/MaxIdleConnsPerHost) and optionally a global Timeout; ensure the
handler still uses NewRequestWithContext but uses this shared httpClient for
client.Do so that io.Copy(c.Writer, resp.Body) cannot hang indefinitely on slow
upstreams and connections are reused rather than allocating a new client per
request.
| type responseBodyWriter struct { | ||
| gin.ResponseWriter | ||
| buf *bytes.Buffer | ||
| } | ||
|
|
||
| func (w *responseBodyWriter) Write(b []byte) (int, error) { | ||
| w.buf.Write(b) | ||
| return w.ResponseWriter.Write(b) | ||
| } | ||
|
|
||
| func (w *responseBodyWriter) WriteString(s string) (int, error) { | ||
| w.buf.WriteString(s) | ||
| return w.ResponseWriter.WriteString(s) | ||
| } |
There was a problem hiding this comment.
Don't buffer whole responses in this middleware.
responseBodyWriter keeps every byte in rbw.buf and only truncates at log time. Because router/relay-router.go installs this globally, the new /pg/video/.../content, /pg/audio/..., and SSE relay responses will all accumulate fully in memory before the request finishes.
Also applies to: 67-87
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@middleware/request_logger.go` around lines 14 - 27, The middleware currently
buffers entire responses in responseBodyWriter.buf via Write and WriteString,
which causes unbounded memory growth for large endpoints (e.g., relay-router
installs this globally); change responseBodyWriter to cap buffering: add a
maxBufSize constant and in responseBodyWriter.Write and WriteString only append
up to remaining bytes (ignore the rest) so the response still streams to the
underlying gin.ResponseWriter but only the first N bytes are kept for logging;
update the type (e.g., responseBodyWriter, buf, maxBufSize) and both
Write/WriteString implementations to enforce the limit and avoid full-memory
accumulation (also apply the same change to the other buffered region referenced
at lines 67-87).
| ImageRatio float64 | ||
| AudioRatio float64 | ||
| AudioCompletionRatio float64 | ||
| AudioMinutePrice float64 // >0 时对 STT 模型按分钟计费（美元/分钟），覆盖 token 计费 |
There was a problem hiding this comment.
Also surface AudioMinutePrice in ToSetting().
This field now affects billing, but the debug string still omits it, so STT minute-based pricing can be active without appearing in logs or diagnostics.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@types/price_data.go` at line 23, The debug/output created by ToSetting()
omits the AudioMinutePrice field so minute-based STT billing can be invisible;
update the ToSetting() implementation to include AudioMinutePrice (e.g., include
the float value when >0 or always include it) alongside the other pricing fields
so it appears in diagnostics/logs—locate the ToSetting() method for the type
that defines AudioMinutePrice and add the field name and its value to the
returned representation (or map/string) consistent with how other price fields
are rendered.
| <ReactMarkdown | ||
| remarkPlugins={[RemarkMath, RemarkGfm, RemarkBreaks]} | ||
| rehypePlugins={rehypePluginsBase} | ||
| urlTransform={(url) => url} |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's examine the MarkdownRenderer.jsx file to see the context around lines 475-478
wc -l web/src/components/common/markdown/MarkdownRenderer.jsxRepository: QuantumNous/new-api
Length of output: 123
🏁 Script executed:
# Read the relevant section of the file
head -520 web/src/components/common/markdown/MarkdownRenderer.jsx | tail -100Repository: QuantumNous/new-api
Length of output: 2762
🏁 Script executed:
# Check if there are custom element renderers defined
rg -n "components\s*=|renderers\s*=|img:|a:" web/src/components/common/markdown/MarkdownRenderer.jsx | head -20Repository: QuantumNous/new-api
Length of output: 166
🏁 Script executed:
# Check the react-markdown package.json to understand version and capabilities
cat web/package.json | grep -A 2 -B 2 "react-markdown"Repository: QuantumNous/new-api
Length of output: 229
🌐 Web query:
react-markdown default URL sanitization urlTransform behavior
💡 Result:
react-markdown provides default URL sanitization through the urlTransform option, which defaults to defaultUrlTransform. This function makes URLs safe by following GitHub's approach: it allows protocols http, https, irc, ircs, mailto, xmpp, and protocol-relative URLs (e.g., /something). Dangerous protocols like javascript:, vbscript:, and file: are blocked (replaced with empty string). Custom transformers can be provided but override the default filter. For complete safety with plugins, use rehype-sanitize. This is the primary built-in sanitization mechanism; HTML is escaped by default unless rehype-raw is used.
Citations:
- 1: https://github.com/remarkjs/react-markdown
- 2: https://github.com/remarkjs/react-markdown/blob/main/readme.md
- 3: https://github.com/remarkjs/react-markdown?tab=readme-ov-file
- 4: https://github.com/remarkjs/react-markdown/commits/8545ebd7
- 5: remarkjs/react-markdown@8545ebd
- 6: https://deepwiki.com/remarkjs/react-markdown/3.3-security-features
- 7: Does react-markdown sanitize the markdown? remarkjs/react-markdown#373
Remove the identity urlTransform to preserve react-markdown's default URL sanitization.
The urlTransform={(url) => url} function disables react-markdown's built-in protocol filtering, which normally blocks dangerous schemes like javascript:, vbscript:, and file:. This allows unsafe URLs to be passed directly to your custom <img> and <a> renderers, creating an XSS vulnerability.
Safer change
- urlTransform={(url) => url}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| <ReactMarkdown | |
| remarkPlugins={[RemarkMath, RemarkGfm, RemarkBreaks]} | |
| rehypePlugins={rehypePluginsBase} | |
| urlTransform={(url) => url} | |
| <ReactMarkdown | |
| remarkPlugins={[RemarkMath, RemarkGfm, RemarkBreaks]} | |
| rehypePlugins={rehypePluginsBase} | |
| /> |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@web/src/components/common/markdown/MarkdownRenderer.jsx` around lines 475 -
478, The ReactMarkdown usage in MarkdownRenderer.jsx currently sets
urlTransform={(url) => url}, which bypasses react-markdown's built-in
URL/protocol sanitization and can allow unsafe schemes; remove the urlTransform
prop from the <ReactMarkdown> invocation (or stop overriding it) so
react-markdown's default URL filtering is preserved and dangerous protocols
(javascript:, vbscript:, file:, etc.) are blocked before URLs reach custom
renderers like your image/link renderers.
| if (/\.(aac|mp3|opus|wav)$/.test(hrefPath)) { | ||
| const title = | ||
| typeof aProps.children === 'string' && | ||
| aProps.children !== href | ||
| ? aProps.children | ||
| : null; | ||
| return <AudioPlayer src={href} title={title} />; | ||
| } |
There was a problem hiding this comment.
Include .flac in playground audio detection.
relay/playground_audio.go can emit /pg/audio/<uuid>.flac, but this regex only matches aac|mp3|opus|wav. Those links will render as plain anchors instead of the new player.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@web/src/components/common/markdown/MarkdownRenderer.jsx` around lines 509 -
516, The audio-detection regex in MarkdownRenderer (the block using hrefPath,
aProps, href and returning <AudioPlayer />) misses .flac links, so update the
regex to include "flac" (e.g., add flac to the group and keep case-insensitive
matching) so /pg/audio/<uuid>.flac is detected and rendered by AudioPlayer
instead of a plain anchor.
| const isVideoUrl = | ||
| /\.(3gp|3g2|webm|ogv|mpeg|mp4|avi)$/.test(hrefPath) || | ||
| /\/videos?\/[^/].*\/content$/.test(hrefPath); | ||
| if (isVideoUrl) { | ||
| return ( | ||
| <video | ||
| controls | ||
| preload='auto' | ||
| style={{ width: '100%', maxWidth: '100%', margin: '12px 0' }} | ||
| > | ||
| <source src={href} /> | ||
| <source src={href} type='video/mp4' /> | ||
| </video> |
There was a problem hiding this comment.
Don't label every detected video as MP4.
The matcher now accepts webm, ogv, mpeg, avi, etc., but Line 527 still hardcodes type='video/mp4'. That can stop otherwise supported files from playing because the declared MIME no longer matches the source.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@web/src/components/common/markdown/MarkdownRenderer.jsx` around lines 517 -
528, The video renderer flags URLs via isVideoUrl (uses hrefPath and href) but
always sets <source type='video/mp4'>; change it to derive the MIME from the
file extension (e.g., .mp4 -> video/mp4, .webm -> video/webm, .ogv -> video/ogg,
.avi/.mpeg -> video/mpeg) or omit the type attribute when unknown, and use that
computed MIME in the <source> for the video element returned by the
MarkdownRenderer component so the declared MIME matches the actual file.
| // formatVideoResponse converts a raw video-generation object into markdown content. | ||
| // Returns null when the payload contains no useful information. | ||
| function formatVideoResponse(payload) { | ||
| if (!payload) return null; | ||
|
|
||
| // Try to find a video URL in common locations | ||
| const url = | ||
| payload.url || | ||
| payload.video_url || | ||
| payload.data?.[0]?.url || | ||
| payload.data?.[0]?.video_url; | ||
|
|
||
| const parts = []; | ||
| if (url) { | ||
| parts.push(`[Generated Video](${url})`); | ||
| } | ||
|
|
||
| const meta = []; | ||
| if (payload.model) meta.push(`模型: ${payload.model}`); | ||
| if (payload.size) meta.push(`分辨率: ${payload.size}`); | ||
| if (payload.seconds != null) meta.push(`时长: ${payload.seconds}s`); | ||
| if (payload.status) meta.push(`状态: ${payload.status}`); | ||
|
|
||
| if (meta.length > 0) { | ||
| parts.push(`**视频生成完成** — ${meta.join(' | ')}`); | ||
| } | ||
|
|
||
| return parts.length > 0 ? parts.join('\n\n') : null; | ||
| } |
There was a problem hiding this comment.
Hard-coded Chinese labels bypass i18n.
formatVideoResponse writes user-visible labels (模型, 分辨率, 时长, 状态, **视频生成完成**, [Generated Video]) directly into the assistant message instead of going through t('...'). Other strings in this file (e.g. t('请求发生错误: '), t('解析响应数据时发生错误')) are translated, so this regresses the i18n contract for video responses and won't switch language.
🛠 Suggested fix: move the helper inside the hook (or pass `t`) and translate the labels
-// formatVideoResponse converts a raw video-generation object into markdown content.
-// Returns null when the payload contains no useful information.
-function formatVideoResponse(payload) {
- if (!payload) return null;
-
- // Try to find a video URL in common locations
- const url =
- payload.url ||
- payload.video_url ||
- payload.data?.[0]?.url ||
- payload.data?.[0]?.video_url;
-
- const parts = [];
- if (url) {
- parts.push(`[Generated Video](${url})`);
- }
-
- const meta = [];
- if (payload.model) meta.push(`模型: ${payload.model}`);
- if (payload.size) meta.push(`分辨率: ${payload.size}`);
- if (payload.seconds != null) meta.push(`时长: ${payload.seconds}s`);
- if (payload.status) meta.push(`状态: ${payload.status}`);
-
- if (meta.length > 0) {
- parts.push(`**视频生成完成** — ${meta.join(' | ')}`);
- }
-
- return parts.length > 0 ? parts.join('\n\n') : null;
-}
+// formatVideoResponse converts a raw video-generation object into markdown content.
+// Returns null when the payload contains no useful information.
+function formatVideoResponse(payload, t) {
+ if (!payload) return null;
+
+ const url =
+ payload.url ||
+ payload.video_url ||
+ payload.data?.[0]?.url ||
+ payload.data?.[0]?.video_url;
+
+ const parts = [];
+ if (url) {
+ parts.push(`[${t('生成的视频')}](${url})`);
+ }
+
+ const meta = [];
+ if (payload.model) meta.push(`${t('模型')}: ${payload.model}`);
+ if (payload.size) meta.push(`${t('分辨率')}: ${payload.size}`);
+ if (payload.seconds != null) meta.push(`${t('时长')}: ${payload.seconds}s`);
+ if (payload.status) meta.push(`${t('状态')}: ${payload.status}`);
+
+ if (meta.length > 0) {
+ parts.push(`**${t('视频生成完成')}** — ${meta.join(' | ')}`);
+ }
+
+ return parts.length > 0 ? parts.join('\n\n') : null;
+}…and at the call site:
- const videoContent = formatVideoResponse(payload);
+ const videoContent = formatVideoResponse(payload, t);Then run bun run i18n:extract / bun run i18n:sync to register the new keys.
As per coding guidelines: web/src/**/*.{tsx,ts,jsx,js}: ... Use useTranslation() hook and call t('中文key') in components.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@web/src/hooks/playground/useApiRequest.jsx` around lines 35 - 63,
formatVideoResponse currently embeds hard-coded Chinese and English labels
(e.g., `模型`, `分辨率`, `时长`, `状态`, `**视频生成完成**`, `[Generated Video]`) which
bypasses i18n; fix by making the helper use the translation function: either
move `formatVideoResponse` inside the hook where `useTranslation()` is available
or change its signature to accept a `t` parameter, replace each literal label
with `t('key')` calls (create descriptive keys like video.model,
video.resolution, video.duration, video.status, video.generatedTitle,
video.generatedLink), update the call site(s) to pass `t` if needed, and run
`bun run i18n:extract` / `bun run i18n:sync` to register the new keys.
1 participant
Important
📝 变更描述 / Description
(简述：做了什么？为什么这样改能生效？请基于你对代码逻辑的理解来写，避免粘贴未经整理的内容)
🚀 变更类型 / Type of change
🔗 关联任务 / Related Issue
✅ 提交前检查项 / Checklist
Bug fix，我已提交或关联对应 Issue，且不会将设计取舍、预期不一致或理解偏差直接归类为 bug。📸 运行证明 / Proof of Work
(请在此粘贴截图、关键日志或测试报告，以证明变更生效)
Summary by CodeRabbit
Release Notes
New Features
Improvements
Chores