feat: add speech and transcription API support#176
feat: add speech and transcription API support#176Pratham-Mishra04 wants to merge 1 commit intomainfrom
Conversation
Summary by CodeRabbit
WalkthroughThis update introduces full support for speech synthesis and audio transcription features across the Bifrost backend, HTTP transport, logging, and UI. It adds new request types, provider interface methods, request/response schemas, HTTP endpoints, logging fields, and UI components for handling and displaying speech and transcription data, including both synchronous and streaming variants. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant HTTP Handler
participant Bifrost
participant Provider
Client->>HTTP Handler: POST /v1/audio/speech or /v1/audio/transcriptions
HTTP Handler->>Bifrost: SpeechRequest / TranscriptionRequest
Bifrost->>Provider: Speech/Transcription (sync or stream)
Provider-->>Bifrost: BifrostResponse or Stream
Bifrost-->>HTTP Handler: Response or Stream
HTTP Handler-->>Client: Audio bytes / transcription or SSE stream
Estimated code review effort5 (~2 hours+) Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🧪 Generate unit tests
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (25)
core/bifrost.go(9 hunks)core/providers/anthropic.go(3 hunks)core/providers/azure.go(1 hunks)core/providers/bedrock.go(2 hunks)core/providers/cohere.go(3 hunks)core/providers/groq.go(1 hunks)core/providers/mistral.go(1 hunks)core/providers/ollama.go(1 hunks)core/providers/openai.go(5 hunks)core/providers/sgl.go(1 hunks)core/providers/vertex.go(1 hunks)core/schemas/bifrost.go(4 hunks)core/schemas/provider.go(1 hunks)transports/bifrost-http/handlers/completions.go(7 hunks)transports/bifrost-http/lib/account.go(2 hunks)transports/bifrost-http/plugins/logging/main.go(11 hunks)transports/bifrost-http/plugins/logging/utils.go(16 hunks)transports/bifrost-http/plugins/telemetry/main.go(1 hunks)transports/go.mod(1 hunks)ui/components/logs/log-detail-sheet.tsx(2 hunks)ui/components/logs/ui/audio-player.tsx(1 hunks)ui/components/logs/ui/speech-view.tsx(1 hunks)ui/components/logs/ui/transcription-view.tsx(1 hunks)ui/lib/constants/logs.ts(2 hunks)ui/lib/types/logs.ts(2 hunks)
🧰 Additional context used
🧠 Learnings (18)
📓 Common learnings
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.875Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#84
File: transports/bifrost-http/main.go:2-2
Timestamp: 2025-06-15T16:05:13.489Z
Learning: For the Bifrost project, HTTP transport integration routers for new providers (like Mistral and Ollama) are implemented in separate PRs from the core provider support, following a focused PR strategy.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#54
File: core/schemas/provider.go:148-148
Timestamp: 2025-06-04T03:57:50.981Z
Learning: Breaking changes in the Bifrost codebase are managed by first merging and tagging core schema changes, then updating dependent code references in subsequent steps after the core version is released.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#63
File: transports/bifrost-http/integrations/openai/router.go:0-0
Timestamp: 2025-06-09T14:03:34.227Z
Learning: In the Bifrost HTTP transport layer (transports/bifrost-http/integrations/), request validation like checking for empty messages should be handled by the provider rather than at the transport layer. The transport layer should forward requests to Bifrost core/providers for validation.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#64
File: transports/bifrost-http/integrations/genai/types.go:470-515
Timestamp: 2025-06-10T11:06:57.716Z
Learning: Bifrost currently only supports images and does not support non-image blob types like audio, PDF, or other binary data formats.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#65
File: transports/bifrost-http/integrations/anthropic/types.go:105-113
Timestamp: 2025-06-09T14:56:14.951Z
Learning: Streaming functionality is not yet supported in the Bifrost system, so Stream flags are intentionally not propagated from integration requests to BifrostRequest.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#55
File: core/providers/anthropic.go:358-388
Timestamp: 2025-06-04T05:37:59.699Z
Learning: User Pratham-Mishra04 prefers not to extract small code duplications (around 2 lines) into helper functions, considering the overhead not worth it for such minor repetition.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#102
File: README.md:62-66
Timestamp: 2025-06-19T17:03:03.639Z
Learning: Pratham-Mishra04 prefers using the implicit 'latest' tag for the maximhq/bifrost Docker image rather than pinning to specific versions.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#143
File: core/mcp.go:155-196
Timestamp: 2025-07-08T15:33:47.698Z
Learning: Pratham-Mishra04 prefers not to add explanatory comments for obvious code patterns, such as the unlock/lock strategy around network I/O operations, considering them self-explanatory to experienced developers.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#141
File: core/bifrost.go:198-272
Timestamp: 2025-07-08T18:30:08.258Z
Learning: Pratham-Mishra04 follows a pattern of implementing core functionality first and deferring non-critical improvements (like race condition fixes, optimizations) to later PRs. This is a reasonable development approach that prioritizes getting the main feature working before addressing edge cases.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#150
File: transports/bifrost-http/lib/store.go:370-466
Timestamp: 2025-07-09T04:58:08.229Z
Learning: Pratham-Mishra04 prefers not to add logging or error handling for unreachable code paths in the Bifrost project. When provider types or similar entities are predefined in the system, defensive programming like logging in default cases is considered unnecessary overhead.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#138
File: docs/usage/go-package/mcp.md:408-412
Timestamp: 2025-07-01T12:40:08.576Z
Learning: Pratham-Mishra04 is okay with keeping bullet list formatting that uses colons after dashes in markdown documentation, even if it triggers linter warnings, preferring functionality over strict formatting rules.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#162
File: tests/core-providers/scenarios/chat_completion_stream.go:103-105
Timestamp: 2025-07-16T04:26:09.288Z
Learning: Pratham-Mishra04 prefers to keep test code simple when it serves its basic functional purpose. For tests that are meant to validate core functionality (like verifying streaming works), they consider hard-coded reasonable limits acceptable rather than making them configurable.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#148
File: transports/bifrost-http/lib/store.go:880-910
Timestamp: 2025-07-08T17:14:21.544Z
Learning: Pratham-Mishra04 prefers resilient system design where missing environment variables for MCP connections should not cause complete system failure. The system should continue processing other MCP connections even when some fail, maintaining partial functionality rather than implementing fail-fast behavior.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#138
File: transports/README.md:26-28
Timestamp: 2025-07-01T12:45:06.906Z
Learning: Pratham-Mishra04 prefers keeping documentation examples simple and concise, trusting users to handle production-specific considerations like version pinning themselves rather than cluttering examples with additional notes.
Learnt from: Pratham-Mishra04
PR: maximhq/bifrost#148
File: transports/bifrost-http/lib/store.go:1081-1098
Timestamp: 2025-07-08T17:16:50.811Z
Learning: Pratham-Mishra04 prefers practical redaction approaches over theoretical security improvements when the threat model is low-risk, such as admin-only interfaces in the Bifrost project. Fixed-length redaction is acceptable when only trusted administrators will see the redacted values.
transports/go.mod (13)
Learnt from: Pratham-Mishra04
PR: #103
File: .github/workflows/transport-dependency-update.yml:53-75
Timestamp: 2025-06-20T16:21:18.912Z
Learning: In the bifrost repository's transport dependency update workflow, when updating the core dependency to a new version using go get, the go.mod and go.sum files will always change in normal operation, making the safety check for changes more of a defensive programming practice rather than handling a common scenario.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/go.mod:38-38
Timestamp: 2025-06-16T03:54:48.005Z
Learning: The core-providers-test module in tests/core-providers/ is an internal testing module that will never be consumed as a dependency by external projects, so the replace directive pointing to ../../core is acceptable for local development and testing purposes.
Learnt from: Pratham-Mishra04
PR: #135
File: docs/core-package.md:105-116
Timestamp: 2025-06-27T17:07:39.462Z
Learning: In Go, when importing a package, the identifier used to access it is determined by the package declaration in the Go source files, not the directory name. For the Bifrost project, the core directory files declare package bifrost, so importing "github.com/maximhq/bifrost/core" automatically makes it accessible as bifrost.Init, bifrost.Ptr, etc., without needing an explicit alias.
Learnt from: Pratham-Mishra04
PR: #149
File: docs/contributing/README.md:22-27
Timestamp: 2025-07-08T18:12:13.590Z
Learning: In the Bifrost project, the tests directory structure has tests/core-providers/ and tests/transports-integrations/ as sibling directories. From tests/core-providers/, the correct relative path to reach tests/transports-integrations/ is ../transports-integrations/, not ../../tests/transports-integrations/.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/go.mod:3-4
Timestamp: 2025-06-16T04:27:53.538Z
Learning: In Go module files, go 1.24.1 (with patch version) can work fine in some setups, contrary to the general rule that go directives should only include major.minor versions.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/openai_test.go:1-2
Timestamp: 2025-06-16T04:29:53.409Z
Learning: In the Bifrost project, the user prefers to use package main for test files in the tests/core-providers directory rather than more descriptive package names like coreproviders_test.
Learnt from: Pratham-Mishra04
PR: #89
File: transports/bifrost-http/integrations/genai/types.go:22-56
Timestamp: 2025-06-16T14:50:46.859Z
Learning: In the Google GenAI integration at transports/bifrost-http/integrations/genai/types.go, the manual URL-safe base64 decoding implementation (converting - to +, _ to /, and adding padding) is required because base64.RawURLEncoding.DecodeString fails for the specific url encoded bytes format being handled.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #89
File: transports/bifrost-http/integrations/genai/types.go:151-155
Timestamp: 2025-06-16T14:45:48.563Z
Learning: In transports/bifrost-http/integrations/genai/types.go, when SystemInstruction has an empty role, the user prefers to let the downstream provider (Google GenAI) handle validation and return errors, rather than implementing validation in the bifrost layer. This represents a design preference for delegating validation to the appropriate service rather than duplicating validation logic in the proxy layer.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/types.go:140-146
Timestamp: 2025-06-10T13:51:52.859Z
Learning: In Bifrost core v1.0.9, ImageContent.Type was a pointer type (*string accessed via bifrost.Ptr), but in v1.0.10 it was changed to a value type (ImageContentType). When reviewing code, check the core version being used to determine the correct assignment pattern.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:168-170
Timestamp: 2025-06-15T14:24:49.882Z
Learning: In the Bifrost codebase, performance is prioritized over defensive copying for HTTP service operations. Specifically, shallow slice assignments in provider response handling are acceptable due to object pool reset patterns and JSON unmarshaling behavior that minimize practical data corruption risks.
transports/bifrost-http/plugins/telemetry/main.go (4)
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: connyay
PR: #92
File: core/providers/utils.go:60-64
Timestamp: 2025-06-17T14:04:21.104Z
Learning: The bifrost codebase uses string alias types (like type ContextKey string) for context keys consistently across multiple packages including plugins/maxim/main.go and transports/bifrost-http/tracking/plugin.go. This pattern should be maintained for consistency rather than switching to the canonical struct{} approach.
Learnt from: Pratham-Mishra04
PR: #79
File: core/bifrost.go:94-103
Timestamp: 2025-06-14T04:06:58.240Z
Learning: In core/bifrost.go, the count parameter in RunPostHooks method is intentionally kept separate from p.executedPreHooks to support circuit breaker plugins that may need to trigger PostHooks for only a subset of executed plugins when detecting failure conditions mid-execution.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
ui/lib/constants/logs.ts (1)
Learnt from: Pratham-Mishra04
PR: #169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
transports/bifrost-http/lib/account.go (11)
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/config/account.go:55-101
Timestamp: 2025-06-16T04:25:00.816Z
Learning: In the Bifrost test account implementation, the user prefers to let Bifrost itself handle missing API key errors rather than adding early validation in the GetKeysForProvider method.
Learnt from: connyay
PR: #92
File: core/providers/utils.go:60-64
Timestamp: 2025-06-17T14:04:21.104Z
Learning: The bifrost codebase uses string alias types (like type ContextKey string) for context keys consistently across multiple packages including plugins/maxim/main.go and transports/bifrost-http/tracking/plugin.go. This pattern should be maintained for consistency rather than switching to the canonical struct{} approach.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and handles the conversion internally, so nil checks are not needed when calling this function.
Learnt from: Pratham-Mishra04
PR: #83
File: core/schemas/bifrost.go:186-190
Timestamp: 2025-06-15T14:18:32.703Z
Learning: In core/schemas/bifrost.go, the ToolChoice UnmarshalJSON validation intentionally only checks for empty Type fields and lets providers handle validation of specific tool choice values. This architectural decision keeps schema validation focused on structure while allowing provider-specific semantic validation.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and always returns a pointer to a valid context. It starts with context.Background() and only adds values to it, so nil checks are not needed when calling this function.
Learnt from: Pratham-Mishra04
PR: #135
File: docs/core-package.md:105-116
Timestamp: 2025-06-27T17:07:39.462Z
Learning: In Go, when importing a package, the identifier used to access it is determined by the package declaration in the Go source files, not the directory name. For the Bifrost project, the core directory files declare package bifrost, so importing "github.com/maximhq/bifrost/core" automatically makes it accessible as bifrost.Init, bifrost.Ptr, etc., without needing an explicit alias.
Learnt from: Pratham-Mishra04
PR: #94
File: core/schemas/bifrost.go:20-23
Timestamp: 2025-06-18T15:16:23.127Z
Learning: In the Bifrost project, BifrostConfig struct is never marshaled/unmarshaled, so serialization tags (json, yaml) are not needed for its fields.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/openai_test.go:1-2
Timestamp: 2025-06-16T04:29:53.409Z
Learning: In the Bifrost project, the user prefers to use package main for test files in the tests/core-providers directory rather than more descriptive package names like coreproviders_test.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
transports/bifrost-http/plugins/logging/utils.go (8)
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:16-18
Timestamp: 2025-07-10T13:44:39.237Z
Learning: In the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go), Pratham-Mishra04 prefers to ignore JSON marshaling errors when storing log entries because logging is not critical for their use case and they are certain the marshaling operations won't fail.
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:378-399
Timestamp: 2025-07-10T13:44:14.518Z
Learning: In the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go), Pratham-Mishra04 prefers not to add error handling for JSON unmarshaling operations, considering logging not very critical and being confident that JSON marshalling won't fail in practice.
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:94-111
Timestamp: 2025-07-10T13:44:23.297Z
Learning: Pratham-Mishra04 prefers not to add error handling for JSON marshaling operations in the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go) because logging is not critical functionality and the structured schema data being marshaled is unlikely to fail. They accept the risk of not handling json.Marshal errors in logging contexts to keep the code simple.
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #149
File: transports/bifrost-http/plugins/logging/main.go:0-0
Timestamp: 2025-07-08T18:09:32.147Z
Learning: In the Bifrost HTTP transport logging plugin (transports/bifrost-http/plugins/logging/main.go), the DatabasePath is explicitly set from main.go via a config struct pointing to appDir/logs, so the default value in the NewLoggerPlugin constructor is only used for standalone plugin usage and doesn't affect the main application flow.
Learnt from: Pratham-Mishra04
PR: #144
File: transports/bifrost-http/plugins/logging/utils.go:509-514
Timestamp: 2025-07-08T16:48:25.386Z
Learning: In the Bifrost logging system (transports/bifrost-http/plugins/logging/utils.go), the Content field in message structures is a struct value type, not a pointer, so it will never be nil and doesn't require nil checks. However, ContentStr within Content is a pointer and should be checked for nil.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/schemas/provider.go (3)
Learnt from: Pratham-Mishra04
PR: #169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
core/providers/anthropic.go (18)
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:0-0
Timestamp: 2025-06-04T05:44:09.141Z
Learning: For the Anthropic provider in core/providers/anthropic.go, it's acceptable to pass potentially malformed messages with invalid roles because Anthropic's API will return suitable error responses for role issues, eliminating the need for additional validation logging.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:483-494
Timestamp: 2025-06-04T10:04:30.181Z
Learning: In the Anthropic provider (core/providers/anthropic.go), parallel tool calls support including the DisableParallelToolUse flag will be implemented in later commits as it's a relatively new feature that's not commonly used yet. The development approach prioritizes core functionality first.
Learnt from: Pratham-Mishra04
PR: #54
File: core/providers/anthropic.go:325-331
Timestamp: 2025-06-04T08:59:29.641Z
Learning: In the Anthropic provider, when handling system messages, prefer to let the API provider validate content (including empty strings) rather than pre-filtering on the client side. Users should receive errors directly from the provider if content is not supported.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:483-494
Timestamp: 2025-06-04T09:36:27.956Z
Learning: In the Anthropic provider (core/providers/anthropic.go), the user has confirmed through practical experience that the tool_choice parameter should always use an object format with a "type" field (e.g., {"type": "auto"}, {"type": "tool", "name": "function_name"}), even though the official documentation examples sometimes show "auto" as a direct string. The current implementation correctly handles all tool choice types with the object format.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/types.go:131-180
Timestamp: 2025-06-10T13:11:37.867Z
Learning: In Anthropic API integration for Bifrost, messages won't contain both image and tool_result content blocks in the same message, so defensive guards against multiple embedded message structs are unnecessary in the content processing loop.
Learnt from: Pratham-Mishra04
PR: #162
File: transports/bifrost-http/integrations/genai/types.go:0-0
Timestamp: 2025-07-16T07:13:29.496Z
Learning: Pratham-Mishra04 prefers to avoid redundant error handling across architectural layers in the Bifrost streaming implementation. When error handling (such as timeouts, context cancellation, and JSON marshaling failures) is already handled at the provider level, they prefer not to duplicate this logic at the transport integration layer to keep the code simple and avoid unnecessary complexity.
Learnt from: Pratham-Mishra04
PR: #169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
Learnt from: Pratham-Mishra04
PR: #83
File: core/schemas/bifrost.go:186-190
Timestamp: 2025-06-15T14:18:32.703Z
Learning: In core/schemas/bifrost.go, the ToolChoice UnmarshalJSON validation intentionally only checks for empty Type fields and lets providers handle validation of specific tool choice values. This architectural decision keeps schema validation focused on structure while allowing provider-specific semantic validation.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #84
File: core/providers/ollama.go:59-61
Timestamp: 2025-06-15T14:34:29.401Z
Learning: The CheckAndSetDefaults() method on *schemas.ProviderConfig in Go does not return any error - it has a void return type and only sets default values on the configuration struct.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:168-170
Timestamp: 2025-06-15T14:24:49.882Z
Learning: In the Bifrost codebase, performance is prioritized over defensive copying for HTTP service operations. Specifically, shallow slice assignments in provider response handling are acceptable due to object pool reset patterns and JSON unmarshaling behavior that minimize practical data corruption risks.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error with "Invalid request" message, making additional nil checks in individual router implementations redundant.
core/providers/azure.go (1)
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:96-101
Timestamp: 2025-06-15T13:50:41.418Z
Learning: In the Bifrost project, when a provider doesn't support a specific operation (like text completion), the IsBifrostError flag should be set to false to mark it as a provider-level error rather than a Bifrost framework error. This is intentional design for unsupported operations.
core/providers/mistral.go (1)
Learnt from: Pratham-Mishra04
PR: #84
File: transports/bifrost-http/main.go:2-2
Timestamp: 2025-06-15T16:05:13.489Z
Learning: For the Bifrost project, HTTP transport integration routers for new providers (like Mistral and Ollama) are implemented in separate PRs from the core provider support, following a focused PR strategy.
core/providers/cohere.go (15)
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/cohere.go:327-335
Timestamp: 2025-06-15T13:46:24.869Z
Learning: For Cohere v1 API in core/providers/cohere.go, the tool_choice parameter formatting uses uppercase strings for the "type" field (e.g., "AUTO", "TOOL") and follows a different structure than initially assumed. The current implementation with strings.ToUpper() for the type field is correct for the v1 API.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/cohere.go:327-335
Timestamp: 2025-06-15T13:46:24.869Z
Learning: Cohere v1 API tool_choice parameter accepts only uppercase string values: "REQUIRED" and "NONE". Unlike other providers, it doesn't use structured objects with "type" and "name" fields. The current implementation in core/providers/cohere.go correctly uses strings.ToUpper() to convert ToolChoiceStruct.Type to uppercase format as expected by the API.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error with "Invalid request" message, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:168-170
Timestamp: 2025-06-15T14:24:49.882Z
Learning: In the Bifrost codebase, performance is prioritized over defensive copying for HTTP service operations. Specifically, shallow slice assignments in provider response handling are acceptable due to object pool reset patterns and JSON unmarshaling behavior that minimize practical data corruption risks.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and handles the conversion internally, so nil checks are not needed when calling this function.
core/providers/bedrock.go (17)
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:96-101
Timestamp: 2025-06-15T13:50:41.418Z
Learning: In the Bifrost project, when a provider doesn't support a specific operation (like text completion), the IsBifrostError flag should be set to false to mark it as a provider-level error rather than a Bifrost framework error. This is intentional design for unsupported operations.
Learnt from: Pratham-Mishra04
PR: #74
File: core/providers/bedrock.go:909-916
Timestamp: 2025-06-11T15:05:45.355Z
Learning: In Bedrock chat responses (core/providers/bedrock.go), image content blocks are only possible in messages with the user role; assistant and tool messages will never contain image blocks.
Learnt from: Pratham-Mishra04
PR: #162
File: transports/bifrost-http/integrations/genai/types.go:0-0
Timestamp: 2025-07-16T07:13:29.496Z
Learning: Pratham-Mishra04 prefers to avoid redundant error handling across architectural layers in the Bifrost streaming implementation. When error handling (such as timeouts, context cancellation, and JSON marshaling failures) is already handled at the provider level, they prefer not to duplicate this logic at the transport integration layer to keep the code simple and avoid unnecessary complexity.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:0-0
Timestamp: 2025-06-04T05:44:09.141Z
Learning: For the Anthropic provider in core/providers/anthropic.go, it's acceptable to pass potentially malformed messages with invalid roles because Anthropic's API will return suitable error responses for role issues, eliminating the need for additional validation logging.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error with "Invalid request" message, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:168-170
Timestamp: 2025-06-15T14:24:49.882Z
Learning: In the Bifrost codebase, performance is prioritized over defensive copying for HTTP service operations. Specifically, shallow slice assignments in provider response handling are acceptable due to object pool reset patterns and JSON unmarshaling behavior that minimize practical data corruption risks.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and handles the conversion internally, so nil checks are not needed when calling this function.
transports/bifrost-http/handlers/completions.go (9)
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error with "Invalid request" message, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and handles the conversion internally, so nil checks are not needed when calling this function.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/types.go:105-113
Timestamp: 2025-06-09T14:56:14.951Z
Learning: Streaming functionality is not yet supported in the Bifrost system, so Stream flags are intentionally not propagated from integration requests to BifrostRequest.
ui/lib/types/logs.ts (1)
Learnt from: Pratham-Mishra04
PR: #64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
core/bifrost.go (12)
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #144
File: transports/bifrost-http/handlers/providers.go:45-49
Timestamp: 2025-07-08T16:50:27.699Z
Learning: In the Bifrost project, breaking API changes are acceptable when features are not yet public. This applies to scenarios like changing struct fields from pointer to non-pointer types in request/response structures for unreleased features.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:0-0
Timestamp: 2025-06-09T14:03:34.227Z
Learning: In the Bifrost HTTP transport layer (transports/bifrost-http/integrations/), request validation like checking for empty messages should be handled by the provider rather than at the transport layer. The transport layer should forward requests to Bifrost core/providers for validation.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:483-494
Timestamp: 2025-06-04T10:04:30.181Z
Learning: In the Anthropic provider (core/providers/anthropic.go), parallel tool calls support including the DisableParallelToolUse flag will be implemented in later commits as it's a relatively new feature that's not commonly used yet. The development approach prioritizes core functionality first.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/automatic_function_calling.go:22-22
Timestamp: 2025-06-16T04:55:11.886Z
Learning: In the Bifrost test suite (tests/core-providers), parallel tests using t.Parallel() are not being implemented currently. The team plans to add parallel test execution in future enhancements.
Learnt from: Pratham-Mishra04
PR: #79
File: core/bifrost.go:94-103
Timestamp: 2025-06-14T04:06:58.240Z
Learning: In core/bifrost.go, the count parameter in RunPostHooks method is intentionally kept separate from p.executedPreHooks to support circuit breaker plugins that may need to trigger PostHooks for only a subset of executed plugins when detecting failure conditions mid-execution.
Learnt from: Pratham-Mishra04
PR: #162
File: transports/bifrost-http/integrations/openai/types.go:340-344
Timestamp: 2025-07-16T05:44:50.544Z
Learning: In the Bifrost streaming implementation, context cancellation is properly propagated through the entire stack to the producer level. The producer goroutines always close their channels when done (including on context cancellation), so manual channel draining in consumers is not necessary to prevent goroutine leaks. The streaming architecture relies on proper context propagation rather than consumer-side cleanup.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
transports/bifrost-http/plugins/logging/main.go (7)
Learnt from: Pratham-Mishra04
PR: #149
File: transports/bifrost-http/plugins/logging/main.go:0-0
Timestamp: 2025-07-08T18:09:32.147Z
Learning: In the Bifrost HTTP transport logging plugin (transports/bifrost-http/plugins/logging/main.go), the DatabasePath is explicitly set from main.go via a config struct pointing to appDir/logs, so the default value in the NewLoggerPlugin constructor is only used for standalone plugin usage and doesn't affect the main application flow.
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:16-18
Timestamp: 2025-07-10T13:44:39.237Z
Learning: In the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go), Pratham-Mishra04 prefers to ignore JSON marshaling errors when storing log entries because logging is not critical for their use case and they are certain the marshaling operations won't fail.
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:378-399
Timestamp: 2025-07-10T13:44:14.518Z
Learning: In the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go), Pratham-Mishra04 prefers not to add error handling for JSON unmarshaling operations, considering logging not very critical and being confident that JSON marshalling won't fail in practice.
Learnt from: Pratham-Mishra04
PR: #152
File: transports/bifrost-http/plugins/logging/utils.go:94-111
Timestamp: 2025-07-10T13:44:23.297Z
Learning: Pratham-Mishra04 prefers not to add error handling for JSON marshaling operations in the Bifrost logging plugin (transports/bifrost-http/plugins/logging/utils.go) because logging is not critical functionality and the structured schema data being marshaled is unlikely to fail. They accept the risk of not handling json.Marshal errors in logging contexts to keep the code simple.
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #144
File: transports/bifrost-http/plugins/logging/utils.go:509-514
Timestamp: 2025-07-08T16:48:25.386Z
Learning: In the Bifrost logging system (transports/bifrost-http/plugins/logging/utils.go), the Content field in message structures is a struct value type, not a pointer, so it will never be nil and doesn't require nil checks. However, ContentStr within Content is a pointer and should be checked for nil.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/bedrock.go:443-468
Timestamp: 2025-06-04T09:32:15.826Z
Learning: In core/providers/bedrock.go, for tool call result messages (ModelChatMessageRoleTool), the Content field represents the actual tool call output. A tool result message should only be created when msg.Content is non-nil, as there's no semantic meaning to a tool result without output content.
core/providers/openai.go (18)
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/types.go:89-119
Timestamp: 2025-06-10T11:00:02.875Z
Learning: In the Bifrost OpenAI integration (transports/bifrost-http/integrations/openai/types.go), the convertOpenAIContent function currently only handles the last image URL when multiple images are present in a content array. The user Pratham-Mishra04 has acknowledged this limitation and indicated it's part of a larger architectural issue that will be addressed comprehensively later, rather than with piecemeal fixes.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:526-550
Timestamp: 2025-06-04T09:29:46.287Z
Learning: In core/providers/anthropic.go, the content field in formattedMessages is always of type []interface{} because it's explicitly constructed that way upstream in the prepareAnthropicChatRequest function. Defensive type casting for multiple types is not needed since the type is guaranteed by the construction logic.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/types.go:223-231
Timestamp: 2025-06-10T11:06:06.670Z
Learning: The OpenAI name field on messages cannot be preserved when converting to Bifrost format because the schemas.BifrostMessage struct in bifrost/core does not support a Name field. This is a known limitation of the Bifrost core schema design.
Learnt from: Pratham-Mishra04
PR: #89
File: transports/bifrost-http/integrations/genai/types.go:22-56
Timestamp: 2025-06-16T14:50:46.859Z
Learning: In the Google GenAI integration at transports/bifrost-http/integrations/genai/types.go, the manual URL-safe base64 decoding implementation (converting - to +, _ to /, and adding padding) is required because base64.RawURLEncoding.DecodeString fails for the specific url encoded bytes format being handled.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/complete_end_to_end.go:39-41
Timestamp: 2025-06-16T04:12:05.427Z
Learning: In the Bifrost system, error returns are of type BifrostError rather than the standard Go error interface. Therefore, use require.Nilf(t, err, ...) instead of require.NoError(t, err) when checking for errors in Bifrost function calls.
Learnt from: Pratham-Mishra04
PR: #55
File: core/providers/anthropic.go:0-0
Timestamp: 2025-06-04T05:44:09.141Z
Learning: For the Anthropic provider in core/providers/anthropic.go, it's acceptable to pass potentially malformed messages with invalid roles because Anthropic's API will return suitable error responses for role issues, eliminating the need for additional validation logging.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/anthropic/router.go:19-33
Timestamp: 2025-06-09T16:46:32.018Z
Learning: In the GenericRouter (transports/bifrost-http/integrations), ResponseFunc is not called if the BifrostResponse parameter is nil, providing built-in protection against nil response marshaling.
Learnt from: Pratham-Mishra04
PR: #149
File: transports/bifrost-http/main.go:259-261
Timestamp: 2025-07-08T18:20:24.086Z
Learning: In transports/bifrost-http/lib/store.go, the LoadFromConfig method already handles missing config files gracefully by initializing defaults and auto-detecting providers from environment variables. It only returns errors for actual parsing/processing failures, not missing files, making log.Fatalf appropriate in main.go since real errors should terminate the program.
Learnt from: Pratham-Mishra04
PR: #83
File: core/schemas/bifrost.go:186-190
Timestamp: 2025-06-15T14:18:32.703Z
Learning: In core/schemas/bifrost.go, the ToolChoice UnmarshalJSON validation intentionally only checks for empty Type fields and lets providers handle validation of specific tool choice values. This architectural decision keeps schema validation focused on structure while allowing provider-specific semantic validation.
Learnt from: Pratham-Mishra04
PR: #84
File: core/providers/ollama.go:59-61
Timestamp: 2025-06-15T14:34:29.401Z
Learning: The CheckAndSetDefaults() method on *schemas.ProviderConfig in Go does not return any error - it has a void return type and only sets default values on the configuration struct.
Learnt from: Pratham-Mishra04
PR: #83
File: core/providers/mistral.go:168-170
Timestamp: 2025-06-15T14:24:49.882Z
Learning: In the Bifrost codebase, performance is prioritized over defensive copying for HTTP service operations. Specifically, shallow slice assignments in provider response handling are acceptable due to object pool reset patterns and JSON unmarshaling behavior that minimize practical data corruption risks.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error, making additional nil checks in individual router implementations redundant.
Learnt from: Pratham-Mishra04
PR: #67
File: transports/bifrost-http/integrations/anthropic/router.go:26-34
Timestamp: 2025-06-10T11:19:29.604Z
Learning: The Generic router in transports/bifrost-http/integrations/utils.go already handles nil pointers from RequestConverter functions. When a RequestConverter returns a nil *schemas.BifrostRequest, the Generic router automatically returns an HTTP 400 error with "Invalid request" message, making additional nil checks in individual router implementations redundant.
core/schemas/bifrost.go (10)
Learnt from: Pratham-Mishra04
PR: #54
File: core/schemas/bifrost.go:46-49
Timestamp: 2025-06-04T09:22:18.123Z
Learning: In core/schemas/bifrost.go, the RequestInput struct uses ChatCompletionInput *[]BifrostMessage (pointer-to-slice) rather than []BifrostMessage to properly represent union type semantics. For text completion requests, ChatCompletionInput should be nil to indicate "no chat payload at all", while for chat completion requests it should be non-nil (even if empty slice). This distinguishes between different request types rather than just empty vs non-empty chat messages.
Learnt from: Pratham-Mishra04
PR: #65
File: transports/bifrost-http/integrations/utils.go:169-173
Timestamp: 2025-06-09T17:33:52.234Z
Learning: The ChatCompletionRequest method in the Bifrost client follows a contract where the result parameter will never be nil if the error parameter is nil. This means when error checking passes (err == nil), the result is guaranteed to be valid and can be safely used without additional nil checks.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/end_to_end_tool_calling.go:43-45
Timestamp: 2025-06-16T04:13:55.437Z
Learning: In the Bifrost codebase, errors returned from client methods like ChatCompletionRequest are of type BifrostError, not the standard error interface. For testing these errors, use require.Nilf instead of require.NoErrorf since BifrostError doesn't work with the standard error assertion methods.
Learnt from: Pratham-Mishra04
PR: #169
File: docs/usage/http-transport/openapi.json:1352-1358
Timestamp: 2025-07-17T08:56:59.907Z
Learning: In the Bifrost project, the fallback format has been updated from object structure {"provider": "...", "model": "..."} to a simpler string format "provider/model" (e.g., "anthropic/claude-3-sonnet-20240229"). The current OpenAPI schema correctly reflects this new format.
Learnt from: Pratham-Mishra04
PR: #80
File: plugins/maxim/main.go:249-264
Timestamp: 2025-06-14T06:17:54.426Z
Learning: In the BifrostError struct, the Error field is a struct (not a pointer), so accessing bifrostErr.Error.Message, bifrostErr.Error.Code, and bifrostErr.Error.Type is safe without nil checks on the Error field itself. The Code and Type fields are of type *string.
Learnt from: Pratham-Mishra04
PR: #64
File: transports/bifrost-http/integrations/genai/types.go:383-395
Timestamp: 2025-06-09T16:26:05.777Z
Learning: In the Bifrost schema (core/schemas/bifrost.go), only UserMessage and ToolMessage structs have ImageContent fields. AssistantMessage does not have an ImageContent field and cannot contain images. The schema design intentionally limits image content to user and tool messages only.
Learnt from: Pratham-Mishra04
PR: #81
File: tests/core-providers/scenarios/simple_chat.go:39-41
Timestamp: 2025-06-16T04:13:42.755Z
Learning: In the Bifrost codebase, errors returned from methods like ChatCompletionRequest are of type BifrostError (a custom error type) rather than the standard Go error interface. Therefore, require.Nilf should be used for error assertions instead of require.NoErrorf.
Learnt from: Pratham-Mishra04
PR: #83
File: core/schemas/bifrost.go:186-190
Timestamp: 2025-06-15T14:18:32.703Z
Learning: In core/schemas/bifrost.go, the ToolChoice UnmarshalJSON validation intentionally only checks for empty Type fields and lets providers handle validation of specific tool choice values. This architectural decision keeps schema validation focused on structure while allowing provider-specific semantic validation.
Learnt from: Pratham-Mishra04
PR: #63
File: transports/bifrost-http/integrations/openai/router.go:62-64
Timestamp: 2025-06-09T11:27:00.925Z
Learning: The lib.ConvertToBifrostContext function in the bifrost HTTP transport never returns nil and handles the conversion internally, so nil checks are not needed when calling this function.
Learnt from: Pratham-Mishra04
PR: #94
File: core/schemas/bifrost.go:20-23
Timestamp: 2025-06-18T15:16:23.127Z
Learning: In the Bifrost project, BifrostConfig struct is never marshaled/unmarshaled, so serialization tags (json, yaml) are not needed for its fields.
🧬 Code Graph Analysis (9)
transports/bifrost-http/plugins/telemetry/main.go (2)
core/schemas/bifrost.go (2)
SpeechInput(70-75)TranscriptionInput(131-136)ui/lib/types/logs.ts (2)
SpeechInput(9-14)TranscriptionInput(16-21)
ui/components/logs/ui/speech-view.tsx (2)
core/schemas/bifrost.go (2)
SpeechInput(70-75)BifrostSpeech(524-529)ui/lib/types/logs.ts (2)
SpeechInput(9-14)BifrostSpeech(69-72)
transports/bifrost-http/lib/account.go (3)
core/schemas/bifrost.go (1)
ModelProvider(37-37)ui/lib/types/config.ts (2)
ModelProvider(4-4)Key(21-28)core/schemas/account.go (1)
Key(8-15)
ui/components/logs/ui/audio-player.tsx (1)
ui/components/ui/button.tsx (1)
Button(71-71)
core/providers/anthropic.go (3)
core/schemas/bifrost.go (7)
BifrostError(624-632)ErrorField(635-642)SpeechInput(70-75)ModelParameters(162-179)BifrostResponse(372-385)BifrostStream(613-616)TranscriptionInput(131-136)core/schemas/account.go (1)
Key(8-15)core/schemas/provider.go (1)
PostHookRunner(144-144)
core/providers/mistral.go (5)
core/schemas/account.go (1)
Key(8-15)ui/lib/types/config.ts (1)
Key(21-28)core/schemas/bifrost.go (6)
SpeechInput(70-75)ModelParameters(162-179)BifrostResponse(372-385)BifrostError(624-632)BifrostStream(613-616)TranscriptionInput(131-136)ui/lib/types/logs.ts (4)
SpeechInput(9-14)ModelParameters(143-155)BifrostError(188-194)TranscriptionInput(16-21)core/schemas/provider.go (1)
PostHookRunner(144-144)
core/providers/cohere.go (3)
core/schemas/bifrost.go (6)
BifrostError(624-632)ErrorField(635-642)SpeechInput(70-75)BifrostResponse(372-385)BifrostStream(613-616)TranscriptionInput(131-136)core/schemas/account.go (1)
Key(8-15)core/schemas/provider.go (1)
PostHookRunner(144-144)
ui/lib/types/logs.ts (1)
core/schemas/bifrost.go (13)
VoiceConfig(82-85)SpeechInput(70-75)TranscriptionInput(131-136)AudioTokenDetails(403-406)AudioLLMUsage(396-401)TranscriptionWord(569-573)TranscriptionSegment(576-587)TranscriptionLogProb(562-566)TranscriptionUsage(590-597)BifrostSpeech(524-529)BifrostTranscribe(535-544)BifrostMessage(283-291)ModelParameters(162-179)
core/schemas/bifrost.go (1)
ui/lib/types/logs.ts (12)
BifrostMessage(100-108)SpeechInput(9-14)TranscriptionInput(16-21)VoiceConfig(4-7)BifrostSpeech(69-72)BifrostTranscribe(74-84)AudioTokenDetails(23-26)AudioLLMUsage(28-33)TranscriptionLogProb(54-58)TranscriptionUsage(60-67)TranscriptionWord(35-39)TranscriptionSegment(41-52)
🔇 Additional comments (68)
transports/go.mod (1)
17-18: LGTM! Standard practice for local development.The replace directive ensures the transports module uses the local development version of the core module, which is necessary for this PR's new speech and transcription functionality.
transports/bifrost-http/plugins/telemetry/main.go (1)
62-65: LGTM! Proper extension of telemetry tracking.The new conditions for
SpeechInputandTranscriptionInputfollow the established pattern and ensure proper categorization of audio request types for metrics collection.transports/bifrost-http/lib/account.go (2)
6-6: LGTM! Context import added for method signature update.
39-39: LGTM! Context parameter addition follows Go best practices.Adding the context parameter to
GetKeysForProvideraligns with Go best practices for context propagation and prepares the interface for future context-aware operations like cancellation and timeouts.ui/components/logs/log-detail-sheet.tsx (3)
13-14: LGTM! New UI components imported for audio features.
173-190: LGTM! Proper conditional rendering for audio sections.The speech and transcription sections are appropriately conditionally rendered based on the presence of relevant data, and the streaming status is properly passed to the components.
193-200: LGTM! Improved conditional rendering for conversation history.Making the conversation history section conditional on non-empty
input_historyprevents displaying empty sections and improves the user experience.ui/components/logs/ui/speech-view.tsx (3)
1-11: LGTM! Well-structured component interface.The component imports are clean, props are properly typed with optional fields, and the interface follows React TypeScript best practices.
16-52: LGTM! Comprehensive speech input rendering.The speech input section handles all input fields appropriately:
- Text to synthesize is displayed clearly
- Instructions are conditionally rendered
- Voice field handles both string and object types correctly
- Response format is conditionally shown
55-67: LGTM! Proper handling of streaming vs non-streaming output.The speech output section correctly:
- Shows streaming message when
isStreamingis true- Uses
AudioPlayercomponent for actual audio playback- Conditionally renders based on output availability
ui/lib/constants/logs.ts (1)
3-3: LGTM! Well-structured constants for audio features.The additions properly extend the existing constant definitions to support the new speech and transcription functionality. The color mappings follow consistent patterns, and the label update for chat completion chunks improves clarity.
Also applies to: 5-14, 32-32, 40-45, 51-56
core/providers/sgl.go (1)
240-254: LGTM! Consistent unsupported operation stubs.The new speech and transcription methods properly implement the provider interface by returning appropriate error messages for unsupported operations, following the established pattern used throughout the codebase.
ui/components/logs/ui/transcription-view.tsx (1)
14-183: Excellent comprehensive transcription UI component!This component provides a thorough and well-organized display of transcription data including:
- Audio playback integration via AudioPlayer
- Detailed metadata display (language, format, duration)
- Word-level timing with intuitive visual representation
- Segment analysis with probability metrics
- JSON log probabilities with proper code formatting
The conditional rendering, time formatting, and streaming state handling are all implemented correctly.
core/providers/vertex.go (1)
504-518: LGTM! Consistent implementation of unsupported operations.The new speech and transcription methods properly implement the provider interface by returning appropriate unsupported operation errors, maintaining consistency with the established patterns across all providers.
core/providers/mistral.go (1)
384-398: LGTM: Audio method stubs properly implemented.The four new methods correctly implement the Provider interface for speech and transcription capabilities, appropriately returning unsupported operation errors since Mistral doesn't offer these audio services.
core/schemas/provider.go (1)
158-165: LGTM: Provider interface properly extended for audio capabilities.The four new methods follow established patterns and provide a clean contract for speech and transcription functionality across all providers. The method signatures are consistent and well-documented.
core/providers/groq.go (1)
234-248: LGTM: Audio method stubs properly implemented.The implementation correctly adds the required audio method stubs, appropriately returning unsupported operation errors since Groq doesn't provide speech or transcription services.
core/providers/ollama.go (1)
240-254: LGTM: Audio method stubs properly implemented.The four new methods correctly implement the Provider interface for audio capabilities, appropriately returning unsupported operation errors since Ollama doesn't offer these services.
core/providers/cohere.go (3)
9-9: LGTM: Import added for enhanced error handling.The
ioimport is correctly added to support reading response bodies for better error reporting.
790-798: LGTM: Enhanced error handling with response body content.The improvement to read and include the HTTP response body in error messages provides valuable debugging information when Cohere API requests fail. This follows good error handling practices by capturing the actual error response from the API.
1027-1041: LGTM: Audio method stubs properly implemented.The four new methods correctly implement the Provider interface for speech and transcription capabilities, appropriately returning unsupported operation errors since Cohere doesn't currently offer these audio services.
core/providers/bedrock.go (2)
1421-1428: Good enhancement for error diagnostics.The addition of response body content to HTTP error messages will significantly improve debugging capabilities when Bedrock API returns error details in the response body.
1722-1736: LGTM: Consistent implementation of unsupported audio operations.The four new audio-related methods properly implement the Provider interface by returning appropriate unsupported operation errors. This follows the established pattern across other providers that don't support speech and transcription functionality.
core/providers/anthropic.go (2)
880-888: Good error handling improvement!Including the response body in the error message when the HTTP status is not OK provides valuable debugging information. This enhancement aligns with similar improvements made in other providers (Bedrock, Cohere) as mentioned in the AI summary.
1297-1311: LGTM! Consistent unsupported operation stubs.The four new methods properly implement the extended Provider interface by returning unsupported operation errors. This is consistent with the implementation pattern used across other providers (Azure, Bedrock, Cohere, etc.) and aligns with the PR objectives where only the OpenAI provider has full speech/transcription support.
core/providers/azure.go (1)
585-599: Consistent stub implementations for audio features.The four new methods correctly implement the extended Provider interface with unsupported operation errors. This maintains consistency with other non-OpenAI providers in the codebase.
transports/bifrost-http/plugins/logging/utils.go (6)
19-20: Proper extension for speech and transcription input logging.The additions correctly serialize and store speech and transcription input data, maintaining consistency with the existing field handling pattern.
Also applies to: 29-31, 37-37
116-128: Consistent handling of speech and transcription outputs.The implementation follows the established pattern for handling optional update fields, properly serializing the output data when present.
155-155: Improved streaming control with explicit final chunk flag.The addition of the
isFinalChunkparameter is a good design improvement that makes the streaming logic more explicit and reliable for determining when to calculate latency and update status.
401-412: Comprehensive content indexing for speech and transcription.The content summary functions have been properly extended to include speech input text, instructions, transcription prompts, and transcription output text. This ensures the new audio-related content is searchable.
Also applies to: 539-555
888-891: Correct object type identification for new audio features.The
determineObjectTypefunction correctly identifies the new "audio.speech" and "audio.transcription" types, which aligns with the constants defined in the broader system.
423-424: Complete integration of speech/transcription fields in data retrieval.The SQL queries and JSON deserialization have been properly updated to handle the new speech and transcription fields throughout the retrieval and search operations. The implementation maintains consistency with the existing field handling patterns.
Also applies to: 431-432, 442-443, 498-509, 637-638, 647-648, 700-711, 748-749
core/providers/openai.go (6)
184-184: Good refactoring of error handling.Consolidating OpenAI error parsing into a dedicated helper function improves code maintainability and reduces duplication.
Also applies to: 340-340
532-540: Enhanced error reporting with response body content.Including the actual response body in streaming errors will help with debugging provider issues.
708-787: Well-implemented Speech method.The implementation correctly handles speech synthesis requests with proper defaults, error handling, and response formatting.
789-1008: Comprehensive streaming speech implementation.The method correctly implements SSE streaming for speech synthesis with proper error handling and response parsing.
1010-1186: Well-structured transcription implementation.The method correctly handles multipart file uploads with all required and optional fields. The response parsing and error handling are appropriate.
1489-1506: Clean error parsing helper implementation.The helper function properly consolidates OpenAI error parsing logic and correctly maps all error fields.
transports/bifrost-http/handlers/completions.go (6)
42-49: Appropriate struct extensions for speech functionality.The new fields correctly support speech synthesis requests with proper JSON tags and types.
54-67: Well-structured API endpoints and constants.The new completion types and routes follow RESTful conventions and maintain consistency with the existing API structure.
71-87: Clean handler method organization.The refactoring to use a generic
handleRequestmethod reduces code duplication, while keeping transcription separate due to its multipart form requirements.
91-226: Comprehensive speech request handling.The implementation correctly validates speech inputs, handles both streaming and non-streaming modes, and properly formats audio responses with appropriate HTTP headers.
334-434: Robust multipart form handling for transcription.The implementation correctly handles file uploads with proper validation, resource cleanup, and optional parameter extraction.
436-486: Consistent streaming transcription implementation.The method follows established streaming patterns with appropriate nil checks and SSE formatting.
core/bifrost.go (5)
28-31: LGTM! Request type constants properly defined.The new speech and transcription request types follow the established naming convention and are correctly typed.
37-43: Map correctly updated with new handlers.The messageExecutors map properly includes the new non-streaming speech and transcription handlers, and the comment clarification is helpful.
1156-1158: Correctly excludes audio requests from MCP tool injection.The condition properly excludes speech and transcription requests from MCP tool injection, which is appropriate since these audio-related operations don't utilize tools.
Also applies to: 1244-1246
1508-1532: Handler functions properly implemented.The new handleSpeech and handleTranscription functions correctly validate inputs and delegate to provider methods, following the established pattern.
1548-1572: Streaming handlers correctly implemented.The new handleSpeechStream and handleTranscriptionStream functions properly validate inputs and delegate to provider streaming methods with the postHookRunner, maintaining consistency with existing patterns.
ui/lib/types/logs.ts (2)
214-214: LogEntry interface correctly extended.The interface properly adds the new audio-related fields as optional properties, and the object field documentation is updated to include the new audio types.
Also applies to: 221-224
9-14: Voice field type matches backend schemaThe Go
SpeechVoiceInputstruct defines:
Voice *stringMultiVoiceConfig []VoiceConfigYour TypeScript union
voice: string | VoiceConfig[]accurately represents the two backend options. No changes needed.transports/bifrost-http/plugins/logging/main.go (8)
44-53: UpdateLogData struct properly extended.The struct correctly adds fields for speech and transcription outputs with appropriate pointer types and clear comments distinguishing non-streaming responses.
77-109: Data structures correctly extended for audio support.Both InitialLogData and LogEntry structs are properly updated with audio-related fields, maintaining consistency with pointer types for optional data.
348-377: Excellent refactoring of migration logic.The migration logic is now more maintainable with the loop-based approach. All necessary columns including the new audio-related ones are properly included.
508-515: PreHook correctly captures audio inputs.The PreHook properly extracts speech and transcription inputs from the request and includes them in both the initial log data and the callback notification.
Also applies to: 538-551
626-660: Comprehensive streaming token usage extraction.The implementation properly extracts token usage from both speech and transcription streaming responses, with correct handling of nullable fields and audio token details mapping.
705-748: Non-streaming audio output handling implemented correctly.The implementation properly captures speech and transcription outputs with comprehensive token usage extraction, maintaining consistency with the streaming handling.
753-757: Streaming detection properly extended for audio.The implementation correctly identifies audio streaming responses and detects final chunks based on usage data, which is appropriate for audio streams.
Also applies to: 798-821
451-452: Pool reset logic correctly updated.The UpdateLogData reset properly clears the new audio output fields to prevent memory leaks.
core/schemas/bifrost.go (9)
54-62: LGTM! Request input types properly extended for audio capabilities.The addition of
SpeechInputandTranscriptionInputfields follows the established pattern for union types inRequestInput, maintaining consistency with existing fields.
69-86: Well-structured speech input types with flexible voice configuration.The design properly supports both single voice string and multi-voice configurations through the
SpeechVoiceInputunion type, providing flexibility for different use cases.
87-103: Correct implementation of union type marshalling.The
MarshalJSONmethod properly validates mutual exclusivity and handles all cases consistently with other union types in the codebase.
131-136: Proper structure for transcription input.The
TranscriptionInputstruct correctly uses[]bytefor file data (which will be base64 encoded in JSON) and follows the established pattern for optional fields.
376-378: Response structure properly extended for audio capabilities.The addition of
SpeechandTranscribefields with appropriate pointer types and the clarification of theEmbeddingfield mapping enhance the response structure while maintaining backward compatibility.
396-406: Well-designed audio usage tracking structures.The
AudioLLMUsageandAudioTokenDetailsstructs appropriately model audio-specific token usage with clear separation between text and audio tokens.
524-532: Speech response structure supports both streaming and non-streaming modes.The
BifrostSpeechstruct properly uses the embedded struct pattern to handle different response modes while maintaining a clean API.
534-559: Excellent separation of streaming and non-streaming transcription fields.The use of embedded structs to separate mode-specific fields while keeping common fields at the top level creates a clean and maintainable API.
561-587: Comprehensive transcription metadata structures.These structs provide rich transcription details including timing, probabilities, and quality metrics, enabling detailed analysis of transcription results.
| func (bifrost *Bifrost) SpeechRequest(ctx context.Context, req *schemas.BifrostRequest) (*schemas.BifrostResponse, *schemas.BifrostError) { | ||
| if err := validateRequest(req); err != nil { | ||
| err.Provider = req.Provider | ||
| return nil, err | ||
| } | ||
|
|
||
| // Try the primary provider first | ||
| primaryResult, primaryErr := bifrost.tryRequest(req, ctx, SpeechRequest) | ||
| if primaryErr == nil { | ||
| return primaryResult, nil | ||
| } | ||
|
|
||
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // Check if this is a short-circuit error that doesn't allow fallbacks | ||
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | ||
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // If primary provider failed and we have fallbacks, try them in order | ||
| if len(req.Fallbacks) > 0 { | ||
| for _, fallback := range req.Fallbacks { | ||
| // Check if we have config for this fallback provider | ||
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | ||
| if err != nil { | ||
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | ||
| continue | ||
| } | ||
|
|
||
| // Create a new request with the fallback provider and model | ||
| fallbackReq := *req | ||
| fallbackReq.Provider = fallback.Provider | ||
| fallbackReq.Model = fallback.Model | ||
|
|
||
| // Try the fallback provider | ||
| result, fallbackErr := bifrost.tryRequest(&fallbackReq, ctx, SpeechRequest) | ||
| if fallbackErr == nil { | ||
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | ||
| return result, nil | ||
| } | ||
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | ||
| fallbackErr.Provider = fallback.Provider | ||
| return nil, fallbackErr | ||
| } | ||
|
|
||
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | ||
| } | ||
| } | ||
|
|
||
| primaryErr.Provider = req.Provider | ||
|
|
||
| // All providers failed, return the original error | ||
| return nil, primaryErr | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add input validation for consistency.
The SpeechRequest method should validate that req.Input.SpeechInput is not nil, similar to how EmbeddingRequest validates its input at line 399-401. This would provide early validation and consistent error handling.
Add this validation after line 461:
if err := validateRequest(req); err != nil {
err.Provider = req.Provider
return nil, err
}
+
+ if req.Input.SpeechInput == nil {
+ return nil, newBifrostErrorFromMsg("speech_input is required")
+ }🤖 Prompt for AI Agents
In core/bifrost.go around lines 457 to 515, the SpeechRequest method lacks
validation to check if req.Input.SpeechInput is nil, which is inconsistent with
EmbeddingRequest's input validation. Add a validation step immediately after
line 461 to verify that req.Input.SpeechInput is not nil, and if it is, return
an appropriate error indicating invalid input. This ensures early validation and
consistent error handling across request types.
| func (bifrost *Bifrost) SpeechStreamRequest(ctx context.Context, req *schemas.BifrostRequest) (chan *schemas.BifrostStream, *schemas.BifrostError) { | ||
| if err := validateRequest(req); err != nil { | ||
| err.Provider = req.Provider | ||
| return nil, err | ||
| } | ||
|
|
||
| // Try the primary provider first | ||
| primaryResult, primaryErr := bifrost.tryStreamRequest(req, ctx, SpeechStreamRequest) | ||
| if primaryErr == nil { | ||
| return primaryResult, nil | ||
| } | ||
|
|
||
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // Check if this is a short-circuit error that doesn't allow fallbacks | ||
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | ||
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // If primary provider failed and we have fallbacks, try them in order | ||
| // This includes both regular provider errors and plugin short-circuit errors with AllowFallbacks=true/nil | ||
| if len(req.Fallbacks) > 0 { | ||
| for _, fallback := range req.Fallbacks { | ||
| // Check if we have config for this fallback provider | ||
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | ||
| if err != nil { | ||
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | ||
| continue | ||
| } | ||
|
|
||
| // Create a new request with the fallback provider and model | ||
| fallbackReq := *req | ||
| fallbackReq.Provider = fallback.Provider | ||
| fallbackReq.Model = fallback.Model | ||
|
|
||
| // Try the fallback provider | ||
| result, fallbackErr := bifrost.tryStreamRequest(&fallbackReq, ctx, SpeechStreamRequest) | ||
| if fallbackErr == nil { | ||
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | ||
| return result, nil | ||
| } | ||
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | ||
| fallbackErr.Provider = fallback.Provider | ||
| return nil, fallbackErr | ||
| } | ||
|
|
||
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | ||
| } | ||
| } | ||
|
|
||
| primaryErr.Provider = req.Provider | ||
|
|
||
| // All providers failed, return the original error | ||
| return nil, primaryErr | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add input validation for streaming speech requests.
For consistency with non-streaming speech requests and other request types, validate that req.Input.SpeechInput is not nil.
Add validation after line 521:
if err := validateRequest(req); err != nil {
err.Provider = req.Provider
return nil, err
}
+
+ if req.Input.SpeechInput == nil {
+ return nil, newBifrostErrorFromMsg("speech_input is required")
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func (bifrost *Bifrost) SpeechStreamRequest(ctx context.Context, req *schemas.BifrostRequest) (chan *schemas.BifrostStream, *schemas.BifrostError) { | |
| if err := validateRequest(req); err != nil { | |
| err.Provider = req.Provider | |
| return nil, err | |
| } | |
| // Try the primary provider first | |
| primaryResult, primaryErr := bifrost.tryStreamRequest(req, ctx, SpeechStreamRequest) | |
| if primaryErr == nil { | |
| return primaryResult, nil | |
| } | |
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // Check if this is a short-circuit error that doesn't allow fallbacks | |
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | |
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // If primary provider failed and we have fallbacks, try them in order | |
| // This includes both regular provider errors and plugin short-circuit errors with AllowFallbacks=true/nil | |
| if len(req.Fallbacks) > 0 { | |
| for _, fallback := range req.Fallbacks { | |
| // Check if we have config for this fallback provider | |
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | |
| if err != nil { | |
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | |
| continue | |
| } | |
| // Create a new request with the fallback provider and model | |
| fallbackReq := *req | |
| fallbackReq.Provider = fallback.Provider | |
| fallbackReq.Model = fallback.Model | |
| // Try the fallback provider | |
| result, fallbackErr := bifrost.tryStreamRequest(&fallbackReq, ctx, SpeechStreamRequest) | |
| if fallbackErr == nil { | |
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | |
| return result, nil | |
| } | |
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | |
| fallbackErr.Provider = fallback.Provider | |
| return nil, fallbackErr | |
| } | |
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | |
| } | |
| } | |
| primaryErr.Provider = req.Provider | |
| // All providers failed, return the original error | |
| return nil, primaryErr | |
| } | |
| func (bifrost *Bifrost) SpeechStreamRequest(ctx context.Context, req *schemas.BifrostRequest) (chan *schemas.BifrostStream, *schemas.BifrostError) { | |
| if err := validateRequest(req); err != nil { | |
| err.Provider = req.Provider | |
| return nil, err | |
| } | |
| if req.Input.SpeechInput == nil { | |
| return nil, newBifrostErrorFromMsg("speech_input is required") | |
| } | |
| // Try the primary provider first | |
| primaryResult, primaryErr := bifrost.tryStreamRequest(req, ctx, SpeechStreamRequest) | |
| if primaryErr == nil { | |
| return primaryResult, nil | |
| } | |
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // Check if this is a short-circuit error that doesn't allow fallbacks | |
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | |
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // If primary provider failed and we have fallbacks, try them in order | |
| // This includes both regular provider errors and plugin short-circuit errors with AllowFallbacks=true/nil | |
| if len(req.Fallbacks) > 0 { | |
| for _, fallback := range req.Fallbacks { | |
| // Check if we have config for this fallback provider | |
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | |
| if err != nil { | |
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | |
| continue | |
| } | |
| // Create a new request with the fallback provider and model | |
| fallbackReq := *req | |
| fallbackReq.Provider = fallback.Provider | |
| fallbackReq.Model = fallback.Model | |
| // Try the fallback provider | |
| result, fallbackErr := bifrost.tryStreamRequest(&fallbackReq, ctx, SpeechStreamRequest) | |
| if fallbackErr == nil { | |
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | |
| return result, nil | |
| } | |
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | |
| fallbackErr.Provider = fallback.Provider | |
| return nil, fallbackErr | |
| } | |
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | |
| } | |
| } | |
| primaryErr.Provider = req.Provider | |
| // All providers failed, return the original error | |
| return nil, primaryErr | |
| } |
🤖 Prompt for AI Agents
In core/bifrost.go around lines 517 to 576, the SpeechStreamRequest function
lacks validation to ensure req.Input.SpeechInput is not nil, which is necessary
for consistency with other request types. Add a validation step after line 521
to check if req.Input.SpeechInput is nil, and if so, return an appropriate error
indicating invalid input before proceeding with the request processing.
| func (bifrost *Bifrost) TranscriptionRequest(ctx context.Context, req *schemas.BifrostRequest) (*schemas.BifrostResponse, *schemas.BifrostError) { | ||
| if err := validateRequest(req); err != nil { | ||
| err.Provider = req.Provider | ||
| return nil, err | ||
| } | ||
|
|
||
| // Try the primary provider first | ||
| primaryResult, primaryErr := bifrost.tryRequest(req, ctx, TranscriptionRequest) | ||
| if primaryErr == nil { | ||
| return primaryResult, nil | ||
| } | ||
|
|
||
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // Check if this is a short-circuit error that doesn't allow fallbacks | ||
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | ||
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // If primary provider failed and we have fallbacks, try them in order | ||
| if len(req.Fallbacks) > 0 { | ||
| for _, fallback := range req.Fallbacks { | ||
| // Check if we have config for this fallback provider | ||
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | ||
| if err != nil { | ||
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | ||
| continue | ||
| } | ||
|
|
||
| // Create a new request with the fallback provider and model | ||
| fallbackReq := *req | ||
| fallbackReq.Provider = fallback.Provider | ||
| fallbackReq.Model = fallback.Model | ||
|
|
||
| // Try the fallback provider | ||
| result, fallbackErr := bifrost.tryRequest(&fallbackReq, ctx, TranscriptionRequest) | ||
| if fallbackErr == nil { | ||
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | ||
| return result, nil | ||
| } | ||
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | ||
| fallbackErr.Provider = fallback.Provider | ||
| return nil, fallbackErr | ||
| } | ||
|
|
||
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | ||
| } | ||
| } | ||
|
|
||
| primaryErr.Provider = req.Provider | ||
|
|
||
| // All providers failed, return the original error | ||
| return nil, primaryErr | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add input validation for transcription requests.
Validate that req.Input.TranscriptionInput is not nil for consistency with other request types.
Add validation after line 582:
if err := validateRequest(req); err != nil {
err.Provider = req.Provider
return nil, err
}
+
+ if req.Input.TranscriptionInput == nil {
+ return nil, newBifrostErrorFromMsg("transcription_input is required")
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func (bifrost *Bifrost) TranscriptionRequest(ctx context.Context, req *schemas.BifrostRequest) (*schemas.BifrostResponse, *schemas.BifrostError) { | |
| if err := validateRequest(req); err != nil { | |
| err.Provider = req.Provider | |
| return nil, err | |
| } | |
| // Try the primary provider first | |
| primaryResult, primaryErr := bifrost.tryRequest(req, ctx, TranscriptionRequest) | |
| if primaryErr == nil { | |
| return primaryResult, nil | |
| } | |
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // Check if this is a short-circuit error that doesn't allow fallbacks | |
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | |
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // If primary provider failed and we have fallbacks, try them in order | |
| if len(req.Fallbacks) > 0 { | |
| for _, fallback := range req.Fallbacks { | |
| // Check if we have config for this fallback provider | |
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | |
| if err != nil { | |
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | |
| continue | |
| } | |
| // Create a new request with the fallback provider and model | |
| fallbackReq := *req | |
| fallbackReq.Provider = fallback.Provider | |
| fallbackReq.Model = fallback.Model | |
| // Try the fallback provider | |
| result, fallbackErr := bifrost.tryRequest(&fallbackReq, ctx, TranscriptionRequest) | |
| if fallbackErr == nil { | |
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | |
| return result, nil | |
| } | |
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | |
| fallbackErr.Provider = fallback.Provider | |
| return nil, fallbackErr | |
| } | |
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | |
| } | |
| } | |
| primaryErr.Provider = req.Provider | |
| // All providers failed, return the original error | |
| return nil, primaryErr | |
| } | |
| func (bifrost *Bifrost) TranscriptionRequest(ctx context.Context, req *schemas.BifrostRequest) (*schemas.BifrostResponse, *schemas.BifrostError) { | |
| if err := validateRequest(req); err != nil { | |
| err.Provider = req.Provider | |
| return nil, err | |
| } | |
| if req.Input.TranscriptionInput == nil { | |
| return nil, newBifrostErrorFromMsg("transcription_input is required") | |
| } | |
| // Try the primary provider first | |
| primaryResult, primaryErr := bifrost.tryRequest(req, ctx, TranscriptionRequest) | |
| if primaryErr == nil { | |
| return primaryResult, nil | |
| } | |
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // Check if this is a short-circuit error that doesn't allow fallbacks | |
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | |
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | |
| primaryErr.Provider = req.Provider | |
| return nil, primaryErr | |
| } | |
| // If primary provider failed and we have fallbacks, try them in order | |
| if len(req.Fallbacks) > 0 { | |
| for _, fallback := range req.Fallbacks { | |
| // Check if we have config for this fallback provider | |
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | |
| if err != nil { | |
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | |
| continue | |
| } | |
| // Create a new request with the fallback provider and model | |
| fallbackReq := *req | |
| fallbackReq.Provider = fallback.Provider | |
| fallbackReq.Model = fallback.Model | |
| // Try the fallback provider | |
| result, fallbackErr := bifrost.tryRequest(&fallbackReq, ctx, TranscriptionRequest) | |
| if fallbackErr == nil { | |
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | |
| return result, nil | |
| } | |
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | |
| fallbackErr.Provider = fallback.Provider | |
| return nil, fallbackErr | |
| } | |
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | |
| } | |
| } | |
| primaryErr.Provider = req.Provider | |
| // All providers failed, return the original error | |
| return nil, primaryErr | |
| } |
🤖 Prompt for AI Agents
In core/bifrost.go around lines 578 to 636, add input validation to ensure that
req.Input.TranscriptionInput is not nil right after the existing validateRequest
call at line 582. If it is nil, return an appropriate error indicating the
missing transcription input to maintain consistency with other request types and
prevent nil pointer dereferences.
| func (bifrost *Bifrost) TranscriptionStreamRequest(ctx context.Context, req *schemas.BifrostRequest) (chan *schemas.BifrostStream, *schemas.BifrostError) { | ||
| if err := validateRequest(req); err != nil { | ||
| err.Provider = req.Provider | ||
| return nil, err | ||
| } | ||
|
|
||
| // Try the primary provider first | ||
| primaryResult, primaryErr := bifrost.tryStreamRequest(req, ctx, TranscriptionStreamRequest) | ||
| if primaryErr == nil { | ||
| return primaryResult, nil | ||
| } | ||
|
|
||
| if primaryErr.Error.Type != nil && *primaryErr.Error.Type == schemas.RequestCancelled { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // Check if this is a short-circuit error that doesn't allow fallbacks | ||
| // Note: AllowFallbacks = nil is treated as true (allow fallbacks by default) | ||
| if primaryErr.AllowFallbacks != nil && !*primaryErr.AllowFallbacks { | ||
| primaryErr.Provider = req.Provider | ||
| return nil, primaryErr | ||
| } | ||
|
|
||
| // If primary provider failed and we have fallbacks, try them in order | ||
| // This includes both regular provider errors and plugin short-circuit errors with AllowFallbacks=true/nil | ||
| if len(req.Fallbacks) > 0 { | ||
| for _, fallback := range req.Fallbacks { | ||
| // Check if we have config for this fallback provider | ||
| _, err := bifrost.account.GetConfigForProvider(fallback.Provider) | ||
| if err != nil { | ||
| bifrost.logger.Warn(fmt.Sprintf("Config not found for provider %s, skipping fallback: %v", fallback.Provider, err)) | ||
| continue | ||
| } | ||
|
|
||
| // Create a new request with the fallback provider and model | ||
| fallbackReq := *req | ||
| fallbackReq.Provider = fallback.Provider | ||
| fallbackReq.Model = fallback.Model | ||
|
|
||
| // Try the fallback provider | ||
| result, fallbackErr := bifrost.tryStreamRequest(&fallbackReq, ctx, TranscriptionStreamRequest) | ||
| if fallbackErr == nil { | ||
| bifrost.logger.Info(fmt.Sprintf("Successfully used fallback provider %s with model %s", fallback.Provider, fallback.Model)) | ||
| return result, nil | ||
| } | ||
| if fallbackErr.Error.Type != nil && *fallbackErr.Error.Type == schemas.RequestCancelled { | ||
| fallbackErr.Provider = fallback.Provider | ||
| return nil, fallbackErr | ||
| } | ||
|
|
||
| bifrost.logger.Warn(fmt.Sprintf("Fallback provider %s failed: %s", fallback.Provider, fallbackErr.Error.Message)) | ||
| } | ||
| } | ||
|
|
||
| primaryErr.Provider = req.Provider | ||
|
|
||
| // All providers failed, return the original error | ||
| return nil, primaryErr | ||
| } |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add input validation for streaming transcription requests.
Validate that req.Input.TranscriptionInput is not nil for consistency.
Add validation after line 642:
if err := validateRequest(req); err != nil {
err.Provider = req.Provider
return nil, err
}
+
+ if req.Input.TranscriptionInput == nil {
+ return nil, newBifrostErrorFromMsg("transcription_input is required")
+ }🤖 Prompt for AI Agents
In core/bifrost.go around lines 638 to 697, add input validation to ensure that
req.Input.TranscriptionInput is not nil immediately after the existing
validateRequest call at line 642. If it is nil, create and return a BifrostError
indicating invalid input, setting the Provider field accordingly. This ensures
consistency and prevents downstream errors during streaming transcription
requests.
| } else if req.Type == TranscriptionStreamRequest { | ||
| pipeline := bifrost.getPluginPipeline() | ||
| defer bifrost.releasePluginPipeline(pipeline) | ||
|
|
||
| postHookRunner := func(ctx *context.Context, result *schemas.BifrostResponse, err *schemas.BifrostError) (*schemas.BifrostResponse, *schemas.BifrostError) { | ||
| resp, bifrostErr := pipeline.RunPostHooks(ctx, result, err, len(bifrost.plugins)) | ||
| if bifrostErr != nil { | ||
| return nil, bifrostErr | ||
| } | ||
| return resp, nil | ||
| } | ||
| stream, bifrostError = handleTranscriptionStream(provider, &req, key, postHookRunner) | ||
| } else { |
There was a problem hiding this comment.
Add missing error handling for transcription stream requests.
The TranscriptionStreamRequest handler is missing the error check that exists for other streaming handlers. This could lead to incorrect retry behavior.
Add the missing error handling after line 1418:
stream, bifrostError = handleTranscriptionStream(provider, &req, key, postHookRunner)
+ if bifrostError != nil && !bifrostError.IsBifrostError {
+ break // Don't retry client errors
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| } else if req.Type == TranscriptionStreamRequest { | |
| pipeline := bifrost.getPluginPipeline() | |
| defer bifrost.releasePluginPipeline(pipeline) | |
| postHookRunner := func(ctx *context.Context, result *schemas.BifrostResponse, err *schemas.BifrostError) (*schemas.BifrostResponse, *schemas.BifrostError) { | |
| resp, bifrostErr := pipeline.RunPostHooks(ctx, result, err, len(bifrost.plugins)) | |
| if bifrostErr != nil { | |
| return nil, bifrostErr | |
| } | |
| return resp, nil | |
| } | |
| stream, bifrostError = handleTranscriptionStream(provider, &req, key, postHookRunner) | |
| } else { | |
| } else if req.Type == TranscriptionStreamRequest { | |
| pipeline := bifrost.getPluginPipeline() | |
| defer bifrost.releasePluginPipeline(pipeline) | |
| postHookRunner := func(ctx *context.Context, result *schemas.BifrostResponse, err *schemas.BifrostError) (*schemas.BifrostResponse, *schemas.BifrostError) { | |
| resp, bifrostErr := pipeline.RunPostHooks(ctx, result, err, len(bifrost.plugins)) | |
| if bifrostErr != nil { | |
| return nil, bifrostErr | |
| } | |
| return resp, nil | |
| } | |
| stream, bifrostError = handleTranscriptionStream(provider, &req, key, postHookRunner) | |
| if bifrostError != nil && !bifrostError.IsBifrostError { | |
| break // Don't retry client errors | |
| } | |
| } else { |
🤖 Prompt for AI Agents
In core/bifrost.go around lines 1407 to 1419, the TranscriptionStreamRequest
handler calls handleTranscriptionStream but does not check for errors afterward,
unlike other streaming handlers. Add error handling immediately after line 1418
to check if bifrostError is not nil and handle it appropriately, ensuring
consistent retry behavior and error management.
| // TranscriptionUsage represents usage information for transcription | ||
| type TranscriptionUsage struct { | ||
| Type string `json:"type"` // "tokens" or "duration" | ||
| InputTokens *int `json:"input_tokens,omitempty"` | ||
| InputTokenDetails *AudioTokenDetails `json:"input_token_details,omitempty"` | ||
| OutputTokens *int `json:"output_tokens,omitempty"` | ||
| TotalTokens *int `json:"total_tokens,omitempty"` | ||
| Seconds *int `json:"seconds,omitempty"` // For duration-based usage | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Consider using float64 for the Seconds field.
The Seconds field uses *int while other duration fields in the codebase (e.g., Duration in BifrostTranscribeNonStreamResponse) use *float64. Consider using *float64 for consistency and to support fractional seconds.
- Seconds *int `json:"seconds,omitempty"` // For duration-based usage
+ Seconds *float64 `json:"seconds,omitempty"` // For duration-based usage🤖 Prompt for AI Agents
In core/schemas/bifrost.go around lines 589 to 597, the Seconds field in the
TranscriptionUsage struct is currently a pointer to int, but other duration
fields use pointer to float64 for fractional seconds. Change the type of Seconds
from *int to *float64 to maintain consistency and support fractional seconds.
| }) | ||
| } | ||
|
|
||
| // handleStreamingChatCompletion handles streaming chat completion requests using Server-Sent Events (SSE) |
There was a problem hiding this comment.
Fix incorrect method documentation.
The comment incorrectly refers to chat completion instead of speech.
-// handleStreamingChatCompletion handles streaming chat completion requests using Server-Sent Events (SSE)
+// handleStreamingSpeech handles streaming speech requests using Server-Sent Events (SSE)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // handleStreamingChatCompletion handles streaming chat completion requests using Server-Sent Events (SSE) | |
| // handleStreamingSpeech handles streaming speech requests using Server-Sent Events (SSE) |
🤖 Prompt for AI Agents
In transports/bifrost-http/handlers/completions.go at line 281, the method
comment incorrectly mentions handling streaming chat completion requests, but it
should refer to handling streaming speech completion requests. Update the
comment to accurately describe that the method handles streaming speech
completion requests using Server-Sent Events (SSE).
| const AudioPlayer = ({ src }: { src: string }) => { | ||
| const [isPlaying, setIsPlaying] = useState(false) | ||
| const [audio] = useState<HTMLAudioElement | null>(typeof window !== 'undefined' ? new Audio() : null) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add cleanup for audio element to prevent memory leaks.
The audio element is created but never cleaned up, which could lead to memory leaks in long-running applications.
Consider using useEffect to handle cleanup:
const AudioPlayer = ({ src }: { src: string }) => {
const [isPlaying, setIsPlaying] = useState(false)
- const [audio] = useState<HTMLAudioElement | null>(typeof window !== 'undefined' ? new Audio() : null)
+ const [audio] = useState<HTMLAudioElement | null>(() =>
+ typeof window !== 'undefined' ? new Audio() : null
+ )
+
+ useEffect(() => {
+ return () => {
+ if (audio) {
+ audio.pause()
+ audio.src = ''
+ }
+ }
+ }, [audio])📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const AudioPlayer = ({ src }: { src: string }) => { | |
| const [isPlaying, setIsPlaying] = useState(false) | |
| const [audio] = useState<HTMLAudioElement | null>(typeof window !== 'undefined' ? new Audio() : null) | |
| const AudioPlayer = ({ src }: { src: string }) => { | |
| const [isPlaying, setIsPlaying] = useState(false) | |
| const [audio] = useState<HTMLAudioElement | null>(() => | |
| typeof window !== 'undefined' ? new Audio() : null | |
| ) | |
| useEffect(() => { | |
| return () => { | |
| if (audio) { | |
| audio.pause() | |
| audio.src = '' | |
| } | |
| } | |
| }, [audio]) | |
| // …rest of component | |
| } |
🤖 Prompt for AI Agents
In ui/components/logs/ui/audio-player.tsx around lines 5 to 7, the AudioPlayer
component creates an audio element but does not clean it up, risking memory
leaks. Wrap the audio element creation in a useEffect hook and return a cleanup
function that pauses the audio and sets it to null when the component unmounts.
This ensures proper resource release and prevents leaks.
| const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], { | ||
| type: 'audio/mpeg', | ||
| }) |
There was a problem hiding this comment.
🛠️ Refactor suggestion
Add error handling for base64 decoding and make audio type configurable.
The base64 decoding could fail with invalid input, and the hard-coded 'audio/mpeg' type assumption may be incorrect for all audio formats.
+const getAudioMimeType = (base64: string): string => {
+ // Simple detection based on base64 header or default to audio/mpeg
+ // You might want to make this more sophisticated or pass as prop
+ return 'audio/mpeg'
+}
const handlePlayPause = () => {
if (!audio || !src) return
if (isPlaying) {
audio.pause()
setIsPlaying(false)
} else {
- // Convert base64 to blob URL
- const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], {
- type: 'audio/mpeg',
- })
+ try {
+ // Convert base64 to blob URL
+ const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], {
+ type: getAudioMimeType(src),
+ })
+ } catch (error) {
+ console.error('Failed to decode base64 audio:', error)
+ return
+ }Also applies to: 35-37
🤖 Prompt for AI Agents
In ui/components/logs/ui/audio-player.tsx around lines 17 to 19 and 35 to 37,
the base64 decoding using atob can throw errors if the input is invalid, and the
audio MIME type is hard-coded to 'audio/mpeg', which may not suit all audio
formats. Add try-catch blocks around the base64 decoding to handle potential
errors gracefully. Also, make the audio MIME type configurable by passing it as
a prop or parameter instead of hard-coding it, so the component can support
different audio formats.
| const handleDownload = () => { | ||
| if (!src) return | ||
|
|
||
| const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], { | ||
| type: 'audio/mpeg', | ||
| }) | ||
| const audioUrl = URL.createObjectURL(audioBlob) | ||
|
|
||
| const a = document.createElement('a') | ||
| a.href = audioUrl | ||
| a.download = 'speech-output.mp3' | ||
| document.body.appendChild(a) | ||
| a.click() | ||
| document.body.removeChild(a) | ||
| URL.revokeObjectURL(audioUrl) | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick (assertive)
Ensure consistent blob URL cleanup in download function.
The download function creates a blob URL but there's a potential race condition where cleanup might not happen if the user navigates away quickly.
Consider adding a small delay or using setTimeout to ensure cleanup:
const handleDownload = () => {
if (!src) return
- const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], {
- type: 'audio/mpeg',
- })
- const audioUrl = URL.createObjectURL(audioBlob)
- const a = document.createElement('a')
- a.href = audioUrl
- a.download = 'speech-output.mp3'
- document.body.appendChild(a)
- a.click()
- document.body.removeChild(a)
- URL.revokeObjectURL(audioUrl)
+ try {
+ const audioBlob = new Blob([Uint8Array.from(atob(src), (c) => c.charCodeAt(0))], {
+ type: getAudioMimeType(src),
+ })
+ const audioUrl = URL.createObjectURL(audioBlob)
+
+ const a = document.createElement('a')
+ a.href = audioUrl
+ a.download = 'speech-output.mp3'
+ document.body.appendChild(a)
+ a.click()
+ document.body.removeChild(a)
+
+ // Cleanup with small delay to ensure download starts
+ setTimeout(() => URL.revokeObjectURL(audioUrl), 100)
+ } catch (error) {
+ console.error('Failed to download audio:', error)
+ }
}Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In ui/components/logs/ui/audio-player.tsx around lines 32 to 47, the
handleDownload function creates a blob URL and revokes it immediately after
triggering the download, which may cause a race condition if the user navigates
away quickly. To fix this, modify the code to revoke the blob URL inside a
setTimeout with a short delay after the click event, ensuring the URL is cleaned
up consistently without interfering with the download process.
1 participant
Add Speech and Transcription API Support
This PR adds support for text-to-speech and speech-to-text capabilities to Bifrost, enabling applications to generate audio from text and transcribe audio to text. The implementation follows the same pattern as existing completion and embedding APIs, with both streaming and non-streaming variants.
Key additions:
The implementation supports all OpenAI speech synthesis features including voice selection, instructions, and format options. For transcription, it supports file uploads, language specification, and timestamps.
This extends Bifrost's capabilities beyond text-based AI to audio processing, making it a more comprehensive AI gateway.