docs: multimodal embeddings - #3367
Conversation
📝 WalkthroughSummary by CodeRabbit
WalkthroughThis PR extends embeddings to support multimodal inputs (text, image, audio, video, file), adds POST /v1/embeddings/batch with per-item overrides, refactors input and response schemas (typed content parts, modality-labeled EmbeddingData, and EmbeddingsByType), and updates provider and quickstart docs and examples. ChangesEmbeddings Multimodal & Batch Specification
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite. This stack of pull requests is managed by Graphite. Learn more about stacking. |
Confidence Score: 5/5This is a purely documentation and OpenAPI schema change with no runtime logic; all changes are additive or clearly intentional breaking changes already called out in the PR description. All changed files are docs, OpenAPI specs, and quickstart guides. The schema additions are well-structured and the prose is accurate; the few schema gaps (missing minimum constraint, missing required guard on EmbeddingMediaPart) are quality improvements rather than correctness blockers. docs/openapi/schemas/inference/embeddings.yaml has schema constraint gaps worth tightening before the spec is consumed by code generators. Important Files Changed
Reviews (2): Last reviewed commit: "docs: multimodal embeddings" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ```json | ||
| { | ||
| "data": [ | ||
| { "index": 0, "modality": "text", "embedding": [0.12, -0.34, ...] }, | ||
| { "index": 1, "modality": "image", "embedding": [0.56, -0.78, ...] } | ||
| ] | ||
| } |
There was a problem hiding this comment.
"embedding", but EmbeddingData.embedding is now EmbeddingsByType — an object with named fields (float, int8, base64, etc.). A reader following this example will expect a plain array from the API and be confused when they receive an object instead.
| ```json | |
| { | |
| "data": [ | |
| { "index": 0, "modality": "text", "embedding": [0.12, -0.34, ...] }, | |
| { "index": 1, "modality": "image", "embedding": [0.56, -0.78, ...] } | |
| ] | |
| } | |
| ```json | |
| { | |
| "data": [ | |
| { "index": 0, "modality": "text", "embedding": { "float": [0.12, -0.34, ...] } }, | |
| { "index": 1, "modality": "image", "embedding": { "float": [0.56, -0.78, ...] } } | |
| ] | |
| } |
| ```bash | ||
| "input": [ | ||
| [ | ||
| { "type": "file", "file": { "url": "https://example.com/doc.pdf", "mime_type": "application/pdf" } } | ||
| ] | ||
| ] | ||
| ``` |
There was a problem hiding this comment.
The PDF file part example is a bare JSON fragment with no surrounding curl command or HTTP context, unlike every other example in this section. Readers can't copy-paste it as-is.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
docs/quickstart/gateway/multimodal.mdx (1)
341-352: ⚡ Quick winClarify provider scope in the “Text Embedding (all providers)” example.
The example uses Cohere-specific
input_type, so the heading can imply broader compatibility than the snippet shows.✏️ Suggested wording tweak
-### Text Embedding (all providers) +### Text Embedding (Cohere example)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/quickstart/gateway/multimodal.mdx` around lines 341 - 352, The heading "Text Embedding (all providers)" is misleading because the snippet uses Cohere-specific fields; update the docs by either renaming the heading to something like "Text Embedding (Cohere example)" or modifying the example payload to be provider-agnostic (remove or replace the Cohere-only "input_type": "search_document" and the "model": "cohere/embed-v4.0" with a generic model placeholder). Ensure you reference the heading text "Text Embedding (all providers)" and the payload keys "model" and "input_type" when making the change so readers know this snippet is Cohere-specific or converted to a generic example.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/openapi/schemas/inference/embeddings.yaml`:
- Around line 211-222: The EmbeddingVideoConfig schema is missing numeric
constraints: update the properties for start_offset_sec and end_offset_sec to
include "minimum: 0" (to enforce non-negative offsets) and add "minimum: 4" to
interval_sec (to enforce the documented 4s lower bound); modify the YAML entries
for the properties start_offset_sec, end_offset_sec, and interval_sec
accordingly so validators and generated clients will enforce these limits.
- Around line 142-178: The EmbeddingContentPart schema currently allows
mismatched or missing payloads; update EmbeddingContentPart to a discriminated
union so that the "type" field drives the required payload shape: replace the
current flat properties with a oneOf array of separate schemas (e.g.,
EmbeddingContentPartText, EmbeddingContentPartImage, EmbeddingContentPartAudio,
EmbeddingContentPartFile, EmbeddingContentPartVideo,
EmbeddingContentPartTokens), each requiring the matching payload (text, image,
audio, file, video with optional video_config, tokens) and forbid other payload
properties; add a discriminator on "type" with mappings for each subtype to
enforce that exactly one payload is present and tied to the declared type in
generated clients and docs.
- Around line 180-204: The EmbeddingMediaPart schema currently allows both or
neither of data and url; update the EmbeddingMediaPart definition to enforce
"exactly one of data or url" by adding a oneOf with two alternatives—one
requiring ["data"] and the other requiring ["url"]—while keeping the existing
properties (data, url, mime_type, filename) and descriptions; follow the same
pattern used by OCRDocument in ocr.yaml so downstream image/audio/file/video
refs inherit the constraint.
In `@docs/quickstart/go-sdk/multimodal.mdx`:
- Around line 485-506: The narrative and code example diverge: the sentence says
“alongside text/image” but the snippet only shows a video part; either update
the sentence to say “For video, add a Video part:” or modify the code to include
text/image parts alongside the video (e.g., add additional entries to the Input
slice such as an EmbeddingContent with Type:
schemas.EmbeddingContentPartTypeText and/or Type:
schemas.EmbeddingContentPartTypeImage using the same client.EmbeddingRequest and
schemas.BifrostEmbeddingRequest/EmbeddingContent structures like videoGCS
shown).
---
Nitpick comments:
In `@docs/quickstart/gateway/multimodal.mdx`:
- Around line 341-352: The heading "Text Embedding (all providers)" is
misleading because the snippet uses Cohere-specific fields; update the docs by
either renaming the heading to something like "Text Embedding (Cohere example)"
or modifying the example payload to be provider-agnostic (remove or replace the
Cohere-only "input_type": "search_document" and the "model": "cohere/embed-v4.0"
with a generic model placeholder). Ensure you reference the heading text "Text
Embedding (all providers)" and the payload keys "model" and "input_type" when
making the change so readers know this snippet is Cohere-specific or converted
to a generic example.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 9bf6eb00-4dc0-4b3f-b82a-03e25aba5a45
📒 Files selected for processing (10)
docs/openapi/openapi.jsondocs/openapi/openapi.yamldocs/openapi/paths/inference/embeddings.yamldocs/openapi/schemas/inference/embeddings.yamldocs/providers/supported-providers/azure.mdxdocs/providers/supported-providers/cohere.mdxdocs/providers/supported-providers/gemini.mdxdocs/providers/supported-providers/vertex.mdxdocs/quickstart/gateway/multimodal.mdxdocs/quickstart/go-sdk/multimodal.mdx
| EmbeddingContentPart: | ||
| type: object | ||
| required: | ||
| - type | ||
| properties: | ||
| type: | ||
| type: string | ||
| enum: [text, image, audio, file, video, tokens] | ||
| description: Modality of this content part | ||
| text: | ||
| type: string | ||
| description: Text payload — required when type is "text" | ||
| image: | ||
| $ref: '#/EmbeddingMediaPart' | ||
| description: Image payload — required when type is "image" | ||
| audio: | ||
| $ref: '#/EmbeddingMediaPart' | ||
| description: Audio payload — required when type is "audio" | ||
| file: | ||
| $ref: '#/EmbeddingMediaPart' | ||
| description: File payload — required when type is "file" | ||
| video: | ||
| $ref: '#/EmbeddingMediaPart' | ||
| description: Video payload — required when type is "video" | ||
| video_config: | ||
| $ref: '#/EmbeddingVideoConfig' | ||
| description: > | ||
| Optional video segment parameters. Only meaningful when type is "video" and the | ||
| provider supports per-segment embeddings (Vertex multimodalembedding@001). | ||
| tokens: | ||
| type: array | ||
| items: | ||
| type: array | ||
| items: | ||
| type: integer | ||
| description: Input for embedding - text or token arrays | ||
| type: integer | ||
| description: Pre-tokenised integer token IDs — required when type is "tokens" (OpenAI only) | ||
| description: > | ||
| Exactly one of text / image / audio / file / video / tokens must be set, | ||
| matching the declared type. |
There was a problem hiding this comment.
Enforce the type → payload mapping in EmbeddingContentPart using a discriminated union.
The schema currently accepts invalid shapes like {type: "text"} (missing payload) and {type: "text", image: {...}} (wrong payload type), despite the description stating exactly one payload must be set and match type. Since this is the core multimodal contract for the stack, generated clients and documentation will advertise impossible requests as valid. Use OpenAPI 3.1's oneOf with discriminator or if/then patterns to enforce this constraint.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/openapi/schemas/inference/embeddings.yaml` around lines 142 - 178, The
EmbeddingContentPart schema currently allows mismatched or missing payloads;
update EmbeddingContentPart to a discriminated union so that the "type" field
drives the required payload shape: replace the current flat properties with a
oneOf array of separate schemas (e.g., EmbeddingContentPartText,
EmbeddingContentPartImage, EmbeddingContentPartAudio, EmbeddingContentPartFile,
EmbeddingContentPartVideo, EmbeddingContentPartTokens), each requiring the
matching payload (text, image, audio, file, video with optional video_config,
tokens) and forbid other payload properties; add a discriminator on "type" with
mappings for each subtype to enforce that exactly one payload is present and
tied to the declared type in generated clients and docs.
| EmbeddingMediaPart: | ||
| type: object | ||
| description: > | ||
| Media payload for image, audio, file, or video parts. | ||
| Exactly one of data or url must be provided. | ||
| properties: | ||
| data: | ||
| type: string | ||
| description: > | ||
| Base64-encoded media data, optionally as a data URI | ||
| (data:<mime>;base64,<data>). Used for inline uploads. | ||
| url: | ||
| type: string | ||
| description: > | ||
| URL pointing to the media. For Vertex multimodal embeddings this must be | ||
| a GCS URI (gs://…). For Gemini this can be any accessible URL. | ||
| mime_type: | ||
| type: string | ||
| description: > | ||
| MIME type of the media (e.g. "image/jpeg", "audio/mpeg", "video/mp4"). | ||
| Optional when the type can be inferred from a data URI or file extension. | ||
| filename: | ||
| type: string | ||
| description: Original filename, used as display name in Gemini File API uploads. | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let me check the file and see the current schema
cat -n docs/openapi/schemas/inference/embeddings.yaml | sed -n '170,220p'Repository: maximhq/bifrost
Length of output: 2175
🏁 Script executed:
# Check if there's a git history or stack context
git log --oneline -n 20 docs/openapi/schemas/inference/embeddings.yaml 2>/dev/null | head -20Repository: maximhq/bifrost
Length of output: 96
🏁 Script executed:
# Search for other uses of EmbeddingMediaPart in the codebase
rg "EmbeddingMediaPart" --type yamlRepository: maximhq/bifrost
Length of output: 600
🏁 Script executed:
# Check git log for related changes
git log --oneline -n 10Repository: maximhq/bifrost
Length of output: 96
🏁 Script executed:
# Check the references to EmbeddingMediaPart in embeddings.yaml to see how it's used
rg -A 5 -B 5 "EmbeddingMediaPart" docs/openapi/schemas/inference/embeddings.yamlRepository: maximhq/bifrost
Length of output: 1206
🏁 Script executed:
# Check if there are any existing uses of oneOf in the OpenAPI schemas
rg "oneOf" docs/openapi/schemas/Repository: maximhq/bifrost
Length of output: 1846
🏁 Script executed:
# Check if there are existing uses of 'required' constraint in the schema
rg "required:" docs/openapi/schemas/inference/embeddings.yaml | head -20Repository: maximhq/bifrost
Length of output: 107
🏁 Script executed:
# Check examples of oneOf usage with required in the codebase
rg -B 5 -A 10 "oneOf:" docs/openapi/schemas/inference/embeddings.yamlRepository: maximhq/bifrost
Length of output: 542
🏁 Script executed:
# Look for patterns of oneOf with required constraints
rg -B 3 -A 8 "oneOf:" docs/openapi/schemas/management/mcp.yaml | head -50Repository: maximhq/bifrost
Length of output: 1340
🏁 Script executed:
# Check if there are examples of 'not' being used with required
rg -B 3 -A 3 "not:" docs/openapi/schemas/Repository: maximhq/bifrost
Length of output: 41
🏁 Script executed:
# Check OpenAPI version used in the project
head -20 docs/openapi/openapi.yamlRepository: maximhq/bifrost
Length of output: 911
🏁 Script executed:
# Look for examples of oneOf with required constraints to see the pattern used
rg -B 2 -A 8 "required:" docs/openapi/schemas/ | grep -A 8 -B 2 "oneOf" | head -40Repository: maximhq/bifrost
Length of output: 746
🏁 Script executed:
# Check if there's a cleaner oneOf pattern with required in existing code
rg -A 15 "type: object" docs/openapi/schemas/inference/embeddings.yaml | grep -A 15 "oneOf"Repository: maximhq/bifrost
Length of output: 41
🏁 Script executed:
# Check for more examples of oneOf with required in the codebase
rg -B 5 -A 12 "oneOf:" docs/openapi/schemas/integrations/openai/embeddings.yamlRepository: maximhq/bifrost
Length of output: 485
🏁 Script executed:
# Look at the git status to check for stack/multiple PRs
git statusRepository: maximhq/bifrost
Length of output: 259
🏁 Script executed:
# Check git branches to see if there's a stack of PRs
git branch -aRepository: maximhq/bifrost
Length of output: 192
🏁 Script executed:
# Verify the exact syntax pattern: look at a complete oneOf example with required
cat -n docs/openapi/schemas/inference/ocr.yaml | grep -A 25 "oneOf:"Repository: maximhq/bifrost
Length of output: 2511
🏁 Script executed:
# Let me verify that the simpler oneOf pattern (without 'not') would work
# by checking if OpenAPI 3.1.0 examples use this pattern
rg -B 3 -A 8 "oneOf:" docs/openapi/schemas/ | grep -A 8 "required:" | head -50Repository: maximhq/bifrost
Length of output: 3021
🏁 Script executed:
# Check the exact context where EmbeddingMediaPart is used to understand the validation impact
cat -n docs/openapi/schemas/inference/embeddings.yaml | sed -n '100,180p'Repository: maximhq/bifrost
Length of output: 3422
🏁 Script executed:
# Let me get the simpler oneOf pattern that's already used in the codebase
cat -n docs/openapi/schemas/inference/ocr.yaml | sed -n '147,156p'Repository: maximhq/bifrost
Length of output: 333
🏁 Script executed:
# Check if there are any other uses of data/url patterns in the schema
rg -B 2 -A 2 "data.*url|url.*data" docs/openapi/schemas/inference/embeddings.yamlRepository: maximhq/bifrost
Length of output: 207
🏁 Script executed:
# Verify if the feature branch is the only one or if there's a stack
git log --oneline --graph --all -n 20Repository: maximhq/bifrost
Length of output: 1442
Use oneOf to enforce exactly one of data/url.
The schema allows both data and url to be omitted or provided simultaneously, contradicting the stated requirement that "Exactly one of data or url must be provided." This underspecified constraint propagates to all four reuses of EmbeddingMediaPart (image, audio, file, video fields).
The codebase already uses a simpler pattern for similar constraints (see OCRDocument in ocr.yaml). Apply the same approach here:
Fix
EmbeddingMediaPart:
type: object
+ oneOf:
+ - required: [data]
+ - required: [url]
description: >
Media payload for image, audio, file, or video parts.
Exactly one of data or url must be provided.
properties:
data:📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| EmbeddingMediaPart: | |
| type: object | |
| description: > | |
| Media payload for image, audio, file, or video parts. | |
| Exactly one of data or url must be provided. | |
| properties: | |
| data: | |
| type: string | |
| description: > | |
| Base64-encoded media data, optionally as a data URI | |
| (data:<mime>;base64,<data>). Used for inline uploads. | |
| url: | |
| type: string | |
| description: > | |
| URL pointing to the media. For Vertex multimodal embeddings this must be | |
| a GCS URI (gs://…). For Gemini this can be any accessible URL. | |
| mime_type: | |
| type: string | |
| description: > | |
| MIME type of the media (e.g. "image/jpeg", "audio/mpeg", "video/mp4"). | |
| Optional when the type can be inferred from a data URI or file extension. | |
| filename: | |
| type: string | |
| description: Original filename, used as display name in Gemini File API uploads. | |
| EmbeddingMediaPart: | |
| type: object | |
| oneOf: | |
| - required: [data] | |
| - required: [url] | |
| description: > | |
| Media payload for image, audio, file, or video parts. | |
| Exactly one of data or url must be provided. | |
| properties: | |
| data: | |
| type: string | |
| description: > | |
| Base64-encoded media data, optionally as a data URI | |
| (data:<mime>;base64,<data>). Used for inline uploads. | |
| url: | |
| type: string | |
| description: > | |
| URL pointing to the media. For Vertex multimodal embeddings this must be | |
| a GCS URI (gs://…). For Gemini this can be any accessible URL. | |
| mime_type: | |
| type: string | |
| description: > | |
| MIME type of the media (e.g. "image/jpeg", "audio/mpeg", "video/mp4"). | |
| Optional when the type can be inferred from a data URI or file extension. | |
| filename: | |
| type: string | |
| description: Original filename, used as display name in Gemini File API uploads. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/openapi/schemas/inference/embeddings.yaml` around lines 180 - 204, The
EmbeddingMediaPart schema currently allows both or neither of data and url;
update the EmbeddingMediaPart definition to enforce "exactly one of data or url"
by adding a oneOf with two alternatives—one requiring ["data"] and the other
requiring ["url"]—while keeping the existing properties (data, url, mime_type,
filename) and descriptions; follow the same pattern used by OCRDocument in
ocr.yaml so downstream image/audio/file/video refs inherit the constraint.
| properties: | ||
| start_offset_sec: | ||
| type: integer | ||
| description: Start of the sampling window in seconds | ||
| end_offset_sec: | ||
| type: integer | ||
| description: End of the sampling window in seconds | ||
| interval_sec: | ||
| type: integer | ||
| description: > | ||
| Interval between consecutive segment embeddings in seconds (minimum 4). | ||
| When set, the model returns multiple EmbeddingData entries, one per segment. |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# First, check if the file exists and read the relevant section
if [ -f "docs/openapi/schemas/inference/embeddings.yaml" ]; then
echo "=== File structure (first 30 lines) ==="
head -30 "docs/openapi/schemas/inference/embeddings.yaml"
echo ""
echo "=== Lines 200-230 ==="
sed -n '200,230p' "docs/openapi/schemas/inference/embeddings.yaml"
echo ""
echo "=== Total lines in file ==="
wc -l "docs/openapi/schemas/inference/embeddings.yaml"
else
echo "File not found"
fiRepository: maximhq/bifrost
Length of output: 2034
🏁 Script executed:
#!/bin/bash
# Check for similar integer constraints in other YAML schema files
echo "=== Searching for 'minimum:' constraints in schema files ==="
rg "minimum:" docs/openapi/schemas/ -A 1 -B 1 | head -50Repository: maximhq/bifrost
Length of output: 3764
🏁 Script executed:
#!/bin/bash
# Check git status to see if this is part of a PR stack
echo "=== Git log for the file (recent commits) ==="
git log --oneline -10 -- docs/openapi/schemas/inference/embeddings.yaml 2>/dev/null || echo "Git history not available"Repository: maximhq/bifrost
Length of output: 142
Add explicit minimum constraints to video segment parameters.
The EmbeddingVideoConfig schema documents interval_sec as having a minimum of 4 seconds and assumes non-negative offsets, but these constraints are not enforced in the schema itself. Without explicit minimum properties, validators and generated clients cannot enforce these limits.
Suggested additions
start_offset_sec:
type: integer
+ minimum: 0
description: Start of the sampling window in seconds
end_offset_sec:
type: integer
+ minimum: 0
description: End of the sampling window in seconds
interval_sec:
type: integer
+ minimum: 4
description: >
Interval between consecutive segment embeddings in seconds (minimum 4).
When set, the model returns multiple EmbeddingData entries, one per segment.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/openapi/schemas/inference/embeddings.yaml` around lines 211 - 222, The
EmbeddingVideoConfig schema is missing numeric constraints: update the
properties for start_offset_sec and end_offset_sec to include "minimum: 0" (to
enforce non-negative offsets) and add "minimum: 4" to interval_sec (to enforce
the documented 4s lower bound); modify the YAML entries for the properties
start_offset_sec, end_offset_sec, and interval_sec accordingly so validators and
generated clients will enforce these limits.
| For video, add a `Video` part alongside text/image: | ||
|
|
||
| ```go | ||
| videoGCS := "gs://my-bucket/clip.mp4" | ||
| resp, err := client.EmbeddingRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), &schemas.BifrostEmbeddingRequest{ | ||
| Provider: schemas.Vertex, | ||
| Model: "multimodalembedding@001", | ||
| Input: []schemas.EmbeddingContent{ | ||
| { | ||
| { | ||
| Type: schemas.EmbeddingContentPartTypeVideo, | ||
| Video: &schemas.EmbeddingMediaPart{URL: &videoGCS}, | ||
| VideoConfig: &schemas.EmbeddingVideoConfig{ | ||
| StartOffsetSec: schemas.Ptr(0), | ||
| EndOffsetSec: schemas.Ptr(30), | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }) | ||
| // resp.Data[].Modality == "video", each entry has a VideoSegment with timestamps | ||
| ``` |
There was a problem hiding this comment.
Narrative and code diverge in the video embedding example.
The text says “alongside text/image,” but the snippet shows only a video part. Align either the sentence or the code sample.
✏️ Suggested wording fix
-For video, add a `Video` part alongside text/image:
+For video embeddings, add a `Video` part (optionally alongside text/image):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/quickstart/go-sdk/multimodal.mdx` around lines 485 - 506, The narrative
and code example diverge: the sentence says “alongside text/image” but the
snippet only shows a video part; either update the sentence to say “For video,
add a Video part:” or modify the code to include text/image parts alongside the
video (e.g., add additional entries to the Input slice such as an
EmbeddingContent with Type: schemas.EmbeddingContentPartTypeText and/or Type:
schemas.EmbeddingContentPartTypeImage using the same client.EmbeddingRequest and
schemas.BifrostEmbeddingRequest/EmbeddingContent structures like videoGCS
shown).
1fd13e8 to
e2a6b4c
Compare
35b2866 to
323d60b
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/openapi/schemas/inference/embeddings.yaml (1)
23-25: 💤 Low valueAdd
minimum: 1constraint todimensions.The
dimensionsproperty should not accept zero or negative values. Adding a constraint prevents invalid requests at the schema validation level.dimensions: type: integer + minimum: 1 description: Number of output dimensions🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/openapi/schemas/inference/embeddings.yaml` around lines 23 - 25, The OpenAPI schema's embeddings property "dimensions" lacks a lower bound; update the "dimensions" property in the embeddings schema to include a "minimum: 1" constraint so zero or negative integers are rejected by schema validation. Locate the "dimensions" entry in the embeddings schema and add the numeric minimum constraint directly under the existing "type: integer" declaration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/openapi/schemas/inference/embeddings.yaml`:
- Around line 23-25: The OpenAPI schema's embeddings property "dimensions" lacks
a lower bound; update the "dimensions" property in the embeddings schema to
include a "minimum: 1" constraint so zero or negative integers are rejected by
schema validation. Locate the "dimensions" entry in the embeddings schema and
add the numeric minimum constraint directly under the existing "type: integer"
declaration.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 410b88b7-ad19-42e5-997e-ce13706ef126
📒 Files selected for processing (10)
docs/openapi/openapi.jsondocs/openapi/openapi.yamldocs/openapi/paths/inference/embeddings.yamldocs/openapi/schemas/inference/embeddings.yamldocs/providers/supported-providers/azure.mdxdocs/providers/supported-providers/cohere.mdxdocs/providers/supported-providers/gemini.mdxdocs/providers/supported-providers/vertex.mdxdocs/quickstart/gateway/multimodal.mdxdocs/quickstart/go-sdk/multimodal.mdx
✅ Files skipped from review due to trivial changes (3)
- docs/providers/supported-providers/azure.mdx
- docs/providers/supported-providers/cohere.mdx
- docs/quickstart/gateway/multimodal.mdx
🚧 Files skipped from review as they are similar to previous changes (6)
- docs/providers/supported-providers/gemini.mdx
- docs/providers/supported-providers/vertex.mdx
- docs/quickstart/go-sdk/multimodal.mdx
- docs/openapi/paths/inference/embeddings.yaml
- docs/openapi/openapi.yaml
- docs/openapi/openapi.json
Security Policy Alert: Secret Policy ViolationThis workflow run has been blocked by StepSecurity's secrets policy because it accesses secrets and the workflow file differs from the default branch. To approve this workflow, please add the Note: The label must be added by someone other than the PR author (TejasGhatte) or automation bots to ensure proper security review. After the label is added, you can re-run the blocked workflow to proceed. This workflow will be automatically approved once merged into the default branch. For more information, see StepSecurity's Secret Exfiltration Policy documentation. |
2 participants
Summary
This PR expands the embedding API to support multimodal inputs (text, image, audio, video, file, and tokens) and introduces a new
/v1/embeddings/batchendpoint. It also refactors theEmbeddingInputschema from a flat union of string/token arrays into a structuredEmbeddingContentmodel composed of typed parts, and replaces the flatEmbeddingStructresponse type with a richerEmbeddingsByTypeobject that supports multiple quantisation formats simultaneously.Changes
/v1/embeddings/batchendpoint backed byBatchEmbeddingRequest, currently supported by Gemini viabatchEmbedContents. Each batch item can carry its own parameter overrides (task_type,dimensions,title,auto_truncate) that override the top-level defaults.EmbeddingInputfrom a rawoneOfof strings/token arrays into a three-way union: single string shorthand, array of strings shorthand, or the full multimodal form ([]EmbeddingContent).EmbeddingContent/EmbeddingContentPartschemas representing typed parts (text, image, audio, file, video, tokens) that together produce one output embedding vector.EmbeddingMediaPartfor inline base64 or URL-referenced media, with optionalmime_typeandfilenamefields.EmbeddingVideoConfigfor Vertexmultimodalembedding@001per-segment video sampling parameters.EmbeddingDatato includemodalityandvideo_segmentfields, enabling providers that return separate vectors per modality (Vertex multimodalembedding@001) to be represented correctly.EmbeddingsByTyperesponse structure supportingfloat,int8,uint8,binary,ubinary, andbase64embedding formats, enabling Cohere embed-v4 multi-quantisation responses.EmbeddingRequestnow explicitly documentstask_type,title,auto_truncate,encoding_format,dimensions, andfallbacksfields with provider-specific notes.[]EmbeddingContentinput structure.Type of change
Affected areas
How to test
go test ./...POST /v1/embeddingsrequest with a multimodalinputarray (e.g. text + image parts) to a Cohere or Gemini model and verify the response contains the expected embedding vectors.POST /v1/embeddings/batchrequest to a Gemini model with multiple items, some with item-levelparamsoverrides, and verify each item produces a separate embedding entry in the response.POST /v1/embeddingsrequest tovertex/multimodalembedding@001with text and image parts and verify the response contains separateEmbeddingDataentries withmodalityset to"text"and"image"respectively.Breaking changes
The
EmbeddingInputtype has changed from a flat union of strings/token arrays to a structured[]EmbeddingContentmodel. Existing Go SDK callers using&schemas.EmbeddingInput{Texts: []string{...}}must be updated to[]schemas.EmbeddingContent{{{Type: schemas.EmbeddingContentPartTypeText, Text: schemas.Ptr("...")}}}. The HTTP API remains backward-compatible for string and string-array shorthands.Related issues
Security considerations
No new authentication mechanisms or secret handling introduced. Media payloads passed as base64 data URIs or URLs are forwarded to the upstream provider as-is; callers should ensure URLs do not expose sensitive resources to unintended providers.
Checklist
docs/contributing/README.mdand followed the guidelines