Repository navigation
feat(providers): complete Jina + Gemini Embedding 2 multimodal via OmniRoute - #10581
Conversation
|
Thanks for the thorough Jina + Gemini Embedding 2 multimodal completion -- the design (native pass-through to Jina, dashboard-key-first with env fallback, Gemini N-item batching) is solid and well tested. Two things need attention before merge: (1) the branch has drifted against the current release tip in open-sse/handlers/embeddings.ts, open-sse/handlers/search.ts and src/lib/embeddings/service.ts (two other PRs landed there since you branched) -- please rebase; (2) |
…odal
Dashboard and env keys share one Jina credential pool, native v5 Omni
{text}/{image}/{content} docs pass through /v1/embeddings intact, and
classify/segment/search are proxied without a third unused Jina card.
Route gemini-embedding-2 through embedContent/batchEmbedContents so N OpenAI input items become N vectors, pass through native multimodal parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback).
- narrow the two new no-explicit-any violations introduced by this PR (validateJinaFoundationProvider's params + catch, search.ts's normalizeJinaSearchResponse data param) - cast credentials to Record<string, unknown> at the two quota-preflight call sites in src/sse/services/auth.ts so the new JinaEnvCredentials / GeminiEnvCredentials union members type-check without loosening the allRateLimited narrowing used elsewhere in the same function Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
d22b255 to
440e5be
Compare
3d0ffb4
into
cheaper-inference:release/v3.8.50
…niRoute (cheaper-inference#10581) * feat(providers): complete Jina AI via OmniRoute including Omni multimodal Dashboard and env keys share one Jina credential pool, native v5 Omni {text}/{image}/{content} docs pass through /v1/embeddings intact, and classify/segment/search are proxied without a third unused Jina card. * chore(changelog): name Jina complete-provider fragment for cheaper-inference#10581 * feat(providers): make Gemini Embedding 2 multimodal work via OmniRoute Route gemini-embedding-2 through embedContent/batchEmbedContents so N OpenAI input items become N vectors, pass through native multimodal parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback). * fix(providers): resolve rebase fallout for Jina/Gemini embeddings - narrow the two new no-explicit-any violations introduced by this PR (validateJinaFoundationProvider's params + catch, search.ts's normalizeJinaSearchResponse data param) - cast credentials to Record<string, unknown> at the two quota-preflight call sites in src/sse/services/auth.ts so the new JinaEnvCredentials / GeminiEnvCredentials union members type-check without loosening the allRateLimited narrowing used elsewhere in the same function Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com> Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Summary
jina-ai(preferred) shares a token withjina-readerandjina-search;JINA_AI_API_KEY/JINA_API_KEYare a real fallback only when no dashboard key exists (connection_id=env:JINA_AI_API_KEY).{ text },{ image: url|base64 },{ content: [...] }, plustask/normalized) pass throughPOST /v1/embeddingsintact tohttps://api.jina.ai/v1/embeddings. Catalog ids stayjina-ai/…;jina-embeddings-v5-omnialiases tojina-embeddings-v5-omni-small.gemini/gemini-embedding-2(aliasgoogle/gemini-embedding-2). Dashboardgeminikeys are used first;GEMINI_API_KEY/GOOGLE_API_KEYonly when none exist. Native parts (text/inline_data/file_data/ fusedparts) are forwarded. N OpenAIinputitems become N:batchEmbedContentsvectors — one fused Content still correctly yields one vector. Default dim 3072.api.jina.ai) from Reader (r.jina.ai). Dashboard Test probesGET /v1/modelsand falls back tojina-embeddings-v5-omni-small. Embed/rerank logs keepconnection_id.POST /v1/classify,POST /v1/segment,jina-search(s.jina.ai) withjina-ai/jinarequest aliases. Catalog addsjina-reranker-v3.5.Fixes / implements #10579.
Related Issues
Validation
jina-omni-multimodal,jina-complete-provider,gemini-embedding-2-multimodal, embeddings multimodal, search registry/route, provider-models, sse-auth Jina poolnpm run check:env-doc-syncrelease/v3.8.50Tests Added Or Updated
tests/unit/jina-omni-multimodal.test.ts— native text+image URL forwarded intact; no image fetch; family alias; text-only SKU rejects imagestests/unit/jina-complete-provider.test.tstests/unit/gemini-embedding-2-multimodal.test.ts— catalog alias; native parts forwarded (notstring[]); N strings → N batch vectors; fused Content → one vector; 001 stays on OpenAI shim; env fallback helpertests/unit/embeddings-multimodal-7956.test.ts— Gemini N canonical items →batchEmbedContentstests/unit/embedding-family-guard.test.ts— embedding-2 default 3072Coverage Notes
jina-omni-multimodal.test.ts.gemini-embedding-2-multimodal.test.ts.sse-auth.test.ts.Reviewer Notes
r.jina.ai. Labels make that obvious; the same token is reused.{ image: "https://…" }is forwarded after a public-HTTPS check; Jina fetches the URL. Canonical{ type, source: { type: "url" } }still inlines server-side as before.x-goog-api-keyagainst:embedContent/:batchEmbedContents. Multiple parts inside one Content still fuse (Google’s documented aggregation). Multiple OpenAIinputarray elements do not fuse.google/gemini-embedding-2is an exact model-string alias togemini/gemini-embedding-2. A custom provider_node whose prefix isgoogleplusgemini-embedding-001is left unaliased.changelog.d/features/10581-jina-complete-provider.md.Test plan
POST /v1/embeddingswithjina-ai/jina-embeddings-v5-omni-smallandinput: [{ "text": "…" }, { "image": "https://…" }]reachesapi.jina.aiwith those objects unchangedPOST /v1/embeddingswithgemini/gemini-embedding-2(orgoogle/gemini-embedding-2) andinput: ["a", "b", "c"]returns 3 vectors via:batchEmbedContents{ text }+{ inline_data }batch is forwarded (not stripped to strings); fused{ parts: [text, image] }returns 1 vectorgeminikey is used when present;GEMINI_API_KEYonly when no dashboard row existsjina-embeddings-v5-omni,jina-embeddings-v5-omni-nano,jina-reranker-v3.5POST /v1/classify,POST /v1/segment,POST /v1/searchwithprovider: "jina-search"/v1/web/fetch