Skip to content

feat(providers): complete Jina + Gemini Embedding 2 multimodal via OmniRoute - #10581

Merged
diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.50from
RaviTharuma:feat/jina-complete-provider
Aug 18, 2026
Merged

diegosouzapw merged 4 commits into
cheaper-inference:release/v3.8.50from
RaviTharuma:feat/jina-complete-provider

Conversation

@RaviTharuma

@RaviTharuma RaviTharuma commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • One Jina credential pool: dashboard jina-ai (preferred) shares a token with jina-reader and jina-search; JINA_AI_API_KEY / JINA_API_KEY are a real fallback only when no dashboard key exists (connection_id=env:JINA_AI_API_KEY).
  • Jina v5 Omni multimodal is first-class: native EmbeddingsV5Request docs ({ text }, { image: url|base64 }, { content: [...] }, plus task / normalized) pass through POST /v1/embeddings intact to https://api.jina.ai/v1/embeddings. Catalog ids stay jina-ai/…; jina-embeddings-v5-omni aliases to jina-embeddings-v5-omni-small.
  • Gemini Embedding 2 multimodal (GA April 2026, not “Gemini 5”): live catalog is gemini/gemini-embedding-2 (alias google/gemini-embedding-2). Dashboard gemini keys are used first; GEMINI_API_KEY / GOOGLE_API_KEY only when none exist. Native parts (text / inline_data / file_data / fused parts) are forwarded. N OpenAI input items become N :batchEmbedContents vectors — one fused Content still correctly yields one vector. Default dim 3072.
  • UI labels distinguish Foundation API (api.jina.ai) from Reader (r.jina.ai). Dashboard Test probes GET /v1/models and falls back to jina-embeddings-v5-omni-small. Embed/rerank logs keep connection_id.
  • New Jina proxies that fit OmniRoute: POST /v1/classify, POST /v1/segment, jina-search (s.jina.ai) with jina-ai / jina request aliases. Catalog adds jina-reranker-v3.5.

Fixes / implements #10579.

Related Issues

Validation

  • Change type: provider
  • Focused tests: jina-omni-multimodal, jina-complete-provider, gemini-embedding-2-multimodal, embeddings multimodal, search registry/route, provider-models, sse-auth Jina pool
  • npm run check:env-doc-sync
  • Reconciled with release/v3.8.50
  • Production-code changes include automated tests

Tests Added Or Updated

  • tests/unit/jina-omni-multimodal.test.ts — native text+image URL forwarded intact; no image fetch; family alias; text-only SKU rejects images
  • tests/unit/jina-complete-provider.test.ts
  • tests/unit/gemini-embedding-2-multimodal.test.ts — catalog alias; native parts forwarded (not string[]); N strings → N batch vectors; fused Content → one vector; 001 stays on OpenAI shim; env fallback helper
  • tests/unit/embeddings-multimodal-7956.test.ts — Gemini N canonical items → batchEmbedContents
  • tests/unit/embedding-family-guard.test.ts — embedding-2 default 3072

Coverage Notes

  • Jina native pass-through covered by jina-omni-multimodal.test.ts.
  • Gemini native transport + N→N covered by gemini-embedding-2-multimodal.test.ts.
  • Auth pool + env fallback covered by the Jina/Gemini provider helper tests and sse-auth.test.ts.

Reviewer Notes

  • Reader stays a separate card because the host is r.jina.ai. Labels make that obvious; the same token is reused.
  • Jina not fitted (wrong OmniRoute model): chat/VLM, DeepSearch, batch embeddings, classifier train/CRUD.
  • Jina-native { image: "https://…" } is forwarded after a public-HTTPS check; Jina fetches the URL. Canonical { type, source: { type: "url" } } still inlines server-side as before.
  • Gemini Embedding 2 uses x-goog-api-key against :embedContent / :batchEmbedContents. Multiple parts inside one Content still fuse (Google’s documented aggregation). Multiple OpenAI input array elements do not fuse.
  • google/gemini-embedding-2 is an exact model-string alias to gemini/gemini-embedding-2. A custom provider_node whose prefix is google plus gemini-embedding-001 is left unaliased.
  • Changelog fragment: changelog.d/features/10581-jina-complete-provider.md.

Test plan

  • POST /v1/embeddings with jina-ai/jina-embeddings-v5-omni-small and input: [{ "text": "…" }, { "image": "https://…" }] reaches api.jina.ai with those objects unchanged
  • POST /v1/embeddings with gemini/gemini-embedding-2 (or google/gemini-embedding-2) and input: ["a", "b", "c"] returns 3 vectors via :batchEmbedContents
  • Gemini native { text } + { inline_data } batch is forwarded (not stripped to strings); fused { parts: [text, image] } returns 1 vector
  • Dashboard gemini key is used when present; GEMINI_API_KEY only when no dashboard row exists
  • Bare / family ids: jina-embeddings-v5-omni, jina-embeddings-v5-omni-nano, jina-reranker-v3.5
  • POST /v1/classify, POST /v1/segment, POST /v1/search with provider: "jina-search"
  • Reader card still only serves /v1/web/fetch

@RaviTharuma RaviTharuma changed the title feat(providers): complete Jina AI via OmniRoute including Omni multimodal feat(providers): complete Jina + Gemini Embedding 2 multimodal via OmniRoute Aug 17, 2026
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks for the thorough Jina + Gemini Embedding 2 multimodal completion -- the design (native pass-through to Jina, dashboard-key-first with env fallback, Gemini N-item batching) is solid and well tested. Two things need attention before merge: (1) the branch has drifted against the current release tip in open-sse/handlers/embeddings.ts, open-sse/handlers/search.ts and src/lib/embeddings/service.ts (two other PRs landed there since you branched) -- please rebase; (2) npm run lint is failing because of a few any types this PR introduces in open-sse/handlers/search.ts and src/lib/providers/validation*.ts, which is an error-level rule in open-sse/. Separately, this PR and #10567 both independently create src/shared/validation/jinaNativeEmbeddingInput.ts to solve the same Jina-native-image-input problem with different approaches -- since this PR's version is broader (also covers Gemini native parts and content-group fusion) we'd like to land this one as the canonical implementation and close #10567 as superseded, unless you'd prefer the other order.

RaviTharuma and others added 4 commits August 17, 2026 21:06
…odal

Dashboard and env keys share one Jina credential pool, native v5 Omni
{text}/{image}/{content} docs pass through /v1/embeddings intact, and
classify/segment/search are proxied without a third unused Jina card.
Route gemini-embedding-2 through embedContent/batchEmbedContents so N
OpenAI input items become N vectors, pass through native multimodal
parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback).
- narrow the two new no-explicit-any violations introduced by this PR
  (validateJinaFoundationProvider's params + catch, search.ts's
  normalizeJinaSearchResponse data param)
- cast credentials to Record<string, unknown> at the two quota-preflight
  call sites in src/sse/services/auth.ts so the new JinaEnvCredentials /
  GeminiEnvCredentials union members type-check without loosening the
  allRateLimited narrowing used elsewhere in the same function

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
@diegosouzapw
diegosouzapw force-pushed the feat/jina-complete-provider branch from d22b255 to 440e5be Compare August 18, 2026 00:19
@diegosouzapw
diegosouzapw merged commit 3d0ffb4 into cheaper-inference:release/v3.8.50 Aug 18, 2026
5 checks passed
@RaviTharuma
RaviTharuma deleted the feat/jina-complete-provider branch September 23, 2026 19:37
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…niRoute (cheaper-inference#10581)

* feat(providers): complete Jina AI via OmniRoute including Omni multimodal

Dashboard and env keys share one Jina credential pool, native v5 Omni
{text}/{image}/{content} docs pass through /v1/embeddings intact, and
classify/segment/search are proxied without a third unused Jina card.

* chore(changelog): name Jina complete-provider fragment for cheaper-inference#10581

* feat(providers): make Gemini Embedding 2 multimodal work via OmniRoute

Route gemini-embedding-2 through embedContent/batchEmbedContents so N
OpenAI input items become N vectors, pass through native multimodal
parts, and use dashboard Gemini keys (GEMINI_API_KEY only as fallback).

* fix(providers): resolve rebase fallout for Jina/Gemini embeddings

- narrow the two new no-explicit-any violations introduced by this PR
  (validateJinaFoundationProvider's params + catch, search.ts's
  normalizeJinaSearchResponse data param)
- cast credentials to Record<string, unknown> at the two quota-preflight
  call sites in src/sse/services/auth.ts so the new JinaEnvCredentials /
  GeminiEnvCredentials union members type-check without loosening the
  allRateLimited narrowing used elsewhere in the same function

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Ravi Tharuma <RaviTharuma@users.noreply.github.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Jina complete + Jina Omni multimodal + Gemini Embedding 2 multimodal via OmniRoute

2 participants