perf(cloud): pass-through fast path for /v1/embeddings (#15512)

[NubsCarson]

Closes #15512.

Summary

• pass-through fast path for /api/v1/embeddings: when INFERENCE_PASSTHROUGH_EMBEDDINGS is on and the direct-OpenAI source serves the model, forward the validated request verbatim and return the upstream JSON untouched — no AI-SDK decode/validate/re-encode of the float arrays
• usage parses once from the same buffer, so the deferred settle chain (billUsage → settler → usageService) bills exactly what the provider reported — identical to the SDK path
• upstream failures throw the same APICallError shape the SDK path produces, so the existing route catch maps them (429/402/503) and releases the credit hold — one failure path, no new error handling
• the Vercel AI Gateway fallback source keeps the SDK path (its wire shape is not guaranteed OpenAI-verbatim)
• X-Eliza-Inference-Path: passthrough response header for probes, same convention as /v1/chat/completions (perf(cloud): opt-in pass-through streaming for openai-compatible providers (8x gateway overhead) #15437)
• flag default off; [env.staging.vars] + [env.production.vars] set true — same soak-then-cutover shape as chore(cloud): enable pass-through streaming on production #15473; rollback = flip off

Why

Measured on prod 2026-07-08 (same input): gateway /v1/embeddings 1.14–3.83s vs direct api.openai.com 0.22–0.28s. Agent runtimes make 2–3 embedding calls per message turn (the always-on recall provider embeds every incoming message), so this is currently the single largest remaining cloud-agent latency item post-#15437/#15508 — worth 2–10s per turn.

Tests

New embeddings-passthrough.test.ts (7 cases, mirrors the sibling suites' module-boundary harness; the flag/qualification/error-mapping/hold-release logic runs real):

• verbatim upstream bytes + header + SDK never called + billed from upstream usage.prompt_tokens via waitUntil
• flag off → SDK path, no upstream fetch
• gateway source → SDK path even with flag on
• upstream 429 → 429 rate_limit_exceeded + hold released (reconcile(0))
• upstream 500 → 503 provider_error + hold released
• missing upstream usage → bills the local estimate, never zero
• OPENAI_BASE_URL override respected

All 5 sibling embeddings suites (34 tests) + the chat pass-through suite (28) green. biome check clean on touched files.

Evidence: prod before/after latency table is in #15512 (live measurements, this branch not yet deployed anywhere — staging soak will produce the after-numbers once merged).

🤖 Generated with Claude Code

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[NubsCarson]

[qa-agent] money-bar review — APPROVE (post-merge confirmation; PR merged while review ran)

Adversarial review — PR #15520 (embeddings pass-through, #15512) — MERGED into develop

Test counts (all run isolated via node test/run-unit-isolated.mjs, process-per-file, in a sparse scratch worktree at PR head 3e75ca8; main tree untouched)

Suite	Result
embeddings-passthrough.test.ts (new)	7/7 pass
5 sibling embeddings suites (affiliate-clamp 5, affiliate-reserve 5, credit-leak 7, optimistic-billing 10, route-billing 7)	34/34 pass
chat-completions-passthrough-streaming.test.ts	28/28 pass
My adversarial probes (written fresh, 6 cases)	6/6 pass

PR-claimed counts (7 + 34 + 28) reproduce exactly.

The #15437 bar — item by item

1. Usage extraction is REAL. packages/cloud/api/v1/embeddings/route.ts:492-496: body buffered once via arrayBuffer(), usage.prompt_tokens parsed from a decode of that same buffer, the identical buffer returned to the client. No tee needed (non-streaming), no double body read (text() only on the !ok branch).
2. Failure shapes settle fail-safe — probed live, not just read:
   • upstream 429 → 429 rate_limit_exceeded, reconcile(0) (PR test)
   • upstream 500 → 503 provider_error, reconcile(0) (PR test)
   • 200 + unparseable JSON (untested by the PR) → JSON.parse throws → outer catch (!billed) → hold released, billUsage never called, ≥500 to client — confirmed by my probe
   • fetch network rejection (untested by the PR) → ≥500, hold released, nothing billed — confirmed by my probe
   • missing usage / prompt_tokens: 0 → bills the chars/4 estimate, never zero (PR test + my probe). Estimate can under-bill CJK, but that fallback is byte-identical to the SDK path's result.usage?.tokens || estimatedInputTokens — not a regression, and OpenAI always returns usage in practice.
3. Exactly-once settle. Pass-through slots between the existing reserve and the existing settleBilling; same createCreditReservationSettler first-call-wins owner, same settlerBackedReservation view into billUsage (affiliate collected-earnings clamp fix(cloud): pay affiliate earnings only from collected markup #11976/money: affiliate markup mints cashable earnings on uncollectable-overage settle via /v1/embeddings (reserve omits affiliateCode — #11972 residual missed by #11976) #12017 stays armed), same billed guard. Upstream failure throws APICallError before billed = true, so the route catch is the single release path.
4. Flag gating. New flag INFERENCE_PASSTHROUGH_EMBEDDINGS (not the streaming flag), inference-passthrough.ts:47-49, same === "true" trim pattern. Flag-off is byte-identical (only diff: let embeddings initialized to [], both SDK branches assign it; dead if (passthroughBody) skipped). wrangler.toml: top-level default "false" (line 178), staging "true" (402), production "true" (560) — identical shape to the sibling streaming flag's current state, meaning prod goes live on next deploy with no separate cutover PR (PR body states this; rollback = flip off).
5. Qualifying gate. isPassthroughEmbeddingsEnabled() && resolveEmbeddingProviderSource() === "openai" + resolver null-check. resolveEmbeddingProviderSource returns "openai" iff OPENAI_API_KEY set — exactly mirroring getTextEmbeddingModel's first branch (same key, same normalizeOpenAIModelId), so routing parity holds for every model id. Gateway source stays SDK (tested). 402 reserve is fail-closed before the forward (reserve block precedes the fetch structurally). Affiliate leg unchanged (shared billUsage with affiliateCode; affiliate requests already forced onto the synchronous reserve).

Embeddings-specific checks

• (a) Billing direction: real prompt_tokens billed; estimate only on absent/zero usage, same as SDK path. Never free (probed).
• (b) Batch: one forward, one billUsage with the batch's single prompt_tokens, one usageService.create — no per-item multiply (my probe, inputTokens: 11 for 3 inputs).
• (c) 402: reserveCredits throws InsufficientCreditsError → 402 before any upstream byte. Fail-closed.
• (d) Agent callers: both real callers verified compatible with verbatim OpenAI JSON. plugins/plugin-elizacloud/src/models/embeddings.ts (priority-50 cloud slot, the 2-3-calls-per-turn path) parses data[].embedding floats + per-batch index + usage.{prompt,total}_tokens, never sends encoding_format → verbatim floats satisfy every check including the width gate. plugins/plugin-embeddings likewise. Upstream headers are stripped (my probe: openai-organization/x-request-id/set-cookie all absent — route builds a fresh 2-header Response).

Confirmed defects

None money-path. Zero scenarios found where credits leak, double-settle, or inference goes free.

Flag-gated behavioral changes (data, not blockers — all fail-loud and money-safe)

1. dimensions is now honored (route.ts:479-482 forwards {...request}; the SDK path silently dropped it — embed/embedMany called with model+value only). plugin-elizacloud always sends dimensions (embeddings.ts:207). Default config (text-embedding-3-small + 1536 = native width) → identical output. Divergent configs: (i) a model rejecting the param (text-embedding-ada-002 + any dimensions) flips from working → upstream 400 → 503 → handler throws → provider fall-through (hold released, unbilled); (ii) 3-small/3-large at non-native width (e.g. 384) flips from always-threw (native width returned, width check failed → fell to local embeddings) → now succeeds at the requested width — this is the designed fix(agent): cloud containers use cloud embeddings, not local gte-small (dim mismatch + boot waste) #8769 fix per the plugin's own comment, but an agent with existing fallback-provider vectors at that same width gets a silent embedding-space mix on recall. Triggered by a billing-flag deploy, not a config change.
2. encoding_format now honored — explicit base64 requesters get base64 (previously ignored → floats). Official OpenAI SDKs handle both; no in-repo caller sends it.
3. Response model field is OpenAI's canonical id, not an echo of the decorated request id (normalization verified: openai/text-embedding-3-small → text-embedding-3-small, spread cannot clobber it — my probe).
4. Single fetch, no AI-SDK retry on the pass-through leg — a transient upstream 5xx surfaces as an immediate 503 instead of ~2 retries; both agent callers retry 503 themselves. Latency-positive, same property the merged chat sibling has.

Verdict

Approve (post-merge confirmation). Money path matches the #15437 bar on all five criteria; the two untested failure shapes (200-with-bad-JSON, network rejection) were probed and settle fail-safe through the existing catch. The dimensions/encoding_format verbatim-forward changes (item 1-2 above) are the only client-visible deltas and are confined to non-default configs; worth one line in the staging-soak checklist: watch for 503 provider_error spikes from any org configured with ada-002 + explicit dimensions after the prod deploy.

Scratch worktree removed; /home/nubs/Git/wt-elevenlabs-dep left clean at 5523874.

— [qa-agent] every money PR in the latency wave now independently reviewed. @vps-backend one soak-checklist ask: watch for 503 provider_error spikes from any org configured ada-002 + explicit dimensions after the prod deploy (the dimensions-now-honored delta).