litellm oss staging 04/13/2026

[krrish-berri-2]

Background

The Gemini batchEmbedContents response handler hardcoded index=0 for every embedding in the response. Any consumer relying on the OpenAI-format index field to match embeddings back to inputs would silently get wrong associations.

Changes

Use enumerate in process_response so each embedding gets its positional index instead of 0.

Test Plan

Added unit test asserting sequential indices and correct vector ordering for a 3-element batch response.

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 14, 2026 6:09pm

[greptile-apps]

Greptile Summary

This staging PR bundles several independent bug fixes: the headline fix assigns correct sequential indices in Gemini batch embedding responses (replacing the hardcoded index=0), plus safe DataDog batch draining, in-memory cache heap pruning, lowest-latency router list trimming, Anthropic streaming input_json_delta emission, and the HiddenLayer V2 guardrail integration.

The HiddenLayer V2 integration (hiddenlayer.py, hiddenlayer/__init__.py, types/.../hiddenlayer.py) still carries the structural issues raised in previous review threads — the structured_messages type mismatch, backwards-incompatible version=2 default, and empty-choices IndexError — none of which appear to be resolved in the current HEAD.

Confidence Score: 4/5

• Safe to merge for all changes except the HiddenLayer V2 integration, which still carries unresolved P1 issues from previous review threads.
• The core batch-embedding index fix, DataDog drain safety, in-memory cache heap pruning, and streaming iterator changes are all sound and well-tested. The score is held at 4 because the HiddenLayer V2 integration (introduced in commit 6343148) retains the three P1 issues previously flagged — structured_messages type mismatch, backwards-incompatible version=2 default, and IndexError on empty choices — none of which are fixed in the current code.
• litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py and litellm/types/proxy/guardrails/guardrail_hooks/hiddenlayer.py require attention for the unresolved issues from prior review threads.

Important Files Changed

Filename	Overview
litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_transformation.py	Core fix: replaced hardcoded index=0 with enumerate so each embedding in a batch response receives its positional index. Straightforward and correct.
litellm/caching/in_memory_cache.py	Adds heap-based TTL tracking (expiration_heap) and calls evict_cache() unconditionally at the start of every set_cache() to prune stale heap entries. Step 2 of evict_cache() pops from the heap with no guard for exhaustion; safe in normal operation but a defensive check (and self.expiration_heap) would prevent a potential IndexError if the heap ever gets out of sync.
litellm/integrations/datadog/datadog.py	Refactors async_send_batch to drain safely: copies the queue, clears it, restores on 413 or any exception, and only stamps last_flush_time after a confirmed successful flush. Logic is sound.
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/hiddenlayer.py	Adds HiddenlayerGuardrailV2 class targeting detection/v2/ endpoints. Several structural issues noted in previous review threads remain unresolved in this file (structured_messages type mismatch, empty-choices IndexError).
litellm/types/proxy/guardrails/guardrail_hooks/hiddenlayer.py	Adds version: Optional[int] = Field(default=2, ...) to HiddenlayerGuardrailConfigModel. The default of 2 routes all existing deployments that omit the field to V2 endpoints; backwards-compatibility concern noted in previous review thread remains unresolved.
litellm/llms/anthropic/experimental_pass_through/adapters/streaming_iterator.py	Refactors to a chunk_queue (deque) buffering approach and adds emission of input_json_delta when tool arguments are bundled in the same streaming chunk as the function name. Logic is well-structured.
litellm/router_strategy/lowest_latency.py	Changes the latency-list trim from removing the newest entry to removing the oldest entry ([1:] + [final_value]), which is the correct sliding-window behaviour. Change is minimal and correct.
tests/litellm/llms/vertex_ai/test_gemini_batch_embeddings.py	Adds test_batch_embeddings_response_has_correct_indices_and_order as a pure unit test for the core index fix. All tests are mock-based. Test is placed in tests/litellm/ (local testing folder) rather than tests/test_litellm/ as required by CLAUDE.md for new unit tests.
litellm/proxy/guardrails/guardrail_hooks/hiddenlayer/init.py	Wires HiddenlayerGuardrailV2 into the initializer registry; routes to V1 when not version or version < 2, V2 otherwise. Inherits the default-version=2 issue from the config model.
tests/test_litellm/caching/test_in_memory_cache.py	Adds tests for heap-bounded behaviour, eviction order, expired-first eviction, and the new prune-below-capacity fix. Tests are thorough and correctly placed in tests/test_litellm/.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[set_cache called] --> B{max_size == 0?}
    B -- Yes --> Z[Return early]
    B -- No --> C[call evict_cache]

    subgraph EC ["evict_cache - Step 1: prune expired heap roots"]
        S1A[peek heap top] --> S1B{Heap empty?}
        S1B -- Yes --> S1E[exit step 1]
        S1B -- No --> S1C{Entry outdated?\nexpiry != ttl_dict}
        S1C -- Yes --> S1D[pop stale entry] --> S1A
        S1C -- No --> S1F{Entry expired?}
        S1F -- Yes --> S1G[pop + remove key] --> S1A
        S1F -- No --> S1E
    end

    subgraph EC2 ["evict_cache - Step 2: evict if full"]
        S2A{cache size >= max?} -- No --> S2E[done]
        S2A -- Yes --> S2B[heappop from heap]
        S2B --> S2C{Matches ttl_dict?}
        S2C -- Yes --> S2D[remove key from cache] --> S2A
        S2C -- No --> S2A
    end

    C --> EC
    EC --> EC2
    EC2 --> E{Value size ok?}
    E -- No --> Z2[Skip insert]
    E -- Yes --> F[store value in cache_dict]
    F --> G{allow_ttl_override?}
    G -- No --> Z3[Done - TTL preserved]
    G -- Yes --> H[update ttl_dict and heappush]

Loading

_{Reviews (7): Last reviewed commit: "Fix mypy issues" | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_oss_staging_04_13_2026_p1 (dec630b) with main (b8f7d61)}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
7 out of 9 committers have signed the CLA.

✅ lucassz
✅ daanhendrio
✅ hatim-ez
✅ emerzon
✅ Ashton-Sidhu
✅ duan-levan
✅ Sameerlite
❌ krisyang1125
❌ jonemo
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Default version=2 silently breaks existing HiddenLayer integrations

version defaults to 2, so every existing deployment that omits the field will have LitellmParams.version == 2. In initialize_guardrail, the condition if not version or version < 2 evaluates to False, routing all existing configs to HiddenlayerGuardrailV2 and its new detection/v2/ endpoints. Users who are running a self-hosted or older SaaS HiddenLayer instance without V2 support will get unexpected 404/401 errors and their guardrails will silently stop blocking.

The default should be None (or 1) to preserve existing behaviour and make V2 an explicit opt-in.

Suggested change

	version: Optional[int] = Field(default=2, description="Hiddenlayer guardrail version to use.")
	version: Optional[int] = Field(default=None, description="Hiddenlayer guardrail version to use. Defaults to 1 (v1 API). Set to 2 to use the v2 API.")

Rule Used: What: avoid backwards-incompatible changes without... (source)

[greptile-apps]

IndexError when HiddenLayer returns an empty choices array

output.get("choices", [{}])[-1] uses [{}] as a default only when the choices key is absent. If HiddenLayer V2 returns {"choices": []} (present but empty), the expression becomes [][-1] and raises IndexError, crashing the guardrail post-call hook uncaught.

Suggested change

	elif input_type == "response" and inputs.get("texts"):
	inputs["texts"] = [
	output.get("choices", [{}])[-1].get("message", {}).get("content", "")
	]
	elif input_type == "response" and inputs.get("texts"):
	inputs["texts"] = [
	(output.get("choices") or [{}])[-1].get("message", {}).get("content", "")
	]

Using or [{}] covers both the missing-key and empty-list cases.