Litellm internal staging 04 11 2026

[krrish-berri-2]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 11, 2026 10:46pm

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_internal_staging_04_11_2026 (c40e459) with main (9a43e32)}

[greptile-apps]

Greptile Summary

This staging PR bundles five features: (1) per-guardrail skip_system_message_in_guardrail toggle (UI + backend) for unified guardrails on OpenAI chat and Anthropic messages; (2) structured Bedrock guardrail assessment details in error responses; (3) Azure container API support with LiteLLM-managed container ID encoding for provider routing affinity; (4) customizable Prometheus latency buckets via prometheus_latency_buckets; and (5) S3 upload retry on transient 500/503 errors. Several correctness fixes are also included (finish_reason overwrite, usage-only streaming chunks, Bedrock prefix-message handling).

• P1 — litellm/types/integrations/prometheus.py: The default LATENCY_BUCKETS constant was silently changed, removing 22 historical bucket boundaries (1.5 s, 2.5 s … 9.5 s, 15 s … 25 s, 180 s, 240 s). Existing Prometheus deployments whose dashboards or alert rules reference those le labels will stop receiving new data after upgrading, with no opt-out path short of manually restoring the old values via the new prometheus_latency_buckets setting.

Confidence Score: 3/5

Mostly safe, but the silent LATENCY_BUCKETS default change will break existing Prometheus dashboards and alerts for any user who upgrades without setting prometheus_latency_buckets.

One P1 finding: the default Prometheus histogram buckets changed in a backwards-incompatible way. This won't crash the proxy but it breaks the monitoring contract for existing deployments — a hard-to-debug production observability regression. All other changes are well-implemented and well-tested.

litellm/types/integrations/prometheus.py — default LATENCY_BUCKETS change; ui/litellm-dashboard/src/components/guardrails/edit_guardrail_form.tsx — fullLitellmParams spread may carry over stale provider-specific keys on provider switch.

Important Files Changed

Filename	Overview
litellm/types/integrations/prometheus.py	Default LATENCY_BUCKETS significantly changed (22 buckets removed, 3 added); breaks existing Prometheus dashboards/alerts keyed on removed le labels.
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py	Adds _extract_blocked_assessments to surface structured policy-violation details; enriches HTTPException with guardrailIdentifier, guardrailVersion, and assessments.
litellm/proxy/utils.py	Adds guardrail context enrichment (name, mode) to HTTPExceptions raised during all hook phases; wraps all guardrail tasks and streaming iterators with enrichment helpers.
litellm/proxy/common_request_processing.py	Fixes dict-detail HTTPException serialization for both streaming and non-streaming proxy surfaces; preserves full guardrail payload as structured fields.
litellm/responses/utils.py	Adds container ID encoding/decoding with provider and model metadata; wires into _update_container_ids_in_response for non-streaming responses.
litellm/integrations/s3_v2.py	Adds 3-attempt exponential-backoff retry (1s, 2s) for transient S3 500/503 errors in both sync and async upload paths.
litellm/llms/base_llm/guardrail_translation/utils.py	New helpers effective_skip_system_message_for_guardrail and openai_messages_without_system extract per-guardrail vs global skip logic cleanly.
ui/litellm-dashboard/src/components/guardrails/edit_guardrail_form.tsx	Preserves full stored litellm_params on edit (avoids silent loss of content-filter fields); adds skip_system_message_choice. Stale provider-specific params can survive a provider switch.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Guardrail Request] --> B{during_call_hook}
    B --> C[_run_guardrail_task_with_enrichment]
    C --> D{Unified guardrail?}
    D -- Yes --> E[async_moderation_hook]
    D -- No --> F[callback.async_moderation_hook]
    E --> G{skip_system_message?}
    F --> G
    G -- Yes --> H[openai_messages_without_system]
    G -- No --> I[Full messages]
    H --> J[Apply guardrail]
    I --> J
    J -- Blocked --> K[HTTPException with dict detail]
    K --> L[_enrich_http_exception_with_guardrail_context]
    L --> M[ProxyException via _serialize_http_exception_detail]
    M --> N[Structured error response]
    J -- Allowed --> O[Request proceeds]

Loading

_{Reviews (2): Last reviewed commit: "fix linting" | Re-trigger Greptile}

[greptile-apps]

pop() mutates the caller's litellm_params dict

litellm_params.pop("metadata", None) removes the metadata key from the dict that was passed in by the caller. If any upstream caller keeps a reference to the same dict (e.g. for retry logic, follow-up logging, or span attribution), it will silently lose its metadata after this call. CLAUDE.md's "use dict spread for immutable copies" guideline recommends building a new dict instead:

lp_metadata = litellm_params.get("metadata", None)
base_litellm_params.update({k: v for k, v in litellm_params.items() if k != "metadata"})

Context Used: CLAUDE.md (source)

[greptile-apps]

Inline import litellm inside function body

CLAUDE.md explicitly says: "Avoid imports within methods — place all imports at the top of the file … The only exception is avoiding circular imports where absolutely necessary." The only reason import litellm is deferred here is to avoid a circular import (litellm/__init__.py imports from litellm.llms.*). If that is the intent, a comment explaining the circular-import rationale would prevent future reviewers from "cleaning it up" back to a module-level import.

def effective_skip_system_message_for_guardrail(guardrail_to_apply: Any) -> bool:
    per = getattr(guardrail_to_apply, "skip_system_message_in_guardrail", None)
    if per is not None:
        return bool(per)
    # Deferred to avoid circular import (litellm.__init__ → litellm.llms.*)
    import litellm  # noqa: PLC0415

    return bool(getattr(litellm, "skip_system_message_in_guardrail", False))

Context Used: CLAUDE.md (source)

[greptile-apps]

replace("cntr_", "") strips all occurrences, not just the prefix

container_id.replace("cntr_", "") would remove every occurrence of the literal string in container_id, not only the leading one. Standard base64 (b64encode) cannot produce cntr_ (underscores don't appear in the standard alphabet), so this is safe today, but the intent is clearly prefix removal. Use removeprefix (Python 3.9+) or an explicit slice to make the intent unambiguous:

Suggested change

	"""
	try:
	# If it doesn't start with cntr_, it's not a managed ID
	if not container_id.startswith("cntr_"):
	return DecodedResponseId(
	custom_llm_provider=None,
	cleaned_id = container_id.removeprefix("cntr_")