litellm oss branch

[krrish-berri-2]

…5855)

Bedrock enforces non-increasing TTL ordering across cache_control blocks (tools → system → messages). The tool cache_control TTL was being unconditionally dropped to the default 5m, while system blocks preserved the user-specified TTL for Claude 4.5+ models. This mismatch caused "a ttl='1h' block must not come after a ttl='5m' block" errors when users set ttl='1h' on both tools and system.

Converse path: add_cache_point_tool_block() now accepts a model param and preserves TTL for Claude 4.5+, matching _get_cache_point_block().

Invoke path: _remove_ttl_from_cache_control() now also processes tools (was only processing system and messages).

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Screenshots / Proof of Fix

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[veria-ai]

All review comments resolved — no active issues remaining.

---

Status: 0 open
Risk: 0/10

Posted by Veria AI · 2026-04-27T06:02:32.365Z

[greptile-apps]

Greptile Summary

This PR bundles several independent changes: (1) the stated Bedrock TTL fix — add_cache_point_tool_block now accepts model and preserves ttl for Claude 4.5+ on the Converse path, and _remove_ttl_from_cache_control is extended to process tools on the Invoke path, resolving the non-increasing-TTL ordering error; (2) a new XecGuard guardrail integration with pre/during/post/logging-only support; (3) ModifyResponseException relocated to litellm/exceptions.py and post-call guardrails wired into the pass-through endpoint; (4) Predibase handler refactored to move response-processing logic into transformation.py; and (5) minor fixes to Ollama tool-call handling and router cost propagation. The Bedrock fix and the structural refactoring are clean. The XecGuard async_logging_hook has a defensiveness gap described in the inline comment; the pass-through logging issues (missing response body and litellm_logging_obj not injected into hook_data) are carried over from prior reviews.

Confidence Score: 4/5

Safe to merge with one low-severity observability gap in the new XecGuard guardrail; the core Bedrock TTL fix is correct and well-tested.

No P0 findings. One P2 in the new XecGuard async_logging_hook (silent data loss in observability when standard_logging_object is absent). Previously flagged P1s in pass_through_endpoints.py remain unaddressed but were already reviewed. Score is 4 rather than 5 because those open P1 threads are still present in the changeset.

litellm/proxy/pass_through_endpoints/pass_through_endpoints.py (open P1 findings from prior review), litellm/proxy/guardrails/guardrail_hooks/xecguard/xecguard.py (async_logging_hook guard).

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_hooks/xecguard/xecguard.py	New XecGuard guardrail integration; async_logging_hook accesses kwargs["standard_logging_object"] without a key/type guard, silently dropping guardrail observability data when that key is absent.
litellm/proxy/pass_through_endpoints/pass_through_endpoints.py	Adds post-call guardrail invocation for pass-through responses and a ModifyResponseException handler; response body may be lost for logging if the hook returns non-dict (flagged in prior reviews).
litellm/litellm_core_utils/prompt_templates/factory.py	Adds model param to add_cache_point_tool_block and _bedrock_tools_pt so TTL is preserved for Claude 4.5+ on the Converse path, mirroring the existing _get_cache_point_block behaviour.
litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py	Extends _remove_ttl_from_cache_control to also sanitize cache_control on tool definitions, matching the system/messages handling and fixing the non-increasing TTL ordering error on the Invoke path.
litellm/exceptions.py	Moves ModifyResponseException from custom_guardrail.py to exceptions.py so it can be imported by the pass-through endpoint without a circular dependency.
litellm/llms/predibase/chat/handler.py	Removes ~160 lines of output_parser/process_response logic from the handler, moving it into transformation.py following the provider separation pattern.
litellm/llms/ollama/chat/transformation.py	Fixes tool_calls assignment by writing to ollama_message instead of mutating the input message m, and adds tool_call_id pass-through.
litellm/router.py	Propagates input_cost_per_token and output_cost_per_token from db_model_info into ModelMapInfo instead of hardcoding None.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Bedrock request with cache_control TTL] --> B{Path?}

    B -->|Converse| C["_bedrock_tools_pt(tools, model=model)"]
    C --> D["add_cache_point_tool_block(tool, model=model)"]
    D --> E{is_claude_4_5_on_bedrock?}
    E -->|Yes + ttl in 5m/1h| F[Preserve TTL in cachePoint block]
    E -->|No| G[Drop TTL — default cachePoint]

    B -->|Invoke / Messages| H["_remove_ttl_from_cache_control()"]
    H --> I[Sanitize tools cache_control]
    H --> J[Sanitize system cache_control]
    H --> K[Sanitize messages cache_control]
    I --> L{Claude 4.5+?}
    J --> L
    K --> L
    L -->|Yes| M[Keep ttl if valid 5m/1h]
    L -->|No| N[Remove ttl]

    subgraph Pass-through guardrail flow
        P[Pass-through response] --> Q{response_body != None and guardrails_to_run?}
        Q -->|Yes| R[post_call_success_hook with enriched hook_data]
        R -->|Returns dict| S[Re-encode content + strip content-length]
        R -->|Returns non-dict| T[Use original content]
        R -->|Raises ModifyResponseException| U[Return 200 with error JSON]
    end

Loading

_{Reviews (8): Last reviewed commit: "fix black issues" | Re-trigger Greptile}

[greptile-apps]

Hardcoded model-specific flag — violates project rule

is_claude_4_5_on_bedrock is a hardcoded list of model-name patterns. The project policy requires model capability flags to live in model_prices_and_context_window.json and be read via get_model_info, so that adding a new Claude 4.x model that supports TTL does not require a code change or a LiteLLM upgrade. The analogous _get_cache_point_block path already calls the same hardcoded function, so this PR extends the pattern rather than introducing it — but each new call site makes the problem harder to fix later.

Rule Used: What: Do not hardcode model-specific flags in the ... (source)

[codecov]

Codecov Report

❌ Patch coverage is 98.68421% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...y/pass_through_endpoints/pass_through_endpoints.py	84.61%	4 Missing ⚠️
...llm/litellm_core_utils/prompt_templates/factory.py	90.90%	1 Missing ⚠️
...xy/guardrails/guardrail_hooks/xecguard/xecguard.py	99.60%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
6 out of 7 committers have signed the CLA.

✅ shubham-arora-clear
✅ tuhinspatra
✅ Zerohertz
✅ clyang
✅ Sameerlite
✅ mverrilli
❌ Jerry-SDE
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

litellm_logging_obj missing from hook_data — unified_guardrail fallback is dead code

The unified_guardrail.async_post_call_success_hook added a fallback in this PR to resolve call_type via data.get("litellm_logging_obj") for pass-through requests. However, hook_data is built solely from _parsed_body (the raw request body), so litellm_logging_obj is never present there. The fallback will always short-circuit with litellm_logging_obj is not None → False, meaning that for routes where neither get_call_types_for_route nor _infer_call_type resolves a call_type, the guardrail silently returns without running — even though the pass-through endpoint's logging object is available in scope as logging_obj.

hook_data = dict(_parsed_body or {})
# add this so the unified_guardrail fallback can detect pass_through call_type
hook_data["litellm_logging_obj"] = logging_obj

[kenany]

Was deleting the litellm_oss_branch branch intended here?