Skip to content

chore(release): backport #32542, #32655 to stable/1.91.x and cut 1.91.3 - #32948

Merged
yuneng-berri merged 4 commits into
stable/1.91.xfrom
litellm_backport_1_91_x_bp-guard-otel-0711
Jul 11, 2026
Merged

chore(release): backport #32542, #32655 to stable/1.91.x and cut 1.91.3#32948
yuneng-berri merged 4 commits into
stable/1.91.xfrom
litellm_backport_1_91_x_bp-guard-otel-0711

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

Relevant issues

Backports two already-merged litellm_internal_staging changes onto stable/1.91.x and cuts 1.91.3. The tip 1.91.2 is already published, so this line needs the patch bump

#32542 fixes a Responses-API bypass in the shared guardrail content helpers: every guardrail that reads litellm/proxy/guardrails/_content_utils.py saw zero text on the /v1/responses path, so AIM turned it into a 422 and every other content guardrail scanned an empty payload and let the request through unscanned. #32655 adds the GenAI-semconv gen_ai.client.operation.exception log event so a failed LLM call is recorded with exception.type / exception.message / exception.stacktrace at WARN, which OTel v2 never emitted before

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

What is included

In staging merge order, then the release bump:

Adaptation notes

#32542 applied verbatim (patch-id identical to the staging commit)

#32655 needed one import resolution in litellm/integrations/otel/logger.py. Staging imports from opentelemetry.context import Context, attach, get_current; this line dropped Context because it predates staging's identity-baggage and MCP-list-tools work that referenced it, so the pick keeps the line's from opentelemetry.context import attach, get_current and adds the fix's own new from opentelemetry.sdk._logs import LoggerProvider verbatim. Keeping Context would be an unused-import failure on this line. Nothing else diverges; the OTel dependency pins are identical between the line and staging (opentelemetry-sdk==1.28.0, opentelemetry-api==1.28.0), and OpenTelemetryV2Config.enable_events already exists on the line

Known noise on this line

None. The pre-pick baseline of the touched test files was 131 passed, 0 failures, so any red on those files is introduced by this PR and should be treated as a real signal

Screenshots / Proof of Fix

Verified end-to-end against a live proxy running this branch's source, hitting real provider APIs. The extraction proof uses a small custom pre_call guardrail that logs what _content_utils extracted (iter_message_text fragments, build_inspection_messages roles, walk_user_text visit count); the OTel proof runs the otel callback with LITELLM_OTEL_V2=true, LITELLM_OTEL_INTEGRATION_ENABLE_EVENTS=true, and the default console log exporter, then fails a call with a deliberately-bogus OpenAI key so OpenAI returns a genuine 401 that maps to AuthenticationError (nothing mocked)

#32542 - /v1/responses before vs after (same curl)

curl -sS http://localhost:4001/v1/responses -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{
  "model":"real-haiku",
  "input":[
    {"role":"user","content":[{"type":"input_text","text":"my aws key is AKIAEXAMPLE, help"}]},
    {"type":"function_call","call_id":"c1","name":"lookup","arguments":"{}"},
    {"type":"function_call_output","call_id":"c1","output":"tool saw AKIAEXAMPLE in logs"}
  ]
}'

Before (pre-pick, the bug - every text guardrail is a no-op on this path):

REPRO32542 call_type=aresponses fragments_seen=0 walk_visited=0 inspection_roles=[] fragments=[]

After (this branch - the Responses taxonomy is walked, and function_call_output surfaces as role: "tool"):

REPRO32542 call_type=aresponses fragments_seen=2 walk_visited=2 inspection_roles=['user', 'tool'] fragments=['my aws key is AKIAEXAMPLE, help', 'tool saw AKIAEXAMPLE in logs']

/v1/chat/completions was the control and reported fragments_seen=1 both before and after, so chat scanning is unchanged

#32655 - failed call before vs after (same curl)

curl -sS http://localhost:4001/v1/chat/completions -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"model":"broken-openai","messages":[{"role":"user","content":"trigger a failure"}]}'

Both before and after, the call returns a real 401 AuthenticationError. Before, grepping the proxy log for the event returns nothing; after, exactly one record appears:

"event.name": "gen_ai.client.operation.exception"
"severity_number": "<SeverityNumber.WARN: 13>"
"exception.type": "AuthenticationError"
"exception.message": "litellm.AuthenticationError: AuthenticationError: OpenAIException - Incorrect API key provided: sk-proj-*************0000. ..."
"exception.stacktrace": "  File \".../litellm/main.py\", line 690, in acompletion ..."

The span-side error.type / error.message / litellm.provider.error.* keys are still present alongside it, so the existing error surface is untouched

Sanity and auth path (this branch)

A real anthropic/claude-haiku-4-5 completion returns pong; /key/generate plus a scoped completion with the generated key both succeed

Tests

Targeted suite on this line: 158 passed, 0 failures (test_content_utils.py, test_aim.py, test_otel_v2_components.py, test_otel_v2_logger.py), up from the 131-pass / 0-fail baseline by the tests these PRs add. As a regression check for existing callers of the modified helpers (build_inspection_messages, iter_message_text, walk_user_text, the SpanEmitter constructor), the full tests/test_litellm/proxy/guardrails/ and tests/test_litellm/integrations/otel/ trees pass 2371, 0 failures on this branch

Type

🆕 New Feature
🐛 Bug Fix

Changes

Two cherry-picks onto stable/1.91.x plus the 1.91.3 version bump and lock refresh. No dependency, schema, migration, auth-default, CI, or UI changes; the diff is code and tests only

yucheng-berri and others added 4 commits July 11, 2026 13:58
…elpers (#32542)

* fix(guardrails): walk Responses-API text taxonomy in shared content helpers

Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.

Three defects, all in _content_utils.py:

1. _iter_text_parts_in_content recognised only part.type == "text", but the
   Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
   Responses input list containing a function_call or function_call_output
   item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
   tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
   reject with a schema error.

Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.

* style: ruff-format changed guardrail files

* test(guardrails): cover function_call_output string form; drop em-dash in new docstring

* fix(guardrails): map function_call_output straight to user role

Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.

* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages

* docs(test): soften AIM-specific claims in LIT-4294 test docstrings

Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.

* refactor(guardrails): move unsupported-role coercion into AIM only

The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).

AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.

function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.

* refactor(guardrails): preserve role fidelity in shared _content_utils

Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).

Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
  keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
  behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
  of the chat-completions tool message shape) instead of role user, so
  Responses and chat completions produce symmetric inspection payloads.
  A caller-supplied role on the item is still preserved.

AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.

(cherry picked from commit e84a19a)
…d LLM calls (#32655)

* feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls

The GenAI semantic conventions record failures of a GenAI client operation as
a log-based event named gen_ai.client.operation.exception, carrying the
exception.type / exception.message / exception.stacktrace trio at severity
WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM
call produced only the deprecated error.* span attributes, a generic exception
span event without a stacktrace, and the stacktrace under the vendor key
litellm.provider.error.stack_trace.

Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring
the metrics plumbing) and record the event behind the enable_events flag, which
until now was defined but consumed nowhere. An operator-configured LoggerProvider
global is reused so the events ride their existing logs pipeline; an explicit
NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all.

The existing span-side error surface (error.type, error.message, the exception
span event, and the litellm.provider.error.* detail keys) is untouched for
backwards compatibility.

* fix(otel): always ride the semconv-required exception pair on the GenAI event

Filtering the event attributes on truthiness conflated "absent" with "empty",
so an empty exception.type or exception.message would have been dropped, leaving
an event with neither semconv-required field. Build the attributes so the pair is
unconditional and only the recommended stacktrace is omitted when the payload
carries none.

* docs(otel): document the events plumbing module in the package README

* test(otel): cover the log exporter selection and logs endpoint normalization

The new logs plumbing had no coverage for exporter-kind selection, the
console fallback for an unrecognized kind, the /v1/logs signal-path rewriting
that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch
processor split.

(cherry picked from commit 99b4c5e)
@yuneng-berri
yuneng-berri requested a review from a team July 11, 2026 22:20
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ yucheng-berri
✅ yuneng-berri
❌ yassin-berriai
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This backport cut (1.91.3) cherry-picks two already-merged fixes onto stable/1.91.x plus a version bump and lock refresh. Both changes are well-tested and the diff is limited to the intended files with no schema, auth, or dependency changes.

  • Guardrail content helpers (_content_utils.py, aim.py): extends TEXT_PART_TYPES to recognise input_text / output_text Responses-API part types, rewrites _coerce_input_to_messages and walk_user_text to walk the full Responses-API item taxonomy (message, function_call, function_call_output), and adds an AIM-specific wrapper that coerces any role outside {system, user, assistant} to user before the AIM POST to avoid schema-validation errors.
  • OTel V2 exception event (events.py, providers.py, emitter.py, logger.py, semconv.py): adds GenAIEventRecorder that emits the gen_ai.client.operation.exception log event at severity WARN on failed LLM-call spans, gated on enable_events (off by default); the existing span-side error.* attributes and exception span event are preserved for backwards compatibility.

Confidence Score: 4/5

Safe to merge; changes are well-scoped cherry-picks with comprehensive unit-test coverage and no modifications to auth, schema, or dependency versions.

Both cherry-picks apply cleanly and the import adaptation described in the PR notes (dropping the unused Context import) is correct. The guardrail content-helper rewrite is logically sound: function_call items (which carry no text content) are intentionally skipped in both the read and write paths; function_call_output is correctly routed to _rewrite_content for the walk path. The AIM role-coercion wrapper accesses m["role"] directly, which is safe because every code path in build_inspection_messages produces messages with an explicit role key. The OTel event plumbing uses opentelemetry.sdk._logs private-module imports, but the OTel dependency is pinned at 1.28.0 so API drift is bounded. Test additions are all unit-only with in-memory exporters; no existing test assertions were weakened.

No files require special attention. providers.py is the most complex addition (log exporter factory + provider resolution) but it closely mirrors the existing metrics pattern and is thoroughly tested.

Important Files Changed

Filename Overview
litellm/proxy/guardrails/_content_utils.py Extends TEXT_PART_TYPES to cover Responses-API part types and rewrites _coerce_input_to_messages / walk_user_text to walk the full Responses taxonomy (message items, function_call, function_call_output); logic is correct and well-tested
litellm/proxy/guardrails/guardrail_hooks/aim/aim.py Adds _build_aim_inspection_messages to coerce non-standard roles to 'user' before the AIM POST; uses direct m['role'] access which is safe given build_inspection_messages always returns role-keyed dicts
litellm/integrations/otel/plumbing/events.py New file; implements GenAIEventRecorder emitting gen_ai.client.operation.exception log events at WARN severity with the required exception.* trio; straightforward and correct
litellm/integrations/otel/plumbing/providers.py Adds log exporter/provider factories (_otlp_logs_endpoint, build_log_exporter, build_logger_provider, resolve_logger_provider, get_event_logger) mirroring the existing metrics pattern; uses opentelemetry.sdk._logs private API which is pinned at 1.28.0
litellm/integrations/otel/emitter.py Wires optional GenAIEventRecorder into SpanEmitter; event emission is correctly guarded on both event_recorder presence and SpanRole.LLM_CALL
litellm/integrations/otel/logger.py Strengthens logger_provider type from Any to LoggerProvider and adds _init_events helper that gates event recorder construction on enable_events config; clean DI pattern
litellm/integrations/otel/model/semconv.py Adds STACKTRACE constant to ExceptionEvent and new GenAIEvent class with OPERATION_EXCEPTION constant; pinned constant values match the semconv spec
tests/test_litellm/proxy/guardrails/test_content_utils.py Adds comprehensive new tests for Responses-API content-part types and function_call_output taxonomy; existing test assertions unchanged (only whitespace reformatting)
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_aim.py New test file; covers role coercion in _build_aim_inspection_messages for tool, developer, and function_call_output inputs; unit tests only, no network calls
tests/test_litellm/integrations/otel/test_otel_v2_components.py New tests for log exporter variants, logger-provider selection, and operation exception event emission; all use in-memory exporters, no network calls
tests/test_litellm/integrations/otel/test_otel_v2_logger.py New integration tests verify enable_events wires through the full failure-callback path and that events are off by default; clean additions
pyproject.toml Version bump 1.91.2 → 1.91.3 in both [project] and [tool.commitizen] blocks

Reviews (1): Last reviewed commit: "chore: refresh uv.lock for 1.91.3" | Re-trigger Greptile

@yuneng-berri
yuneng-berri enabled auto-merge July 11, 2026 22:36
@yuneng-berri
yuneng-berri merged commit 7a4a68f into stable/1.91.x Jul 11, 2026
51 of 52 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_backport_1_91_x_bp-guard-otel-0711 branch July 11, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants