fix(otel): hashable scope for _emit_once when guardrail_mode is list by yucheng-berri · Pull Request #31262 · BerriAI/litellm

yucheng-berri · 2026-06-25T02:06:07Z

Relevant issues

Fixes #28486

Linear ticket

Resolves LIT-3428
Resolves LIT-3764

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

Branch creation CI run
Link: https://github.com/BerriAI/litellm/actions?query=branch%3Alitellm_lit_3428_otel_emit_once_hashable
CI run for the last commit
Link: https://github.com/BerriAI/litellm/actions?query=branch%3Alitellm_lit_3428_otel_emit_once_hashable
Merge / cherry-pick CI run
Links: TBD

Screenshots / Proof of Fix

Live proxy on this PR's worktree, real Anthropic API, OTEL callback on, custom guardrail with mode: ["pre_call", "post_call"] (the shape Presidio expands to when output_parse_pii: true). Same curl on the broken commit and the fixed commit; the only difference is whether _freeze_for_dedupe is applied inside _emit_once.

Repro config (custom guardrail records to standard_logging_object so _create_guardrail_span actually runs):

model_list:
  - model_name: anthropic-haiku
    litellm_params:
      model: anthropic/claude-haiku-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

guardrails:
  - guardrail_name: lit-3428-list-mode
    litellm_params:
      guardrail: lit_3428_repro_guardrail.Lit3428Guardrail
      default_on: true
      mode: ["pre_call", "post_call"]

general_settings:
  master_key: sk-1234

litellm_settings:
  callbacks: ["otel"]
  drop_params: true
  telemetry: false

Launch:

export PYTHONPATH=<worktree-root>
export OTEL_EXPORTER=console
python litellm/proxy/proxy_cli.py \
  --config litellm/proxy/lit_3428_repro_config.yaml \
  --port 4000 --detailed_debug 2>&1 | tee litellm.log

Curl (identical on both runs):

curl -sS -o /tmp/resp.json -w "HTTP_STATUS=%{http_code}\n" \
  -X POST http://localhost:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{"model":"anthropic-haiku","messages":[{"role":"user","content":"reply with just the word: OK"}]}'

Before (origin/litellm_internal_staging, no fix)

HTTP_STATUS=200

The user sees a 200 because the crash happens inside the logging callback and is swallowed; the request itself succeeds. The damage is silent observability loss. Proxy log:

$ grep -c unhashable litellm.log
2
$ grep -c '"name": "guardrail"' litellm.log
0

Traceback in the log (each occurrence — sync + async success callbacks both fire):

File "litellm/integrations/opentelemetry.py", line 1214, in _handle_success
    self._create_guardrail_span(kwargs=kwargs, context=guardrail_ctx)
File "litellm/integrations/opentelemetry.py", line 1862, in _create_guardrail_span
    if not self._emit_once(
        kwargs,
        "guardrail",
        guardrail_information.get("guardrail_name"),
        start_time_float,
        guardrail_information.get("guardrail_mode"),
    ):
File "litellm/integrations/opentelemetry.py", line 1102, in _emit_once
    if spans_logged.get(dedupe_key) is True:
       ~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: unhashable type: 'list'

On the blocking path (e.g. callbacks that mark guardrail-failure synchronously), the same error surfaces as HTTP 500; see GH #28486.

After (this PR)

HTTP_STATUS=200
$ grep -c unhashable litellm.log
0
$ grep -c '"name": "guardrail"' litellm.log
2

One of the emitted guardrail spans (truncated to the relevant attributes):

{
  "name": "guardrail",
  "attributes": {
    "openinference.span.kind": "GUARDRAIL",
    "guardrail_name": "lit-3428-list-mode",
    "guardrail_mode": "['pre_call', 'post_call']",
    "guardrail_response": "{\"checked\": true, \"stage\": \"pre_call\"}",
    "guardrail_status": "success"
  }
}

Regression check that the existing mode: pre_call (string) path is unaffected: the same curl with mode: pre_call returns 200, emits the guardrail span, no crash.

Type

🐛 Bug Fix

Changes

_emit_once keys spans_logged by (class, id, *scope). _create_guardrail_span passes guardrail_information["guardrail_mode"] into that scope; its type is Optional[Union[GuardrailEventHooks, List[GuardrailEventHooks], GuardrailMode]] (litellm/types/utils.py), so a YAML mode: [pre_call, post_call] makes it a real list at runtime. Lists are not hashable, so spans_logged.get(dedupe_key) raises TypeError: unhashable type: 'list'.

The fix is a small recursive normalizer applied inside _emit_once before the dict lookup:

def _freeze_for_dedupe(value: object) -> object:
    if isinstance(value, (list, tuple)):
        return tuple(_freeze_for_dedupe(item) for item in value)
    if isinstance(value, set):
        return frozenset(_freeze_for_dedupe(item) for item in value)
    if isinstance(value, dict):
        return frozenset((key, _freeze_for_dedupe(item)) for key, item in value.items())
    try:
        hash(value)
    except TypeError:
        return repr(value)
    return value

It is applied at the helper, not at the guardrail callsite, so all three _emit_once callsites ("success", "failure", and the guardrail one) are protected without per-site work. The helper assumes acyclic input; guardrail_mode values are built fresh from config (str enums, lists of str enums, TypedDict of str/list-of-str leaves), so a self-referential value cannot arise in practice.

The dedupe contract is preserved: distinct list scopes produce distinct keys (a fixed-string mutant fails the new test), and the existing string-scope behavior is byte-for-byte the same (the helper passes already-hashable scalars straight through).

Regression coverage in TestOpenTelemetrySpanDedupe (tests/test_litellm/integrations/test_opentelemetry.py):

test_emit_once_accepts_list_valued_scope_part — the exact crash case
test_emit_once_distinct_list_scopes_dont_collide — distinctness is preserved
test_emit_once_accepts_dict_and_set_scope_parts — covers the GuardrailMode TypedDict shape and a future set-shaped scope
test_create_guardrail_span_does_not_raise_on_list_mode — end-to-end through _create_guardrail_span confirming exactly one guardrail span emits across repeated lifecycle entrypoints

Each new test fails when the _freeze_for_dedupe application is reverted; mutation kill rate 4/4

Scope

This is a surgical patch for v1 (litellm/integrations/opentelemetry.py), not an architectural change. The reason v1 needs a dedupe at all is that _create_guardrail_span is registered at three lifecycle hooks (async_post_call_success_hook, _handle_success, _handle_failure) and re-reads the guardrail entry list from standard_logging_payload each time; _emit_once is the workaround that collapses the three-way fan-out into one span. That fan-out is the structural reason a stray list-valued guardrail_mode could ever reach a dict key in the first place.

v2 (litellm/integrations/otel/) does not have this shape: it emits each guardrail span directly from the guardrail-recording code at the moment a guardrail finishes (emit_guardrail_span in litellm/integrations/otel/logger.py), so the emitter never sees the same entry twice and never needs to hash guardrail_mode. v2 also normalizes guardrail_mode to a display string at the front door via _guardrail_mode_str in litellm/integrations/otel/model/payloads.py, so the wider type can never crash a downstream callsite.

The right long-term move is to retire v1's three-hook registration in favor of v2's direct-emit pattern; that is out of scope here and tracked separately. This PR is the minimum patch that unblocks v1 customers hitting the crash today

Note

Low Risk
Narrow change to v1 OTEL span dedupe on the logging callback path; behavior for hashable scopes is unchanged aside from fixing the list/dict/set crash.

Overview
Fixes TypeError: unhashable type: 'list' when OpenTelemetry dedupes spans and guardrail_mode is a list (e.g. Presidio with output_parse_pii: true or YAML mode: [pre_call, post_call]).

Adds _freeze_for_dedupe to turn list/tuple, set, and dict scope parts into hashable keys (depth cap 16, repr fallback), and applies it inside _emit_once so all dedupe sites are covered without changing string-scope behavior. Guardrail span emission and dedupe are restored; regression tests cover list/dict/set scopes, distinct list keys, cyclic input, and end-to-end _create_guardrail_span.

^{Reviewed by Cursor Bugbot for commit 1483a1b. Bugbot is set up for automated code reviews on this repo. Configure here.}

yucheng-berri · 2026-06-25T02:06:14Z

@greptileai

CLAassistant · 2026-06-25T02:06:14Z

All committers have signed the CLA.

greptile-apps · 2026-06-25T02:07:59Z

Greptile Summary

Fixes TypeError: unhashable type: 'list' in the OTel v1 span-dedupe path when guardrail_mode is a list (e.g. Presidio with output_parse_pii: true or YAML mode: [pre_call, post_call]). The crash silently dropped all guardrail spans and could surface as HTTP 500 on blocking guardrail paths.

Adds _freeze_for_dedupe — a small, depth-capped (16) recursive normalizer that converts lists → tuples, sets → frozensets, and dicts → frozensets of pairs, then applies it inside _emit_once when building the dedupe tuple key. Scalars and strings pass through unchanged, so existing string-scope behavior is byte-for-byte identical.
Adds four new TestOpenTelemetrySpanDedupe tests covering the exact crash case, distinct list key non-collision, dict/set scope parts, cyclic input (depth-cap fallback to repr), and an end-to-end _create_guardrail_span round-trip confirming exactly one span emits across repeated lifecycle entrypoints.
Registers _freeze_for_dedupe in the recursive-detector ignore list alongside all other depth-guarded helpers, with a matching justification comment.

Confidence Score: 5/5

Safe to merge. The change is confined to the v1 OTel logging callback path; the dedupe fix is additive, all hashable-scope call sites are unaffected, and the new helper has a depth cap with a safe fallback.

The change is narrow: one new module-level helper, one four-line substitution in _emit_once, and a one-line addition to the recursive-detector ignore list. The helper is thoroughly tested (list, dict, set, cyclic input, and an end-to-end guardrail span test), existing string-scope behavior is provably unchanged, and the depth cap prevents any unbounded recursion on adversarial input.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/integrations/opentelemetry.py	Adds `_freeze_for_dedupe` helper (depth-capped recursive normalizer) and applies it inside `_emit_once` when building the dedupe tuple key; fixes the `TypeError: unhashable type: 'list'` crash when `guardrail_mode` is a list.
tests/test_litellm/integrations/test_opentelemetry.py	Adds four new unit tests to `TestOpenTelemetrySpanDedupe`: list/dict/set scope parts, distinct list keys, cyclic input depth cap, and end-to-end `_create_guardrail_span` with list `guardrail_mode`. All tests are properly mocked with no real network calls.
tests/code_coverage_tests/recursive_detector.py	Adds `_freeze_for_dedupe` to `IGNORE_FUNCTIONS` with an accurate justification comment — consistent with all other depth-guarded recursive helpers in this list.

_{Reviews (6): Last reviewed commit: "fix: avoid explicit casting" | Re-trigger Greptile}

greptile-apps · 2026-06-25T02:09:04Z

Greptile Summary

Fixes a TypeError: unhashable type: 'list' crash in OpenTelemetry._emit_once that silently swallowed guardrail OTEL spans (or raised HTTP 500 on blocking paths) when guardrail_mode was configured as a list (e.g. mode: ["pre_call", "post_call"]).

Introduces _freeze_for_dedupe, a small recursive normalizer that converts lists/tuples → tuples, sets → frozensets, and dicts → frozensets of pairs before the scope parts are used as dict keys, with a repr() safety fallback for any remaining unhashable type.
Applies _freeze_for_dedupe to every element of scope inside _emit_once, protecting all three call-sites (success, failure, guardrail) without per-site changes.
Adds four focused regression tests to TestOpenTelemetrySpanDedupe covering the crash case, key distinctness, dict/set scope shapes, and end-to-end _create_guardrail_span behavior — all using InMemorySpanExporter with no network calls.

Confidence Score: 5/5

Safe to merge — the change is contained to a single helper function and one call site, with no modifications to existing tests or observable behavior for the already-working string-mode path.

The fix is minimal and surgical: _freeze_for_dedupe is pure, has no side effects, and its recursive normalization correctly handles every container type that can appear in guardrail_mode. The deduplication contract (same scope → same key, distinct scopes → distinct keys) is verified by new tests that each fail on the unfixed commit. No existing test behavior was altered.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/integrations/opentelemetry.py	Adds `_freeze_for_dedupe` module-level helper and applies it inside `_emit_once` to make the dedupe key hashable when `guardrail_mode` is a list; docstring updated to match
tests/test_litellm/integrations/test_opentelemetry.py	Adds four regression tests to `TestOpenTelemetrySpanDedupe`: list scope crash, distinct list scope uniqueness, dict/set scope parts, and end-to-end `_create_guardrail_span` with list mode — all using in-memory OTEL components, no network calls

_{Reviews (2): Last reviewed commit: "fix(otel): hashable scope for _emit_once..." | Re-trigger Greptile}

codecov · 2026-06-25T02:10:09Z

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/opentelemetry.py	93.33%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

mateo-berri · 2026-06-25T03:55:09Z

@greptileai

mateo-berri · 2026-06-25T03:56:31Z

+_FREEZE_MAX_DEPTH = 16
+
+
+def _freeze_for_dedupe(value: object, _depth: int = 0) -> object:


q: why can't this return a union of all the different possible return types?

Tried it — the passthrough return value would need a cast(Hashable, value) to satisfy basedpyright, which CLAUDE.md says don't. So object → object it is.

passthrough return value would need a cast(Hashable, value) to satisfy basedpyright

Why? Is value not hashable?

Hashable would need a cast to compile, which the repo rules forbid.

if that's not the case, I can use Union

OK just use cast

Better to have type info. If it's not possible to have type info without casting, I would rather cast

Wait actually can you swap the hash() try/except + cast for if isinstance(value, (str, int, float, bytes)) or value is None: return value else repr(value). Type narrow instead of cast

yucheng-berri · 2026-06-25T17:05:14Z

@greptileai re-review please

mateo-berri

LGTM; thanks!

mateo-berri · 2026-06-25T17:19:54Z

@greptileai

mateo-berri · 2026-06-25T17:20:00Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 1483a1b. Configure here.}

`_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]` (the shape Presidio expands to with `output_parse_pii: true`, and the shape `event_hook` carries for any `mode: [...]` in config), the tuple contains a list and `spans_logged.get(dedupe_key)` raises `TypeError: unhashable type: 'list'`. On the post-call path this fires inside the logging callback and is swallowed; the request returns 200 but the OTEL `guardrail` span is silently dropped. On the blocking path the same error surfaces as HTTP 500. Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists and tuples into tuples, sets into frozensets, dicts into frozensets of `(key, value)` pairs, and falls back to `repr` for arbitrary unhashables. Applied inside `_emit_once` before the dict lookup, so all three callsites are protected without touching the guardrail-specific callsite. Helper assumes acyclic input; `guardrail_mode` values are built fresh from config (str enums, lists of str enums, TypedDict of str/list-of-str), so no cycle can arise in practice. Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash, distinct-list-scope collision, dict and set scope parts, and an end-to-end `_create_guardrail_span` exercise that confirms exactly one `guardrail` span is emitted across repeated lifecycle entrypoints. Each new test fails on a reverted helper (4/4 mutation kill)

…sive detector CI's recursive_detector blocks new recursive functions in litellm/ unless they are in the allowlist with a documented bound. Cap the helper at 16 levels and return repr(value) past the cap; this is well past the realistic depth of guardrail_mode (1-3 levels) and means a future caller passing a cyclic container can no longer push the proxy logging path into a RecursionError. Add a regression test that exercises the cycle path.

@mateo-berri

… union Per review feedback from @mateo-berri: replace the loose `-> object` annotation with a recursive `HashableScope` union (str | int | float | bool | bytes | None | Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract is visible at the signature. Replace the `try/except hash(value); return value` passthrough with an explicit isinstance check over the hashable-scalar types so the type checker can narrow without requiring `cast(Hashable, value)` on the return. Symmetric: dict keys also flow through the freezer (a TypedDict key is already a string in practice, so behaviorally identical). All 16 regression tests still pass; mutation kill behavior preserved

yucheng-berri · 2026-06-25T18:06:56Z

@greptileai please re-review on the latest HEAD (rebased onto staging, includes @mateo-berri's avoid-cast commit)

yucheng-berri requested review from mateo-berri and yassin-berriai June 25, 2026 03:23

mateo-berri requested changes Jun 25, 2026

View reviewed changes

mateo-berri approved these changes Jun 25, 2026

View reviewed changes

cursor Bot reviewed Jun 25, 2026

View reviewed changes

yucheng-berri added 3 commits June 25, 2026 10:26

yucheng-berri force-pushed the litellm_lit_3428_otel_emit_once_hashable branch from 1483a1b to 12c8eda Compare June 25, 2026 17:27

fix: avoid explicit casting

8034e59

yucheng-berri removed the request for review from yassin-berriai June 25, 2026 17:53

yucheng-berri enabled auto-merge (squash) June 25, 2026 18:57

yucheng-berri merged commit a545c49 into litellm_internal_staging Jun 25, 2026
117 of 120 checks passed

yucheng-berri deleted the litellm_lit_3428_otel_emit_once_hashable branch June 25, 2026 18:59

LohitG mentioned this pull request Jun 29, 2026

test(otel): add guardrail list mode regression coverage #29155

Closed

mateo-berri mentioned this pull request Jun 29, 2026

fix(databricks): split parallel tool calls so each tool message follows tool_calls #31633

Merged

4 tasks

igiornotigi mentioned this pull request Jul 5, 2026

[Bug]: Compliance checks (GDPR / EU AI Act) report NON-COMPLIANT when guardrail_mode is a list (e.g. Presidio with output_parse_pii) #32206

Open

1 task

Anai-Guo mentioned this pull request Jul 26, 2026

fix(opentelemetry): normalize list-valued guardrail_mode for span dedupe (#28486) #28500

Closed

2 tasks

		_FREEZE_MAX_DEPTH = 16


		def _freeze_for_dedupe(value: object, _depth: int = 0) -> object:

Uh oh!

Uh oh!

Conversation

yucheng-berri commented Jun 25, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

CI (LiteLLM team)

Screenshots / Proof of Fix

Before (origin/litellm_internal_staging, no fix)

After (this PR)

Type

Changes

Scope

Uh oh!

yucheng-berri commented Jun 25, 2026

Uh oh!

CLAassistant commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

greptile-apps Bot commented Jun 25, 2026

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

codecov Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mateo-berri commented Jun 25, 2026

Uh oh!

mateo-berri Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yucheng-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

yucheng-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

yucheng-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Jun 25, 2026

Uh oh!

mateo-berri left a comment

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Jun 25, 2026

Uh oh!

mateo-berri commented Jun 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

yucheng-berri commented Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

yucheng-berri commented Jun 25, 2026 •

edited by cursor Bot

Loading

CLAassistant commented Jun 25, 2026 •

edited

Loading

greptile-apps Bot commented Jun 25, 2026 •

edited

Loading

codecov Bot commented Jun 25, 2026 •

edited

Loading

mateo-berri Jun 25, 2026 •

edited

Loading

mateo-berri Jun 25, 2026 •

edited

Loading