Skip to content

test(otel): add guardrail list mode regression coverage - #29155

Closed
LohitG wants to merge 1 commit into
BerriAI:litellm_internal_stagingfrom
LohitG:fix-otel-guardrail-mode-dedupe
Closed

test(otel): add guardrail list mode regression coverage#29155
LohitG wants to merge 1 commit into
BerriAI:litellm_internal_stagingfrom
LohitG:fix-otel-guardrail-mode-dedupe

Conversation

@LohitG

@LohitG LohitG commented May 28, 2026

Copy link
Copy Markdown

What

Adds regression coverage for OpenTelemetry guardrail span dedupe when guardrail metadata contains list-valued modes, e.g. mode: ["pre_call", "post_call"].

Refs #28486.

Context

The current litellm_internal_staging branch already contains the implementation that normalizes _emit_once scope parts via _freeze_for_dedupe. After rebasing to resolve conflicts, this PR is intentionally test-only.

Changes

  • Add direct _emit_once coverage for list-valued scope parts
  • Add _create_guardrail_span coverage proving list-valued guardrail_mode emits exactly one guardrail span across repeated lifecycle calls

Tests

  • uv run --no-sync pytest tests/test_litellm/integrations/test_opentelemetry.py::TestOpenTelemetrySpanDedupe -q
  • uv run --no-sync black --check tests/test_litellm/integrations/test_opentelemetry.py
  • git diff --check

@greptile-apps

greptile-apps Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a TypeError: unhashable type: 'list' crash that occurred when guardrail metadata contained list-valued mode fields (e.g. mode: [\"pre_call\", \"post_call\"]). The root cause was that _emit_once used raw scope parts — including the unhashable list — directly as dict key components.

  • Introduces _make_hashable_dedupe_scope_part, a recursive static method that converts lists, tuples, dicts, and sets into hashable equivalents before the dedupe key is assembled.
  • Updates _emit_once to apply the normalization and updates its docstring to reflect that scope parts no longer need to be hashable at the call site.
  • Adds two regression tests covering both the _emit_once unit path and the full _create_guardrail_span integration path with list-valued modes.

Confidence Score: 4/5

The change is narrowly scoped to the OTEL dedupe helper and carries no risk to the request hot path or auth layer.

The normalisation logic correctly handles the reported crash and preserves existing dedupe behaviour for hashable values. The one minor gap is that list and tuple scope parts are collapsed to the same tuple representation, so a caller that mixes the two types for the same logical scope value would get a false dedupe hit — harmless today since the field always arrives as a JSON-decoded list, but silently asymmetric.

The _make_hashable_dedupe_scope_part method in litellm/integrations/opentelemetry.py is worth a second look for the list/tuple type collapse.

Important Files Changed

Filename Overview
litellm/integrations/opentelemetry.py Adds _make_hashable_dedupe_scope_part static method to normalize unhashable scope parts (lists, dicts, sets) before constructing the _emit_once dedupe key, and updates _emit_once to apply it; list and tuple values are collapsed to the same tuple representation.
tests/test_litellm/integrations/test_opentelemetry.py Adds two new unit tests: test_emit_once_handles_list_scope_parts (unit-level dedupe with a list-valued mode) and test_create_guardrail_span_handles_list_guardrail_mode (integration-level span emission). Both use only in-memory exporters — no real network calls.

Reviews (1): Last reviewed commit: "Fix OTEL guardrail mode dedupe" | Re-trigger Greptile

Comment thread litellm/integrations/opentelemetry.py Outdated
@codecov

codecov Bot commented May 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jorgesnzgza

Copy link
Copy Markdown

We’re also affected by this issue in production.

We can consistently reproduce the error when using OpenTelemetry callbacks together with guardrails configured with multiple modes:

mode:
- pre_call
- post_call

This results in:

TypeError: unhashable type: 'list'

After reviewing the proposed changes, #29155 appears to address our scenario by making the OTEL deduplication key generation resilient to unhashable values, including lists.

This issue is currently blocking our OpenTelemetry observability rollout.

We would appreciate getting this PR reviewed and merged. If helpful, we can also help validate the fix in a production-like environment.

@LohitG
LohitG force-pushed the fix-otel-guardrail-mode-dedupe branch from 8158269 to e3621d3 Compare June 29, 2026 13:34
@LohitG LohitG changed the title Fix OTEL guardrail span dedupe for list modes test(otel): add guardrail list mode regression coverage Jun 29, 2026
@LohitG

LohitG commented Jun 29, 2026

Copy link
Copy Markdown
Author

We’re also affected by this issue in production.

We can consistently reproduce the error when using OpenTelemetry callbacks together with guardrails configured with multiple modes:

mode: - pre_call - post_call

This results in:

TypeError: unhashable type: 'list'

After reviewing the proposed changes, #29155 appears to address our scenario by making the OTEL deduplication key generation resilient to unhashable values, including lists.

This issue is currently blocking our OpenTelemetry observability rollout.

We would appreciate getting this PR reviewed and merged. If helpful, we can also help validate the fix in a production-like environment.

I believe #31262 solves the same issue!

@LohitG

LohitG commented Jun 29, 2026

Copy link
Copy Markdown
Author

Closing as superseded by #31262, which merged the underlying OTEL _emit_once hashable-scope fix for list-valued guardrail_mode and broader regression coverage. Thanks!

@LohitG LohitG closed this Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants