Skip to content

refactor(otel): move litellm error detail keys under the litellm.* namespace - #32591

Merged
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_otel_v2_error_keys_litellm_ns
Jul 9, 2026
Merged

refactor(otel): move litellm error detail keys under the litellm.* namespace#32591
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_otel_v2_error_keys_litellm_ns

Conversation

@yassin-berriai

@yassin-berriai yassin-berriai commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Live proxy with the v2 OTel integration and console exporter, hitting the real OpenAI API. The request is oversized on purpose so OpenAI returns a 400 and the proxy emits an error span:

LITELLM_OTEL_V2=true OTEL_EXPORTER=console python litellm/proxy/proxy_cli.py --config config.yaml --port 4187

curl -sS http://localhost:4187/v1/chat/completions \
  -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "hi"}], "max_tokens": 999999999}'

Before (captured at 142d5aa), the litellm-specific detail keys squat on the semconv error.* namespace:

"error.type": "BadRequestError",
"error.message": "litellm.BadRequestError: OpenAIException - max_tokens is too large: 999999999. ...",
"error.code": "invalid_value",
"error.stack_trace": "  File \"litellm/utils.py\", line 1689, in wrapper_async ...",
"error.llm_provider": "openai"

After (captured at 6258d31), the same span carries them under the litellm.* vendor namespace while the semconv-defined error.type and error.message stay put:

"error.type": "BadRequestError",
"error.message": "litellm.BadRequestError: OpenAIException - max_tokens is too large: 999999999. ...",
"litellm.provider.error.code": "invalid_value",
"litellm.provider.error.stack_trace": "  File \"litellm/utils.py\", line 1689, in wrapper_async ...",
"litellm.provider.error.llm_provider": "openai"

Type

🧹 Refactoring

Changes

The v2 OTel integration stamped its litellm-specific error detail attributes as error.code, error.stack_trace, and error.llm_provider, which places vendor keys inside the semconv-owned error.* namespace. This PR moves them to litellm.provider.error.code, litellm.provider.error.stack_trace, and litellm.provider.error.llm_provider, alongside the rest of the litellm.* vendor-extension keys. These details describe the mapped provider exception of a failed LLM call, hence the litellm.provider.error.* prefix. The semconv-defined error.type and error.message are unchanged

The test_error_attribute_keys_are_pinned test pins all five key strings as literals so the vocabulary cannot drift silently; it fails on the previous error.* values

This changes the keys emitted on v2 spans, so any dashboard reading the old detail keys off v2 spans would need updating. The v2 integration is gated behind the LITELLM_OTEL_V2 env flag and has no public docs yet, so real-world exposure is minimal; the v1 integration keeps emitting the old keys and its tests are untouched

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov

codecov Bot commented Jul 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@greptile-apps

greptile-apps Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR moves three LiteLLM-specific error detail attributes (error.code, error.stack_trace, error.llm_provider) out of the semconv-owned error.* namespace and into the litellm.provider.error.* vendor namespace. The change is scoped entirely to the v2 OTel integration, which is gated behind LITELLM_OTEL_V2=true (off by default, no public docs), and the v1 integration's attribute keys are untouched.

  • semconv.py: LiteLLMError constants renamed to litellm.provider.error.{code,stack_trace,llm_provider}; the emitter accesses them through class constants so no string literals needed updating elsewhere.
  • test_otel_v2_components.py: test_v2_error_attribute_keys_match_v1_error_attributes_byte_for_byte replaced by test_error_attribute_keys_are_pinned, which asserts the five key strings as literals; v1 keys remain locked by the separate v1 OTel test files.
  • test_otel_v2_sources_of_truth.py: Stale v1-parity comment removed from test_attribute_keys_are_unique_across_namespaces; the uniqueness assertion itself is unchanged and still passes with the new namespace.

Confidence Score: 5/5

Safe to merge; the rename is fully contained within the pre-GA v2 OTel integration, which is off by default, and the emitter references the constants through the class rather than hardcoded strings.

The rename is mechanically complete — the emitter uses LiteLLMError.CODE/STACK_TRACE/LLM_PROVIDER as class-constant lookups, so no stale string literals remain. The uniqueness test in the sources-of-truth file still covers LiteLLMError against all other namespaces. The replacement pin test preserves the same number of assertions with the correct expected values. No production code path outside the v2 flag is affected.

No files require special attention.

Important Files Changed

Filename Overview
litellm/integrations/otel/model/semconv.py Renames LiteLLMError constants from error.* to litellm.provider.error.* and updates docstring; no logic changes, emitter uses class constants so the new strings propagate automatically.
tests/test_litellm/integrations/otel/test_otel_v2_components.py Replaces v1-parity cross-check test with literal-string pin test matching the new litellm.provider.error.* keys; updated comment on existing assertion; no coverage lost given intentional v1/v2 divergence.
tests/test_litellm/integrations/otel/test_otel_v2_sources_of_truth.py Removes stale v1-parity comment from uniqueness test; no assertions changed; uniqueness check still covers all relevant classes including LiteLLMError.

Reviews (2): Last reviewed commit: "refactor(otel): move litellm error detai..." | Re-trigger Greptile

Comment thread litellm/integrations/otel/model/semconv.py Outdated
Comment thread tests/test_litellm/integrations/otel/test_otel_v2_components.py Outdated
…mespace

The v2 OTel integration stamped litellm-specific error details as
error.code, error.stack_trace, and error.llm_provider, squatting on the
semconv-owned error.* namespace. They now live at
litellm.provider.error.code, litellm.provider.error.stack_trace, and
litellm.provider.error.llm_provider alongside the other vendor-extension
keys. error.type and error.message stay on the semconv keys.
@yassin-berriai
yassin-berriai force-pushed the litellm_otel_v2_error_keys_litellm_ns branch from 5442086 to 6258d31 Compare July 9, 2026 07:02
@yassin-berriai

Copy link
Copy Markdown
Contributor Author

@greptileai please review the current head 6258d31

@codspeed-hq

codspeed-hq Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 30 untouched benchmarks


Comparing litellm_otel_v2_error_keys_litellm_ns (6258d31) with litellm_internal_staging (e84a19a)

Open in CodSpeed

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

The CodSpeed failure is a measurement artifact, not a real regression. The report claims test_completion_multi_turn regressed 25% while its two sibling benchmarks in the same file improved 42% and 31%. This PR renames three string constants in litellm/integrations/otel/, a module the inference benchmarks never import since the v2 integration is gated behind LITELLM_OTEL_V2, which defaults off; a diff like that cannot move inference benchmarks in opposite directions.

Two checks confirm it. A rerun of the benchmark workflow on the same commit reproduced the exact same three deltas, so the artifact is deterministic for the build rather than run-to-run noise. And PR #32577, a completely unrelated diff, currently shows the same fingerprint (-25.23%, +42.06%, +30.94% on the same three benchmarks). This needs an acknowledge on the CodSpeed dashboard rather than a fix in this PR

@yassin-berriai
yassin-berriai enabled auto-merge (squash) July 9, 2026 07:50
@yassin-berriai
yassin-berriai merged commit 1d87084 into litellm_internal_staging Jul 9, 2026
127 checks passed
@yassin-berriai
yassin-berriai deleted the litellm_otel_v2_error_keys_litellm_ns branch July 9, 2026 07:51
yucheng-berri added a commit that referenced this pull request Jul 15, 2026
…vider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message
yucheng-berri added a commit that referenced this pull request Jul 15, 2026
…vider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message
yucheng-berri added a commit that referenced this pull request Jul 15, 2026
…vider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message
yucheng-berri added a commit that referenced this pull request Jul 15, 2026
…sage and status (LIT-4179) (#33304)

* test(e2e): failed request error span carries the full untruncated message and status

Covers logging.otel.failure.exports_metric on chat_completions: a request that
fails at the provider (invalid upstream key deployment) must export one
complete trace whose gen-AI span carries the LIT-4179 error contract, declared
as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated
error.message proven by parsing the embedded provider error JSON back out of
the attribute. The root SERVER span must record the 401 the client received.
Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which
the suite's model-registering tests already assume

* test(e2e): clean failure diagnostics on the error-span contract per review

A truncated error.message with missing braces now fails with a readable
assertion instead of an unhandled ValueError, an unparseable embedded JSON
fails via pytest.fail with the truncation context, and the retry loop now
asserts the upstream provider failure was actually observed so a fresh-key
propagation deadline cannot masquerade as a trace-export failure

* test(e2e): pin the full error attribute set including the litellm.provider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message

* test(e2e): author the error-span test docstring
AloysJehwin pushed a commit to AloysJehwin/litellm that referenced this pull request Jul 15, 2026
…sage and status (LIT-4179) (BerriAI#33304)

* test(e2e): failed request error span carries the full untruncated message and status

Covers logging.otel.failure.exports_metric on chat_completions: a request that
fails at the provider (invalid upstream key deployment) must export one
complete trace whose gen-AI span carries the LIT-4179 error contract, declared
as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated
error.message proven by parsing the embedded provider error JSON back out of
the attribute. The root SERVER span must record the 401 the client received.
Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which
the suite's model-registering tests already assume

* test(e2e): clean failure diagnostics on the error-span contract per review

A truncated error.message with missing braces now fails with a readable
assertion instead of an unhandled ValueError, an unparseable embedded JSON
fails via pytest.fail with the truncation context, and the retry loop now
asserts the upstream provider failure was actually observed so a fresh-key
propagation deadline cannot masquerade as a trace-export failure

* test(e2e): pin the full error attribute set including the litellm.provider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (BerriAI#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message

* test(e2e): author the error-span test docstring
edelauna pushed a commit to edelauna/litellm that referenced this pull request Jul 22, 2026
…mespace (BerriAI#32591)

The v2 OTel integration stamped litellm-specific error details as
error.code, error.stack_trace, and error.llm_provider, squatting on the
semconv-owned error.* namespace. They now live at
litellm.provider.error.code, litellm.provider.error.stack_trace, and
litellm.provider.error.llm_provider alongside the other vendor-extension
keys. error.type and error.message stay on the semconv keys.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants