refactor(otel): move litellm error detail keys under the litellm.* namespace - #32591
Conversation
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR moves three LiteLLM-specific error detail attributes (
Confidence Score: 5/5Safe to merge; the rename is fully contained within the pre-GA v2 OTel integration, which is off by default, and the emitter references the constants through the class rather than hardcoded strings. The rename is mechanically complete — the emitter uses LiteLLMError.CODE/STACK_TRACE/LLM_PROVIDER as class-constant lookups, so no stale string literals remain. The uniqueness test in the sources-of-truth file still covers LiteLLMError against all other namespaces. The replacement pin test preserves the same number of assertions with the correct expected values. No production code path outside the v2 flag is affected. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/integrations/otel/model/semconv.py | Renames LiteLLMError constants from error.* to litellm.provider.error.* and updates docstring; no logic changes, emitter uses class constants so the new strings propagate automatically. |
| tests/test_litellm/integrations/otel/test_otel_v2_components.py | Replaces v1-parity cross-check test with literal-string pin test matching the new litellm.provider.error.* keys; updated comment on existing assertion; no coverage lost given intentional v1/v2 divergence. |
| tests/test_litellm/integrations/otel/test_otel_v2_sources_of_truth.py | Removes stale v1-parity comment from uniqueness test; no assertions changed; uniqueness check still covers all relevant classes including LiteLLMError. |
Reviews (2): Last reviewed commit: "refactor(otel): move litellm error detai..." | Re-trigger Greptile
…mespace The v2 OTel integration stamped litellm-specific error details as error.code, error.stack_trace, and error.llm_provider, squatting on the semconv-owned error.* namespace. They now live at litellm.provider.error.code, litellm.provider.error.stack_trace, and litellm.provider.error.llm_provider alongside the other vendor-extension keys. error.type and error.message stay on the semconv keys.
5442086 to
6258d31
Compare
|
@greptileai please review the current head 6258d31 |
|
The CodSpeed failure is a measurement artifact, not a real regression. The report claims test_completion_multi_turn regressed 25% while its two sibling benchmarks in the same file improved 42% and 31%. This PR renames three string constants in litellm/integrations/otel/, a module the inference benchmarks never import since the v2 integration is gated behind LITELLM_OTEL_V2, which defaults off; a diff like that cannot move inference benchmarks in opposite directions. Two checks confirm it. A rerun of the benchmark workflow on the same commit reproduced the exact same three deltas, so the artifact is deterministic for the build rather than run-to-run noise. And PR #32577, a completely unrelated diff, currently shows the same fingerprint (-25.23%, +42.06%, +30.94% on the same three benchmarks). This needs an acknowledge on the CodSpeed dashboard rather than a fix in this PR |
…vider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message
…vider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message
…vider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message
…sage and status (LIT-4179) (#33304) * test(e2e): failed request error span carries the full untruncated message and status Covers logging.otel.failure.exports_metric on chat_completions: a request that fails at the provider (invalid upstream key deployment) must export one complete trace whose gen-AI span carries the LIT-4179 error contract, declared as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated error.message proven by parsing the embedded provider error JSON back out of the attribute. The root SERVER span must record the 401 the client received. Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which the suite's model-registering tests already assume * test(e2e): clean failure diagnostics on the error-span contract per review A truncated error.message with missing braces now fails with a readable assertion instead of an unhandled ValueError, an unparseable embedded JSON fails via pytest.fail with the truncation context, and the retry loop now asserts the upstream provider failure was actually observed so a fresh-key propagation deadline cannot masquerade as a trace-export failure * test(e2e): pin the full error attribute set including the litellm.provider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message * test(e2e): author the error-span test docstring
…sage and status (LIT-4179) (BerriAI#33304) * test(e2e): failed request error span carries the full untruncated message and status Covers logging.otel.failure.exports_metric on chat_completions: a request that fails at the provider (invalid upstream key deployment) must export one complete trace whose gen-AI span carries the LIT-4179 error contract, declared as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated error.message proven by parsing the embedded provider error JSON back out of the attribute. The root SERVER span must record the 401 the client received. Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which the suite's model-registering tests already assume * test(e2e): clean failure diagnostics on the error-span contract per review A truncated error.message with missing braces now fails with a readable assertion instead of an unhandled ValueError, an unparseable embedded JSON fails via pytest.fail with the truncation context, and the retry loop now asserts the upstream provider failure was actually observed so a fresh-key propagation deadline cannot masquerade as a trace-export failure * test(e2e): pin the full error attribute set including the litellm.provider.error keys The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later refactor (BerriAI#32591) moved the litellm-specific keys under litellm.provider.error.*, which the initial contract missed. The payload now pins error, error.type, otel.status_code, litellm.provider.error.code=401, and litellm.provider.error.llm_provider=anthropic exactly, plus non-empty litellm.provider.error.stack_trace and the untruncated error.message * test(e2e): author the error-span test docstring
…mespace (BerriAI#32591) The v2 OTel integration stamped litellm-specific error details as error.code, error.stack_trace, and error.llm_provider, squatting on the semconv-owned error.* namespace. They now live at litellm.provider.error.code, litellm.provider.error.stack_trace, and litellm.provider.error.llm_provider alongside the other vendor-extension keys. error.type and error.message stay on the semconv keys.
Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
@greptileaito re-request a review after pushing changes)Screenshots / Proof of Fix
Live proxy with the v2 OTel integration and console exporter, hitting the real OpenAI API. The request is oversized on purpose so OpenAI returns a 400 and the proxy emits an error span:
Before (captured at 142d5aa), the litellm-specific detail keys squat on the semconv
error.*namespace:After (captured at 6258d31), the same span carries them under the
litellm.*vendor namespace while the semconv-definederror.typeanderror.messagestay put:Type
🧹 Refactoring
Changes
The v2 OTel integration stamped its litellm-specific error detail attributes as
error.code,error.stack_trace, anderror.llm_provider, which places vendor keys inside the semconv-ownederror.*namespace. This PR moves them tolitellm.provider.error.code,litellm.provider.error.stack_trace, andlitellm.provider.error.llm_provider, alongside the rest of thelitellm.*vendor-extension keys. These details describe the mapped provider exception of a failed LLM call, hence thelitellm.provider.error.*prefix. The semconv-definederror.typeanderror.messageare unchangedThe
test_error_attribute_keys_are_pinnedtest pins all five key strings as literals so the vocabulary cannot drift silently; it fails on the previouserror.*valuesThis changes the keys emitted on v2 spans, so any dashboard reading the old detail keys off v2 spans would need updating. The v2 integration is gated behind the
LITELLM_OTEL_V2env flag and has no public docs yet, so real-world exposure is minimal; the v1 integration keeps emitting the old keys and its tests are untouched