feat(langfuse)!: migrate the sdk callback to langfuse v4

[yucheng-berri]

TLDR

Problem this solves:

• The langfuse callback only works with Langfuse SDK v2, which Langfuse retired
• With langfuse>=4 installed every request fails with module 'langfuse' has no attribute 'version'
• v4 removed trace(), generation(), span(), langfuse.version and Langfuse.base_url

How it solves it:

• Exports the generation and its child spans as OpenTelemetry spans on a litellm-owned provider
• Uses Langfuse's documented langfuse.* span attributes on the v4 OTLP route
• Prompts, auth check and the Slack project id go through the SDK's REST client, never a Langfuse() client
• One export channel per credential set, refcounted, retired 60 s after the last logger is evicted
• Oversized spans (HTTP 413) are truncated with v2's marker and resent instead of dropped
• Rejects SDK 2.x and 3.x at construction with an upgrade message
• Pins langfuse>=4.7,<5 with opentelemetry 1.33.1 in pyproject.toml and uv.lock

User Flow

Before: a proxy admin who installs the current Langfuse SDK gets a broken proxy, since Langfuse logging fails before the model is called

1. The admin installs the proxy with langfuse>=4 and sets success_callback: ["langfuse"] in litellm_settings
2. The admin sends GET http://localhost:4000/health/services?service=langfuse and gets HTTP 500 with module 'langfuse' has no attribute 'version'
3. A developer sends POST http://localhost:4000/v1/chat/completions with {"model":"gpt-5.5","messages":[...]} and gets HTTP 400 with Invalid request format: module 'langfuse' has no attribute 'version'. No model reply, nothing billed, nothing in Langfuse
4. The developer opens the Langfuse project and the traces list for that session is empty

After: the same install works, and traces land in Langfuse with the fields the v2 integration produced

1. The admin installs the proxy (the pin now pulls langfuse>=4.7,<5) and sets success_callback: ["langfuse"] in litellm_settings
2. The admin sends GET http://localhost:4000/health/services?service=langfuse and gets HTTP 200 {"status":"success",...}
3. The developer sends POST http://localhost:4000/v1/chat/completions with {"model":"gpt-5.5","messages":[...]} and gets HTTP 200 with the model reply and usage
4. The developer opens the Langfuse project and sees one trace per request with the same name, user, session, tags, cost, usage and error level as before; trace ids are now 32 hex characters, since v4 requires that shape

Relevant issues

Fixes #24123. Relates to #33383 and Pylon #6665. Supersedes #33391. Docs: BerriAI/litellm-docs#1103

Affected release

Linear ticket

Resolves LIT-5050
Resolves LIT-8021

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.). At 09cc332 every check is green except documentation (required) and code-quality, both failing on one line of tests/documentation_tests/test_env_keys.py until docs(langfuse): update for Python SDK v4 migration litellm-docs#1103 merges, and oauth, which waits on the environment approval. CircleCI integration matrix is 8 of 9 green; integration-mcp fails on an UnboundLocalError inside a test this PR does not touch, which fails the same way on main
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Two real proxies, no mocks in the product path. Before is the merge base 5beac4f (its own pin, langfuse==2.59.7) on port 4200, After is 09cc332 (langfuse==4.15.2, opentelemetry-sdk==1.33.1) on port 4100, each started with python litellm/proxy/proxy_cli.py --config config.yaml --port <port> --num_workers 2 --use_v2_migration_resolver against its own PostgreSQL. The tip fdbd084 on top of 09cc332 changes only the pip Dockerfile pins and a test docstring, no production file. Provider calls hit real OpenAI gpt-5.5 and Gemini gemini-3.8-flash. The destination is a real Langfuse Cloud project and every readback is GET /api/public/traces?sessionId=... then GET /api/public/traces/<id>, matched by response id or x-litellm-call-id. For the prompt cells the proxy reaches Langfuse Cloud through a local TLS front that forwards bytes unchanged and can inject one fault. Admin UI and Langfuse UI screenshots and recordings are in #36741 (comment) and #36741 (comment); paired Before/After Langfuse UI captures of the same six requests at exactly 5beac4f vs 09cc332 are in #36741 (comment)

model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gemini-3.8-flash
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: os.environ/GEMINI_API_KEY
  - model_name: langfuse/gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

guardrails:
  - guardrail_name: mask-post
    litellm_params:
      guardrail: litellm_content_filter
      mode: post_call
      default_on: true

general_settings:
  master_key: sk-litaudit-master
  database_url: os.environ/DATABASE_URL

Every request carries -H "Authorization: Bearer sk-litaudit-master" -H "Content-Type: application/json" and a session id in metadata.session_id (chat) or litellm_metadata.session_id (responses) so the readback can find it

Before (5beac4f)

Health check

1. curl -s -w ' http=%{http_code} wall=%{time_total}s' 'http://localhost:4200/health/services?service=langfuse'
2. {"status":"success","message":"Mock LLM request made - check langfuse."} http=200 wall=0.677894s

Chat, Responses and streaming matrix

1. 12 cells: /v1/chat/completions and /v1/responses, streamed and not, curl plus the openai SDK sync and async, gpt-5.5 and gemini-3.8-flash, each with its own session id
2. Readback: 10 of 12 traces found with model, cost and usage, for example H17 PASS 200 time-20-47-32-595141_5Dq0avf-LNHVz7IP2ua2iAs gemini/gemini-3.8-flash 0.000414
3. The two streamed /v1/responses cells read FAIL(no generation): v2 stores them under the encrypted response id, so the plain readback cannot find them (pre-existing)

Guardrail child span with trace user and session

1. curl -s http://localhost:4200/v1/chat/completions -d '{"model":"gpt-5.5","messages":[{"role":"user","content":"hi"}],"metadata":{"trace_user_id":"r9u-base-<run>","session_id":"r9s-base-<run>","version":"v9","trace_name":"r9-base"}}' with the mask-post guardrail on
2. Readback: trace name=r9-base userId=r9u-base-<run> sessionId=r9s-base-<run> version=v9, observations GENERATION litellm-acompletion and SPAN guardrail (a sibling of the generation), both with userId=None sessionId=None
3. GET /api/public/observations?userId=r9u-base-<run> returns both the guardrail span and the generation

Prompt fetch on a cold miss (captured at the previous merge base bc6b540, same langfuse==2.59.7)

1. prompt_id: "litaudit-lf5050-what?" through langfuse/gpt-5.5: GET /api/public/v2/prompts/litaudit-lf5050-what%3F -> 200, answer QMARK-OK
2. prompt_id: "litaudit-lf5050-r7/greeting": GET .../prompts/litaudit-lf5050-r7/greeting -> 404, caller HTTP 404 (v2 left / unescaped)
3. Real Cloud 404 for litaudit-lf5050-does-not-exist: caller HTTP 404 in 0.89 s after 2 GETs, only litellm's own response headers
4. One injected 503 then the real prompt: caller HTTP 200 with the prompt's answer, upstream 503, 200, 200 (the third GET is the logging hook's own v2 client)
5. Persistent 503 with Retry-After: 30, /health/liveliness probed 30 times during the request: caller HTTP 503 in 0.44 s after 2 GETs, slowest probe 0.13 s

After (09cc332, no production change through tip fdbd084)

Health check

1. curl -s -w ' http=%{http_code} wall=%{time_total}s' 'http://localhost:4100/health/services?service=langfuse'
2. {"status":"success","message":"Mock LLM request made - check langfuse."} http=200 wall=0.638937s

Chat, Responses and streaming matrix

1. The same 12 cells against port 4100
2. Readback: 12 of 12 traces found with model, cost and usage, for example H16 PASS 200 3152bcbac633e621 openai/gpt-5.5 0.00071 and H17 PASS 200 fc5b51cea769f34f gemini/gemini-3.8-flash 0.00041025; the trace id is now the 16 or 32 hex form
3. The two streamed /v1/responses cells are found by x-litellm-call-id

Guardrail child span with trace user and session

1. The same request against port 4100 with trace_user_id: r9u-head-<run>, session_id: r9s-head-<run>, trace_name: r9-head
2. Readback: trace id=c5b84f5140a3c179ccc3ac10ea501840 name=r9-head userId=r9u-head-<run> sessionId=r9s-head-<run> version=v9, observations GENERATION litellm-acompletion (root) and SPAN guardrail nested under it, both with userId=None sessionId=None as on Before
3. GET /api/public/observations?userId=r9u-head-<run> returns both the guardrail span and the generation

Prompt fetch on a cold miss

1. prompt_id: "litaudit-lf5050-what?": GET /api/public/v2/prompts/litaudit-lf5050-what%3F -> 200, answer QMARK-OK (the tip before round 7 fetched .../what and served the wrong prompt)
2. prompt_id: "litaudit-lf5050-r7/greeting": GET .../prompts/litaudit-lf5050-r7%2Fgreeting -> 200, answer FOLDER-OK
3. Real Cloud 404 for litaudit-lf5050-does-not-exist: caller HTTP 404 in 0.57 s after 1 GET, response headers x-litellm-call-id, x-litellm-response-cost, x-litellm-key-spend, content-type, x-frame-options, content-security-policy, x-content-type-options only, body {"error":{"message":"status_code: 404, body: {'message': \"Prompt not found: 'litaudit-lf5050-does-not-exist' with label 'production'\", ...}","code":"404"}}
4. One injected 503 then the real prompt: caller HTTP 200 with the prompt's answer after exactly 2 GETs (503, 200)
5. Persistent 503 with Retry-After: 30, /health/liveliness probed 30 times during the request: caller HTTP 503 in 0.36 s after 2 GETs, no upstream header forwarded, slowest probe 0.04 s

Type

🐛 Bug Fix

Changes

litellm/integrations/langfuse/langfuse_sdk.py (new) is litellm's tracing layer for Langfuse v4:

• trace_attributes and observation_attributes build the langfuse.trace.* and langfuse.observation.* span attributes from the SDK's public LangfuseOtelSpanAttributes constants, serialising input, output, usage, cost, model parameters, prompt and metadata the way the SDK's wire format does
• resolve_trace_id and resolve_observation_id lowercase and de-dash the caller's id, keep it when it is 32 or 16 hex characters, and sha256-hash anything else
• start_generation opens the generation with the model call's start_time under a remote parent context built from existing_trace_id and parent_observation_id (or an empty context for a fresh trace, so a caller's active span is never adopted); start_child_span nests the guardrail and provider spans under it
• Each credential set gets one LangfuseTracing channel: TracerProvider with a TraceIdHashSampler for LANGFUSE_SAMPLE_RATE, explicit SpanLimits, a Resource built without env detectors, a BatchSpanProcessor with a 100k queue and LANGFUSE_FLUSH_AT, and a LangfuseSpanExporter posting OTLP protobuf through litellm's HTTPHandler with LANGFUSE_TIMEOUT, LANGFUSE_MAX_RETRIES (1, 2, 4 s backoff on 408, 429, 5xx and transport errors) and the x-langfuse-ingestion-version: 4 header
• A 413 splits the batch in halves; a single span that still draws 413 has its largest input, output or metadata group replaced by <truncated due to size exceeding limit> and is resent, up to three times
• acquire_langfuse_tracing and release_langfuse_tracing refcount the channel across loggers; the last release parks it on a 60 s langfuse-retire timer, cancelled if the credentials come back, otherwise the provider is shut down and its batch thread ends
• LangfuseApiClient wraps the SDK's generated LangfuseAPI REST client: auth_check returns None or an AuthCheckFailure reason (status and body, transport error, or no project found), project_id for the Slack link, and get_prompt with a litellm-owned 60 s stale-while-revalidate cache. projects.get and prompts.get run with max_retries=0; a cold prompt fetch URL-encodes the name, makes one immediate second attempt on 5xx or transport error, and raises LangfusePromptError with the status and body but never Langfuse's response headers
• flush_langfuse_tracing flushes every channel for the proxy shutdown hook

litellm/integrations/langfuse/langfuse.py: LangFuseLogger rejects SDK 2.x and 3.x, acquires its channel and REST client from langfuse_sdk, and _log_langfuse_v2 emits the generation with the trace-level attributes (name, user, session, tags, version, release, public flag, trace_metadata, trace input and output), the guardrail and provider-info spans under it, and a stringified generation_name. release falls back to the CI commit variables the v2 SDK read. UPSTREAM_LANGFUSE_* logs a warning and creates nothing. stop() releases the channel lease. The v1 path is deleted

litellm/integrations/langfuse/langfuse_prompt_management.py uses build_langfuse_client and acquire_langfuse_tracing. litellm/litellm_core_utils/specialty_caches/dynamic_logging_cache.py releases the initialized-client count on eviction and calls the evicted logger's stop(). litellm/litellm_core_utils/specialty_caches/service_trace_id_cache.py takes the per-service trace id cache out of litellm_logging.py. litellm/integrations/SlackAlerting/utils.py builds the alert's trace link from the recorded trace id and host instead of constructing a LangFuseLogger. litellm/proxy/proxy_server.py flushes the channels on shutdown in a worker thread with a 10 s cap; /health/services calls auth_check and raises with its reason. litellm/types/integrations/langfuse.py adds LangfuseLoggedEvent

Tests: tests/test_litellm/integrations/langfuse/test_langfuse_sdk.py (new) and the mapped test_langfuse.py, test_langfuse_prompt_management.py, test_dynamic_logging_cache.py and health tests pin every behaviour above. tests/integration/observability/test_langfuse_delivery.py (new) runs a real proxy with Postgres and Redis against scripted provider and Langfuse wire servers, decodes the OTLP protobuf and checks the trace fields, auth, headers and the prompt fetch path; it fails both tests on the merge base. tests/logging_callback_tests/test_langfuse_e2e_test.py decodes OTLP instead of v2 ingestion batches; tests/local_testing/test_alangfuse.py reads back through observations.get_many()

pyproject.toml and uv.lock move to langfuse>=4.7,<5 (locked 4.15.2), opentelemetry 1.33.1, instrumentation 0.54b1 and the traceloop 0.34.0 bump that pin forces. docker/build_from_pip/Dockerfile.build_from_pip is untouched: it installs the published litellm==1.83.0 wheel, whose callback still reads langfuse.version, so its langfuse==2.59.7 and opentelemetry 1.28.0 pins stay and move together with the next LITELLM_VERSION bump

Caveats (if any)

Severe

• langfuse 2.x and 3.x are rejected at logger construction with an upgrade message; on those SDKs the old code lost every event silently. The rejection follows litellm's callback bootstrap: with success_callback = ["langfuse"] the first completion() raises the ImportError and later calls succeed without a Langfuse logger, so no event is sent; with callbacks = ["langfuse"] the error goes to the log only. Base behaves the same way when its constructor raises (an SDK without langfuse.version gives one AttributeError, then silence). Changing that to fail on every call means changing function_setup for every callback, left out of this PR
• Self-hosted Langfuse must be 3.63.0 or newer (the v4 OTLP route); older servers answer 404 and the log names the floor
• UPSTREAM_LANGFUSE_* is no longer supported; a warning is logged. v4 has no equivalent
• A failed request's trace shows the error string as its output; v2 left it null. The observation keeps level: ERROR and the same statusMessage

Medium

• Trace, parent and generation ids are lowercased and de-dashed (a UUID becomes 32 hex characters); anything still not 32 or 16 hex is sha256-hashed. Dashboards keyed on the raw id need the normalised form
• Metadata keys whose value is null are not exported (OpenTelemetry has no null attribute; the v4 SDK drops them too). v2 stored the key with null
• trace_version and generation version share one v4 field (langfuse.version on the span); trace_version wins, at the root and under parent_observation_id alike, as the v4 SDK's propagated attributes overwrite a span's own. v2 kept both on separate events
• modelParameters read back as strings ("stream": "false"), as with the v4 SDK; v2 kept the boolean
• A generation with no usage (a provider error, or a response without a usage block) gets server-side estimated tokens and cost in Langfuse: Cloud tokenizes input and output on the v4 ingestion path when usage is absent, as it does for any v4 SDK sender (live 401: 18 in, 42 out, $0.00135 on head, nothing on v2). Failed requests now add estimated cost to Langfuse totals. Disclosed, not changed
• Nested dicts with non-string keys lose those entries in safe_dumps (Python SDK callers only; proxy JSON always has string keys). Disclosed, not changed
• Input, output and metadata are serialised with litellm's safe_dumps, not the SDK's EventSerializer, so values that are not JSON-native render differently: datetime as 2024-01-02 03:04:05+00:00 (SDK: 2024-01-02T03:04:05Z), an Enum as Color.RED (SDK: its value), bytes as b'hello' (SDK: decoded), a dataclass as its repr (SDK: a dict), float('nan') as a bare NaN token (v2: null, v4 SDK: "NaN"). Python SDK callers only; proxy JSON bodies are unaffected. Disclosed, not changed
• A generation appended to a continued trace (existing_trace_id only) reports a random, unexported parentObservationId instead of null, as the v4 SDK does; the UI still renders it at the root
• Guardrail and provider spans nest under the generation; v2 emitted them as siblings. They do not carry user.id or session.id themselves (neither did v2's); Langfuse reads those from the trace
• A 413 truncation removes a whole input, output or metadata group where v2 trimmed each field; base64 media travels inline (Cloud extracts it server side, a self-hosted ingress body cap triggers the truncation)
• LANGFUSE_SAMPLE_RATE outside 0..1 or non-numeric is logged and treated as 1.0 (v2 returned HTTP 500 or exported nothing); LANGFUSE_FLUSH_INTERVAL not a whole number above 0 warns and uses 1 s; LANGFUSE_MAX_RETRIES above 1000 is capped; LANGFUSE_TIMEOUT is a float, default 20 s, for OTLP and REST; LANGFUSE_DEBUG accepts true or 1; LANGFUSE_BASE_URL is a fallback for LANGFUSE_HOST
• LANGFUSE_PROMPT_CACHE_DEFAULT_TTL_SECONDS is read by the v4 SDK at import with int(); a non-integer raises a named ValueError before that import
• auth_check, project_id and a cold prompt fetch are synchronous on the event loop, as on v2, but with zero sleeping retries: a Langfuse 5xx or 429 fails at once (v2 slept 1 s and retried). Worst case is two attempts of LANGFUSE_TIMEOUT when the destination hangs
• A metadata["prompt"] dict that does not validate as a typed prompt raises during attribute building and loses the trace; v2 kept a separate trace event
• Evicting a logger frees its client-budget slot at once while its channel retires 60 s later, so with MAX_LANGFUSE_INITIALIZED_CLIENTS raised above the 200-entry logger cache, fast credential churn can hold up to one extra channel per evicted key for 60 s (live: 300 keys, 427 threads at burst end, 227 after 60 s; base peaked at 631 and never reclaimed). Default budget 50 is unaffected
• A BatchSpanProcessor that was shut down leaves an otel-sdk at-fork hook that prints TypeError: 'NoneType' object is not callable in a child forked afterwards (SDK users who fork after logging; proxy workers fork before any channel exists)
• Pre-existing and untouched: /health/services builds a fresh logger per probe without releasing its slot; DynamicLoggingCache TTL bookkeeping is unlocked; the lru_cached REST client is not closed on eviction; /v1/messages post-call masking reaches Langfuse unmasked

Low

• Export is best effort as on v2: a batch is dropped after LANGFUSE_MAX_RETRIES failed sends, a killed worker loses its queue, the shutdown hook waits 10 s and logs flushed or flush incomplete. Live chaos losses at this rig: outage 5 of 30, pause 0 of 10, worker kill 1 of 20, restart 0 of 12 (v2: 28, 1, 6, 6)
• Not exercised live: a real Slack workspace (local webhook receiver used), a named end-user client, Datadog or Arize as sibling OTel consumers (the generic otel callback stood in), retry on 501/507/599, prompt names with %, space or #, a transport error on the first prompt attempt, the eviction race (unit tests only for each)
• Anthropic cells could not be compared at this tip: the shared account answers HTTP 400 credit balance is too low; both legs logged the failure generation, last full Anthropic comparison was at an earlier tip
• documentation and code-quality fail on tests/documentation_tests/test_env_keys.py until docs(langfuse): update for Python SDK v4 migration litellm-docs#1103 lands the three new LANGFUSE_* rows; merge that first and rerun both checks
• docker/build_from_pip/Dockerfile.build_from_pip keeps langfuse==2.59.7 and opentelemetry 1.28.0 because it installs the published litellm==1.83.0, whose callback reads langfuse.version; whoever bumps LITELLM_VERSION to a release carrying this PR bumps those pins with it (or drops the langfuse line, since litellm[proxy] then requires langfuse>=4.7). Its CircleCI config has no langfuse callback, so that job would not catch a mismatch
• CodeQL notes one py/import-own-module on from langfuse import ... in langfuse.py; absolute imports resolve to the SDK package, false positive

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

High Risk
This is a breaking observability migration: unsupported Langfuse SDK versions fail at logger construction, trace/id normalization and span nesting differ from v2, and all tracing depends on Langfuse server OTLP support and new export lifecycle on proxy shutdown.

Overview
Migrates the langfuse success/failure callback from Langfuse SDK v2 ingestion to v4’s OTLP path, with dependency pins moved to langfuse>=4.7,<5 and OpenTelemetry 1.33.1.

A new langfuse_sdk layer owns trace export: generations and child spans (guardrails, provider metadata) are emitted as OTel spans with Langfuse v4 attributes, posted via a custom OTLP exporter (retries, 413 batch-split/truncation, mock-mode discarding). REST-only LangfuseApiClient handles prompts, auth_check, and project id—without constructing the SDK’s Langfuse() client. Export channels are refcounted per credential set; loggers get flush()/stop() and cache eviction releases channels instead of calling v2 shutdown() on every client.

LangFuseLogger now enforces supported SDK versions at init, drops UPSTREAM_LANGFUSE_* (warn-only), normalizes hosts via resolve_langfuse_host, and returns typed LangfuseLoggedEvent. Prompt management and Slack trace URLs use the REST client / resolved host; trace ids for alerting are cached after log events. Proxy shutdown flushes all channels (LANGFUSE_SHUTDOWN_FLUSH_TIMEOUT_MILLIS); health checks use api_client.auth_check(). Tests and expected fixtures shift from v2 batch JSON to OTLP-shaped payloads, plus new integration coverage for delivery and prompt fetch behavior.

^{Reviewed by Cursor Bugbot for commit fdbd084. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge

Findings

1.  Eviction retires retained tracing ▶

Summary

The PR migrates the Langfuse callback from the retired v2 SDK interface to a v4 OpenTelemetry and REST implementation

• Adds refcounted tracing channels, OTLP delivery, retry and truncation handling
• Moves prompt, authentication, and project lookups to the Langfuse REST client
• Updates callback lifecycle, Slack trace links, health checks, dependencies, and Langfuse coverage

_{Reviews (62) · Last reviewed commit: "build(docker): keep the pip image's lang..."}

[codecov]

Codecov Report

❌ Patch coverage is 96.40957% with 27 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/langfuse/langfuse_sdk.py	97.59%	13 Missing ⚠️
litellm/integrations/langfuse/langfuse.py	92.91%	9 Missing ⚠️
...ntegrations/langfuse/langfuse_prompt_management.py	89.65%	3 Missing ⚠️
...re_utils/specialty_caches/dynamic_logging_cache.py	83.33%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yucheng-berri]

Eviction flushes queued observations rather than dropping them, now pinned by a test. Exposure matches the pre-existing client shutdown. @greptileai please re-review 3eca824

[yucheng-berri]

This race predates the PR: base already calls flush then shutdown here. v4 only widens it; skipping provider shutdown leaks threads. @greptileai re-review 3eca824

[yucheng-berri]

@greptileai review latest head

[veria-ai]

PR overview

This pull request migrates the Langfuse SDK callback integration to Langfuse v4, including updated span batching and encoding behavior.

Three security issues have been addressed, with one availability concern still open. An authenticated caller can submit oversized prompt or metadata attributes that are encoded without preemptive size limits, potentially causing excessive memory allocation and service disruption. Local per-span and per-batch limits are still needed before serialization.

Open issues (1)

• Low: Unbounded batch encoding enables memory exhaustion — litellm/integrations/langfuse/langfuse_sdk.py:698

Fixed/addressed: 3 · PR risk: 6/10

[yucheng-berri]

All three fixed and verified on a live proxy with a real guardrail: float timings, missing cost_details, and prompt-management client isolation. @greptileai re-review bbad301

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_langfuse_sdk_v4 (fdbd084) with main (57eb3ff)}

[yucheng-berri]

Race predates this PR; base flushed and shut down here too. Triggering it needs the one-hour TTL or 200 credential sets. Both nits fixed. @greptileai re-review 27d8d67

[devin-ai-integration]

Verified end to end against a real Langfuse Cloud project on langfuse 4.7.0. Streaming start times are now the model call, not the callback.

Streaming generation: stored latency 5.17s / TTFT 3.46s vs curl 5.201s / 3.491s

mask_input / mask_output redaction

Failed request recorded as ERROR with statusMessage

existing_trace_id continuation: both generations in one trace

Other scenarios

Full metadata surface, non-uuid trace_id hashing into one trace, langfuse_trace_release header, /health/services?service=langfuse healthy, and the version guard raising ImportError: litellm requires langfuse>=4.7,<5 on 3.15.0 and 2.59.7 all check out. Repo langfuse unit tests: 52 passed

Two things to consider. langfuse_release as a header does not set the trace release since only the langfuse_ prefix is stripped, so langfuse_trace_release is required. On a continuation, the trace API reports the second request's name/release/version (null where unset) while the Cloud UI shows the root observation's, so trace-level fields from the first request effectively get overwritten

Devin session: https://app.devin.ai/sessions/85ed7dd169f0475992a5636b8c2b51b2

[ZhuoRoger]

update?

[yucheng-berri]

@greptileai please review the current head b623e05

[yucheng-berri]

bugbot run

[devin-ai-integration]

Paired Langfuse Cloud UI captures, same six requests through the merge base 5beac4f (Before, langfuse 2.59.7, proxy 4200) and this tip 09cc332 (After, langfuse 4.15.2, proxy 4100), real OpenAI gpt-5.5, one authenticated Cloud project, full 1600x1200 windows. Every request carried the same metadata: trace_user_id, session_id, two tags, generation_name, trace_name, version, custom key foo, a top-level nullkey: null and trace_metadata {foo, nullkey: null}. Before rows took about 15 minutes to appear (Cloud's stated delay for the legacy ingestion route), After rows were visible within seconds. Each image is explained under it

00 Combined trace list, both legs

Search traceTags:uipair-1790230361. Before contributes a separate trace-root row plus its generation (and a guardrail row where one ran); After has the generation as the trace root, so fewer rows per request. After trace ids are 32 hex, Before shows the caller-supplied readable id

01 Plain chat with metadata (POST /v1/chat/completions, "Reply with exactly: UI pair ok")

Trace list filtered to this cell's tag. Before: root row named uipair-before-plain plus a generation row. After: one row, the generation is the root, id is 32 hex

Trace detail. Same trace name, user id, session id, tags, version, input and output on both. Before shows the root with a nested generation, After shows the generation at the root

Generation view. Both: openai/gpt-5.5, 30 tokens, $0.000575, same reply. Model parameters: stream is a boolean on Before and the string "false" on After (disclosed, v4 SDK parity); max_tokens stays numeric on both

Metadata tab. Before lists the top-level nullkey: null. After omits that key (OpenTelemetry has no null attribute, disclosed); foo: bar, requester metadata and the litellm keys match otherwise

After only: the nested nullkey: null inside requester_metadata and trace_metadata is kept, since nested dicts travel as one JSON string

Sessions page for this cell's session_id. One trace with the same reply on both legs

02 Guardrail masking (POST /v1/chat/completions with the mask-post guardrail)

Trace list. Before: root, generation and guardrail as three rows. After: generation root with the guardrail nested under it

Trace detail. Output is I like [KEYWORD_REDACTED] pizza on both. Tree shape is the disclosed change: Before has generation and guardrail as siblings under the root, After has the guardrail as a child of the generation

Generation view. Same model, tokens, cost and masked output on both

Metadata tab. Same keys apart from the dropped top-level null, guardrail information present on both

Guardrail observation. Same name, mode, keyword and MASK action on both; only its parent differs

03 Prompt management (alias langfuse/gpt-5.5 with prompt_id litaudit-lf5050-greeting)

Trace list. Same row shape difference as the plain cell

Trace detail. Compiled prompt messages with who=auditor and token=UI-PAIR match on both. Before's assistant content is empty with finish_reason: length (gpt-5.5 reasoning consumed max_tokens 100, provider side, not the callback), After got Arr auditor, UI-PAIR token acknowledged

Generation view. Both carry the prompt link litaudit-lf5050-greeting v2, so the prompt-to-generation association survives the migration

Metadata tab. Prompt name and version recorded on both

04 turn_off_message_logging (guardrail request with redaction on)

Trace list. Same shape difference, rows present on both

Trace detail. Input and output read redacted-by-litellm on both legs while the client still received the masked reply

Generation view. Redacted bodies, real model, tokens and cost on both

Metadata tab. Same keys on both, no message content

Guardrail observation. Pre-existing gap visible on both legs: the guardrail's own metadata still shows the matched keyword and MASK action under redaction. Unchanged by this PR

05 Provider 401 (alias with an invalid OpenAI key)

Trace list. Both legs logged the failed request; After shows a token and cost badge on the row, Before does not

Trace detail. Both trees show the generation as ERROR. Before's trace page has only the user input and the trace metadata (foo, nullkey), After's trace page shows the sanitized invalid-key message as the assistant output (disclosed, v2 left trace output null) plus the generation's full metadata, because the generation is the trace root. The header also carries the estimated cost $0.00135 and tokens 60 badges on After only

Generation view. Neither leg sent usage. Before shows no tokens or cost, After shows 60 tokens and $0.00135: Cloud tokenizes input and output itself on the v4 ingestion route when usage is absent, the same for any v4 SDK sender. Now in the Caveats

Metadata tab. Same request metadata on both, no key material in either

06 Responses endpoint (POST /v1/responses, same prompt)

Trace list. Same shape difference as the chat cells

Trace detail. UI pair ok on both, Cloud renders the encrypted reasoning item as Thinking (redacted) on both

Generation view. openai/gpt-5.5, 31 tokens, $0.000605 on both

Metadata tab. Same litellm_metadata keys on both apart from the dropped top-level null

Not captured in this pair: /v1/messages. Request and response payloads plus the Cloud API readback for every cell are in the audit evidence directory named in the PR body

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fdbd084. Configure here.}

[yucheng-berri]

@greptileai review latest head and rescore, the Eviction retires retained tracing finding is the two resolved threads above, measured there: base loses the event at once, head keeps it for 60 s. Pre-existing and disclosed