Skip to content

fix(langfuse): send v4 ingestion header for otel callback - #33907

Merged
yucheng-berri merged 6 commits into
litellm_internal_stagingfrom
litellm_langfuse_otel_v4_header
Jul 19, 2026
Merged

fix(langfuse): send v4 ingestion header for otel callback#33907
yucheng-berri merged 6 commits into
litellm_internal_stagingfrom
litellm_langfuse_otel_v4_header

Conversation

@yucheng-berri

@yucheng-berri yucheng-berri commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Adopted from #26105

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

No live-proxy capture against a Langfuse project yet. What is verified so far is the header construction itself, at commit afca40da82:

$ python -m pytest tests/test_litellm/integrations/test_langfuse_otel.py \
    tests/test_otel_thread_leak.py tests/test_service_logger_otel.py -q
35 passed in 3.74s

The three previously-passing assertions in tests/test_litellm/integrations/test_langfuse_otel.py that pinned the exporter headers to {"Authorization": ...} alone now assert the v4 header alongside it, which is the behavior change this PR is making

Type

Bug Fix

Changes

langfuse_otel was sending only the OTLP Authorization header. Langfuse routes spans that arrive without x-langfuse-ingestion-version: 4 through its older transformation path rather than the real-time v4 ingestion path, which surfaces as delayed trace visibility

This adds x-langfuse-ingestion-version: 4 to every place LiteLLM builds Langfuse OTLP headers:

  • _build_langfuse_otel_config, which now backs both the env-based config and the per-request key/team dynamic config
  • construct_dynamic_otel_headers, the dynamic per-request header path

Two helpers carry it: _build_langfuse_otel_headers returns the header mapping, _format_otel_headers serializes that mapping into the comma-separated OTLP header string

Behavior changes

  • Langfuse OTLP exports now carry x-langfuse-ingestion-version: 4 in addition to Authorization, on both the env-configured exporter and dynamically-credentialed per-key exporters. Langfuse will ingest these spans through its v4 path instead of the legacy transformation path
  • No change to endpoint resolution, auth, or span content

Credit

Adopted from #26105 by @hassiebp. Mirrored onto a litellm_ branch so CircleCI and the internal lint workflow run

The original branch conflicted with litellm_internal_staging, which had since refactored both config paths into a single _build_langfuse_otel_config helper. The v4 header is reapplied on top of that refactor, so the change now lands in one place instead of two. The original author's commits are preserved in this branch's history

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Open in Devin Review

Note

Low Risk
Narrow observability export change: only OTLP HTTP headers are extended; auth, endpoints, and span payloads are unchanged.

Overview
Langfuse OTLP exports from LangfuseOtelLogger now include x-langfuse-ingestion-version: 4 alongside Basic auth, so spans use Langfuse’s v4 ingestion path instead of the legacy transformation route (which could delay trace visibility).

Header construction is centralized in _build_langfuse_otel_headers and _format_otel_headers, used by env-based _build_langfuse_otel_config and per-key construct_dynamic_otel_headers. Tests were updated to expect the new header on static config, dynamic config, and exporter instances; test_service_logger_otel adds coverage for env and dynamic header paths.

Reviewed by Cursor Bugbot for commit 181b880. Bugbot is set up for automated code reviews on this repo. Configure here.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ yucheng-berri
❌ hassiebp
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@greptile-apps

greptile-apps Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds x-langfuse-ingestion-version: 4 to every Langfuse OTLP header path so spans are ingested via Langfuse's v4 pipeline instead of the legacy transformation path, which had been causing delayed trace visibility.

  • Two static helpers (_build_langfuse_otel_headers, _format_otel_headers) centralize header construction and serialization; both the env-based _build_langfuse_otel_config path and the per-request construct_dynamic_otel_headers path now call them.
  • Existing test assertions are strengthened to require the new header, and two new tests cover the env-config and dynamic-header paths end-to-end.

Confidence Score: 5/5

Safe to merge — the change is additive (an extra OTLP header), touches no auth logic or endpoint resolution, and is fully covered by updated and new tests.

Header construction is centralized in two small static helpers, both the env-based and dynamic-credential paths are updated consistently, and all affected test assertions are strengthened rather than relaxed. No pre-existing concerns were introduced or left unaddressed.

No files require special attention.

Important Files Changed

Filename Overview
litellm/integrations/langfuse/langfuse_otel.py Adds two constants and two static helpers; wires both the env-config and dynamic-header paths through _build_langfuse_otel_headers to include x-langfuse-ingestion-version: 4. Logic is correct and serialization is consistent with the pre-existing key=value OTLP header format.
tests/test_litellm/integrations/test_langfuse_otel.py Existing assertions on config.headers and exporter._headers are strengthened (not weakened) to require the new x-langfuse-ingestion-version=4 entry; no regression risk.
tests/test_service_logger_otel.py Adds two new async tests (env-config path and dynamic-header path) that verify the v4 header is present; relies on the existing setUp credentials, which are set unconditionally before every test in the class.

Reviews (5): Last reviewed commit: "chore(langfuse): drop stale development ..." | Re-trigger Greptile

@greptile-apps

greptile-apps Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR appends x-langfuse-ingestion-version: 4 to every Langfuse OTLP export so that spans are routed through Langfuse's v4 ingestion path rather than the legacy transformation path, which previously caused delayed trace visibility. The header is injected via two new static helpers (_build_langfuse_otel_headers, _format_otel_headers) and flows through both the env-based exporter config (_build_langfuse_otel_config) and the per-request dynamic headers (construct_dynamic_otel_headers).

  • _build_langfuse_otel_headers returns a {\"Authorization\": …, \"x-langfuse-ingestion-version\": \"4\"} dict; _format_otel_headers serializes it to the comma-separated OTLP header string, which _get_headers_dictionary already parses correctly via first-= splitting.
  • Existing tests in test_langfuse_otel.py were correctly updated to assert both headers; new tests were added in test_service_logger_otel.py, but one of them (test_langfuse_otel_env_config_includes_v4_ingestion_header) is not self-contained — it needs LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY set in the environment to reach the Langfuse config branch rather than falling back to the generic from_env() path.

Confidence Score: 4/5

The production code change is straightforward and correct; the only concern is a new test that will silently pass or fail depending on whether Langfuse env vars happen to be set in the runner's environment.

The core change — adding a constant header via two well-isolated helper methods — is clean and matches the intended fix. The existing test_langfuse_otel.py assertions are updated correctly and cover the key paths. The test_langfuse_otel_env_config_includes_v4_ingestion_header test in test_service_logger_otel.py does not set up LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY, so in a clean environment it exercises the wrong code branch and raises KeyError rather than verifying the v4 header; this makes the test unreliable across environments.

tests/test_service_logger_otel.py — the new env-config test needs explicit env-var setup to be reliable.

Important Files Changed

Filename Overview
litellm/integrations/langfuse/langfuse_otel.py Adds LANGFUSE_INGESTION_VERSION_HEADER constant and two new helpers (_build_langfuse_otel_headers, _format_otel_headers) to attach x-langfuse-ingestion-version: 4 to every Langfuse OTLP export, both env-configured and per-request dynamic. Minor stale comment on import json line.
tests/test_litellm/integrations/test_langfuse_otel.py Updates three existing assertions to include x-langfuse-ingestion-version: 4 alongside the Authorization header; changes correctly reflect the new production behavior.
tests/test_service_logger_otel.py Adds two new async tests for the v4 header, but test_langfuse_otel_env_config_includes_v4_ingestion_header is environment-dependent — it raises KeyError on x-langfuse-ingestion-version when LANGFUSE_PUBLIC_KEY/SECRET_KEY are absent, making it unreliable in clean CI runs.

Reviews (2): Last reviewed commit: "test(langfuse): assert v4 ingestion head..." | Re-trigger Greptile

greptile-apps[bot]

This comment was marked as resolved.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai on the 4/5 concern about test_langfuse_otel_env_config_includes_v4_ingestion_header being environment-dependent: this is not accurate.

TestServiceLoggerOTEL.setUp already sets both credentials before every test in the class, so the Langfuse config branch is always reached and from_env() is never hit:

def setUp(self):
    # Reset callbacks before each test
    litellm.service_callback = []
    os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-lf-123"
    os.environ["LANGFUSE_SECRET_KEY"] = "sk-lf-123"

Those four lines are pre-existing on litellm_internal_staging and are untouched by this PR - git show origin/litellm_internal_staging:tests/test_service_logger_otel.py shows them identically.

Verified empirically at commit 6e38363ea0 with the variables explicitly unset, both for the file and for the flagged test in isolation:

$ env -u LANGFUSE_PUBLIC_KEY -u LANGFUSE_SECRET_KEY -u LANGFUSE_HOST \
    python -m pytest tests/test_service_logger_otel.py -q -p no:randomly
5 passed in 3.17s

$ env -u LANGFUSE_PUBLIC_KEY -u LANGFUSE_SECRET_KEY python -m pytest \
    'tests/test_service_logger_otel.py::TestServiceLoggerOTEL::test_langfuse_otel_env_config_includes_v4_ingestion_header' -q
1 passed in 2.94s

No KeyError, and the test asserts the v4 header on the real Langfuse branch rather than a fallback.

Separately, the previous commit fixed the ruff format lint failure, so please re-score on 6e38363ea0

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@codecov

codecov Bot commented Jul 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 6e38363. Configure here.

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

Both findings from the last review are addressed on 181b880f15:

  • P2 stale # <--- NEW annotation on the json import: removed
  • P1 environment-dependent test: rebutted on the thread. TestServiceLoggerOTEL.setUp already sets LANGFUSE_PUBLIC_KEY/LANGFUSE_SECRET_KEY for every test in the class, and those lines are pre-existing on litellm_internal_staging. Verified passing with both variables explicitly unset

Please re-score on 181b880f15

@yucheng-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 181b880. Configure here.

@yucheng-berri
yucheng-berri merged commit bd44c9e into litellm_internal_staging Jul 19, 2026
80 checks passed
@yucheng-berri
yucheng-berri deleted the litellm_langfuse_otel_v4_header branch July 19, 2026 03:36
@codspeed-hq

codspeed-hq Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_langfuse_otel_v4_header (181b880) with litellm_internal_staging (ac1b4b3)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants