feat(langfuse): accept OTLP trace exports for the Langfuse v4 callback

[yuneng-berri]

TLDR

Problem this solves:

• LiteLLM's Langfuse callback now exports over OTLP (feat(langfuse)!: migrate the sdk callback to langfuse v4 litellm#36741)
• This mock has no OTLP route, so exports get 404
• test_e2e_langfuse_callbacks_in_db in litellm has been red on every CircleCI run since

How it solves it:

• New POST /api/public/otel/v1/traces parses the OTLP protobuf export
• It records each span's langfuse.observation.metadata.response_id into the set /langfuse/trace/{id} already reads
• A body that isn't valid OTLP protobuf gets a 400
• Adds opentelemetry-proto==1.33.1, which supports the Dockerfile's Python 3.9

Screenshots / Proof of Fix

Shared setup: this mock on 127.0.0.1:18090 (Python 3.9 venv from requirements.txt), and a litellm proxy from BerriAI/litellm main (1a58162630) on 127.0.0.1:14100 with one model fake-gpt pointed at the mock and success_callback: ["langfuse"], LANGFUSE_HOST=http://127.0.0.1:18090

Before (ab53af6)

1. curl -s http://127.0.0.1:14100/v1/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model":"fake-gpt","messages":[{"role":"user","content":"hi"}]}' returns id chatcmpl-83c4c6cb255b4dab8c0f3b96959596e0
2. Mock log: POST /api/public/otel/v1/traces HTTP/1.1" 404 Not Found, and the proxy logs Langfuse rejected an export ... with HTTP 404
3. curl -s http://127.0.0.1:18090/langfuse/trace/chatcmpl-83c4c6cb255b4dab8c0f3b96959596e0 returns {"exists":false,...}, the same failure CircleCI shows

After (f6d4d0c)

1. The same completion request returns id chatcmpl-74a6c7ce65494a9c9cb50a5f063e8e85
2. Mock log: Stored OTLP response IDs (total: 1): {'chatcmpl-74a6c7ce65494a9c9cb50a5f063e8e85'} then POST /api/public/otel/v1/traces HTTP/1.1" 200 OK
3. curl -s http://127.0.0.1:18090/langfuse/trace/chatcmpl-74a6c7ce65494a9c9cb50a5f063e8e85 returns {"exists":true,...}
4. curl -s http://127.0.0.1:18090/langfuse/trace/chatcmpl-never-sent still returns {"exists":false,...}
5. Posting a garbage body to /api/public/otel/v1/traces returns 400

Caveats (if any)

Medium

• The litellm test only goes green once this is deployed to the Railway service it calls