fix(langfuse): stop a collected httpx handler from closing a shared client

[yassin-berriai]

TLDR

Problem this solves:

• Langfuse traces stop silently until restart, no error surfaced
• A collected HTTP handler closed a client other holders still used
• Hits per-key and per-team Langfuse credentials

How it solves it:

• A handler finalizer closes only a client it solely referred to
• Pooled sockets still released promptly, measured against base
• close() respects _owns_client, so injected clients survive

Relevant issues

Linear ticket

Resolves LIT-5228

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on port 15228, real gpt-4o-mini calls billed to a real key, real HTTP ingestion. There are no Langfuse Cloud credentials on this machine, so a local HTTP server on port 15229 stands in for the Langfuse ingest endpoint and prints every event it receives. Everything else is the real path: the real Langfuse SDK and its real background flush thread posting to /api/public/ingestion over the client under test

The bug needs the shared client cache entry to go away, which normally happens on the 1 hour TTL or under the 200 entry size cap. Neither is drivable from curl, so evict_callback.py below is loaded as a proxy callback and forces exactly that event on demand when a request carries "user": "lit5228-evict"

Setup, identical for both runs:

# evict_callback.py
import gc

import litellm
from litellm.caching.llm_caching_handler import LLMClientCache
from litellm.integrations.custom_logger import CustomLogger


class CacheEvictor(CustomLogger):
    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        if "lit5228-evict" in str(data.get("user", "")):
            litellm.in_memory_llm_clients_cache = LLMClientCache()
            print(f"[LIT-5228] evicted cache, gc.collect() reclaimed {gc.collect()} objects", flush=True)
        return data


cache_evictor_instance = CacheEvictor()

# config.yaml
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  success_callback: ["langfuse"]
  callbacks: evict_callback.cache_evictor_instance

general_settings:
  master_key: sk-lit5228
  allow_client_side_credentials: true

1. Start the ingest stand-in, then the proxy

python langfuse_sink.py 15229 &
python litellm/proxy/proxy_cli.py --config config.yaml --port 15228 --num_workers 1

2. Send a request carrying per-tenant Langfuse credentials

curl -s http://127.0.0.1:15228/v1/chat/completions -H 'Authorization: Bearer sk-lit5228' -H 'Content-Type: application/json' \
 -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"say LIT5228-PROXY-BEFORE"}],"max_tokens":10,
      "langfuse_public_key":"pk-lf-lit5228-tenant","langfuse_secret":"sk-lf-lit5228-tenant","langfuse_host":"http://127.0.0.1:15229",
      "metadata":{"generation_name":"LIT5228-PROXY-BEFORE"}}'

3. Expire the shared client cache entry

curl -s http://127.0.0.1:15228/v1/chat/completions -H 'Authorization: Bearer sk-lit5228' -H 'Content-Type: application/json' \
 -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"evict"}],"max_tokens":5,"user":"lit5228-evict"}'

4. Send the same tenant request again, with LIT5228-PROXY-AFTER

Before, at 2792887e47d698edaeb8a2d0ad1abfc9576609a6

Steps 2 and 4 both return 200 and the caller sees nothing wrong. An instrumented build shows the tenant logger's ingestion client dying across step 3:

[LIT-5228] before evict: LangFuseLogger client id=4924741584 is_closed=False
[LIT-5228] evicted cache, gc.collect() reclaimed 204 objects
[LIT-5228] after evict:  LangFuseLogger client id=4924741584 is_closed=True

Full ingest log for the run:

SINK RECEIVED trace-create name=litellm-acompletion
SINK RECEIVED generation-create name=LIT5228-PROXY-BEFORE
SINK RECEIVED trace-create name=litellm-acompletion
SINK RECEIVED generation-create name=litellm-acompletion

LIT5228-PROXY-AFTER never arrives. The two middle lines are the eviction request itself, logged through the global callback, which uses a different client

After, at 9478d25b63

Same four steps, same proxy, same real OpenAI calls. Step 3 now reports the client staying open:

[LIT-5228] evicted cache, gc.collect() reclaimed 204 objects
[LIT-5228] after evict: LangFuseLogger client id=4950011152 is_closed=False

Full ingest log for the run:

SINK RECEIVED trace-create name=litellm-acompletion
SINK RECEIVED generation-create name=LIT5228-PROXY-BEFORE
SINK RECEIVED trace-create name=litellm-acompletion
SINK RECEIVED generation-create name=litellm-acompletion
SINK RECEIVED trace-create name=litellm-acompletion
SINK RECEIVED generation-create name=LIT5228-PROXY-AFTER

LIT5228-PROXY-AFTER now lands

Pooled socket reclamation, before and after

Touching a finalizer that closes clients risks trading a silent logging outage for a socket leak, so this is measured rather than argued. The first version of this harness could not have found anything: its test server sent Connection: close, so nothing was ever pooled and both arms reported a flat sockets=1, which is indistinguishable from a clean result. A harness that cannot observe the resource it is measuring reports success. With keep-alive the difference was immediate, and that is why the finalizer is guarded here rather than deleted. 2000 handler create-request-drop cycles against a local keep-alive server, no forced gc.collect() at any point, sockets counted with psutil, live objects counted with gc.get_objects(). Both runs print litellm.__file__ so the tree under test is on the record

[BASE]        start  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=0
[BASE]   cycle 1000  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=8
[BASE]   cycle 2000  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=15

[BRANCH]      start  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=0
[BRANCH] cycle 1000  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=15
[BRANCH] cycle 2000  sockets=1    httpx.Client=0  HTTPHandler=0  ConnectionPool=11

Sockets stay pinned at 1, the listening socket, on both. Live pool counts oscillate in the same bounded band on both. An earlier revision of this change dropped the finalizer outright and that showed real drift, branch sockets climbing to 29 while base held at 1, which is why the sole-referrer guard exists instead

Repeated LangFuseLogger() construction, the shape at _health_endpoints.py:291, also tracks base exactly: 40 constructions leave 1 live httpx.Client on both builds

Why the finalizer has to stay at all, rather than trusting refcounting. The handler has no reference cycle and is reclaimed by refcounting, but on the httpcore transport the client's connection pool IS cyclic, so dropping the client does not return its file descriptor. Server in a separate process so only the client side is counted, gc disabled, handler confirmed unreferenced at the point of measurement:

AsyncHTTPHandler, httpcore transport, WITH the guarded finalizer
  baseline                          EST=0  ALL=0  FDS=9
  after request                     EST=1  ALL=1  FDS=10
  after `del h`, gc OFF, no yield   EST=1  ALL=1  FDS=10   pool alive? True
  after a loop turn, gc still OFF   EST=0  ALL=0  FDS=9    <- finalizer released it, no gc involved
  after gc.collect()                EST=0  ALL=0  FDS=9

same handler, same transport, with the guard forced to never fire
  after `del h`, gc OFF, no yield   EST=1  ALL=1  FDS=10   pool alive? True
  after a loop turn, gc still OFF   EST=1  ALL=1  FDS=10   <- still held
  after gc.collect()                EST=0  ALL=0  FDS=9    <- only the cycle collector frees it

That pair is the argument for keeping a guarded finalizer instead of deleting one: with it, the descriptor comes back without any collection pass; without it, the descriptor waits for the cycle collector

This is transport-dependent and the PR should not overstate it. Under litellm's default async aiohttp transport the session is NOT cyclic, dies by refcount, and the descriptor returns on the next loop turn whether or not the finalizer fires. The cyclic case is httpcore, which is every HTTPHandler (there is no sync aiohttp transport) and any AsyncHTTPHandler with aiohttp disabled. HTTPHandler is the one Langfuse uses and the one the 2000-cycle measurement above exercises, so the finalizer is load-bearing exactly where this change touches

These are two different experiments and are not interchangeable: the 2000-cycle keep-alive numbers are about REMOVING the finalizer under handler churn, and the transcripts here are about whether a single dropped handler returns its descriptor. Both matter, neither is evidence for the other

One related change on the async side, and it is an improvement on the previous behaviour rather than a restoration of it: the finalizer now schedules self._client.aclose() rather than self.close(). The old form handed self to a coroutine mid-collection, which resurrected the handler and delayed its reclamation by a loop turn. Closing over the client instead gets both properties, the handler is reclaimed immediately by refcounting and the client it owned is still torn down

Known limitation

_CLIENT_REFCOUNT_WHEN_HANDLER_IS_SOLE_REFERRER is a CPython reference-counting assumption. requires-python allows <3.15, so free-threaded 3.13t and 3.14t are in scope, and sys.getrefcount is not meaningful under deferred reference counting there. The expectation is that it reads high and so fails safe, declining to close and leaving the client to the collector rather than closing one that is still in use, which is the same behaviour as the guard being conservative. That is reasoning, not a measurement, and this is untested on a free-threaded build

The constant is also sensitive to how it is read: binding the client to a local before the call adds a reference and reads 3 instead of 2, which would silently stop the guard from ever firing. Both finalizers are covered by a test that fails in exactly that case, so the drift cannot land unnoticed

Type

🐛 Bug Fix

Changes

_get_httpx_client() and get_async_httpx_client() return a handler cached process-wide in litellm.in_memory_llm_clients_cache, and at the Langfuse call site the handler itself was a local that went out of scope immediately. A consumer that takes handler.client and keeps the raw httpx.Client therefore outlives the handler. Once the cache entry expires on its TTL or is evicted under the 200 entry cap, nothing references the handler, it is collected, and its __del__ called close() on the client the consumer is still using

A finalizer firing proves only that nothing references the handler. It proves nothing about the client, which may be held by anyone the handler gave it to. Both handlers now close during finalization only when they built the client and are still its sole referrer, read from the refcount at the call site. That preserves prompt teardown for the ordinary unshared case, which is what the finalizer was actually buying, while a handed-out client is left for its real owners. LLMClientCache's own docstring already promised this: "This cache intentionally does NOT close clients on eviction ... Clients that are no longer referenced will be garbage-collected normally"

Explicit close() stays, that is a caller deliberately shutting a handler down, but it now honours _owns_client. The setter and the client= constructor argument both record that a caller supplied the client, and a wrapper should not close something it was handed. __aexit__ routes through close() so the two agree

The second half is the Langfuse call site keeping a reference to the handler that owns the client it hands the SDK, so the handler cannot be collected out from under it while the logger lives. It still takes the client from the shared cache, so no additional clients are created per logger. This is defence in depth rather than the load-bearing fix: the sole-referrer guard alone stops the outage, and this makes the Langfuse client's liveness independent of that guard being right

Which half stops the outage: the finalizer guard. Which half is hardening: the Langfuse reference

Reachability

LangFuseLogger has four constructors in-tree and they are not equally exposed. success_callback: ["langfuse"] resolves through custom_logger_registry.py to LangfusePromptManagement, which builds its own standalone client and never touches the cached handler, so that object is not affected. Instrumenting the constructor on a live proxy shows litellm_logging.py:3514 does still fire once per process for that configuration, so a vulnerable logger is constructed on the default setup, but it is not the object that performs the logging and no trace loss follows. It fires once, not per request

The path that both constructs the vulnerable logger and logs through it is LangFuseHandler._create_langfuse_logger_from_credentials, reached with per-key or per-team Langfuse credentials and cached long-lived in the DynamicLoggingCache. That is the configuration the proof above reproduces. _health_endpoints.py:291 constructs one per health check, measured above as no worse than base

Tests

tests/test_litellm/llms/custom_httpx/test_http_handler.py covers both directions: a handler must not close a client it does not own, a consumer holding a client obtained from the cache must survive eviction plus a forced collection, and an exclusively owned client's connection pool must still be closed when its handler is collected, so the guard cannot be quietly widened into a no-op. The eviction tests use a weakref to assert the handler really was collected rather than trusting that it was, and the pool tests assert the pooled connection existed before asserting it went away

tests/test_litellm/integrations/test_langfuse.py covers the client handed to the Langfuse SDK surviving cache eviction, and the loggers continuing to share one cached client

This closes a gap in tests/test_litellm/llms/test_lifecycle_fix.py, which drops a handler and then asserts only that an unrelated client is still open, never asserting anything about the client the dropped handler was holding

Every executable line this PR adds is exercised: 15 of 15, checked with coverage locally per changed line. Both directions are pinned for both handlers, verified by mutation rather than assumed: widening the guard so it fires on a shared client is caught by the survival tests, narrowing it so it never fires is caught by the pool-closure tests, and rebinding the client to a local at either call site is caught by the matching pool-closure test. An earlier revision of the async closure test passed under that last mutant because it allowed the client to have been collected instead of closed, so it was rewritten to assert on the connection pool the same way the sync one does

QA runbook

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

The PR changes HTTP client ownership handling so cache eviction and handler finalization do not close clients still used by Langfuse or other consumers.

• Retains the cached HTTP handler for each Langfuse logger’s lifetime.
• Makes explicit close and context-manager cleanup respect client ownership.
• Adds sync and async lifecycle tests covering shared-client survival and owned-client cleanup.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/integrations/langfuse/langfuse.py	Retains the cached HTTP handler alongside the client passed to the Langfuse SDK, preventing premature finalization.
litellm/llms/custom_httpx/http_handler.py	Introduces ownership- and refcount-aware client cleanup for synchronous and asynchronous HTTP handlers.
tests/test_litellm/integrations/test_langfuse.py	Adds coverage for Langfuse client survival across cache eviction and reuse of the shared cached client.
tests/test_litellm/llms/custom_httpx/test_http_handler.py	Adds lifecycle coverage for injected clients, cache eviction, finalization, and connection-pool cleanup.

_{Reviews (4): Last reviewed commit: "fix(langfuse): stop a collected httpx ha..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please re-review f266024. The design changed since the initial open: the finalizer is kept but guarded, rather than removed.

[yassin-berriai]

@greptileai please re-review bebf74c. Adds regression tests pinning the finalizer guard in both directions for the sync and async handlers.

[yassin-berriai]

@greptileai please re-review 9478d25. Adds an __aexit__ ownership test, bringing every changed line under test.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_langfuse_shared_client_close (9478d25) with litellm_internal_staging (2792887)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (32deaff) during the generation of this report, so 2792887 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩