fix(langfuse): shutdown client on session finalize to avoid interpreter-teardown TypeError - #81054
Closed
bgodlin wants to merge 2 commits into
Closed
fix(langfuse): shutdown client on session finalize to avoid interpreter-teardown TypeError#81054bgodlin wants to merge 2 commits into
bgodlin wants to merge 2 commits into
Conversation
…er-teardown TypeError The langfuse plugin never called client.shutdown(), relying on the SDK's atexit handler. That fires during interpreter finalization, after opentelemetry.trace.Span is torn down to None — use_span's isinstance(span, Span) raises TypeError, surfaced as 'Exception ignored in: <generator>' on quit. Register on_session_finalize to call client.shutdown() while the interpreter is alive.
alt-glitch
added
type/bug
Collaborator
Duplicate of #80674 — it already implements the same session-finalize Langfuse client shutdown, and also closes suspended root contexts. |
…down TypeError The on_session_finalize hook (added in the previous commit) calls client.shutdown() but that only flushes the SDK's internal queues. It does not unwind the root observation context managers the plugin itself created: _start_root_trace enters start_as_current_observation(...).__enter__() but _finish_trace only called root_span.end(), never root_ctx.__exit__(). The generator stays suspended inside 'with otel_trace_api.use_span(parent_span):' until the GC collects it during interpreter teardown. By then opentelemetry.trace.Span has been torn down to None, and use_span's isinstance(span, Span) raises: TypeError: isinstance() arg 2 must be a type surfaced as 'Exception ignored in: <generator>' on every CLI exit. Fix: call root_ctx.__exit__(None, None, None) right after root_span.end() in both _finish_trace and _evict_stale_locked. This unwinds the generator while all modules are intact. Regression test: test_finish_trace_exits_root_context_manager verifies __exit__ is called and fails on the pre-fix code.
Contributor
|
adopted into #83437 with authorship preserved (both commits cherry-picked). one interaction fix on top: the shutdown is gated on reason == "shutdown" there, since on_session_finalize also fires on /new, /reset, and gateway session expiry where the process keeps running and the client must stay alive. |
kshitijk4poor
added a commit
that referenced
this pull request
Aug 13, 2026
… fan-out Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054), @aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress (#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797), and @liuhao1024 (#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes #29482, #43129, #72661. Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130. Partially addresses #67544 (capture modes + secret redaction; user_id remains open).
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Quitting a Hermes session with the
observability/langfuseplugin enabled prints a noisy traceback:Root cause
The langfuse plugin never called
client.shutdown(). It relied entirely on the Langfuse SDK's ownatexit.register(self.shutdown)handler (resource_manager.py:279).That atexit handler fires during interpreter finalization — by then module globals (notably
opentelemetry.trace.Span) may already be torn down toNone. The SDK's span-finalization path runsuse_span→isinstance(span, Span)(opentelemetry/trace/__init__.py:597), andSpanbeingNoneraises theTypeError. Python suppresses it as "Exception ignored in: ".This is cosmetic (no data loss — spans were already queued), but every langfuse-enabled session sees it on quit.
Fix
Register an
on_session_finalizehook in the plugin that explicitly callsclient.shutdown()while the interpreter is still alive.on_session_finalizefires from the normal CLI exit path (cli.py:1228→lifecycle.finalize_session), not from atexit — so all modules are intact.client.shutdown()flushes pending spans and joins the background export threads. The SDK's own atexit handler then becomes a no-op (it checks_shutdownand unregisters itself), so the race with interpreter teardown never starts.Test plan
tests/plugins/test_langfuse_plugin.py— 24/24 pass (updated manifest assertion for the new hook)tests/hermes_cli/test_lifecycle.py— 3/3 pass