fix(langfuse): finalize open root spans at exit so short-lived processes export complete traces - #82332
Closed
aldoeliacim wants to merge 1 commit into
Closed
Conversation
…ses export complete traces Langfuse-driven correctness review, bug 3: kanban workers / hermes chat -q / cron one-shots exit while the final LLM response still has tool calls queued, so _finish_trace never runs. The SDK's own atexit flush exports the ENDED children but the root span never lands: the backend shows an anonymous trace — name '', no sessionId, empty metadata — with 17-37 orphaned observations (4 seen live in one kanban tick, 33/50 of recent traces anonymous). _finalize_all_traces drains _TRACE_STATE and ends every open root; registered with atexit AFTER Langfuse client construction (atexit is LIFO → runs before the SDK's shutdown flush, so the ended spans still export). Idempotent; fail-open per span. RED-verified: both new tests fail with the fix stashed; 29/29 green with it.
Contributor
alt-glitch
added
type/bug
kshitijk4poor
added a commit
that referenced
this pull request
Aug 13, 2026
… fan-out Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054), @aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress (#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797), and @liuhao1024 (#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes #29482, #43129, #72661. Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130. Partially addresses #67544 (capture modes + secret redaction; user_id remains open).
Collaborator
|
Merged via #85439 — your fix was adopted and composed into the wider Langfuse tracing PR by @erosika. Your contribution is credited in the commit body. Thanks @aldoeliacim! |
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
… fan-out Salvaged from PR NousResearch#83437 by @erosika, with adopted fixes from @bgodlin (NousResearch#81054), @aldoeliacim (NousResearch#82332), @nftpoetrist (NousResearch#42326), @rodboev (NousResearch#39653), @FnExpress (NousResearch#64292, supersedes NousResearch#32175 by @db-aeon), @Per0-1 (NousResearch#61166), @NaMinhyeok (NousResearch#64797), and @liuhao1024 (NousResearch#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes NousResearch#29482, NousResearch#43129, NousResearch#72661. Supersedes NousResearch#81054, NousResearch#82332, NousResearch#42326, NousResearch#39653, NousResearch#64292, NousResearch#32175, NousResearch#61166, NousResearch#64797, NousResearch#43130. Partially addresses NousResearch#67544 (capture modes + secret redaction; user_id remains open).
bobaba76
pushed a commit
to bobaba76/hermes-agent
that referenced
this pull request
Aug 27, 2026
… fan-out Salvaged from PR NousResearch#83437 by @erosika, with adopted fixes from @bgodlin (NousResearch#81054), @aldoeliacim (NousResearch#82332), @nftpoetrist (NousResearch#42326), @rodboev (NousResearch#39653), @FnExpress (NousResearch#64292, supersedes NousResearch#32175 by @db-aeon), @Per0-1 (NousResearch#61166), @NaMinhyeok (NousResearch#64797), and @liuhao1024 (NousResearch#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes NousResearch#29482, NousResearch#43129, NousResearch#72661. Supersedes NousResearch#81054, NousResearch#82332, NousResearch#42326, NousResearch#39653, NousResearch#64292, NousResearch#32175, NousResearch#61166, NousResearch#64797, NousResearch#43130. Partially addresses NousResearch#67544 (capture modes + secret redaction; user_id remains open).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Short-lived Hermes processes (kanban workers,
hermes chat -qone-shots, cron jobs) export anonymous traces to Langfuse: no name, nosessionId, empty metadata — just a bag of orphaned observations. On a board running a few parallel workers this quickly becomes the majority of traces (33 of the most recent 50 on the instance where this was diagnosed).Root cause: the plugin ends the root span in
_finish_trace(), which only runs when a turn finalizes (final assistant message with no tool calls). A process that exits while the last LLM response still has tool calls queued never reaches it. The SDK's own atexit flush then exports the ENDED child observations, but the root span never lands — so the backend renders a nameless trace shell around orphaned children.Fix:
_finalize_all_traces()drains_TRACE_STATEand ends every open root span (children first), registered withatexitimmediately after the Langfuse client is constructed. atexit is LIFO, so it runs BEFORE the SDK's own shutdown hook and the spans ended here are still picked up by the exporter. Idempotent; fail-open per span; no-op when nothing is pending.Why
How to test
29 tests (2 new):
_finalize_all_tracesends every open root and drains state (idempotent on re-entry), and the atexit hook is registered on client init. Both new tests fail without the fix (verified by stashing it).Live verification on the diagnosed instance: after the fix, a
hermes chat -qone-shot exports a complete "Hermes turn" trace with session id; before, the same flow produced a nameless trace with 17+ orphaned observations.Tested on Linux (aarch64).