feat(langfuse): widen tracing to errors, sessions, subagents, and MoA fan-out by erosika · Pull Request #83437 · NousResearch/hermes-agent

erosika · 2026-08-10T20:39:32Z

Widens the bundled Langfuse plugin from 6 hooks to 11 and fixes two attribution bugs. Also adopts the shutdown/atexit lifecycle fixes from #81054 and #82332 (original authorship preserved) with two interaction-fix follow-up commits.

Model attribution

on_pre_llm_request and on_post_llm_call read model, the agent's attribute at hook time. It goes stale after a mid-session /model switch or a provider fallback, so generations file under the wrong model and cost estimation keys off it. Both now prefer the wire value — request["body"]["model"] and response_model. agent/conversation_loop.py already passed both and the plugin ignored them.

Cost total

Langfuse does not derive calculatedTotalCost from the per-type cost_details keys. _usage_and_cost wrote the breakdown and no total, so a priced generation read 0 in the dashboard while its components were correct. Both cost paths now send a summed total. A priced model that billed no tokens writes no breakdown and still sends no total.

New coverage

Errors and session lifecycle. api_request_error closes failed generations with level=ERROR and retry metadata. on_session_finalize and on_session_end close still-open traces so tool-only and interrupted turns stop dangling. Adds HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full, default sanitized.

Subagents. tools/delegate_tool.py emits subagent_start/subagent_stop and the sibling nemo_relay plugin already consumes them, so every delegated child was missing here. The payloads carry parent_turn_id but no task_id, and _scope_prefix prefers task_id when the LLM hooks minted the key — so _state_for_turn matches on the turn-id suffix instead of rebuilding a key that would miss.

MoA advisors. A MoA turn runs N advisor models before its aggregator and returns only the aggregator's response, so the fan-out showed up as one generation priced at the aggregator's model. _RefAccounting already computes each advisor's usage and dollars, precisely because advisors run on a different provider and cannot be priced at the aggregator's rate. moa_trace.slot_metrics renders that and post_api_request carries it as moa_references.

MoA plumbing

The read is deliberately non-consuming. post_api_request fires on a different branch of run_conversation than consume_reference_usage and consume_and_save_trace, so a consuming read would race them. The client holds its last fan-out until the next one, so the plugin fingerprints each fan-out and does not re-emit the same advisors on every API call of a tool-loop turn.

The payload derives from the already privacy-redacted _trace_refs, so an active privacy mode redacts it too. slot_metrics drops input_messages rather than crossing the hook boundary with a full transcript per advisor per turn.

Verification

184 tests pass across the langfuse, MoA, subagent-hook, and relay-metrics suites. moa_references cannot reach the nemo relay's metrics payload: model_call_fields allowlists model and provider. The 11 whole-tree pytest collection errors are identical on main.

Adopted lifecycle fixes

#81054 (bgodlin, 2 commits). Quitting Hermes with the plugin enabled printed an "Exception ignored in: " TypeError traceback. The SDK's own atexit shutdown runs during interpreter finalization, after opentelemetry.trace.Span is torn down to None, so its span-cleanup isinstance(span, Span) check blows up. The fix shuts the client down at session finalize while the interpreter is alive, and exits the root observation's context manager instead of leaving its generator suspended for GC to unwind at teardown.

#82332 (aldoeliacim, 1 commit). Short-lived processes (kanban workers, hermes chat -q, cron jobs) could exit with tool calls still queued, so the root span never ended and the backend showed an anonymous trace with no name, session, or metadata. An atexit finalizer — registered after the SDK client so LIFO ordering runs it before the SDK's own flush — ends every open root span at exit.

Follow-ups (interaction fixes found while composing). The adopted shutdown fired on every on_session_finalize, but that hook also fires on /new, /reset, and gateway session expiry, where the process lives on — the first rotation would have killed the cached client and silently stopped exports for every later session. The shutdown is now gated on reason == "shutdown". The adopted atexit finalizer also skipped subagent observations (added in this PR) and never exited the root context manager, reintroducing the same teardown TypeError it sat next to; both closed.

Verification

74 langfuse + MoA-bridge tests pass; full tests/agent/ + tests/hermes_cli/ sweep is 8018 passed with 8 failures identical on the unmodified base (provider-routing/env, not langfuse). Live smoke against Langfuse Cloud: three sessions (clean turn, rotation-closed dangling turn, shutdown-closed dangling turn) all exported with correct names and session ids; the client survived rotation and shut down cleanly at exit.

Adopted fixes (tier 2)

Adopted with follow-up integration on top; each commit thanks and co-author-credits the original contributor.

#42326 (@nftpoetrist). _get_langfuse() double-checked a global with no lock, so two concurrent first callers could both construct a client and leak the loser's HTTP connection and flush thread. First build is now serialized; the settled fast path stays lock-free.

#39653 (@rodboev). Reasoning models that expose their scratchpad as reasoning_content or structured reasoning_details traced as reasoning: None. An accessor now checks the three fields in precedence order. Closes #29482.

#64292 (@FnExpress). Providers that move the system prompt out of messages (Anthropic system, Codex instructions, Bedrock Converse blocks) produced generation inputs with no system prompt at all — no skills, memory, or instructions visible in traces. The loop now forwards the prompt as actually sent and the plugin prepends a role: system entry. Extends and supersedes #32175 (@db-aeon), whose Anthropic-only fix is credited in the commit.

#61166 (@Per0-1). The plugin called set_trace_io(), which does not exist in SDK v3 — the AttributeError skipped root_span.end(), so traces listed with blank Input/Output columns and no CHAIN root. Trace I/O now uses v3 update_trace(), individually fail-open so no export step can block the root end.

#64797 (@NaMinhyeok). Replaces this PR's earlier sum-of-components total with the canonical Hermes estimate: summing the breakdown undercounts when a component can't be priced or request-level pricing applies. Both cost paths now share one helper; a partial breakdown with no valid estimate exports no total. Together with the original cost-total commit this closes #72661.

#43130 (@liuhao1024). Subscription-included routes (openai-codex) sent explicit $0 cost_details, which Langfuse treats as authoritative — blocking its own model-based estimation, so every generation showed $0 forever. Included routes now send no cost keys at all. Closes #43129.

Integration fixes made while composing: system-prompt and reasoning serialization route through _capture_content so capture modes and secret redaction govern them; the canonical total adds a zero-guard so a priced model that billed nothing doesn't export an authoritative 0.0.

Closes / supersedes

Closes #29482, closes #43129, closes #72661.

Supersedes (can be closed in favor of this PR): #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130.

Partially addresses #67544: capture modes (metadata/sanitized/full) plus secret redaction at the export boundary cover the masking half; user_id attribution remains open.

Verification (tier 2)

88 plugin tests pass; full tests/plugins/ + tests/agent/ + tests/hermes_cli/ sweep is 9380 passed with every failure reproduced on the unmodified base or current main (env-dependent provider tests, hindsight extra not installed locally, and a _preset_cache conftest collection error that exists on main). Live smoke against Langfuse Cloud confirmed server-side: system prompt visible in generation input, reasoning_content in output, costDetails.total matching calculatedTotalCost, trace-level Input/Output columns filled.

… attribute pre_api_request now prefers request.body.model and post_api_request prefers response_model over the agent's model attribute, which goes stale across mid-session /model switches and provider fallbacks. cost estimation keys off the served model.

langfuse ingests per-type cost_details keys but does not derive calculatedTotalCost from them — every generation showed cost 0 in the dashboard despite correct input/output/cache components. add a summed 'total' key on both the response-object and usage-summary paths.

The cost fix shipped without tests. Cover the response-object path and the usage-summary path, and pin the empty-breakdown case: a priced model that billed no tokens must not report a 0.0 total.

…lush - HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized): metadata replaces content with shape/size stubs; sanitized redacts secret-shaped substrings before truncation; full is explicit opt-in. active mode recorded as trace metadata.capture_mode - subscribe api_request_error: failed generations close with level=ERROR, status/retry metadata; non-retryable failures finish the turn trace - on_session_finalize/on_session_end: close still-open traces for the session and flush, so tool-only/interrupted turns no longer dangle

delegate_tool emits subagent_start/subagent_stop and the sibling nemo_relay plugin already consumes them; langfuse ignored both, so every delegated child was missing from the trace. The payloads carry parent_turn_id but no task_id, and _scope_prefix prefers task_id when the LLM hooks minted the key, so rebuilding the key from session_id would miss. _state_for_turn matches on the turn-id suffix instead. Spans key on child_session_id because subagent_stop carries no child_subagent_id.

A MoA turn runs N advisor models before its aggregator and returns only the aggregator's response, so the whole fan-out showed up as one generation priced at the aggregator's model. moa_loop emits no plugin hooks, so nothing observable ever saw the advisors. The per-advisor numbers were already computed: _RefAccounting carries each advisor's usage and dollars precisely because advisors run on a different provider than the aggregator and cannot be priced at its rate. moa_trace.slot_metrics renders that for the hook boundary, dropping input_messages so a full transcript per advisor per turn does not cross it. The payload derives from the privacy-redacted _trace_refs, so an active privacy mode redacts it too. post_api_request carries it as moa_references. The read is deliberately non-consuming: the hook fires on a different branch than consume_reference_usage and consume_and_save_trace, so a consuming read would race them. The client holds its last fan-out until the next one, so the plugin fingerprints each fan-out to avoid re-emitting the same advisors on every API call of a tool-loop turn.

…er-teardown TypeError The langfuse plugin never called client.shutdown(), relying on the SDK's atexit handler. That fires during interpreter finalization, after opentelemetry.trace.Span is torn down to None — use_span's isinstance(span, Span) raises TypeError, surfaced as 'Exception ignored in: <generator>' on quit. Register on_session_finalize to call client.shutdown() while the interpreter is alive.

…down TypeError The on_session_finalize hook (added in the previous commit) calls client.shutdown() but that only flushes the SDK's internal queues. It does not unwind the root observation context managers the plugin itself created: _start_root_trace enters start_as_current_observation(...).__enter__() but _finish_trace only called root_span.end(), never root_ctx.__exit__(). The generator stays suspended inside 'with otel_trace_api.use_span(parent_span):' until the GC collects it during interpreter teardown. By then opentelemetry.trace.Span has been torn down to None, and use_span's isinstance(span, Span) raises: TypeError: isinstance() arg 2 must be a type surfaced as 'Exception ignored in: <generator>' on every CLI exit. Fix: call root_ctx.__exit__(None, None, None) right after root_span.end() in both _finish_trace and _evict_stale_locked. This unwinds the generator while all modules are intact. Regression test: test_finish_trace_exits_root_context_manager verifies __exit__ is called and fails on the pre-fix code.

…ses export complete traces Langfuse-driven correctness review, bug 3: kanban workers / hermes chat -q / cron one-shots exit while the final LLM response still has tool calls queued, so _finish_trace never runs. The SDK's own atexit flush exports the ENDED children but the root span never lands: the backend shows an anonymous trace — name '', no sessionId, empty metadata — with 17-37 orphaned observations (4 seen live in one kanban tick, 33/50 of recent traces anonymous). _finalize_all_traces drains _TRACE_STATE and ends every open root; registered with atexit AFTER Langfuse client construction (atexit is LIFO → runs before the SDK's shutdown flush, so the ended spans still export). Idempotent; fail-open per span. RED-verified: both new tests fail with the fix stashed; 29/29 green with it.

…ation The cherry-picked shutdown fix called client.shutdown() on every on_session_finalize — but that hook also fires on /new, /reset, and gateway session expiry, where the process lives on and the cached client must keep exporting for later sessions. Gate the shutdown on reason == "shutdown" (CLI exit / gateway shutdown), keeping the interpreter-teardown TypeError fix for the case it targets while preserving multi-session export in long-lived processes. Also merges the shutdown into the existing session-scoped finalize handler rather than keeping a second duplicate hook function. Co-authored-by: bgodlin <37313677+bgodlin@users.noreply.github.com>

…lizer The adopted atexit finalizer ended generations, tools, and the root span but skipped subagent observations and never exited the root observation's context manager — the same suspended-generator teardown path the CM-exit fix closes for _finish_trace/_evict_stale_locked. Close both in _finalize_all_traces so short-lived-process exits don't reintroduce the interpreter-teardown TypeError or drop subagent spans. Co-authored-by: Aldo <github@aldo.pw>

@nftpoetrist

_get_langfuse() double-checked a global with no lock, so two concurrent first callers (e.g. two gateway sessions firing hooks at once) could both pass the None guard, both construct a client, and leak the loser's HTTP connection and background flush thread. First build is now serialized by a module lock with a re-check inside; the settled fast path stays lock-free. The atexit finalizer registration moves inside the locked section so it registers exactly once, after the winning client. Adopted from NousResearch#42326 — thanks @nftpoetrist for the report and fix. Co-authored-by: nftpoetrist <264138787+nftpoetrist@users.noreply.github.com>

@rodboev

Assistant-message serialization only read message.reasoning, but reasoning models and adapters can expose their scratchpad under reasoning_content or structured reasoning_details — those traces showed reasoning: None despite the data being right there. A small accessor now checks the three fields in precedence order. Routed through _capture_content so capture modes and secret redaction apply to reasoning text like everything else. Adopted from NousResearch#39653 (closes NousResearch#29482) — thanks @rodboev. Co-authored-by: Rod Boev <rod.boev@gmail.com>

@FnExpress

Providers that move the system prompt out of messages made it vanish from traces: Anthropic Messages carries it as a separate system kwarg (str or content-block list) and the Responses/Codex API as top-level instructions, so generation inputs showed conversations without the agent's instructions, skills, or memory. conversation_loop now derives the system prompt as actually sent to the provider and forwards it to hooks; the plugin prepends a role: system entry when messages don't already carry one. Serialization routes through _capture_content so capture modes apply to system prompts too. Adopted from NousResearch#64292, which extends NousResearch#32175's Anthropic fix to the Codex/Responses path — thanks @FnExpress and @db-aeon. Co-authored-by: FnExpress <37214785+FnExpress@users.noreply.github.com> Co-authored-by: Dan Benyamin <db@project-aeon.com>

@Per0-1

The plugin called root_span.set_trace_io(), which does not exist in Langfuse SDK v3 — the AttributeError inside _finish_trace's try block skipped root_span.end(), so generations and tools exported without a CHAIN root and the trace list showed blank Input/Output columns with levels and latency populated. Trace-level I/O now goes through the v3 update_trace() API, with each call individually fail-open so no export step can block the root end(), plus a last-chance end() on unexpected errors. Adopted from NousResearch#61166 — thanks @Per0-1. Co-authored-by: Bizon Tech <bizon@example.com>

@NaMinhyeok

Supersedes this branch's earlier sum-of-components total with the canonical Hermes estimate. Summing the per-type breakdown undercounts whenever a component can't be priced (cache rates missing) or when request-level pricing applies; the estimate_usage_cost amount is the number Hermes itself reports, so both response-object and summary-dict paths now share one _canonical_usage_and_cost helper that exports it as the explicit total alongside the per-type breakdown. A partial breakdown with no valid estimate exports no total at all, so Langfuse can't mistake a subtotal for the full cost. Subscription-included routes keep their component-only payload, and a zero estimate is not exported as an authoritative 0.0. Adopted from NousResearch#64797 — thanks @NaMinhyeok for the thorough repro and test matrix. Co-authored-by: NaMinhyeok <nmh9097@gmail.com>

@liuhao1024

Subscription-included routes (e.g. openai-codex) priced every request at explicit $0: get_pricing_entry returns zero rates for these routes, and Langfuse treats provided cost_details as authoritative — the zeros blocked its own model-based cost estimation, so every generation showed $0 forever (NousResearch#43129). _canonical_usage_and_cost now resolves the billing route first and sends no cost keys at all for included routes, letting Langfuse fall back to its own pricing. Usage details still export. Adopted from NousResearch#43130 — thanks @liuhao1024. Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>

@erosika

… fan-out Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054), @aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress (#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797), and @liuhao1024 (#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes #29482, #43129, #72661. Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130. Partially addresses #67544 (capture modes + secret redaction; user_id remains open).

Follow-up fixes from /hermes-pr-review + /simplify-code on PR #83437: 1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True) — the plugin's 11-pattern list was a strict subset of the 50+ patterns in agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens, HuggingFace tokens, DB connection strings, and Telegram bot tokens would all leak through the plugin's list but are caught by the existing redactor. Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py. 2. Remove dead 'not isinstance(client, object)' check in on_session_finalize — always False for any Python value. 3. Fix MoAClient.last_reference_metrics() to call the public self.chat.completions.last_reference_metrics() instead of reaching into the private _last_reference_metrics attribute via getattr. 4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass pre_coerced=input_messages to _messages_for_langfuse_input to avoid double-coercion + double _capture_content serialization per API request. 5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py for consistency with the other HERMES_LANGFUSE_* env vars. 6. Fix test_sanitized_mode_redacts_secrets test data — the old samples ('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too short to match the regex thresholds and never actually tested redaction. Updated to realistic-length secrets and changed assertions to check that the output differs from input (redact_sensitive_text masks rather than inserting the literal string 'REDACTED').

kshitijk4poor · 2026-08-13T17:40:26Z

Merged via #85439 — your commits were applied via diff-apply salvage (the PR was 1246 commits behind main), with all 3 contributors credited in the commit body. Follow-up fixes from /hermes-pr-review + /simplify-code were applied on top: reused redact_sensitive_text instead of a thinner 11-pattern subset, fixed a leaky getattr on a private attribute, deduplicated _coerce_request_messages, added HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS, and fixed weak test data that wasn't actually exercising the redaction regexes. Stale-base revert hunks in conversation_loop.py and moa_loop.py were detected and restored.

Thank you for the excellent work — this is a substantial, well-tested contribution.

Follow-up fixes from /hermes-pr-review + /simplify-code on PR NousResearch#83437: 1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True) — the plugin's 11-pattern list was a strict subset of the 50+ patterns in agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens, HuggingFace tokens, DB connection strings, and Telegram bot tokens would all leak through the plugin's list but are caught by the existing redactor. Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py. 2. Remove dead 'not isinstance(client, object)' check in on_session_finalize — always False for any Python value. 3. Fix MoAClient.last_reference_metrics() to call the public self.chat.completions.last_reference_metrics() instead of reaching into the private _last_reference_metrics attribute via getattr. 4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass pre_coerced=input_messages to _messages_for_langfuse_input to avoid double-coercion + double _capture_content serialization per API request. 5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py for consistency with the other HERMES_LANGFUSE_* env vars. 6. Fix test_sanitized_mode_redacts_secrets test data — the old samples ('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too short to match the regex thresholds and never actually tested redaction. Updated to realistic-length secrets and changed assertions to check that the output differs from input (redact_sensitive_text masks rather than inserting the literal string 'REDACTED'). (cherry picked from commit ace8301)

@erosika

… fan-out Salvaged from PR NousResearch#83437 by @erosika, with adopted fixes from @bgodlin (NousResearch#81054), @aldoeliacim (NousResearch#82332), @nftpoetrist (NousResearch#42326), @rodboev (NousResearch#39653), @FnExpress (NousResearch#64292, supersedes NousResearch#32175 by @db-aeon), @Per0-1 (NousResearch#61166), @NaMinhyeok (NousResearch#64797), and @liuhao1024 (NousResearch#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes NousResearch#29482, NousResearch#43129, NousResearch#72661. Supersedes NousResearch#81054, NousResearch#82332, NousResearch#42326, NousResearch#39653, NousResearch#64292, NousResearch#32175, NousResearch#61166, NousResearch#64797, NousResearch#43130. Partially addresses NousResearch#67544 (capture modes + secret redaction; user_id remains open).

Follow-up fixes from /hermes-pr-review + /simplify-code on PR NousResearch#83437: 1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True) — the plugin's 11-pattern list was a strict subset of the 50+ patterns in agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens, HuggingFace tokens, DB connection strings, and Telegram bot tokens would all leak through the plugin's list but are caught by the existing redactor. Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py. 2. Remove dead 'not isinstance(client, object)' check in on_session_finalize — always False for any Python value. 3. Fix MoAClient.last_reference_metrics() to call the public self.chat.completions.last_reference_metrics() instead of reaching into the private _last_reference_metrics attribute via getattr. 4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass pre_coerced=input_messages to _messages_for_langfuse_input to avoid double-coercion + double _capture_content serialization per API request. 5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py for consistency with the other HERMES_LANGFUSE_* env vars. 6. Fix test_sanitized_mode_redacts_secrets test data — the old samples ('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too short to match the regex thresholds and never actually tested redaction. Updated to realistic-length secrets and changed assertions to check that the output differs from input (redact_sensitive_text masks rather than inserting the literal string 'REDACTED').

@erosika

… fan-out Salvaged from PR NousResearch#83437 by @erosika, with adopted fixes from @bgodlin (NousResearch#81054), @aldoeliacim (NousResearch#82332), @nftpoetrist (NousResearch#42326), @rodboev (NousResearch#39653), @FnExpress (NousResearch#64292, supersedes NousResearch#32175 by @db-aeon), @Per0-1 (NousResearch#61166), @NaMinhyeok (NousResearch#64797), and @liuhao1024 (NousResearch#43130). Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes 8 prior community PRs with interaction-fix follow-ups. Model attribution: on_pre_llm_request and on_post_llm_call now prefer the wire value (request body model, response model) over the agent attribute, which goes stale after /model switch or provider fallback. Cost total: both cost paths now send a summed total alongside the per-type breakdown, since Langfuse does not derive calculatedTotalCost from cost_details keys. Subscription-included routes send no cost keys at all. New coverage: api_request_error closes failed generations with ERROR level; on_session_finalize/on_session_end close dangling traces for tool-only and interrupted turns; subagent_start/subagent_stop trace delegated children as spans; MoA advisor fan-out emits one generation per advisor priced at the advisor's own model. Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default sanitized). Sanitized mode redacts secret patterns before truncation. Adopted lifecycle fixes: shutdown client at session finalize when reason=shutdown (not on session rotation); atexit finalizer ends open root spans for short-lived processes; root context manager exited to prevent interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock; reasoning_content surfaced in traces; system prompt included in generation input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io. Closes NousResearch#29482, NousResearch#43129, NousResearch#72661. Supersedes NousResearch#81054, NousResearch#82332, NousResearch#42326, NousResearch#39653, NousResearch#64292, NousResearch#32175, NousResearch#61166, NousResearch#64797, NousResearch#43130. Partially addresses NousResearch#67544 (capture modes + secret redaction; user_id remains open).

Follow-up fixes from /hermes-pr-review + /simplify-code on PR NousResearch#83437: 1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True) — the plugin's 11-pattern list was a strict subset of the 50+ patterns in agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens, HuggingFace tokens, DB connection strings, and Telegram bot tokens would all leak through the plugin's list but are caught by the existing redactor. Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py. 2. Remove dead 'not isinstance(client, object)' check in on_session_finalize — always False for any Python value. 3. Fix MoAClient.last_reference_metrics() to call the public self.chat.completions.last_reference_metrics() instead of reaching into the private _last_reference_metrics attribute via getattr. 4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass pre_coerced=input_messages to _messages_for_langfuse_input to avoid double-coercion + double _capture_content serialization per API request. 5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py for consistency with the other HERMES_LANGFUSE_* env vars. 6. Fix test_sanitized_mode_redacts_secrets test data — the old samples ('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too short to match the regex thresholds and never actually tested redaction. Updated to realistic-length secrets and changed assertions to check that the output differs from input (redact_sensitive_text masks rather than inserting the literal string 'REDACTED').

erosika added 6 commits August 10, 2026 14:30

test(langfuse): cover the explicit cost total on both cost paths

be5a0ea

The cost fix shipped without tests. Cover the response-object path and the usage-summary path, and pin the empty-breakdown case: a priced model that billed no tokens must not report a 0.0 total.

bgodlin and others added 5 commits August 10, 2026 19:10

This was referenced Aug 10, 2026

fix(langfuse): shutdown client on session finalize to avoid interpreter-teardown TypeError #81054

Closed

fix(langfuse): finalize open root spans at exit so short-lived processes export complete traces #82332

Closed

erosika and others added 6 commits August 10, 2026 19:37

erosika mentioned this pull request Aug 10, 2026

fix(langfuse): omit explicit zero cost_details for subscription-included providers (#43129) #43130

Closed

13 tasks

kshitijk4poor mentioned this pull request Aug 13, 2026

feat(langfuse): widen tracing to errors, sessions, subagents, and MoA fan-out #85439

Merged

kshitijk4poor closed this Aug 13, 2026

Navigation Menu

Uh oh!

feat(langfuse): widen tracing to errors, sessions, subagents, and MoA fan-out - #83437

feat(langfuse): widen tracing to errors, sessions, subagents, and MoA fan-out#83437
erosika wants to merge 17 commits into
NousResearch:mainfrom
erosika:eri/langfuse-model-attribution-and-cost

erosika commented Aug 10, 2026 •

edited

Loading

Uh oh!

kshitijk4poor commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

erosika commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Model attribution

Cost total

New coverage

MoA plumbing

Verification

Adopted lifecycle fixes

Verification

Adopted fixes (tier 2)

Closes / supersedes

Verification (tier 2)

Uh oh!

kshitijk4poor commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

erosika commented Aug 10, 2026 •

edited

Loading