feat(observability): add trusted trace correlation

[zepdevelopers]

Summary

• add disabled-by-default trusted-edge observability correlation config
• derive a strict environment-only K1 HMAC identifier on Langfuse root traces
• cache configuration at plugin registration and omit correlation metadata on invalid or disabled configuration
• document privacy and restart requirements

Verification

• ................................................................ [100%]
  64 passed in 0.49s
• 64 passed on current upstream main

The HMAC key remains environment-only and is never logged or serialized.

[teknium1]

Thanks for the careful key parsing, disabled default, and documentation. Current main does not already provide this metadata (plugins/observability/langfuse/__init__.py:717-726), so the feature is not superseded.

Problems

• plugins/observability/langfuse/__init__.py:727-730 sends a new third-party correlation identifier, but the change adds only a config gate. Current policy requires a config gate, setup prompt, and hermes tools toggle (AGENTS.md:118-121); current Langfuse setup only prompts for its existing credentials (hermes_cli/tools_config.py:555-579).
• history_execution_writer_enabled is added at hermes_cli/config.py:3015 but is never read by _load_correlation_config() (plugins/observability/langfuse/__init__.py:162-173).
• The registration test mocks the config loader (tests/plugins/test_langfuse_plugin.py:329-376), so it does not cover persisted-config/plugin-discovery propagation.

Suggested changes

• Wire this through the required explicit consent surfaces before emitting metadata.
• Remove the unused writer flag, or ship its concrete consumer and coverage together.
• Add an isolated-HERMES_HOME integration test for enabled and disabled persisted configuration.

This is an automated hermes-sweeper review.

[teknium1]

This flag has no consumer: _load_correlation_config() reads only scheme and emitter_metadata_enabled, and the new test asserts this field is absent from metadata. Please remove it until a concrete writer uses it, or implement that writer and its behavior in this change.

[teknium1]

This is where the new identifier becomes outbound Langfuse metadata. Current policy requires a generic consent flow with config gate, setup prompt, and hermes tools toggle (AGENTS.md:118-121); this PR adds the gate but does not update the Langfuse setup surface. Please add the required explicit opt-in flow before emitting it.

[teknium1]

Please add an isolated-HERMES_HOME integration test using persisted config and normal plugin discovery. Mocking load_config_readonly verifies the helper, but not the configuration propagation and consent boundary.

[GottZ]

This was generated by AI during triage.

Summary

Fourteen PRs address or reference this Langfuse observability complex, spanning the merged trace-I/O and wrong-prefix credential baseline, residual right-prefix template validation, credential diagnostics, system-prompt capture, privacy and endpoint handling, profile metadata, and trusted correlation. For target #64779, the diff adds disabled-by-default HMAC-derived trace correlation, but the contributor review identifies missing setup consent surfaces, an unused writer flag, and missing persisted-config propagation coverage.

Related pull requests

• fix(langfuse): fixes langfuse observability - llm input/output and tool output missing #22345 [closed] fixes — (+295/-21) — n/a: This closed trace-I/O precursor forwards real request/response payloads and deterministically matches tool outputs; it remains relevant because fix(langfuse): complete observability fix — trace I/O, tool outputs, placeholder credentials (closes #22342, #22763) #26320 superseded it by preserving its commits and authorship.
• fix(langfuse): reject placeholder credentials #23188 [closed] partial — (+41/-0) — n/a: This closed placeholder-credential fix uses a finite blacklist and remains relevant as the narrower duplicate of Fix/langfuse placeholder key warning #23831, superseded by the prefix-based implementation in fix(langfuse): complete observability fix — trace I/O, tool outputs, placeholder credentials (closes #22342, #22763) #26320.
• Fix/langfuse placeholder key warning #23831 [closed] fixes — (+349/-0) — n/a: This closed precursor adds prefix validation, a one-shot warning, and the _INIT_FAILED path for wrong-prefix placeholders; it remains relevant because fix(langfuse): complete observability fix — trace I/O, tool outputs, placeholder credentials (closes #22342, #22763) #26320 incorporated its implementation and tests.
• fix(langfuse): complete observability fix — trace I/O, tool outputs, placeholder credentials (closes #22342, #22763) #26320 [merged] best fix — (+705/-22) — n/a: This merged reference implementation combines request/response tracing, tool-output matching, and wrong-prefix credential rejection, while its prefix-only check leaves right-prefix templates such as pk-lf-... to fix(langfuse): reject right-prefix template keys, not just wrong prefixes #64309/fix(langfuse): reject right-prefix template keys, not just wrong prefixes #77373.
• feat(langfuse): tag traces with active profile #26462 [closed] best fix — (+182/-39) — n/a: This closed adjacent feature threads the active profile through hooks and sets Langfuse user_id plus profile metadata, remaining relevant as the recorded best existing fix for feat(langfuse): expose profile_name as trace userId for per-profile observability #26455 rather than for correlation.
• fix(langfuse): probe server credentials so misconfigured keys fail loudly (#29332) #29340 partial — (+492/-0) — n/a: This adds a live auth_check() for correctly shaped but rejected credentials; keep it on its reviewed redesign path because production probing conflicts with SDK guidance, the behavioral opt-out belongs in config, and remote I/O lacks real-path validation.
• fix(langfuse): include Anthropic system prompt in generation input #32175 partial — (+115/-10) — n/a: This restores Anthropic's detached system prompt to generation input and should remain open on the visible keep_open salvage path requiring a regression across the conversation-loop hook boundary.
• fix(langfuse): default base URL to localhost; reap stale per-turn trace state #45326 related — (+28/-3) — n/a: This changes the default endpoint and adds stale-state reaping; despite the APPROVED review, the later keep_open verdict identifies Cloud-setup breakage, a refresh race, overlap with current state bounds, missing tests, and incomplete documentation.
• fix(observability/langfuse): surface placeholder & runtime ingestion failures (#60961) #60974 [closed] best fix — (+635/-156) — n/a: This closed broader predecessor rejects several right-prefix templates and attempts first-failure ingestion logging, remaining relevant as recorded best-fix prior art while its focused validator portion is superseded by fix(langfuse): reject right-prefix template keys, not just wrong prefixes #64309/fix(langfuse): reject right-prefix template keys, not just wrong prefixes #77373.
• fix(langfuse): require explicit export consent #64236 related — (+200/-17) — n/a: This adds explicit metadata/content export consent and should remain open on the contributor-reviewed salvage path to integrate consent into hermes tools, update shipped guides, and test persisted configuration and payload boundaries.
• fix(langfuse): include system prompt in generation input (Anthropic system + Codex instructions) #64292 partial — (+384/-7) — n/a: This broadens detached system-prompt tracing across Anthropic, Codex, and Bedrock, and its shown diff addresses the keep_open review by accepting Bedrock {"text": ...} blocks and adding hook-boundary coverage.
• fix(langfuse): reject right-prefix template keys, not just wrong prefixes #64309 best fix — (+116/-6) — n/a: This focused validator rejects right-prefix template remainders while preserving UUID-shaped keys; despite its keep_open review, fix(langfuse): reject right-prefix template keys, not just wrong prefixes #77373 carries the same production change and focused tests as an authorship-preserving current-base salvage, making fix(langfuse): reject right-prefix template keys, not just wrong prefixes #64309 duplicative.
• feat(observability): add trusted trace correlation #64779 related — (+419/-2) — n/a: This adds strict, disabled-by-default HMAC trace correlation, but its contributor keep_open review requires a setup prompt and hermes tools toggle, removal or implementation of history_execution_writer_enabled, and persisted-config/plugin-discovery coverage.
• fix(langfuse): reject right-prefix template keys, not just wrong prefixes #77373 best fix — (+116/-6) — n/a: This current-base salvage of fix(langfuse): reject right-prefix template keys, not just wrong prefixes #64309 rejects docs-literal, bare-prefix, and fragment-bearing template keys while retaining real-shaped-key acceptance and warn-and-skip tests, directly covering the residual credential cause left by fix(langfuse): complete observability fix — trace I/O, tool outputs, placeholder credentials (closes #22342, #22763) #26320.

Duplicates

#22345 and #23831 supplied components consolidated into #26320, while #23188 is the narrower competing placeholder guard; #64309 and #77373 contain the same focused right-prefix template-key change, #60974 overlaps that validator work while adding runtime diagnostics, and #32175 is the Anthropic-only subset of #64292.

Suggested consolidation

Keep #64779 open with a salvage path that adds the required setup prompt and hermes tools consent toggle, resolves the unused writer flag, and verifies persisted-config propagation before emitting correlation metadata. Separately, close #64309 as a duplicate of #77373 despite #64309's keep_open review because their supplied production diffs and focused tests are the same; retain merged #26320 and closed #22345/#23188/#23831/#60974 as reference or superseded implementations, and keep the reviewed diagnostic, prompt-tracing, endpoint/state, privacy, profile, and credential-salvage paths independent.

Cross-PR triage: Reviewed 14 pull requests and 18 issues in this complex. Each diff was read against this issue; Assessment working set: 274 kB of PR diffs, 86 kB of issue/PR text, 37 kB of discussion (57 comments), 82 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.