fix(langfuse): include system prompt in generation input (Anthropic system + Codex instructions)

[FnExpress]

Summary

Langfuse LLM call generation observations show conversations without the agent's system prompt (skills, memory, SOUL, tool guidance) for every provider that moves the system prompt out of messages:

• Anthropic Messages API → separate system kwarg (agent/anthropic_adapter.py), reported in fix(langfuse): include Anthropic system prompt in generation input #32175
• Responses/Codex API → top-level instructions (agent/transports/codex.py:132-139 extracts messages[0] into instructions and drops it from the payload) — the sibling call path, previously uncovered

The pre_api_request hook only forwarded request_messages, so traces for both paths looked like conversations with no instructions at all — e.g. a Codex-backed Hermes turn whose LLM call 1 shows 18k prompt tokens but only a one-line user message as input.

This PR salvages #32175 by @db-aeon onto current main (cherry-picked so authorship survives, per the AGENTS.md salvage etiquette), then fixes the whole bug class:

• agent/conversation_loop.py derives the system prompt the provider actually receives — api_kwargs["system"] → api_kwargs["instructions"] → messages[0] fallback, extracted into a _system_prompt_for_hooks() helper — and passes it to pre_api_request as system_prompt
• plugins/observability/langfuse prepends a serialized role: system entry to the generation input (after the last-12 window, so long conversations never drop it), flattens Anthropic content-block lists, skips the prepend when messages[0] already carries the system role (chat_completions), and records system_prompt_chars metadata

Commits

• test(langfuse): system prompt must reach generation input across providers — RED-first coverage: plugin contract (prepend, block-list flatten, no duplication, window-safe, metadata) plus the cross-hook-boundary regression the fix(langfuse): include Anthropic system prompt in generation input #32175 review asked for — real ResponsesApiTransport().build_kwargs() / AnthropicTransport().build_kwargs() output flows through the loop's derivation into the plugin hook with no Hermes internals mocked (only the Langfuse client is faked)
• fix(langfuse): include Anthropic system prompt in generation input (@db-aeon, cherry-picked from fix(langfuse): include Anthropic system prompt in generation input #32175; conflicts with the current middleware/hook block resolved)
• fix(langfuse): carry Codex/Responses instructions into hook system_prompt — the Codex sibling path + _system_prompt_for_hooks() extraction

Test plan

• tests/plugins/test_langfuse_plugin.py — 59 passed (48 pre-existing + 9 new + 2 from fix(langfuse): include Anthropic system prompt in generation input #32175)
• Affected surface: tests/plugins/, tests/hermes_cli/test_plugins.py, tests/run_agent/test_run_agent_codex_responses.py — no regressions vs clean main
• scripts/run_tests.sh full suite — remaining failures reproduce identically on clean main (environment-specific, macOS), zero delta from this change
• E2E against a self-hosted Langfuse: enabled observability/langfuse, ran hermes chat on an openai-codex model — generation input now starts with role: system (102,659 chars, system_prompt_chars metadata matches); chat_completions sessions unchanged (single system entry, no duplication)

Closes #32175 (supersedes it while preserving its commit; happy to rebase if @db-aeon prefers to land his PR first — this one then shrinks to the Codex path + tests).

[tonydwb]

Code Review Summary

Verdict: Approved

This PR includes the system prompt in generation input for Langfuse.

Looks Good

• Targeted fix for Langfuse integration.
• Well-scoped: 4 files.

No Issues Found

---

Reviewed by Hermes Agent

[teknium1]

Thanks for extending the Langfuse fix to the Codex/Responses request shape; the current Anthropic and Codex premise is verified.

Problems

• agent/bedrock_adapter.py:580-583 also removes the system prompt from messages, encoding it as Bedrock blocks shaped {"text": content} and sending it through a separate system kwarg at agent/bedrock_adapter.py:951-962. The new check at plugins/observability/langfuse/__init__.py:497 only accepts dict blocks where type == "text", so the proposed helper returns no system entry for Bedrock and leaves the same trace gap on that provider.

Suggested changes

• Accept Bedrock text blocks in _serialize_system_prompt() and add a BedrockTransport().build_kwargs() regression through the hook-boundary helper, alongside the Anthropic and Codex cases.

Automated hermes-sweeper review.

[teknium1]

Bedrock Converse sends separate system blocks as {"text": ...} (agent/bedrock_adapter.py:580-583,951-962), not {"type": "text", ...}. This condition drops every Bedrock system prompt, leaving the same Langfuse gap there; please accept that block shape and cover it with a transport-boundary regression.

[erosika]

adopted into #83437 with co-author credit — thanks for extending the fix across Anthropic, Codex, and Bedrock shapes. integration changes: system-prompt serialization routes through the branch's capture modes, and the Bedrock premise test tolerates the transport's cachePoint block on current main.

[kshitijk4poor]

Merged via #85439 — your fix was adopted and composed into the wider Langfuse tracing PR by @erosika. Your contribution is credited in the commit body. Thanks @FnExpress!