Phoenix LLM spans now show with real prompts and completions

[mpenn]

Summary

End-to-end NeMo-Relay observability for Hermes via a single persistent sidecar gateway. One daemon serves every Hermes surface — PID-1 gateway, Slack/Outlook bridge-driven turns, and interactive hermes chat TUI — producing per-turn ATIF trajectory files at /tmp/atif/*.json plus Phoenix LLM spans carrying the real prompt + completion, paired tool spans (no orphans), and per-turn agent root spans. All four surfaces land in the same correlated Phoenix session.

Architecture

start.sh launches a long-running NeMo-Relay daemon at sandbox startup (start_nemo_relay_sidecar helper, listens on 127.0.0.1:4040). Every Hermes process discovers it via NEMO_RELAY_GATEWAY_URL in env — exported explicitly on PID-1's launch list and via _PROXY_ENV_FILE (sourced by /sandbox/.bashrc) for interactive shells. The daemon fails hard on /healthz timeout: dumps the last 30 lines of /tmp/nemo-relay.log to stderr and exit 1s. Silent telemetry loss is worse than a noisy startup error.

Two event paths feed the daemon:

• Shell hooks (generate-config.ts): on_session_*, pre/post_llm_call, subagent_stop via nemo-relay hook-forward hermes.
• In-process plugin (plugins/nemo-relay/): pre/post_api_request + pre/post_tool_call — forwards the rich kwargs (full request messages, response SDK object, stable tool_call_ids) that the shell path strips.

nemo-relay-finalize-hook runs as a second on_session_end command to synthesize a per-turn on_session_finalize (Hermes's native finalize only fires on idle expiry). Daemon dispatches to ATIF (always-on) and OpenInference (gated by PHOENIX_COLLECTOR_ENDPOINT).

Interactive hermes invocations from sandbox shells resolve through a PATH-prepended CLI shim at /usr/local/lib/nemoclaw/bin/hermes — warns on setup/doctor about the in-sandbox config lifecycle and execs the upstream binary for everything else. Telemetry flows because the shell's NEMO_RELAY_GATEWAY_URL propagates to the exec'd process; the in-process plugin POSTs to the sidecar like any other Hermes surface.

Sandbox workarounds

Seven non-obvious places where the OpenShell L7 proxy, OpenShell exec-session allowlist, OpenTelemetry batching, and PTY relay break observability or TUI bootstrap. Diagnosed empirically — keep these in mind if anything regresses.

1. BSP batching silently dropped by L7 proxy. Default OTel batches (5s / 512 spans) get 200-acked by the proxy but not forwarded. Single-span POSTs land. → OTEL_BSP_MAX_EXPORT_BATCH_SIZE=1 / SCHEDULE_DELAY=100 / EXPORT_TIMEOUT=2000 (_export_otel_bsp_tunings() helper in start.sh, applied to the sidecar's env).
2. Asymmetric tool_call_id between pre and post. Hermes fires pre_tool_call with empty id, post_tool_call with the real one. NeMo-Relay's adapter synthesizes a fresh UUID per call → 2 unpaired spans per invocation (one carries {"status":"closed_by_agent_end"}). → Plugin synthesizes stable IDs from SHA1(task_id, tool_name, args) and pairs pre/post via a bounded deque(maxlen=512) FIFO.
3. L7 proxy fail-closed scan rejects OTLP bodies containing openshell:resolve:env:. SOUL.md taught the model that pattern by literal example → it rode in every LLM request's system prompt → OTLP egress carrying it got rejected, silently losing all LLM spans. → SOUL.md "Credential placeholders" section rewritten to teach the rule (recognize structured placeholders, use verbatim, don't refuse/parse/echo) without printing the trigger string.
4. Phoenix input.value was a lossy "Requested tools: …" summary. OpenInference's display fn finds messages via content.get("messages"). Sending the body as a bare list missed that path; the fallback picked up tool-role messages' name field. → Plugin wraps as {"messages":[...], "model":..., "max_tokens":...} per NeMo-Relay's documented LlmRequest.content shape. Phoenix now shows system: … \n\n user: <prompt> \n\n assistant: ….
5. TUI invocations bypassed telemetry. Raw hermes/hermes chat launched the upstream binary without NEMO_RELAY_GATEWAY_URL in env, so the in-process plugin no-op'd → ATIF and Phoenix never saw TUI turns. Direct-path /usr/local/bin/hermes and non-bash shells bypassed any bash-function fix entirely. → PATH-prepended CLI shim at /usr/local/lib/nemoclaw/bin/hermes (installed via Dockerfile, prepended to PATH from _PROXY_ENV_FILE). Works for any shell — bash, zsh, sh, scripts. Shim blocks setup/doctor with a one-line lifecycle warning, then execs /usr/local/bin/hermes for every subcommand. Telemetry rides on NEMO_RELAY_GATEWAY_URL already in shell env.
6. hermes chat banner hung on PyPI fetches. TUI banner enumeration triggers lazy installs (e.g. _lazy_ensure("tts.edge") → uv pip install edge-tts), but the sandbox can't reach PyPI by policy. → Pre-install the relevant extras via HERMES_UV_EXTRAS="messaging web cli edge-tts"; set HERMES_DISABLE_LAZY_INSTALLS=1 as belt-and-suspenders for anything not pre-installed (set in both image ENV — for the gateway PID 1 — and _PROXY_ENV_FILE — for interactive shells, since the allowlist strips it from ENV).

Plumbing

• Hermes v0.14.0 for rich-kwargs hooks (request_messages, assistant_message, response SDK object). uv-tarball install with SHA256 verification.
• NeMo-Relay CLI 0.3.0-beta.2 built from nemo-relay-cli in the existing builder stage. (Upstream renamed nemo-flow-cli → nemo-relay-cli at 0.3.0; this PR includes the rebrand migration.)
• Layout: in-process plugins under agents/hermes/plugins/{nemoclaw,nemo-relay}/; non-plugin assets under agents/hermes/nemo-relay/{finalize-hook,hermes-cli-shim,plugins.toml.in}.
• Install paths: /usr/local/bin/nemo-relay (upstream daemon); /usr/local/lib/nemoclaw/bin/{hermes,nemo-relay-finalize-hook} (example-owned helpers — kept off default PATH so PID-1 hermes resolves the upstream binary directly).
• /etc/nemo-relay/plugins.toml carries the observability config (ATIF + OpenInference). No config.toml — daemon mode doesn't use [agents.hermes].
• policy.yaml: phoenix_collector egress for /usr/local/bin/nemo-relay.
• NemoClaw plugin: registers nemoclaw_status / nemoclaw_info / nemoclaw_reload_skills tools. No startup banner — info is on-demand via the tools.

Test plan

• bash scripts/tear-down.sh && bash scripts/bring-up.sh builds clean
• pgrep -af nemo-relay shows exactly one long-running daemon process (no per-invocation ephemeral gateways)
• curl -sf http://127.0.0.1:4040/healthz from inside the sandbox returns {"status":"ok"}
• ls /sandbox/.hermes-data/plugins/ shows nemoclaw/ and nemo-relay/
• Drive a Slack DM that uses a tool. In Phoenix: LLM spans with full prompt in input.value; one TOOL span per invocation; agent root closes per turn
• Drive an Outlook message → bridge → PID-1 hermes → confirm telemetry correlates to the same sidecar process as Slack
• /tmp/atif/nemo-relay-atif-*.json lands per turn with full messages array
• Phoenix logs show POST /v1/traces 200 OK per span
• Unset PHOENIX_COLLECTOR_ENDPOINT, rebuild → ATIF still works, no Phoenix export
• openshell sandbox connect → which hermes returns /usr/local/lib/nemoclaw/bin/hermes; type hermes shows the file path (not a function)
• hermes chat from the connected shell. Banner renders cleanly (no ]11;rgb:… garbage in the input field, no PyPI-fetch hang). Drive a turn that uses a tool. Confirm the Phoenix span and /tmp/atif/*.json land for the TUI turn and correlate to the same sidecar PID as the Slack/Outlook turns above.
• In the TUI, Ctrl+C (or /exit) returns the user to the sandbox shell prompt — the openshell sandbox connect session stays alive
• bash -c 'hermes setup': shim prints the in-sandbox config lifecycle note, then runs the upstream wizard
• /usr/local/bin/hermes setup (full path) bypasses the shim warning (known and intentional — integrity hash at /sandbox/.hermes/.config-hash is the real boundary)

Pending upstream work

Two Hermes upstream PRs may obsolete parts of this PR when they land:

• NousResearch/hermes-agent#29722 — adds a plugin middleware contract with first-class telemetry hooks and adaptive-middleware execution intercepts.
• NousResearch/hermes-agent#29724 — adds an optional NeMo-Flow (NeMo-Relay) telemetry plugin built directly into Hermes.

When these land, the follow-up cleanup is small and well-scoped:

• Retire the in-process plugin (plugins/nemo-relay/__init__.py + its directory + the "nemo-relay" entry in plugins.enabled in generate-config.ts + the Dockerfile COPY of it). The upstream plugin from #29724 takes over pre/post_api_request and pre/post_tool_call forwarding.
• Workaround item 2 (asymmetric tool_call_id) becomes unnecessary if the middleware refactor in #29722 smooths Hermes's pre/post call sites or if the upstream plugin emits stable IDs.
• Workaround item 4 (Phoenix input.value lossy summary) becomes unnecessary if the upstream plugin already wraps the request body in {messages, model, max_tokens} shape.

[pastorsj]

Testing this based on the test plan. Things seem good to me.

Screen.Recording.2026-05-25.at.4.37.46.PM.mov

• bash scripts/tear-down.sh && bash scripts/bring-up.sh builds clean (confirmed)
• pgrep -af nemo-flow shows the gateway with Hermes as its child (confirmed)
• ls /sandbox/.hermes-data/plugins/ shows nemoclaw/ and nemo-flow/ (confirmed)
• Drive a Slack DM that uses a tool. In Phoenix: LLM spans with full prompt in input.value; one TOOL span per invocation; agent root closes per turn
• /tmp/atif/nemo-flow-atif-*.json lands per turn with full messages array (confirmed)
• Phoenix logs show POST /v1/traces 200 OK per span (see video)
• Unset PHOENIX_COLLECTOR_ENDPOINT, rebuild → ATIF still works, no Phoenix export
  openshell sandbox connect → run hermes chat. Banner renders cleanly (no ]11;rgb:… garbage in the input field, no PyPI-fetch hang). Drive a turn that uses a tool. Confirm a Phoenix agent root span and a /tmp/atif/*.json land for the TUI turn
• In the TUI, Ctrl+C (or /exit) returns the user to the sandbox shell prompt — the openshell sandbox connect session stays alive

- [x] TUI banner shows Hermes hooks at /sandbox/.hermes/config.yaml (not the "not yet installed" warning)

[pastorsj]

LGTM