fix(observability): restore Relay metrics without Anthropic context reentry

[afourniernv]

What does this PR do?

Reapplies the Relay runtime and shared-metrics work from #67607 after its revert in #73053, with the Anthropic streaming regression fixed.

The regression came from reusing one captured contextvars.Context across provider iteration, Relay collection, stream cleanup, and finalization. Those callbacks can overlap or nest, and Python rejects entering the same Context twice. Streaming callbacks now enter a fresh copy of the captured context while preserving the intended context values.

This PR adds a deterministic test that reproduced the exact production error on the reverted merge commit. It also adds an end-to-end test using the real Anthropic SDK parser against deterministic mock SSE, through Hermes' worker thread and Relay managed execution.

Related Issue

Restores #67607 after the regression-driven revert in #73053.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Reapply the reverted Relay runtime, plugin, shared-metrics, local export, and documentation changes on current main.
• Copy the captured callback context for each managed streaming surface instead of re-entering one shared Context.
• Add a deterministic regression for the concurrent Anthropic/Relay collector failure in tests/agent/test_relay_llm.py.
• Add an Anthropic SDK streaming test through Hermes' interruptible worker-thread path and Relay managed execution in tests/run_agent/test_streaming.py.

How to Test

1. Run .venv/bin/python -m pytest tests/agent/test_relay_llm.py tests/run_agent/test_streaming.py -q.
2. Run the broader Relay, streaming, delegation, lifecycle, and tool-security matrix described below.
3. Run .venv/bin/python scripts/smoke_nemo_relay_shared_metrics.py --hermes-repo . and verify the SQLite counters and schema-validated export.
4. For a live provider check, run a streamed NVIDIA one-shot call with shared metrics enabled in an isolated HERMES_HOME.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS arm64

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

N/A.

Screenshots / Logs

Before the fix, the new regression reproduced the production failure on merge commit 1470022:

RuntimeError: internal error: Python collector error: RuntimeError:
cannot enter context: <_contextvars.Context object ...> is already entered

Validation on the replacement branch with released nemo-relay==0.6.0:

• 715/715 across the merge-sensitive Relay, streaming, delegation, lifecycle, plugin, and tool-security matrix.
• 105 passed and 2 unrelated skips in a fresh frozen environment for the focused Relay and streaming suites.
• Ruff and git diff --check are clean.
• Native Hermes CLI smoke passed through Relay to SQLite and schema-validated JSON export; all expected base metric families were present and privacy canaries were absent.

Live-provider validation used credentials only from the process environment and isolated temporary HERMES_HOME directories:

• A real streamed NVIDIA call to nvidia/nemotron-3-nano-30b-a3b returned RELAY_METRICS_LIVE_OK in one API call.
• A real native Anthropic streamed call to claude-haiku-4-5-20251001 returned ANTHROPIC_RELAY_LIVE_OK in one API call.
• Neither run produced a context re-entry, collector, cleanup, or finalizer error.
• The Anthropic run persisted one successful remote Claude model call with provider_family=direct, model_family=claude, locality=remote, and outcome=success, plus successful task start/finish counters.
• Credentials were not printed or persisted in Hermes configuration.

The native Anthropic production path that originally regressed has therefore been exercised against Anthropic's live endpoint on this PR head.

[afourniernv]

@jquesnelle The replacement is ready for review. I reproduced the exact error from 1470022, fixed the shared streaming context re-entry, and clarified the validation boundary in the description: native Anthropic uses the real SDK with deterministic mock SSE, while the additional live network test uses NVIDIA and exercises the same Relay managed-stream implementation. CI is running on the replacement PR.

[afourniernv]

@jquesnelle Follow-up: the remaining live Anthropic validation gap is now closed. A real native Anthropic streamed Hermes turn through Relay against claude-haiku-4-5-20251001 returned ANTHROPIC_RELAY_LIVE_OK in one call. There were no context re-entry, collector, cleanup, or finalizer errors, and the isolated Relay SQLite store recorded one successful remote Claude model call plus successful task start/finish counters. The PR description has been updated with the exact boundary and result.

[afourniernv]

@mnajafian-nv Tagging for visibility: #67607 was merged, then reverted in #73053 after Anthropic streaming calls began failing with a contextvars.Context ... is already entered error. We reproduced the exact failure and traced it to one captured context being reused across overlapping provider, Relay collector, cleanup, and finalizer callbacks.

This PR reapplies the original Relay/shared-metrics change on current main and uses a fresh context copy for each streaming callback. The exact regression tests, all required CI checks, and a live native Anthropic streamed call now pass.

[afourniernv]

Caught this branch up with upstream main at f228e145b. The only textual conflict was in agent/chat_completion_helpers.py, where upstream added per-request client reuse and error-specific cleanup while this PR added Relay-managed streaming. The resolution preserves Relay stream cleanup first, then reuses the client only after a clean stream and closes it on error/cancellation.

The new upstream abort-race tests also exposed a semantic overlap beyond the conflict: Relay was masking provider stream-close failures, which could allow a client with a checked-out/broken connection to be cached. Explicit Relay stream close now surfaces that provider failure once, allowing both chat and Codex paths to poison the reuse slot correctly; normal exhaustion and destructor cleanup remain fail-safe.

Validation on head b1a5d67e7:

• 774/774 across Relay, shared metrics, streaming, Anthropic, Codex, request-client reuse, abort races, lifecycle, plugin, and tool-security suites
• Ruff and git diff --check clean
• Native Relay -> SQLite -> schema-validated metrics smoke passed
• Branch is current with upstream main

[afourniernv]

Incident triage

For visibility, here is the blameless readout for the Anthropic regression from #67607 and the follow-up work in this PR.

Timeline

• 4788994bd added caller ContextVar propagation after the July 23 deep review correctly found that managed Relay callbacks lost caller context.
• feat(observability): integrate NeMo Relay runtime and shared metrics #67607 merged at 2026-07-28 02:04 UTC as 1470022ad.
• Main was reverted through Revert "feat(observability): integrate NeMo Relay runtime and shared metrics" #73053 at 02:28 UTC, about 25 minutes later. The maintainer report was posted at 02:31 UTC, with b6244959a confirmed good and 1470022ad bad.
• The exact failure was reproduced, fixed in eda54775e, and replacement PR fix(observability): restore Relay metrics without Anthropic context reentry #73120 opened at 04:16 UTC.
• Current upstream was merged again in b1a5d67e7. That merge exposed a separate semantic overlap in provider stream cleanup; the resolution, request-client regression, and required E2E coverage are now pushed in 2dcd7448d and 956fc87ee.

What failed

The streaming adapter captured one contextvars.Context and reused that same object for the provider factory, iterator, collector callback, cleanup, and finalizer. A Python Context cannot be entered recursively or concurrently. Native Anthropic streaming through Relay's managed execution could overlap those surfaces and raised:

RuntimeError: cannot enter context: <_contextvars.Context ...> is already entered

The fix keeps the caller's values but enters a fresh callback_context.copy() for every streaming callback.

Impact

The observed production impact was native Anthropic streaming while a Relay managed-execution consumer was active; those calls failed before delivering a response. Because the unsafe code was in the generic managed stream adapter, other managed streaming providers had the same latent re-entry risk, although Anthropic is the provider we reproduced and live-verified.

Non-streaming calls used independent context copies and were not affected. Calls on the no-consumer bypass, unsupported/no-wheel platforms, and tool execution were also outside this failure path. This was a runtime availability regression, not a telemetry privacy or data-integrity incident.

Why green validation missed it

All required checks on #67607 were green, including all eight Python slices and E2E. The issue was missing coverage, not a skipped or broken runner.

The context regression added with 4788994bd asserted that each callback saw the caller's value, but consumed the stream serially. It proved propagation and did not exercise Context ownership or re-entry. The live checks also tested Anthropic request fidelity, generic stream fidelity, and caller-context propagation as separate invariants. They did not combine the real Anthropic SDK stream, Hermes' worker thread, Relay's native collector, and overlapping callback entry on the final head. The native metrics smoke used a local OpenAI-compatible server, so it could not cover this provider-specific execution shape.

The PR's size and repeated main merges made final-head review harder, but a merge conflict did not introduce this bug; the unsafe context reuse was already present in a normal feature commit and survived the merges unchanged.

Recovery and safeguards

The revert was the right containment action and landed quickly. The replacement now has:

• a deterministic test that holds the captured Context entered on another thread and reproduces the exact exception on the bad merge;
• a real anthropic==0.87.0 SDK stream test using httpx.MockTransport through Hermes' worker and Relay's managed lifecycle;
• live native Anthropic validation on claude-haiku-4-5-20251001 through Relay, with the response and persisted metrics verified;
• explicit provider-close error propagation and a Relay-managed regression proving the correct request-local client is poisoned when stream cleanup fails;
• the real SDK contract in tests/e2e, so the existing required E2E job runs it without API keys, network access, or a CI workflow change.

Focused validation is 4 passed; the affected unit files are 114 passed; the required E2E directory is 61 passed, 7 skipped; the CI classifier is 33 passed; Ruff, YAML parsing, and git diff --check are clean. Earlier validation on the replacement head also passed the 774-test Relay/streaming/client-lifecycle suite and the native Relay-to-SQLite metrics smoke.

For future core transport changes, the practical rule should be: test each native provider with its real SDK and mocked transport through the managed runtime, rerun that contract gate after the final main merge, and record any manual live canary against the exact head being proposed for merge. Live API calls should remain confirmation rather than the required CI gate.

[jquesnelle]

hey @afourniernv we merged this but are not getting the following errors in the logs:

Traceback (most recent call last):
  File "C:\Users\Jeffrey Quesnelle\AppData\Local\hermes\hermes-agent\agent\relay_runtime.py", line 727, in _finish_logical_calls
    lease.host.run_in_session(
  File "C:\Users\Jeffrey Quesnelle\AppData\Local\hermes\hermes-agent\agent\relay_runtime.py", line 217, in run_in_session
    return context.run(invoke)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jeffrey Quesnelle\AppData\Local\hermes\hermes-agent\agent\relay_runtime.py", line 213, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Jeffrey Quesnelle\AppData\Local\hermes\hermes-agent\venv\Lib\site-packages\nemo_relay\scope.py", line 144, in pop
    _native_pop_scope(handle, output=output, metadata=metadata, timestamp=timestamp)
RuntimeError: invalid argument: scope handle is not at the top of the stack```

[afourniernv]

Addressed in #73493, which has been merged. The fix was validated against the released NeMo Relay native binding and live Anthropic Messages, OpenAI Chat Completions, and OpenAI Responses endpoints, with 142 focused regression tests passing.