feat(relay)!: initialize static/dynamic plugins via native integration, remove opt-in plugin

[bbednarski9]

Warning

Breaking change — migration is required for existing NeMo Relay plugin users.

This PR removes the bundled Hermes observability/nemo_relay plugin and moves
plugin.initialize() lifecycle ownership into the core RelayRuntime. Before
upgrading, remove observability/nemo_relay (or the legacy nemo_relay alias)
from plugins.enabled and move exporter/middleware configuration into a Relay
plugins.toml selected with HERMES_NEMO_RELAY_PLUGINS_TOML.

Legacy HERMES_NEMO_RELAY_ATOF_* and HERMES_NEMO_RELAY_ATIF_* variables no
longer activate exporters. When HERMES_NEMO_RELAY_PLUGINS_TOML is unset,
Hermes does not initialize Relay plugins and Relay performs no plugin discovery
or configuration layering. When it points to a valid file, Relay performs its
normal static discovery/layering and Hermes owns the resulting process-global
activation. If another host already has an active Relay plugin configuration,
Hermes leaves it unchanged and disables Hermes-managed Relay middleware rather
than replacing it.

Relay plugin policy is a process boundary. The selected static configuration,
discovered layers, middleware, exporters, guardrails, and dynamic plugins apply
to every profile hosted by that Hermes process. ATIF keeps top-level Agent
trajectories separate, while ATOF and other global subscribers observe events
from every hosted profile. Run profiles in separate Hermes processes when they
require different trust levels, plugin credentials, exporter destinations, or
guardrail policies.

Upgrade safeguards in this PR migrate stale plugins.enabled entries out of
config v35, warn through hermes doctor and runtime diagnostics, and reject
user-installed copies of the removed plugin identity.

What does this PR do?

This PR adds explicitly configured NeMo Relay plugin activation to Hermes's native in-process Relay runtime and removes the overlapping bundled observability/nemo_relay Hermes plugin.

The native SDK integrations remain in place: Hermes provides Relay session, turn, task, and manual scopes, shared-metrics lifecycle events, and SDK-managed LLM/tool adapter paths independently of whether plugin activation is enabled.

The native runtime calls neither nemo_relay.plugin.initialize(...) nor initialize_with_dynamic_plugins(...) unless HERMES_NEMO_RELAY_PLUGINS_TOML is set to a non-empty path and the selected file loads successfully. With the variable unset, native Relay instrumentation continues normally, but Hermes does not activate Relay middleware, exporters, dynamic plugins, or ambient plugins.toml discovery.

When explicitly enabled and the selected file loads successfully, Hermes passes its static components to Relay as a programmatic overlay. Relay performs its normal static plugins.toml discovery and layering, with the selected static configuration taking precedence. Hermes delegates canonical [[plugins.dynamic]] manifest resolution and validation to Relay 0.7.1 before opening the first Relay session scope; dynamic records are loaded from the selected file only. The unreleased Hermes-specific top-level [[dynamic_plugins]] format and parser have been removed. If the selected file is missing, unreadable, malformed, contains an invalid dynamic declaration, or Relay rejects it, Hermes reports the failure and continues without native plugin activation. Because Hermes does not invoke Relay's initializer on that failure path, it does not fall back to ambient discovery.

Initialization and activation are process-wide. The first hosted profile's plugin decision is represented explicitly as disabled, active, foreign, or failed, and additional profile-scoped Hermes hosts share that decision rather than retrying or selecting a different policy. Successful initialization emits Relay plugins are active process-wide and apply to all profiles hosted by this Hermes process. exactly once. After the final host stops admitting work and active operations drain, Hermes closes sessions, flushes subscribers/exporters, and clears or closes the Relay plugin activation. Per-session close does not perform a global subscriber flush, avoiding a concurrent-session event-loop deadlock.

Profile scopes preserve causal isolation inside that shared policy. ATIF produces separate trajectories for separate top-level Agent scopes, while ATOF and other global subscribers observe activity from every hosted profile. Static and dynamic middleware applies to calls from every profile, and worker invocations retain the invoking profile's Relay scope stack. Profiles requiring different trust, plugin credentials, exporter destinations, or guardrail policies must run in separate Hermes processes.

Removing plugins/observability/nemo_relay establishes a single lifecycle owner for Relay configuration. Deployments that previously enabled that Hermes plugin should remove it from their enabled-plugin configuration and express middleware/exporter configuration in the TOML selected by HERMES_NEMO_RELAY_PLUGINS_TOML. Hermes-native shared metrics do not require the removed plugin.

This PR relies on the Relay >=0.7.1,<0.8 requirement already present on main; it does not modify pyproject.toml or uv.lock.

Because Relay 0.7.1 is the minimum supported version, the native runtime calls initialize, initialize_with_dynamic_plugins, load_dynamic_plugin_activation_specs, flush_async, and clear_async directly. It retains fail-open handling for operational failures, but no longer carries Relay 0.6-era missing-method checks or synchronous cleanup fallbacks.

Why remove observability/nemo_relay?

The bundled Hermes plugin and the native runtime were two independent owners of the same process-global Relay state. If both were enabled, each could initialize Relay from its own configuration and register middleware, subscribers, exporters, or dynamic plugins. That made behavior dependent on startup order and allowed several invalid or buggy configurations:

• the same exporter or subscriber could be installed twice, producing duplicate events, traces, or trajectory output;
• middleware could wrap the same Hermes LLM or tool operation more than once;
• one integration could replace or clear configuration owned by the other;
• one integration could flush subscribers or close dynamic-plugin activation while another Hermes host or managed operation was still using it;
• separate configuration paths could disagree about which components were active, making failures difficult to diagnose.

The native integration now provides the complete supported path: Hermes owns session and operation context, while Relay owns plugin parsing, activation, exporter/subscriber registration, and dynamic-plugin lifecycle. Removing the overlapping Hermes plugin gives that process-global state one owner and one teardown boundary. It does not remove Relay observability or shared metrics; deployments configure those components in the plugins.toml selected by HERMES_NEMO_RELAY_PLUGINS_TOML.

Related Issue

Relates to NVIDIA/NeMo-Relay#694

Type of Change

• ⚠️ Breaking change (requires migration from observability/nemo_relay to core RelayRuntime ownership)
• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Relay cutover safeguards
  • migrate config v35 by removing only the legacy Relay identities from plugins.enabled;
  • warn on stale plugin keys and ignored legacy ATOF/ATIF exporter variables;
  • reject user/entry-point copies of the removed plugin identity;
  • leave a pre-existing foreign process-global Relay configuration unchanged instead of replacing it.
• agent/relay_runtime.py
  • gates native Relay plugin initialization on an explicit HERMES_NEMO_RELAY_PLUGINS_TOML path;
  • enables Relay's normal static discovery and layering only after the selected TOML loads successfully;
  • layers selected static components over discovered configuration and loads canonical [[plugins.dynamic]] records from the selected file only;
  • rejects the unreleased Hermes-specific [[dynamic_plugins]] format instead of maintaining a second parser;
  • uses the required Relay 0.7.1 plugin and subscriber APIs directly, without obsolete capability checks or synchronous cleanup fallbacks;
  • fails open without invoking Relay initialization or falling back to ambient discovery when the selected file is invalid;
  • owns activation and final cleanup process-wide;
  • records the first hosted profile's process decision with an explicit state enum and shares it with later hosts;
  • emits one process-wide diagnostic only after plugin initialization completes successfully;
  • drains deferred managed operations before teardown and avoids per-session global subscriber flushes.
• agent/relay_llm.py
  • preserves managed-operation ownership through stream completion and cleanup;
  • uses canonical Relay operation names and keeps client-only timeouts off provider wire payloads;
  • safely handles completed-response and streaming provider paths.
• agent/chat_completion_helpers.py and agent/transports/chat_completions.py
  • tolerate sparse SDK response and stream-delta objects when optional fields are absent.
• plugins/observability/nemo_relay/
  • removed to prevent duplicate initialization, activation, exporter, and teardown ownership;
  • native shared metrics and Relay plugin activation remain available through core.
• Documentation and scripts/toolperf_abeval
  • migrate references and exporter setup to the Hermes-native TOML path.
• Tests
  • cover explicit configuration gating, initialization ordering, canonical dynamic loading, legacy-format rejection, static/dynamic ownership, exporter output, stream cleanup, and concurrent shutdown;
  • prove static and dynamic middleware reaches two profile hosts with distinct Relay scope stacks;
  • prove ATOF observes two profiles while ATIF keeps their top-level Agent trajectories separate;
  • remove tests for the deleted legacy integration.

How to Test

scripts/run_tests.sh \
  tests/agent/test_relay_runtime_plugins.py \
  tests/agent/test_relay_llm.py \
  tests/agent/test_relay_tools.py \
  tests/hermes_cli/test_plugins_cmd_enable_disable_nested.py \
  tests/hermes_cli/test_plugins_cmd_category_discovery.py \
  tests/hermes_cli/test_relay_shared_metrics_runtime.py \
  tests/hermes_cli/test_relay_shared_metrics.py
uv run ruff check \
  hermes_cli/plugins_cmd.py \
  scripts/toolperf_abeval/ab_eval.py \
  tests/hermes_cli/test_plugins_cmd_enable_disable_nested.py
uv lock --check

Cutover guard validation: 166 passed across the full config, doctor, and plugin-manager suites, plus 31 passed in the focused Relay ownership/migration suite. Ruff and whitespace checks passed.

Focused validation after removing the legacy plugin: 224 passed. The final canonical configuration and Relay 0.7.1 API tightening was additionally validated with 217 passed across the native Relay, nested-execution, shared-metrics, and smoke suites. After rebasing onto current main, all 12 Python CI slices and every required check passed.

Latest process-policy follow-up: 26 passed in the focused plugin-ownership suite and 208 passed across Relay runtime, LLM, tool, and shared-metrics tests. Ruff and whitespace checks passed.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS arm64

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A; this uses an existing deployment environment variable
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

For New Skills

Not applicable.

Screenshots / Logs

224 focused Relay, shared-metrics, and plugin-discovery tests passed
217 native Relay, nested-execution, shared-metrics, and smoke tests passed after canonical config and 0.7.1 API tightening
Post-rebase CI: all 12 Python slices and all required checks passed
Process-policy follow-up: 208 Relay runtime, LLM, tool, and shared-metrics tests passed
Ruff passed
uv lock --check passed

[yczhang-nv]

I tested this with a real AIAgent turn through Hermes → Relay #694 → the Switchyard #270 native plugin → NVIDIA-hosted models. Plugin activation succeeds, but the unmodified full-agent path exposes three pre-existing Hermes Relay-adapter gaps that prevent the integration from working end to end:

1. Relay operation identity

   Hermes passes name=agent.provider (for example, custom) while separately providing metadata.api_mode. Switchyard manages the canonical Relay operations:

   • chat_completions → openai.chat_completions
   • codex_responses → openai.responses
   • anthropic_messages → anthropic.messages

   With name="custom", Switchyard treats the call as unmanaged and Hermes’ original provider callback runs instead.

   The minimal fix is to derive the Relay operation name from api_mode at the relay_llm adapter boundary for synchronous, asynchronous, and streaming managed execution. Unknown modes should retain the original provider name.

2. Client-only timeout leaks into the Relay wire request

   _relay_request_body() serializes the provider SDK kwargs, including timeout=1800.0. Switchyard’s same-protocol preservation correctly forwards that field, but timeout is an OpenAI Python client option—not part of the Chat Completions JSON protocol—and NVIDIA rejects it with HTTP 400.

   Removing timeout only from the Relay-facing copy fixes this while preserving it for Hermes’ original provider callback when Relay does not intercept.

3. Optional OpenAI-compatible response fields are treated as required

   Managed responses can validly omit optional message.tool_calls, delta.content, or delta.tool_calls. Direct attribute access causes Hermes normalization to fail after a successful plugin response. These accesses should use defensive getattr(..., None) handling.

The e2e test works after I applied those changes locally.

The issues are pre-existing before the PR, so either creating a separate PR for those fixes, or including the fixes in this PR works for me.

[bbednarski9]

@yczhang-nv

1. is addressed in 8cd3ecf
2. addressed in 8a84e4a
3. addressed in 623be67

[afourniernv]

Blocking: concurrent session close can deadlock the active asyncio loop

I reproduced a hard deadlock at the current PR head (08c76bb6) when one session closes while another session has an active managed Relay publication.

The path is:

1. _close_session() calls the process-wide _flush_relay_subscribers().
2. _resolve_plugin_awaitable() starts a helper thread for flush_async().
3. The asyncio event-loop thread synchronously waits in thread.join().
4. Relay's flush waits for the other active managed publication.
5. That publication needs the event loop, which is blocked waiting for the flush thread.

A four-way comparison isolated the interaction:

• Parent Hermes + previous Relay dependency: completes.
• PR head + previous Relay dependency: completes.
• Parent Hermes + the Relay dependency required by this PR: reports that synchronous flush cannot block a running event loop, then continues.
• PR head + the required Relay dependency: hangs after the first session begins closing; watchdog terminated it after 3 seconds.

The sequential control completed in about 2 ms. The trigger is concurrent sessions sharing one Relay runtime, not normal sequential close.

The per-session process-wide flush predates this PR, but the new awaitable bridge changes the failure mode into a hard deadlock. I think this needs to be fixed before merge. The narrow options are to remove the global subscriber flush from per-session close and perform it once during process/runtime shutdown after active operations drain, or make session close fully asynchronous and await it without blocking the event-loop thread.

Separate non-regression found during the same lifecycle audit: one-shot execution hard-exits without emitting the root Relay session end. I filed that independently as #79471.

[afourniernv]

Stable nemo-relay 0.7.0 is now published on PyPI. I updated the dependency constraint and uv.lock and opened a focused PR directly against this PR's head branch: bbednarski9#1

Validation: uv lock --check passes and all 18 Relay plugin-runtime tests pass against the stable package.

[bbednarski9]

Intentionally differed items from review:

• Public Relay activation/readiness status. Reason: this is out of scope for this PR
• Relay 0.7 cross-call-starvation regression test. Reason: this would add tech debt
• One-shot root Relay session end [Bug]: One-shot execution exits without closing the Relay session lifecycle #79471. Reason: out of scope addressed in separate work
• Release comms. Reason: out of scope.

@andrexibiza we have addressed your comments above, caught up to main, and CI is green.

CC @afourniernv

[andrexibiza]

Approval-style re-review of exact current head 0e5499b15300f10ad51a211ccf1c76824193feb7 against PR base b5455fdd16fe608214f91149233660e1836b067c and current main d21cd51d860454b921abf9f2fe1d5dfd4c071e33. This is a materially new head since my prior review at 00d108e075480fd3d22a3d03f5121cbb1998ef4d, so I rechecked the blocker, lifecycle ownership, tests, CI, and adjacent Relay work rather than repeating the old review.

Previous blocker is closed

The earlier problem was a state-collapse at the configuration load boundary: an unset HERMES_NEMO_RELAY_PLUGINS_TOML and an explicitly selected but missing/malformed/invalid configuration both became DISABLED. That is fixed structurally now.

_configured_plugin_inputs() returns None only when no non-empty config path was selected. Once a path is explicitly selected, load/parse/dynamic-spec failures raise _RelayPluginConfigurationLoadError; _ProcessRelayPluginConfiguration.acquire() catches that under the failure path and records FAILED. The process state now preserves the intended four-way policy:

• no explicit selection → DISABLED;
• explicit selection that cannot be loaded/validated/initialized → FAILED;
• pre-existing process-global Relay activation not owned by Hermes → FOREIGN;
• successful Hermes-owned activation → ACTIVE.

The new tests close the exact acceptance edges I asked for: missing selected config is FAILED for the first and later hosted profile; malformed config is FAILED; an invalid plugins section is FAILED; legacy [[dynamic_plugins]] is rejected into FAILED; initialization failures stay FAILED for concurrent hosts; and only after the final failed owner exits may a later host retry and become ACTIVE. The old DISABLED/FAILED ambiguity is gone.

Lifecycle / other side of the shape

I also rechecked the process-global teardown boundary because that is the dangerous half of this cutover.

• @afourniernv's reproduced concurrent-session deadlock is addressed at the ownership layer, not papered over: per-session close no longer performs a process-wide subscriber flush. Final teardown waits for tracked operations to drain, then closes session scopes and flushes/clears the process plugin configuration once.
• RelayOperationLease extends runtime/plugin lifetime across managed LLM streams and their deferred logical completion/cleanup, so teardown cannot invalidate the process-global activation while a stream still owns Relay work.
• The two-profile tests prove one process activation with distinct profile/session scope stacks, no teardown after the first host exits, and teardown only after the final host. The real binding witness also proves ATOF sees both hosted profiles while ATIF trajectories remain separated by top-level Agent scope.
• @yczhang-nv's earlier full-agent integration findings are represented in the current implementation: canonical operation identity, removal of client-only timeout from the Relay wire payload, and tolerant handling of sparse optional response fields.
• Removing plugins/observability/nemo_relay now leaves one lifecycle/configuration owner instead of the previous core-plugin double ownership. That is the right class-level fix.

I do not see a new merge blocker on this head.

CI / merge state

Exact-head CI is now real and green: the main CI workflow completed successfully, as did Docker and Nix. The executed matrix includes blocking Ruff, Ruff+ty diff, E2E, Windows footguns, Windows-only tests, macOS-only tests, docs, attribution and supply-chain/OSV checks. GitHub currently reports the PR mergeable against main.

Interlocks / provenance

• NVIDIA/NeMo-Relay#694 is the upstream API/format dependency this consumes; complementary, not duplicated here.
• @poisdahl's deterministic Relay 0.6 cross-call-starvation regression remains valuable adjacent evidence and should retain attribution. It was explicitly dispositioned rather than absorbed into this PR. The production dependency floor is now nemo-relay>=0.7.1,<0.8, so this cutover no longer claims compatibility with the 0.6 worker-pool behavior that regression demonstrates.
• #79471 (one-shot root Relay session finalization) remains separate work rather than a reason to hold this lifecycle-owner cutover.
• A public non-secret Relay readiness/status surface remains a downstream/integration follow-up. The important prerequisite from this PR is now satisfied: FAILED and FOREIGN remain truthful process states instead of being collapsed into DISABLED, so a future admission/readiness consumer can fail closed without reconstructing intent from logs.

Verdict: clear on exact head 0e5499b15300f10ad51a211ccf1c76824193feb7. I attempted to submit this as an APPROVE review, but the connected GitHub integration is not permitted to file an approval review on this repository; posting the same exact-head verification as a formal COMMENT review instead. A later head that changes Relay configuration classification, operation leases, shutdown/drain ordering, or the plugin cutover should be re-reviewed.