rapids-bot · Aug 18, 2026 · Aug 16, 2026 · Aug 16, 2026 · Aug 18, 2026 · Aug 18, 2026
@@ -32,7 +32,7 @@ evaluation products, see [the Ecosystem guide](https://docs.nvidia.com/nemo/rela
 | Instrument app-owned LLM or tool calls | [Quick Start Application](https://docs.nvidia.com/nemo/relay/getting-started/quick-start) |
 | Use LangChain, LangGraph, Deep Agents, or OpenClaw | [Supported Integrations](https://docs.nvidia.com/nemo/relay/supported-integrations/about) |
 | Build a framework or provider integration | [Integrate into Frameworks](https://docs.nvidia.com/nemo/relay/integrate-into-frameworks/about) |
-| Export ATOF, ATIF, or typed OpenTelemetry projections | [Observability Plugin](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about) |
+| Export ATOF events, ATIF trajectories, or OpenTelemetry traces, logs, and metrics | [Observability Plugin](https://docs.nvidia.com/nemo/relay/configure-plugins/observability/about) |
 | Package reusable middleware or exporters | [Build Plugins](https://docs.nvidia.com/nemo/relay/build-plugins/about) |
 | Develop or test this repository from source | [CONTRIBUTING.md](CONTRIBUTING.md) |
 
@@ -256,8 +256,8 @@ Relay gives those systems:
   route, retry, or replace execution.
 - **Plugins** so reusable observability, guardrail, adaptive, and exporter
   behavior can be turned on from configuration.
-- **Events and subscribers** so raw ATOF, normalized ATIF, and typed
-  OpenTelemetry projections all come from the same runtime stream.
+- **Events and subscribers** so raw ATOF, normalized ATIF, and OpenTelemetry
+  traces, logs, and metrics all come from the same runtime stream.
 
 Relay does not replace frameworks, model provider, application logic,
 observability backend, or guardrail authoring system. It gives those systems a

@@ -46,8 +46,13 @@ binding consumes it through CGo.
   isolation.
 - **Typed OpenTelemetry export**:
   `nemo_relay_otel_subscriber_create` constructs one `full`, `gen_ai`, or
-  `openinference` endpoint subscriber. The endpoint and projection type are
-  required.
+  `openinference` trace subscriber. Independently managed log and metric
+  subscribers use `nemo_relay_otel_log_subscriber_create` and
+  `nemo_relay_otel_metric_subscriber_create`.
+- **Structured marks and metrics**: `nemo_relay_event_v2` adds optional data
+  schema JSON and `NemoRelayLogSeverity` to the compatible mark API.
+  `nemo_relay_metric_json` and `nemo_relay_metric` emit atomically validated
+  Relay metric measurements.
 - **Generated header**: A committed `nemo_relay.h` file for C-compatible
   consumers.
 - **Native library outputs**: Shared and static libraries for platform
@@ -63,6 +68,28 @@ long-running callback work therefore occupy that thread and can reduce
 middleware throughput. The FFI does not expose completion-based middleware
 registration.
 
+## OTLP Logs and Metrics
+
+Use `nemo_relay_event_v2` when a mark needs a data schema or OTLP log severity.
+Pass `data_schema_json` as `{"name":"example.schema","version":"1"}` and a
+`NemoRelayLogSeverity` pointer when applicable. The legacy `nemo_relay_event`
+function remains valid for untyped marks.
+
+Use `nemo_relay_metric_json` for a nonempty JSON array of canonical metric
+measurements, or `nemo_relay_metric` for `NemoRelayMetricMeasurement` entries.
+Relay validates a complete measurement group before it emits any recording
+operation. Do not construct the reserved metric mark schema manually.
+
+Create each direct OTLP log or metric subscriber independently. Register it
+with `nemo_relay_otel_log_subscriber_register` or
+`nemo_relay_otel_metric_subscriber_register`. During graceful shutdown,
+deregister the name, force-flush the subscriber, shut it down, and free its
+handle with the matching `nemo_relay_otel_*_subscriber_free` function. The log
+and metric APIs also expose bounded JSON runtime-diagnostics snapshots through
+their `nemo_relay_otel_*_subscriber_runtime_diagnostics_json` functions. The
+caller owns each returned `out_json` string and must release it with
+`nemo_relay_string_free`.
+
 ## Installation
 
 Build the FFI library from a repository checkout:

@@ -53,6 +53,12 @@ the dynamic-library boundary on the stable C-compatible ABI.
   continuations use a pull-based host handle.
 - **Canonical tool results**: `ToolNext` returns `ToolExecutionResult`, keeping
   application results and opaque annotations adjacent across native API 1.
+- **Typed telemetry marks**: `PluginRuntime::emit_mark_with_options` carries a
+  `DataSchema` and `LogSeverity`; `emit_metric` validates and emits typed
+  `MetricMeasurement` values through the reserved Relay metric schema.
+- **Runtime diagnostics**: `PluginRuntime::runtime_diagnostics()` reads the
+  active host-level `RuntimeDiagnostics` snapshot, with ordered entries and
+  `get(code)` lookup.
 
 ## Installation
 
@@ -151,6 +157,16 @@ can override that method when they need different configuration rules.
 During plugin teardown, the SDK stops accepting new callbacks and drains
 already accepted typed middleware before the plugin library unloads.
 
+## Runtime Diagnostics
+
+`PluginRuntime::runtime_diagnostics()` returns `RuntimeDiagnostic { code,
+message, count }` entries for the active host configuration. Relay aggregates
+repeated codes, retains the latest message, saturates counts, orders entries by
+code, and caps the snapshot at 32 entries. The result is host-level and does
+not attribute an entry to the plugin that emitted it. ABI v4 requires the
+complete finalized host table. Plugins loaded through the frozen ABI-v3 or
+ABI-v2 compatibility tables cannot use runtime diagnostics.
+
 Relay scope context is restored around every poll of a registered middleware
 future. Child tasks created with `tokio::spawn` do not automatically inherit
 that scope context.

@@ -71,6 +71,13 @@ JSON values use `JsonValue`, whose bytes contain exactly one JSON value; this
 preserves JSON integers and other application data without the numeric coercion
 of `google.protobuf.Value`. Hosts and SDKs reject a missing required `result`
 or invalid JSON bytes. JSON null annotations normalize to absence.
+- **Additive mark options**: `EmitMarkRequest.data_schema` carries a
+  `nemo.relay.DataSchema@1` envelope and `severity` carries the canonical log
+  severity string. Omitting both is wire-compatible with legacy workers.
+- **Runtime diagnostics**: Authenticated `GetRuntimeDiagnostics` returns the
+  bounded active-host `{ code, message, count }` snapshot. Older hosts return
+  gRPC `UNIMPLEMENTED`; current SDKs surface that as an explicit unsupported
+  runtime-diagnostics error.
 
 ## Installation
 

@@ -93,6 +93,18 @@ Relay supplies the socket, activation ID, and authentication token through the
 worker environment. Use `serve_plugin` for Relay-spawned workers; explicit
 server configuration is intended for tests and custom launchers.
 
+`PluginRuntime::emit_mark` retains its original positional contract.
+`emit_mark_with_options` adds an optional `DataSchema` and `LogSeverity`, and
+`emit_metric` validates typed `MetricMeasurement` values before emitting the
+reserved Relay metric schema.
+
+`PluginRuntime::runtime_diagnostics().await` returns an ordered host-level
+snapshot with `RuntimeDiagnostic { code, message, count }` entries and
+`get(code)` lookup. Relay aggregates repeated codes, retains the latest
+message, saturates counts, and returns at most 32 entries. It does not provide
+per-plugin attribution. An older host returns `UNIMPLEMENTED`, which the SDK
+reports as an unsupported-runtime-diagnostics error.
+
 ## Concurrency and Cancellation
 
 Unary and streaming callbacks run concurrently. Cancellation is cooperative:

@@ -68,8 +68,10 @@ Every event carries a shared envelope:
 
 Scope events add `scope_category`, `category`, `attributes`, and optional
 `category_profile`. Mark events can optionally carry `category` and
-`category_profile`, though the current public mark helpers emit generic marks
-without those optional category fields.
+`category_profile`. Generic mark helpers also accept an optional `data_schema`
+and log `severity`. Relay stores typed severity in sanitizer-visible metadata
+under `nemo_relay.log.severity`; that Relay convention does not add a field to
+the ATOF `0.1` envelope.
 
 Runtime helpers assign timestamps automatically by default. Manual lifecycle
 helpers for scopes, marks, tool calls, and LLM calls also accept an explicit
@@ -325,7 +327,12 @@ Exporters handle the mark as follows:
 - ATIF omits marks because it models trajectory steps rather than independent
   point-in-time events.
 - The `full` and `openinference` OpenTelemetry projections use their fixed
-  native mark handling. The `gen_ai` projection omits marks.
+  native handling for non-metric marks. The `gen_ai` projection omits marks.
+- The independent OTLP log pipeline exports sanitized non-metric marks. The
+  independent metric pipeline exclusively consumes valid marks with the
+  Relay-owned `nemo.relay.metric_measurements` schema. Invalid reserved metric
+  marks reach no OTLP signal and produce an operational diagnostic. This
+  routing does not define an ATOF-wide metric semantic.
 
 ## How Events Are Produced
 
@@ -336,5 +343,5 @@ the runtime record important state transitions even when there is no full nested
 callback to wrap. Subscriber delivery is downstream from event construction and
 does not block the emitting call on native bindings.
 
-ATOF export writes the raw ATOF event stream. ATIF and the typed
-OpenTelemetry formats are downstream projections over the same events.
+ATOF export writes the raw ATOF event stream. ATIF and OpenTelemetry traces,
+logs, and metrics are downstream projections over the same events.
@@ -240,14 +240,23 @@ The [Agent Trajectory Observability Format (ATOF) JSONL exporter](/configure-plu
 writes the canonical event stream to a native filesystem path as one raw ATOF
 event per line.
 
-### OpenTelemetry Subscriber
-
-The OpenTelemetry subscriber maps runtime events into OTLP traces for tracing
-backends. Each subscriber uses one `full`, `gen_ai`, or `openinference`
-projection. Select `openinference` for model-centric OpenInference semantics;
-there is no separate OpenInference subscriber.
-
-Detailed setup, configuration, and API behavior for this subscriber belongs in
+### OpenTelemetry Subscribers
+
+The trace subscriber maps runtime events into OTLP traces. Each trace
+subscriber uses one `full`, `gen_ai`, or `openinference` projection. Select
+`openinference` for model-centric OpenInference semantics; there is no separate
+OpenInference subscriber.
+
+Independent log and metric subscribers consume the same sanitized event
+stream. The log subscriber exports non-metric marks with typed severity and
+scope correlation. Valid marks that use the Relay-owned
+`nemo.relay.metric_measurements` schema route exclusively to the metric
+subscriber and are not projected as traces or logs. Invalid reserved metric
+marks reach no OTLP signal and produce an operational diagnostic. Scope events
+maintain log lineage but do not become logs or metrics. Operational stderr and
+file records are not input to either telemetry subscriber.
+
+Detailed setup, configuration, and API behavior for these subscribers belongs in
 [Observability](/configure-plugins/observability/about).
 For configuration-driven setup, use the built-in
 [`observability` plugin](/configure-plugins/observability/configuration)

@@ -34,12 +34,13 @@ continuations. This is a source-breaking change for typed native Rust plugins;
 rebuild them with `nemo-relay-plugin` 0.8.0 and declare
 `compat.relay = ">=0.8.0,<1.0"`.
 
-Native ABI v4 adds completion-scoped codec operations and pull-based LLM
-stream continuations. Relay also negotiates the frozen ABI v3 and v2 table
-layouts for Relay 0.8-built plugins that target them, and native subscribers
-remain synchronous. Follow the [native Rust migration
+Native ABI v4 is current for Relay 0.8. It includes typed asynchronous
+middleware operations such as completion-scoped codecs and pull-based LLM
+stream continuations, then appends `emit_mark_v2` for typed data-schema and log
+severity options. Relay still accepts existing raw ABI v2 and v3 binaries, and
+native subscribers remain synchronous. Follow the [native Rust migration
 guide](/reference/migration-guides#upgrade-native-rust-plugins-to-nemo-relay-08)
-for callback and executor changes.
+for callback, executor, and mark-option changes.
 
 Managed and manual tool execution now uses the canonical
 `ToolExecutionResult` shape across Rust, Python, Node.js, Go, the public C
@@ -53,8 +54,9 @@ The following updates expand native Rust plugin capabilities:
 
 - Typed native Rust middleware can await Tokio timers, I/O, codecs, unary
   continuations, and downstream LLM streams on an SDK-owned executor.
-- Native ABI v4 provides completion-scoped codec operations and pull-based LLM
-  stream continuations.
+- Native ABI v4 provides completion-scoped codec operations, pull-based LLM
+  stream continuations, and appended `emit_mark_v2` support for typed data
+  schemas and log severity.
 - Tool callbacks and execution-intercept continuations share one canonical
   result contract with an optional opaque annotation.
 - ATOF, ATIF, and the `full` and `openinference` OpenTelemetry projections

@@ -80,6 +80,13 @@ Workers return declarative registrations. Relay owns registry mutation,
 namespacing, rollback, and deregistration. Relay DTOs use `JsonEnvelope`
 values, while protobuf handles control flow.
 
+Worker SDKs also expose `PluginRuntime::runtime_diagnostics()` as a read-only,
+host-level snapshot. Entries contain `code`, `message`, and `count`; repeated
+codes are aggregated, ordered by code, and limited to 32 entries. The host does
+not attribute entries to individual plugins. An older Relay host returns
+`UNIMPLEMENTED`; current SDKs surface that as an unsupported-runtime-diagnostics
+error without a handshake capability flag.
+
 Run `nemo-relay plugins add` and `nemo-relay plugins validate` to evaluate the
 configured host policy and artifact trust evidence. Refer to [Configure
 Discoverable Plugins](/configure-plugins/discoverable-plugins) for the

@@ -148,6 +148,7 @@ Relay implements these RPCs for worker callbacks:
 | --- | --- |
 | Marks and scopes | `EmitMark`, `PushScope`, `PopScope` |
 | Isolated scope stacks | `CreateScopeStack`, `DropScopeStack` |
+| Runtime diagnostics | `GetRuntimeDiagnostics` |
 | Execution continuations | `ToolNext`, `LlmNext`, `LlmStreamNext` |
 | Codec capabilities | `DecodeLlmCodecRequest`, `EncodeLlmCodecRequest`, `DecodeLlmCodecResponse` |
 
@@ -178,6 +179,23 @@ baseline. The package and RPC method names remain v1, but workers must
 regenerate their protobuf bindings and rebuild. Future incompatible worker
 protobuf or payload semantic changes must increment `worker_protocol`.
 
+`EmitMarkRequest` keeps its original `name`, `data`, and `metadata` fields and
+additively supports `data_schema` (a `nemo.relay.DataSchema@1` JSON envelope)
+and canonical `severity` (`trace`, `debug`, `info`, `warn`, or `error`). An
+empty severity and an absent schema preserve legacy behavior. Worker SDKs keep
+the original `emit_mark(name, data, metadata)` call and expose optional fields.
+The Rust and Python SDKs also provide `emit_metric` for validated
+`nemo.relay.metric_measurements` marks.
+
+`GetRuntimeDiagnosticsRequest` carries the activation ID and token. Its closed
+response contains repeated `RuntimeDiagnostic { code, message, count }`
+entries. Relay returns the active host-level snapshot only: it aggregates by
+code, retains the latest message, saturates counts, sorts by code, and limits
+the response to 32 entries. There is no per-plugin attribution. The RPC was
+added without changing the `grpc-v1` handshake; an older host naturally returns
+gRPC `UNIMPLEMENTED`, which SDKs translate into an unsupported-runtime-
+diagnostics error.
+
 A worker may call an execution continuation repeatedly or concurrently while
 the corresponding `Invoke` or `InvokeStream` callback is active. Each call gets
 an isolated scope-stack branch containing the scopes visible at invocation.

@@ -165,6 +165,58 @@ registers a tool request intercept that updates the request JSON and emits a
 mark through `PluginContext.runtime`. The `main` entrypoint blocks until Relay
 requests shutdown.
 
+## Read Host Runtime Diagnostics
+
+Use the runtime handle inside an asynchronous callback to inspect the active
+host-level snapshot:
+
+```python
+diagnostics = await ctx.runtime.runtime_diagnostics()
+diagnostic = diagnostics.get("otel.metric_mark_invalid")
+if diagnostic is not None:
+    print(f"{diagnostic.message} occurred {diagnostic.count} times")
+```
+
+`RuntimeDiagnostics.entries` is an immutable tuple ordered by code. The host
+aggregates repeated codes, retains the latest message, saturates counts, and
+returns at most 32 entries. Entries do not identify the emitting plugin. If a
+worker connects to an older Relay host, the SDK translates gRPC `UNIMPLEMENTED`
+into an unsupported-runtime-diagnostics error.
+
+## Emit Typed Marks and Metrics
+
+Worker plugins can assign a schema and OTLP log severity to a mark without
+changing the existing positional call shape. Use `emit_metric` rather than
+constructing Relay's reserved metric schema directly:
+
+```python
+from nemo_relay_plugin import DataSchema, LogSeverity
+
+await ctx.runtime.emit_mark(
+    "examples.python_grpc_worker.policy_decision",
+    {"allowed": True},
+    data_schema=DataSchema("examples.policy.decision", "1"),
+    severity=LogSeverity.INFO,
+)
+
+await ctx.runtime.emit_metric(
+    "examples.python_grpc_worker.requests",
+    [
+        {
+            "name": "example.requests",
+            "kind": "counter",
+            "value_type": "u64",
+            "value": 1,
+            "attributes": {"plugin": "python-worker"},
+        }
+    ],
+)
+```
+
+Relay validates the schema and severity at the host boundary. It validates the
+metric mark after sanitization, so an invalid measurement group is rejected
+atomically and is not exported as a log.
+
 ## Create the Manifest
 
 Create `relay-plugin.toml` with the following content:

@@ -69,6 +69,29 @@ identity from `WorkerPlugin::plugin_id()`; custom launchers can use
 credentials and runs until Relay requests shutdown. Do not start the worker
 directly for normal operation.
 
+## Read Host Runtime Diagnostics
+
+Use the runtime handle inside an asynchronous callback to read the active
+host-level diagnostic snapshot:
+
+```rust
+let diagnostics = context
+    .runtime()
+    .expect("Relay provides a runtime handle")
+    .runtime_diagnostics()
+    .await?;
+
+if let Some(diagnostic) = diagnostics.get("otel.metric_mark_invalid") {
+    eprintln!("{} occurred {} times", diagnostic.message, diagnostic.count);
+}
+```
+
+`entries()` is stable in code order. The snapshot aggregates by code, retains
+the latest message, saturates counts, and has at most 32 entries. It is a
+host-level view and has no per-plugin attribution. An older Relay host returns
+`UNIMPLEMENTED`; the SDK reports that as an unsupported-runtime-diagnostics
+error.
+
 ## Package the Worker
 
 Create `relay-plugin.toml` with the following content. The artifact digest must