docs: document Event metadata injection

docs/about-nemo-relay/concepts/middleware.mdx

@@ -9,12 +9,13 @@ import { MermaidStyles } from "@/components/MermaidStyles";
 SPDX-License-Identifier: Apache-2.0 */}
 
 This page explains the runtime behavior that runs around managed tool and LLM
-calls and sanitizes emitted mark and scope events.
+calls, injects metadata into events, and sanitizes emitted mark and scope events.
 
 ## What Middleware Is
 
-Middleware controls or transforms tool and LLM execution and sanitizes emitted
-events. NeMo Relay applies each middleware type at a specific lifecycle point.
+Middleware can control or transform tool and LLM execution. It can also enrich
+or sanitize emitted `Event` objects. NeMo Relay applies each middleware type at
+a specific lifecycle point.
 
 Middleware is organized by lifecycle meaning rather than as one undifferentiated
 hook system.
@@ -27,8 +28,9 @@ registrations accept callbacks that return a value or an awaitable when invoked
 through an asynchronous Relay API or queued event publication. Worker and
 native-plugin middleware can also complete asynchronously. Within each
 middleware chain, Relay awaits entries sequentially in priority order so later
-callbacks observe earlier middleware output. Payload and event sanitizer chains
-run on the queued publication path and do not delay managed execution.
+callbacks observe earlier middleware output. Event payload transformations,
+event metadata injectors, and event sanitizer chains run on the queued
+publication path and do not delay managed execution.
 
 The experimental raw C FFI and Go binding retain synchronous middleware
 callbacks. Relay invokes each callback on a native thread and waits for it to
@@ -49,10 +51,11 @@ sanitization and publication rather than awaiting it. Manual lifecycle APIs
 synchronous and create or close their handle immediately.
 
 <Warning>
-Event sanitizers, conditional-execution guardrails, request intercepts,
-execution intercepts, and subscribers are not re-entrant. These callbacks must
-not invoke another NeMo Relay API that runs middleware, flushes subscriber
-delivery, waits on an exporter, or clears plugins. Scope APIs remain supported:
+Event metadata injectors, event sanitizers, conditional-execution guardrails,
+request intercepts, execution intercepts, and subscribers are not re-entrant.
+These callbacks must not invoke another NeMo Relay API that runs middleware,
+flushes subscriber delivery, waits on an exporter, or clears plugins. Scope
+APIs remain supported:
 callbacks may create, push, or pop scopes at any nesting level and may replace
 the active scope stack with an arbitrary stack. Emitting a new event is the
 only supported operation that can enqueue additional callback work; Relay
@@ -83,8 +86,8 @@ separately from this application-visible result.
 
 ## Registration Levels
 
-Middleware and subscribers can be registered at different levels depending on their
-lifetime and visibility.
+Middleware and subscribers can be registered at different levels depending on
+their lifetime and visibility.
 
 ### Global Registrations
 
@@ -107,10 +110,11 @@ everything in application code.
 
 ## Middleware Families
 
-NeMo Relay has two major middleware families with three distinct purposes:
+NeMo Relay provides four middleware purposes:
 
 - **Intercepts** change the real request or callback execution path.
 - **Conditional-execution guardrails** decide whether the real work runs.
+- **Event metadata injectors** add flat metadata to emitted events.
 - **Sanitize guardrails** change emitted observability without changing the real
   request or result.
 
@@ -122,6 +126,8 @@ Choose the middleware type that matches the behavior you need:
   rejected.
 - Use a **request intercept** when the real request must change before the call.
 - Use an **execution intercept** when code must run before or after the callback.
+- Use an **Event metadata injector** when every emitted `Event` should receive
+  additional flat metadata before sanitization.
 - Use a **sanitize guardrail** when only subscribers and exporters should see
   rewritten data.
 - Use a **mark or scope event sanitizer** when the sensitive fields are in
@@ -144,6 +150,11 @@ execution intercept cannot undo work from a `next` continuation that it already
 invoked. This fail-closed contract is the same whether the callback completes
 directly or asynchronously.
 
+Event metadata injectors fail open. If an injector fails, panics, or returns
+invalid attributes, Relay logs the failure, ignores only that injector's output
+for the current `Event`, and continues with later injectors using the `Event` as
+it stood before the failed injector.
+
 ## Intercepts
 
 Intercepts are middleware that change the real request or execution path.
@@ -177,6 +188,33 @@ LLM streaming has a stream execution path for wrappers that need to run around
 chunk delivery and finalization rather than only around a single response
 object.
 
+## Event Metadata Injection
+
+Event metadata injectors add values to the existing Event `metadata` object.
+They run on every delivered Event and receive the current Event as immutable
+context, so a callback can inspect the Event kind and name before deciding what
+to return.
+
+Event metadata keys contain one or more non-empty segments made of ASCII
+letters, digits, underscores, or hyphens. Segments may be separated by single
+dots, so keys such as `region.zone.name` are valid. Leading dots, trailing dots,
+consecutive dots, whitespace, and unsupported punctuation are invalid. Relay
+stores accepted keys literally; dots do not create nested objects. Values may
+be strings, numbers, booleans, or homogeneous lists of those primitive types.
+Nested objects, `null`, and mixed-type lists are rejected as one atomic callback
+result.
+
+Injection is insert-only. Existing Event metadata is never overwritten.
+Injectors run in ascending numeric priority and then by registration name, so
+the first injector to insert a missing key wins. A later injector that returns
+the same key cannot replace it.
+
+Register injectors globally, on an owning scope, or through a plugin
+registration context. Injectors execute on the queued serial publication path,
+before Event sanitizers, so they do not delay the managed tool or LLM callback.
+Sanitizers retain the final decision to preserve, rewrite, or remove injected
+metadata before subscriber and exporter delivery.
+
 ## Guardrails
 
 Guardrails are middleware that block execution or sanitize observability payloads.
@@ -229,16 +267,17 @@ arguments passed to the callback or the real value returned to the caller.
 ## Queued Event Publication
 
 Scope operations, marks, and manual or managed tool/LLM lifecycle calls do not
-await observability sanitizers. At emission time Relay snapshots the event-only
-payload, visible sanitizer chains, and subscribers, then places the work on a
-serial dispatcher. The dispatcher awaits the specialized tool or LLM payload
-sanitizers, then the event sanitizers, and publishes the event later in FIFO
-order.
+await observability middleware. At emission time Relay snapshots the event-only
+payload, visible Event metadata injectors, sanitizer chains, and subscribers,
+then places the work on a serial dispatcher. The dispatcher completes the
+internal tool or LLM Event payload transformation, applies Event metadata
+injectors, runs Event sanitizers, and publishes the Event later in FIFO order.
 
 Subscriber and exporter delivery is therefore delayed, while start/end/mark
 order is preserved. Closing a scope or deregistering middleware after emission
-does not affect queued snapshots. Sanitizer failures fail closed: Relay records
-the callback failure and withholds the governed observability payload.
+does not affect queued snapshots. Injector failures fail open and omit that
+injector's output. Sanitizer failures fail closed: Relay records the callback
+failure and withholds the governed observability payload.
 
 ## Managed Execution Order
 
@@ -247,6 +286,8 @@ execution, and response phases.
 
 <MermaidStyles />
 
+The following diagram shows the managed execution and serial publication paths.
+
 ```mermaid
 flowchart LR
     subgraph RequestPhase[Request Phase]
@@ -272,15 +313,20 @@ flowchart LR
     Callback --> QueueEnd
 
     subgraph PublicationPath[Serial Publication Path]
-        RequestSanitizers[Request Sanitizers]
-        StartEvent[Scope-Start Sanitizers and Deliver Start]
-        ResponseSanitizers[Response Sanitizers]
-        EndEvent[Scope-End Sanitizers and Deliver End]
-        RequestSanitizers --> StartEvent --> ResponseSanitizers --> EndEvent
+        RequestTransform[Request Event Payload Transform]
+        StartInjection[Event Metadata Injectors]
+        StartSanitizers[Scope-Start Event Sanitizers]
+        DeliverStart[Deliver Start Event]
+        ResponseTransform[Response Event Payload Transform]
+        EndInjection[Event Metadata Injectors]
+        EndSanitizers[Scope-End Event Sanitizers]
+        DeliverEnd[Deliver End Event]
+        RequestTransform --> StartInjection --> StartSanitizers --> DeliverStart
+        DeliverStart --> ResponseTransform --> EndInjection --> EndSanitizers --> DeliverEnd
     end
 
-    QueueStart -.-> RequestSanitizers
-    QueueEnd -.-> ResponseSanitizers
+    QueueStart -.-> RequestTransform
+    QueueEnd -.-> ResponseTransform
 ```
 
 The phases run as follows:
@@ -291,9 +337,10 @@ The phases run as follows:
 2. **Execution phase:** Execution intercepts wrap or replace the real callback.
 3. **Response phase:** Relay snapshots and enqueues the end event's
    observability copy, then returns the real result.
-4. **Publication path:** The serial dispatcher runs request or response
-   sanitizers, then the matching scope-event sanitizers, and finally delivers
-   the event to subscribers and exporters.
+4. **Publication path:** The serial dispatcher completes the internal request or
+   response Event payload transformation, applies Event metadata injectors, runs
+   the matching scope-Event sanitizers, and finally delivers the Event to
+   subscribers and exporters.
 
 The start event is submitted before execution begins, but its sanitizers run
 later on the publication path and may overlap application execution. The serial
@@ -303,6 +350,7 @@ for queued sanitization and delivery when a caller needs that barrier.
 This ordering preserves the distinction between the families:
 
 - Use an intercept to change real execution.
+- Use an event metadata injector to add observability context.
 - Use a sanitize guardrail to change only emitted observability.
 
 ### Rejection Path