From d579c7fa8eec152bb197eb799dc4bd7033ff440f Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Thu, 6 Aug 2026 14:59:02 -0600 Subject: [PATCH 1/3] docs: document analyst agent name requirement Signed-off-by: Brian Newsom --- .../src/nemo_platform_ext/skills/nemo-intake/SKILL.md | 11 ++++++++--- plugins/nemo-insights/README.md | 9 +++++++++ .../src/nemo_platform/skills/nemo-intake/SKILL.md | 11 ++++++++--- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md index f7a6183a9c..f6e83ab825 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. + Preserve parent-child IDs and set a stable `session.id` for related traces. If NeMo Insights will + analyze the telemetry, also set `gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name` + to the analyst's configured agent. Without a normalized `agent_name`, the spans can ingest but + are excluded from the analyst's scoped queries. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare -them in a leaderboard, hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. For NeMo Insights, also query with +`filter[agent_name]=` and confirm the returned `agent_name` matches the analyst's `agent` +or `--agent` value. If the goal is to create named evaluation runs and compare them in a leaderboard, +hand off to `nemo-experiments-upload`. diff --git a/plugins/nemo-insights/README.md b/plugins/nemo-insights/README.md index 531fe58287..f5a9905be4 100644 --- a/plugins/nemo-insights/README.md +++ b/plugins/nemo-insights/README.md @@ -43,6 +43,15 @@ explicit command-line flags, then profile values (for `agent`, `agent_spec`, and `workspace`) or `NMP_BASE_URL` (for the base URL), then the built-in defaults. `--base-url` takes precedence over `NMP_BASE_URL`. +### Telemetry requirement + +The analyst scopes Intake span queries to the configured `agent`; it does not +fall back to another attribute. The normalized `agent_name` on each span must +therefore match `agent` in `optimizer.yaml` or `--agent`. For OTLP, set +`gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name`; ATIF maps +its required `agent.name` automatically. Spans without an agent name can +ingest successfully, but the analyst cannot retrieve or group them. + ### Where insights are written Insights always go to the platform, through the Insights plugin API. There is no diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md index f7a6183a9c..f6e83ab825 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. + Preserve parent-child IDs and set a stable `session.id` for related traces. If NeMo Insights will + analyze the telemetry, also set `gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name` + to the analyst's configured agent. Without a normalized `agent_name`, the spans can ingest but + are excluded from the analyst's scoped queries. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare -them in a leaderboard, hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. For NeMo Insights, also query with +`filter[agent_name]=` and confirm the returned `agent_name` matches the analyst's `agent` +or `--agent` value. If the goal is to create named evaluation runs and compare them in a leaderboard, +hand off to `nemo-experiments-upload`. From 2c7e810f619abfad33135ffd766190da3bd59edb Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Thu, 6 Aug 2026 15:21:56 -0600 Subject: [PATCH 2/3] docs: require agent names for all OTLP spans Signed-off-by: Brian Newsom --- .../skills/nemo-intake/SKILL.md | 16 ++++++++-------- plugins/nemo-insights/README.md | 9 +++++---- .../nemo_platform/skills/nemo-intake/SKILL.md | 16 ++++++++-------- 3 files changed, 21 insertions(+), 20 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md index f6e83ab825..d7062519ff 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md @@ -110,10 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. If NeMo Insights will - analyze the telemetry, also set `gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name` - to the analyst's configured agent. Without a normalized `agent_name`, the spans can ingest but - are excluded from the analyst's scoped queries. + Preserve parent-child IDs and set a stable `session.id` for related traces. Always set + `gen_ai.agent.name` on every span to a stable name for the agent. Do this even when NeMo Insights + is not currently in use so historical spans remain queryable if it is enabled later. Intake also + normalizes `llm.agent.name` and `agent.name` from instrumentation that emits those conventions. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -149,7 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. For NeMo Insights, also query with -`filter[agent_name]=` and confirm the returned `agent_name` matches the analyst's `agent` -or `--agent` value. If the goal is to create named evaluation runs and compare them in a leaderboard, -hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. Also query with `filter[agent_name]=` and +confirm the returned `agent_name` matches the stable value emitted by the agent. NeMo Insights must +use that same value for its `agent` or `--agent` setting. If the goal is to create named evaluation +runs and compare them in a leaderboard, hand off to `nemo-experiments-upload`. diff --git a/plugins/nemo-insights/README.md b/plugins/nemo-insights/README.md index f5a9905be4..58ff7df833 100644 --- a/plugins/nemo-insights/README.md +++ b/plugins/nemo-insights/README.md @@ -47,10 +47,11 @@ defaults. `--base-url` takes precedence over `NMP_BASE_URL`. The analyst scopes Intake span queries to the configured `agent`; it does not fall back to another attribute. The normalized `agent_name` on each span must -therefore match `agent` in `optimizer.yaml` or `--agent`. For OTLP, set -`gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name`; ATIF maps -its required `agent.name` automatically. Spans without an agent name can -ingest successfully, but the analyst cannot retrieve or group them. +therefore match `agent` in `optimizer.yaml` or `--agent`. For OTLP, always set +`gen_ai.agent.name` on every span; Intake also normalizes `llm.agent.name` and +`agent.name` from instrumentation that emits those conventions. ATIF maps its +required `agent.name` automatically. Spans without an agent name can ingest +successfully, but the analyst cannot retrieve or group them later. ### Where insights are written diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md index f6e83ab825..d7062519ff 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md @@ -110,10 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. If NeMo Insights will - analyze the telemetry, also set `gen_ai.agent.name` (preferred), `llm.agent.name`, or `agent.name` - to the analyst's configured agent. Without a normalized `agent_name`, the spans can ingest but - are excluded from the analyst's scoped queries. + Preserve parent-child IDs and set a stable `session.id` for related traces. Always set + `gen_ai.agent.name` on every span to a stable name for the agent. Do this even when NeMo Insights + is not currently in use so historical spans remain queryable if it is enabled later. Intake also + normalizes `llm.agent.name` and `agent.name` from instrumentation that emits those conventions. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -149,7 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. For NeMo Insights, also query with -`filter[agent_name]=` and confirm the returned `agent_name` matches the analyst's `agent` -or `--agent` value. If the goal is to create named evaluation runs and compare them in a leaderboard, -hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. Also query with `filter[agent_name]=` and +confirm the returned `agent_name` matches the stable value emitted by the agent. NeMo Insights must +use that same value for its `agent` or `--agent` setting. If the goal is to create named evaluation +runs and compare them in a leaderboard, hand off to `nemo-experiments-upload`. From 3efa02c767e320f18a3257410496a9e85a9125a8 Mon Sep 17 00:00:00 2001 From: Brian Newsom Date: Thu, 6 Aug 2026 15:30:44 -0600 Subject: [PATCH 3/3] docs: refine agent name guidance Signed-off-by: Brian Newsom --- .../nemo_platform_ext/skills/nemo-intake/SKILL.md | 12 ++++++------ plugins/nemo-insights/README.md | 12 +++++------- .../src/nemo_platform/skills/nemo-intake/SKILL.md | 12 ++++++------ 3 files changed, 17 insertions(+), 19 deletions(-) diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md index d7062519ff..f3d6da98ce 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md @@ -111,9 +111,9 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. Preserve parent-child IDs and set a stable `session.id` for related traces. Always set - `gen_ai.agent.name` on every span to a stable name for the agent. Do this even when NeMo Insights - is not currently in use so historical spans remain queryable if it is enabled later. Intake also - normalizes `llm.agent.name` and `agent.name` from instrumentation that emits those conventions. + `gen_ai.agent.name` (or `llm.agent.name` / `agent.name` for instrumentation that emits those + conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed + by NeMo Insights in the future. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -150,6 +150,6 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, status/errors, and any evaluator results. Also query with `filter[agent_name]=` and -confirm the returned `agent_name` matches the stable value emitted by the agent. NeMo Insights must -use that same value for its `agent` or `--agent` setting. If the goal is to create named evaluation -runs and compare them in a leaderboard, hand off to `nemo-experiments-upload`. +confirm the returned `agent_name` matches the stable value emitted by the agent. If the goal is to +create named evaluation runs and compare them in a leaderboard, hand off to +`nemo-experiments-upload`. diff --git a/plugins/nemo-insights/README.md b/plugins/nemo-insights/README.md index 58ff7df833..799a1b3d42 100644 --- a/plugins/nemo-insights/README.md +++ b/plugins/nemo-insights/README.md @@ -45,13 +45,11 @@ defaults. `--base-url` takes precedence over `NMP_BASE_URL`. ### Telemetry requirement -The analyst scopes Intake span queries to the configured `agent`; it does not -fall back to another attribute. The normalized `agent_name` on each span must -therefore match `agent` in `optimizer.yaml` or `--agent`. For OTLP, always set -`gen_ai.agent.name` on every span; Intake also normalizes `llm.agent.name` and -`agent.name` from instrumentation that emits those conventions. ATIF maps its -required `agent.name` automatically. Spans without an agent name can ingest -successfully, but the analyst cannot retrieve or group them later. +The analyst scopes Intake span queries to the configured `agent`. The normalized +`agent_name` on each span must therefore match `agent` in `optimizer.yaml` or +`--agent`. For OTLP, always set `gen_ai.agent.name` on every span; Intake also +normalizes `llm.agent.name` and `agent.name` from instrumentation that emits +those conventions. ATIF maps its required `agent.name` automatically. ### Where insights are written diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md index d7062519ff..f3d6da98ce 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md @@ -111,9 +111,9 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. Preserve parent-child IDs and set a stable `session.id` for related traces. Always set - `gen_ai.agent.name` on every span to a stable name for the agent. Do this even when NeMo Insights - is not currently in use so historical spans remain queryable if it is enabled later. Intake also - normalizes `llm.agent.name` and `agent.name` from instrumentation that emits those conventions. + `gen_ai.agent.name` (or `llm.agent.name` / `agent.name` for instrumentation that emits those + conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed + by NeMo Insights in the future. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -150,6 +150,6 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, status/errors, and any evaluator results. Also query with `filter[agent_name]=` and -confirm the returned `agent_name` matches the stable value emitted by the agent. NeMo Insights must -use that same value for its `agent` or `--agent` setting. If the goal is to create named evaluation -runs and compare them in a leaderboard, hand off to `nemo-experiments-upload`. +confirm the returned `agent_name` matches the stable value emitted by the agent. If the goal is to +create named evaluation runs and compare them in a leaderboard, hand off to +`nemo-experiments-upload`.