diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md index f7a6183a9c..f3d6da98ce 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. + Preserve parent-child IDs and set a stable `session.id` for related traces. Always set + `gen_ai.agent.name` (or `llm.agent.name` / `agent.name` for instrumentation that emits those + conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed + by NeMo Insights in the future. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare -them in a leaderboard, hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. Also query with `filter[agent_name]=` and +confirm the returned `agent_name` matches the stable value emitted by the agent. If the goal is to +create named evaluation runs and compare them in a leaderboard, hand off to +`nemo-experiments-upload`. diff --git a/plugins/nemo-insights/README.md b/plugins/nemo-insights/README.md index 531fe58287..799a1b3d42 100644 --- a/plugins/nemo-insights/README.md +++ b/plugins/nemo-insights/README.md @@ -43,6 +43,14 @@ explicit command-line flags, then profile values (for `agent`, `agent_spec`, and `workspace`) or `NMP_BASE_URL` (for the base URL), then the built-in defaults. `--base-url` takes precedence over `NMP_BASE_URL`. +### Telemetry requirement + +The analyst scopes Intake span queries to the configured `agent`. The normalized +`agent_name` on each span must therefore match `agent` in `optimizer.yaml` or +`--agent`. For OTLP, always set `gen_ai.agent.name` on every span; Intake also +normalizes `llm.agent.name` and `agent.name` from instrumentation that emits +those conventions. ATIF maps its required `agent.name` automatically. + ### Where insights are written Insights always go to the platform, through the Insights plugin API. There is no diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md index f7a6183a9c..f3d6da98ce 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel ``` 5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. - Preserve parent-child IDs and set a stable `session.id` for related traces. + Preserve parent-child IDs and set a stable `session.id` for related traces. Always set + `gen_ai.agent.name` (or `llm.agent.name` / `agent.name` for instrumentation that emits those + conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed + by NeMo Insights in the future. 6. Run one representative interaction, then verify it through the spans query below. Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session ``` Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, -status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare -them in a leaderboard, hand off to `nemo-experiments-upload`. +status/errors, and any evaluator results. Also query with `filter[agent_name]=` and +confirm the returned `agent_name` matches the stable value emitted by the agent. If the goal is to +create named evaluation runs and compare them in a leaderboard, hand off to +`nemo-experiments-upload`.