Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel
```

5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans.
Preserve parent-child IDs and set a stable `session.id` for related traces.
Preserve parent-child IDs and set a stable `session.id` for related traces. Always set
`gen_ai.agent.name` (or `llm.agent.name` / `agent.name` for instrumentation that emits those
conventions) on every span to a stable name for the agent. This ensures the spans can be analyzed
by NeMo Insights in the future.
6. Run one representative interaction, then verify it through the spans query below.

Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool,
Expand Down Expand Up @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session
```

Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs,
status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare
them in a leaderboard, hand off to `nemo-experiments-upload`.
status/errors, and any evaluator results. Also query with `filter[agent_name]=<agent-name>` and
confirm the returned `agent_name` matches the stable value emitted by the agent. If the goal is to
create named evaluation runs and compare them in a leaderboard, hand off to
`nemo-experiments-upload`.
8 changes: 8 additions & 0 deletions plugins/nemo-insights/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ explicit command-line flags, then profile values (for `agent`, `agent_spec`,
and `workspace`) or `NMP_BASE_URL` (for the base URL), then the built-in
defaults. `--base-url` takes precedence over `NMP_BASE_URL`.

### Telemetry requirement

The analyst scopes Intake span queries to the configured `agent`. The normalized
`agent_name` on each span must therefore match `agent` in `optimizer.yaml` or
`--agent`. For OTLP, always set `gen_ai.agent.name` on every span; Intake also
normalizes `llm.agent.name` and `agent.name` from instrumentation that emits
those conventions. ATIF maps its required `agent.name` automatically.

### Where insights are written

Insights always go to the platform, through the Insights plugin API. There is no
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading