Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,10 @@ tool, token, cost, input/output, and session fields needed for useful Intake tel
```

5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans.
Preserve parent-child IDs and set a stable `session.id` for related traces.
Preserve parent-child IDs and set a stable `session.id` for related traces. Always set
`gen_ai.agent.name` on every span to a stable name for the agent. Do this even when NeMo Insights
is not currently in use so historical spans remain queryable if it is enabled later. Intake also
normalizes `llm.agent.name` and `agent.name` from instrumentation that emits those conventions.
6. Run one representative interaction, then verify it through the spans query below.

Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool,
Expand Down Expand Up @@ -146,5 +149,7 @@ curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session
```

Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs,
status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare
them in a leaderboard, hand off to `nemo-experiments-upload`.
status/errors, and any evaluator results. Also query with `filter[agent_name]=<agent-name>` and
confirm the returned `agent_name` matches the stable value emitted by the agent. NeMo Insights must
use that same value for its `agent` or `--agent` setting. If the goal is to create named evaluation
runs and compare them in a leaderboard, hand off to `nemo-experiments-upload`.
10 changes: 10 additions & 0 deletions plugins/nemo-insights/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,16 @@ explicit command-line flags, then profile values (for `agent`, `agent_spec`,
and `workspace`) or `NMP_BASE_URL` (for the base URL), then the built-in
defaults. `--base-url` takes precedence over `NMP_BASE_URL`.

### Telemetry requirement

The analyst scopes Intake span queries to the configured `agent`; it does not
fall back to another attribute. The normalized `agent_name` on each span must
therefore match `agent` in `optimizer.yaml` or `--agent`. For OTLP, always set
`gen_ai.agent.name` on every span; Intake also normalizes `llm.agent.name` and
`agent.name` from instrumentation that emits those conventions. ATIF maps its
required `agent.name` automatically. Spans without an agent name can ingest
successfully, but the analyst cannot retrieve or group them later.

### Where insights are written

Insights always go to the platform, through the Insights plugin API. There is no
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.