diff --git a/AGENTS.md b/AGENTS.md index 018207a705..7832f70f8e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,6 +31,8 @@ User-facing skills in `packages/nemo_platform_ext/src/nemo_platform_ext/skills/` - `nemo-spec`: writes an agent spec from explore output. - `nemo-build-agent`: scaffolds NAT workflow YAML from the spec and deploys. - `nemo-try-agent`: test a deployed agent or chat with a model. +- `nemo-intake`: instrument agents, choose an ingest format, upload/query telemetry, and attach evaluator results. +- `nemo-experiments-upload`: publish named evaluation runs and scores to the Experiments leaderboard. - `nemo-status`: read-only health dashboard. - `nemo-teardown`: guided shutdown with confirmation. diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md index 4b4faa537d..16400ac9e8 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/SKILL.md @@ -1,21 +1,22 @@ --- name: nemo-experiments-upload -description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up as Experiments. Create an Experiment, create an Evaluation, then log traces and evaluator results via the ATIF (Harbor), chat-completions, or OTLP ingest endpoint — and view the rollups in Studio. Use when a user wants to upload, log, ingest, publish, or send evaluation runs, agent traces, or scores to NeMo Experiments / Intake. +description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up in the Experiments leaderboard. Create an Experiment, create an Evaluation, then log traces and evaluator results via ATIF (Harbor), chat-completions, or OTLP and view the rollups in Studio. Use when a user wants to create named evaluation runs, publish evaluation results, or compare runs in NeMo Experiments. triggers: - - log traces to intake + - log evaluation traces to experiments - upload experiment results - ingest evaluation data - how do I log to experiments - - send traces to nemo intake + - send evaluation traces to nemo experiments - publish evaluation results - log to experiments - upload harbor / atif results - get my eval data into nemo not-for: + - nemo-intake (use for general instrumentation, telemetry ingestion, trace queries, or evaluator results outside an Experiments leaderboard) - nemo-evaluator (use to AUTHOR and RUN evaluations/metrics; this skill UPLOADS results) - nemo-status (use for a read-only platform health dashboard) - nemo-skill-selection (use for dispatch when intent is unclear) -compatibility: nemo-platform >= 0.1.0; needs the intake service running (auth, entities, intake) and ClickHouse for rollups/results; talks HTTP to /apis/intake/v2 (curl only, no Docker); Experiments viewing in Studio is behind the VITE_FF_EXPERIMENT feature flag. +compatibility: nemo-platform >= 0.1.0; needs a reachable local or remote intake service (with auth and entities) backed by ClickHouse for rollups/results; talks HTTP to /apis/intake/v2 (curl only, no Docker); Experiments viewing in Studio is behind the VITE_FF_EXPERIMENT feature flag. maturity: beta license: Apache-2.0 user-invocable: true @@ -26,20 +27,45 @@ allowed-tools: [Bash, Read, Write] Get evaluation runs into the platform end-to-end: **create an Experiment → create an Evaluation → log traces + scores to an ingest endpoint → see the rollups.** The API calls the parent (the leaderboard) an **Experiment** and each row an **Evaluation**; the whole feature is called **Experiments**. -Everything below uses `${NMP_BASE_URL}` (default `http://localhost:8080`) and a `${WORKSPACE}` (default `default`). All routes are under `/apis/intake/v2/workspaces/${WORKSPACE}`. +Everything below uses `${NMP_BASE_URL}` (default `http://localhost:8080`) and a `${WORKSPACE}` +(default `default`). Point `NMP_BASE_URL` at the local platform or a remote HTTPS origin. Reject +non-loopback `http://` targets, and never send authentication across an HTTP redirect. All routes +are under `/apis/intake/v2/workspaces/${WORKSPACE}`. ## Pre-flight -Confirm intake is up before doing anything. If this fails, the platform isn't running — route to `setup`/`nemo-status` and stop. +Confirm the target platform is reachable before doing anything. If this fails, report the target as +unreachable and stop; route to `setup`/`nemo-status` only for a local platform. ```bash set -euo pipefail : "${NMP_BASE_URL:=http://localhost:8080}" : "${WORKSPACE:=default}" +nmp_authority=${NMP_BASE_URL#*://} +nmp_authority=${nmp_authority%%/*} +case "${nmp_authority}" in + *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;; +esac +case "${NMP_BASE_URL}" in + https://*) ;; + http://*) + case "${nmp_authority}" in + localhost|127.0.0.1) ;; + localhost:*|127.0.0.1:*) + nmp_port=${nmp_authority#*:} + case "${nmp_port}" in + ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;; + esac + ;; + *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;; + esac + ;; + *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;; +esac if curl -sf "${NMP_BASE_URL}/health/ready" >/dev/null; then echo "platform ready" else - echo "NOT READY — platform isn't running; route to setup/nemo-status and stop" >&2 + echo "NOT READY — target platform is unreachable or not ready" >&2 exit 1 fi ``` @@ -97,20 +123,22 @@ curl -sf -X POST \ ### 3. Log traces + evaluator results -Pick the ingest endpoint that matches your producer. **Read `references/ingest-formats.md` for the full schema and a copy-pasteable example for each.** How you attach evaluation identity depends on the endpoint: +Pick the ingest endpoint that matches your producer. **Read `../nemo-intake/references/ingest-formats.md` for the full schema and a copy-pasteable example for each.** How you attach evaluation identity depends on the endpoint: - **ATIF and chat-completions** (JSON body) — add an `evaluation_context = {evaluation_id: "", test_case_id: ""}` object to the payload. -- **OTLP** — there is no body field; set identity as **root-span resource attributes** `nemo.experiment.id` (the Evaluation **name**) and `nemo.test_case.id`. Spans missing these still ingest but won't associate to an Evaluation. +- **OTLP** — there is no body field; set `nemo.experiment.id` (the Evaluation **name**) and + `nemo.test_case.id` (the task ID) as **attributes on the root span**. Spans missing these still + ingest but won't associate to an Evaluation. | Producer | Endpoint | Read | |---|---|---| | **Harbor / agent trajectories** (most common) | `POST .../ingest/atif` | `references/harbor-quickstart.md` | -| A single captured model call | `POST .../ingest/chat-completions` | `references/ingest-formats.md` | -| OpenTelemetry spans | `POST .../ingest/otlp/v1/traces` | `references/ingest-formats.md` | +| A single captured model call | `POST .../ingest/chat-completions` | `../nemo-intake/references/ingest-formats.md` | +| OpenTelemetry spans | `POST .../ingest/otlp/v1/traces` | `../nemo-intake/references/ingest-formats.md` | Evaluator **scores** arrive one of two ways (both covered in the references): - **Automatically** with ATIF — put rewards under `extra.verifier_result.rewards` (one key per criterion). -- **Explicitly** — `POST .../evaluator-results` with `{span_id, session_id, name, data_type, value}` — use `string_value` instead of `value` for `CATEGORICAL`/`TEXT` results (see `references/ingest-formats.md`). +- **Explicitly** — `POST .../evaluator-results` with `{span_id, session_id, name, data_type, value}` — use `string_value` instead of `value` for `CATEGORICAL`/`TEXT` results (see `../nemo-intake/references/ingest-formats.md`). ### 4. Verify the data landed @@ -134,7 +162,7 @@ Open Studio → the **Experiments** area (behind the `VITE_FF_EXPERIMENT` flag) Read these before hand-writing a payload: -- **`references/ingest-formats.md`** — the three ingest endpoints in full: request schemas, how `evaluation_context` and evaluator results attach, and a working example for each (ATIF, chat-completions, OTLP). +- **`../nemo-intake/references/ingest-formats.md`** — the shared Intake request schemas, evaluation context, evaluator results, and examples for ATIF, chat-completions, and OTLP. - **`references/harbor-quickstart.md`** — the Harbor path specifically: mapping a Harbor trial result → an ATIF payload, including verifier rewards → evaluator scores. - **`references/troubleshooting.md`** — every common `400`/`422`/`503` from the ingest and CRUD endpoints, with the fix. @@ -151,7 +179,7 @@ If `run_count` is 0 after ingesting, the traces didn't associate — almost alwa | Symptom | Cause | Recovery | |---|---|---| | `400 "…must be created before it can be logged."` | Ingested before the Evaluation existed, or `evaluation_id` doesn't match | Create the Evaluation (step 2); ensure `evaluation_context.evaluation_id` equals its **name** | -| `422 Unprocessable` on ingest | Unknown/typo'd top-level key (ATIF/chat-completions are `extra="forbid"`) or bad `schema_version` | Check the exact schema in `references/ingest-formats.md`; remove stray keys | +| `422 Unprocessable` on ingest | Unknown/typo'd top-level key (ATIF/chat-completions are `extra="forbid"`) or bad `schema_version` | Check the exact schema in `../nemo-intake/references/ingest-formats.md`; remove stray keys | | Ingest 2xx but `run_count` stays 0 | Evaluation identity missing/wrong — `evaluation_context.evaluation_id` (ATIF/chat-completions) or the `nemo.experiment.id` root-span attribute (OTLP) ≠ the Evaluation's name | Attach the identity for your endpoint; use the Evaluation **name**, not its id | | `503` on GET evaluation / sessions | ClickHouse (telemetry store) not running | Start ClickHouse; rollups and sessions require it | | Scores don't show up | Rewards not under `extra.verifier_result.rewards`, or wrong `data_type` on `/evaluator-results` | See `references/troubleshooting.md` | diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md index 736edc1a66..8cd5170a5c 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/troubleshooting.md @@ -18,7 +18,7 @@ string — read it first. |---|---|---| | `400 "Evaluation '…' must be created before it can be logged."` | Ingested before the Evaluation existed, or `evaluation_id` typo | Create the Evaluation first; set `evaluation_context.evaluation_id` to its **name** | | `400 "Evaluation '…' has been deleted…"` | The referenced Evaluation is soft-deleted | Recreate it or target a live one | -| `422` on ATIF/chat-completions | Unknown top-level key (both are `extra="forbid"`) | Remove stray keys; check the schema in `ingest-formats.md` | +| `422` on ATIF/chat-completions | Unknown top-level key (both are `extra="forbid"`) | Remove stray keys; check the schema in `../../nemo-intake/references/ingest-formats.md` | | `422` bad `schema_version` (ATIF) | Not one of `ATIF-v1.0` … `ATIF-v1.7` | Use a supported literal | | `422` non-sequential `step_id` / duplicate `tool_call_id` (ATIF) | Step/tool-call invariants violated | 1-based sequential `step_id`; unique `tool_call_id`; observation `source_call_id` must resolve | | `422 cost_total_usd` unexpected (chat-completions) | Used the wrong cost key | Use top-level `cost_usd` (not `cost_total_usd`, not nested in `response`) | diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json index 9b807227bd..8942047a01 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/tests.json @@ -1,7 +1,7 @@ { "skill": "nemo-experiments-upload", "tests": [ - { "type": "explicit", "prompt": "How do I log traces to NeMo intake?", "expected_skill": "nemo-experiments-upload" }, + { "type": "explicit", "prompt": "How do I log evaluation traces to NeMo Experiments?", "expected_skill": "nemo-experiments-upload" }, { "type": "explicit", "prompt": "Upload my Harbor eval results to NeMo Experiments", "expected_skill": "nemo-experiments-upload" }, { "type": "explicit", "prompt": "log to experiments", "expected_skill": "nemo-experiments-upload" }, { "type": "implicit", "prompt": "I ran an agent eval and want to see it in the platform leaderboard", "expected_skill": "nemo-experiments-upload" }, diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md new file mode 100644 index 0000000000..f7a6183a9c --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/SKILL.md @@ -0,0 +1,150 @@ +--- +name: nemo-intake +description: Instrument agents, ingest telemetry into NeMo Intake, and query spans, traces, sessions, and evaluator results. Use when connecting agent code or existing telemetry to Intake, choosing among OTLP, chat-completions, or ATIF, checking Intake and ClickHouse readiness, inspecting agent runs, or attaching evaluation scores outside the Experiments leaderboard workflow. +license: Apache-2.0 +allowed-tools: [Bash, Read] +--- + +# Use NeMo Intake + +Use Intake to normalize agent telemetry into queryable spans and traces. The Intake endpoint may be +local or remote. An agent does not need to run on NeMo Platform; it only needs network access and +credentials, when required, to send a supported format to that endpoint. + +## Requirements + +Set the target to the local or remote NeMo Platform origin: + +```bash +export NMP_BASE_URL=http://127.0.0.1:8080 +export WORKSPACE=default + +nmp_authority=${NMP_BASE_URL#*://} +nmp_authority=${nmp_authority%%/*} +case "${nmp_authority}" in + *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;; +esac +case "${NMP_BASE_URL}" in + https://*) ;; + http://*) + case "${nmp_authority}" in + localhost|127.0.0.1) ;; + localhost:*|127.0.0.1:*) + nmp_port=${nmp_authority#*:} + case "${nmp_port}" in + ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;; + esac + ;; + *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;; + esac + ;; + *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;; +esac +``` + +Require: + +- Reachable ClickHouse storage. +- A reachable local or remote NeMo Platform `intake` service with its `auth` and `entities` + dependencies. ClickHouse must be reachable from Intake; it does not need to be reachable from the + telemetry producer. +- One supported telemetry source. NeMo Studio is optional. + +For a remote deployment, set `NMP_BASE_URL` to its HTTPS origin and skip local startup. Use the +deployment's authentication mechanism. Do not send authentication across an HTTP redirect: validate +the final HTTPS origin and do not add `curl -L` to authenticated requests. For a local source +checkout, follow `SETUP.md`, then start ClickHouse before the backend: + +```bash +services/intake/scripts/spans/run_clickhouse.sh +uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 +``` + +Verify the Intake read path before ingesting: + +```bash +curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1" +``` + +Continue only on `200`; an empty list is healthy. `503` means Intake cannot reach ClickHouse. For +other failures, report the response and route platform startup problems to `setup` or `nemo-status`. + +## Choose an ingest path + +| Path | Use it for | Endpoint | +|---|---|---| +| **OTLP/HTTP protobuf** | Live, granular telemetry when instrumentation emits **OpenInference** or **OTel GenAI semantic conventions**. Prefer this for ongoing observability and complete agent hierarchies. | `POST .../ingest/otlp/v1/traces` | +| **Chat completions** | Captured OpenAI-compatible request/response logs, proxy instrumentation, or runtimes without OpenInference or OTel GenAI instrumentation. It represents one model interaction, not a full agent trajectory. | `POST .../ingest/chat-completions` | +| **ATIF** | Complete agent trajectories with ordered steps and metadata, especially Harbor evaluation trials. | `POST .../ingest/atif` | + +All endpoint prefixes are: +`$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE`. + +Read `references/ingest-formats.md` before hand-writing an ATIF or chat-completions payload, or when +you need the semantic attributes and response behavior for OTLP. + +Do not translate a native source format without need. Preserve the source's IDs, timestamps, +hierarchy, inputs, outputs, statuses, errors, and semantic attributes. + +## Instrument code for OTLP + +Only choose OTLP when the instrumentation emits **OpenInference** or **OTel GenAI semantic +conventions**. Generic OpenTelemetry spans may ingest, but they do not provide the semantic model, +tool, token, cost, input/output, and session fields needed for useful Intake telemetry. + +1. Recommend **NeMo Relay** when it supports the agent runtime and emits a supported semantic + format; it avoids hand-written span capture and records agent, model, and tool activity + consistently. Use the format Relay actually exports: send semantic OTLP to the OTLP endpoint, or + Relay ATIF output to the ATIF endpoint. +2. Otherwise use the runtime's **OpenInference instrumentor**, or its native **OTel GenAI** + instrumentation, with an OpenTelemetry SDK OTLP exporter. Choose the framework-specific + instrumentor from its current documentation; do not invent package names or APIs. +3. If neither semantic convention is available, capture each OpenAI-compatible request and response + and send it directly to `.../ingest/chat-completions`. Do not recommend generic OTLP as a + substitute. +4. For supported OTLP instrumentation, point the exporter at Intake: + + ```bash + export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces" + export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf + ``` + +5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. + Preserve parent-child IDs and set a stable `session.id` for related traces. +6. Run one representative interaction, then verify it through the spans query below. + +Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, +status, token, cost, and error fields while retaining unhandled attributes. + +## Customer-facing data model + +The hierarchy is `session -> trace -> span`; evaluator results attach to a span and session. + +| Record | Meaning | +|---|---| +| **Span** | One timed operation, such as an agent step, model call, tool call, retrieval, guardrail, evaluator, or chain step. It carries IDs, timing, status/error, input/output, semantic fields, and source attributes. | +| **Trace** | One end-to-end agent run. Its spans share a trace ID and form a parent-child tree. | +| **Session** | Related traces, such as a multi-turn conversation or the traces for one evaluation case. A stable session ID is the main grouping key across ingest paths. | +| **Evaluator result** | A score attached to an exact span and session. `NUMERIC` and `BOOLEAN` use `value`; `CATEGORICAL` and `TEXT` use `string_value`. | + +For ATIF, top-level `extra.verifier_result.rewards = {criterion: score}` automatically creates a +`harbor.verifier` evaluator span and one evaluator result per criterion. Stock Harbor keeps rewards +in a separate `reward.json`; enrich the trajectory before ingesting or post results explicitly: + +```bash +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \ + -H 'Content-Type: application/json' \ + -d '{"span_id":"","session_id":"","name":"faithfulness/v1","data_type":"NUMERIC","value":0.82}' +``` + +## Verify ingestion + +Always query the interaction back; a successful POST alone is insufficient: + +```bash +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=&page=1&page_size=100" +``` + +Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, +status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare +them in a leaderboard, hand off to `nemo-experiments-upload`. diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md similarity index 93% rename from packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md rename to packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md index b46d7312eb..a029753b8d 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-experiments-upload/references/ingest-formats.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/references/ingest-formats.md @@ -69,7 +69,8 @@ automatically; you don't call `/evaluator-results` separately for Harbor runs. - Two rewards under `extra.verifier_result.rewards` → two `evaluator_results` rows named `correctness` and `structure`, aggregated per-evaluator on the read model. -See `harbor-quickstart.md` for mapping a Harbor trial result to this shape. +See `../../nemo-experiments-upload/references/harbor-quickstart.md` for mapping a Harbor trial result +to this shape and publishing it as an Evaluation. --- @@ -113,11 +114,15 @@ See `harbor-quickstart.md` for mapping a Harbor trial result to this shape. --- -## 3. OTLP (OpenTelemetry spans) +## 3. OTLP (OpenInference or OTel GenAI spans) `POST .../ingest/otlp/v1/traces` with `Content-Type: application/x-protobuf` (a standard OTLP/HTTP trace export). Response `{ "errors": [] }` (per-span errors collected; HTTP stays 200). +Use this path only when spans carry OpenInference or OTel GenAI semantic conventions. If the runtime +has neither, capture its OpenAI-compatible requests and responses and use `ingest/chat-completions`; +generic OTel spans do not provide the semantic fields needed for useful model and tool telemetry. + There is **no JSON `evaluation_context`** here — evaluation identity travels as **span attributes** on the root span: diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/tests.json b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/tests.json new file mode 100644 index 0000000000..89f9909be8 --- /dev/null +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-intake/tests.json @@ -0,0 +1,16 @@ +{ + "skill": "nemo-intake", + "tests": [ + { "type": "explicit", "prompt": "Use nemo-intake to instrument my agent and send its traces to Intake.", "expected_skill": "nemo-intake" }, + { "type": "explicit", "prompt": "Upload these OpenTelemetry spans to NeMo Intake over OTLP.", "expected_skill": "nemo-intake" }, + { "type": "implicit", "prompt": "I have OpenAI-compatible request and response logs. Make them queryable as agent telemetry.", "expected_skill": "nemo-intake" }, + { "type": "implicit", "prompt": "Should I use OTLP, chat completions, or ATIF for these traces?", "expected_skill": "nemo-intake" }, + { "type": "contextual", "prompt": "Connect my LangChain agent to Intake with an OpenInference exporter.", "expected_skill": "nemo-intake" }, + { "type": "contextual", "prompt": "My runtime has generic OTel spans but no OpenInference or GenAI semantic conventions. How should I log its model calls?", "expected_skill": "nemo-intake" }, + { "type": "contextual", "prompt": "Send telemetry from my local agent to a remote NeMo Intake deployment.", "expected_skill": "nemo-intake" }, + { "type": "contextual", "prompt": "Attach a categorical evaluator result to this Intake span.", "expected_skill": "nemo-intake" }, + { "type": "negative-control", "prompt": "Create an Experiment and Evaluation and publish my Harbor scores to the leaderboard.", "expected_skill_not": "nemo-intake" }, + { "type": "negative-control", "prompt": "Author an LLM-as-a-judge metric and run it.", "expected_skill_not": "nemo-intake" }, + { "type": "negative-control", "prompt": "Show the health of every NeMo Platform service.", "expected_skill_not": "nemo-intake" } + ] +} diff --git a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md index 6a22f1ea79..c53ec9a5bc 100644 --- a/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md +++ b/packages/nemo_platform_ext/src/nemo_platform_ext/skills/nemo-skill-selection/SKILL.md @@ -1,6 +1,6 @@ --- name: nemo-skill-selection -description: Top-level skill selector for ambiguous tasks involving NeMo Platform (NVIDIA's agent platform). Picks the right downstream skill for setup, design, specification, agent configuration, build, deployment, testing, status, teardown, evaluation, optimization, security, or model customization. Use when the user needs help deciding where a NeMo Platform task should start. +description: Top-level skill selector for ambiguous tasks involving NeMo Platform (NVIDIA's agent platform). Picks the right downstream skill for setup, design, specification, agent configuration, build, deployment, testing, observability, status, teardown, evaluation, optimization, security, or model customization. Use when the user needs help deciding where a NeMo Platform task should start. triggers: - build an agent - create an agent @@ -20,18 +20,20 @@ not-for: - nemo-build-agent (use for the actual scaffold/deploy flow) - nemo-explore (use to reason about agent design) - superpowers:brainstorming (use for design work unrelated to NeMo Platform) - - running platform commands (each downstream skill owns its own commands) + - running downstream workflow or state-changing platform commands (each downstream skill owns its own commands) - loading multiple downstream skills in one turn -compatibility: nemo-platform >= 0.1.0; pure selection (no commands run from this skill); safe under macOS or Linux sandbox; works without an installed CLI (selector can pick setup, which then tells the user how to run the CLI install). +compatibility: nemo-platform >= 0.1.0; selection plus a host scan on macOS or Linux; works without an installed CLI (selector can pick setup, which then tells the user how to run the CLI install). maturity: active license: Apache-2.0 user-invocable: true -allowed-tools: [Read] --- # NeMo Platform skill selection -You are deciding which downstream NeMo Platform skill should run. This skill never executes commands. It picks the next skill, announces the choice, and hands off. +Decide which downstream NeMo Platform skill should run. Bash access is unrestricted at runtime and +can execute state-changing commands; the scope below is a behavioral constraint, not an enforced +allowlist. Execute only the host scan in this skill's Pre-flight section, then announce the choice +and hand off. Never run downstream workflow or state-changing platform commands from this skill. New NeMo Platform agent builds use a Platform-managed `agent.yaml` with `config_format: nemo-agents-spec-v1` and a supported harness. NVIDIA NeMo Agent @@ -55,6 +57,8 @@ Match the user's intent to one downstream skill. Pick exactly one. | "write agent.yaml", "validate agent.yaml", "choose a harness", "migrate this NAT YAML", "convert to nemo-agents-spec-v1" | `nemo-agent-config` | Author or migrate the Platform-managed machine-readable config without running the full build | | "build the agent", "create the agent", "deploy", "scaffold from spec" | `nemo-build-agent` | Build from the approved spec, default to Platform `agent.yaml`, register, deploy, evaluate, and optionally apply guardrails | | "ask my agent", "try the agent", "test it", "invoke this agent.yaml" | `nemo-try-agent` | Invoke a named deployment or run a local agent YAML config once | +| "instrument my agent", "send traces", "use Intake", "agent observability", "query spans or traces" | `nemo-intake` | Choose an ingest path, instrument the source, ingest telemetry, and verify spans, traces, sessions, or evaluator results | +| "create an experiment", "publish evaluation runs", "evaluation leaderboard" | `nemo-experiments-upload` | Create Experiments and Evaluations, ingest their telemetry and scores, and verify leaderboard rollups | | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | @@ -83,7 +87,8 @@ lsof -iTCP:8080 -sTCP:LISTEN 2>/dev/null curl -sS --connect-timeout 2 --max-time 5 http://localhost:8080/health/ready -o /dev/null -w "%{http_code}\n" 2>/dev/null || echo "no-response" # 3. Conflict check: other platform processes / data dirs / configs on this host? -ps -eo pid,user,command 2>/dev/null | grep -E "nemo services (run|start)|nemo-platform run" | grep -v grep +ps -eo pid=,user=,comm=,args= 2>/dev/null \ + | awk '$0 ~ /[n]emo services (run|start)|[n]emo-platform run/ {print $1, $2, $3}' ls -d ~/.local/share/nemo* 2>/dev/null ls ~/.config/nmp*/config.yaml 2>/dev/null ``` @@ -94,7 +99,7 @@ Interpretation: |---|---|---| | (1) returns a listener AND (2) returns `200` | the requested downstream skill | Platform is up and ready. Skip `setup`. | | (1) returns a listener but (2) returns `no-response` or non-200 | `nemo-status` | Something is bound to :8080 but the platform is not ready. Do not start a second platform. | -| (1) empty but (3) finds another `nemo services` process OR more than one data dir / config | **stop, do not hand off yet** | Another install on this host, possibly on a different port. Surface the inventory verbatim. Ask whether to tear that one down first, pick a different port + data dir, or abort. Two installs writing to the same `~/.config/nmp/config.yaml` is how users end up with one Studio frontend pointing at the wrong backend. | +| (1) empty but (3) finds another `nemo services` process OR more than one data dir / config | **stop, do not hand off yet** | Another install on this host, possibly on a different port. Surface only the redacted PID, user, and executable inventory emitted above. Ask whether to tear that one down first, pick a different port + data dir, or abort. Two installs writing to the same `~/.config/nmp/config.yaml` is how users end up with one Studio frontend pointing at the wrong backend. | | (1), (2), and (3) all empty | `setup` | Clean machine, no platform installed. | Read-only callers (this skill, `nemo-status`, the build/try pre-flights) should not trust `nemo services status` or `nemo services ls` as an up-check. Both report stale "running" from a held instance lock after the underlying process has died. The lock reconciles automatically the next time `nemo services run` is invoked, but until that happens, `lsof` is ground truth. (Tracking a CLI-side fix for this so we can drop the workaround from skills.) @@ -103,7 +108,7 @@ Read-only callers (this skill, `nemo-status`, the build/try pre-flights) should Tell the user, in one sentence, which skill is next and what it will do. For example: "Handing off to `setup` to verify the platform is installed and running. If it isn't, the skill will tell you the CLI command to run; install is a 5-minute shell step that this skill cannot do reliably for you." -Then hand off. Do not run any platform commands from this skill. +Then hand off. Do not run downstream workflow or state-changing platform commands from this skill. ## If nothing matches @@ -117,6 +122,8 @@ NeMo Platform skills I can route to: nemo-agent-config author, validate, or migrate Platform agent.yaml nemo-build-agent build from the spec, register, deploy, evaluate, and sign off nemo-try-agent invoke a named deployment or local agent YAML config + nemo-intake instrument agents, ingest/query telemetry, attach scores + nemo-experiments-upload publish named evaluation runs to an Experiments leaderboard nemo-status read-only platform health dashboard nemo-teardown guided shutdown @@ -135,7 +142,15 @@ Which one fits what you're trying to do? For things outside this catalog (for example, "show me how Switchyard routes between models"), point at the relevant repo skill (`nemo-evaluator`, `nemo-auditor`, etc.) or tell the user no skill claims that intent yet. Do not invent a path. -If the pre-flight finds no platform but the user insists they have installed one: ask them to report the output of `lsof -iTCP:8080 -sTCP:LISTEN` and `ps -eo pid,user,command | grep -E "nemo services|nemo-platform run" | grep -v grep` from the shell where they ran setup. The platform may be bound to a non-default port, or the install may be in a venv whose `nemo` binary is not on `PATH`. +If the pre-flight finds no platform but the user insists they have installed one: ask them to report +the output of `lsof -iTCP:8080 -sTCP:LISTEN` and the redacted scan below from the shell where they ran +setup. The platform may be bound to a non-default port, or the install may be in a venv whose `nemo` +binary is not on `PATH`. + +```bash +ps -eo pid=,user=,comm=,args= 2>/dev/null \ + | awk '$0 ~ /[n]emo services|[n]emo-platform run/ {print $1, $2, $3}' +``` ## If the user asks about Studio (web UI) diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.md index 4b4faa537d..16400ac9e8 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/SKILL.md @@ -1,21 +1,22 @@ --- name: nemo-experiments-upload -description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up as Experiments. Create an Experiment, create an Evaluation, then log traces and evaluator results via the ATIF (Harbor), chat-completions, or OTLP ingest endpoint — and view the rollups in Studio. Use when a user wants to upload, log, ingest, publish, or send evaluation runs, agent traces, or scores to NeMo Experiments / Intake. +description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up in the Experiments leaderboard. Create an Experiment, create an Evaluation, then log traces and evaluator results via ATIF (Harbor), chat-completions, or OTLP and view the rollups in Studio. Use when a user wants to create named evaluation runs, publish evaluation results, or compare runs in NeMo Experiments. triggers: - - log traces to intake + - log evaluation traces to experiments - upload experiment results - ingest evaluation data - how do I log to experiments - - send traces to nemo intake + - send evaluation traces to nemo experiments - publish evaluation results - log to experiments - upload harbor / atif results - get my eval data into nemo not-for: + - nemo-intake (use for general instrumentation, telemetry ingestion, trace queries, or evaluator results outside an Experiments leaderboard) - nemo-evaluator (use to AUTHOR and RUN evaluations/metrics; this skill UPLOADS results) - nemo-status (use for a read-only platform health dashboard) - nemo-skill-selection (use for dispatch when intent is unclear) -compatibility: nemo-platform >= 0.1.0; needs the intake service running (auth, entities, intake) and ClickHouse for rollups/results; talks HTTP to /apis/intake/v2 (curl only, no Docker); Experiments viewing in Studio is behind the VITE_FF_EXPERIMENT feature flag. +compatibility: nemo-platform >= 0.1.0; needs a reachable local or remote intake service (with auth and entities) backed by ClickHouse for rollups/results; talks HTTP to /apis/intake/v2 (curl only, no Docker); Experiments viewing in Studio is behind the VITE_FF_EXPERIMENT feature flag. maturity: beta license: Apache-2.0 user-invocable: true @@ -26,20 +27,45 @@ allowed-tools: [Bash, Read, Write] Get evaluation runs into the platform end-to-end: **create an Experiment → create an Evaluation → log traces + scores to an ingest endpoint → see the rollups.** The API calls the parent (the leaderboard) an **Experiment** and each row an **Evaluation**; the whole feature is called **Experiments**. -Everything below uses `${NMP_BASE_URL}` (default `http://localhost:8080`) and a `${WORKSPACE}` (default `default`). All routes are under `/apis/intake/v2/workspaces/${WORKSPACE}`. +Everything below uses `${NMP_BASE_URL}` (default `http://localhost:8080`) and a `${WORKSPACE}` +(default `default`). Point `NMP_BASE_URL` at the local platform or a remote HTTPS origin. Reject +non-loopback `http://` targets, and never send authentication across an HTTP redirect. All routes +are under `/apis/intake/v2/workspaces/${WORKSPACE}`. ## Pre-flight -Confirm intake is up before doing anything. If this fails, the platform isn't running — route to `setup`/`nemo-status` and stop. +Confirm the target platform is reachable before doing anything. If this fails, report the target as +unreachable and stop; route to `setup`/`nemo-status` only for a local platform. ```bash set -euo pipefail : "${NMP_BASE_URL:=http://localhost:8080}" : "${WORKSPACE:=default}" +nmp_authority=${NMP_BASE_URL#*://} +nmp_authority=${nmp_authority%%/*} +case "${nmp_authority}" in + *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;; +esac +case "${NMP_BASE_URL}" in + https://*) ;; + http://*) + case "${nmp_authority}" in + localhost|127.0.0.1) ;; + localhost:*|127.0.0.1:*) + nmp_port=${nmp_authority#*:} + case "${nmp_port}" in + ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;; + esac + ;; + *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;; + esac + ;; + *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;; +esac if curl -sf "${NMP_BASE_URL}/health/ready" >/dev/null; then echo "platform ready" else - echo "NOT READY — platform isn't running; route to setup/nemo-status and stop" >&2 + echo "NOT READY — target platform is unreachable or not ready" >&2 exit 1 fi ``` @@ -97,20 +123,22 @@ curl -sf -X POST \ ### 3. Log traces + evaluator results -Pick the ingest endpoint that matches your producer. **Read `references/ingest-formats.md` for the full schema and a copy-pasteable example for each.** How you attach evaluation identity depends on the endpoint: +Pick the ingest endpoint that matches your producer. **Read `../nemo-intake/references/ingest-formats.md` for the full schema and a copy-pasteable example for each.** How you attach evaluation identity depends on the endpoint: - **ATIF and chat-completions** (JSON body) — add an `evaluation_context = {evaluation_id: "", test_case_id: ""}` object to the payload. -- **OTLP** — there is no body field; set identity as **root-span resource attributes** `nemo.experiment.id` (the Evaluation **name**) and `nemo.test_case.id`. Spans missing these still ingest but won't associate to an Evaluation. +- **OTLP** — there is no body field; set `nemo.experiment.id` (the Evaluation **name**) and + `nemo.test_case.id` (the task ID) as **attributes on the root span**. Spans missing these still + ingest but won't associate to an Evaluation. | Producer | Endpoint | Read | |---|---|---| | **Harbor / agent trajectories** (most common) | `POST .../ingest/atif` | `references/harbor-quickstart.md` | -| A single captured model call | `POST .../ingest/chat-completions` | `references/ingest-formats.md` | -| OpenTelemetry spans | `POST .../ingest/otlp/v1/traces` | `references/ingest-formats.md` | +| A single captured model call | `POST .../ingest/chat-completions` | `../nemo-intake/references/ingest-formats.md` | +| OpenTelemetry spans | `POST .../ingest/otlp/v1/traces` | `../nemo-intake/references/ingest-formats.md` | Evaluator **scores** arrive one of two ways (both covered in the references): - **Automatically** with ATIF — put rewards under `extra.verifier_result.rewards` (one key per criterion). -- **Explicitly** — `POST .../evaluator-results` with `{span_id, session_id, name, data_type, value}` — use `string_value` instead of `value` for `CATEGORICAL`/`TEXT` results (see `references/ingest-formats.md`). +- **Explicitly** — `POST .../evaluator-results` with `{span_id, session_id, name, data_type, value}` — use `string_value` instead of `value` for `CATEGORICAL`/`TEXT` results (see `../nemo-intake/references/ingest-formats.md`). ### 4. Verify the data landed @@ -134,7 +162,7 @@ Open Studio → the **Experiments** area (behind the `VITE_FF_EXPERIMENT` flag) Read these before hand-writing a payload: -- **`references/ingest-formats.md`** — the three ingest endpoints in full: request schemas, how `evaluation_context` and evaluator results attach, and a working example for each (ATIF, chat-completions, OTLP). +- **`../nemo-intake/references/ingest-formats.md`** — the shared Intake request schemas, evaluation context, evaluator results, and examples for ATIF, chat-completions, and OTLP. - **`references/harbor-quickstart.md`** — the Harbor path specifically: mapping a Harbor trial result → an ATIF payload, including verifier rewards → evaluator scores. - **`references/troubleshooting.md`** — every common `400`/`422`/`503` from the ingest and CRUD endpoints, with the fix. @@ -151,7 +179,7 @@ If `run_count` is 0 after ingesting, the traces didn't associate — almost alwa | Symptom | Cause | Recovery | |---|---|---| | `400 "…must be created before it can be logged."` | Ingested before the Evaluation existed, or `evaluation_id` doesn't match | Create the Evaluation (step 2); ensure `evaluation_context.evaluation_id` equals its **name** | -| `422 Unprocessable` on ingest | Unknown/typo'd top-level key (ATIF/chat-completions are `extra="forbid"`) or bad `schema_version` | Check the exact schema in `references/ingest-formats.md`; remove stray keys | +| `422 Unprocessable` on ingest | Unknown/typo'd top-level key (ATIF/chat-completions are `extra="forbid"`) or bad `schema_version` | Check the exact schema in `../nemo-intake/references/ingest-formats.md`; remove stray keys | | Ingest 2xx but `run_count` stays 0 | Evaluation identity missing/wrong — `evaluation_context.evaluation_id` (ATIF/chat-completions) or the `nemo.experiment.id` root-span attribute (OTLP) ≠ the Evaluation's name | Attach the identity for your endpoint; use the Evaluation **name**, not its id | | `503` on GET evaluation / sessions | ClickHouse (telemetry store) not running | Start ClickHouse; rollups and sessions require it | | Scores don't show up | Rewards not under `extra.verifier_result.rewards`, or wrong `data_type` on `/evaluator-results` | See `references/troubleshooting.md` | diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.md index 736edc1a66..8cd5170a5c 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/troubleshooting.md @@ -18,7 +18,7 @@ string — read it first. |---|---|---| | `400 "Evaluation '…' must be created before it can be logged."` | Ingested before the Evaluation existed, or `evaluation_id` typo | Create the Evaluation first; set `evaluation_context.evaluation_id` to its **name** | | `400 "Evaluation '…' has been deleted…"` | The referenced Evaluation is soft-deleted | Recreate it or target a live one | -| `422` on ATIF/chat-completions | Unknown top-level key (both are `extra="forbid"`) | Remove stray keys; check the schema in `ingest-formats.md` | +| `422` on ATIF/chat-completions | Unknown top-level key (both are `extra="forbid"`) | Remove stray keys; check the schema in `../../nemo-intake/references/ingest-formats.md` | | `422` bad `schema_version` (ATIF) | Not one of `ATIF-v1.0` … `ATIF-v1.7` | Use a supported literal | | `422` non-sequential `step_id` / duplicate `tool_call_id` (ATIF) | Step/tool-call invariants violated | 1-based sequential `step_id`; unique `tool_call_id`; observation `source_call_id` must resolve | | `422 cost_total_usd` unexpected (chat-completions) | Used the wrong cost key | Use top-level `cost_usd` (not `cost_total_usd`, not nested in `response`) | diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md new file mode 100644 index 0000000000..f7a6183a9c --- /dev/null +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/SKILL.md @@ -0,0 +1,150 @@ +--- +name: nemo-intake +description: Instrument agents, ingest telemetry into NeMo Intake, and query spans, traces, sessions, and evaluator results. Use when connecting agent code or existing telemetry to Intake, choosing among OTLP, chat-completions, or ATIF, checking Intake and ClickHouse readiness, inspecting agent runs, or attaching evaluation scores outside the Experiments leaderboard workflow. +license: Apache-2.0 +allowed-tools: [Bash, Read] +--- + +# Use NeMo Intake + +Use Intake to normalize agent telemetry into queryable spans and traces. The Intake endpoint may be +local or remote. An agent does not need to run on NeMo Platform; it only needs network access and +credentials, when required, to send a supported format to that endpoint. + +## Requirements + +Set the target to the local or remote NeMo Platform origin: + +```bash +export NMP_BASE_URL=http://127.0.0.1:8080 +export WORKSPACE=default + +nmp_authority=${NMP_BASE_URL#*://} +nmp_authority=${nmp_authority%%/*} +case "${nmp_authority}" in + *@*) echo "NMP_BASE_URL must not contain userinfo" >&2; exit 1 ;; +esac +case "${NMP_BASE_URL}" in + https://*) ;; + http://*) + case "${nmp_authority}" in + localhost|127.0.0.1) ;; + localhost:*|127.0.0.1:*) + nmp_port=${nmp_authority#*:} + case "${nmp_port}" in + ""|*[!0-9]*) echo "loopback NMP_BASE_URL has an invalid port" >&2; exit 1 ;; + esac + ;; + *) echo "HTTP NMP_BASE_URL must use exactly localhost or 127.0.0.1" >&2; exit 1 ;; + esac + ;; + *) echo "remote NMP_BASE_URL must use https://" >&2; exit 1 ;; +esac +``` + +Require: + +- Reachable ClickHouse storage. +- A reachable local or remote NeMo Platform `intake` service with its `auth` and `entities` + dependencies. ClickHouse must be reachable from Intake; it does not need to be reachable from the + telemetry producer. +- One supported telemetry source. NeMo Studio is optional. + +For a remote deployment, set `NMP_BASE_URL` to its HTTPS origin and skip local startup. Use the +deployment's authentication mechanism. Do not send authentication across an HTTP redirect: validate +the final HTTPS origin and do not add `curl -L` to authenticated requests. For a local source +checkout, follow `SETUP.md`, then start ClickHouse before the backend: + +```bash +services/intake/scripts/spans/run_clickhouse.sh +uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 +``` + +Verify the Intake read path before ingesting: + +```bash +curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1" +``` + +Continue only on `200`; an empty list is healthy. `503` means Intake cannot reach ClickHouse. For +other failures, report the response and route platform startup problems to `setup` or `nemo-status`. + +## Choose an ingest path + +| Path | Use it for | Endpoint | +|---|---|---| +| **OTLP/HTTP protobuf** | Live, granular telemetry when instrumentation emits **OpenInference** or **OTel GenAI semantic conventions**. Prefer this for ongoing observability and complete agent hierarchies. | `POST .../ingest/otlp/v1/traces` | +| **Chat completions** | Captured OpenAI-compatible request/response logs, proxy instrumentation, or runtimes without OpenInference or OTel GenAI instrumentation. It represents one model interaction, not a full agent trajectory. | `POST .../ingest/chat-completions` | +| **ATIF** | Complete agent trajectories with ordered steps and metadata, especially Harbor evaluation trials. | `POST .../ingest/atif` | + +All endpoint prefixes are: +`$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE`. + +Read `references/ingest-formats.md` before hand-writing an ATIF or chat-completions payload, or when +you need the semantic attributes and response behavior for OTLP. + +Do not translate a native source format without need. Preserve the source's IDs, timestamps, +hierarchy, inputs, outputs, statuses, errors, and semantic attributes. + +## Instrument code for OTLP + +Only choose OTLP when the instrumentation emits **OpenInference** or **OTel GenAI semantic +conventions**. Generic OpenTelemetry spans may ingest, but they do not provide the semantic model, +tool, token, cost, input/output, and session fields needed for useful Intake telemetry. + +1. Recommend **NeMo Relay** when it supports the agent runtime and emits a supported semantic + format; it avoids hand-written span capture and records agent, model, and tool activity + consistently. Use the format Relay actually exports: send semantic OTLP to the OTLP endpoint, or + Relay ATIF output to the ATIF endpoint. +2. Otherwise use the runtime's **OpenInference instrumentor**, or its native **OTel GenAI** + instrumentation, with an OpenTelemetry SDK OTLP exporter. Choose the framework-specific + instrumentor from its current documentation; do not invent package names or APIs. +3. If neither semantic convention is available, capture each OpenAI-compatible request and response + and send it directly to `.../ingest/chat-completions`. Do not recommend generic OTLP as a + substitute. +4. For supported OTLP instrumentation, point the exporter at Intake: + + ```bash + export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces" + export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf + ``` + +5. Emit a root agent/chain span plus granular model, tool, retrieval, guardrail, and error spans. + Preserve parent-child IDs and set a stable `session.id` for related traces. +6. Run one representative interaction, then verify it through the spans query below. + +Intake maps OpenInference and OTel GenAI semantic attributes into queryable model, provider, tool, +status, token, cost, and error fields while retaining unhandled attributes. + +## Customer-facing data model + +The hierarchy is `session -> trace -> span`; evaluator results attach to a span and session. + +| Record | Meaning | +|---|---| +| **Span** | One timed operation, such as an agent step, model call, tool call, retrieval, guardrail, evaluator, or chain step. It carries IDs, timing, status/error, input/output, semantic fields, and source attributes. | +| **Trace** | One end-to-end agent run. Its spans share a trace ID and form a parent-child tree. | +| **Session** | Related traces, such as a multi-turn conversation or the traces for one evaluation case. A stable session ID is the main grouping key across ingest paths. | +| **Evaluator result** | A score attached to an exact span and session. `NUMERIC` and `BOOLEAN` use `value`; `CATEGORICAL` and `TEXT` use `string_value`. | + +For ATIF, top-level `extra.verifier_result.rewards = {criterion: score}` automatically creates a +`harbor.verifier` evaluator span and one evaluator result per criterion. Stock Harbor keeps rewards +in a separate `reward.json`; enrich the trajectory before ingesting or post results explicitly: + +```bash +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \ + -H 'Content-Type: application/json' \ + -d '{"span_id":"","session_id":"","name":"faithfulness/v1","data_type":"NUMERIC","value":0.82}' +``` + +## Verify ingestion + +Always query the interaction back; a successful POST alone is insufficient: + +```bash +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=&page=1&page_size=100" +``` + +Confirm the response contains the expected session, trace/span hierarchy, inputs and outputs, +status/errors, and any evaluator results. If the goal is to create named evaluation runs and compare +them in a leaderboard, hand off to `nemo-experiments-upload`. diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/references/ingest-formats.md similarity index 93% rename from sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.md rename to sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/references/ingest-formats.md index b46d7312eb..a029753b8d 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-experiments-upload/references/ingest-formats.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-intake/references/ingest-formats.md @@ -69,7 +69,8 @@ automatically; you don't call `/evaluator-results` separately for Harbor runs. - Two rewards under `extra.verifier_result.rewards` → two `evaluator_results` rows named `correctness` and `structure`, aggregated per-evaluator on the read model. -See `harbor-quickstart.md` for mapping a Harbor trial result to this shape. +See `../../nemo-experiments-upload/references/harbor-quickstart.md` for mapping a Harbor trial result +to this shape and publishing it as an Evaluation. --- @@ -113,11 +114,15 @@ See `harbor-quickstart.md` for mapping a Harbor trial result to this shape. --- -## 3. OTLP (OpenTelemetry spans) +## 3. OTLP (OpenInference or OTel GenAI spans) `POST .../ingest/otlp/v1/traces` with `Content-Type: application/x-protobuf` (a standard OTLP/HTTP trace export). Response `{ "errors": [] }` (per-span errors collected; HTTP stays 200). +Use this path only when spans carry OpenInference or OTel GenAI semantic conventions. If the runtime +has neither, capture its OpenAI-compatible requests and responses and use `ingest/chat-completions`; +generic OTel spans do not provide the semantic fields needed for useful model and tool telemetry. + There is **no JSON `evaluation_context`** here — evaluation identity travels as **span attributes** on the root span: diff --git a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md index 6a22f1ea79..c53ec9a5bc 100644 --- a/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md +++ b/sdk/python/nemo-platform/src/nemo_platform/skills/nemo-skill-selection/SKILL.md @@ -1,6 +1,6 @@ --- name: nemo-skill-selection -description: Top-level skill selector for ambiguous tasks involving NeMo Platform (NVIDIA's agent platform). Picks the right downstream skill for setup, design, specification, agent configuration, build, deployment, testing, status, teardown, evaluation, optimization, security, or model customization. Use when the user needs help deciding where a NeMo Platform task should start. +description: Top-level skill selector for ambiguous tasks involving NeMo Platform (NVIDIA's agent platform). Picks the right downstream skill for setup, design, specification, agent configuration, build, deployment, testing, observability, status, teardown, evaluation, optimization, security, or model customization. Use when the user needs help deciding where a NeMo Platform task should start. triggers: - build an agent - create an agent @@ -20,18 +20,20 @@ not-for: - nemo-build-agent (use for the actual scaffold/deploy flow) - nemo-explore (use to reason about agent design) - superpowers:brainstorming (use for design work unrelated to NeMo Platform) - - running platform commands (each downstream skill owns its own commands) + - running downstream workflow or state-changing platform commands (each downstream skill owns its own commands) - loading multiple downstream skills in one turn -compatibility: nemo-platform >= 0.1.0; pure selection (no commands run from this skill); safe under macOS or Linux sandbox; works without an installed CLI (selector can pick setup, which then tells the user how to run the CLI install). +compatibility: nemo-platform >= 0.1.0; selection plus a host scan on macOS or Linux; works without an installed CLI (selector can pick setup, which then tells the user how to run the CLI install). maturity: active license: Apache-2.0 user-invocable: true -allowed-tools: [Read] --- # NeMo Platform skill selection -You are deciding which downstream NeMo Platform skill should run. This skill never executes commands. It picks the next skill, announces the choice, and hands off. +Decide which downstream NeMo Platform skill should run. Bash access is unrestricted at runtime and +can execute state-changing commands; the scope below is a behavioral constraint, not an enforced +allowlist. Execute only the host scan in this skill's Pre-flight section, then announce the choice +and hand off. Never run downstream workflow or state-changing platform commands from this skill. New NeMo Platform agent builds use a Platform-managed `agent.yaml` with `config_format: nemo-agents-spec-v1` and a supported harness. NVIDIA NeMo Agent @@ -55,6 +57,8 @@ Match the user's intent to one downstream skill. Pick exactly one. | "write agent.yaml", "validate agent.yaml", "choose a harness", "migrate this NAT YAML", "convert to nemo-agents-spec-v1" | `nemo-agent-config` | Author or migrate the Platform-managed machine-readable config without running the full build | | "build the agent", "create the agent", "deploy", "scaffold from spec" | `nemo-build-agent` | Build from the approved spec, default to Platform `agent.yaml`, register, deploy, evaluate, and optionally apply guardrails | | "ask my agent", "try the agent", "test it", "invoke this agent.yaml" | `nemo-try-agent` | Invoke a named deployment or run a local agent YAML config once | +| "instrument my agent", "send traces", "use Intake", "agent observability", "query spans or traces" | `nemo-intake` | Choose an ingest path, instrument the source, ingest telemetry, and verify spans, traces, sessions, or evaluator results | +| "create an experiment", "publish evaluation runs", "evaluation leaderboard" | `nemo-experiments-upload` | Create Experiments and Evaluations, ingest their telemetry and scores, and verify leaderboard rollups | | "status", "what is running", "platform health", "is the platform up", "what's deployed", "show me what's running" | `nemo-status` | Read-only dashboard: platform, agents, providers, models | | "shut down", "stop NeMo", "tear down", "clean up" | `nemo-teardown` | Stop the cluster (keep data, delete platform data, or full cleanup) | | "fine-tune", "customize the model", "train on my data", "SFT", "LoRA" | `nemo-customizer` | Model customization via installed customization contributor plugins (`nemo-customizer-plugin`). Requires plugin skills to be installed (`nemo skills install` / enabled-plugins). | @@ -83,7 +87,8 @@ lsof -iTCP:8080 -sTCP:LISTEN 2>/dev/null curl -sS --connect-timeout 2 --max-time 5 http://localhost:8080/health/ready -o /dev/null -w "%{http_code}\n" 2>/dev/null || echo "no-response" # 3. Conflict check: other platform processes / data dirs / configs on this host? -ps -eo pid,user,command 2>/dev/null | grep -E "nemo services (run|start)|nemo-platform run" | grep -v grep +ps -eo pid=,user=,comm=,args= 2>/dev/null \ + | awk '$0 ~ /[n]emo services (run|start)|[n]emo-platform run/ {print $1, $2, $3}' ls -d ~/.local/share/nemo* 2>/dev/null ls ~/.config/nmp*/config.yaml 2>/dev/null ``` @@ -94,7 +99,7 @@ Interpretation: |---|---|---| | (1) returns a listener AND (2) returns `200` | the requested downstream skill | Platform is up and ready. Skip `setup`. | | (1) returns a listener but (2) returns `no-response` or non-200 | `nemo-status` | Something is bound to :8080 but the platform is not ready. Do not start a second platform. | -| (1) empty but (3) finds another `nemo services` process OR more than one data dir / config | **stop, do not hand off yet** | Another install on this host, possibly on a different port. Surface the inventory verbatim. Ask whether to tear that one down first, pick a different port + data dir, or abort. Two installs writing to the same `~/.config/nmp/config.yaml` is how users end up with one Studio frontend pointing at the wrong backend. | +| (1) empty but (3) finds another `nemo services` process OR more than one data dir / config | **stop, do not hand off yet** | Another install on this host, possibly on a different port. Surface only the redacted PID, user, and executable inventory emitted above. Ask whether to tear that one down first, pick a different port + data dir, or abort. Two installs writing to the same `~/.config/nmp/config.yaml` is how users end up with one Studio frontend pointing at the wrong backend. | | (1), (2), and (3) all empty | `setup` | Clean machine, no platform installed. | Read-only callers (this skill, `nemo-status`, the build/try pre-flights) should not trust `nemo services status` or `nemo services ls` as an up-check. Both report stale "running" from a held instance lock after the underlying process has died. The lock reconciles automatically the next time `nemo services run` is invoked, but until that happens, `lsof` is ground truth. (Tracking a CLI-side fix for this so we can drop the workaround from skills.) @@ -103,7 +108,7 @@ Read-only callers (this skill, `nemo-status`, the build/try pre-flights) should Tell the user, in one sentence, which skill is next and what it will do. For example: "Handing off to `setup` to verify the platform is installed and running. If it isn't, the skill will tell you the CLI command to run; install is a 5-minute shell step that this skill cannot do reliably for you." -Then hand off. Do not run any platform commands from this skill. +Then hand off. Do not run downstream workflow or state-changing platform commands from this skill. ## If nothing matches @@ -117,6 +122,8 @@ NeMo Platform skills I can route to: nemo-agent-config author, validate, or migrate Platform agent.yaml nemo-build-agent build from the spec, register, deploy, evaluate, and sign off nemo-try-agent invoke a named deployment or local agent YAML config + nemo-intake instrument agents, ingest/query telemetry, attach scores + nemo-experiments-upload publish named evaluation runs to an Experiments leaderboard nemo-status read-only platform health dashboard nemo-teardown guided shutdown @@ -135,7 +142,15 @@ Which one fits what you're trying to do? For things outside this catalog (for example, "show me how Switchyard routes between models"), point at the relevant repo skill (`nemo-evaluator`, `nemo-auditor`, etc.) or tell the user no skill claims that intent yet. Do not invent a path. -If the pre-flight finds no platform but the user insists they have installed one: ask them to report the output of `lsof -iTCP:8080 -sTCP:LISTEN` and `ps -eo pid,user,command | grep -E "nemo services|nemo-platform run" | grep -v grep` from the shell where they ran setup. The platform may be bound to a non-default port, or the install may be in a venv whose `nemo` binary is not on `PATH`. +If the pre-flight finds no platform but the user insists they have installed one: ask them to report +the output of `lsof -iTCP:8080 -sTCP:LISTEN` and the redacted scan below from the shell where they ran +setup. The platform may be bound to a non-default port, or the install may be in a venv whose `nemo` +binary is not on `PATH`. + +```bash +ps -eo pid=,user=,comm=,args= 2>/dev/null \ + | awk '$0 ~ /[n]emo services|[n]emo-platform run/ {print $1, $2, $3}' +``` ## If the user asks about Studio (web UI)