Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
218 changes: 75 additions & 143 deletions openapi/ga/individual/platform.openapi.yaml

Large diffs are not rendered by default.

218 changes: 75 additions & 143 deletions openapi/ga/openapi.yaml

Large diffs are not rendered by default.

218 changes: 75 additions & 143 deletions openapi/openapi.yaml

Large diffs are not rendered by default.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
name: nemo-experiments-upload
description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up as Experiments. Create an Experiment Group, create an Evaluation, then log traces and evaluator results via the ATIF (Harbor), chat-completions, or OTLP ingest endpoint — and view the rollups in Studio. Use when a user wants to upload, log, ingest, publish, or send evaluation runs, agent traces, or scores to NeMo Experiments / Intake.
description: End-to-end guide for getting evaluation data into NeMo Platform Intake so it shows up as Experiments. Create an Experiment, create an Evaluation, then log traces and evaluator results via the ATIF (Harbor), chat-completions, or OTLP ingest endpoint — and view the rollups in Studio. Use when a user wants to upload, log, ingest, publish, or send evaluation runs, agent traces, or scores to NeMo Experiments / Intake.
triggers:
- log traces to intake
- upload experiment results
Expand All @@ -24,7 +24,7 @@ allowed-tools: [Bash, Read, Write]

# Log evaluation data to NeMo Intake

Get evaluation runs into the platform end-to-end: **create an Experiment Group → create an Evaluation → log traces + scores to an ingest endpoint → see the rollups.** The API says "Evaluation" and "Experiment Group"; the whole feature is called **Experiments**.
Get evaluation runs into the platform end-to-end: **create an Experiment → create an Evaluation → log traces + scores to an ingest endpoint → see the rollups.** The API calls the parent (the leaderboard) an **Experiment** and each row an **Evaluation**; the whole feature is called **Experiments**.

Everything below uses `${NMP_BASE_URL}` (default `http://localhost:8080`) and a `${WORKSPACE}` (default `default`). All routes are under `/apis/intake/v2/workspaces/${WORKSPACE}`.

Expand All @@ -48,33 +48,33 @@ fi

Run the steps in order. Steps 1–2 create the entities; step 3 logs the data; steps 4–5 verify.

### 1. Create an Experiment Group
### 1. Create an Experiment

A group is the leaderboard container. You need its `id` for the next step.
An Experiment is the leaderboard container. You need its `id` for the next step.

```bash
set -euo pipefail
: "${NMP_BASE_URL:=http://localhost:8080}"
: "${WORKSPACE:=default}"
groups="${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/experiment-groups"
# Create the group. 201 = created, 409 = already exists; any other status is a real failure.
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "${groups}" \
experiments="${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/experiments"
# Create the experiment. 201 = created, 409 = already exists; any other status is a real failure.
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST "${experiments}" \
-H 'Content-Type: application/json' \
-d '{"name": "my-experiment-group", "description": "example run"}')
-d '{"name": "my-experiment", "description": "example run"}')
case "${code}" in
201|409) ;;
*) echo "group create failed: HTTP ${code}" >&2; exit 1 ;;
*) echo "experiment create failed: HTTP ${code}" >&2; exit 1 ;;
esac
# Fetch the id (works whether it was just created or already existed).
GROUP_ID=$(curl -sf "${groups}/my-experiment-group" \
EXPERIMENT_ID=$(curl -sf "${experiments}/my-experiment" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])')
[ -n "${GROUP_ID}" ] || { echo "could not resolve experiment group id" >&2; exit 1; }
echo "group id: ${GROUP_ID}"
[ -n "${EXPERIMENT_ID}" ] || { echo "could not resolve experiment id" >&2; exit 1; }
echo "experiment id: ${EXPERIMENT_ID}"
```

The POST accepts only `201` (created) or `409` (already exists) — any other status stops the step
instead of masking it. The `id` is then read with a GET, so this works on both first run and re-run;
`set -euo pipefail` + the `[ -n ]` guard keep it from continuing with an empty `GROUP_ID`.
`set -euo pipefail` + the `[ -n ]` guard keep it from continuing with an empty `EXPERIMENT_ID`.

### 2. Create an Evaluation

Expand All @@ -85,13 +85,13 @@ curl -sf -X POST \
"${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/evaluations" \
-H 'Content-Type: application/json' \
-d "{\"name\": \"my-eval-baseline\",
\"experiment_group_id\": \"${GROUP_ID}\",
\"experiment_ids\": [\"${EXPERIMENT_ID}\"],
\"dataset_name\": \"my-dataset\",
\"dataset_version\": \"v1\",
\"metadata\": {\"model\": \"provider/model\", \"job_name\": \"baseline\"}}"
```

- `experiment_group_id` is the group's **`id`** (from step 1).
- `experiment_ids` is a list holding the Experiment's **`id`** (from step 1).
- `metadata` values must be **strings** (`dict[str, str]`).
- **You must create the Evaluation before you can log to it** — ingesting with an unknown `evaluation_id` returns `400 "…must be created before it can be logged."`

Expand Down Expand Up @@ -128,7 +128,7 @@ curl -sf "${NMP_BASE_URL}/apis/intake/v2/workspaces/${WORKSPACE}/evaluations/my-

### 5. View in Studio

Open Studio → the **Experiments** area (behind the `VITE_FF_EXPERIMENT` flag) → your group → your evaluation. You'll see the leaderboard row with score/cost/latency rollups and can drill into individual sessions and traces.
Open Studio → the **Experiments** area (behind the `VITE_FF_EXPERIMENT` flag) → your experiment → your evaluation. You'll see the leaderboard row with score/cost/latency rollups and can drill into individual sessions and traces.

## Reference files

Expand Down Expand Up @@ -159,8 +159,8 @@ If `run_count` is 0 after ingesting, the traces didn't associate — almost alwa
## Gotchas

- **Create before you log.** The Evaluation entity must exist before any ingest referencing it — otherwise `400`.
- **`evaluation_id` is the Evaluation's `name`, not its entity id.** But **`experiment_group_id` is the group's `id`.** Different identifiers; easy to swap.
- **`evaluation_id` is the Evaluation's `name`, not its entity id.** But **`experiment_ids` holds the Experiment's `id`.** Different identifiers; easy to swap.
- **OTLP uses the attribute key `nemo.experiment.id`** (and `nemo.test_case.id`) — the span-attribute key still says "experiment" even though the JSON body field is `evaluation_context`. Set `nemo.experiment.id` on your root span.
- **Use `/evaluations`, not `/experiments`.** `/experiments` still works as a deprecated hidden alias but you should log to `/evaluations`.
- **The parent lives at `/experiments`; `/experiment-groups` is a deprecated hidden alias.** Prefer `/experiments`. Evaluations are created and logged under `/evaluations`.
- **`metadata` is `dict[str, str]`** — stringify non-string values or you'll get a `422`.
- **ATIF and chat-completions are `extra="forbid"`** (unknown keys → 422); `evaluation_context` itself is lenient (`extra="ignore"`).
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ each task runs several times (trials). You upload **one ATIF payload per trial**

| Harbor concept | NeMo entity | How |
|---|---|---|
| A benchmark / sweep | **Experiment Group** | `POST /experiment-groups` once |
| A benchmark / sweep | **Experiment** | `POST /experiments` once |
| One agent+config on that benchmark | **Evaluation** | `POST /evaluations` once (its `name` is your `evaluation_id`) |
| A task / test case | `test_case_id` | field inside `evaluation_context` |
| One trial (attempt) of a task | one ingested **session** | one `POST /ingest/atif` |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@
Common failures across the create + ingest endpoints, and the fix. Most 4xx bodies include a `detail`
string — read it first.

## Create Experiment Group / Evaluation
## Create Experiment / Evaluation

| Status | Meaning | Fix |
|---|---|---|
| `409` on create group/evaluation | Name already exists in this workspace | Reuse it — `GET .../experiment-groups/{name}` or `.../evaluations/{name}` |
| `400 "…group … does not exist"` on create evaluation | `experiment_group_id` is wrong or the group was deleted | Use the group's **`id`** from the create-group response (not its name) |
| `409` on create experiment/evaluation | Name already exists in this workspace | Reuse it — `GET .../experiments/{name}` or `.../evaluations/{name}` |
| `400 "…experiment … does not exist"` on create evaluation | `experiment_ids` is wrong or the experiment was deleted | Use the Experiment's **`id`** from the create-experiment response (not its name) |
| `400 "…parent … does not exist"` | `parent_evaluation_id` doesn't resolve | Omit it, or pass the parent Evaluation's entity **`id`** |
| `422` on create evaluation | Missing required field or non-string metadata value | Required: `name`, `experiment_group_id`, `dataset_name`. `metadata` must be `dict[str, str]` |
| `422` on create evaluation | Missing required field or non-string metadata value | Required: `name`, `experiment_ids`, `dataset_name`. `metadata` must be `dict[str, str]` |

## Ingest (all endpoints)

Expand Down Expand Up @@ -38,6 +38,6 @@ string — read it first.
## Identifier cheat-sheet (the #1 source of bugs)

- `evaluation_context.evaluation_id` → the Evaluation's **`name`**.
- `experiment_group_id` (on create evaluation) → the group's **`id`**.
- `experiment_ids` (on create evaluation) → a list with the Experiment's **`id`**.
- OTLP evaluation attribute key → **`nemo.experiment.id`** (test case → `nemo.test_case.id`).
- Log to **`/evaluations`** (the `/experiments` path is a deprecated hidden alias).
- Parent → **`/experiments`** (`/experiment-groups` is a deprecated hidden alias); evaluations → **`/evaluations`**.
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
{ "type": "implicit", "prompt": "How do I get my evaluation runs and scores into NeMo?", "expected_skill": "nemo-experiments-upload" },
{ "type": "implicit", "prompt": "Send my ATIF trajectory to intake with evaluation context", "expected_skill": "nemo-experiments-upload" },
{ "type": "contextual", "prompt": "I have a Harbor run with verifier rewards and I need them to show up as experiment scores", "expected_skill": "nemo-experiments-upload" },
{ "type": "contextual", "prompt": "Create an experiment group and an evaluation, then publish traces to it", "expected_skill": "nemo-experiments-upload" },
{ "type": "contextual", "prompt": "Create an experiment and an evaluation, then publish traces to it", "expected_skill": "nemo-experiments-upload" },
{ "type": "negative-control", "prompt": "Help me author an LLM-as-a-judge metric for my benchmark", "expected_skill_not": "nemo-experiments-upload" },
{ "type": "negative-control", "prompt": "Is the NeMo platform healthy right now?", "expected_skill_not": "nemo-experiments-upload" },
{ "type": "negative-control", "prompt": "What is the status of PR #612 in the platform repo?", "expected_skill_not": "nemo-experiments-upload" }
Expand Down
Loading