diff --git a/docs/agents/index.mdx b/docs/agents/index.mdx index ca7b029fd8..e403ab177b 100644 --- a/docs/agents/index.mdx +++ b/docs/agents/index.mdx @@ -111,6 +111,8 @@ undeploying the candidate. - [Deploy Agents](/documentation/agents/deploy-agents): run an agent as a local subprocess or as a durable container on Docker or Kubernetes, and invoke it through the Agents gateway. +- [Observe Agents](/documentation/agents/observe-agents): ingest and query agent telemetry with NeMo + Intake, then review traces, feedback, and evaluator results. - [Optimize Agents](/documentation/agents/optimize-agents): analyze deployed agents for model routing, skill, prompt, and new-model opportunities. - [Secure Agents](/documentation/agents/secure-agents): check guardrail coverage and scan recent diff --git a/docs/agents/observability.mdx b/docs/agents/observability.mdx new file mode 100644 index 0000000000..e5a9aa9d81 --- /dev/null +++ b/docs/agents/observability.mdx @@ -0,0 +1,342 @@ +--- +title: "Observe Agents" +description: "Ingest, store, and query agent telemetry with NeMo Intake — OTLP, chat-completion, and ATIF ingest paths, annotations, and evaluator results, reviewable in NeMo Studio." +--- + + + +Observability on NeMo Platform is provided by **NeMo Intake**, the trace ingestion and query service +for agent telemetry, and optionally by [NeMo Studio](/documentation/studio), the web UI for browsing +that telemetry. Observability helps humans and agents understand what happened in an agent system: +which models were called, which tools were used, what inputs and outputs were produced, where +failures occurred, and what feedback or evaluation scores were attached. + +## Before You Start + +### Who This Is For + +NeMo Intake is for anyone who owns, operates, evaluates, or optimizes an agent. It is written for +engineers wiring telemetry into an agent runtime, and for human and agent reviewers who need to +diagnose failures from traces, in a production system or in offline evaluation flows. + +### Requirements + +- A reachable ClickHouse database. +- A NeMo Platform environment with the `intake` service running. +- A telemetry source: an OpenTelemetry/OpenInference exporter, NeMo Agent Toolkit (NAT), NeMo Flow, + NeMo Evaluator runs, captured OpenAI-compatible chat-completion payloads, or ATIF trajectories. +- (Optional) [NeMo Studio](/documentation/studio) for the UI review flow. + +You do not need to deploy your agents on NeMo Platform. The lightweight path is to run the platform, +post one representative interaction, and confirm it appears. + +## How It Works + +### What It Does + +Intake normalizes agent telemetry across multiple ingestion paths into queryable spans and traces. It +preserves captured inputs and outputs, stores semantic fields such as model, provider, tool name, +status, token counts, cost, and errors, and keeps unhandled source attributes available for +debugging. + +Studio reads the same Intake APIs to provide trace lists, span lists, trace detail, span detail, and +annotation review pages. The UI is a human review surface; Intake is the ingestion and storage +service. + +### When to Use It + +Use Intake when you need to: + +- Feed the optimization loop with production traces, staged replays, seed interactions, or synthetic + cases. +- Diagnose specific failures from concrete runs instead of reconstructing behavior from logs. +- Establish baselines before changing a prompt, model, tool, guardrail, or agent workflow. +- Standardize telemetry across teams so everyone shares one vocabulary, data model, and review + surface. + +Production traffic is the best fuel for optimization. New agents can start with staged traffic or +generated cases, then improve coverage as real traffic arrives. + +Intake accepts three standardized ingest formats depending on your instrumentation: + +| Format | Example use cases | Endpoint | +|--------|-------------------|----------| +| OTLP/HTTP protobuf (OTel GenAI or OpenInference semantic conventions) | NeMo Relay, LangChain Deep Agents | `/apis/intake/v2/workspaces/{workspace}/ingest/otlp/v1/traces` | +| Chat completions | Importing raw logs, instrumenting a proxy server, custom logging | `/apis/intake/v2/workspaces/{workspace}/ingest/chat-completions` | +| ATIF | Running Harbor evaluations | `/apis/intake/v2/workspaces/{workspace}/ingest/atif` | + +### Core Concepts and Data Model + +Telemetry uses a three-level hierarchy: + +- A **span** is one timed operation: an LLM call, tool invocation, retrieval, guardrail, evaluator, or + chain step. +- A **trace** is one end-to-end agent run, made of spans that share a trace ID. +- A **session** groups related traces, such as a multi-turn conversation or a multi-system evaluation. + +Additional records attach signal to the same session or span: + +- **Annotations** store post-hoc feedback, labels, notes, and metadata. +- **Evaluator results** store numeric, boolean, categorical, or text scores for a span. +- **Experiments** and **Evaluations** organize evaluation runs into leaderboard rollups for + comparison. An Evaluation is a named run whose sessions are individual test cases; an Experiment + rolls related Evaluations into one leaderboard. See + [Experiments](/documentation/evaluate-models/experiments). + +The most useful instrumentation logs granular steps: every model call, tool call, final response, and +error. A practical test: the trace can answer _where did the system go wrong, and why?_ + +## Get Started + +### Setup + +The examples assume a running NeMo Platform reachable at `$NMP_BASE_URL`. Point at whatever you have, +whether a deployed platform or a local one from `nemo setup` or `nemo quickstart up`: + +```shell +export NMP_BASE_URL=http://127.0.0.1:8080 +export WORKSPACE=default +``` + + + +If you are bringing the pieces up yourself from a repository checkout, start them in this order. + +1. **ClickHouse** (the telemetry datastore): + + ```shell + services/intake/scripts/spans/run_clickhouse.sh + ``` + +1. **Backend services**: `intake` plus its `auth` (access checks) and `entities` (entity store) + dependencies. `--port` defaults to `8080`; drop `uv run` if you installed the `nemo` CLI: + + ```shell + uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 + ``` + +1. **Studio** (optional, for the UI review flow). From the `web/` workspace, with the intake feature + flag on and pointed at the backend: + + ```shell + VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ + pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 + ``` + + + +Confirm the Intake read path can reach ClickHouse: + +```shell +curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?page=1&page_size=1" +``` + +A `200` response with an empty list is healthy. A `503` response means the Intake service is running +but cannot reach ClickHouse. + +### First Workflow + +Send one captured chat-completion interaction. This is the lowest-friction smoke test — it needs no +OpenTelemetry exporter: + +```shell +export SESSION_ID="demo-session-001" +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/chat-completions" \ + -H "Content-Type: application/json" \ + -d '{ + "session_id": "'"$SESSION_ID"'", + "provider": "example", + "request": { + "model": "example-model", + "messages": [ + { "role": "user", "content": "Summarize this alert and identify the likely next action." } + ] + }, + "response": { + "choices": [ + { "message": { "role": "assistant", + "content": "This appears to be a developing incident. Verify the source, check related signals, then decide whether to escalate." } } + ] + } + }' +``` + +### Confirm It Worked + +Read the span back from Intake: + +```shell +curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[session_id]=$SESSION_ID&page=1&page_size=10" +``` + +Then open Studio and navigate to `/workspaces/default/intake/traces`. + +**What good looks like:** + +- The API call returns your interaction: one span with `session_id` `demo-session-001`, model + `example-model`, and the prompt and response you sent as its input and output. +- Studio, pointed at the same workspace, lists that trace, and opening it shows the same request and + response in the span tree. + +That is the full loop: something you posted is now queryable through the API and reviewable in the UI. +The fields Intake captured here — session, trace, span, request, response, status, error, plus any +feedback or evaluator results — are the vocabulary the rest of these docs build on. + +## Common Workflows + +### Send OTLP Traces from an Instrumented Agent + +Use OTLP when your framework or collector already emits OpenTelemetry traces: + +```shell +export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/otlp/v1/traces" +export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf +``` + +Then run the instrumented agent. Intake maps OpenInference and OTel GenAI semantic attributes onto +span fields so model, tool, status, token, and error data are queryable. + +### Send ATIF Trajectories + +Use ATIF when your source system exports complete agent trajectories — steps, agent metadata, or final +metrics: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/ingest/atif" \ + -H "Content-Type: application/json" \ + --data-binary @trajectory.json +``` + +Each trajectory is stored as a structured session of spans. If the trajectory's top-level +`extra.verifier_result.rewards` is populated, Intake writes those as evaluator results automatically. +Stock Harbor output does not populate it — Harbor's rewards live in a separate `reward.json` — so it +must be enriched first. See [Capture Evaluator Results](#capture-evaluator-results) for the automatic +and explicit paths. + +### Add Feedback and Labels + +Use annotations to attach review signal after a trace lands: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/annotations" \ + -H "Content-Type: application/json" \ + -d '{ "kind": "feedback", "session_id": "'"$SESSION_ID"'", "value": "negative" }' +``` + +Use `kind: "note"` for reviewer notes, `kind: "label"` for categorical or numeric labels, and +`kind: "metadata"` for structured key/value context. + +### Capture Evaluator Results + + + +Evaluator results — a judge's rating, a pass/fail check, a similarity score — attach to a span. There +are two ways to get them into Intake. + +**Automatically, from ATIF.** Put a `verifier_result.rewards` object in the trajectory's **top-level** +`extra`, keyed by criterion (`{criterion: score}`). On ingest, Intake synthesizes an evaluator span +named `harbor.verifier` and writes one evaluator result per key onto it; a bare scalar +`extra.verifier_result.score` instead becomes a single `reward` result. Nothing else to send. + +A stock Harbor `trajectory.json` does **not** carry this — Harbor writes rewards to a separate +`reward.json` — so you must copy them into the ATIF `extra` before ingest, or use the explicit path +below. For example: + +```json +"extra": { "verifier_result": { "rewards": { "solved": 1, "groundedness": 0.8 } } } +``` + +**Explicitly, through the API.** Post a result against a span and session. Unlike the ATIF automatic +path, which lands on the synthesized `harbor.verifier` span, this attaches to the exact `span_id` you +post. `data_type` picks which field carries the score: + +| `data_type` | Where the score goes | +|-------------|----------------------| +| `NUMERIC` | A number in `value` | +| `BOOLEAN` | `value`, either `0` or `1` | +| `CATEGORICAL` | A label in `string_value` | +| `TEXT` | Free text in `string_value` | + +For example, a numeric score: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results" \ + -H "Content-Type: application/json" \ + -d '{ + "span_id": "", + "session_id": "'"$SESSION_ID"'", + "name": "faithfulness/v1", + "data_type": "NUMERIC", + "value": 0.82 + }' +``` + +Read them back for one span, or list and filter across the workspace by evaluator name, data type, or +value range: + +```shell +curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans//evaluator-results" +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluator-results?filter[name]=faithfulness/v1&filter[value][\$gte]=0.8" +``` + +### Find Recurring Failures + +Query traces or spans with filters, then group spans by session or trace to find repeated errors: + +```shell +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans?filter[status]=ERROR&page=1&page_size=20" +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/spans/groups?by=session_id&filter[status]=ERROR" +``` + +In Studio, start from the trace list, filter to negative feedback or error status, then open trace +detail to inspect the span tree. + +### Turn Evaluation Telemetry into a Leaderboard + +Once you are ingesting evaluation runs, group them into an **Experiment** to compare them side by +side. See [Experiments](/documentation/evaluate-models/experiments), or use the agent-assisted +`nemo-experiments-upload` skill to go from zero to a populated leaderboard. + +## Operations + +### Limits + +- OTLP request bodies are capped at 5 MiB by default. +- Read endpoints default to a 30-day lookback when no time filter is supplied. +- Page size is capped at 1000 records. +- Span data and trace-index data expire after 90 days. + +Raise the OTLP body cap with `NMP_INTAKE_OTLP_MAX_BODY_BYTES` when larger batches are required. For +high-volume producers, prefer smaller export batches over very large single requests. + +### Security and Access Control + +All Intake endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`. The service +depends on platform auth and checks workspace access before ingesting or reading telemetry. + + + +Do not send secrets, credentials, raw PII, or regulated data unless your deployment, retention, and +access-control policy permits it. Intake preserves request and response payloads so reviewers can +diagnose behavior — useful for debugging, and important for data governance. + + + +### Retention and Storage + +ClickHouse is the telemetry datastore. Intake owns the ClickHouse schema and lazily initializes the +tables on first use. + +Span and trace-index tables have a 90-day TTL. Annotations and evaluator results are retained in their +ClickHouse tables without that 90-day span TTL. + +For production storage sizing, estimate span volume across the retained window, then size ClickHouse +for interactive reads over recent traces and periodic aggregate queries. + +## Related Topics + +- [Experiments](/documentation/evaluate-models/experiments): roll evaluation runs up into a comparable + leaderboard built from this telemetry. +- [Optimize Agents](/documentation/agents/optimize-agents): use captured traces as the baseline for + optimization. +- [Secure Agents](/documentation/agents/secure-agents): scan recent telemetry for sensitive data. diff --git a/docs/evaluator/experiments.mdx b/docs/evaluator/experiments.mdx new file mode 100644 index 0000000000..024765f83b --- /dev/null +++ b/docs/evaluator/experiments.mdx @@ -0,0 +1,458 @@ +--- +title: "Experiments" +description: "Compare evaluation runs on one leaderboard with NeMo Experiments — cost, latency, token, and evaluator rollups computed at read time from NeMo Intake telemetry." +--- + + + +NeMo Experiments is the comparison layer for agent optimization tasks, such as evaluating the impact +of changes to a harness, infrastructure, tools, or agent code. It gives you one place to visualize and +compare evaluation results from the runner of your choice — Harbor, the NeMo Optimizer, NeMo +Evaluator, or your own — ranked on the metrics that matter, such as cost, latency, and evaluator +scores. + +Those metrics are derived from [NeMo Intake](/documentation/agents/observe-agents) observability data, +the same traces and evaluator results your runs already produce, so any runner that lands telemetry in +Intake feeds the same comparison. + +## Before You Start + +### Who This Is For + +NeMo Experiments is for anyone running evaluations on their agent who needs a way to compare, analyze, +and promote the results. For example: engineers iterating on prompts, models, tools, or routing; teams +benchmarking many models or configurations at once; and reviewers who need a shared leaderboard to +decide what to promote. + +### Requirements + +- A NeMo Platform environment with the `intake` service running. +- The platform entity store (Postgres). +- A reachable ClickHouse database. +- A producer that creates Evaluations and sends their telemetry: the NeMo Optimizer, a benchmark or + evaluation framework such as Harbor, or a direct API integration. +- (Recommended) [NeMo Studio](/documentation/studio) for the full UI — trace comparison views, a + customizable leaderboard, and Pareto charts. Everything is also available through the API, but + Studio is where the experience really lives. The Experiments UI is gated by the + `VITE_FF_EXPERIMENT` feature flag, which is off by default. + +## How It Works + +### What It Does + +Experiments computes comparable, always-current metrics for your evaluation runs. For each run it +rolls up cost, latency, tokens, and per-evaluator scores from the underlying telemetry in ClickHouse +and returns them as ranked rows. Rollups are computed at read time, so a leaderboard always reflects +current telemetry, with no denormalized score table to maintain. + +You group runs however makes sense — an insight to investigate, a model bake-off, a benchmark +leaderboard, or the top runs promoted from several groups — and a single run can belong to more than +one group. Sorting, filtering, and pinning within a group surface the runs that matter. + +### When to Use It + +Use Experiments when you need to: + +- **Compare candidates** from the optimizer, a prompt or model change, or a routing strategy against a + baseline. +- **Run a benchmark leaderboard**: import many runs into one Experiment and rank them by the metric + that matters. +- **Track iteration over time**: keep one Experiment per project so successive attempts stay side by + side. +- **Standardize comparison across a team**: everyone reads the same metrics, in the same place, with + the same vocabulary. + +Experiments are most useful once real evaluation telemetry exists. A new group can start empty and +fill in as runs land. + +### Core Concepts and Data Model + +Experiments sit on top of Intake's telemetry hierarchy (span → trace → session): + +- An **Experiment** is a named container of Evaluations. It holds durable metadata (`description`, + `summary`, free-form `metadata`, optional `insight_id`) and view configuration: a `default_sort` and + a `pareto` (default X/Y metrics for the Pareto view). +- An **Evaluation** is one run and one leaderboard row. It records producer-supplied fields + (`dataset_name`, `dataset_version`, `source_link`, `metadata`, `description`, `status`, + `root_cause`), the `experiment_ids` it belongs to (at least one — an Evaluation can live in more + than one Experiment), and an optional `parent_evaluation_id` linking a variant back to the run it + was derived from. +- A **Session** is one test-case execution within an Evaluation, a single ingested run made of spans. + Sessions carry per-case status, latency, token and cost totals, and evaluator scores. + +At read time, each Evaluation is enriched with rollups derived from its sessions in ClickHouse: + +| Rollup | Meaning | +|--------|---------| +| `test_case_count` | Number of distinct test cases (distinct non-empty `test_case_id` values). Sessions with no `test_case_id` don't count toward it or the rollups. | +| `cost_usd` | Cost aggregate across the Evaluation's sessions. | +| `latency_ms` | Latency aggregate across the Evaluation's sessions. | +| `tokens` | Average total tokens (input + output) per test case. | +| `evaluators.` | Aggregate of a named evaluator's session scores. | +| `model_names`, `agent_names`, `agent_versions` | Distinct models, agents, and versions observed in the telemetry. | + +Metric aggregates expose these statistics: `sum`, `mean`, `median`, `p90`, `p95`, `p99`, `count`. A +**metric path** is therefore `test_case_count`, `cost_usd.`, `latency_ms.`, +`tokens.`, or `evaluators..` — the same grammar used for sorting and filtering below. + +Two things about evaluator rollups specifically: + +- **Response shape.** `evaluators..` is the sort/filter *query* grammar. In the Evaluation + JSON the same scores come back under `aggregate_scores` — a map keyed by evaluator name, each value + carrying the stats above — alongside an `evaluator_names` list. There is no `evaluators` field in + the response; `evaluators..` (query) reads `aggregate_scores[].` (response). +- **Missing-value semantics.** Evaluator rollups are test-case-weighted: each stat is computed over + `test_case_count`, the full set of test cases, with a test case that didn't report a given evaluator + counted as `0` rather than dropped. + +Two Experiment-level behaviors are worth knowing: + +- **Default sort.** An Experiment stores a `default_sort` (a sort-param string such as + `-evaluators.solved.mean`) so its leaderboard opens ordered by the metric the team cares about. It + defaults to `-created_at` (newest first). +- **Pinning.** Any Evaluation can be pinned to the top of its Experiment — for example, the current + baseline. Pins are workspace-shared: everyone with access sees the same pinned set, regardless of + the active sort. + +## Get Started + +### Setup + +Experiments are part of the `intake` service, so any running NeMo Platform already serves them. Point +at whatever you have, whether a deployed platform or a local one from `nemo setup` or +`nemo quickstart up`: + +```shell +export NMP_BASE_URL=http://127.0.0.1:8080 +export WORKSPACE=default +``` + +ClickHouse must be reachable either way, since the leaderboard rollups are computed from it at read +time. + + + +If you are bringing the pieces up yourself from a repository checkout, start them in this order. + +1. **ClickHouse** (required for the rollups): + + ```shell + services/intake/scripts/spans/run_clickhouse.sh + ``` + +1. **Backend services**: `intake` plus its `auth` and `entities` dependencies. `--port` defaults to + `8080`; drop `uv run` if you installed the `nemo` CLI: + + ```shell + uv run nemo services run --services auth,entities,intake --host 127.0.0.1 --port 8080 + ``` + +1. **Studio** (optional, for the leaderboard and drill-down UI). From the `web/` workspace, with the + Experiments feature flag on and intake enabled for trace drill-down: + + ```shell + VITE_FF_EXPERIMENT=true VITE_FF_INTAKE_ENABLED=true VITE_PLATFORM_BASE_URL=http://127.0.0.1:8080 \ + pnpm --filter nemo-studio-ui start -- --host 127.0.0.1 + ``` + + + +Confirm the Experiments read path is reachable: + +```shell +curl -i "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments?page=1&page_size=1" +``` + +A `200` with a (possibly empty) list is healthy. A `503` means the service is running but cannot reach +ClickHouse; reads that need metric rollups will fail until it recovers. + +### First Workflow + +Create an Experiment, add an Evaluation, send it telemetry, then see it in Studio and through the API. + +**1. Create the Experiment** (the leaderboard container): + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "reranker-prompt-iteration", + "description": "Iterating on the support-bench RAG agent'\''s reranker and system prompt." + }' +``` + +Capture its `id` for the next step: + +```shell +export EXPERIMENT_ID=$(curl -sf \ + "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ + | python3 -c 'import sys,json; print(json.load(sys.stdin)["id"])') +echo "experiment id: $EXPERIMENT_ID" +``` + +**2. Add an Evaluation** to it. `experiment_ids` is a list, so an Evaluation belongs to one or more +existing Experiments: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "reranker-add-cross-encoder", + "experiment_ids": ["'"$EXPERIMENT_ID"'"], + "dataset_name": "support-bench", + "dataset_version": "v3", + "metadata": { "reranker": "cross-encoder" } + }' +``` + +An Evaluation is a durable record on its own; its leaderboard metrics appear once evaluation telemetry +for it lands in Intake. + +**3. Send it telemetry.** Send the run's telemetry to Intake just as you would for any agent run, +whether it comes from the optimizer, a benchmark framework like Harbor, or your own code. Tag each +session with the Evaluation's identity: + +- For ATIF and chat-completions, add a **top-level** `evaluation_context` object to the ingest payload + carrying `evaluation_id` (the Evaluation's **name**) and `test_case_id`. +- For OTLP, set the `nemo.experiment.id` and `nemo.test_case.id` root-span attributes. + +The per-evaluator scores on the leaderboard come from **evaluator results** captured on those +sessions, either automatically from ATIF verifier rewards or explicitly through the evaluator-results +endpoint. See +[Capture Evaluator Results](/documentation/agents/observe-agents#capture-evaluator-results) for both +paths, and [Observe Agents](/documentation/agents/observe-agents) for the ingestion paths themselves. +The `nemo-experiments-upload` skill walks this through end to end. + + + +**`test_case_id` is required for a populated leaderboard.** A session tagged with only `evaluation_id` +still ingests and appears in the Evaluation's session list, but it doesn't count toward +`test_case_count` or any rollup — so the row reads as all zeros, with tokens, model, and agent blank +too. Always send `test_case_id` alongside `evaluation_id`. + + + +### Confirm It Worked + +List the Experiment's Evaluations and confirm the new row is present: + +```shell +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&page=1&page_size=20" +``` + +Then open Studio and navigate to `/workspaces/default/experiment`, then open +**reranker-prompt-iteration**. + +**What good looks like:** the Experiment opens in Studio, your Evaluation appears as a row, and once +its sessions are ingested the row shows non-zero `test_case_count`, cost, latency, tokens, and +evaluator scores. Sorting by a metric reorders the table, and opening the Evaluation lists its +individual test cases. + +## Common Workflows + +### Add an Evaluation to an Existing Experiment + +This is the everyday operation: an Experiment already exists and you want to record another run in it. +Create the Evaluation with the Experiment's `id` in `experiment_ids`: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "reranker-bge-large", + "experiment_ids": ["'"$EXPERIMENT_ID"'"], + "dataset_name": "support-bench", + "dataset_version": "v3", + "metadata": { "reranker": "bge-large" } + }' +``` + +To **move or re-scope** an existing Evaluation's membership, `PATCH` its `experiment_ids`, which must +stay non-empty: + +```shell +curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ + -H "Content-Type: application/json" \ + -d '{ "experiment_ids": ["'"$EXPERIMENT_ID"'", "'"$OTHER_EXPERIMENT_ID"'"] }' +``` + +An Evaluation can belong to several Experiments at once, which is useful when the same run should +appear on both a per-project board and a cross-project benchmark. + +### Update an Evaluation + +Use `PATCH` for partial updates: only the fields you send change. `name`, `dataset_name`, and +`dataset_version` are immutable: + +```shell +curl -X PATCH "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" \ + -H "Content-Type: application/json" \ + -d '{ "status": "winner", "root_cause": "Best groundedness at acceptable cost." }' +``` + +`PUT` does a full replace of the mutable fields — omitted fields reset — so prefer `PATCH` for one-off +edits. + +### Rank an Experiment by the Metric That Matters + +List an Experiment's Evaluations sorted by a metric. Prefix the field with `-` for descending: + +```shell +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&sort=-evaluators.solved.mean&page=1&page_size=20" +``` + +You can sort by an entity column (`name`, `created_at`) or any rollup metric (`test_case_count`, +`cost_usd.`, `latency_ms.`, `tokens.`, `evaluators..`). In Studio, click +a column header. If the Experiment has a `default_sort`, the leaderboard opens already ordered that +way. + +### Filter to the Evaluations You Care About + +Filter by a metric range to narrow the leaderboard — for example, only Evaluations whose average cost +is under $0.50: + +```shell +curl -g "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations?filter[experiment_id]=$EXPERIMENT_ID&filter[cost_usd.mean][\$lte]=0.5" +``` + +Metric filters use the same grammar as sort, such as `filter[test_case_count][\$gte]=5` or +`filter[evaluators.groundedness.mean][\$gte]=0.8`. You can also filter by +`filter[metadata.]=`, `filter[status]=…`, `filter[is_pinned]=true`, and +`created_at`/`updated_at` ranges. In Studio, use the column filters on cost, latency, test-case count, +and evaluator columns. + +### Set an Experiment's Default Sort + +Store the ordering the team should see first so no one has to re-sort each visit: + +```shell +curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ + -H "Content-Type: application/json" \ + -d '{ "name": "reranker-prompt-iteration", "default_sort": "-evaluators.solved.mean" }' +``` + +In Studio, set it from the Experiment's Edit dialog. + +### Pin a Baseline to the Top + +Keep the current baseline, or any reference run, at the top of the Experiment for everyone: + +```shell +curl -X POST "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" + +# Unpin: +curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-main-baseline/pin" +``` + +Pinned Evaluations float to the top of the leaderboard regardless of the active sort. List just the +pinned set with `filter[is_pinned]=true`. + +### Drill into an Evaluation's Test Cases + +Open one Evaluation's sessions to see per-test-case behavior: status, latency, cost, and evaluator +scores: + +```shell +curl "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-add-cross-encoder/sessions?page=1&page_size=20" +``` + +Use this to move from "this candidate scores lower" to "these specific cases regressed." In Studio, +open the Evaluation row to reach its sessions, then follow a session into its trace in Intake. + +### Configure the Pareto View + +Each Experiment stores a `pareto` config: the default X/Y metrics for its Pareto (trade-off) chart in +Studio, defaulting to cost vs. latency. The axes are `x_metric` and `y_metric`, and each takes a base +metric id — `cost_usd`, `latency_ms`, or `evaluators.`, with no `.` suffix. Set it on +create or update: + +```shell +curl -X PUT "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" \ + -H "Content-Type: application/json" \ + -d '{ "name": "reranker-prompt-iteration", + "pareto": { "x_metric": "cost_usd", "y_metric": "evaluators.solved" } }' +``` + +### Delete an Experiment or Evaluation + +Deleting an Experiment soft-deletes it and cascades to its Evaluations; deleting an Evaluation +soft-deletes just that row. Soft-deleted records are hidden from list and get operations unless +explicitly requested with `filter[is_deleted]=true`: + +```shell +curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/evaluations/reranker-bge-large" +curl -X DELETE "$NMP_BASE_URL/apis/intake/v2/workspaces/$WORKSPACE/experiments/reranker-prompt-iteration" +``` + +### Where Evaluations Come From + +You usually don't create Evaluations by hand. Common producers: + +- **Optimizer**: records each run as an Experiment and its candidates as Evaluations, so results land + automatically. See [Optimize Agents](/documentation/agents/optimize-agents). +- **Evaluation framework**: a framework like Harbor sends complete trajectories through Intake's ATIF + ingest, and each run becomes an Evaluation with its final metrics as evaluator results. See + [Observe Agents](/documentation/agents/observe-agents). +- **Direct API**: create the Experiment and Evaluations, then ingest their sessions, for a custom + evaluation pipeline. + +### Do It with an Agent + +If you are working with a coding agent, the `nemo-experiments-upload` skill automates this whole path: +create an Experiment, add an Evaluation, log traces and scores to an ingest endpoint, and verify the +rollups. It ships with reference guides for the ATIF/Harbor, chat-completions, and OTLP ingest formats +and a troubleshooting matrix. Invoke it when you want to upload, log, ingest, or publish evaluation +runs to NeMo Experiments. + +## Operations + +### Limits + +- An Experiment is sorted and filtered in memory over the full set of its Evaluations, bounded to + **1,000 Evaluations** per Experiment. A request that would select more is rejected with `413` rather + than returned partially sorted. Narrow it with filters. +- Page size is capped at 1,000 records. +- Metric sorting and filtering require rollups. If ClickHouse is unavailable, a metric sort or filter + returns `503` rather than a silently unsorted result. Sorting and filtering by entity columns + (`name`, `created_at`) still works. + +### Security and Access Control + +All Experiments endpoints are workspace-scoped under `/apis/intake/v2/workspaces/{workspace}/`, and +the service checks workspace access before reading or writing. + + + +Evaluation `metadata`, `source_link`, and related fields are producer-supplied and surfaced to +reviewers, so treat them like any other telemetry payload. Do not store secrets, credentials, or +regulated data unless your deployment's policy permits it. + + + +### Retention and Storage + +Experiment and Evaluation metadata lives in the platform entity store and persists until you delete +it; it does not expire on the telemetry TTL. Rollups are derived from ClickHouse at read time, so +leaderboard metrics reflect whatever telemetry is still retained: span and trace-index data expire +after 90 days, while evaluator results are retained without that span TTL. An Experiment older than +the span window keeps its records and evaluator-based scores, but cost, latency, and run-count rollups +reflect only the retained telemetry window. + +### Troubleshooting + +| Symptom | Cause and fix | +|---------|---------------| +| **Rows show zero metrics** | Three causes: no sessions have been ingested for that Evaluation yet; the sessions were ingested without `test_case_id`, so they don't count toward `test_case_count` or the rollups; or ClickHouse is unreachable. Confirm ingestion in Intake, that sessions carry `test_case_id`, and that the read path returns `200` rather than `503`. | +| **A metric sort or filter returns `503`** | Rollups can't be computed because ClickHouse is down. Retry once the read path is healthy, or fall back to an entity-column sort. | +| **A list returns `413`** | The Experiment selected more than 1,000 Evaluations for an in-memory sort. Add filters to narrow the set. | +| **An Evaluation isn't in the Experiment** | Confirm you created it with the correct Experiment `id` in `experiment_ids` and are querying the right workspace (`filter[experiment_id]=`). | +| **`422` on create Evaluation** | An Evaluation must belong to at least one Experiment: provide `experiment_ids`. Required fields are `name`, `experiment_ids`, and `dataset_name`; `metadata` values must be strings. | + +## Related Topics + +- [Observe Agents](/documentation/agents/observe-agents): the telemetry these leaderboards are + computed from, and how to ingest it. +- [Agent Evaluation](/documentation/evaluate-models/agent-eval): the task-driven evaluation model that + produces these runs. +- [Optimize Agents](/documentation/agents/optimize-agents): the optimizer that creates Experiments and + Evaluations automatically. diff --git a/docs/evaluator/index.mdx b/docs/evaluator/index.mdx index 10fa88d258..ab124840ad 100644 --- a/docs/evaluator/index.mdx +++ b/docs/evaluator/index.mdx @@ -203,6 +203,8 @@ Most teams get the best results by starting metric-first and scaling up: custom metrics, dataset sources, and how scores aggregate. - **Evaluate agents:** [Agent Evaluation](/documentation/evaluate-models/agent-eval) — the task-driven model, targets, and runners. +- **Compare runs:** [Experiments](/documentation/evaluate-models/experiments) — roll evaluation runs up + into one leaderboard ranked by cost, latency, and evaluator scores. - **Run it in code:** [SDK Resources](/documentation/evaluate-models/sdk-resources) — the `run()` and `submit()` calling surface. - **Full API:** [Evaluator API Reference](/documentation/reference/api-reference). diff --git a/docs/fern/versions/latest.yml b/docs/fern/versions/latest.yml index 888de0b3ce..4be250e722 100644 --- a/docs/fern/versions/latest.yml +++ b/docs/fern/versions/latest.yml @@ -261,6 +261,8 @@ navigation: contents: - page: Deploy Agents path: ../../agents/deploy-agents.mdx + - page: Observe Agents + path: ../../agents/observability.mdx - page: Optimize Agents path: ../../agents/optimization.mdx - page: Secure Agents @@ -403,6 +405,9 @@ navigation: path: ../../evaluator/tutorials/run-llm-judge-evaluation.mdx - page: Define and Run Custom Python Metrics path: ../../evaluator/tutorials/define-run-custom-python-metrics.mdx + - page: Experiments + slug: experiments + path: ../../evaluator/experiments.mdx - page: SDK Resources path: ../../evaluator/sdk-resources.mdx - page: Manage Tasks & Tasksets