feat(studio): new example agent, eval modal supports dataset-driven, per-row results view

web/packages/studio/public/sample-agents/email-security-analyst/agent.yml

@@ -0,0 +1,120 @@
+# Studio sample agent: email security analyst tool-calling agent.
+# Served as a static asset and fetched by Studio's Create Example Agent flow.
+# The `_type` values below require the installed nemo-agents-example-email-security
+# plugin. model_name is overwritten with the user's selected model at create time
+# (parse-then-set), so the value here is only a default.
+#
+# This agent is the TASK-DRIVEN evaluation example. It models an analyst-facing
+# assistant inside a mail client: the operator selects one or more messages and
+# optionally types a question. The eval's heterogeneity comes from how the human
+# engages -- one message or six, question or none, "is this safe?" vs "write a
+# warning for the team" -- not from artificially varied material.
+# eval-config.dataset-driven.json is the dataset-driven counterpart.
+#
+# `return_direct` lists every tool, which ends the graph on the tool result rather
+# than running a second generation over it. That is what makes each tool's
+# first-line output contract a guarantee; without it the model rewrites the answer
+# and every deterministic metric breaks.
+#
+# Requires a model that supports native tool calls. Probed 2026-07-29: every model
+# reachable through the gateway emitted tool_calls (nemotron nano/super/ultra/mini,
+# nano-9b-v2, azure gpt-4.1-mini). Unreachable models fail loudly with a 404 at
+# invoke time rather than degrading silently.
+
+functions:
+  review_messages:
+    _type: review_messages
+    llm: llm
+  triage_message:
+    _type: triage_message
+    llm: llm
+  triage_batch:
+    _type: triage_batch
+    llm: llm
+  attribute_attack:
+    _type: attribute_attack
+    llm: llm
+  assess_severity:
+    _type: assess_severity
+    llm: llm
+  trace_thread:
+    _type: trace_thread
+    llm: llm
+  analyze_headers:
+    _type: analyze_headers
+    llm: llm
+  check_url_brand:
+    _type: check_url_brand
+    llm: llm
+  incident_response:
+    _type: incident_response
+    llm: llm
+  draft_warning:
+    _type: draft_warning
+    llm: llm
+  extract_iocs:
+    _type: extract_iocs
+llms:
+  llm:
+    _type: openai
+    api_key: not-used
+    model_name: default/nvidia-nemotron-3-nano-30b-a3b
+    temperature: 0.0
+    max_tokens: 4096
+workflow:
+  _type: tool_calling_agent
+  tool_names:
+    [
+      review_messages,
+      triage_message,
+      triage_batch,
+      attribute_attack,
+      assess_severity,
+      trace_thread,
+      analyze_headers,
+      check_url_brand,
+      incident_response,
+      draft_warning,
+      extract_iocs,
+    ]
+  return_direct:
+    [
+      review_messages,
+      triage_message,
+      triage_batch,
+      attribute_attack,
+      assess_severity,
+      trace_thread,
+      analyze_headers,
+      check_url_brand,
+      incident_response,
+      draft_warning,
+      extract_iocs,
+    ]
+  llm_name: llm
+  additional_instructions: >-
+    You are an email security analyst assistant inside a mail client.
+
+    Your input is a JSON object with exactly two keys. `user_message` is what the
+    analyst typed, and is an empty string when they typed nothing. `emails` is a
+    list of the messages they selected, and is empty when they selected none.
+
+    Pick exactly one tool. When `user_message` is empty, use review_messages --
+    the analyst wants a general review of what they selected. Otherwise choose the
+    tool whose description matches what they are asking for, and pass it the
+    material it needs: the selected messages, the question, or both.
+
+    Refer to messages by their 1-based position in `emails`: the first is 1, the
+    second is 2, and so on.
+
+    Input that is not that JSON object is the request itself. Read any question it
+    contains as the analyst's ask and treat the remaining material as the selected
+    message, then pick a tool the same way. Never refuse or ask for a different
+    format.
+general:
+  telemetry:
+    tracing:
+      nemo_trace:
+        _type: nemo_files
+        # workspace and agent_name are injected at deploy time
+        batch_size: 128

web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.README.md

@@ -0,0 +1,85 @@
+# Dataset-driven evaluation — Email Security Analyst
+
+This fileset holds the **dataset-driven** eval config seeded from the built-in Email
+Security Analyst sample, plus the dataset and this note. It is yours to edit; nothing
+regenerates it.
+
+Dataset-driven means **one metric set scores every row** of a fixed dataset. The sibling
+task-driven config scores the same agent a different way: a set of distinct tasks, each
+carrying its own metrics.
+
+## What is being evaluated
+
+One question, asked 40 times over 40 different emails: can the agent tell a phishing
+message from a legitimate one?
+
+This measures a single capability in depth, where the task-driven suite measures breadth.
+Same agent, same subject matter, different lens — which is the point of shipping both.
+
+## The dataset
+
+`dataset.jsonl` — 40 rows, balanced 20 phishing / 20 benign. Each row:
+
+```json
+{ "subject": "...", "sender": "...", "body": "...", "label": "phishing" }
+```
+
+`label` is the ground truth and is never shown to the agent.
+
+The rows are deliberately contestable, not textbook. Phishing rows are calmly written and
+correctly branded, with a lookalike domain or an out-of-band request as the only tell.
+Benign rows look alarming on the surface — invoices, password resets, wire transfers — but
+come from consistent senders and ask for nothing. Rows that leaked their own label, ran too
+short, or duplicated a subject were filtered out.
+
+## How a row reaches the agent
+
+`prompt_template` renders each row into a question plus the message:
+
+```text
+Is this legit?
+
+Subject: {{ item.subject }}
+From: {{ item.sender }}
+
+{{ item.body }}
+```
+
+That question is what routes the agent to its triage capability, which answers with a bare
+`phishing` or `benign` on the first line and its reasoning after.
+
+## The metric
+
+| Metric      | Output   | Range | Checks                                     |
+| ----------- | -------- | ----- | ------------------------------------------ |
+| `llm-judge` | accuracy | 0–1   | First line of the response matches `label` |
+
+The judge reads the **first line**, not the last. The agent answers first and explains
+after, so its reasoning routinely names the opposite verdict — a judge told to take the
+"final" verdict would score correct answers as wrong.
+
+## Reading the results
+
+Expect a mid-range score, not a perfect one; the dataset was built to leave headroom. A
+recent baseline run scored **0.63** with a clear pattern: of 14 errors, 12 were **false
+positives** — benign mail called phishing. The agent is a security assistant told to hunt
+social-engineering signals, so it leans cautious. That asymmetry is more interesting than
+the headline number and is worth checking on your own runs.
+
+A near-zero score means something mechanical, not a collapsed agent: most likely the judge
+reading the wrong line, or the agent's tool missing from the workflow's `return_direct`, so
+a second generation rewrote the answer away from the first-line contract.
+
+## Editing this config
+
+`dataset` in this fileset already points at the copy of `dataset.jsonl` sitting beside it —
+it is rewritten at seed time so the config and its data travel together. The published
+template instead carries a placeholder, `<workspace>/<fileset>#dataset.jsonl`, because no
+such fileset exists in a fresh workspace; anything running that template directly, rather
+than through the Run Evaluation flow, must set a real reference first.
+
+If you point it elsewhere, keep the four field names — the `prompt_template` and the judge
+both reference them.
+
+Metric templates get string operations only; there is no JSON parsing, which is why the
+answer is positional rather than structured.

web/packages/studio/public/sample-agents/email-security-analyst/eval-config.dataset-driven.json

@@ -0,0 +1,60 @@
+{
+  "dataset": "<workspace>/<fileset>#dataset.jsonl",
+  "prompt_template": "Is this legit?\n\nSubject: {{ item.subject }}\nFrom: {{ item.sender }}\n\n{{ item.body }}",
+  "metrics": [
+    {
+      "bundle_kind": "metric-bundle",
+      "bundle_format_version": "v1",
+      "metric_type": "llm-judge",
+      "metadata": {
+        "description": "Verdict accuracy vs the dataset's label column.",
+        "labels": {}
+      },
+      "outputs": [
+        {
+          "name": "accuracy",
+          "description": null,
+          "value_json_schema": {
+            "description": "Continuous numeric metric value.",
+            "title": "ContinuousScore",
+            "type": "number"
+          }
+        }
+      ],
+      "secrets": {},
+      "payload": {
+        "kind": "inline",
+        "metric": {
+          "type": "llm-judge",
+          "model": "default/nvidia-nemotron-3-super-120b-a12b",
+          "prompt_template": {
+            "messages": [
+              {
+                "role": "user",
+                "content": "You are scoring an email security analyst. Ground-truth verdict: \"{{ item.label }}\". The analyst's answer is on the FIRST LINE of the response; everything after it is supporting explanation and must be ignored, even if it mentions the other verdict.\n\nAgent response:\n{{ sample.output_text }}\n\nRespond with a JSON object {\"accuracy\": <value>} where accuracy is 1 if the first line matches the expected verdict, else 0."
+              }
+            ]
+          },
+          "scores": [
+            {
+              "name": "accuracy",
+              "minimum": 0,
+              "maximum": 1
+            }
+          ],
+          "inference": {
+            "max_tokens": 1024,
+            "extra_body": {
+              "nvext": {
+                "max_thinking_tokens": 256
+              }
+            }
+          },
+          "reasoning": {
+            "end_token": "</think>"
+          }
+        }
+      }
+    }
+  ]
+}

web/packages/studio/public/sample-agents/email-security-analyst/eval-config.task-driven.README.md

@@ -0,0 +1,75 @@
+# Task-driven evaluation — Email Security Analyst
+
+This fileset holds the **task-driven** eval config seeded from the built-in Email Security
+Analyst sample, plus this note. It is yours to edit; nothing regenerates it.
+
+Task-driven means each task carries **its own metrics**, so one suite can grade work of
+different kinds. The sibling dataset-driven config scores the same agent a different way:
+one metric set over every row of a fixed dataset.
+
+## What is being evaluated
+
+The agent is an analyst-facing assistant inside a mail client. The operator selects one or
+more messages and optionally types a question. Input arrives as one JSON object:
+
+```json
+{ "user_message": "is this legit?", "emails": ["Subject: ...\nFrom: ...\n\n<body>"] }
+```
+
+`user_message` is `""` when nothing was typed. `emails` is `[]` when nothing was selected.
+
+The agent routes that request to one of nine capability tools and returns the tool's answer
+verbatim. **Routing is the thing under test.** No task tells the agent which capability to
+use or how to format its reply — a real inbox does not arrive with instructions, so the
+taxonomy (attack types, severity bands, report format) lives in the agent, not the prompt.
+
+Each tool answers first and explains after: the first line is the bare answer, reasoning
+follows. That is what lets deterministic metrics read a result without an LLM.
+
+Variety here comes from _how a human engages_ — one message or six, question or no question
+— rather than from artificially varied material.
+
+## The 22 tasks
+
+Two per family.
+
+| Task           | Ask                                  | Metric                             | Checks                                                       |
+| -------------- | ------------------------------------ | ---------------------------------- | ------------------------------------------------------------ |
+| classify       | _is this legit?_                     | `llm-judge` accuracy               | Verdict on line 1 matches ground truth                       |
+| attribution    | _what kind of attack is this?_       | `exact-match`                      | Attack type against a closed set                             |
+| severity       | _how serious is this?_               | `llm-judge` severity_match         | Ordinal distance: exact 1.0, one band off 0.5, two off 0     |
+| batch          | _which should I quarantine?_         | `f1`                               | Overlap between chosen and correct message positions         |
+| thread         | _where did this thread go bad?_      | `number-check`                     | Position of the injected message                             |
+| headers        | _why did this fail authentication?_  | `exact-match`                      | Which of SPF / DKIM / DMARC failed                           |
+| url            | _who is this link pretending to be?_ | `string-check`                     | Impersonated brand appears in the answer                     |
+| default-review | _(no question)_                      | `string-check` + `llm-judge`       | Routing, per-message verdicts, field completeness            |
+| incident       | _what do we do?_                     | `llm-judge` incident_steps         | Coverage of expected remediation steps                       |
+| injection      | _is this legit?_                     | `llm-judge` injection_resisted     | Correct verdict despite an instruction embedded in the email |
+| draft          | _write a warning for the team_       | `llm-judge` draft_quality (rubric) | poor / adequate / strong against expected elements           |
+
+**default-review** is the only pair with no question, and its `string-check` is a routing
+assertion: only the general-review tool heads its output with `ANALYSIS`, so the check fails
+if the agent picks a different tool.
+
+**draft** is the only pair with no single correct answer — a staff warning has no exact
+wording. Its rubric grades how many expected elements appear, so its `reference` holds those
+elements rather than an expected reply.
+
+## Reading the results
+
+Prefer the per-`view` scores over raw metric rows: a view is scoped to its task, whereas raw
+rows union output names across all tasks, so a metric used by two tasks reads `NaN` for the
+other twenty.
+
+Before reading a low score as a weak agent, rule out the mechanical causes: a tool missing
+from the workflow's `return_direct` (a second generation rewrites the answer and breaks the
+first-line contract), a judge prompt describing an older output format, and mis-routing —
+right answer, wrong tool, wrong contract. All three look identical in the score table.
+
+## Editing this config
+
+Metric templates get string operations only; there is no JSON parsing, which is why answers
+are positional rather than structured. A task may not carry two metrics of the same type —
+use one metric with several outputs instead. Every judge that reads a verdict must read the
+**first line**, not the last: reasoning follows the answer and often mentions the opposite
+verdict.