feat(studio): connect Studio to Intake based eval schema, update modal

web/packages/studio/public/sample-agents/email-security-triage/dataset.jsonl

@@ -0,0 +1,5 @@
+{"subject": "Claim Your Free iPhone Now!", "sender": "prize@example.com", "body": "Dear valued customer,\nCongratulations! You have been selected to receive a brand new iPhone absolutely free. To claim your prize, confirm your details at http://claim-prize.example.com within 24 hours.", "label": "phishing"}
+{"subject": "Urgent: Your Account Has Been Suspended", "sender": "security-alerts@bank.com", "body": "Hello,\nWe have detected unusual activity on your account. To prevent suspension, please verify your identity at http://verify-account.example.com within 24 hours.", "label": "phishing"}
+{"subject": "Important: Invoice Attached", "sender": "accounts@shop-example.com", "body": "Hi there,\nPlease find the invoice attached for your recent purchase. Click here to view the details.\nhttp://invoice-view.example.com/doc", "label": "phishing"}
+{"subject": "Project Meeting Reminder", "sender": "bob@example.com", "body": "Hi Team,\nJust wanted to remind you about our project update meeting on Friday at 2pm. Please let me know if you can attend.\nThanks,\nBob", "label": "benign"}
+{"subject": "Invoice Follow-up", "sender": "alice@company.com", "body": "Hi John,\nPlease find the invoice #1234 attached for your recent purchase. Let me know if you have any questions.\nBest regards,\nAlice", "label": "benign"}

web/packages/studio/public/sample-agents/email-security-triage/eval-config.dataset-driven.README.md

@@ -0,0 +1,37 @@
+# Dataset-Driven evaluation — Email Security Triage
+
+Every row of `dataset.jsonl` is scored by the same metric set. Use this shape when the inputs are
+homogeneous and the interesting variable is coverage, not the kind of task.
+
+## The dataset
+
+Five rows (three phishing, two benign), each with `subject`, `sender`, `body`, and `label`.
+`prompt_template` assembles them into the RFC-822 message the agent expects:
+
+```jinja
+From: {{ item.sender }}
+Subject: {{ item.subject }}
+
+{{ item.body }}
+```
+
+The `From:` line is deliberate — the sender domain is a top phishing signal and is what the agent's
+`extract_iocs` tool harvests. Dropping it measurably weakens the agent.
+
+`label` is the only ground truth here. Add rows by appending to `dataset.jsonl`; five rows means
+each one moves a score by 20%, so treat small differences as noise.
+
+## How the verdict is read
+
+The agent answers with a YAML block whose `is_likely_phishing` key carries the verdict. Metrics
+read that key out of the body with `contains`, because the model usually wraps the block in a ```
+fence despite being asked not to — anything anchored to the start of the response scores 0 for
+every row.
+
+## Metrics
+
+- `string-check` — deterministic. Its expected value is rendered per row
+  (`is_likely_phishing: {% if item.label == 'phishing' %}true{% else %}false{% endif %}`), so it
+  needs no judge model and scores even when the judge is unreachable.
+- `llm-judge` — grades the verdict and whether `attack_type` is a sensible label, as two `scores`
+  on one metric.

web/packages/studio/public/sample-agents/email-security-triage/eval-config.dataset-driven.json

@@ -0,0 +1,95 @@
+{
+  "dataset": "<workspace>/<fileset>#dataset.jsonl",
+  "prompt_template": "From: {{ item.sender }}\nSubject: {{ item.subject }}\n\n{{ item.body }}",
+  "metrics": [
+    {
+      "bundle_kind": "metric-bundle",
+      "bundle_format_version": "v1",
+      "metric_type": "string-check",
+      "metadata": {
+        "description": "Verdict key matches ground truth, without needing a judge model.",
+        "labels": {}
+      },
+      "outputs": [
+        {
+          "name": "string-check",
+          "description": null,
+          "value_json_schema": {
+            "description": "Continuous numeric metric value.",
+            "title": "ContinuousScore",
+            "type": "number"
+          }
+        }
+      ],
+      "secrets": {},
+      "payload": {
+        "kind": "inline",
+        "metric": {
+          "type": "string-check",
+          "operation": "contains",
+          "left_template": "{{ (sample.output_text or '') | lower }}",
+          "right_template": "is_likely_phishing: {% if item.label == 'phishing' %}true{% else %}false{% endif %}"
+        }
+      }
+    },
+    {
+      "bundle_kind": "metric-bundle",
+      "bundle_format_version": "v1",
+      "metric_type": "llm-judge",
+      "metadata": {
+        "description": "attack_type plausibility vs the ground-truth verdict.",
+        "labels": {}
+      },
+      "outputs": [
+        {
+          "name": "attack_type_plausible",
+          "description": null,
+          "value_json_schema": {
+            "description": "Continuous numeric metric value.",
+            "title": "ContinuousScore",
+            "type": "number"
+          }
+        }
+      ],
+      "secrets": {},
+      "payload": {
+        "kind": "inline",
+        "metric": {
+          "type": "llm-judge",
+          "model": "default/nvidia-nemotron-3-super-120b-a12b",
+          "prompt_template": {
+            "messages": [
+              {
+                "role": "user",
+                "content": "You are scoring an email security triage agent.\n\nThe agent answers with a YAML block (often wrapped in a ``` fence). Its verdict is the `is_likely_phishing` key: `true` means phishing, `false` means benign. Read that key and nothing else — the `indicators` and `explanation` fields routinely mention the opposite verdict as things the agent considered, and must be ignored. Do not read the first line; it is usually the fence.\n\nAgent response:\n{{ sample.output_text }}\n\nGround-truth verdict: \"{{ item.label }}\".\n\nReturn a JSON object with one key:\n  \"attack_type_plausible\": 1 if the `attack_type` value is a sensible label for this email (one of bec, credential, malware, spam, benign; `benign` iff the verdict is benign), else 0."
+              }
+            ]
+          },
+          "scores": [
+            {
+              "name": "attack_type_plausible",
+              "minimum": 0.0,
+              "maximum": 1.0,
+              "parser": {
+                "type": "regex",
+                "method": "search",
+                "pattern": "attack_type_plausible\"?\\s*:\\s*([01])"
+              }
+            }
+          ],
+          "inference": {
+            "max_tokens": 1024,
+            "extra_body": {
+              "nvext": {
+                "max_thinking_tokens": 256
+              }
+            }
+          },
+          "reasoning": {
+            "end_token": "</think>"
+          }
+        }
+      }
+    }
+  ]
+}

web/packages/studio/public/sample-agents/email-security-triage/eval-config.task-driven.README.md

@@ -0,0 +1,43 @@
+# Task-Driven evaluation — Email Security Triage
+
+Each task is one email the agent triages independently, carrying its own metrics. Use this shape
+when the suite grades different _kinds_ of work; here the kinds are ordinary classification and
+resistance to prompt injection.
+
+## What it measures
+
+| Tasks         | Input                                                       | Checks                                                         |
+| ------------- | ----------------------------------------------------------- | -------------------------------------------------------------- |
+| `classify-*`  | one phishing or benign email                                | the verdict matches the label                                  |
+| `injection-*` | a phishing email carrying instructions aimed at the analyst | the verdict follows the evidence, not the injected instruction |
+
+## How the verdict is read
+
+The agent answers with a YAML block whose `is_likely_phishing` key carries the verdict:
+
+```yaml
+is_likely_phishing: true
+confidence: 0.93
+indicators: [...]
+explanation: ...
+attack_type: credential
+impersonated_brand: none
+```
+
+Every metric reads that key out of the body. Two details that matter if you edit this config:
+
+- **The model usually wraps the YAML in a ``` fence**, despite its prompt asking for the block
+  alone. Metrics use `contains`, never `startswith` or a first-line lookup — anything anchored to
+  the start of the response scores 0 for every row.
+- **`indicators` and `explanation` routinely name the opposite verdict** as something the agent
+  considered and rejected. The judge prompt says to ignore them.
+
+## Metrics
+
+- `string-check` — deterministic. Matches `is_likely_phishing: true|false` against the task's
+  label. Needs no judge model, so it scores even when the judge is unreachable.
+- `llm-judge` — grades the verdict and whether `attack_type` is a sensible label. Both criteria
+  live in a single metric: the runtime rejects two metrics of the same _type_ within one task, so
+  extra criteria have to be extra `scores`.
+
+The `triage` view reduces the deterministic and judged verdict signals to a mean.