feat(evaluator): taskset references for agent evaluation + Tasks/Tasksets docs

docs/evaluator/manage-tasks-tasksets.mdx

@@ -0,0 +1,246 @@
+---
+title: "Manage Tasks & Tasksets"
+description: ""
+---
+<a id="eval-manage-tasks-tasksets"></a>
+
+A **task** is a stored, reusable definition of an agent-eval unit of work: an intent (what the agent
+should do), the inputs it receives, and the metrics that score it. A **taskset** is a named grouping
+of tasks. Both are first-class entities in the Evaluator plugin, addressed by `workspace/name` and
+managed through the `nemo_platform` SDK.
+
+Use stored tasks and tasksets when you want to define an evaluation unit once and reference it across
+runs, share it across a team, or assemble suites — rather than re-declaring the intent, inputs, and
+metrics inline every time.
+
+## Concepts
+
+| Concept | What it is | Members |
+|---------|------------|---------|
+| **Task** | A reusable agent-eval unit: `intent`, `inputs`, and the `metrics` that score it. | References the metrics that score it. |
+| **Taskset** | A flexible grouping of tasks with a description and metadata. | References member tasks by `workspace/name`. Membership is a **set** — order is not significant and duplicate references are rejected. |
+
+Both are addressed by `workspace/name`. Names are unique within a workspace, limited to 255
+characters, and must match `^[\w\-\.]+$`.
+
+<Note>
+Tasks and tasksets support **create, retrieve, list, and delete** — there is no update. To change a
+stored task or taskset, delete it and create a new one, or store a new version under a different
+name.
+</Note>
+
+## Initialize the SDK
+
+```python
+import os
+
+from nemo_platform import NeMoPlatform
+
+
+client = NeMoPlatform(
+    base_url=os.environ.get("NMP_BASE_URL", "http://localhost:8080"),
+    workspace="default",
+)
+
+tasks = client.evaluator.tasks        # EvaluatorTasksResource
+tasksets = client.evaluator.tasksets  # EvaluatorTasksetsResource
+```
+
+## Manage Tasks
+
+A task scores its output with metrics. Store the metric first, then reference it from the task by
+`workspace/name`. See [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) for the
+metric classes and options.
+
+```python
+from nemo_evaluator_sdk import ExactMatchMetric
+
+# Store a metric the task will reference.
+client.evaluator.metrics.create(
+    "answer-exact-match",
+    metric=ExactMatchMetric(reference="{{item.expected}}", candidate="{{item.output}}"),
+)
+```
+
+Send the task as a `TaskInput` (the authorable subset of a task) and address it by name on create.
+Reference the stored metric with a `MetricRef` (`workspace/name`, or a bare `name` resolved against
+the task's workspace). The service returns the stored `Task`.
+
+```python
+from nemo_evaluator.api.schemas import MetadataItem, MetricRef, TaskInput, TaskInputs
+
+task = TaskInput(
+    intent="Answer the user's geography question with the capital city.",
+    inputs=TaskInputs(instruction="What is the capital of France?"),
+    metrics=[MetricRef("default/answer-exact-match")],
+    metadata=[MetadataItem(key="suite", value="geography")],
+)
+
+stored = tasks.create("capital-of-france", task=task)
+print(stored.id, stored.metrics)
+```
+
+### `TaskInput` fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `intent` | `str` | Yes | Human-readable description of the desired agent behavior. |
+| `inputs` | `TaskInputs` | No | The task's recognized input fields. `instruction` is the agent's prompt; it falls back to `intent` when unset. |
+| `metrics` | `list[MetricRefOrInline]` | No | The metrics that score the task, as `MetricRef` references (`workspace/name`) to stored metrics. Pre-built inline metric bundles (`MetricInline`) are also accepted and are normalized to stored metrics on create. |
+| `views` | `dict[str, SemanticView]` | No | Optional reporting views mapping metric outputs into named semantic scores. |
+| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
+
+<Note>
+A stored task holds **metric references only**. Any inline metric bundle you pass on create is stored
+as a content-addressed *derived* metric, and the task record is normalized to reference it. This is
+why `stored.metrics` always comes back as a list of `MetricRef` references.
+</Note>
+
+### Retrieve, list, and delete
+
+```python
+# Retrieve one task by name
+task = tasks.retrieve("capital-of-france")
+
+# List tasks in the workspace (paginated)
+page = tasks.list(page=1, page_size=100, sort="-created_at")
+for item in page.data:
+    print(item.name, item.intent)
+
+# Delete a task
+tasks.delete("capital-of-france")
+```
+
+`sort` accepts `name`, `created_at`, or `updated_at`, each optionally prefixed with `-` for
+descending order.
+
+## Manage Tasksets
+
+A taskset references existing tasks by `workspace/name`. All referenced tasks must already exist when
+the taskset is created; a missing or duplicate reference is rejected.
+
+```python
+from nemo_evaluator.api.schemas import TaskRef, TasksetInput
+
+taskset = TasksetInput(
+    description="Geography questions for smoke-testing the agent.",
+    tasks=[
+        TaskRef("default/capital-of-france"),
+        TaskRef("default/capital-of-japan"),
+    ],
+)
+
+stored = tasksets.create("geography-suite", taskset=taskset)
+print(stored.tasks)
+```
+
+### `TasksetInput` fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `description` | `str` | No | Human-readable description of the grouping. |
+| `tasks` | `list[TaskRef]` | No | References to member tasks (`workspace/name`, or bare `name` within the same workspace). Set semantics — duplicates rejected. |
+| `metadata` | `list[MetadataItem]` | No | Key/value annotations. Keys must be unique. |
+
+### Retrieve, list, and delete
+
+```python
+taskset = tasksets.retrieve("geography-suite")
+
+page = tasksets.list(page=1, page_size=100, sort="name")
+for item in page.data:
+    print(item.name, len(item.tasks))
+
+tasksets.delete("geography-suite")
+```
+
+Deleting a taskset does not delete its member tasks — a taskset only holds references.
+
+## Run an evaluation over a taskset
+
+An agent evaluation is submitted with an `AgentEvalInputSpec`, whose `tasks` field is either an
+inline list of tasks or a **reference to a stored taskset**. Referencing a taskset lets you keep the
+task definitions in one place and evaluate the whole set by name, instead of inlining every task on
+each run.
+
+```python
+from nemo_evaluator.api.schemas import TasksetRef
+from nemo_evaluator.jobs.agent_spec import AgentEvalInputSpec, ModelTarget
+from nemo_evaluator_sdk.values import Model
+from nemo_evaluator_sdk.enums import ModelFormat
+
+# Instead of inlining AgentEvalTaskInput objects, point `tasks` at a stored taskset.
+input_spec = AgentEvalInputSpec(
+    tasks=TasksetRef("default/geography-suite"),
+    target=ModelTarget(
+        model=Model(url="https://integrate.api.nvidia.com/v1", name="meta/llama-3.3-70b-instruct", format=ModelFormat.OPEN_AI),
+    ),
+)
+```
+
+When the job runs, the taskset reference is resolved: its member tasks are loaded, and each task's
+stored metric references are hydrated into runnable metrics — exactly as if you had inlined them. The
+same spec is submitted as the agent-evaluate job input; see
+[Agent Evaluation](/documentation/evaluate-models/agent-eval) for the full run, target, and
+results flow.
+
+The inline form remains available for one-off tasks — swap `tasks=TasksetRef(...)` for
+`tasks=[AgentEvalTaskInput(...), ...]`.
+
+<Note>
+Stored tasks carry no grader-only `reference` (held-out ground truth): that field lives only on inline
+`AgentEvalTaskInput`. Taskset-driven tasks therefore run with an empty `reference`, so use a taskset
+when your metrics score the agent's output directly rather than against per-task held-out data.
+</Note>
+
+## Async usage
+
+`AsyncNeMoPlatform` exposes the same surface; await each call.
+
+```python
+import asyncio
+
+from nemo_platform import AsyncNeMoPlatform
+
+
+async def main() -> None:
+    client = AsyncNeMoPlatform(base_url="http://localhost:8080", workspace="default")
+    page = await client.evaluator.tasks.list()
+    for item in page.data:
+        print(item.name)
+
+
+asyncio.run(main())
+```
+
+## Workspaces and projects
+
+Every method accepts an optional `workspace` argument that overrides the client's default workspace.
+On create, an optional `project` argument associates the task or taskset with a project. When you
+omit `workspace`, the client's configured workspace is used.
+
+## REST API
+
+The SDK resources are a thin client over the Evaluator plugin REST API, mounted under
+`/apis/evaluator/v2/workspaces/{workspace}`:
+
+| Method | Path | Description |
+|--------|------|-------------|
+| `GET` | `/tasks` | List tasks (paginated). |
+| `POST` | `/tasks/{name}` | Create a task. |
+| `GET` | `/tasks/{name}` | Retrieve a task. |
+| `DELETE` | `/tasks/{name}` | Delete a task. |
+| `GET` | `/tasksets` | List tasksets (paginated). |
+| `POST` | `/tasksets/{name}` | Create a taskset. |
+| `GET` | `/tasksets/{name}` | Retrieve a taskset. |
+| `DELETE` | `/tasksets/{name}` | Delete a taskset. |
+
+Creating a name that already exists returns `409`. An invalid metric reference (task) or a missing or
+duplicate task reference (taskset) returns `422`. Retrieving or deleting a name that does not exist
+returns `404`.
+
+## Related Topics
+
+- [Manage Metrics](/documentation/evaluate-models/metrics/manage-metrics) - Define and reuse the metrics that score a task
+- [SDK Resources](/documentation/evaluate-models/sdk-resources) - Run and submit evaluations through the Evaluator plugin
+- [Agent Evaluation](/documentation/evaluate-models/agent-eval) - How agent-eval tasks are executed and scored

docs/fern/versions/latest.yml

@@ -398,6 +398,9 @@ navigation:
         path: ../../evaluator/tutorials/define-run-custom-python-metrics.mdx
     - page: SDK Resources
       path: ../../evaluator/sdk-resources.mdx
+    - page: Manage Tasks & Tasksets
+      slug: manage-tasks-tasksets
+      path: ../../evaluator/manage-tasks-tasksets.mdx
     - section: Metrics
       path: ../../evaluator/metrics/index.mdx
       contents:

plugins/nemo-evaluator/src/nemo_evaluator/api/schemas.py

@@ -180,6 +180,19 @@ class TaskRef(RootModel[str]):
     )
 
 
+class TasksetRef(RootModel[str]):
+    """Reference to a persisted taskset (format: ``workspace/name`` or ``name``).
+
+    Same shape and charset as :class:`TaskRef`. Lets an evaluation reference a stored taskset in place
+    of an inline task list; the taskset's member tasks are loaded and expanded during spec resolution.
+    """
+
+    root: str = Field(
+        pattern=_ENTITY_REF_PATTERN,
+        description="Reference to a stored taskset (format: workspace/taskset-name, or taskset-name in the job workspace).",
+    )
+
+
 class Metric(BaseModel):
     """API representation of a stored metric.
 

plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py

@@ -20,7 +20,7 @@
 import logging
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Any, ClassVar
+from typing import Any, ClassVar, cast
 from urllib.parse import urlsplit
 
 import nemo_evaluator.agent_seeds  # noqa: F401 - registers the platform 'fileset' workspace-seed handler
@@ -39,6 +39,7 @@
 from nemo_evaluator.jobs.metric_resolution import resolve_metrics_to_inline, to_runtime_bundle
 from nemo_evaluator.jobs.result_persistence import persist_agent_eval_result
 from nemo_evaluator.shared.metric_bundles.bundles import unbundle_metric
+from nemo_evaluator.task_refs import resolve_agent_eval_tasks
 from nemo_evaluator_sdk.agent_eval.evaluator import AgentEvaluator
 from nemo_evaluator_sdk.agent_eval.persistence import persist_run
 from nemo_evaluator_sdk.agent_eval.results import AgentEvalResult
@@ -49,6 +50,7 @@
 from nemo_evaluator_sdk.metrics.protocol import Metric
 from nemo_evaluator_sdk.values import RunConfigOnline, RunConfigOnlineModel
 from nemo_platform import AsyncNeMoPlatform, NeMoPlatform
+from nemo_platform_plugin.entities import EntityClient
 from nemo_platform_plugin.job import NemoJob
 from nemo_platform_plugin.job_context import JobContext
 from nemo_platform_plugin.jobs.api_factory import PlatformJobSpec
@@ -135,8 +137,15 @@ async def to_spec(
             if isinstance(input_spec, AgentEvalInputSpec)
             else AgentEvalInputSpec.model_validate_json(input_spec.model_dump_json())
         )
+        entity_client = cast(EntityClient | None, entity_client)
+        # A `tasks` taskset reference is loaded and expanded into inline task DTOs first, so the
+        # metric-ref resolution below is identical whether the tasks were submitted inline or via a
+        # stored taskset.
+        task_inputs = await resolve_agent_eval_tasks(
+            submit_spec.tasks, workspace=workspace, entity_client=entity_client
+        )
         resolved_tasks: list[AgentEvalTaskSpec] = []
-        for task in submit_spec.tasks:
+        for task in task_inputs:
             metrics = await resolve_metrics_to_inline(
                 task.metrics,
                 workspace=workspace,