feat(evaluator-sdk): grader-only reference field on agent-eval tasks - #566
Merged
Conversation
SandyChapman
force-pushed
the
codex-seed-sources/schapman
branch
from
July 3, 2026 18:00
7e41506 to
73dc920
Compare
SandyChapman
force-pushed
the
agent-eval-reference/schapman
branch
from
July 3, 2026 18:13
17568e1 to
73ac4b2
Compare
SandyChapman
force-pushed
the
codex-seed-sources/schapman
branch
from
July 3, 2026 18:38
73dc920 to
be2593b
Compare
SandyChapman
force-pushed
the
agent-eval-reference/schapman
branch
from
July 3, 2026 19:07
73ac4b2 to
c115cc5
Compare
arpitsardhana
approved these changes
Jul 3, 2026
Extends the workspace seeding added for the general Codex runtime. A task's inputs["files"] map now accepts three JSON-serializable seed shapes, chosen by a `kind` discriminator (a bare string stays sugar for inline text): - inline — contents in the task; resolvable anywhere. `encoding="base64"` for binary. - path — a file on the authoring host; local-only. - fileset — a workspace/name#glob reference; platform-resolved. The three differ in *where* they resolve, which is what governs a task's portability. inline + path resolve in the pure-SDK runtime; fileset raises a clear "cannot be resolved in local execution" error (same pattern as metric-ref resolution) and is reserved for the service-side path, where a follow-up will resolve filesets and normalize inline/path seeds into a fileset at submit time. Adds `agent_eval/workspace_seeds.py` (the SeedFile union + `seed_workspace` resolver, with workspace-escape protection and binary support). The Codex runtime delegates to it; AgentEvalTask.inputs stays a generic dict — seeding remains a documented convention, not a core task field. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
codex-seed-sources/schapman
branch
from
July 6, 2026 12:31
be2593b to
6fd4691
Compare
…tasks Held-out ground truth for grading (canonical tests, expected outputs, rubric data) had no home on an agent-eval task, so graders could only read artifacts living in the agent's own writable workspace — which the agent can edit, making metrics like a pytest scorer trivially gameable (edit the test file the metric grades on). Add `AgentEvalTask.reference: dict[str, Any]`, surfaced to metrics via `_metric_row` as `row.data['reference']` but never routed through the agent-visible `_task_row` or seeded into the workspace. The field round-trips through the job wire/canonical DTOs (`_AgentEvalTaskCommon`) and `_to_runtime_task`/`to_spec`, so it survives remote submit. Also stop seeding the serialized task object (`task.json`) into the Docker sandbox workspace. Nothing in the runtime consumes it, and dumping the whole DTO would expose grader-only fields to the agent; the workspace now receives only the agent-facing projection (the prompt plus any declared workspace files). Stored-task persistence of `reference` (Task/TaskInput schemas + entity store) is a follow-up; those surface in the OpenAPI/SDK and are out of scope here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
agent-eval-reference/schapman
branch
from
July 6, 2026 12:35
c115cc5 to
c350b09
Compare
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds a grader-only ChangesReference field addition
Sequence Diagram(s)sequenceDiagram
participant AgentEvalJob
participant AgentEvalTaskSpec
participant AgentEvalTask
participant MetricRow
AgentEvalJob->>AgentEvalTaskSpec: to_spec(reference=task.reference)
AgentEvalJob->>AgentEvalTask: _to_runtime_task(reference=task.reference)
AgentEvalTask->>MetricRow: _metric_row includes reference
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
|
CI's frozen ruff reformats the b"".join(...) expression in test_manifest_omits_serialized_task_to_avoid_leaking_grader_fields; split it into a list + join so it's stable across ruff versions. Fixes lint-python-style on #566. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
arpitsardhana
pushed a commit
that referenced
this pull request
Jul 9, 2026
#566) * feat(evaluator-sdk): typed workspace seed sources (inline/path/fileset) Extends the workspace seeding added for the general Codex runtime. A task's inputs["files"] map now accepts three JSON-serializable seed shapes, chosen by a `kind` discriminator (a bare string stays sugar for inline text): - inline — contents in the task; resolvable anywhere. `encoding="base64"` for binary. - path — a file on the authoring host; local-only. - fileset — a workspace/name#glob reference; platform-resolved. The three differ in *where* they resolve, which is what governs a task's portability. inline + path resolve in the pure-SDK runtime; fileset raises a clear "cannot be resolved in local execution" error (same pattern as metric-ref resolution) and is reserved for the service-side path, where a follow-up will resolve filesets and normalize inline/path seeds into a fileset at submit time. Adds `agent_eval/workspace_seeds.py` (the SeedFile union + `seed_workspace` resolver, with workspace-escape protection and binary support). The Codex runtime delegates to it; AgentEvalTask.inputs stays a generic dict — seeding remains a documented convention, not a core task field. Signed-off-by: Sandy Chapman <schapman@nvidia.com> * feat(evaluator-sdk): add grader-only `reference` field to agent-eval tasks Held-out ground truth for grading (canonical tests, expected outputs, rubric data) had no home on an agent-eval task, so graders could only read artifacts living in the agent's own writable workspace — which the agent can edit, making metrics like a pytest scorer trivially gameable (edit the test file the metric grades on). Add `AgentEvalTask.reference: dict[str, Any]`, surfaced to metrics via `_metric_row` as `row.data['reference']` but never routed through the agent-visible `_task_row` or seeded into the workspace. The field round-trips through the job wire/canonical DTOs (`_AgentEvalTaskCommon`) and `_to_runtime_task`/`to_spec`, so it survives remote submit. Also stop seeding the serialized task object (`task.json`) into the Docker sandbox workspace. Nothing in the runtime consumes it, and dumping the whole DTO would expose grader-only fields to the agent; the workspace now receives only the agent-facing projection (the prompt plus any declared workspace files). Stored-task persistence of `reference` (Task/TaskInput schemas + entity store) is a follow-up; those surface in the OpenAPI/SDK and are out of scope here. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> * style(evaluator-sdk): format leak-check test to satisfy CI ruff (0.15.7) CI's frozen ruff reformats the b"".join(...) expression in test_manifest_omits_serialized_task_to_avoid_leaking_grader_fields; split it into a list + join so it's stable across ruff versions. Fixes lint-python-style on #566. Signed-off-by: Sandy Chapman <schapman@nvidia.com> --------- Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 7, 2026
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a member whose kind the target cannot run, rather than
projecting it onto an agent-eval DTO. A Harbor task's content is a directory of
files, not fields — a pure projection would silently produce an empty task.
Mixed tasksets remain storable; the mismatch surfaces at submit as a 422.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 7, 2026
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 7, 2026
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 7, 2026
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Aug 9, 2026
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
ryana
pushed a commit
to ryana/nemo-platform
that referenced
this pull request
Aug 12, 2026
…IA-NeMo#1071) * feat(evaluator): make a stored task runner-polymorphic via kind A task is an evaluation unit; how it runs is a property of the task, not a different kind of record. The target side already models this — `AgentRunnerTarget` is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now matches, and a user manages every evaluation unit in one place regardless of which runner executes it. Task content moves under a discriminated `spec`: - `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference, metrics, views - `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged directory in the Files service, plus Harbor's own config Nested rather than flattened with nullable per-kind fields, so each variant's required fields stay required and the revision digest covers the spec as a unit; two kinds with coincidentally similar metadata cannot collide on content. `kind` is a `Literal`, matching how the runner targets discriminate. The two definitions live in their own modules under `api/task_definitions/`; the shared field types they need moved to `api/fields.py`, since the definitions are imported *by* `schemas` and cannot import back from it. A single model per kind, rather than a stored/input pair: only `metrics` widens on the way in, and the service narrows it to references when storing. That keeps the API surface small at the cost of making the narrowing a service invariant rather than a type-level one. `EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground truth, surfaced to metrics but never seeded into the agent's workspace. It has existed on the inline `AgentEvalTaskInput` since NVIDIA-NeMo#566, where persisting it was deferred because the stored schemas then lived in the root OpenAPI/SDK; they are plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run expanded to an empty reference, so any task needing ground truth the agent cannot edit had to give up stored tasks and tasksets entirely. It is covered by the revision digest. The rule: the digest covers anything that affects a task's execution output or how it is graded, and `reference` decides what a metric grades against — two revisions that score differently must not share a digest, or publish-time dedup would collapse them and a pin would stop fixing the grading. Held out from the *agent*, not from the API: anyone who can read the task can read it. Harbor's `config` is the one exclusion, and it does not contradict that rule. It is a projection of `task.toml`; Harbor reads the real file out of the materialized archive at run time, and `archive_digest` is authoritative over every file in that directory. A config change that genuinely alters execution or grading therefore already moves the digest, while hashing the projection would make revision history sensitive to Harbor's serialization. That makes `archive_digest` load-bearing: a Harbor field ever read from the stored record rather than from the archive would have to be digested. Harbor specifics: - One fileset per task, so a task shared by several tasksets is stored once. - `archive_ref` is shape-validated, so a malformed reference is rejected at publish rather than surfacing as a download failure mid-run. - `config` is stored but excluded from the revision digest, as above. - Which agent runs a task is not stored: that comes from the run's target, so the same stored task can be evaluated against different agents. Taskset expansion rejects a `harbor` member rather than projecting it onto an agent-eval DTO: that content is a directory of files, not fields, so a pure projection would silently produce a task with no intent and no metrics — an evaluation that runs and scores nothing. Mixed tasksets stay storable; the mismatch surfaces at submit as a 422. The rejection is unconditional, not target-dependent. Storage landed ahead of the execution bridge, so no target can run a stored `harbor` task yet and the message says so plainly instead of suggesting the reader find a compatible one. Bridging the two — and encoding runner/task-kind compatibility declaratively rather than as an isinstance check here — is AALGO-481. Note for anyone with existing task rows: this is a breaking schema change with no migration. Rows stored in the previous flat shape fail validation on read, which surfaces as a 500 when listing tasks. Clear them before upgrading. Signed-off-by: Sandy Chapman <schapman@nvidia.com> * fix(evaluator): require the kind discriminator on a task definition `TaskDefinition` is discriminated on `kind`, but both variants defaulted it, so the generated schema left `kind` out of `required` while the validator demanded it. A raw create or replace body without `kind` fails with `union_tag_not_found` — meaning a client generated from that spec would omit the field and 422 on every write. Make `kind` a required field on both definitions, matching how the metric payload DTOs in the same package already declare their discriminator, and regenerate the plugin spec. Tests cover both halves of the mismatch: raw POST and PUT bodies without `kind` are rejected, and the published schema keeps `kind` in `required`. Signed-off-by: Sandy Chapman <schapman@nvidia.com> * refactor(evaluator): adopt the platform's entity-ref parser and fix the task docs Review follow-ups on NVIDIA-NeMo#1071. Reference parsing was duplicated. `nmp.common.entities.utils` already re-exports `nemo_platform_plugin.refs.parse_entity_ref`, which ~10 services and three other plugins use; the evaluator was the last place carrying its own copy under the same name. Delete it and delegate: `parse_subentity_ref` now adds only the `#fragment` that a revisioned entity needs on top of the shared split, and `ENTITY_REF_PATTERN` / `FILESET_REF_PATTERN` move next to the parser and the `FilesetRef` type they describe. `_SUBENTITY_REF_PATTERN` is spliced from the shared constant, so widening what counts as a `workspace/name` widens both shapes at once instead of leaving one behind. One behavior detail this makes explicit: taskset duplicate-detection relied on the old parser silently stripping `#fragment`, so `task-a` and `task-a#<digest>` deduped as one member. The platform parser does not strip, so that path now discards the fragment deliberately. Restore the `CloudpickleMetricPayload` / `InlineMetricPayload` / `MetricPayload` re-exports from `api.schemas`, which `fields.py` promises in its module docstring and lost when they moved. The `manage-tasks-tasksets` revision snippets still passed the pre-`spec` flat shape. `make docs-check-python-snippets` did not catch it because the snippet linter passes ty a rule name that was renamed upstream, so ty answered with `warning[unknown-rule]` and the check failed for every doc regardless of its content. Fix the rule name, fix the snippets, and name both task kinds before the sentence that refers to "both kinds". Type-checking a snippet would not have caught one that type-checks and then fails at run time, nor a documented output gone stale — which is the shape of what review found here. So add an integration test that walks the doc top to bottom against a real platform and asserts the results it claims. No OpenAPI change: the spec regenerates byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> * chore(evaluator): move the skill updates out of this PR The evaluator skill's `resources.md` and `plugin_sdk_examples.py` were updated here for the new `spec` shape, which put a `skills/` file in the diff and so put the PR behind the NVSkills gate. That gate cannot currently pass: tier 3 is invoked with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the runners, so no evaluation runs and the gate blocks on empty coverage. It is an infrastructure problem with the nvcarps pipeline, already reported, and nothing in this repo can resolve it. With no `skills/` file touched, the gate no longer applies to this PR and the storage change can land on its own merits. The skill updates are not lost — they move to a stacked follow-up PR, which can sit behind the gate for as long as it takes without holding this one. Reverting them costs nothing in tests: no test invokes `store_resources`, and the one assertion in `test_skill_examples.py` that pinned the new wording is reverted alongside the content it describes. Known cost while the two are apart: the skill documents the pre-`spec` task shape, which no longer validates. Anyone following the skill in that window writes a task the API rejects. That is the price of unblocking, and it ends when the follow-up lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> * chore(lint): scope a ty override to the stale evaluator skill example Moving a stored task's content under a discriminated `spec` invalidates the skill example's `TaskInput(intent=..., inputs=..., metrics=...)`, and ty checks `skills/**/*.py`, so the example fails the type gate. The fix for the example is written and sits in NVIDIA-NeMo#1237. It cannot ride along here: editing any file under `skills/` puts the PR behind the NVSkills gate, and that gate currently cannot pass — tier 3 runs with `--env-mode local`, whose bubblewrap sandbox fails its smoke test on the nvcarps runners, so nothing is evaluated and it blocks on empty coverage. Keeping the example correct and keeping this PR out of the gate are mutually exclusive until that is fixed. Chosen as an override rather than a `[tool.ty.src].exclude` entry, which that list's own header asks contributors not to grow: an override keeps every other rule live on the file and names the two the stale call actually produces, so it cannot quietly widen into cover for unrelated drift. NVIDIA-NeMo#1237 removes it in the same commit that corrects the example. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com> --------- Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a grader-only
referencefield to agent-eval tasks: held-out ground truth (canonical tests, expected outputs, rubric data) that metrics can grade against but that is never seeded into the agent's workspace or shown to the agent.Why
Held-out ground truth had no home on an
AgentEvalTask. Graders could only read artifacts living in the agent's own writable workspace — which the agent can edit — making a pytest-style scorer trivially gameable (edit the very test file the metric grades on, and "do not edit the tests" is only a polite request).referencegives graders a channel the agent cannot influence.Changes
AgentEvalTask.reference: dict[str, Any](SDK) — defaults to{}._metric_rowsurfaces it to metrics asrow.data["reference"]; the agent-visible_task_rowdeliberately does not carry it._AgentEvalTaskCommon→AgentEvalTaskInput/AgentEvalTaskSpec) and_to_runtime_task/to_specthread it through, so it round-trips to remotesubmit().DockerSandboxAgentRuntime._build_manifestwas dumping the whole task DTO astask.jsoninto the agent workspace. Nothing in the runtime consumes it, and dumping the full DTO would expose grader-only fields (reference, and anything added later) to the agent. The workspace now receives only the agent-facing projection: the prompt (instruction.md) plus any declared workspace files. (The Codex runtime was already clean — itstask.jsongoes to the server-side evidence dir, and it seeds onlyinputs["files"].)sdk/python/...) and the evaluator plugin OpenAPI spec.Tests
_metric_rowexposesreference;_task_rowhides it.AgentEvalInputSpec→AgentEvalSpec→ runtime-task round-trip preservesreference.Out of scope / follow-up
Persisting
referenceon the stored-taskTask/TaskInputschemas + entity store (AALGO-307 path) — those surface in the root OpenAPI/SDK and would require regen. Not needed for the inline/run_sync/submitpaths this PR targets.Notes
Stacked on #563 (
codex-seed-sources/schapman). Motivating consumer is the fix-bug / write-tests grading in #565.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
referencepayload that’s carried through evaluation and surfaced to metrics.referenceis preserved from input to runtime.Bug Fixes
referencedata from being exposed in the agent workspace.Tests