feat(evaluator): make a stored task runner-polymorphic via kind

[SandyChapman]

#1023 has merged. This branch is rebased onto main and is now a single commit; the earlier stacked-review note no longer applies.

Why

A task is an evaluation unit; how it runs is a property of the task, not a different kind of record. The target side already models this — AgentRunnerTarget is a kind-discriminated union of codex | fabric | harbor, with a comment saying "widen with more members as runners land."

This makes the stored side match, so a user manages every evaluation unit in one place regardless of which runner executes it. It's the foundation for publishing Harbor tasks into NeMo.

What

Task content moves under a discriminated spec:

kind	content
evaluator	intent, inputs, reference, metrics, views
harbor	archive_ref (fileset), archive_digest, instruction, opaque Harbor config

Nested rather than flattened with nullable per-kind fields. Flat would make every field optional with nothing enforcing a coherent set; nesting keeps each variant's required fields required, and the revision digest then covers the spec as a unit — so two kinds with coincidentally similar metadata can't collide on content. There's a test for that.

Named …Definition, not …Spec. AgentEvalTaskSpec already names the runtime task DTO in jobs.agent_spec; two models sharing a class name across modules break OpenAPI generation with a schema-name collision. (Found by regenerating the spec — see below.)

One fileset per Harbor task, so a task shared by several tasksets is stored once. archive_ref is shape-validated, so a malformed reference is rejected at publish rather than surfacing as a download failure mid-run — matching how a metric reference is checked when a task is stored.

Grader-only reference on stored tasks

EvaluatorTaskDefinition gains reference — held-out ground truth, surfaced to metrics but never seeded into the agent's workspace or shown to the agent. It has existed on the inline AgentEvalTaskInput since #566, where persisting it was deferred because the stored schemas then lived in the root OpenAPI/SDK; they're plugin-owned now, so that reason has lapsed.

Until this, a taskset-driven run expanded to an empty reference, so any task needing ground truth the agent cannot edit had to give up stored tasks and tasksets entirely — which is the reward-hacking hole #566 was opened to close.

Held out from the agent, not from the API: anyone who can read the task can read it.

What the revision digest covers

The rule: the digest covers anything that affects a task's execution output or how it is graded. A field may be excluded only if it's a derived view of content the digest already covers by another route.

reference is therefore in — it decides what a metric grades against, so two revisions that score differently must not share a digest, or publish-time dedup would collapse them and a pin would stop fixing the grading.

Harbor's config is the one exclusion, and it doesn't contradict that rule: it's a projection of task.toml, Harbor reads the real file out of the materialized archive at run time, and archive_digest is authoritative over every file in that directory. So a config change that genuinely alters execution or grading already moves the digest, while hashing the projection would make revision history sensitive to Harbor's serialization — a release that reordered keys would cut a revision for byte-identical files.

That makes archive_digest load-bearing, and there's a test guarding it: a Harbor field ever read from the stored record rather than from the archive would have to be digested.

Scope: storage, not execution

This PR makes the storage model runner-polymorphic. It touches no execution-path source file.

	evaluator kind	harbor kind
Stored-task management (CRUD, revisions, tasksets)	✅	✅
Execution from a stored task	✅ today	❌ follow-up

resolve_taskset_ref previously projected every member onto an agent-eval DTO. A Harbor task's content is a directory of files, not fields, so that projection would silently produce a task with no intent and no metrics — an evaluation that runs and scores nothing. It now raises UnsupportedTaskKindError, surfaced as a 422 before the run starts.

Harbor execution itself already works on main (HarborRunnerTarget, driven by harbor_dataset_path metadata) and is unaffected by this PR — it just isn't fed by stored task entities yet.

Follow-up: AALGO-481 — bridge stored harbor tasks to the harbor runner, and encode runner ↔ task-kind compatibility declaratively instead of via an isinstance check in the expansion path. That ticket also covers fixing the current rejection message, which unhelpfully suggests submitting against "a target whose runner executes 'harbor' tasks" when no such target exists yet.

Breaking change

No migration. Rows stored in the previous flat shape fail validation on read, which surfaces as a 500 when listing tasks. Clear them before upgrading.

Verification

• 733 unit tests
• tools/lint/lint-python-types.sh — 0 errors
• ruff check / ruff format --check — clean
• Plugin OpenAPI spec regenerated

Docs updated: examples, the TaskInput field table, the task-kinds section, and the evaluator skill reference.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added support for evaluator and Harbor task types with runner-specific task details.
  • Tasks now use a unified specification for inputs, metrics, references, and Harbor configuration.
  • Added grader-only references and pinned taskset grading content.
  • Changes to grading references or Harbor archives now create appropriately tracked revisions.
  • Added validation for task definitions, metric references, metadata, and taskset membership.

• Documentation

  • Updated task creation, retrieval, stored-task, taskset, and held-out-reference examples for the new format.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-1071-harbor-task-kind-schapman.docs.buildwithfern.com/nemo-platform

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	32000/40624	78.8%	63.5%
Integration Tests	18554/38550	48.1%	20.8%

[SandyChapman]

/nvskills-ci

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Tasks now use nested, discriminated evaluator or Harbor specifications. Persistence, revision hashing, taskset expansion, tests, examples, and documentation use the new model and pinned grader references.

Changes

Unified task definitions and task execution

Layer / File(s)	Summary
Task definition contracts plugins/nemo-evaluator/openapi/openapi.yaml, plugins/nemo-evaluator/src/nemo_evaluator/api/*, plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/*	Adds evaluator and Harbor task definitions. Task and TaskInput now expose a discriminated spec field with shared validation.
Specification persistence and hashing plugins/nemo-evaluator/src/nemo_evaluator/entities.py, plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py, plugins/nemo-evaluator/src/nemo_evaluator/revisions.py	Stores task content under spec. Supports nested hash exclusions and separates revision pointer and self-field exclusions.
Task creation and taskset resolution plugins/nemo-evaluator/src/nemo_evaluator/api/service/task_service.py, plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py	Normalizes inline evaluator metrics, preserves pinned evaluator references, and rejects unsupported Harbor task expansion.
Validation and examples plugins/nemo-evaluator/tests/*, docs/evaluator/manage-tasks-tasksets.mdx, skills/nemo-evaluator-plugin/*	Updates tests, examples, and documentation for nested specifications, Harbor fields, stored references, revision behavior, and taskset pinning.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant TaskService
  participant RevisionStore
  participant TasksetResolver

  Client->>TaskService: Submit TaskInput.spec
  TaskService->>RevisionStore: Store normalized task specification
  RevisionStore-->>TaskService: Return task revision
  TasksetResolver->>RevisionStore: Load pinned revision
  RevisionStore-->>TasksetResolver: Return evaluator spec and reference
  TasksetResolver-->>Client: Return expanded task input

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#779: Extends taskset resolution for nested task specifications and pinned grader references.
• NVIDIA-NeMo/nemo-platform#1023: Relates to revision pinning, content hashing, and nested task definitions.
• NVIDIA-NeMo/nemo-platform#1129: Relates to task-reference validation and pinned taskset-member revisions.

Suggested reviewers: arpitsardhana, ngoncharenko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: stored tasks become runner-polymorphic through a kind discriminator.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 2

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch harbor-task-kind/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

skills/nemo-evaluator-plugin/references/resources.md (1)

64-99: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add supported CLI variants in tab sets.

The evaluator CLI does not manage stored metrics, tasks, or tasksets. Add curl REST examples for the CRUD calls in both affected workflows instead of undocumented nemo evaluator commands.

• skills/nemo-evaluator-plugin/references/resources.md#L64-L99
• docs/evaluator/manage-tasks-tasksets.mdx#L76-L89

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/nemo-evaluator-plugin/references/resources.md` around lines 64 - 99,
Add supported curl REST examples for creating the metric and stored task in
skills/nemo-evaluator-plugin/references/resources.md lines 64-99, replacing any
undocumented nemo evaluator CLI usage while preserving the reference-backed
workflow. Apply the same CLI-to-curl REST replacement for the task and taskset
CRUD workflow in docs/evaluator/manage-tasks-tasksets.mdx lines 76-89, covering
each documented CRUD call.

Source: Coding guidelines

🧹 Nitpick comments (3)

plugins/nemo-evaluator/tests/test_content_hash.py (1)

231-257: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Add the Harbor head/revision digest-agreement test.

Both tests here exercise head_digest only. entities.py states that REVISION_POINTER_EXCLUDE and REVISION_SELF_EXCLUDE must exclude the same derived fields, otherwise publish-time dedup never fires. That mirror is asserted for the evaluator kind in test_revision_entity.py::test_task_head_and_revision_digests_agree, but not for Harbor — the only kind the exclusion actually applies to.

💚 Proposed test

def test_harbor_head_and_revision_digests_agree() -> None:
    """The exclusion is declared twice, once per constant. If they drift, a Harbor task
    republishes an identical archive as a new revision on every PUT."""
    head = _harbor_task(config={"verifier": {"type": "pytest"}})
    revision = TaskRevisionEntity(
        spec=head.spec,
        name="rev.1",
        workspace="default",
        content_hash="a" * 64,
        revision=1,
    )
    assert head_digest(head) == content_hash(revision, exclude=REVISION_SELF_EXCLUDE)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/tests/test_content_hash.py` around lines 231 - 257,
Add a Harbor-specific test near test_harbor_config_does_not_change_digest that
constructs a configured task via _harbor_task, creates the matching
TaskRevisionEntity, and asserts head_digest(head) equals content_hash(revision,
exclude=REVISION_SELF_EXCLUDE). Import or reuse the existing TaskRevisionEntity
and REVISION_SELF_EXCLUDE symbols, preserving the test’s focus on agreement
between the two exclusion definitions.

plugins/nemo-evaluator/src/nemo_evaluator/entities.py (1)

53-84: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Make the digest exclusion kind-aware, or assert it.

_DERIVED_SPEC_FIELDS excludes any spec field named config, for every variant in the TaskDefinition union. The safety argument documented above holds only for HarborTaskDefinition, where archive_digest covers the same content. EvaluatorTaskDefinition has no config today, so there is no current defect.

If a future variant adds a config that is a real execution or grading input, it is silently dropped from the revision digest. Two tasks that grade differently would then share a digest, and a pinned taskset would re-grade without cutting a revision. Name-based exclusion cannot detect that.

Bind the exclusion to the kind that owns it, or fail loudly when another variant declares the name.

♻️ Kind-scoped exclusion

-_DERIVED_SPEC_FIELDS = {"config"}
+#: Keyed by the variant that owns the field, so the safety argument travels with the kind.
+_DERIVED_SPEC_FIELDS_BY_KIND: dict[type, set[str]] = {HarborTaskDefinition: {"config"}}
+_DERIVED_SPEC_FIELDS = {
+    field for fields in _DERIVED_SPEC_FIELDS_BY_KIND.values() for field in fields
+}
+# A variant that is not the owner must not declare a name on the exclusion list, or its field
+# would be dropped from the digest by accident.
+for _variant in (EvaluatorTaskDefinition, HarborTaskDefinition):
+    _unowned = (set(_variant.model_fields) & _DERIVED_SPEC_FIELDS) - _DERIVED_SPEC_FIELDS_BY_KIND.get(_variant, set())
+    if _unowned:
+        raise RuntimeError(
+            f"{_variant.__name__} declares digest-excluded field(s) {sorted(_unowned)} it does not own; "
+            "digest every field that affects execution or grading."
+        )

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/src/nemo_evaluator/entities.py` around lines 53 - 84,
Make the config exclusion in REVISION_POINTER_EXCLUDE and REVISION_SELF_EXCLUDE
kind-aware rather than applying _DERIVED_SPEC_FIELDS to every TaskDefinition
variant. Scope config removal to HarborTaskDefinition, or add validation that
fails loudly if another kind declares config without an explicit digest policy;
preserve config exclusion for Harbor while preventing silently omitted execution
or grading inputs.

plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py (1)

50-96: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Update the exclude argument documentation

Pydantic accepts nested set syntax and ignores unknown exclusion keys. The shared REVISION_POINTER_EXCLUDE is therefore safe for TasksetEntity, which has no spec field. Update the Args entry to describe both flat sets and nested mappings.

📝 Docstring fix

-        exclude: Extra field names to drop on top of ``__base_fields__``. Revisioned entities pass
-            their own revision/tag bookkeeping here — a revision's digest must not cover the
-            revision index that was assigned *because of* that digest.
+        exclude: Extra fields to drop on top of ``__base_fields__``, as either a flat set of field
+            names or pydantic's nested mapping form (``{"spec": {"config"}}``) for dropping a
+            field inside a sub-model. Revisioned entities pass their own revision/tag bookkeeping
+            here — a revision's digest must not cover the revision index that was assigned
+            *because of* that digest.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py` around lines 50 -
96, Update the Args documentation for canonical_payload’s exclude parameter to
describe support for both flat field-name sets and nested mappings, including
that unknown exclusion keys are ignored by Pydantic. Keep the existing behavior
and implementation of _as_exclude_map unchanged.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/evaluator/manage-tasks-tasksets.mdx`:
- Around line 91-112: Update docs/evaluator/manage-tasks-tasksets.mdx lines
91-112 to use the public schema name TaskDefinition instead of TaskSpecInput,
and change the adjacent stored.metrics reference to stored.spec.metrics. At
docs/evaluator/manage-tasks-tasksets.mdx line 139, replace agent_eval with
evaluator.

In `@plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py`:
- Around line 61-64: Update the UnsupportedTaskKindError message in task_refs.py
lines 61-64 to state that stored Harbor task execution is not yet available,
removing guidance to select a compatible target. Update
docs/evaluator/manage-tasks-tasksets.mdx lines 123-126 to state that taskset
expansion rejects Harbor members until stored Harbor execution is implemented.

In `@plugins/nemo-evaluator/tests/api/service/test_task_service.py`:
- Around line 74-75: The Task.spec union must be narrowed before
variant-specific fields are accessed. In
plugins/nemo-evaluator/tests/api/service/test_task_service.py at lines 74-75,
99-101, 123-125, 145-146, 253, 390-393, and 423, and in
plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py at lines
77-84, plugins/nemo-evaluator/tests/integration/test_task_revisions.py at lines
81-83, 175, 225-226, and 324-325, plus
plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py at line 89, add an
appropriate EvaluatorTaskDefinition or HarborTaskDefinition isinstance assertion
before accessing intent, metrics, reference, or config.

In `@skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py`:
- Around line 76-80: Update
skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py lines 76-80
to use the bare resource references MetricRef("answer-exact") and
TaskRef("capital-france") so they resolve in the caller workspace. Update
skills/nemo-evaluator-plugin/references/resources.md lines 44-60 to use the same
bare metric and task references for resources created by the example.

---

Outside diff comments:
In `@skills/nemo-evaluator-plugin/references/resources.md`:
- Around line 64-99: Add supported curl REST examples for creating the metric
and stored task in skills/nemo-evaluator-plugin/references/resources.md lines
64-99, replacing any undocumented nemo evaluator CLI usage while preserving the
reference-backed workflow. Apply the same CLI-to-curl REST replacement for the
task and taskset CRUD workflow in docs/evaluator/manage-tasks-tasksets.mdx lines
76-89, covering each documented CRUD call.

---

Nitpick comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py`:
- Around line 50-96: Update the Args documentation for canonical_payload’s
exclude parameter to describe support for both flat field-name sets and nested
mappings, including that unknown exclusion keys are ignored by Pydantic. Keep
the existing behavior and implementation of _as_exclude_map unchanged.

In `@plugins/nemo-evaluator/src/nemo_evaluator/entities.py`:
- Around line 53-84: Make the config exclusion in REVISION_POINTER_EXCLUDE and
REVISION_SELF_EXCLUDE kind-aware rather than applying _DERIVED_SPEC_FIELDS to
every TaskDefinition variant. Scope config removal to HarborTaskDefinition, or
add validation that fails loudly if another kind declares config without an
explicit digest policy; preserve config exclusion for Harbor while preventing
silently omitted execution or grading inputs.

In `@plugins/nemo-evaluator/tests/test_content_hash.py`:
- Around line 231-257: Add a Harbor-specific test near
test_harbor_config_does_not_change_digest that constructs a configured task via
_harbor_task, creates the matching TaskRevisionEntity, and asserts
head_digest(head) equals content_hash(revision, exclude=REVISION_SELF_EXCLUDE).
Import or reuse the existing TaskRevisionEntity and REVISION_SELF_EXCLUDE
symbols, preserving the test’s focus on agreement between the two exclusion
definitions.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ede6d620-8093-4dbc-a705-f7bd33e5b261

📥 Commits

Reviewing files that changed from the base of the PR and between d985222 and 4ccd57c.

📒 Files selected for processing (25)

• docs/evaluator/manage-tasks-tasksets.mdx
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/api/fields.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/schemas.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/service/task_service.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/evaluator.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/harbor.py
• plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py
• plugins/nemo-evaluator/src/nemo_evaluator/entities.py
• plugins/nemo-evaluator/src/nemo_evaluator/revisions.py
• plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py
• plugins/nemo-evaluator/tests/api/service/test_task_service.py
• plugins/nemo-evaluator/tests/api/v2/test_tasks_routes.py
• plugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.py
• plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py
• plugins/nemo-evaluator/tests/integration/test_task_revisions.py
• plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py
• plugins/nemo-evaluator/tests/test_content_hash.py
• plugins/nemo-evaluator/tests/test_revision_entity.py
• plugins/nemo-evaluator/tests/test_revisions.py
• plugins/nemo-evaluator/tests/test_skill_examples.py
• plugins/nemo-evaluator/tests/test_task_entity.py
• plugins/nemo-evaluator/tests/test_task_refs.py
• skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
• skills/nemo-evaluator-plugin/references/resources.md

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/tests/api/service/test_task_service.py`:
- Around line 376-382: Update _FakeMetricService to record get_metric
invocations, then extend test_a_harbor_task_never_reaches_the_metric_service to
assert that the recorded get_metric call list is empty alongside
metric_service.stored. Preserve the existing assertion covering
store_derived_metric.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ca32a535-4ee6-493d-b7c8-c6b2353f1437

📥 Commits

Reviewing files that changed from the base of the PR and between 4ccd57c and ca82875.

📒 Files selected for processing (8)

• docs/evaluator/manage-tasks-tasksets.mdx
• plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py
• plugins/nemo-evaluator/tests/api/service/test_task_service.py
• plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py
• plugins/nemo-evaluator/tests/integration/test_task_revisions.py
• plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py
• skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
• skills/nemo-evaluator-plugin/references/resources.md

🚧 Files skipped from review as they are similar to previous changes (5)

• plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py
• plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py
• plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py
• skills/nemo-evaluator-plugin/references/resources.md
• plugins/nemo-evaluator/tests/integration/test_task_revisions.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/tests/api/service/test_task_service.py`:
- Around line 425-430: Update the test around replace_task to fetch “fix-test”
again after the no-revision replacement, then assert the Harbor config on that
freshly retrieved task rather than on same; retain the existing assertions that
no revision was published and the revision remains 1.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: edb05a94-f83a-4de8-8711-00ff1f596280

📥 Commits

Reviewing files that changed from the base of the PR and between ca82875 and 2345ddb.

📒 Files selected for processing (1)

• plugins/nemo-evaluator/tests/api/service/test_task_service.py

[SandyChapman]

/nvskills-ci

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/openapi/openapi.yaml`:
- Around line 3466-3470: Make the discriminator property kind required in both
TaskDefinition DTO variants, remove any schema behavior that permits it to be
omitted, and regenerate the OpenAPI schema. Update the documentation example to
include kind set to evaluator, and add raw JSON POST and PUT coverage verifying
bodies without kind are rejected while valid bodies include it.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5da970de-6c33-4b03-b88d-8a752d845b53

📥 Commits

Reviewing files that changed from the base of the PR and between 021afd1 and 9852905.

📒 Files selected for processing (25)

• docs/evaluator/manage-tasks-tasksets.mdx
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/api/fields.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/schemas.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/service/task_service.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/evaluator.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/harbor.py
• plugins/nemo-evaluator/src/nemo_evaluator/content_hash.py
• plugins/nemo-evaluator/src/nemo_evaluator/entities.py
• plugins/nemo-evaluator/src/nemo_evaluator/revisions.py
• plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py
• plugins/nemo-evaluator/tests/api/service/test_task_service.py
• plugins/nemo-evaluator/tests/api/v2/test_tasks_routes.py
• plugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.py
• plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py
• plugins/nemo-evaluator/tests/integration/test_task_revisions.py
• plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py
• plugins/nemo-evaluator/tests/test_content_hash.py
• plugins/nemo-evaluator/tests/test_revision_entity.py
• plugins/nemo-evaluator/tests/test_revisions.py
• plugins/nemo-evaluator/tests/test_skill_examples.py
• plugins/nemo-evaluator/tests/test_task_entity.py
• plugins/nemo-evaluator/tests/test_task_refs.py
• skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
• skills/nemo-evaluator-plugin/references/resources.md

🚧 Files skipped from review as they are similar to previous changes (21)

• plugins/nemo-evaluator/tests/integration/test_task_derived_metrics.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/service/task_service.py
• plugins/nemo-evaluator/src/nemo_evaluator/revisions.py
• plugins/nemo-evaluator/tests/test_revision_entity.py
• plugins/nemo-evaluator/tests/test_skill_examples.py
• skills/nemo-evaluator-plugin/assets/examples/plugin_sdk_examples.py
• plugins/nemo-evaluator/tests/test_task_entity.py
• plugins/nemo-evaluator/tests/test_content_hash.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/harbor.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/task_definitions/evaluator.py
• plugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.py
• plugins/nemo-evaluator/src/nemo_evaluator/task_refs.py
• plugins/nemo-evaluator/tests/sdk/test_task_sdk_resources.py
• plugins/nemo-evaluator/tests/test_task_refs.py
• skills/nemo-evaluator-plugin/references/resources.md
• plugins/nemo-evaluator/tests/test_revisions.py
• plugins/nemo-evaluator/src/nemo_evaluator/entities.py
• plugins/nemo-evaluator/src/nemo_evaluator/api/schemas.py
• plugins/nemo-evaluator/tests/integration/test_task_revisions.py
• plugins/nemo-evaluator/tests/api/service/test_task_service.py
• plugins/nemo-evaluator/tests/api/v2/test_tasks_routes.py

[SandyChapman]

/nvskills-ci

[ngoncharenko]

Manual review against origin/main: three actionable issues.

[SandyChapman]

/nvskills-ci

[SandyChapman]

/nvskills-ci