feat(harbor-nemo): add a NeMo registry backend for Harbor - #1235
Draft
SandyChapman wants to merge 4 commits into
Draft
feat(harbor-nemo): add a NeMo registry backend for Harbor#1235SandyChapman wants to merge 4 commits into
SandyChapman wants to merge 4 commits into
Conversation
A task is an evaluation unit; how it runs is a property of the task, not a
different kind of record. The target side already models this — `AgentRunnerTarget`
is a `kind`-discriminated union of codex/fabric/harbor — so the stored side now
matches, and a user manages every evaluation unit in one place regardless of
which runner executes it.
Task content moves under a discriminated `spec`:
- `EvaluatorTaskDefinition` (kind="evaluator") — intent, inputs, reference,
metrics, views
- `HarborTaskDefinition` (kind="harbor") — a reference to the task's packaged
directory in the Files service, plus Harbor's own config
Nested rather than flattened with nullable per-kind fields, so each variant's
required fields stay required and the revision digest covers the spec as a unit;
two kinds with coincidentally similar metadata cannot collide on content.
`kind` is a `Literal`, matching how the runner targets discriminate. The two
definitions live in their own modules under `api/task_definitions/`; the shared
field types they need moved to `api/fields.py`, since the definitions are
imported *by* `schemas` and cannot import back from it.
A single model per kind, rather than a stored/input pair: only `metrics` widens
on the way in, and the service narrows it to references when storing. That keeps
the API surface small at the cost of making the narrowing a service invariant
rather than a type-level one.
`EvaluatorTaskDefinition` gains the grader-only `reference` — held-out ground
truth, surfaced to metrics but never seeded into the agent's workspace. It has
existed on the inline `AgentEvalTaskInput` since #566, where persisting it was
deferred because the stored schemas then lived in the root OpenAPI/SDK; they are
plugin-owned now, so that reason has lapsed. Until this, a taskset-driven run
expanded to an empty reference, so any task needing ground truth the agent cannot
edit had to give up stored tasks and tasksets entirely.
It is covered by the revision digest. The rule: the digest covers anything that
affects a task's execution output or how it is graded, and `reference` decides
what a metric grades against — two revisions that score differently must not
share a digest, or publish-time dedup would collapse them and a pin would stop
fixing the grading. Held out from the *agent*, not from the API: anyone who can
read the task can read it.
Harbor's `config` is the one exclusion, and it does not contradict that rule. It
is a projection of `task.toml`; Harbor reads the real file out of the
materialized archive at run time, and `archive_digest` is authoritative over
every file in that directory. A config change that genuinely alters execution or
grading therefore already moves the digest, while hashing the projection would
make revision history sensitive to Harbor's serialization. That makes
`archive_digest` load-bearing: a Harbor field ever read from the stored record
rather than from the archive would have to be digested.
Harbor specifics:
- One fileset per task, so a task shared by several tasksets is stored once.
- `archive_ref` is shape-validated, so a malformed reference is rejected at
publish rather than surfacing as a download failure mid-run.
- `config` is stored but excluded from the revision digest, as above.
- Which agent runs a task is not stored: that comes from the run's target, so
the same stored task can be evaluated against different agents.
Taskset expansion rejects a `harbor` member rather than projecting it onto an
agent-eval DTO: that content is a directory of files, not fields, so a pure
projection would silently produce a task with no intent and no metrics — an
evaluation that runs and scores nothing. Mixed tasksets stay storable; the
mismatch surfaces at submit as a 422.
The rejection is unconditional, not target-dependent. Storage landed ahead of
the execution bridge, so no target can run a stored `harbor` task yet and the
message says so plainly instead of suggesting the reader find a compatible one.
Bridging the two — and encoding runner/task-kind compatibility declaratively
rather than as an isinstance check here — is AALGO-481.
Note for anyone with existing task rows: this is a breaking schema change with
no migration. Rows stored in the previous flat shape fail validation on read,
which surfaces as a 500 when listing tasks. Clear them before upgrading.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
`TaskDefinition` is discriminated on `kind`, but both variants defaulted it, so the generated schema left `kind` out of `required` while the validator demanded it. A raw create or replace body without `kind` fails with `union_tag_not_found` — meaning a client generated from that spec would omit the field and 422 on every write. Make `kind` a required field on both definitions, matching how the metric payload DTOs in the same package already declare their discriminator, and regenerate the plugin spec. Tests cover both halves of the mismatch: raw POST and PUT bodies without `kind` are rejected, and the published schema keeps `kind` in `required`. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
…he task docs Review follow-ups on #1071. Reference parsing was duplicated. `nmp.common.entities.utils` already re-exports `nemo_platform_plugin.refs.parse_entity_ref`, which ~10 services and three other plugins use; the evaluator was the last place carrying its own copy under the same name. Delete it and delegate: `parse_subentity_ref` now adds only the `#fragment` that a revisioned entity needs on top of the shared split, and `ENTITY_REF_PATTERN` / `FILESET_REF_PATTERN` move next to the parser and the `FilesetRef` type they describe. `_SUBENTITY_REF_PATTERN` is spliced from the shared constant, so widening what counts as a `workspace/name` widens both shapes at once instead of leaving one behind. One behavior detail this makes explicit: taskset duplicate-detection relied on the old parser silently stripping `#fragment`, so `task-a` and `task-a#<digest>` deduped as one member. The platform parser does not strip, so that path now discards the fragment deliberately. Restore the `CloudpickleMetricPayload` / `InlineMetricPayload` / `MetricPayload` re-exports from `api.schemas`, which `fields.py` promises in its module docstring and lost when they moved. The `manage-tasks-tasksets` revision snippets still passed the pre-`spec` flat shape. `make docs-check-python-snippets` did not catch it because the snippet linter passes ty a rule name that was renamed upstream, so ty answered with `warning[unknown-rule]` and the check failed for every doc regardless of its content. Fix the rule name, fix the snippets, and name both task kinds before the sentence that refers to "both kinds". Type-checking a snippet would not have caught one that type-checks and then fails at run time, nor a documented output gone stale — which is the shape of what review found here. So add an integration test that walks the doc top to bottom against a real platform and asserts the results it claims. No OpenAPI change: the spec regenerates byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Harbor publishes to and downloads from the public Hub. This implements Harbor's
pluggable registry-backend interface against NeMo, so
export HARBOR_REGISTRY_BACKEND=nemo
harbor publish ./my-task
harbor run -d nvidia/my-dataset
use NeMo instead, with no change to Harbor. Registration is the
`harbor.registry_backends` entry point, which is what makes the stock CLI work.
A Harbor task becomes a stored task with `kind="harbor"`, its archive a file in
a fileset; a Harbor dataset becomes a taskset. The org is folded into the entity
name (`nvidia/my-task` -> `nvidia.my-task`) rather than mapped to a workspace: a
workspace is a tenancy boundary with its own lifecycle and authorization, while
a Harbor org is a cheap namespace that `publish` creates on demand, and mapping
one to the other would make publishing a tenancy operation.
Two digests are in play and are not interchangeable. NeMo addresses a revision
by a digest of the stored spec; Harbor addresses a version by a digest of the
task directory's files, kept in `spec.archive_digest`. A `sha256:` reference
arriving from Harbor is always the latter and is not a valid revision selector,
so a content-pinned lookup scans revisions rather than fetching directly. Two
adjacent traps are handled with it: a revision *ordinal* is read by the platform
as a tag name, and NeMo digests are bare hex, so Harbor's `sha256:` prefix must
be stripped before any digest is used as a selector.
Publishing a dataset translates between the two spaces so a manifest's pins
survive. This is not optional: the taskset service re-resolves bare member refs
at write time, so an unpinned member would silently pin whatever was `latest` at
publish rather than what the manifest named.
`_create_archive`, `remote_path` and `publish_tasks` are inherited untouched, so
a package published here is byte-identical to the same package on the Hub and
their content hashes agree.
Dataset-level files currently ride in taskset `metadata` as JSON. The taskset
`files` field that replaces this is a separate change; this package moves over
once it lands.
This depends on Harbor's registry-backend abstraction, which is not upstream
yet, so it is not usable as it stands and is not wired into the uv workspace.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Harbor publishes to and downloads from the public Hub. This implements Harbor's pluggable registry-backend interface against NeMo, so the stock Harbor CLI uses NeMo instead — with no change to Harbor:
Registration is the
harbor.registry_backendsentry point, which is what makes the packaged CLI work:pip installplus the variable is the whole integration.Changes
A new
packages/harbor_nemo/implementing the five interfaces:NemoRegistryBackend,NemoPublisher,NemoStorage,NemoDatasetClient,NemoTaskResolver, plus name mapping, config, and an HTTP client whose failures are already Harbor's error types.Deliberately not a
[tool.uv.workspace]member: it depends onharbor, which the platform keeps as a marker-gated optional extra, and listing it would pull harbor into every bareuv sync --all-packages. It is also outsidepytest.ini'stestpaths, so CI does not run its tests — see Verification.Design decisions worth review
nvidia/my-task→nvidia.my-task), not mapped to a workspace. A workspace is a tenancy boundary with its own lifecycle and authorization; a Harbor org is a cheap namespace thatpublishcreates on demand. Mapping one to the other would make publishing a tenancy operation. The cost is that the org prefix is a convention, not an enforced boundary.spec.archive_digest. Asha256:reference arriving from Harbor is always the latter and is not a valid revision selector, so a content-pinned lookup scans revisions instead of fetching directly.latestat publish rather than what the manifest named._create_archive,remote_pathandpublish_tasksare inherited untouched. That is what keeps a package published here byte-identical to the same package on the Hub, so their content hashes agree and historical eval results stay comparable across a migration.record_downloadis left as the inherited no-op. No counter primitive exists, so implementing it would mean a read-modify-write on the hottest entity per package for best-effort telemetry.Known gaps, called out in the package README
metadataas JSON. feat(evaluator): give a taskset its own files #1234 adds the tasksetfilesfield that replaces this; the switchover is a follow-up.ResolvedTaskVersion.yanked_atis alwaysNone.harbor version list|show|tagis Supabase-pinned in Harbor itself and shows Hub data regardless of the selector.Type of Change
Quality Gates
packages/harbor_nemo/README.mddocuments the mapping, configuration, the two-digest hazard, and the known gaps; nothing user-facing changes until Harbor's side is upstream.Verification
Signed-off-by:traileruv run pre-commit run -apasses, or any blocked checks are identified belowTargeted validation, all against a local platform (
entities,files,evaluator) on this branch:pytest packages/harbor_nemo/tests), in a venv withharborinstalled. CI will not run these — the package is outsidetestpathsandharboris not installed in the default environment. Reviewers wanting to run them:uv pip install -e <harbor checkout> -e packages/harbor_nemo.harbor publish ./my-task→ revision 1; re-run →exists/skipped, no upload.harbor download→ recomputed content hash matches the source exactly (the byte-identical-archive contract).harbor download nvidia/my-task@sha256:<older digest>→ returns the older revision's content, hash matches.harbor add→ pins the digest;harbor sync --upgrade→ moves the pin,Source: registry.harbor publish ./my-dataset(with a dataset-levelmetric.py) → revision 1; re-run → skipped.harbor run -t nvidia/my-task --agent nop→ 1 trial, 0 exceptions, Reward 1.000.harbor run -d nvidia/my-dataset --agent nop→ 1 trial, 0 exceptions, Reward 1.000, dataset-level metric executed.Package '...' not found; unreachable platform →Could not reach the NeMo platform, not a not-found.uv run pre-commit run -a— all substantive hooks pass (ruff,ruff format,Run ty typechecks,Check config reference doc is up to date,Check for uv.lock drift,Fix copyright headers,Plugins must not import from nmp-common,check for merge conflicts). Two hooks could not run locally and are not reported as passing:Run uv lock with platform uv(needs uv 0.9.14; nopyproject.tomlin the workspace is touched) andRun UI lint-staged(no localpnpm; noweb/files touched).