feat: first-class v1 harnesses/tasksets packages (+ README, serve/CLI fixes)

[mikasenghaas]

Summary

• First-class built-in packages — packages/harnesses and packages/tasksets are now real namespaced packages (harnesses/{default,rlm}, tasksets/{harbor_v1,textarena_v1}). Each package's __init__ re-exports its plugins' classes (from harnesses import RLMHarness, from tasksets import HarborTaskset); the loader resolves a built-in id to its namespaced module directly (find_spec), falling back to a flat import for local examples and hub ids. textarena_v1 stays lazily imported (it pulls the optional textarena dep), so import tasksets doesn't require it. Package descriptions are just "Built-in harnesses" / "Built-in tasksets".
• Renamed the built-in harbor taskset → harbor-v1 (module tasksets.harbor_v1), consistent with textarena-v1 and the -v1 examples.
• verifiers/v1/README.md — the README for the v1 framework (originally vf-nano), framed as the next version of verifiers. Mirrors the vf-nano README's command-driven structure (Highlights / Install / Quickstart + alias table / Patterns) and folds in the features that landed since on feat/nano-as-v1 (modal runtime + managed runtime lifecycle; per-rollout limits + native retries + wall-clock timeouts; hub ids; v0 --id bridge; vf.User; token-id/logprob traces).
• serve accepts @ file.toml — serve rejected config-only argv with USAGE before parsing even though @ file.toml supplies the ids; now mirrors eval's references_config_file. (Also: the legacy --id <env-id> path for serving v0 envs through LegacyEnvServer.)
• prime-pydantic-config bump (dev83 → dev86) — the README uses single-dash short flags (-n / -r / …); on dev83 those parsed as values. Bumped (lock + a >=0.3.0.dev86 floor) so they work.
• README API/section fixes — tool_servers → the tools() method; removed the User-simulation section; split the examples enumeration into tasksets/harnesses (matching examples/{tasksets,harnesses}).
• Removed the hello-rlm-v1 example — dropped the example package + its pyproject source/group + lock entries.

Breaking

• Built-in plugins are namespaced. The built-in harnesses/tasksets are no longer importable as top-level modules (import rlm / default / harbor / textarena_v1); they're now harnesses.rlm, harnesses.default, tasksets.harbor_v1, tasksets.textarena_v1 (or re-exported: from harnesses import RLMHarness, from tasksets import HarborTaskset). Selection by id (the public surface: --harness.id rlm, eval harbor-v1, …) is unchanged.
• harbor taskset id → harbor-v1 (e.g. eval harbor → eval harbor-v1).
• hello-rlm-v1 example removed.

Verification

• Every uv run eval … README example is dry-run-checked on dev86 (all short flags resolve; limits/retries/timeouts resolve).
• First-class packages: from harnesses.rlm import RLMHarness / from tasksets.harbor_v1 import HarborTaskset (and the re-exports from harnesses import RLMHarness / from tasksets import HarborTaskset) resolve from a tmp dir; import tasksets works without textarena; the loader resolves built-in ids (rlm → harnesses.rlm, harbor-v1 → tasksets.harbor_v1, textarena-v1 → tasksets.textarena_v1) and flat example ids (gsm8k-v1 → gsm8k_v1); terminal_bench_2_v1 / wordle_v1 import; eval … --harness.id rlm / eval harbor-v1 dry-run resolve.
• serve @ <serve-config>.toml --dry-run resolves the taskset/harness ids (no longer exits with USAGE).

---

Note

Medium Risk
Breaking import paths and harbor taskset id affect configs and downstream code; loader resolution change is central to all eval/serve plugin loading.

Overview
Reorganizes built-in v1 plugins into unified harnesses and tasksets packages (harnesses.default / harnesses.rlm, tasksets.harbor_v1 / tasksets.textarena_v1), with package __init__ re-exports and wheel layout updated to ship a single top-level package each. loaders.py now tries find_spec on {group}.{id} before falling back to flat imports for examples and Hub ids.

Breaking: built-in taskset id harbor → harbor-v1 (e.g. configs/harbor.toml); direct import harbor / import rlm style imports must use tasksets.* / harnesses.*. Example wrappers (terminal-bench-2-v1, wordle-v1) and TextArena user server entrypoint switch to namespaced imports and -m tasksets.textarena_v1.server.

Adds verifiers/v1/README.md (v1 quickstart, patterns, CLI/TOML). serve accepts @ file.toml without a positional taskset (like eval) and routes --id v0 envs through LegacyEnvServer. Bumps prime-pydantic-config to dev86 so README short flags (-n, -r, …) parse correctly. Removes the hello-rlm-v1 example from workspace sources and lockfile.

^{Reviewed by Cursor Bugbot for commit db03f33. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Reorganize harnesses and tasksets into namespaced packages with updated CLI serve support

• Moves harnesses and tasksets built-ins under unified namespaced packages (harnesses.*, tasksets.*), adding top-level __init__.py re-exports for each.
• Updates plugin loader in loaders.py to prefer the namespaced import form (e.g. tasksets.<id>) via importlib.util.find_spec, falling back to the bare module name.
• Updates serve.py to support legacy v0 environments via --id or configs with is_legacy=true, constructing a LegacyEnvServer in those cases.
• Removes the hello-rlm-v1 example and renames the harbor taskset id to harbor-v1 throughout.
• Behavioral Change: existing code importing from harbor import ... or from textarena_v1 import ... must update to the namespaced paths (e.g. from tasksets.harbor_v1 import ...).

^{Macroscope summarized db03f33.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR reorganizes harness/taskset packages and adds documentation. However, an unresolved review comment identifies that the new plugin resolution logic could silently load built-in plugins instead of hub-installed ones with the same normalized name, which warrants human review.

^{You can customize Macroscope's approvability policy. Learn more.}

Dismissing prior approval to re-evaluate db03f33

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit db03f33. Configure here.}

[cursor]

Hub ids shadowed by builtins

Medium Severity

_import_plugin always prefers tasksets.{module} / harnesses.{module} whenever find_spec succeeds, using only the normalized name from ensure_installed. Environments Hub ids like org/harbor-v1 or org/rlm normalize to the same module names as built-ins, so hub installs can silently load the vendored plugin instead of the hub package.

 

^{Reviewed by Cursor Bugbot for commit db03f33. Configure here.}