V1 runtime/taskset/harness refactor

.github/workflows/README.md

@@ -13,7 +13,7 @@ This directory contains automated workflows for the verifiers project.
 
 **What it does**:
 - Runs ruff for linting and formatting checks
-- Runs ty type checks with `uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses`
+- Runs ty type checks with `uv run ty check verifiers/v1 packages/tasksets/tasksets packages/harnesses/harnesses`
 - Runs Semgrep policy checks from the isolated `policy` dependency group.
 - Uses configuration from `pyproject.toml`, `.pre-commit-config.yaml`, and `.semgrep/verifiers.yml`
 
@@ -55,7 +55,7 @@ To run checks locally the same way they run in CI:
 
 ```bash
 # Ty parity with CI (Python 3.13 target configured in `pyproject.toml`)
-uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses
+uv run ty check verifiers/v1 packages/tasksets/tasksets packages/harnesses/harnesses
 
 # Verifiers-specific policy lint
 env PYTHONWARNINGS=ignore::SyntaxWarning uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet

.github/workflows/style.yml

@@ -41,7 +41,7 @@ jobs:
       - name: Install dependencies
         run: uv sync
       - name: Run ty
-        run: uv run ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses
+        run: uv run ty check verifiers/v1 packages/tasksets/tasksets packages/harnesses/harnesses
   semgrep:
     name: Semgrep
     runs-on: ubuntu-latest

.pre-commit-config.yaml

@@ -28,7 +28,7 @@ repos:
         pass_filenames: false
       - id: ty
         name: ty (ci parity)
-        entry: uv run --python 3.13 ty check verifiers packages/tasksets/tasksets packages/harnesses/harnesses
+        entry: uv run --python 3.13 ty check verifiers/v1 packages/tasksets/tasksets packages/harnesses/harnesses
         language: system
         pass_filenames: false
         stages: [pre-push]

.semgrep/verifiers.yml

@@ -1,10 +1,4 @@
 rules:
-  - id: verifiers-no-future-annotations
-    languages: [python]
-    severity: ERROR
-    message: Do not use `from __future__ import annotations`; quote only the specific forward references that need it.
-    pattern: from __future__ import annotations
-
   - id: verifiers-no-skip-validation
     languages: [python]
     severity: ERROR
@@ -40,89 +34,16 @@ rules:
               metavariable: $ANNOT
               regex: "(Any|Mapping\\[str, object\\]|dict\\[str, object\\]|.*\\|.*\\|.*)"
 
-  - id: verifiers-v1-load-environment-config-required
+  - id: verifiers-v1-no-env-config-subclasses
     languages: [python]
     severity: ERROR
-    message: v1 load_environment must take a strict config object; do not accept None or synthesize config with config-or-Config fallback.
+    message: EnvConfig is final. Put defaults and public settings on TasksetConfig and HarnessConfig; the library loader owns vf.Env construction.
     paths:
       include:
         - /environments/**/*.py
-    pattern-either:
-      - pattern: |
-          def load_environment(..., config: $CONFIG = None, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: $CONFIG | None, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: $CONFIG | None = $DEFAULT, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: Optional[$CONFIG], ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: Optional[$CONFIG] = $DEFAULT, ...):
-              ...
-
-  - id: verifiers-v1-no-load-environment-config-fallback
-    languages: [python]
-    severity: ERROR
-    message: v1 load_environment receives a concrete config object from the framework; do not use config or Config() fallback.
-    paths:
-      include:
-        - /environments/**/*.py
-    patterns:
-      - pattern-inside: |
-          def load_environment(...):
-              ...
-      - pattern: $CONFIG = $CONFIG or $DEFAULT()
-      - metavariable-regex:
-          metavariable: $DEFAULT
-          regex: ".*Config"
-
-  - id: verifiers-v1-load-environment-canonical-shim
-    languages: [python]
-    severity: ERROR
-    message: v1 load_environment is only the Taskset/Harness loader shim. Keep load_environment as-is; implement the config surface through TasksetConfig, HarnessConfig, load_taskset, and load_harness instead of patching root-loader behavior.
-    paths:
-      include:
-        - /environments/**/*.py
-        - /verifiers/scripts/init.py
-      exclude:
-        - /environments/alphabet_sort/alphabet_sort_v1.py
-        - /environments/bfcl_v3/bfcl_v3.py
-        - /environments/dspy_flights/dspy_flights.py
-        - /environments/dspy_rlm/dspy_rlm.py
-        - /environments/hello_group_reward_v1/hello_group_reward_v1.py
-        - /environments/hello_parallel_sandbox_v1/hello_parallel_sandbox_v1.py
-        - /environments/hello_self_judge_v1/hello_self_judge_v1.py
-        - /environments/hello_subagent_v1/hello_subagent_v1.py
-        - /environments/langchain_deep_agents_wikispeedia/langchain_deep_agents_wikispeedia.py
-        - /environments/math_python/math_python_v1.py
-        - /environments/mcp_search_env/mcp_search_env.py
-        - /environments/nemo_gym_env/nemo_gym_env/env.py
-        - /environments/nested_harness_v1/nested_harness_v1.py
-        - /environments/openai_agents_env/openai_agents_env.py
-        - /environments/rlm_swe_v1/rlm_swe_v1.py
-        - /environments/tau2_bench_v1/tau2_bench_v1.py
-        - /environments/wiki_search/wiki_search_v1.py
-    patterns:
-      - pattern: |
-          def load_environment(config: $CONFIG) -> vf.Env:
-              ...
-      - pattern-not: |
-          def load_environment(config: vf.EnvConfig) -> vf.Env:
-              return vf.Env(
-                  taskset=vf.load_taskset(config=config.taskset),
-                  harness=vf.load_harness(config=config.harness),
-              )
-      - pattern-not: |
-          def load_environment(config: vf.EnvConfig) -> vf.Env:
-              """Loader pattern for all Taskset/Harness environments."""
-              return vf.Env(
-                  taskset=vf.load_taskset(config=config.taskset),
-                  harness=vf.load_harness(config=config.harness),
-              )
+    pattern: |
+      class $CONFIG(vf.EnvConfig):
+          ...
 
   - id: verifiers-v1-no-config-object-defaults
     languages: [python]
@@ -159,59 +80,9 @@ rules:
         - /environments/**/*.py
       exclude:
         - /verifiers/v1/types.py
-        - /environments/openenv_*/proj/**/*.py
+        - /environments/**/proj/**/*.py
     pattern-regex: "\\bAny\\b"
 
-  - id: verifiers-no-raw-object-containers-v1
-    languages: [python]
-    severity: ERROR
-    message: Do not spell broad object containers in v1 or environment code; use concrete config models, ConfigData, Handler, BindingsConfig, Objects, or a narrower type. Do not patch around weak typing with generic containers.
-    paths:
-      include:
-        - /verifiers/v1/**/*.py
-        - /environments/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-        - /verifiers/v1/utils/object_utils.py
-        - /verifiers/v1/utils/task_freeze_utils.py
-        - /environments/openenv_*/proj/**/*.py
-    pattern-regex: "(?x)(\\b(?:Mapping|MutableMapping|dict|list|Sequence|Iterable|Callable|Awaitable|tuple)\\[[^\\n\\]]*\\bobject\\b|\\bcast\\([^\\n)]*\\bobject\\b)"
-
-  - id: verifiers-no-raw-mapping-annotations-v1
-    languages: [python]
-    severity: ERROR
-    message: Do not use raw Mapping annotations in v1 or environment code; normalize at the boundary to a dict, a typed Config model, or a named v1 boundary type. Do not patch around unclear contracts with loose Mapping types.
-    paths:
-      include:
-        - /verifiers/v1/**/*.py
-        - /environments/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-        - /environments/openenv_*/proj/**/*.py
-    pattern-regex: "\\b(?:Mapping|MutableMapping)\\["
-
-  - id: verifiers-v1-package-no-mapping
-    languages: [python]
-    severity: ERROR
-    message: Package taskset/harness implementations should not use Mapping as a loose shape escape hatch; use concrete dict payloads, typed Config models, or package utils that encode a real protocol boundary.
-    paths:
-      include:
-        - /packages/tasksets/tasksets/**/*.py
-        - /packages/harnesses/harnesses/**/*.py
-        - /verifiers/v1/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-    pattern-regex: "\\b(?:Mapping|MutableMapping)\\b"
-
-  - id: verifiers-get-messages-typed
-    languages: [python]
-    severity: ERROR
-    message: "`get_messages` must return typed `Message` objects, not raw dictionaries."
-    paths:
-      include:
-        - /verifiers/utils/message_utils.py
-    pattern-regex: "def\\s+get_messages\\b[\\s\\S]*?->\\s*list\\s*\\[\\s*dict\\b"
-
   - id: verifiers-v1-no-signal-factories
     languages: [python]
     severity: ERROR
@@ -350,7 +221,7 @@ rules:
   - id: verifiers-v1-no-env-var-reads-in-signals
     languages: [python]
     severity: ERROR
-    message: v1 reward/update handlers must not read env vars directly; use state.get_client(api="chat") and optionally read state.get_endpoint_config(api="chat").model for model selection.
+    message: v1 reward/update handlers must not read env vars directly; put model/backend settings on typed configs and resolve clients through the harness runtime.
     paths:
       include:
         - /environments/**/*.py
@@ -503,24 +374,6 @@ rules:
               os.environ[$KEY]
               ...
 
-  - id: verifiers-v1-no-judge-endpoint-config-fields
-    languages: [python]
-    severity: ERROR
-    message: Judge rewards must use state.get_endpoint_config(api="chat"); expose only judge_model, not judge_base_url or judge_api_key_var config fields.
-    paths:
-      include:
-        - /environments/**/*.py
-      exclude:
-        - /environments/**/proj/**/*.py
-        - /environments/**/tasks/**/*.py
-    patterns:
-      - pattern-inside: |
-          class $CONFIG(vf.TasksetConfig):
-              ...
-      - pattern-either:
-          - pattern: "judge_base_url: $TYPE = $VALUE"
-          - pattern: "judge_api_key_var: $TYPE = $VALUE"
-
   - id: verifiers-v1-no-raw-kwargs-in-subclass-init
     languages: [python]
     severity: ERROR
@@ -572,27 +425,3 @@ rules:
       - pattern: |
           def __init__(...):
               ...
-
-  - id: verifiers-v1-package-no-static-classmethod-helpers
-    languages: [python]
-    severity: ERROR
-    message: Package taskset/harness implementations should use direct instance methods and standard lifecycle hooks; do not patch around weak ownership with staticmethod/classmethod helpers.
-    paths:
-      include:
-        - /packages/tasksets/tasksets/*.py
-        - /packages/harnesses/harnesses/*.py
-    pattern-either:
-      - pattern: "@staticmethod"
-      - pattern: "@classmethod"
-
-  - id: verifiers-v1-no-transcript-user-callback-param
-    languages: [python]
-    severity: ERROR
-    message: Do not use transcript as a user callback parameter name in v1 environments; use messages for the rendered prompt/completion messages.
-    paths:
-      include:
-        - /environments/**/*.py
-      exclude:
-        - /environments/**/proj/**/*.py
-        - /environments/**/tasks/**/*.py
-    pattern-regex: "def\\s+[A-Za-z_][A-Za-z0-9_]*\\s*\\([^)]*\\btranscript\\b"

AGENTS.md

@@ -9,7 +9,7 @@ These points are direct restatements of Verifiers docs so agents can follow the
 - Environments are expected to expose `load_environment(...) -> vf.Environment` and be installable with `prime env install <env-name>`. (See `docs/overview.md` and `docs/environments.md`.)
 - Validate environment behavior with `prime eval run <env-name> ...` before sharing/publishing changes. Treat `prime eval run` as the canonical eval path: it saves results automatically, and agents should not add opt-out flags such as `--skip-upload` unless the user explicitly requests that deviation so runs stay visible in the private Evaluations tab and in `prime eval view`. (See `docs/overview.md` and `docs/development.md`.)
 - Agents should assume they are allowed to make live model calls through the user's authenticated Prime CLI when a live smoke test is useful. For Prime Inference models, use `prime eval run <env-name>` with the base eval configuration from the environment's `pyproject.toml`; do not edit that `pyproject.toml`, and do not add model/config flags unless the task truly requires them. Agents do not need to manage API keys. If sandboxing blocks outbound requests, request elevated permissions for `prime eval run`, preferably as an ongoing approval instead of per run.
-- For new taskset/harness environments, start with `prime env init <name> --v1` or `prime env init <name> --v1 --with-harness`. Edit the generated `TasksetConfig` for task settings, `Taskset.load_tasks()` for train/eval task records, `Taskset.load_toolsets()` for task-owned tools, `User` subclasses for user behavior, and `@vf.*` methods for lifecycle, metrics, rewards, and advantages. Keep the generated `load_taskset(config: MyTasksetConfig)`, optional `load_harness(config: MyHarnessConfig)`, and `load_environment(config: vf.EnvConfig)` shapes as the component entrypoints. Environment READMEs must use the generated `prime env init` section structure; freeform environment READMEs are not allowed. Treat [BYO Harness](docs/byo-harness.md) as the canonical authoring guide for reusable tasksets, reusable harnesses, framework programs, endpoint interception, and sandboxed Python/command programs.
+- For new taskset/harness environments, start with `prime env init <name> --v1` or `prime env init <name> --v1 --with-harness`. Edit the generated `TasksetConfig` for task settings and `ToolsetConfig` entries, `Taskset.load_tasks()` for train/eval task records, `servers/toolset.py` for task-owned tools, `servers/user.py` for user behavior, and `@vf.*` methods for lifecycle, metrics, rewards, and advantages. Keep `taskset.py` with `load_taskset(config: MyTasksetConfig)` and optional `harness.py` with `load_harness(config: MyHarnessConfig)` as the component entrypoints. Environment READMEs must use the generated `prime env init` section structure; freeform environment READMEs are not allowed. Treat [BYO Harness](docs/byo-harness.md) as the canonical authoring guide for reusable tasksets, reusable harnesses, framework programs, endpoint interception, and sandboxed Python/command programs.
 - Use `ToolEnv`/`MCPEnv` for stateless tools and `StatefulToolEnv` when per-rollout state must persist (sandbox/session/db handles). (See `docs/environments.md`.)
 - If external API keys are required, validate them in `load_environment()` with `vf.ensure_keys(...)` so failures are explicit and early. (See `docs/environments.md`.)
 
@@ -21,7 +21,7 @@ Use these rules when shaping user-facing Verifiers APIs, configs, and environmen
 - Keep user-facing APIs incredibly minimal and elegant. The best surface is usually golfy but intuitive: one obvious field, one obvious constructor, and no redundant knobs unless there is a concrete long-term reason.
 - Use Pydantic config models wherever structured configuration is needed. Pydantic is always acceptable and preferred over loose dictionaries when it clarifies the contract.
 - Prefer strict, narrow types. Use `object`, broad unions, or untyped mappings only at explicit framework boundaries where arbitrary user values are genuinely part of the contract.
-- Basic v1 environments should fit in a few dozen self-contained, idiomatic lines: import `verifiers as vf`, define typed taskset/harness config classes when needed, keep policy values in config subclasses, and put implementation logic on the owning taskset or harness class. Static prompts should usually be `system_prompt` config fields; override `load_system_prompt` only for computed prompt loading. Use bindings for shared resources owned by tasksets, toolsets, users, programs, or harnesses; object entries should be loader specs, not pre-initialized resources.
+- Basic v1 environments should fit in a few dozen self-contained, idiomatic lines: import `verifiers.v1 as vf`, define typed taskset/harness/toolset config classes when needed, keep policy values in config subclasses, and put implementation logic on the owning taskset, harness, toolset, or user class. Static prompts should usually be `system_prompt` config fields; override `load_system_prompt` only for computed prompt loading. Configure tools and users with `ToolsetConfig.loader` / `UserConfig.loader`; pass serializable config, not pre-initialized resources.
 - Avoid module globals. Acceptable globals are imports, immutable literals, factory functions, and carefully managed process-level resource constraints such as locks or semaphores. Put all other behavior and state in well-named utility modules, taskset/harness classes, toolsets, users, programs, or user code.
 - Additional code should have a clear home. Do not hide utilities at the bottom of files or scatter one-off helpers through environment entrypoints.
 
@@ -39,7 +39,7 @@ Use this guidance when contributing to the `verifiers` repository itself.
 - Before v0.2.0, breaking backward compatibility inside v1 Taskset/Harness APIs is acceptable and encouraged when it improves the core design. Preserve v0 multi-turn environment compatibility unless the user explicitly asks for a v0 migration.
 - Treat public configuration and docs as part of the API. Keep TOML shapes consistent across eval, GEPA, RL, and Hosted Training; normalize legacy inputs at the ingestion boundary instead of spreading compatibility branches through examples.
 - For v1 Taskset/Harness work, make the taskset own task data, task controls, task tools, user behavior, metrics, rewards, and task-specific configuration. Make the harness own reusable execution mechanisms such as programs, command agents, primary sandboxes, endpoint interception, framework adapters, and execution artifacts. Use the base `vf.Harness` unless the harness really owns such a mechanism.
-- Keep v1 construction explicit: `vf.Env` receives concrete taskset/harness objects, while `load_taskset(config: MyTasksetConfig)` and `load_harness(config: MyHarnessConfig)` define child config types. `EnvConfig` only carries the two child configs; ordinary environment packages use the generated `load_environment(config: vf.EnvConfig)` shim.
+- Keep v1 construction explicit: `vf.Env` receives concrete taskset/harness objects, while `taskset.py` / `harness.py` expose `load_taskset(config: MyTasksetConfig)` and `load_harness(config: MyHarnessConfig)` to define child config types. `EnvConfig` only carries the two child configs; the package loader assembles `vf.Env` from those components.
 - Put class-owned behavior on the taskset or harness class through config fields, `load_*` methods, `User` subclasses, `Toolset`, and `@vf.*` lifecycle methods. `load_taskset` and `load_harness` provide the typed entrypoints to those classes.
 - Do not override `Taskset.__init__`, `Harness.__init__`, or `User.__init__` in v1 implementations. Put initialization policy in config fields, public load methods, lifecycle handlers, task rows, `Toolset`, `User.get_response`, or utility modules when genuinely shared.
 - Do not add one-off private helper methods or bottom-of-file helper functions to make taskset/harness classes look shorter. Core lifecycle logic should live on the class with standard public method names or `@vf.*` decorators; reusable multi-line plumbing belongs in a named utility module.