V1 runtime/taskset/harness refactor

.semgrep/verifiers.yml

@@ -1,10 +1,4 @@
 rules:
-  - id: verifiers-no-future-annotations
-    languages: [python]
-    severity: ERROR
-    message: Do not use `from __future__ import annotations`; quote only the specific forward references that need it.
-    pattern: from __future__ import annotations
-
   - id: verifiers-no-skip-validation
     languages: [python]
     severity: ERROR
@@ -40,89 +34,16 @@ rules:
               metavariable: $ANNOT
               regex: "(Any|Mapping\\[str, object\\]|dict\\[str, object\\]|.*\\|.*\\|.*)"
 
-  - id: verifiers-v1-load-environment-config-required
+  - id: verifiers-v1-no-env-config-subclasses
     languages: [python]
     severity: ERROR
-    message: v1 load_environment must take a strict config object; do not accept None or synthesize config with config-or-Config fallback.
+    message: EnvConfig is final. Put defaults and public settings on TasksetConfig and HarnessConfig; the library loader owns vf.Env construction.
     paths:
       include:
         - /environments/**/*.py
-    pattern-either:
-      - pattern: |
-          def load_environment(..., config: $CONFIG = None, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: $CONFIG | None, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: $CONFIG | None = $DEFAULT, ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: Optional[$CONFIG], ...):
-              ...
-      - pattern: |
-          def load_environment(..., config: Optional[$CONFIG] = $DEFAULT, ...):
-              ...
-
-  - id: verifiers-v1-no-load-environment-config-fallback
-    languages: [python]
-    severity: ERROR
-    message: v1 load_environment receives a concrete config object from the framework; do not use config or Config() fallback.
-    paths:
-      include:
-        - /environments/**/*.py
-    patterns:
-      - pattern-inside: |
-          def load_environment(...):
-              ...
-      - pattern: $CONFIG = $CONFIG or $DEFAULT()
-      - metavariable-regex:
-          metavariable: $DEFAULT
-          regex: ".*Config"
-
-  - id: verifiers-v1-load-environment-canonical-shim
-    languages: [python]
-    severity: ERROR
-    message: v1 load_environment is only the Taskset/Harness loader shim. Keep load_environment as-is; implement the config surface through TasksetConfig, HarnessConfig, load_taskset, and load_harness instead of patching root-loader behavior.
-    paths:
-      include:
-        - /environments/**/*.py
-        - /verifiers/scripts/init.py
-      exclude:
-        - /environments/alphabet_sort/alphabet_sort_v1.py
-        - /environments/bfcl_v3/bfcl_v3.py
-        - /environments/dspy_flights/dspy_flights.py
-        - /environments/dspy_rlm/dspy_rlm.py
-        - /environments/hello_group_reward_v1/hello_group_reward_v1.py
-        - /environments/hello_parallel_sandbox_v1/hello_parallel_sandbox_v1.py
-        - /environments/hello_self_judge_v1/hello_self_judge_v1.py
-        - /environments/hello_subagent_v1/hello_subagent_v1.py
-        - /environments/langchain_deep_agents_wikispeedia/langchain_deep_agents_wikispeedia.py
-        - /environments/math_python/math_python_v1.py
-        - /environments/mcp_search_env/mcp_search_env.py
-        - /environments/nemo_gym_env/nemo_gym_env/env.py
-        - /environments/nested_harness_v1/nested_harness_v1.py
-        - /environments/openai_agents_env/openai_agents_env.py
-        - /environments/rlm_swe_v1/rlm_swe_v1.py
-        - /environments/tau2_bench_v1/tau2_bench_v1.py
-        - /environments/wiki_search/wiki_search_v1.py
-    patterns:
-      - pattern: |
-          def load_environment(config: $CONFIG) -> vf.Env:
-              ...
-      - pattern-not: |
-          def load_environment(config: vf.EnvConfig) -> vf.Env:
-              return vf.Env(
-                  taskset=vf.load_taskset(config=config.taskset),
-                  harness=vf.load_harness(config=config.harness),
-              )
-      - pattern-not: |
-          def load_environment(config: vf.EnvConfig) -> vf.Env:
-              """Loader pattern for all Taskset/Harness environments."""
-              return vf.Env(
-                  taskset=vf.load_taskset(config=config.taskset),
-                  harness=vf.load_harness(config=config.harness),
-              )
+    pattern: |
+      class $CONFIG(vf.EnvConfig):
+          ...
 
   - id: verifiers-v1-no-config-object-defaults
     languages: [python]
@@ -159,50 +80,9 @@ rules:
         - /environments/**/*.py
       exclude:
         - /verifiers/v1/types.py
-        - /environments/openenv_*/proj/**/*.py
+        - /environments/**/proj/**/*.py
     pattern-regex: "\\bAny\\b"
 
-  - id: verifiers-no-raw-object-containers-v1
-    languages: [python]
-    severity: ERROR
-    message: Do not spell broad object containers in v1 or environment code; use concrete config models, ConfigData, Handler, BindingsConfig, Objects, or a narrower type. Do not patch around weak typing with generic containers.
-    paths:
-      include:
-        - /verifiers/v1/**/*.py
-        - /environments/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-        - /verifiers/v1/utils/object_utils.py
-        - /verifiers/v1/utils/task_freeze_utils.py
-        - /environments/openenv_*/proj/**/*.py
-    pattern-regex: "(?x)(\\b(?:Mapping|MutableMapping|dict|list|Sequence|Iterable|Callable|Awaitable|tuple)\\[[^\\n\\]]*\\bobject\\b|\\bcast\\([^\\n)]*\\bobject\\b)"
-
-  - id: verifiers-no-raw-mapping-annotations-v1
-    languages: [python]
-    severity: ERROR
-    message: Do not use raw Mapping annotations in v1 or environment code; normalize at the boundary to a dict, a typed Config model, or a named v1 boundary type. Do not patch around unclear contracts with loose Mapping types.
-    paths:
-      include:
-        - /verifiers/v1/**/*.py
-        - /environments/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-        - /environments/openenv_*/proj/**/*.py
-    pattern-regex: "\\b(?:Mapping|MutableMapping)\\["
-
-  - id: verifiers-v1-package-no-mapping
-    languages: [python]
-    severity: ERROR
-    message: Package taskset/harness implementations should not use Mapping as a loose shape escape hatch; use concrete dict payloads, typed Config models, or package utils that encode a real protocol boundary.
-    paths:
-      include:
-        - /packages/tasksets/tasksets/**/*.py
-        - /packages/harnesses/harnesses/**/*.py
-        - /verifiers/v1/**/*.py
-      exclude:
-        - /verifiers/v1/types.py
-    pattern-regex: "\\b(?:Mapping|MutableMapping)\\b"
-
   - id: verifiers-get-messages-typed
     languages: [python]
     severity: ERROR
@@ -350,7 +230,7 @@ rules:
   - id: verifiers-v1-no-env-var-reads-in-signals
     languages: [python]
     severity: ERROR
-    message: v1 reward/update handlers must not read env vars directly; use state.get_client(api="chat") and optionally read state.get_endpoint_config(api="chat").model for model selection.
+    message: v1 reward/update handlers must not read env vars directly; put model/backend settings on typed configs and resolve clients through the harness runtime.
     paths:
       include:
         - /environments/**/*.py
@@ -506,7 +386,7 @@ rules:
   - id: verifiers-v1-no-judge-endpoint-config-fields
     languages: [python]
     severity: ERROR
-    message: Judge rewards must use state.get_endpoint_config(api="chat"); expose only judge_model, not judge_base_url or judge_api_key_var config fields.
+    message: Judge rewards should use typed model config; do not expose raw judge_base_url or judge_api_key_var fields.
     paths:
       include:
         - /environments/**/*.py
@@ -584,15 +464,3 @@ rules:
     pattern-either:
       - pattern: "@staticmethod"
       - pattern: "@classmethod"
-
-  - id: verifiers-v1-no-transcript-user-callback-param
-    languages: [python]
-    severity: ERROR
-    message: Do not use transcript as a user callback parameter name in v1 environments; use messages for the rendered prompt/completion messages.
-    paths:
-      include:
-        - /environments/**/*.py
-      exclude:
-        - /environments/**/proj/**/*.py
-        - /environments/**/tasks/**/*.py
-    pattern-regex: "def\\s+[A-Za-z_][A-Za-z0-9_]*\\s*\\([^)]*\\btranscript\\b"

environments/README.md

@@ -23,8 +23,8 @@ This folder contains installable example environments that showcase common usage
 - **sentence_repeater**: Multi-turn Q/A over a paragraph; rewards compare assistant messages to expected answers.
 - **wordle**: Game-style interaction via `TextArenaEnv`; multiple rewards (correctness, partial credit, few-turn bonus) and XML formatting.
 - **wordle_v1**: Wordle on the reusable v1 `TextArenaTaskset`, with Wordle-specific prompt, feedback, and rewards kept in the environment package.
-- **openenv_echo**: OpenEnv MCP integration example using upstream `echo_env`.
-- **openenv_textarena**: OpenEnv gym integration example using upstream `textarena_env` (default `Wordle-v0`).
+- **openenv_echo_v1**: OpenEnv MCP integration example using upstream `echo_env`.
+- **openenv_textarena_v1**: OpenEnv gym integration example using upstream `textarena_env` (default `Wordle-v0`).
 
 ### Tool use
 - **ToolEnv (native function-calling)**
@@ -40,26 +40,26 @@ This folder contains installable example environments that showcase common usage
 
 ### Experimental environments
 - **MCPEnv (MCP server integration)**
-  - **mcp_search_env**: Example environment demonstrating `vf.MCPEnv` for Model Context Protocol server integration.
+  - **mcp_search_env_v1**: Example environment demonstrating `vf.MCPEnv` for Model Context Protocol server integration.
 
 - **RLM (Recursive Language Model)**
   - **hello_rlm_v1**: v1 packaged `RLM` harness example with endpoint interception and metrics collection.
 
 - **V1 Taskset/Harness**
-  - **dspy_rlm**: DSPy RLM harness on GSM8K through `vf.Env`; DSPy uses the V1 interception endpoint from rollout state.
-  - **openai_agents_env**: OpenAI Agents SDK harness with a calculator tool on GSM8K through `vf.Env`.
-  - **langchain_deep_agents_wikispeedia**: LangChain Deep Agents harness on Wikispeedia navigation, where tool use is load-bearing.
+  - **dspy_rlm_v1**: DSPy RLM harness on GSM8K through `vf.Env`; DSPy uses the V1 interception endpoint from rollout state.
+  - **openai_agents_env_v1**: OpenAI Agents SDK harness with a calculator tool on GSM8K through `vf.Env`.
+  - **langchain_deep_agents_wikispeedia_v1**: LangChain Deep Agents harness on Wikispeedia navigation, where tool use is load-bearing.
 
 - **HarborEnv / CliAgentEnv (agent sandboxes)**
-  - **opencode_harbor**: Runs the OpenCode CLI agent on Harbor tasks with API interception via Prime Tunnel.
+  - **opencode_harbor_v1**: Runs the OpenCode CLI agent on Harbor tasks with API interception via Prime Tunnel.
   - **terminus_harbor**: Runs the Terminus agent on Harbor tasks with API interception via Prime Tunnel.
   - **hello_mcp_harbor**: Smallest runnable `HarborEnv` exercising framework-managed MCP server lifecycle (FastMCP `get_secret` server + OpenCode agent).
 
 - **Taskset/Harness v1**
-  - **bfcl_v3**: BFCL v3 function-calling eval using task-local dynamic tool schemas and v1 rewards.
-  - **dspy_flights**: Sandboxed DSPy flight-support `program.fn` entrypoint installed from its package `pyproject.toml` and configured against the v1 interception endpoint.
+  - **bfcl_v3_v1**: BFCL v3 function-calling eval using task-local dynamic tool schemas and v1 rewards.
+  - **dspy_flights_v1**: Sandboxed DSPy flight-support `program.fn` entrypoint installed from its package `pyproject.toml` and configured against the v1 interception endpoint.
   - **hello_group_reward_v1**: Deterministic v1 reference for group updates, metrics, rewards, advantages, and cleanup.
-  - **nemo_gym_env**: Minimal v1 example that wraps a packaged NeMo Gym task with `NeMoGymTaskset` and `NeMoGymHarness`.
+  - **nemo_gym_env_v1**: Minimal v1 example that wraps a packaged NeMo Gym task with `NeMoGymTaskset` and `NeMoGymHarness`.
   - **sft-replay**: Thin v1 replay environment using `ReplayTaskset` and `ReplayHarness` to turn stored transcripts into trajectory steps without model calls.
   - **tau2_bench_v1**: `tau2-bench-v1` τ²-bench taskset/user/tool pattern on the v1 harness runtime.
   - **wordle_v1**: TextArena Wordle through the packaged v1 `TextArenaTaskset` boundary.
@@ -87,17 +87,17 @@ This folder contains installable example environments that showcase common usage
 - **mmmu**: Demonstrates passing images via chat `content` items with `{type: "image_url", image_url: {url: ...}}` and standard answer parsing.
 
 ## What to look at for each pattern
-- **Minimal SingleTurnEnv**: `reverse_text`, `gsm8k`
+- **Minimal SingleTurnEnv**: `reverse_text_v1`, `gsm8k`
 - **JudgeRubric end-to-end**: `continuation_quality`, `toxicity_explanation`, `self_reward`
-- **ToolEnv with real tools**: `wiki_search`, `math_python`
-- **Custom MultiTurnEnv**: `alphabet_sort`, `doublecheck`, `sentence_repeater`, `wordle`
+- **ToolEnv with real tools**: `wiki_search_v1`, `math_python_v1`
+- **Custom MultiTurnEnv**: `alphabet_sort_v1`, `doublecheck`, `sentence_repeater`, `wordle`
 - **GymEnv integration**: `gem_wordle`
-- **OpenEnv integration (gym + MCP)**: `openenv_textarena`, `openenv_echo`
-- **CLI agent sandboxes**: `opencode_harbor`, `terminus_harbor`, `hello_mcp_harbor`
-- **MCP integration**: `mcp_search_env`, `hello_mcp_harbor`
-- **Taskset/Harness v1**: use this pattern for new environments that need reusable tasksets, reusable harnesses, framework programs, endpoint interception, or sandboxed Python/command programs. Examples include `dspy_rlm`, `openai_agents_env`, `langchain_deep_agents_wikispeedia`, `reverse_text`, `alphabet_sort`, `wiki_search`, `math_python`, `mcp_search_env`, `opencode_harbor`, `bfcl_v3`, `hello_subagent_v1`, `nested_harness_v1`, `hello_self_judge_v1`, `hello_parallel_sandbox_v1`, `hello_group_reward_v1`, `hello_rlm_v1`, `rlm_swe_v1`, `dspy_flights`, `tau2-bench-v1`, and `wordle-v1`.
-  - `opencode_harbor` uses the packaged `HarborTaskset` + `OpenCode` boundary from `tasksets` and `harnesses`.
-- **Environment and rubric composition**: `math_group`, `math_python`, `wiki_search`
+- **OpenEnv integration (gym + MCP)**: `openenv_textarena_v1`, `openenv_echo_v1`
+- **CLI agent sandboxes**: `opencode_harbor_v1`, `terminus_harbor`, `hello_mcp_harbor`
+- **MCP integration**: `mcp_search_env_v1`, `hello_mcp_harbor`
+- **Taskset/Harness v1**: use this pattern for new environments that need reusable tasksets, reusable harnesses, framework programs, endpoint interception, or sandboxed Python/command programs. Examples include `dspy_rlm_v1`, `openai_agents_env_v1`, `langchain_deep_agents_wikispeedia_v1`, `reverse_text_v1`, `alphabet_sort_v1`, `wiki_search_v1`, `math_python_v1`, `mcp_search_env_v1`, `opencode_harbor_v1`, `bfcl_v3_v1`, `hello_subagent_v1`, `nested_harness_v1`, `hello_self_judge_v1`, `hello_parallel_sandbox_v1`, `hello_group_reward_v1`, `hello_rlm_v1`, `rlm_swe_v1`, `dspy_flights_v1`, `tau2-bench-v1`, and `wordle-v1`.
+  - `opencode_harbor_v1` uses the packaged `HarborTaskset` + `OpenCode` boundary from `tasksets` and `harnesses`.
+- **Environment and rubric composition**: `math_group`, `math_python_v1`, `wiki_search_v1`
 - **Procedural datasets**: `reasoning_gym_env`
 - **Multimodal**: `mmmu`
 

environments/alphabet_sort/alphabet_sort.py

@@ -200,35 +200,11 @@ def load_environment(
     dataset_name: str = "kalomaze/alphabetic-arxiv-authors-it1",
     dataset_split: str = "train",
     seed: int = 1337420,
-    v1: bool = False,
     **kwargs,
 ) -> vf.Environment:
-    if v1:
-        if kwargs:
-            unexpected = ", ".join(sorted(kwargs))
-            raise TypeError(f"Unsupported v1 load_environment kwargs: {unexpected}")
-
-        from alphabet_sort_v1 import (
-            AlphabetSortEnvConfig,
-            AlphabetSortTasksetConfig,
-            load_environment as load_v1,
-        )
-
-        return load_v1(
-            config=AlphabetSortEnvConfig(
-                taskset=AlphabetSortTasksetConfig(
-                    max_turns=max_turns,
-                    min_turns=min_turns,
-                    min_names_per_turn=min_names_per_turn,
-                    max_names_per_turn=max_names_per_turn,
-                    similarity_power=similarity_power,
-                    power_per_turn=power_per_turn,
-                    dataset_name=dataset_name,
-                    dataset_split=dataset_split,
-                    seed=seed,
-                )
-            )
-        )
+    if kwargs:
+        unexpected = ", ".join(sorted(kwargs))
+        raise TypeError(f"Unsupported load_environment kwargs: {unexpected}")
 
     assert min_turns >= 1, "min_turns must be at least 1"
     assert min_turns <= max_turns, "min_turns must be less than or equal to max_turns"

environments/alphabet_sort/pyproject.toml

@@ -13,7 +13,7 @@ requires = ["hatchling"]
 build-backend = "hatchling.build"
 
 [tool.hatch.build]
-include = ["alphabet_sort.py", "alphabet_sort_v1.py"]
+include = ["alphabet_sort.py"]
 
 [tool.verifiers.eval]
 num_examples = 5