feat(mini_swe_env): add SWE-Gym async GRPO environment with Pi interception and HF Space deployment

[rycerzes]

Summary

This PR adds a new mini_swe_env and a complete async RL training path for SWE tasks: Pi agent in sandbox, host-side interception, vLLM generation, and Async GRPO updates. It is self-contained for review against feat/multi-harness and supports both direct black-box rollouts and interception-driven async training. The goal is to make SWE-Gym-style training runnable end-to-end on local Docker and HF Space + HF Sandbox.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• New environment
• Refactoring

Alignment Checklist

Before submitting, verify:

• I have read .claude/docs/PRINCIPLES.md and this PR aligns with our principles
• I have checked .claude/docs/INVARIANTS.md and no invariants are violated
• I have run /pre-submit-pr (or bash .claude/hooks/lint.sh and tests) and addressed all issues

---

What this PR does (self-contained)

1) Adds a new SWE environment package

New package: envs/mini_swe_env/

• SWE task models + SWE-Gym loader (models.py, task_loader_swegym.py)
• host-side grading (grading.py) with binary resolved/not-resolved semantics
• harness integration (harness.py) for Pi/OpenCode sessions
• server + MCP environment (server/swe_environment.py, server/app.py)
• client (client.py) and env package metadata (pyproject.toml, uv.lock)

2) Supports both rollout modes needed by SWE workflows

• black_box mode: agent calls upstream LLM directly
• interception_gate mode: trainer owns generation + logprob capture

3) Adds async GRPO control plane + rollout worker

New modules in envs/mini_swe_env/async_grpo/:

• control_plane.py — starts/owns interception server lifecycle
• rollout_worker.py — custom worker implementing TRL rollout protocol
  • consumes intercepted chat requests
  • tokenizes with apply_chat_template
  • calls vLLM /v1/completions with return_token_ids=True, logprobs=0
  • delivers chat responses back to Pi
  • assembles input_ids, completion_mask, old_log_probs for trainer

4) Adds runnable training/examples and HF Space deploy path

• examples/mini_swe_env/train_swe_async_grpo.py
• examples/mini_swe_env/run_swe_sample.py
• HF Space app assets:
  • envs/mini_swe_env/async_grpo/space_app/Dockerfile
  • envs/mini_swe_env/async_grpo/space_app/start.sh
  • envs/mini_swe_env/async_grpo/space_app/deploy_hf_space.sh
  • envs/mini_swe_env/async_grpo/space_app/README.md

5) Core harness compatibility updates used by this env

• src/openenv/core/harness/agents/cli_driver.py updated for interception + queue handling needed by async rollouts.

---

Execution modes

Area	black_box mode	interception_gate mode
Agent LLM calls	Direct to upstream provider	Routed through host InterceptionServer
Training signal	No token/logprob capture	Exact token IDs + logprobs captured via vLLM
Main use	Eval / smoke / demos	Async GRPO training
Host tools	Optional	Required (answer host-side)
Endpoints	provider-specific	/rollout/{id}/v1/chat/completions, /rollout/{id}/v1/tools/answer, internal vLLM /v1/completions

---

Network diagram (async training path)

flowchart LR
    subgraph SPACE[HF Space / Trainer Host]
        IS["InterceptionServer<br/>0.0.0.0:7860<br/>/rollout/:rollout_id/v1/chat/completions<br/>/rollout/:rollout_id/v1/tools/answer"]
        RW["SWERolloutWorker<br/>threads + queues"]
        TR["AsyncGRPOTrainer"]
        VLLM["vLLM<br/>127.0.0.1:8000<br/>/v1/completions"]

        IS <--> RW
        RW --> TR
        RW <--> VLLM
        TR -. weight sync .-> VLLM
    end

    SB["Sandbox (HF or Docker)<br/>Pi agent + /testbed"] -->|chat completion request| IS
    SB -->|answer tool call| IS
    RW -->|create/exec/kill sandboxes| HFAPI["HF Sandbox API"]

Loading

black_box mode at a glance

Pi agent (sandbox) -> upstream LLM directly
verify() runs at end; no host-side token/logprob interception loop

---

Reward and grading integrity

• Reward source is host-side grading path.
• answer() is routed to host tool handler (not sandbox-owned grader).
• Grading uses SWE-Gym-style case outcomes (FAIL_TO_PASS, PASS_TO_PASS) with binary reward 1.0/0.0.
• This keeps reward authority on host and aligns with OpenEnv invariants.

Note

This PR depends and should be merged after #694

CC: @burtenshaw

[rycerzes]

Thanks @AmineDiro for the detailed review. It helped me catch a few core issues. Addressing each point:

1. The weight-transfer

All send_weights/pause/resume/_init_weight_transfer NCCL setups are basically unreachable. Injecting a worker sets self.weight_transfer = None and skips every sync path so vLLM never receives updated weights.

I verified this, weight sync does happen. No code change needed, but I added a clarifying comment. The trainer's _sync_weight() calls self.rollout_worker.send_weights() regardless of injection, so my SWERolloutWorker.send_weights() is always reached. The hardcoded language_model. prefix remap is necessary due to a VLM param-name mismatch that TRL's default sync doesn't handle. I'll propose this fix upstream.

2. The loss is REINFORCE, not GRPO

advantage = reward = 0/1 and you have no baseline, and no group-relative normalization.

Thank you for catching this, I already had this fixed but havent pushed yet. I implemented proper group-relative normalization: accumulate N rollouts per task, compute advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8), exclude overflow/timeout samples. Bumped gradient_accumulation_steps to 4 and weight_decay to 0.1 to match Polar's hyperparameters.

3. Import helpers from TRL

Subtle logic in tool-suffix tokenization / multiturn handling / token length clamping is tricky. I would advise reusing the logic.

Fixed for suffix/template. I now import TRL's get_training_chat_template and is_chat_template_prefix_preserving, which resolved a silent bug where Qwen3.5's <think> tokens misaligned. For length clamping, I use progressive truncation before tokenization rather than post-hoc clamping.

---

Staleness: staleness handling matches Polar's approach for now but this results in wasted sandbox compute. I will visit this at a later revision.

Duplicated RolloutSample: TRL's version includes fields my pipeline doesn't use; my local version covers what RolloutQueueDataset actually reads, but I can update it as well if needed

Defensive scaffolding: Fair point. The getattr and try/except is higher than it should be — a lot of it accrued while debugging Qwen3.5-specific crashes. I will do a pass to tighten these up.

Reinvent turn-termination heuristics: The post_response_grace_s idle-stop is needed for remote sandboxes where process exit isn't synchronously observable. Documented inline.

[rycerzes]

quick summary of what changed after the GRPO normalization work:

• moved to proper group-normalized GRPO with configurable num_generations (default 16)
• fixed Qwen3.5/vLLM compatibility issues (weight sync + chat-template/tool-call tokenization edge cases)
• hardened long-run stability with better context-window handling (trimming + overflow guards) so rollouts degrade gracefully instead of looping/crashing
• integrated your key harness stabilizations: reward mode toggle, stronger prompt/instructions, anti-test-edit protections, and configurable git checkout timeout
• improved control-plane reliability with explicit agent exit signaling so workers don’t sit on long idle timeouts
• stabilized training stack for Spaces with multi-GPU/LoRA path, FSDP2-safe optimizer settings, and kernel/dependency fixes

[Darktex]

Automated Alignment Review — PR #695

Reviewer: Claude Code (alignment-reviewer)
Scope: 77 changed files across 4 review areas: core harness infrastructure, mini_swe_env, training examples/deployment, and coding_agent_env refactor.
Verdict: REQUEST_CHANGES — 4 critical issues, 6 high-severity issues, and 5 alignment flags requiring human decision.

---

Critical Issues (must fix before merge)

1. api_key exposed in MCP tool schema — envs/mini_swe_env/server/swe_environment.py:120-140
   FastMCP generates JSON Schema from the run_swe_rollout function signature, so api_key appears in tools/list responses visible to any MCP client (including agents). Any MCP-level logging captures the key in plaintext. Fix: remove api_key/base_url from the tool signature; read from env vars server-side, matching the pattern used everywhere else in the codebase. [Violates: No credential exposure — INVARIANTS.md §3]

2. HF_TOKEN shell injection in deploy script — examples/mini_swe_env/async_grpo/space_app/deploy_hf_space.sh:114,125
HF_TOKEN is interpolated directly into a Python -c string literal (api = HfApi(token='$HF_TOKEN')). A token containing a single quote causes a syntax error; the literal token also appears in ps aux output. The same script correctly uses os.environ['HF_TOKEN'] elsewhere (lines 157, 212, 339). Fix: use a heredoc or env var read for the --pause/--resume blocks.

3. _put_queue_threadsafe broken on Python 3.10+ — src/openenv/core/harness/agents/interception_server.py:550-566
   The fallback path calls q.put_nowait(item) on an asyncio.Queue from a non-event-loop thread when getattr(q, "_loop", None) returns None (which it does on Python 3.10+). This silently corrupts internal queue state. Fix: store the event loop reference at creation time and use loop.call_soon_threadsafe(q.put_nowait, item).

4. TOCTOU race in InterceptionServer — src/openenv/core/harness/agents/interception_server.py:456-462
_handle_chat_completions checks for rollout context inside _state_lock, exits the lock, then re-enters to write the intercept. Between the two acquisitions, unregister_rollout could remove the entry, leaving a stranded intercept. Fix: do the entire context-fetch, intercept-write, and queue-put under a single lock acquisition.

High-Severity Issues

5. API_KEY string interpolation — examples/mini_swe_env/deploy_collect_space.sh:175
$API_KEY is expanded by shell before Python sees it in an unquoted heredoc. Tokens with \, ", or $ corrupt the Python string. Fix: pass as env var, read with os.environ.

6. CPU Dockerfile runs as root — examples/mini_swe_env/deploy_collect_space.sh (inline Dockerfile ~line 281)
   No USER directive. The GPU Dockerfile (space_app/Dockerfile:56) correctly switches to uid 1000. HF Spaces strongly recommends non-root. Fix: add USER 1000.

7. DockerSandboxBackend start_bg lacks shell quoting — src/openenv/core/harness/sandbox/docker_backend.py:131-148
cmd is embedded directly in a bash -c wrapper without shell_quote. Commands containing single quotes or special characters break the wrapper. Fix: apply shell_quote to cmd in the wrapper string.

8. state property returns Any — envs/mini_swe_env/server/swe_environment.py:226
   Should be -> SWEState to maintain generic type safety per INVARIANTS.md API Invariants §2.

9. Docstring/implementation mismatch — envs/mini_swe_env/task_loader_swegym.py
   Docstring says __ is replaced with "_1776_" but implementation uses "_s_". Fix: update docstring.

10. Dataclasses inside method bodies — envs/mini_swe_env/server/swe_environment.py:707-732
_AgentConfig and _AgentTask are created inside method bodies, accumulating stale class objects on every call. Fix: move to module level.

Medium-Severity Issues

11. README/script variable mismatch — README says SWE_ASYNC_MODEL but train_swe_async_grpo.py:233 reads SWE_MODEL. Every user following the README will fail immediately.
12. Missing openenv.yaml — envs/mini_swe_env/ lacks this file, required for deployment tooling and CI discovery.
13. Missing server/Dockerfile — envs/mini_swe_env/server/ has no Dockerfile for standalone environment deployment.
14. JSON-RPC notification response — envs/mini_swe_env/server/sandbox_mcp_server.py:263-266 sends a response to notifications/initialized in HTTP mode, violating JSON-RPC 2.0 spec.
15. Unused parameters — envs/coding_agent_env/server/coding_environment.py:490 _build_session_factory accepts disable_thinking, top_logprobs, max_tokens_cap but never uses them. max_tokens_cap/top_logprobs silently dropped for Pi agent without warning.
16. Private attribute coupling — envs/coding_agent_env/harness.py:138,143,161 accesses CLIAgentDriver._interception_server and ._interception_base_url directly. Fix: promote to public properties or delegate through driver's public API.
17. Duplicated grading logic — _apply_test_patch, _run_swegym_case_tests, etc. appear in both swe_environment.py and harness.py with near-identical implementations. A fix in one won't reach the other.
18. Missing None guard on logprobs — examples/mini_swe_env/async_grpo/rollout_worker.py:651 — choice["logprobs"]["token_logprobs"] accessed without a None guard.

---

Alignment Flags (human decision needed)

FLAG 1: Two parallel harness frameworks (openenv.core.harness vs openenv.core.harnesses) with nearly identical names and incompatible abstractions. The PR should either merge these or clearly mark one as superseding the other.
Principle: One canonical way to build environments

FLAG 2: Server-wide bearer secret written into sandbox environment variables as OPENAI_API_KEY/ANTHROPIC_API_KEY and into Pi models.json on the sandbox filesystem (cli_driver.py:508-509,528). Any process in the sandbox can read these. Consider per-sandbox, short-lived tokens rotated after each rollout.
Principle: No credential exposure — INVARIANTS.md §3

FLAG 3: Reward computation in harness.py runs on the host process (outside the environment container), triggered by the InterceptionServer's _answer_handler. The grading path is also duplicated between swe_environment.py (in-container) and harness.py (host-side), creating two independently-evolvable reward definitions.
Principle: Rewards inside environment — RFC 002

FLAG 4: _noop_reward in train_swe_async_grpo.py bypasses TRL's standard reward_funcs mechanism entirely, carrying the actual reward via RolloutSample.advantage. Need confirmation that TRL's AsyncGRPOTrainer reads advantage as gradient signal and ignores the zero-valued reward_funcs output.
Principle: Rewards inside environment — RFC 002

FLAG 5: RESERVED_TOOL_NAMES defined in two places (openenv.core.harness/__init__.py:27 and openenv.core.env_server/mcp_types.py:321). No single source of truth — if someone adds a reserved name in one location but not the other, the invariant silently breaks.
Principle: Agents cannot reset — INVARIANTS.md §1

---

What's Good

• The opencode_env → coding_agent_env rename is clean — no stale references remain
• RESERVED_TOOL_NAMES enforcement blocking reset/step/state/close from MCP tools is well-implemented
• InterceptionServer architecture (host-owned generation for trainer-controlled logprob capture) is sound
• Test coverage is solid across the new harness modules (5 new test files with substantive tests)
• The PR description and mermaid diagrams are excellent documentation

Suggested reviewer for alignment flags: @Darktex

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Tier 1: Bugs & Issues

• Private driver method access: swe_environment.py lines 338-339 call driver._bootstrap_sandbox(...) and driver._start_agent(...) directly, bypassing the public create() path. Same issue in coding_agent_env/harness.py lines 138, 143, 161 accessing self._driver._interception_server. These will break silently on driver refactors.

• Inline @dataclass definitions per-call: swe_environment.py lines 711-725 and 728-745 define @dataclass classes inside methods, creating fresh anonymous types on every invocation. These defeat type-checking and should be module-level definitions (or reuse SWEAgentConfig from harness.py).

• Dangling doc reference: harness.py line 42-44 says "Requires core changes — see CORE_CHANGES.md" but that file doesn't exist.

• Config type divergence: coding_environment.py lines 67-78 introduce _GenericAgentConfig for Pi with thinking: str | None = "off", while CodingAgentConfig uses disable_thinking: bool = False. Two separate config shapes for what should be one abstraction.

• Merge conflict risk: This PR overlaps significantly with #694 (same rename, same core harness files). Must resolve ordering before merge.

Tier 2: Alignment

ALIGNMENT FLAG: Host-side grading runs outside environment server boundary

• Invariant at risk: "Rewards inside environment" (RFC 002, INVARIANTS.md). SWESessionFactory._register_answer_tool and _grade_answer_submission in harness.py run grading in the trainer's process, not the environment server. The swe_environment.py path (black_box) correctly keeps grading server-side, but the harness path does not. This needs an explicit RFC-level decision about whether the harness is considered "inside" the environment boundary.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG: grading.py exported at package level enables trainer-side reward computation

• Invariant at risk: "Rewards inside environment" (RFC 002). __init__.py exports grade_from_case_results publicly, inviting trainers to compute rewards entirely outside the environment.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG: No RFC for new SWE environment category with interception_gate training path

• Invariant at risk: RFC process. This PR adds a new environment category (real-repo SWE tasks), a new agent mode (interception_gate as full RL training path with host-side grading), and a new async GRPO integration pattern. RFC 005 covers agentic harnesses but not this specific grading boundary architecture.
• Suggested reviewer: @Darktex

Summary

Several mechanical issues (private attribute access, inline dataclass definitions, dangling doc references) need fixing. The key alignment concern is that the harness-path grading runs in the trainer process rather than the environment server, which may violate the "rewards inside environment" invariant. The new SWE + interception_gate architecture warrants an RFC before merge.

---

Automated review by Claude Code | Learn more

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Alignment Review Report

Automated Checks

• Lint: SKIP — Manual structural review performed.
• Debug code: CLEAN — No stray print, breakpoint, or TODO artifacts in new files.

---

Tier 1: Fixes Required

• envs/mini_swe_env/harness.py:496,519 — _build_agent_task(swe_task) is called twice in SWESessionFactory.create. The first call's result is used only for _bootstrap_sandbox, then discarded and rebuilt on line 519. Call once and reuse.

• envs/mini_swe_env/server/swe_environment.py:110-113 — CLIAgentSessionFactory is imported and stored as self._CLIAgentSessionFactory but never referenced again. Dead code — remove the import and instance attribute.

• envs/coding_agent_env/harness.py:138,143,161 — Directly accesses self._driver._interception_server and self._driver._interception_base_url (private attributes of CLIAgentDriver). This bypasses encapsulation and will break silently on any internal refactor. Use the driver's public create_session path, or promote these to properties.

---

Tier 2: Alignment Discussion

ALIGNMENT FLAG: In-sandbox reward computation returned to agent in sandbox_mcp_server.py

• Principle at stake: Rewards inside environment (INVARIANTS.md §Architectural Invariants 3)
• The concern: sandbox_mcp_server.py runs inside the agent's sandbox, executes verify commands, computes reward = passed / total when the agent calls terminal(final_answer=...), and returns that reward value to the agent in the tool response. While the host-side SWEEnvironment._grade_submission also computes reward independently, the sandbox-side path exposes reward to the agent during the episode. The agent can condition its behavior on the reward signal it sees in tool output.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG: interception_gate mode adds trainer-owned LLM generation without an RFC

• Principle at stake: RFC-backed architectural decisions
• The concern: The interception_gate mode is a substantial architecture where the trainer process owns the forward pass of the agent's LLM, intercepting every chat/completions call. This goes beyond RFC 005's scope and changes the dual-API boundary. The InterceptionServer sits between the agent's MCP boundary and the LLM as a third party. The PR states "RFC not required" but the rationale isn't documented.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG: answer tool returns resolved status to agent

• Principle at stake: Agents cannot access reward/grading outcome
• The concern: The answer tool handler returns "✅ Resolved: true" or "false" to the agent. While answer is terminal (sets _done = True), nothing in CLIAgentSession hard-prevents the agent from reading the response. The session relies on the agent obeying the "you cannot continue" instruction — a soft guarantee. A hard session kill before the response is delivered would be more aligned.
• Suggested reviewer: @Darktex

---

Summary

• 3 mechanical issues to fix (double task build, dead import, private attribute access)
• 3 alignment points for human review (in-sandbox reward exposure, interception mode without RFC, answer tool leaking grading result)
• The core architecture is well-structured. InterceptionServer has HMAC auth and RESERVED_TOOL_NAMES enforcement. Grading is cleanly separated host-side by default.

---

Automated review by Claude Code | Learn more

[Darktex]

Alignment Review: feat(mini_swe_env) — Two-Tier Report

Reviewed against .claude/docs/PRINCIPLES.md, .claude/docs/INVARIANTS.md, and RFC 005 (agentic harnesses).

Verdict: Request Changes — two mechanical fixes required, four alignment flags need human sign-off.

---

Tier 1 — Fixes Required

T1-1 · sandbox_mcp_server.py · subprocess.run(shell=True) with agent-controlled command string

envs/mini_swe_env/server/sandbox_mcp_server.py execute_command():

proc = subprocess.run(
    command,      # ← fully agent-controlled string
    shell=True,   # ← unsafe
    ...
)

command is passed verbatim from the agent's terminal tool call. shell=True enables shell metacharacter injection. Since this runs inside an already-isolated sandbox the blast radius is bounded, but the pattern is contrary to defense-in-depth.

Fix: Use ["bash", "-c", command] with shell=False.

---

T1-2 · sandbox_mcp_server.py · Agent-writeable reward.txt overrides computed reward

run_verify() reads /home/user/logs/verifier/reward.txt and, if present, substitutes its float as the reward. The agent has full shell access via the terminal tool. Before calling final_answer, the agent can write any float to reward.txt and receive that value as its reward. This is a direct agent→reward channel.

The PR body correctly says "The agent cannot influence the training reward" — but that claim only holds for the Pi/interception path. The sandbox_mcp_server.py (legacy terminal-tool path) breaks this guarantee.

Fix: Either remove the reward.txt override entirely, or chmod 000 the file before the agent session starts, or gate the read behind a host-side verify step outside the agent's writable filesystem.

---

Tier 2 — Alignment Flags

ALIGNMENT FLAG 1: sandbox_mcp_server.py reward.txt — violates "rewards inside environment"

• Principle at stake: RFC 002 + INVARIANTS.md "Reward computation must stay inside environment boundary"
• The concern: Agent can write arbitrary floats to /home/user/logs/verifier/reward.txt, which is read and used as the training reward in the legacy terminal-tool path. Even if this path is secondary, the invariant is unconditional.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG 2: InterceptionServer multiplexes concurrent rollouts — needs RFC coverage

• Principle at stake: PRINCIPLES.md "One env = one trajectory" (RFC 004)
• The concern: InterceptionServer is a single server that routes concurrent rollouts by rollout_id at the trainer level. RFC 004 says environments don't support multiplexed trajectories and batching is via environment stacking. The PR argues correctly that the InterceptionServer is trainer-side infrastructure, but this architectural extension is not covered by any existing RFC. Should be documented (even as a short addendum to RFC 004/005).
• Suggested reviewer: @Darktex

ALIGNMENT FLAG 3: answer tool returns live reward signal to agent during episode

• Principle at stake: RFC 005 §"Harness Security Boundary": "domain-specific validation … should return a factual result, not a reward signal"
• The concern: The answer host-side tool calls SWE-Gym grading and returns "Resolved: true/false" directly to the Pi agent mid-episode. The agent can use this signal to decide whether to retry. This is a deliberate SWE-Gym design choice but was not discussed in any RFC and sits in tension with RFC 005's guidance.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG 4: HF Space deployment — InterceptionServer secret accessible inside sandbox

• Principle at stake: INVARIANTS.md "No credential exposure" and "Container isolation"
• The concern: The InterceptionServer is exposed on 0.0.0.0:7860 (public HF Space port) protected by a bearer token. That token is injected into the sandbox as SWE_VLLM_API_KEY / SWE_LLM_API_KEY. The agent can read its own environment variables via the terminal tool (printenv), exfiltrate the token, and call the host-side tool endpoints directly. Please confirm this is acceptable in the HF Space demo context, or scope the secret so it only grants access to /rollout/{id}/v1/chat/completions (LLM interception) and not /rollout/{id}/v1/tools/answer (grading).
• Suggested reviewer: @Darktex

---

What's Well Done

• The host-side grading path for the Pi/interception mode is correctly architected.
• The RESERVED_TOOL_NAMES guard in InterceptionServer._validate_tool_registration is correct.
• The anti-test-tampering logic in _list_changed_test_paths + _revert_test_files is good hardening.

---

Note: This is an automated review by Claude Code, not a human review.

---

Automated review by Claude Code | Learn more

[Darktex]

Alignment Review — PR #695

Tier 1: Fixes Required

1. HF_TOKEN exposed in process args (deploy_hf_space.sh:114,125): python -c "...HfApi(token='$HF_TOKEN')..." expands the token into the command line, visible via ps aux. The heredoc blocks later in the same script correctly use os.environ['HF_TOKEN'] — convert the two -c blocks to match.
2. Unguarded logprobs dict access (rollout_worker.py:651): choice["logprobs"]["token_logprobs"] crashes with TypeError if logprobs is None (some vLLM builds). Use .get() with a descriptive RuntimeError.
3. SWEGymTask/SWETask are plain @dataclass, not Pydantic (models.py): These types cross the MCP wire as task_json. INVARIANTS.md requires all wire types to be Pydantic models. Migrate to BaseModel.
4. Inline _AgentConfig dataclass recreated per-call (swe_environment.py:709-721): Defeats type checking and produces confusing tracebacks. Move to module level or reuse SWEAgentConfig from harness.py.

Tier 2: Alignment Flags (for human review)

1. Reward visible inside sandbox: sandbox_mcp_server.py writes reward to /home/user/logs/verifier/reward.txt inside the sandbox, potentially readable by the agent before calling final_answer. The harness path correctly routes reward computation host-side — but standalone Docker mode retains file-based reward with weaker isolation.
2. interception_gate mode lacks an RFC: New architectural boundary (trainer-owned interception between MCP and model provider) landed without RFC documentation. RFC 007 was referenced as future work for this.
3. SUPPORTS_CONCURRENT_SESSIONS = True but shared mutable _state: Concurrent run_swe_rollout calls race on self._state. Either set to False or use per-session state with locking.
4. Harness boundary ambiguity: harness.py imports from core and straddles client/server boundary. Needs __all__ restriction and docstring clarifying it's trainer-side only.

Verdict: request_changes — Credential exposure in deploy script, wire-type invariant violation, and crash risk in training worker must be fixed. Alignment flags need human decision.

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Alignment Review — PR #695: mini_swe_env

Automated Checks

• Lint: SKIP (network unavailable in sandbox)
• Debug code: CLEAN
• CORE_CHANGES.md: NOT FOUND — referenced in harness.py docstring but absent from repo

---

Tier 1: Fixes Required

T1-1 Shell injection — base_commit not shell-quoted

envs/mini_swe_env/server/swe_environment.py:652 and envs/mini_swe_env/harness.py:558:

# CURRENT (unsafe):
f"git checkout --quiet {task.base_commit}"

# FIXED:
import shlex
f"git checkout --quiet {shlex.quote(task.base_commit)}"

A malicious base_commit value would execute arbitrary commands in the sandbox.

T1-2 Shell injection — task.repo not validated or quoted

envs/mini_swe_env/server/swe_environment.py:641-643 and envs/mini_swe_env/harness.py:548-550:

Validate task.repo against a pattern like r'^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$' before interpolating into shell commands.

T1-3 Agent-writeable reward.txt override in sandbox_mcp_server.py

envs/mini_swe_env/server/sandbox_mcp_server.py:126-132: An agent can call terminal(command="echo 0.99 > /home/user/logs/verifier/reward.txt") before calling final_answer, overriding the host-side grading. This breaks reward integrity. The reward file override should either be removed or written atomically by the host after agent exit.

T1-4 Module-level _done global is not thread-safe

envs/mini_swe_env/server/sandbox_mcp_server.py:53: Two concurrent final_answer calls would both read _done=False, both set it, and both run verify. Use a threading lock or make the state per-connection.

T1-5 Missing CORE_CHANGES.md document

envs/mini_swe_env/harness.py references CORE_CHANGES.md for required core prerequisites, but the file doesn't exist in the repo. Either ship the doc or add runtime assertions.

T1-6 Server bypasses SWESessionFactory, calls private driver methods

envs/mini_swe_env/server/swe_environment.py:333-349 directly calls driver._bootstrap_sandbox(...) and driver._start_agent(...) (private methods). SWESessionFactory.create() already encapsulates this logic — the server should use it.

---

Tier 2: Alignment Discussion

ALIGNMENT FLAG 1: Reward computed inside the sandbox, observable by agent

• Principle at stake: "Rewards inside environment" (RFC 002, PRINCIPLES.md)
• The concern: sandbox_mcp_server.py runs run_verify() inside the sandbox and returns {"reward": ..., "done": true} directly to the agent. The agent sees its own score mid-episode. Combined with T1-3, the agent can also manipulate the score. Host-side _grade_submission() re-grades, but the in-sandbox reward path blurs the boundary.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG 2: No RFC for interception_gate architecture

• Principle at stake: Dual API boundary (RFC 001, INVARIANTS.md)
• The concern: interception_gate places the host in-band on the agent's LLM forward pass — a new execution model not covered by existing RFCs. This warrants a formal RFC.
• Suggested reviewer: @Darktex

ALIGNMENT FLAG 3: New HTTP in-sandbox server contradicts WebSocket-first direction

• Principle at stake: Communication patterns (INVARIANTS.md)
• The concern: sandbox_mcp_server.py uses BaseHTTPRequestHandler on port 8765. MCP stdio transport is already supported by the same file. The HTTP server creates a new HTTP dependency against the stated deprecation direction.
• Suggested reviewer: @Darktex

---

Other Notes

• The grading anti-gaming logic (_revert_test_files, _list_changed_test_paths) is well-designed and correctly reverts agent-modified test files. Good defensive design.
• The coding_agent_env rename from opencode_env is clean and consistent.

---

Automated review by Claude Code | Learn more

[Darktex]

Note: This is an automated review by Claude Code, not a human review.

---

Alignment Review Report

Automated Checks

• Lint: FAIL — 4 unused import/variable errors in examples/mini_swe_env/trajectory_store.py
• Format: PASS
• Debug code: CLEAN

Tier 1: Fixes Required (blocking)

• Lint failures — trajectory_store.py: unused imports (os, time, typing.Iterator) and unused variable (local_path). Fix with ruff --fix.
• Security: Unauthenticated _handle_exit — Same issue as PR #694. interception_server.py _handle_exit has no auth check while other handlers do.
• Silently dropped tunables — coding_environment.py _build_session_factory: disable_thinking, top_logprobs, max_tokens_cap are silently dropped for the Pi branch. Either document/raise or thread through.
• Broken doc reference — harness.py references CORE_CHANGES.md which doesn't exist in the PR or repo.

Tier 2: Alignment Discussion

FLAG 1: In-sandbox reward computation — sandbox_mcp_server.py computes reward = passed / total and writes reward.txt inside the sandbox. The agent could write to reward.txt to override its own reward. Reward authority should flow host→sandbox, not sandbox→host.

• Principle: "Rewards inside environment" (INVARIANTS.md §3)

FLAG 2: _handle_exit unauthenticated — beyond the security fix, should agents be able to signal their own exit, or only the host process?

• Principle: Agent isolation (INVARIANTS.md §1)

FLAG 3: interception_gate bypasses the Gymnasium step loop — introduces a third communication channel (HTTP endpoints) alongside WebSocket and MCP.

• Principle: "WebSocket for all environment communication" (INVARIANTS.md §4)

FLAG 4: No RFC for InterceptionServer as a new 674-line core primitive.

• Principle: Core changes require RFC (PRINCIPLES.md)

FLAG 5: opencode_env → coding_agent_env rename without backward-compatibility shim. Any downstream users will break silently.

• Principle: Minimize lifecycle deltas (PRINCIPLES.md)

Summary

• 4 blocking mechanical issues
• 5 alignment points for human review
• Structural foundation is solid; main blockers are security fix + architectural sign-off.

---

Automated review by Claude Code

[rycerzes]

Closing alongside #694 — same reason: re-landing as a reviewable stack, not dropping the work.

Context:

• This PR is stacked on #694 and patches cli_driver.py, a file that only exists on the #694 branch — so it can't rebase onto main independently. Closing #694 alone would strand it, so both close together.
• The work splits cleanly: mini_swe_env in black-box mode is reviewable standalone and doesn't depend on the interception rebuild or the RFC. Re-basing its task model on the SWE-rebench V2 schema, with the SWE-Gym loader kept as a compat adapter.
• The async_grpo/ control plane is the piece gated on the RFC — the cross-framework research (Polar, verifiers, AReaL, TRL-native) reshaped several decisions there (token-level trace contract, TRL owning IS correction, session-ID-as-API-key auth, prefix-break forking).

The env package and grading logic (anti-gaming test-file revert, binary all-pass reward) carry forward largely intact and land as the mini_swe_env split PR.