[Sync][D] coding_agent_rl: agent-in-sandbox RL subsystem (slime #1923/#1956/#1960/#1954/#1957/#1958/#1963/#1979/#1981/#1982/#1961)

.github/workflows/pr-test.yml

@@ -505,6 +505,76 @@ jobs:
           pip install torch --index-url https://download.pytorch.org/whl/cpu
           pip install pytest numpy packaging pyyaml omegaconf tqdm httpx pybase64 pylatexenc sympy aiohttp pillow
 
+
+      - name: Install
+        shell: bash
+        run: cd $GITHUB_WORKSPACE && pip install -e . --no-deps
+
+
+      - name: Execute
+        shell: bash
+        run: |
+
+          TEST_PATH="${{ matrix.info.test_file }}"
+          if [[ "$TEST_PATH" != tests/* ]]; then
+            TEST_PATH="tests/$TEST_PATH"
+          fi
+          TEST_ARGS="${{ matrix.info.test_args || '' }}"
+          if [[ -n "$TEST_ARGS" ]]; then
+            read -r -a TEST_ARGS_ARRAY < <(printf '%s\n' "$TEST_ARGS")
+          else
+            TEST_ARGS_ARRAY=()
+          fi
+          if [ "${{ matrix.info.num_gpus }}" = "0" ]; then
+            python "$TEST_PATH" "${TEST_ARGS_ARRAY[@]}"
+          else
+            python tests/ci/gpu_lock_exec.py --count ${{ matrix.info.num_gpus }} -- python "$TEST_PATH" "${TEST_ARGS_ARRAY[@]}"
+          fi
+
+
+  agent-adapter-test:
+    needs: pre-commit
+
+    if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
+
+
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        info: [{"num_gpus": 0, "test_file": "test_agent_trajectory.py"}, {"num_gpus": 0, "test_file": "test_agent_adapters.py"}, {"num_gpus": 0, "test_file": "test_agent_sdk_adapters.py"}]
+    defaults:
+      run:
+        working-directory: ${{ github.workspace }}
+    env:
+      GITHUB_COMMIT_NAME: ${{ github.sha }}_${{ github.event.pull_request.number || 'non-pr' }}
+      WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
+      SLIME_TEST_ENABLE_INFINITE_RUN: ${{ (github.event_name == 'workflow_dispatch' && github.event.inputs.infinite_run) || 'false' }}
+      SLIME_TEST_USE_DEEPEP: ${{ matrix.info.use_deepep || '0' }}
+      SLIME_TEST_USE_FP8_ROLLOUT: ${{ matrix.info.use_fp8_rollout || '0' }}
+      SLIME_TEST_ENABLE_EVAL: ${{ matrix.info.enable_eval || '1' }}
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+          cache: 'pip'
+
+      - name: Install dependencies
+        shell: bash
+        run: |
+          pip install torch --index-url https://download.pytorch.org/whl/cpu
+          pip install pytest numpy packaging pyyaml omegaconf tqdm httpx pybase64 pylatexenc sympy aiohttp pillow
+
+          pip install openai openai-agents anthropic
+
+
       - name: Install
         shell: bash
         run: cd $GITHUB_WORKSPACE && pip install -e . --no-deps

.github/workflows/pr-test.yml.j2

@@ -69,6 +69,18 @@
       ],
     },
 
+    'agent-adapter-test': {
+      'label': 'run-ci-agent-adapter',
+      'always': True,
+      'cpu': True,
+      'extra_pip_deps': 'openai openai-agents anthropic',
+      'tests': [
+        {'test_file': 'test_agent_trajectory.py', 'num_gpus': 0},
+        {'test_file': 'test_agent_adapters.py', 'num_gpus': 0},
+        {'test_file': 'test_agent_sdk_adapters.py', 'num_gpus': 0},
+      ],
+    },
+
     'e2e-test-image': {
       'label': 'run-ci-image',
       'image': 'inferactinc/public:vime-vllm-cu129-latest',
@@ -175,6 +187,9 @@ jobs:
         run: |
           pip install torch --index-url https://download.pytorch.org/whl/cpu
           pip install pytest numpy packaging pyyaml omegaconf tqdm httpx pybase64 pylatexenc sympy aiohttp pillow
+<% if config.get('extra_pip_deps') %>
+          pip install << config.extra_pip_deps >>
+<% endif %>
 
       - name: Install
         shell: bash

examples/coding_agent_rl/README.md

@@ -0,0 +1,179 @@
+# Coding-Agent RL
+
+This directory provides an example of running end-to-end **SWE (Software-Engineering) coding-agent RL** with vime: a real coding agent (claude-code CLI) drives `Read/Edit/Grep/Bash/Agent` tools inside a fresh sandbox per sample, the model produces a `git diff`, and the diff is graded against the dataset's test harness in a second clean sandbox (no test-cheating).
+
+Two example files and one shared adapter implement the loop:
+
+- `generate.py` — per-sample `generate()` registered via `--custom-generate-function-path`. Boots the sandbox, runs claude-code, captures the diff, scores it, and emits one or more `Sample`s back to vime.
+- `vime.agent.adapters.AnthropicAdapter` — the shared Anthropic Messages adapter. claude-code talks to it as if it were Anthropic; the adapter tokenizes the current message history each turn, records prompt/output token snapshots, preserves model-generated tokens (`loss_mask=1`) only while later prompts stitch onto them, masks template/observation tokens (`0`), and emits **three kinds of segments** per trajectory: `subagent` (completed `Task/Agent` dispatch), `wipe` (chain frozen by auto-compact), `final` (tail of the main chain).
+- `sandbox.py` — coding-agent/SWE helpers built on `vime.agent.sandbox`: install bootstraps, spawn claude-code, capture patches, and run the fresh-sandbox evaluator. The shared sandbox contract lives in `vime.agent.sandbox.Sandbox`.
+
+`generate.py` owns one `AnthropicAdapter` instance. For each sample it calls
+`adapter.open_session(...)` before starting claude-code, serves `adapter.app` as
+the Anthropic-compatible endpoint, and drains trainable `TokenSegment`s with
+`await adapter.finish_session(...)` when the trajectory ends.
+
+## Environment Setup
+
+The vime training stack itself follows the standard setup. On top of that you need:
+
+1. **An E2B-compatible sandbox cluster** (or any provider that speaks the E2B SDK). Configure via `E2B_API_KEY` (e.g. the standard `e2b_xxx` key from https://e2b.dev, or any internal endpoint that accepts the same SDK). The official SDK validates this value locally, so internal gateways that ignore auth still need a syntactically valid `e2b_` + 40 hex-character placeholder.
+2. **Host-side tarballs** that get uploaded into each sandbox at boot:
+   - Node 22 (`node-v22.x-linux-x64.tar.xz`) — exported as `SWE_HOST_NODE_TARBALL`.
+   - Claude Code CLI npm tarball (`anthropic-ai-claude-code-local-linux-x64.tgz`) — exported as `SWE_HOST_CC_TARBALL`.
+3. **A sandbox metadata file** (`SWE_SANDBOX_METADATA_FILE`, or the generic `VIME_AGENT_SANDBOX_METADATA_FILE`) — JSON dict whose keys are passed as routing tags when booting an E2B sandbox. Must contain the image key referenced by `SWE_SANDBOX_IMAGE_METADATA_KEY` / `VIME_AGENT_SANDBOX_IMAGE_METADATA_KEY` (e.g. `image`).
+4. **Network reachability**: each sandbox dials back to the vime head node's Anthropic adapter over `http://${VIME_HEAD_HOST}:${SHIM_PORT}`. The head host must be reachable from inside the sandboxes (set `VIME_HEAD_HOST` to a routable IP, not `127.0.0.1`).
+
+## Dataset Format
+
+Standard vime JSONL with three keys:
+
+```jsonc
+{
+  "prompt": "<falls back here if metadata.problem_statement is missing>",
+  "label": "<instance_id or grader label>",
+  "metadata": {
+    "image": "swedev/scaleswe.oh.34:<tag>",   // sandbox image reference
+    "workdir": "/workspace/<repo>",            // repo path inside the sandbox
+    "problem_statement": "<issue body>",
+    // exactly one of the following two graders:
+    "swepro": { /* SWE-bench Pro test harness — preferred */ },
+    "eval_cmd": "pytest -x tests/..."          // last-resort: exit 0 = solved
+    // sweb-style rows: metadata.remote_env_info.f2p_script (Python file
+    // ending in `sys.exit(pytest.main(...))`) is auto-wrapped into eval_cmd.
+  }
+}
+```
+
+Wire it up with `--input-key prompt --label-key label --metadata-key metadata`.
+
+## Running the Script
+
+Override the paths at the top of the launcher, then run from a long-lived shell on the Ray head node (do **not** wrap in `nohup` — Ray child processes get cleaned up with it):
+
+```bash
+cd vime/
+
+export HF_CHECKPOINT=/path/to/Qwen3.6-35B-A3B
+export REF_MODEL_PATH=/path/to/Qwen3.6-35B-A3B_torch_dist
+export PROMPT_DATA=/path/to/swe_train.jsonl
+export SANDBOX_METADATA_FILE=/path/to/sandbox_metadata.json
+export SWE_HOST_NODE_TARBALL=/path/to/node-v22.20.0-linux-x64.tar.xz
+export SWE_HOST_CC_TARBALL=/path/to/anthropic-ai-claude-code-local-linux-x64.tgz
+
+bash examples/coding_agent_rl/run_qwen36_35b_a3b_swe_8nodes.sh
+```
+
+The launcher brings up Ray across all hosts in `/root/mpi_rack_hostfile`, dumps every rollout to `runs/${EXP_TAG}_${STAMP}/rollout_dumps/`, and tees stdout into `runs/${EXP_TAG}_${STAMP}/run.log`.
+
+## New Arguments
+
+`generate.py` is wired in through vime's standard custom-generate hook:
+
+```bash
+ROLLOUT_ARGS=(
+   --custom-generate-function-path examples.coding_agent_rl.generate.generate
+   --prompt-data "${PROMPT_DATA}"
+   --input-key prompt
+   --label-key label
+   --metadata-key metadata
+   --rollout-batch-size 8
+   --n-samples-per-prompt 8
+   --rollout-max-context-len 96000
+   --rollout-max-response-len 32768
+   --rollout-stop-token-ids 248046 248044
+   --save-debug-rollout-data "${RUN_ROOT}/rollout_dumps/rollout_{rollout_id}.pt"
+)
+```
+
+claude-code's tool invocations are parsed from the model output. By default the
+adapter uses the built-in XML tool-call fallback (`parse_xml_tool_uses`), which
+handles the `<tool_call><function=...>` text Qwen3-Coder emits, so no extra
+engine-side parser configuration is required. (The adapter can optionally
+delegate to SGLang's reasoning/function-call parsers when
+`vllm_tool_call_parser` / `vllm_reasoning_parser` are set on `args`, but this is
+not used by this example and is not wired into vime's vLLM arguments.)
+
+## SWE-specific Environment Knobs
+
+All set in the launcher; tune per cluster.
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `VIME_HEAD_HOST` | `${MASTER_ADDR}` | Public IP the sandbox uses to reach the Anthropic adapter. **Must be routable from inside the sandbox.** |
+| `SHIM_BIND_HOST` / `SHIM_PORT` | `0.0.0.0` / `18001` | Bind address of the adapter shim on the head node. |
+| `E2B_API_KEY` | — | E2B (or compatible) API key. |
+| `SWE_SANDBOX_METADATA_FILE` / `VIME_AGENT_SANDBOX_METADATA_FILE` | — | JSON dict of routing metadata passed at sandbox boot. |
+| `SWE_SANDBOX_IMAGE_METADATA_KEY` / `VIME_AGENT_SANDBOX_IMAGE_METADATA_KEY` | — | Which key in the metadata file holds the image reference (e.g. `image`). |
+| `SWE_HOST_NODE_TARBALL` | — | Host path to Node 22 tarball uploaded into each sandbox. |
+| `SWE_HOST_CC_TARBALL` | — | Host path to the Claude Code CLI npm tarball. |
+| `SWE_TIME_BUDGET_SEC` | `1800` | Wallclock budget for one agent run. |
+| `SWE_EVAL_TIMEOUT_SEC` | `600` | Wallclock cap on the evaluator sandbox. |
+| `SWE_BOOT_CONCURRENCY` | `6` | Cap on simultaneous sandbox boots (eases h2/SSL long-tail). |
+| `SWE_CLAUDE_EXTRA_ARGS` | (see launcher) | Extra flags appended to the `claude` CLI invocation — registers the read-only `investigator` sub-agent, disables `WebFetch`/`WebSearch`, disables slash commands. |
+| `SWE_CC_PROMPT` | unset | Optional override for the user-turn prompt. Setting this to require sub-agent dispatch is the most reliable way to maximize fan-out. |
+
+`--rollout-max-response-len` is the per-turn generation cap passed to each vLLM
+`/inference/v1/generate` call as the sampling-params `max_tokens`.
+`--rollout-max-context-len` is the multi-turn prompt+response budget enforced
+only during generation: each turn clamps the generation length to the remaining
+context. Trajectory merge/export keeps the emitted segments and does not drop
+them for length.
+
+## String-in, Token-out Trajectories
+
+The coding-agent environment is string/message based: claude-code sends
+Anthropic Messages requests, receives streamed text/thinking/tool-use blocks,
+and later sends back rendered tool observations. Training, however, must stay
+token based. A trajectory is only a valid RL target when the optimized tokens
+are the same tokens the rollout model actually sampled.
+
+The Anthropic adapter therefore follows a **string in, token out** contract:
+
+- Each incoming message history is rendered with the served model's chat
+  template and sent to vLLM as `token_ids`.
+- vLLM's `/inference/v1/generate` is called with `logprobs` set; the adapter
+  records the exact `prompt_ids`, sampled `output_ids` (`choices[0].token_ids`),
+  and per-token rollout logprobs (`choices[0].logprobs.content[i].logprob`) for
+  that turn.
+- At training export time, samples are assembled from those saved token ids.
+  The decoded `response` field is only a readable sidecar; it is not
+  re-tokenized to recover the training sequence.
+
+Multi-turn agents still force the adapter to tokenize later message
+histories, because tool observations and claude-code's own compacted messages
+arrive as strings. `vime.agent.trajectory.merge_turns` stitches those later
+prompts against the saved token stream:
+
+- New prompt suffixes that are tool/user/environment context are appended with
+  `loss_mask=0`.
+- Fresh model outputs from vLLM are appended with `loss_mask=1`.
+- If a later prompt no longer token-matches an earlier sampled output, the
+  unmatched suffix is dropped. If the drift cuts through the middle of a
+  previous model output, the retained prefix of that whole output turn is also
+  assigned `loss_mask=0`.
+
+That last case is the important correctness guard. A re-tokenization mismatch
+can make a string-level conversation look continuous while token-level
+provenance is broken. vime keeps the context needed to continue the agent, but
+does not backprop through tokens whose sampled origin can no longer be proven.
+
+## Fan-out Semantics
+
+- `generate()` returns `list[Sample]` — one Sample per trajectory **segment** (`subagent` / `wipe` / `final`).
+- Per-trajectory reward is split as `reward / K` across segments; `rollout_id` is shared so the per-rollout-mean loss reducer still counts the trajectory once.
+- Sub-agent dispatch increases `K` (each completed `Agent` turn block becomes its own segment), so the effective batch after flatten can be much larger than `rollout_batch_size * n_samples_per_prompt`.
+
+## Porting to a New Sandbox Backend
+
+`vime.agent.sandbox.Sandbox` exposes the shared sandbox contract, and
+`vime.agent.sandbox.E2BSandbox` is the E2B implementation:
+
+```python
+await sb.exec(cmd, user=..., check=..., timeout=...)
+await sb.write_file(sandbox_path, content_or_host_path, user=...)
+await sb.read_file(sandbox_path, user=...)
+async with E2BSandbox(...) as sb: ...
+```
+
+Reimplement those on Docker / Modal / a local VM and everything in `generate.py` keeps working unchanged.

examples/coding_agent_rl/aiohttp_threaded.py

@@ -0,0 +1,90 @@
+"""Run an ``aiohttp.web.Application`` in a background daemon thread."""
+
+from __future__ import annotations
+
+import asyncio
+import threading
+from dataclasses import dataclass
+from typing import Any
+
+from aiohttp import web
+
+
+@dataclass
+class AppHandle:
+    host: str
+    port: int
+    thread: threading.Thread
+    loop: asyncio.AbstractEventLoop
+    runner: web.AppRunner
+
+    @property
+    def url(self) -> str:
+        return f"http://{self.host}:{self.port}"
+
+    def stop(self) -> None:
+        async def _shutdown() -> None:
+            await self.runner.cleanup()
+
+        try:
+            fut = asyncio.run_coroutine_threadsafe(_shutdown(), self.loop)
+            fut.result(timeout=10)
+        except Exception:
+            pass
+        self.loop.call_soon_threadsafe(self.loop.stop)
+        self.thread.join(timeout=5)
+
+
+def run_app_in_thread(
+    app: web.Application,
+    *,
+    host: str = "0.0.0.0",
+    port: int = 0,
+    thread_name: str = "aiohttp-app",
+    start_timeout_sec: float = 15.0,
+    runner_kwargs: dict[str, Any] | None = None,
+) -> AppHandle:
+    """Spin up ``app`` on a daemon thread; block until it is listening.
+
+    ``runner_kwargs`` is forwarded to ``web.AppRunner`` (e.g. pass
+    ``{"handler_cancellation": True}`` to make a client disconnect cancel
+    the in-flight handler coroutine).
+    """
+    started = threading.Event()
+    err_box: list[BaseException] = []
+    box: dict[str, Any] = {}
+
+    def _run() -> None:
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        try:
+            runner = web.AppRunner(app, **(runner_kwargs or {}))
+            loop.run_until_complete(runner.setup())
+            site = web.TCPSite(runner, host, port)
+            loop.run_until_complete(site.start())
+            actual_port = port
+            for sock in site._server.sockets:  # type: ignore[attr-defined]
+                actual_port = sock.getsockname()[1]
+                break
+            box["loop"] = loop
+            box["runner"] = runner
+            box["port"] = actual_port
+            started.set()
+            loop.run_forever()
+        except BaseException as e:  # pragma: no cover
+            err_box.append(e)
+            started.set()
+            raise
+
+    thread = threading.Thread(target=_run, name=thread_name, daemon=True)
+    thread.start()
+    started.wait(timeout=start_timeout_sec)
+    if err_box:
+        raise err_box[0]
+    return AppHandle(
+        host=host,
+        port=int(box["port"]),
+        thread=thread,
+        loop=box["loop"],
+        runner=box["runner"],
+    )