diff --git a/examples/README.md b/examples/README.md index 335cc6ef16a..bdd1ae28cd2 100644 --- a/examples/README.md +++ b/examples/README.md @@ -39,4 +39,5 @@ Not fully verified — for experimental and development use. - **[reproducibility](./experimental/reproducibility)**: Guides on achieving bitwise experiment reproduction using deterministic modes. - **[search-r1](./experimental/search-r1)**: A minimal reproduction of Search-R1, featuring multi-turn conversation and tool-calling. - **[strands_sglang](./experimental/strands_sglang)**: Integration example with the Strands-Agents scaffolding framework. +- **[swe-agent-harbor-daytona](./experimental/swe-agent-harbor-daytona)**: The swe-agent Harbor pipeline with task sandboxes hosted on Daytona instead of local Docker. - **[tau-bench](./experimental/tau-bench)**: Training in an agentic multi-turn tool use environment (Tau-bench). diff --git a/examples/experimental/swe-agent-harbor-daytona/README.md b/examples/experimental/swe-agent-harbor-daytona/README.md new file mode 100644 index 00000000000..3d40c79da55 --- /dev/null +++ b/examples/experimental/swe-agent-harbor-daytona/README.md @@ -0,0 +1,184 @@ +# SWE-Agent training with Harbor on Daytona sandboxes + +This example trains GLM-4.7-Flash on agentic terminal and coding tasks, with +task sandboxes hosted on [Daytona](https://www.daytona.io/) instead of local +Docker. Miles runs synchronous GRPO and serves the policy through its session +server; a Harbor agent server drives the agent and returns verifier rewards. +It is meant to run on a single node of 8 H200 GPUs. + +It is the `examples/swe-agent` pipeline with two changes: + +- **Daytona sandboxes.** The agent-server host needs outbound HTTPS but no + Docker daemon, no local image builds, and no local disk for task images. This + is the practical option when the trainer runs on a GPU node where you cannot + or do not want to run Docker-in-Docker. +- **terminus-2 agent.** terminus-2 runs as a host process and calls the model + endpoint itself, rather than from inside the sandbox, so the model endpoint + must be reachable from the agent-server host. + +Everything else — TITO, the session server, GRPO, the reward path — is identical +to `examples/swe-agent`, and so is the trainer side: this example has no +launcher of its own and runs `examples/swe-agent/run.py` unchanged. Daytona is +selected entirely by the agent server's environment, which the trainer never +sees. + +## Files + +| File | Purpose | +| --- | --- | +| `launch_agent_server.sh` | Starts the Harbor agent server in Daytona mode. | + +## 1. Provision Daytona + +Create an API key in the Daytona dashboard and export it on the agent-server +host: + +```bash +export DAYTONA_API_KEY= +``` + +Daytona accounts have a total-disk quota, so keep concurrent sandboxes times +`HARBOR_DAYTONA_DISK_GB` under it. + +## 2. Start the Harbor agent server + +Use the `harbor-miles-v0.20.0` branch of `harbor-framework/harbor`, which +carries the Miles integration: + +```bash +git clone https://github.com/harbor-framework/harbor.git +cd harbor +git checkout harbor-miles-v0.20.0 +uv sync + +export DAYTONA_API_KEY= +export HARBOR_TASKS_DIR=/path/to/harbor_tasks +export TRIALS_DIR=/path/to/trials +bash /path/to/miles/examples/experimental/swe-agent-harbor-daytona/launch_agent_server.sh +``` + +`HARBOR_TASKS_DIR` must contain one Harbor task directory for every +`metadata.instance_id` in the training data; a missing directory makes the trial +score 0 rather than raise. Set `--max-concurrent` to at least one sandbox per +trajectory in a rollout step (`--rollout-batch-size` times +`--n-samples-per-prompt`). Keep the agent timeout generous — agentic trials +routinely run past an hour. + +Run the agent server under a process supervisor or a detached terminal +multiplexer on its own host, not in a foreground shell over SSH: if that shell +dies it takes the agent server and every live sandbox with it, and the trainer +then starves without an obvious error. + +Verify `http://:11000/health` before launching Miles. + +## 3. Prepare data + +`examples/swe-agent/download_and_process_data.py` converts a local JSONL into +Miles format. For terminus-2, set the agent name accordingly: + +```bash +python examples/swe-agent/download_and_process_data.py \ + --input /path/to/terminal-bench.jsonl \ + --output /path/to/tb2_train.jsonl \ + --agent-name terminus-2 \ + --prompt-key instruction +``` + +## 4. Launch training + +The shape below is what a multi-day Terminal-Bench 2 run used, with the agent +server colocated on the same host as the trainer: + +```bash +export WANDB_API_KEY= + +python examples/swe-agent/run.py \ + --num-nodes 1 \ + --num-gpus-per-node 8 \ + --skip-prepare \ + --megatron-path /root/Megatron-LM \ + --hf-checkpoint /path/to/GLM-4.7-Flash \ + --ref-load /path/to/GLM-4.7-Flash_torch_dist \ + --save-dir /path/to/checkpoints \ + --prompt-data /path/to/tb2_train.jsonl \ + --max-seq-len 65536 \ + --rollout-batch-size 4 \ + --n-samples-per-prompt 8 \ + --global-batch-size 32 \ + --num-rollout 200 \ + --save-interval 10 \ + --agent-server-url http://127.0.0.1:11000 \ + --router-external-host \ + --save-traces-dir /path/to/traces \ + --wandb-project +``` + +For a smoke test, set `--num-rollout 1`. + +`--router-external-host` is the address the agent server uses to reach the Miles +session server, substituted into the base URL handed to the agent. It only has +to resolve from the agent-server host, so a hostname is fine — use one when the +agent server reaches the trainer over a tailnet or other overlay. Do not confuse +it with `--miles-host-ip`, which is bound locally on the trainer and must be an +address that already exists on one of its interfaces. Ports 30000 and 31000 must +be reachable from the agent-server host. + +## Sizing the per-turn response cap + +`--rollout-max-response-len` and the agent server's `AGENT_MAX_OUTPUT_TOKENS` +both cap a **single turn**, not the whole trajectory. Agentic trajectories are +routinely several times longer than one turn, so a cap that looks generous +against `--max-seq-len` can still abort most trials. + +When a turn exceeds the cap, `HARBOR_RESPONSE_LENGTH_POLICY=abort` ends the +trial and **none of that turn's tool calls are performed**, so the trial scores +0 and dilutes its GRPO group. The symptoms are +`SingleTurnMaxSeqLenExceededError` and `ContextLengthExceededError` in the trial +exception files, with `rollout/truncated_ratio` well above 0. + +To size these, compare `rollout/response_len/mean` and `rollout/response_len/max` +against the cap, and keep `AGENT_MAX_INPUT_TOKENS` above the largest observed +context. +`--max-seq-len 65536` leaves plenty of headroom to raise both. + +`examples/swe-agent/run.py` hardcodes `--rollout-max-response-len 8192`, so raise +it there; `AGENT_MAX_OUTPUT_TOKENS` is an environment variable on the agent +server and is set in `launch_agent_server.sh`. Raise the two together — leaving +either one behind reintroduces the aborts. + +## Verify progress + +Read `rollout/raw_reward` for the task solve rate. `rollout/rewards` is the +GRPO-centered advantage and sits near zero by construction, so it never shows +learning. + +Two properties of this shape make per-step reward misleading: + +- With `--rollout-batch-size 4` there are only 4 GRPO groups per step. Uniform + groups (all solved or all failed) contribute no gradient, and which 4 tasks + were drawn dominates the step reward. Judge progress on repeated tasks across + many batches, never on consecutive steps. +- A long run's headline health number is the fraction of trials that return + successfully, not the reward. Census the trial directories under + `--trials-dir` by outcome: no `exception.txt` means success, and the last + exception class named in that file is the failure mode. Key the census on the + **trial start time** — the mtime of the trial's `config.json`, written at + launch — because writing `exception.txt` bumps the directory mtime and makes + any mtime-sorted listing look like everything is failing. + +Confirm a suspected stall on disk before believing a dashboard. W&B uploads can +fail partway through a long run, dropping some metric rows while others keep +arriving, which looks exactly like a frozen reward curve. The per-step +`train_data/` and `rollout_data/.pt` dumps under `--save-traces-dir` +are written by the trainer itself and are authoritative. + +## Troubleshooting + +| Symptom | Cause | +| --- | --- | +| `DaytonaValidationError` on sandbox create | Daytona disk quota exhausted. | +| `EnvironmentStartTimeoutError` in bursts | Sandbox creation is slow because the account is near its disk quota. | +| `SingleTurnMaxSeqLenExceededError` | Per-turn output cap too low; see the sizing section. | +| `ContextLengthExceededError` | `AGENT_MAX_INPUT_TOKENS` below the observed context length. | +| sgl-router fails to bind | `--miles-host-ip` is not an address the trainer host can bind; leave it unset to auto-detect. | +| Every trial scores 0 | `metadata.instance_id` values have no matching directory under `HARBOR_TASKS_DIR`. | diff --git a/examples/experimental/swe-agent-harbor-daytona/launch_agent_server.sh b/examples/experimental/swe-agent-harbor-daytona/launch_agent_server.sh new file mode 100755 index 00000000000..44c89927303 --- /dev/null +++ b/examples/experimental/swe-agent-harbor-daytona/launch_agent_server.sh @@ -0,0 +1,41 @@ +#!/bin/bash +# Start the Harbor agent server in Daytona mode. +# +# Run this from the root of a harbor-framework/harbor checkout on the +# harbor-miles-v0.20.0 branch, which carries the Miles integration, before +# launching examples/swe-agent/run.py. Trials are graded inside Daytona cloud +# sandboxes, so this host needs outbound HTTPS but no Docker daemon. +set -euo pipefail + +: "${DAYTONA_API_KEY:?set DAYTONA_API_KEY to a Daytona API key}" +: "${HARBOR_TASKS_DIR:?set HARBOR_TASKS_DIR to the directory holding Harbor task dirs}" + +TRIALS_DIR="${TRIALS_DIR:-/tmp/harbor_trials}" +PORT="${PORT:-11000}" +MAX_CONCURRENT="${MAX_CONCURRENT:-32}" + +export HARBOR_ENV_TYPE=daytona +export HARBOR_DAYTONA_DISK_GB="${HARBOR_DAYTONA_DISK_GB:-10}" +# Snapshot each task image on first use so later trials skip the build. +export HARBOR_DAYTONA_AUTO_SNAPSHOT=1 + +# terminus-2 runs as a host process and calls the model itself, so the model +# endpoint must be reachable from here rather than from inside the sandbox. +export OPENAI_API_KEY="${OPENAI_API_KEY:-dummy}" +export OPENAI_API_BASE="${OPENAI_API_BASE:-http://127.0.0.1:30000/v1}" +export OPENAI_BASE_URL="$OPENAI_API_BASE" + +# Keep these consistent with --rollout-max-response-len and --max-seq-len on the +# trainer side; see the README section on sizing them. +export AGENT_MAX_INPUT_TOKENS="${AGENT_MAX_INPUT_TOKENS:-32768}" +export AGENT_MAX_OUTPUT_TOKENS="${AGENT_MAX_OUTPUT_TOKENS:-8192}" +export HARBOR_RESPONSE_LENGTH_POLICY=abort + +mkdir -p "$TRIALS_DIR" + +exec python miles_agent_server.py \ + --host 0.0.0.0 \ + --port "$PORT" \ + --max-concurrent "$MAX_CONCURRENT" \ + --agent-timeout 5400 \ + --trials-dir "$TRIALS_DIR"