Add examples/swe-agent: GLM-4.7-Flash agentic training with Harbor

examples/README.md

@@ -18,6 +18,7 @@ These examples provide concrete examples to leverage miles in your own RL workfl
 - **[retool](./retool)**: Demonstrates the retool functionality for tool-enabled language model generation.
 - **[search-r1](./search-r1)**: A minimal reproduction of Search-R1, featuring multi-turn conversation and tool-calling.
 - **[strands-agents](./strands-agents)**: Integration example with the Strands-Agents scaffolding framework.
+- **[swe-agent](./swe-agent)**: Trains coding and terminal agents with Harbor-managed sandboxes and verifier rewards.
 - **[tau-bench](./tau-bench)**: Training in an agentic multi-turn tool use environment (Tau-bench).
 - **[train_infer_mismatch_helper](./train_infer_mismatch_helper)**: Algorithmic methods for rollout correction (e.g., TIS, MIS).
 - **[true_on_policy](./true_on_policy)**: Ensures strictly equal log probabilities between inference (SGLang) and training engines.

examples/swe-agent/README.md

@@ -0,0 +1,125 @@
+# SWE-Agent training with Harbor
+
+This example trains GLM-4.7-Flash on agentic coding and terminal tasks. Miles
+runs synchronous GRPO and serves the policy through its session server; a
+separate [Harbor](https://github.com/harbor-framework/harbor) agent server
+creates the task sandboxes, runs the agents, and returns verifier rewards.
+
+The same pipeline supports Terminal-Bench, SWE-bench, and custom Harbor tasks.
+Training records must contain a `prompt` and `metadata.instance_id` identifying
+the Harbor task.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `run.py` | Validated synchronous GLM-4.7-Flash launcher. |
+| `run-glm47-flash-agentic-async.py` | Disaggregated fully asynchronous launcher. |
+| `swe_agent_function.py` | Sends each rollout to the Harbor agent server. |
+| `generate.py` | Builds rewards, metrics, and training samples. |
+| `download_and_process_data.py` | Converts supported datasets to Miles JSONL. |
+
+## 1. Start the Harbor agent server
+
+Use the `harbor-miles-v0.20.0` branch of the `harbor-framework/harbor`
+repository, which carries the Miles integration:
+
+```bash
+git clone https://github.com/harbor-framework/harbor.git
+cd harbor
+git checkout harbor-miles-v0.20.0
+uv sync
+
+HARBOR_TASKS_DIR=/path/to/harbor_tasks uv run python miles_agent_server.py \
+    --host 0.0.0.0 \
+    --port 30000 \
+    --dashboard-port 0 \
+    --max-concurrent 32 \
+    --agent-timeout 5400 \
+    --trials-dir /path/to/trials
+```
+
+`HARBOR_TASKS_DIR` must contain one Harbor task directory for every
+`metadata.instance_id` in the training data. The agent-server machine must have
+Docker and enough capacity for the requested number of concurrent sandboxes;
+set `--max-concurrent` to at least one sandbox per trajectory in a rollout step
+(`--rollout-batch-size` times `--n-samples-per-prompt`). Keep `--agent-timeout`
+generous — agentic trials routinely run past an hour, and a short timeout kills
+them mid-episode. Verify `http://<agent-server>:30000/health` before launching
+Miles.
+
+If the trainer reaches the agent server through a proxy or an in-cluster service
+rather than directly, point `--agent-server-url` at that stable name rather than
+an ephemeral pod address. The rollout client enables TCP keepalive probes so
+long-running trials do not lose an idle connection while Harbor is working.
+
+## 2. Prepare Terminal-Bench data
+
+Convert a local JSONL whose rows include a task instruction and instance name:
+
+```bash
+python examples/swe-agent/download_and_process_data.py \
+    --input /path/to/terminal-bench.jsonl \
+    --output /path/to/tb2_train.jsonl \
+    --agent-name mini-swe-agent \
+    --prompt-key instruction
+```
+
+The resulting `metadata.instance_id` values must match task directories known to
+the Harbor agent server.
+
+## 3. Launch synchronous GLM-4.7-Flash training
+
+The shape below is what a multi-day Terminal-Bench 2 run used on one node of 8
+H200 GPUs: 32 trajectories per GRPO step (4 prompts times 8 samples), each one a
+full mini-swe-agent episode in its own Harbor sandbox.
+
+```bash
+python examples/swe-agent/run.py \
+    --num-nodes 1 \
+    --num-gpus-per-node 8 \
+    --skip-prepare \
+    --megatron-path /root/Megatron-LM \
+    --hf-checkpoint /path/to/GLM-4.7-Flash \
+    --ref-load /path/to/GLM-4.7-Flash_torch_dist \
+    --save-dir /path/to/checkpoints \
+    --prompt-data /path/to/tb2_train.jsonl \
+    --max-seq-len 65536 \
+    --rollout-batch-size 4 \
+    --n-samples-per-prompt 8 \
+    --global-batch-size 32 \
+    --num-rollout 200 \
+    --save-interval 20 \
+    --agent-server-url http://<agent-server>:30000 \
+    --router-external-host <trainer-host-reachable-from-agent-server> \
+    --miles-host-ip 0.0.0.0 \
+    --save-traces-dir /path/to/traces
+```
+
+For a smoke test, set `--num-rollout 1`. Expect roughly 10 minutes per step at
+this shape; because synchronous rollout waits for the slowest trajectory in the
+batch, a step that draws an unusually slow task can take several times that.
+
+`--router-external-host` is the address Harbor sandboxes use to call the Miles
+session server and SGLang router. It must resolve and route from the agent-server
+machine. `--miles-host-ip 0.0.0.0` is useful when those services must accept
+connections forwarded from another host. Ensure ports 30000 and 31000 are
+reachable end to end; Tailscale is one option when the machines are on different
+networks.
+
+## 4. Verify progress
+
+Check both layers:
+
+1. Miles logs emit rollout metrics and write `rollout_data/*.pt` under the trace
+   directory.
+2. Megatron logs emit `train/step` and the Ray job exits successfully.
+
+Confirm a suspected stall on disk before believing a dashboard. W&B uploads can
+fail partway through a long run — dropping some metric rows while others keep
+arriving — which looks exactly like a frozen reward curve. The per-step
+`train_data/<step>` and `rollout_data/<step>.pt` dumps under `--save-traces-dir`
+are written by the trainer itself and are the authoritative progress signal.
+
+The synchronous launcher uses GLM-4.7 tool-call and reasoning parsers, TITO,
+the Miles session server, and the Megatron backend.

examples/swe-agent/download_and_process_data.py

@@ -0,0 +1,171 @@
+#!/usr/bin/env python3
+"""Download and convert datasets to Miles format for SWE-agent training.
+
+Supports any task type — SWE-bench, Terminal-Bench, custom datasets, etc.
+Each record's metadata is enriched with ``agent_name`` so that
+the Harbor agent server can route to the correct agent.
+All original fields are preserved in metadata so that
+prepare_harbor_tasks.py can infer the right task directory layout.
+
+Usage examples:
+
+    # SWE-bench from HuggingFace (defaults)
+    python download_and_process_data.py \\
+        --input SWE-Gym/SWE-Gym --output /root/swe_train.jsonl
+
+    # Terminal-Bench from local JSONL
+    python download_and_process_data.py \\
+        --input /data/tb_tasks.jsonl --output /root/tb_train.jsonl \\
+        --agent-name terminus-2 --prompt-key instruction
+
+    # Custom dataset
+    python download_and_process_data.py \\
+        --input /data/my_tasks.jsonl --output /root/custom_train.jsonl \\
+        --agent-name my-agent --prompt-key task_description
+
+    # Merge multiple outputs into one mixed JSONL
+    cat /root/swe_train.jsonl /root/tb_train.jsonl > /root/mixed.jsonl
+"""
+
+import argparse
+import json
+import tempfile
+from pathlib import Path
+
+from datasets import load_dataset
+
+_PROMPT_KEY_FALLBACKS = ("problem_statement", "instruction", "prompt")
+
+
+def convert_to_miles_format(
+    input_path: str,
+    output_path: str,
+    *,
+    limit: int | None = None,
+    split: str = "train",
+    agent_name: str = "mini-swe-agent",
+    prompt_key: str = "problem_statement",
+    append: bool = False,
+) -> int:
+    """Convert JSONL to Miles format.
+
+    Returns the number of records written.
+    """
+    count = 0
+    mode = "a" if append else "w"
+    with open(input_path) as fin, open(output_path, mode) as fout:
+        for line in fin:
+            if limit is not None and count >= limit:
+                break
+
+            instance = json.loads(line)
+
+            metadata = dict(instance)
+            metadata["agent_name"] = agent_name
+            metadata["split"] = split
+
+            prompt = instance.get(prompt_key, "")
+            if not prompt:
+                for fallback in _PROMPT_KEY_FALLBACKS:
+                    prompt = instance.get(fallback, "")
+                    if prompt:
+                        break
+
+            miles_sample = {
+                "prompt": prompt,
+                "metadata": metadata,
+            }
+
+            fout.write(json.dumps(miles_sample) + "\n")
+            count += 1
+
+    print(f"Converted {count} samples: {input_path} -> {output_path}")
+    return count
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Download dataset and convert to Miles format",
+    )
+    parser.add_argument(
+        "--input",
+        type=str,
+        required=True,
+        help="HuggingFace dataset path or local JSONL file",
+    )
+    parser.add_argument(
+        "--output",
+        type=str,
+        required=True,
+        help="Output JSONL file path",
+    )
+    parser.add_argument(
+        "--split",
+        type=str,
+        default="train",
+        help="Dataset split (default: train)",
+    )
+    parser.add_argument("--limit", type=int, help="Limit number of samples")
+    parser.add_argument(
+        "--agent-name",
+        type=str,
+        default="mini-swe-agent",
+        help="Harbor agent name injected into metadata " "(default: mini-swe-agent)",
+    )
+    parser.add_argument(
+        "--prompt-key",
+        type=str,
+        default="problem_statement",
+        help="JSON key to use as prompt text " "(default: problem_statement)",
+    )
+    parser.add_argument(
+        "--append",
+        action="store_true",
+        help="Append to output file instead of overwriting",
+    )
+
+    args = parser.parse_args()
+
+    input_path = Path(args.input)
+
+    common_kwargs = dict(
+        limit=args.limit,
+        split=args.split,
+        agent_name=args.agent_name,
+        prompt_key=args.prompt_key,
+        append=args.append,
+    )
+
+    if input_path.exists() and input_path.suffix == ".jsonl":
+        print(f"Processing local file: {args.input}")
+        convert_to_miles_format(args.input, args.output, **common_kwargs)
+    else:
+        print(f"Loading HuggingFace dataset: " f"{args.input} (split={args.split})")
+        ds = load_dataset(args.input, split=args.split)
+
+        if args.limit:
+            ds = ds.select(range(min(args.limit, len(ds))))
+
+        tmp_path = None
+        try:
+            with tempfile.NamedTemporaryFile(
+                mode="w",
+                suffix=".jsonl",
+                delete=False,
+            ) as tmp:
+                tmp_path = tmp.name
+
+            print(f"Downloading to temporary file: {tmp_path}")
+            ds.to_json(tmp_path)
+
+            print(f"Converting to Miles format: {args.output}")
+            convert_to_miles_format(tmp_path, args.output, **common_kwargs)
+        finally:
+            if tmp_path and Path(tmp_path).exists():
+                Path(tmp_path).unlink()
+
+    print("Done.")
+
+
+if __name__ == "__main__":
+    main()