Add SWE-bench inference & evaluation

[ludwig-n]

This PR adds SWE-bench inference (featuring 2 agentic frameworks: SWE-agent and OpenHands) and evaluation using the official SWE-bench harness.

Sample evaluation

Here's how to evaluate Qwen3-Coder-30B-A3B-Instruct with OpenHands on a Slurm cluster.
First, prepare the data (SWE-bench Verified by default) by running

ns prepare_data swe-bench

The inference and evaluation runs inside of prebuilt container images from the SWE-bench team. By default, this will configure them to be downloaded from Dockerhub every time you run ns eval.
If you have the SWE-bench images downloaded somewhere on the cluster, add that folder to the mounts in your cluster config and use the option --container_formatter to specify the mounted path to the images, e.g.

ns prepare_data swe-bench \
    --container_formatter "/swe-bench-images/swebench_sweb.eval.x86_64.{instance_id}.sif"

Then, to run the agent on all instances and evaluate the generated patches, run

ns eval \
    --cluster=<CLUSTER_NAME> \
    --model=Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --server_type=vllm \
    --server_args="--enable-auto-tool-choice --tool-call-parser qwen3_coder" \
    --server_nodes=1 \
    --server_gpus=8 \
    --benchmarks=swe-bench \
    --expname=<EXPNAME> \
    --output_dir=<OUTPUT_DIR> \
    --num_chunks=2 \
    ++agent_framework=openhands \
    ++inference.temperature=0.7 \
    ++inference.top_p=0.8 \
    ++inference.top_k=20

replacing the <...> with your desired parameters. Of course, if you have the model downloaded on the cluster, you can modify the --model parameter to specify the path to the weights.

I ran 5 identical evaluations with this command and got the following scores on SWE-bench Verified: 50.0, 49.8, 49.6, 51.8, 51.0, with an average of 50.4. The official reported score for this model is 51.6.

[Kipok]

Thanks!

[Kipok]

@ludwig-n when you get a chance, can you please update description of this PR with an example eval command and expected score? We will eventually port it to the evaluation docs we are going to have at some point in the near future