Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ uv venv && uv sync --extra dev --group docs
pre-commit install

# Run servers
ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[resources_servers/example_single_tool_call/configs/example_single_tool_call.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"

# Run tests for a specific server (creates .venv per server, installs deps, runs pytest)
# First run is slow. Use skip_venv_if_present config or place a .venv to skip venv creation.
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/aalcr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ng_prepare_benchmark "+config_paths=[$config_paths]"

# Run
```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/aalcr/config.yaml"
ng_e2e_collect_rollouts \
"+config_paths=[${config_paths}]" \
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/aalcr/config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Refer to the params in responses_api_models/local_vllm_model/configs/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8.yaml
# Refer to the params in responses_api_models/vllm_server/configs/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8.yaml
Qwen3-235B-A22B-Instruct-2507-FP8:
responses_api_models:
vllm_model:
vllm_endpoint:
entrypoint: app.py
model: Qwen3-235B-A22B-Instruct-2507-FP8
replace_developer_role_with_system: true
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/aime24-x/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ python benchmarks/aime24-x/prepare.py --prompt_language en
## Quickstart

```bash
ng_run "+config_paths=[benchmarks/aime24-x/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/aime24-x/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"
```

Then in another shell:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/aime25-x/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ python benchmarks/aime25-x/prepare.py --prompt_language en
## Quickstart

```bash
ng_run "+config_paths=[benchmarks/aime25-x/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/aime25-x/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"
```

Then in another shell:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/aime26/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/aime26/config.yaml]"
## Run servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/aime26/config.yaml"
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,benchmarks/aime26/config.yaml"
ng_run "+config_paths=[$config_paths]"
```

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/answer-judge/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ here is the same deterministic `Judgement: Yes/No` parsing used by Skills'
ng_prepare_benchmark "+config_paths=[benchmarks/answer-judge/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/answer-judge/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/apex_shortlist/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Writes `data/apex_shortlist_benchmark.jsonl` with one row per problem:
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/apex_shortlist/config.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/arena_hard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ to pick the standard judge prompt.
ng_prepare_benchmark "+config_paths=[benchmarks/arena_hard/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/arena_hard/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/arena_hard_v2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ repo, joins by `uid`, and emits one row per question with `question`,
ng_prepare_benchmark "+config_paths=[benchmarks/arena_hard_v2/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/arena_hard_v2/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/asr_leaderboard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Downloads the 8 ESB subsets (~tens of GB of FLAC) and writes
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/asr_leaderboard/config.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/bigcodebench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ the `full` split (~1140 problems) is `bigcode/bigcodebench@v0.1.4`.
ng_prepare_benchmark "+config_paths=[benchmarks/bigcodebench/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/bigcodebench/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/birdbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ and writes `data/birdbench_benchmark.jsonl`. Each row has
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/birdbench/config.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/browsecomp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ browsecomp_benchmark_resources_server:
exclude_domains_file_path: ???
Qwen3-235B-A22B-Instruct-2507-FP8:
responses_api_models:
vllm_model:
vllm_endpoint:
base_url: ???
api_key: ???
```
Expand All @@ -23,7 +23,7 @@ ng_prepare_benchmark "+config_paths=[$config_paths]"
```bash
WANDB_PROJECT=
EXPERIMENT_NAME=
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/browsecomp/config.yaml"
ng_e2e_collect_rollouts \
"+config_paths=[${config_paths}]" \
Expand Down
6 changes: 3 additions & 3 deletions benchmarks/browsecomp/config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Refer to the params in responses_api_models/local_vllm_model/configs/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8.yaml
# Refer to the params in responses_api_models/vllm_server/configs/Qwen/Qwen3-235B-A22B-Instruct-2507-FP8.yaml
Qwen3-235B-A22B-Instruct-2507-FP8:
responses_api_models:
vllm_model:
vllm_endpoint:
entrypoint: app.py
model: Qwen3-235B-A22B-Instruct-2507-FP8
replace_developer_role_with_system: true
Expand All @@ -11,7 +11,7 @@ Qwen3-235B-A22B-Instruct-2507-FP8:
policy_model_no_interleaved_reasoning:
_copy: policy_model
responses_api_models:
vllm_model:
vllm_endpoint:
uses_interleaved_reasoning: false

browsecomp_benchmark_resources_server:
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/finance_sec_search/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ JSONL to `data/`.
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/finance_sec_search/config_no_web_search.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/flores200/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ only advertised on multi-node SLURM deployments via NeMo-Skills'
override and rely on corpus-BLEU only:

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/flores200/config.yaml"
ng_run "+config_paths=[$config_paths]" \
"++flores200_wmt_translation_resources_server.resources_servers.wmt_translation.compute_comet=false"
Expand Down Expand Up @@ -102,7 +102,7 @@ ns nemo_gym_rollouts \
--server_nodes 2 \
--server_args "--tensor-parallel-size 8 --data-parallel-size 1 --data-parallel-size-local 1 --data-parallel-backend ray --distributed-executor-backend ray --api-server-count 1 --reasoning-parser deepseek_r1 --trust-remote-code --dtype auto --enforce-eager" \
--model "$MODEL" \
--config_paths "benchmarks/flores200/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml" \
--config_paths "benchmarks/flores200/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml" \
--input_file benchmarks/flores200/data/flores200_devtest_benchmark.jsonl \
--output_dir /workspace/flores200_smoke \
--expname flores200_smoke \
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/frontierscience_olympiad/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ example, the original Skills configuration uses `o3-mini-2025-01-31` via
ng_prepare_benchmark "+config_paths=[benchmarks/frontierscience_olympiad/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/frontierscience_olympiad/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/gdpval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/gdpval/config.yaml]"
Each deliverable is scored 0–1 against the task rubric.

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/gdpval/config.yaml"
ng_e2e_collect_rollouts \
"+config_paths=[${config_paths}]" \
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/gdpval/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Run:
# ng_prepare_benchmark "+config_paths=[benchmarks/gdpval/config.yaml]"
# ng_e2e_collect_rollouts \
# "+config_paths=[responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/gdpval/config.yaml]" \
# "+config_paths=[responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,benchmarks/gdpval/config.yaml]" \
# ++split=benchmark \
# ++output_jsonl_fpath=results/gdpval.jsonl
#
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/global-piqa/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ server.
ng_prepare_benchmark "+config_paths=[benchmarks/global-piqa/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/global-piqa/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/gpqa-x/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ python benchmarks/gpqa-x/prepare.py --prompt_language en
## Quickstart

```bash
ng_run "+config_paths=[benchmarks/gpqa-x/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/gpqa-x/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"
```

Then in another shell:
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/gpqa/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ This benchmark uses the `mcqa` resource server with the `mcqa_simple_agent`.
ng_prepare_benchmark "+config_paths=[benchmarks/gpqa/config.yaml]"

# Start servers
ng_run "+config_paths=[benchmarks/gpqa/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/gpqa/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"

# Collect rollouts
ng_collect_rollouts \
"+config_paths=[benchmarks/gpqa/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
"+config_paths=[benchmarks/gpqa/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]" \
+output_jsonl_fpath=results/gpqa.jsonl
```
2 changes: 1 addition & 1 deletion benchmarks/graphwalks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ python benchmarks/graphwalks/prepare.py \
## Start environment

```bash
ng_run "+config_paths=[benchmarks/graphwalks/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/graphwalks/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"
```

## Collect rollouts
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/gsm8k/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ the expected answer is integer-valued), then renames `problem` ->
ng_prepare_benchmark "+config_paths=[benchmarks/gsm8k/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/gsm8k/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/hendrycks_math/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ applies Skills' renames (`answer` -> `expected_answer`, `question` ->
ng_prepare_benchmark "+config_paths=[benchmarks/hendrycks_math/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/hendrycks_math/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/hle/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Downloads `cais/hle`, filters to text-only questions, and writes
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/hle/config.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/hmmt_feb25/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Start the benchmark's servers (inherits `math_with_judge` in symbolic-only mode
plus a vLLM model server — adjust the model config to match your deployment):

```
ng_run "+config_paths=[benchmarks/hmmt_feb25/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"
ng_run "+config_paths=[benchmarks/hmmt_feb25/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]"
```

In a separate shell, collect rollouts against the full 30-problem set. `num_repeats`
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/hmmt_nov25/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ few-shots).
Start vLLM with the `--reasoning-parser` that matches your model
(e.g. `deepseek_r1` for models with a `<think>…</think>` convention;
the parser name is declared in
`responses_api_models/local_vllm_model/configs/nvidia/*.yaml`). Without
`responses_api_models/vllm_server/configs/nvidia/*.yaml`). Without
one, `math_with_judge` may extract intermediate expressions from
truncated rollouts, and Skills' `parse_reasoning=True` default diverges
on the same inputs.
Expand All @@ -38,7 +38,7 @@ on the same inputs.
ng_prepare_benchmark "+config_paths=[benchmarks/hmmt_nov25/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/hmmt_nov25/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/hotpotqa_closedbook/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/hotpotqa_closedbook/config.yaml]
## Run servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/hotpotqa_closedbook/config.yaml"
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,benchmarks/hotpotqa_closedbook/config.yaml"
ng_run "+config_paths=[$config_paths]"
```

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/human_eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ holds the dataset definition + prompt + prepare script.
ng_prepare_benchmark "+config_paths=[benchmarks/human_eval/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/human_eval/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/human_eval_infilling/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ holds only the dataset definition + prompt + prepare script.
ng_prepare_benchmark "+config_paths=[benchmarks/human_eval_infilling/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/human_eval_infilling/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/ifbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/ifbench/config.yaml]"

```bash
ng_e2e_collect_rollouts \
"+config_paths=[benchmarks/ifbench/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]" \
"+config_paths=[benchmarks/ifbench/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]" \
++output_jsonl_fpath=results/benchmarks/ifbench.jsonl \
++overwrite_metrics_conflicts=true \
++split=benchmark \
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/ifeval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/ifeval/config.yaml]"
## Running servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/ifeval/config.yaml"
ng_run "+config_paths=[$config_paths]"
```
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/imo_answerbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/imo_answerbench/config.yaml]"
## Run servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,benchmarks/imo_answerbench/config.yaml"
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,benchmarks/imo_answerbench/config.yaml"
ng_run "+config_paths=[$config_paths]"
```

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/imo_gradingbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ details.
ng_prepare_benchmark "+config_paths=[benchmarks/imo_gradingbench/config.yaml]"

# Running servers
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/imo_gradingbench/config.yaml"
ng_run "+config_paths=[$config_paths]"

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/imo_proofbench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ ng_prepare_benchmark "+config_paths=[benchmarks/imo_proofbench/config.yaml]"
## Run servers

```bash
config_paths="responses_api_models/vllm_model/configs/vllm_model.yaml,\
config_paths="responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml,\
benchmarks/imo_proofbench/config.yaml"
ng_run "+config_paths=[$config_paths]" \
+judge_base_url=https://generativelanguage.googleapis.com/v1beta/openai \
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/ioi/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ ng_prepare_data +config_paths=[benchmarks/ioi/config.yaml] \
+output_dirpath=benchmarks/ioi/data \
+mode=benchmark_preparation

ng_run +config_paths=[benchmarks/ioi/config.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]
ng_run +config_paths=[benchmarks/ioi/config.yaml,responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml]

ng_collect_rollouts +agent_name=ioi_simple_agent \
+input_jsonl_fpath=benchmarks/ioi/data/ioi24_benchmark.jsonl \
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/labbench2_vlm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ the agent.
The benchmark chains in `resources_servers/labbench2_vlm/configs/judge_model_openai.yaml`,
which targets an OpenAI-compatible hosted endpoint. To use a different judge
(e.g. a local vLLM), drop that chain and supply your own `responses_api_models`
instance named `judge_model` — see `responses_api_models/vllm_model/configs/vllm_model.yaml`
instance named `judge_model` — see `responses_api_models/vllm_endpoint/configs/vllm_endpoint.yaml`
for the vLLM form.

Credentials go in `env.yaml` at the **repository root** (the parser loads
Expand Down
Loading
Loading