[AMD][AgentX] DSv4 FP4 MI355X agentX vLLM - #2109
Merged
Merged
Conversation
Pin LMCache to a specific commit to work around vLLM scheduler ValueError with hybrid KV cache manager. Bump vLLM image to nightly, tune LMCache config (blocking timeout, transfer mode), and narrow sweep to lmcache conc=72 for focused debugging. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
seungrokj
requested review from
1am9trash,
billishyahao,
chunfangamd and
yctseng0211
as code owners
July 7, 2026 03:48
Contributor
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
seungrokj
changed the title
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Comment on lines
+227
to
+246
| trap cleanup_lmcache_server EXIT | ||
|
|
||
| cleanup_agentic_services() { | ||
| local exit_code=$? | ||
| trap - EXIT INT TERM | ||
| set +e | ||
| stop_background_process_tree "$ROUTER_PID" "vLLM router" | ||
| stop_background_process_tree "$SERVER_PID" "vLLM server" 60 | ||
| stop_background_process_tree "$MOONCAKE_MASTER_PID" "Mooncake master" | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup_agentic_services EXIT | ||
| trap 'exit 130' INT | ||
| trap 'exit 143' TERM | ||
|
|
||
| wait_for_lmcache_ready() { | ||
| { set +x; } 2>/dev/null | ||
| local attempts="${LMCACHE_READY_ATTEMPTS:-120}" | ||
| local tail_pid="" | ||
|
|
Contributor
There was a problem hiding this comment.
🟡 The lmcache branch installs two EXIT traps sequentially at lines 227 and 238 — bash only keeps the last handler per signal, so cleanup_agentic_services silently replaces cleanup_lmcache_server and (because it starts with trap - EXIT INT TERM and never touches $LMCACHE_PID) leaves the LMCache MP server orphaned on every exit path. The server holds LMCACHE_L1_SIZE_GB (~2.5 TB) of pinned host DRAM and stays bound to ports 5555/8080, which can block same-node retries. Fix by calling cleanup_lmcache_server from cleanup_agentic_services (or add stop_background_process_tree "$LMCACHE_PID" "LMCache server"), matching the single-trap pattern in kimik2.5_fp4_b200.sh:54.
Extended reasoning...
The bug\n\nIn the lmcache) branch of dsv4_fp4_mi355x_vllm.sh, two trap ... EXIT statements are registered in sequence:\n\nbash\n# line 227\ntrap cleanup_lmcache_server EXIT\n...\n# line 238\ntrap cleanup_agentic_services EXIT\ntrap 'exit 130' INT\ntrap 'exit 143' TERM\n\n\nBash only keeps one handler per signal — the second trap call silently overwrites the first. The replacement handler cleanup_agentic_services explicitly clears any residual handler with trap - EXIT INT TERM at its top, then only stops ROUTER_PID, SERVER_PID, and MOONCAKE_MASTER_PID. It never references LMCACHE_PID and never calls cleanup_lmcache_server. The MOONCAKE_MASTER_PID cleanup is also dead in this branch — that variable is only assigned in the mooncake) branch.\n\n### Why stop_background_process_tree $SERVER_PID does not reach it\n\nThe LMCache server is launched at line 341-342 as "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 &, so it is a sibling of the vLLM SERVER_PID under the main script's process group, not a descendant. stop_background_process_tree (benchmarks/benchmark_lib.sh:190) kills the given PID plus its descendants — walking down from SERVER_PID cannot reach a sibling.\n\n### Reference pattern\n\nbenchmarks/single_node/agentic/kimik2.5_fp4_b200.sh:54 uses only trap cleanup_lmcache_server EXIT with no override, which is what the author of this new script almost certainly intended before copy-pasting the mooncake-branch cleanup_agentic_services block below the lmcache-branch trap.\n\n### Step-by-step proof\n\n1. KV_OFFLOAD_BACKEND=lmcache, we enter the lmcache) branch.\n2. Line 227 registers trap cleanup_lmcache_server EXIT.\n3. Line 238 executes trap cleanup_agentic_services EXIT — the EXIT handler slot now points to cleanup_agentic_services; cleanup_lmcache_server is defined but unreachable via signal.\n4. LMCache server is spawned at line 341-342; LMCACHE_PID=$! captured.\n5. Script runs, then exits (normal, error, or SIGTERM → exit 143 → EXIT handler fires).\n6. cleanup_agentic_services runs: does trap - EXIT INT TERM; stops $ROUTER_PID, $SERVER_PID, $MOONCAKE_MASTER_PID (unset in this branch, no-op); exits.\n7. $LMCACHE_PID still alive — the process is not a child of SERVER_PID, and no code path kills it. It continues to hold LMCACHE_L1_SIZE_GB (TOTAL_CPU_DRAM_GB / (8 / TP) ≈ 2.5 TB with TP=8) of pinned host memory and to bind LMCACHE_PORT=5555 + LMCACHE_HTTP_PORT=8080.\n\n### Impact under the current sweep\n\nThe current amd-master.yaml sweep is narrowed to a single point (conc-list: [72]), so there is no in-sweep same-container retry. Under slurm, job-scoped cgroup teardown will reap the orphan at job boundary, bounding the practical damage. Any local re-run inside the same container/pod, uncommented sweep expansion back to multiple conc values, or interactive debugging on the runner will collide on ports 5555/8080 and/or fail cudaHostAlloc from leftover pinned pages.\n\n### Fix\n\nAny one of:\n\n1. Invoke cleanup_lmcache_server from inside cleanup_agentic_services (recommended — keeps the composed handler).\n2. Add stop_background_process_tree "$LMCACHE_PID" "LMCache server" to cleanup_agentic_services.\n3. Drop the second trap cleanup_agentic_services EXIT and let the single cleanup_lmcache_server handler stand (matches kimik2.5_fp4_b200.sh).
Comment on lines
+148
to
+161
| git clone https://github.com/kvcache-ai/Mooncake.git | ||
| cd Mooncake | ||
| bash dependencies.sh | ||
| mkdir build | ||
| cd build | ||
| cmake .. | ||
| make -j | ||
| sudo make install # optional, make it ready to be used by vLLM/SGLang | ||
| cd .. | ||
| cd .. | ||
|
|
||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| export INFERENCEX_MOONCAKE_MAX_TRANSFER_BATCH_KEYS=32 | ||
| python3 "$(dirname "$0")/patch_vllm_mooncake_transfer_batches.py" |
Collaborator
There was a problem hiding this comment.
hi @seungrokj whats the reason for needing to build mooncake from source? i thought mooncake has prebuilt ROCm wheels
pip install mooncake-transfer-engine-non-cuda
https://kvcache-ai.github.io/Mooncake/getting_started/quick-start.html
+viz @cquil11
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28840175467 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28843511906 |
Apply vllm-project/vllm#45497 scheduler fix at runtime via gist to resolve ValueError in _update_requests_with_invalid_blocks with hybrid KV cache manager. Expand lmcache conc-list to [16, 32, 48, 64]. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28850302356 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DP-attention conc=[40,56,72] sweep, narrow lmcache to conc=[32,40], and reduce dram-utilization from 0.80 to 0.60. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28856284153 |
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28861274175 |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28869263846 |
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28869505738 |
4 tasks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28910441198 |
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
2 similar comments
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
Contributor
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28911223583 |
Collaborator
Author
|
/reuse-sweep-run |
seungrokj
changed the title
seungrokj
commented
Jul 8, 2026
| # MI355X nodes have ~2.7 TiB of host DRAM available for offload; | ||
| # reserve 2.5 TB for the offload pool (leaves ~200 GB headroom for | ||
| # worker RSS / page cache / slurm cgroup). | ||
| TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))" |
Collaborator
Author
There was a problem hiding this comment.
This will not be needed anymore. will not hurt the CPU DRAM size for TP8 but will remove it in the next PR.
seungrokj
commented
Jul 8, 2026
|
|
||
| python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null | ||
|
|
||
| TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))" |
Collaborator
Author
There was a problem hiding this comment.
This will not be needed anymore. will not hurt the CPU DRAM size for TP8 but will remove it in the next PR.
seungrokj
changed the title
cquil11
added
the
agentx
ChatGPT Codex Connector
adibarra
added a commit
that referenced
this pull request
Jul 12, 2026
…fresh EVALS.md Merge-readiness sweep findings: - BLOCKER: dsv4_fp4_mi355x_vllm.sh arrived via a main-merge (#2109) after the eval-gating rollout, so it was the only 1 of 23 single-node agentic recipes without the EVAL_ONLY/maybe_run_eval block. A live config targets it (configs/amd-master.yaml: dsv4-fp4-mi355x-vllm-agentic), which the generator marks for eval -- under EVAL_ONLY=true the recipe would fall through to the throughput replay and produce no score (failing the eval-scores gate). Appended the standard gating block (matches its sglang sibling). - swebench install pinned to ==4.1.0: the harness CLI flags and the _patch_swebench_scoring anchors are verified against 4.1.0; an unpinned upgrade could drift either. - EVALS.md refreshed to the shipped behavior: agentic-only default, 0.50 gate, 50-slice/full run sizing, and the current knob set/defaults.
Collaborator
|
Recovered the missing agentic ingest for this PR (no GPU rerun).
Follow-up: `recover-reused-ingest.yml`'s `trigger-agentic-ingest` step targets the wrong dispatch endpoint and should be switched to a `repository_dispatch` (this also caused the 07-10 403 failures). 中文：已恢复本 PR 缺失的 agentic 结果入库（未重跑 GPU）。失败目标为合并提交 `cbc9f656` 上的 push-to-main run 29047522866 的 `reuse-ingest-artifacts` 任务：`setup` 通过，但复用校验拒绝了固定来源运行 28911223583（第 3 次尝试）——其 agentic 产物因该运行的重试残留了 2 个重复标识（`conc32 dram-lmcache dpafalse` 与 `conc64 none dpatrue`），`trigger-ingest` 被跳过，未写入数据库。恢复方式：触发 `recover-reused-ingest.yml`（run 29320514843），对来源产物去重（每个标识保留最新分片），校验通过 12 条 agentic 记录，并打包 `reused-ingest-artifacts` 与 `changelog-metadata`。其末步 `trigger-agentic-ingest` 报 403：它调用了 `ingest-agentic-results.yml` 的 workflow_dispatch 端点，但该工作流仅支持 `repository_dispatch`/`workflow_call`。改用正确的 `repository_dispatch`（`event_type: ingest-agentic-results`）完成入库。下游入库 InferenceX-app run 29320820943 成功，插入 12 条 `trace_replay`（两个原重复点现各一条），基准行与服务日志已关联，生产数据库，缓存已失效。后续：`recover-reused-ingest.yml` 的 `trigger-agentic-ingest` 步骤指向了错误的 dispatch 端点，应改为 `repository_dispatch`（这也是 07-10 两次 403 失败的原因）。 |
adibarra
added a commit
that referenced
this pull request
Jul 17, 2026
* feat(evals): add SWE-bench Lite eval (lm-eval generation + swebench harness scoring, Modal-capable)
Add a SWE-bench Lite accuracy eval that generates patches via the lm-eval
harness and scores them with the official swebench evaluation harness.
- utils/evals/swebench_lite.yaml: lm-eval task config for SWE-bench Lite
generation (prompt/doc-to-text, generation kwargs, dataset wiring).
- utils/evals/swebench_score.py: post-processing + scoring. Extracts model
patches from lm-eval output, feeds them to the swebench harness, and emits
a "resolved" rate. Supports running the harness locally or on Modal via
SWEBENCH_USE_MODAL (Modal pass-through so scoring can run off-box).
- utils/collect_eval_results.py: extract_lm_metrics learns a "resolved"
filter branch so the swebench resolved metric is collected alongside the
existing lm-eval metrics.
- utils/evals/thresholds.json: add the swebench_lite threshold entry.
- utils/evals/EVALS.md: document the SWE-bench Lite eval and how scoring works.
- benchmarks/benchmark_lib.sh: add run_swebench_eval, _install_swebench_deps,
maybe_run_eval, and Modal pass-through. run_eval now picks a per-scenario
default framework (agentic-coding -> swebench, fixed-seq-len -> lm-eval);
an explicit EVAL_FRAMEWORK env var or --framework arg overrides the default.
EVAL_TASKS_DIR selects the task yaml.
- utils/evals/test_swebench_eval.py, utils/evals/test_run_eval_dispatch.py:
tests for the scorer and the scenario/framework dispatch precedence.
* feat(evals): agentic-scenario eval selection + routing (swebench on agentic configs)
Wire the SWE-bench Lite eval into the sweep matrix so it runs on agentic
coding configs, and route it through e2e-tests.
- utils/matrix_logic/generate_sweep_configs.py: add mark_eval_entries and
mark_all_eval_entries. For agentic configs these mark exactly one eval
entry per (model, runner, framework, precision) group at the highest
concurrency, single-node only, so each unique agentic config gets one
swebench eval run rather than one per concurrency point.
- utils/matrix_logic/test_generate_sweep_configs.py: add
test_marks_agentic_entry_for_swebench and update TestMarkAllEvalEntries
to cover the agentic marking behavior.
- .github/workflows/e2e-tests.yml: add the agentic-eval-config bucket, a
test-sweep-agentic-evals job, and make collect-evals depend on it. The
AGENTIC_EVAL filter (agentic + no prefill + run-eval) selects the eval
entries; the throughput AGENTIC filter (agentic + not run-eval) excludes
them so throughput and eval runs don't collide.
- benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh: add the eval hook so
the recipe triggers the agentic swebench eval.
* fix(evals): swebench Modal uses --max_workers (no --parallelism in 4.1.0) + bootstrap Modal creds from env
swebench 4.1.0 exposes --max_workers in both Docker and Modal modes; --parallelism
does not exist. Fix run_harness() to emit --max_workers in the Modal branch.
Add _ensure_modal_credentials() to benchmark_lib.sh: swebench's credential
check only looks for ~/.modal.toml, but CI supplies MODAL_TOKEN_ID/
MODAL_TOKEN_SECRET env vars (GitHub secret). The helper bootstraps the file
from the env vars when the file is absent, so the harness check passes. Called
in run_swebench_eval() right after _install_swebench_deps, scoring path only.
Update the Modal test name and assertions, the run_swebench_eval docstring,
and the EVALS.md knobs bullet to document the credential bootstrapping.
* feat(evals): eval-only gating for all single-node agentic recipes
Apply the EVAL_ONLY=true if/else gating pattern (already present in
kimik2.5_fp4_b300.sh) to the remaining 24 single-node agentic recipes in
benchmarks/single_node/agentic/. In eval-only mode each recipe skips the
multi-turn agentic replay and calls maybe_run_eval "$PORT" against the live
server; run_eval auto-selects swebench for the agentic-coding scenario.
The deprecated/ subdirectory was not touched.
* feat(evals): thread Modal credentials + SWEBENCH_USE_MODAL into eval job env
GitHub secrets MODAL_TOKEN_ID/MODAL_TOKEN_SECRET are now available; bootstrap
into ~/.modal.toml happens in benchmark_lib.sh:_ensure_modal_credentials.
SWEBENCH_USE_MODAL is only read by swebench-path functions, so it is inert for
lm-eval/gsm8k jobs.
* fix(evals): adapt agentic eval wiring to AgentX v1.0
- Re-sync test-sweep-agentic-evals inputs with main's test-sweep-agentic:
offloading -> kv-offloading + kv-offload-backend + total-cpu-dram-gb.
- Add EVAL_ONLY/maybe_run_eval tail gating to the agentic recipes AgentX
v1.0 added (dsv4_fp4_b200_sglang, dsv4_fp4_b300_sglang, minimaxm3_fp8_h100/
h200/mi300x/mi325x) so eval-only runs skip the replay like the others.
- test_run_eval_dispatch: set KV_OFFLOADING=none so the new source-time
agentic guard in benchmark_lib.sh is satisfied (dispatch logic unaffected).
* feat(evals): eval-limit smoke knob + robust Modal credential HOME handling
Add EVAL_LIMIT env var to run_lm_eval() so --limit N is appended to the
lm_eval invocation when set, enabling small smoke runs (e.g. 10 instances)
without touching the full ~300-instance swebench suite. Wire the knob through
benchmark-tmpl.yml (new eval-limit input + EVAL_LIMIT env) and e2e-tests.yml
(both workflow_dispatch and workflow_call inputs; passed through to
test-sweep-evals and test-sweep-agentic-evals with: blocks). Document the
variable in utils/evals/EVALS.md.
Harden _ensure_modal_credentials against b300 slurm/pyxis containers where
--export=ALL propagates the HOST's HOME into the container; if HOME is unset,
mkdir -p fails, or the directory isn't writable, remap HOME to
/tmp/inferencex-modal-home before writing ~/.modal.toml. Remap is scoped to
the write path (SWEBENCH_USE_MODAL=true, file absent, tokens present).
Tests: functional shim tests for --limit presence/absence; HOME-remap tests
covering writable home (no remap), read-only parent (remap + 600 perms), and
non-writable existing dir (remap); and a no-op test when SWEBENCH_USE_MODAL=false.
* fix(evals): gate agentic eval marking to evals-only/all-evals + fix empty SWEBENCH_NAMESPACE arg
- Add `include_agentic: bool = False` to `mark_eval_entries`; wrap the
`ag_sn_groups` agentic-marking block in `if include_agentic:` so that
default sweeps no longer set `run-eval: true` on any agentic entry.
The e2e-tests.yml AGENTIC filter (`not x.get('run-eval', False)`) then
routes all agentic entries to the throughput job, restoring main parity.
- Pass `include_agentic=args.evals_only or args.all_evals` in `main()` so
--evals-only and --all-evals continue to mark and select agentic entries.
- Replace `${SWEBENCH_NAMESPACE+--namespace "$SWEBENCH_NAMESPACE"}` with an
`ns_args` array in `run_swebench_eval`; when `SWEBENCH_NAMESPACE=""` the
old form word-split to a bare `--namespace` (argparse error); the array
form safely expands `--namespace ""` or nothing when unset.
- Tests: `test_marks_agentic_entry_for_swebench` updated to pass
`include_agentic=True`; new `test_default_mode_does_not_mark_agentic`
asserts zero agentic entries marked in default mode; new ns_args unit
tests cover unset/empty/value cases plus a static assertion that the old
pattern is gone from benchmark_lib.sh.
* fix(evals): register swebench task via --include_path (pinned lm-eval KeyErrors on unregistered task-name paths)
The pinned lm-eval (0.4.9.2, ref b315ef3) crashes with
KeyError: '<task_name>' in pretty_print_task (tasks/__init__.py:681) when
--tasks is given a file path to an external YAML whose task: name is not in
lm-eval's bundled registry. gsm8k/gpqa_diamond are immune because those
names exist in the bundled registry; swebench_lite is not.
Fix: in run_lm_eval(), add optional EVAL_INCLUDE_PATH support — when set,
injects --include_path "$EVAL_INCLUDE_PATH" just before --tasks; inert when
unset (gsm8k/gpqa production invocations are byte-identical).
In run_swebench_eval(), switch the generation call from
EVAL_TASKS_DIR="$yaml_path" (path form → KeyError)
to
EVAL_TASKS_DIR="$task_name" (name form)
EVAL_INCLUDE_PATH="$(dirname "$yaml_path")" (registers the dir)
with save/restore of both vars so EVAL_INCLUDE_PATH does not leak to
subsequent lm-eval invocations. The dataset_path-from-YAML derivation
(awk over yaml_path) is unchanged — generation and scoring remain in lockstep.
Tests: two shim-based dynamic tests (EVAL_INCLUDE_PATH set/unset → flag
present/absent in argv; --tasks carries name vs. yaml path) and one static
assertion that run_swebench_eval source contains EVAL_INCLUDE_PATH wiring.
* fix(evals): sanitize Modal token env vars (CI secrets pasted with trailing newline fail validation)
Live probe proved it: MODAL_TOKEN_SECRET secret has a trailing whitespace char;
raw auth fails ('Token validation failed'), whitespace-stripped auth succeeds.
Strip whitespace/quotes and re-export in _ensure_modal_credentials so both the
modal client (env) and the bootstrapped ~/.modal.toml are clean.
* fix(evals): scoring timeout guard + artifact preservation on eval failure
- run_swebench_eval: wrap scoring in timeout ${SWEBENCH_SCORE_TIMEOUT:-7200}s.
The overnight 300-instance run stalled ~7h in Modal image builds and held the
b300 allocation until the slurm wall; a stalled backend now fails fast.
- maybe_run_eval: always stage eval artifacts (append_lm_eval_summary) even when
the eval fails, then propagate the rc — samples/predictions survive for
diagnosis instead of dying in the job sandbox.
* feat(evals): --predictions-file mode in swebench_score (agentic generation input)
Agent harnesses (SWE-agent / mini-swe-agent) emit standard predictions.jsonl
directly; this bypasses lm-eval samples parsing and feeds the existing Modal
scoring + results pipeline unchanged. Groundwork for agentic swebench.
* feat(evals): agentic SWE-bench generation via mini-swe-agent + Modal sandboxes
SWEBENCH_GEN_MODE=agentic runs a real agent loop per instance instead of the
single-shot prompt: mini-swe-agent (2.4.5) drives the local OpenAI-compatible
endpoint; each instance's shell executes in a Modal sandbox (swe-rex[modal],
official swebench per-instance images -- no docker needed on the GPU node).
preds.json feeds the existing Modal scoring via --predictions-file (which now
also accepts the dict-keyed preds.json format directly).
- benchmark_lib.sh: _run_swebench_agentic_generation (config overlay, slice via
EVAL_LIMIT, workers/step/timeout knobs), _install_swebench_agent_deps
(mini-swe-agent==2.4.5 + swe-rex[modal]==1.4.0), gen-mode branch in
run_swebench_eval feeding scoring via score_input array.
- swebench_score.py: --predictions-file accepts dict preds.json or JSONL.
- workflows: swebench-gen-mode input threaded e2e-tests -> benchmark-tmpl env.
- tests: shim-driven agentic-generation test + predictions-file format tests.
Single-shot remains the default; agentic is the real SWE-bench setting.
* fix(evals): mini-swe-agent import banner pollutes config-path capture
Fresh installs print a multi-line version banner on import; take only the last
stdout line and validate it is a file. Shim test now emulates the banner.
* fix(evals): raise swe-rex Modal sandbox startup timeout for agentic mode
mini's default startup_timeout=60s is consumed by the cold GB-scale swebench
image pull alone ('Runtime did not start within 0s'). Default 900s via
SWEBENCH_AGENT_STARTUP_TIMEOUT; command timeout 300s (mini default 60s is too
tight for running repo test suites) via SWEBENCH_AGENT_CMD_TIMEOUT.
* feat(evals): preserve agent trajectories + predictions as run artifacts
Trajectories are the primary forensic artifact for agent tuning; they
previously died with the job's temp dir. Copy *.traj* flat into the eval
output (append_lm_eval_summary flattens *.json* into the workspace root),
upload via new globs, and clean up post-upload.
* feat(evals): trajectory-forensics guidance in the agent template
Findings from 10-trajectory deep-dive (first-10 Lite, DSv4):
- 3/5 unresolved agents submitted without ever running the failing test
- 1 agent had the CORRECT fix on disk at step 31, burned 44 steps fighting an
unfixable sandbox C-extension build, and hit the step cap without submitting
- CoT leaks into visible content (deepseek_v4 reasoning parser init failure,
recipe-side follow-up) -- 'execute over prose' guidance mitigates
Replace the static config heredoc with a runtime merger that appends targeted
guidance to mini's instance_template: verify-before-submit, build-failure
escape hatch, submission discipline, step-budget framing. Single merged config
replaces the dual -c chain.
* fix(swebench): stop leaking Modal sandboxes to the 1h runtime_timeout
Every agent sandbox was billing a full hour for ~7-minute instances
(observed: batches dying at 59m59s on the Modal dashboard). Three leaks:
- mini-swe-agent 2.4.5 process_instance() never calls env.stop(), even on
success, so every sandbox lives until runtime_timeout (3600s default).
- swe-rex 1.4.0 ModalDeployment.stop() has its poll check inverted: it
terminates only sandboxes that already exited and skips running ones.
- ModalDeployment.start() leaks the sandbox when the runtime never comes
alive (the startup-timeout failure mode).
Fix: _patch_swebench_agent_cleanup() patches the installed files at dep
install (idempotent, anchor-checked against the pinned versions) so
sandboxes terminate the moment their instance finishes; a post-generation
workspace sweep reaps anything that slips through (crashed workers, outer
timeout kills; SWEBENCH_SANDBOX_SWEEP=0 disables for tests); and the
merged config now sets runtime_timeout explicitly
(SWEBENCH_AGENT_RUNTIME_TIMEOUT, default 3600) as a pure backstop.
No agent-visible behavior change: cleanup happens after instance
completion, so resolved-rate comparisons across runs stay clean.
* fix(swebench): correct resolved-rate denominator; submit tree diff on budget exhaustion
Run-1/3 findings (50 instances, tuned template):
- Metric bug: the harness report's total_instances is the full dataset size
(300) even with EVAL_LIMIT=50, so a 32/50 (64%) run was published as
0.107 and nearly tripped the 0.10 threshold gate. parse_resolved now
prefers submitted_instances over total_instances (identical for
full-split runs).
- 6/50 instances hit LimitsExceeded after 75 steps and submitted NOTHING,
despite forensics showing fixes can be complete mid-run. patched
process_instance now falls back to submitting `git diff` of the working
tree when an instance ends abnormally with a live sandbox (requires rc 0
and a `diff --git` prefix so an error string can never become a patch).
Empty submissions score zero, so the fallback is strictly >=.
- Stage the swebench harness report as swebench_report_<task>.json and
upload it; it names resolved/unresolved per instance and was previously
left behind on the node.
* fix(swebench): hook budget-exhaustion fallback on the normal-return path
Run-2/3 verified the sandbox-cleanup patches (applied on the node, sweep
found 0 lingering sandboxes) but 0 fallback submissions fired while 6
instances still ended LimitsExceeded with empty patches. Root cause: mini's
agent run loop absorbs InterruptAgentFlow (Submitted, LimitsExceeded, ...)
and RETURNS normally with an empty submission -- LimitsExceeded never
reaches process_instance's except branch, which is where the fallback hook
lived (their trajectories carry no traceback/exception_str keys,
confirming the normal-return path).
Move the primary hook to just after agent.run(): any empty submission with
a live sandbox now submits `git diff` of the tree (same rc-0 +
"diff --git"-prefix guards). The except-path hook stays for real
exceptions.
* fix(swebench): cut scoring cost ~2x by patching eval-sandbox cpu=4 -> 2
Two full-300 Modal scorings measured ~$80 each in eval sandboxes alone
(vs $0.99-5.91 for image builds -- caching was never the cost driver).
Root cause: swebench's run_evaluation_modal.py hardcodes cpu=4 per
sandbox; Modal bills reserved cores and the test runs are predominantly
single-threaded pytest.
Patch the installed file at dep install (idempotent, anchor-checked,
numeric-validated) to SWEBENCH_EVAL_SANDBOX_CPU (default 2). Per-instance
tests run somewhat slower on fewer cores; scoring parallelism absorbs it.
* fix(swebench): completion watchdog + partial-preds salvage for generation
Run 29039988325 (full-300, workers=144): all 300 predictions were on disk
by t+60min but mini-extra never exited -- a probabilistic hang-on-exit at
high worker counts (the identical previous run exited cleanly). The
process idled 3h into SWEBENCH_AGENT_TIMEOUT, and the rc!=0 path then
deleted the complete preds.json.
- Completion watchdog: run mini-extra in the background and poll
preds.json (written incrementally per instance); once all expected
instances are present, grant SWEBENCH_AGENT_EXIT_GRACE (300s) for a
clean exit, then kill and count generation complete. Overall
SWEBENCH_AGENT_TIMEOUT deadline retained.
- Salvage: if generation fails with N>0 predictions written, warn and
score the partial set instead of discarding real work (denominator is
submitted instances, so partial runs report honestly over what ran).
- Tests: hung-mini watchdog kill, partial-preds salvage, zero-preds
still-fails.
* feat(swebench): production defaults -- 0.50 threshold, 50-slice CI default, workers=64
Decisions 2026-07-09 after three full-300 validation runs (162/162/163
resolved, 54.0-54.3%):
- Threshold 0.10 -> 0.50: the old value predates the denominator fix and
was effectively decorative; 0.50 sits ~4pts under the observed full-run
floor and well under the 50-slice range (62-68%).
- EVAL_LIMIT empty now defaults to the 50-instance CI slice (~45min GPU +
~$9 Modal); EVAL_LIMIT=full runs the whole split (~1.75h + ~$44) for
release-grade checks. Applies to the agentic swebench path only.
- SWEBENCH_AGENT_WORKERS default 8 -> 64: saturates the serving point
without entering the 144-worker teardown-race territory; full-300
generation drops ~3h -> ~55min.
- Known-bad b300 nodes (005: NCCL init death, 006: wedged nvidia driver)
excluded via SALLOC_EXCLUDE with a tracking comment; remove as infra
repairs them.
* fix(swebench): scenario implies generation mode when SWEBENCH_GEN_MODE unset
Label/changelog-triggered evals pass no swebench-gen-mode, which fell
through to single-shot generation -- ~10% resolved on a healthy config,
an instant false-negative against the new 0.50 gate. Agentic scenarios
now default to the agent loop; explicit SWEBENCH_GEN_MODE still wins.
* feat(swebench): agentic-only generation
Unset SWEBENCH_GEN_MODE now means the agent loop unconditionally, not
just for agentic scenarios -- SWE-bench without the agent loop is not a
meaningful eval (~10% resolved) and the 0.50 gate is calibrated to
agentic scores. single-shot remains solely as an explicit
SWEBENCH_GEN_MODE=single-shot debugging escape hatch.
* fix(swebench): launch-line step_limit display default matches the real default (75)
* fix(swebench): review-wave hardening -- 17 confirmed findings across the eval surface
From a 6-dimension review (bash correctness, python/patch code, workflow
wiring, docs-vs-behavior, credential hygiene, failure modes) with
adversarial verification (41 raw -> 17 confirmed, 24 refuted):
- SWEBENCH_USE_MODAL=false no longer passes --modal (the :+ expansion
fired on any non-empty value, including "false")
- EVAL_LIMIT is validated in the agentic path: positive integer, "full",
or 0 -- a negative/garbage value silently short-circuited the
completion watchdog
- fail-fast when the task YAML dataset_path is not SWE-bench_Lite:
agentic generation is hardcoded to the lite subset, and a divergent
scoring dataset would mis-score every instance
- set -u safety: bare ${RESULT_FILENAME} and ${SPEC_DECODING} in
append_lm_eval_summary
- explicit UTF-8 (reads tolerant, writes clean) on all swebench_score.py
file I/O
- sandbox-sweep scope documented precisely: confined to the current Modal
environment; concurrent agentic-eval legs need per-leg MODAL_ENVIRONMENT
(workflow follow-up) before the matrix ever fans out in parallel
- docs/comments made truthful: knob defaults (workers 64, step limit 75),
gen-mode input descriptions (empty = agentic), stale 0.10-threshold and
dev-Mac-only framing, dangling _patch_swebench_agent_cleanup reference
* fix(swebench): terminate eval sandboxes on instance completion (scoring utilization)
run_instance_modal never finalizes its ModalSandboxRuntime -- the __exit__
that terminates the sandbox exists but nothing calls it -- so every eval
sandbox idle-bills after its tests finish until the 30-min sandbox timeout
or ephemeral-app teardown. Invisible on the 50-slice (all-fast tests, the
app ends in ~2 min and reaps everything); on full-300 the slow tail keeps
the app alive ~40 min and all 300 sandboxes bill ~30 min for ~3 min of
work: measured 152 sandbox-hours ($41.57 at cpu=2) where real test time is
~15-20 sandbox-hours.
Patch (same install-time mechanism, idempotent, anchor-checked): a
finally: on run_instance_modal's main try/except chain terminates the
sandbox on every exit path. Expected full-300 scoring: ~$41 -> ~$5-8.
_patch_swebench_scoring_cpu renamed _patch_swebench_scoring (cpu +
lifecycle hunks).
* feat(swebench): SWEBENCH_AGENT_SANDBOX_CPU knob for agent execution sandboxes
Modal's default sandbox reservation is fractional-core, and the agents run
real test suites inside these sandboxes (verify-before-submit), where a
starved CPU can eat the 300s command timeout and waste agent steps. Optional
knob threads through mini's modal_sandbox_kwargs -> swe-rex ->
modal.Sandbox.create; unset preserves the Modal default (current behavior).
Added for the cost/time pareto sweep.
* feat(swebench): SWEBENCH_EVAL_TIMEOUT knob for per-instance scoring timeout
Post-lifecycle-patch, real test runs bill ~33s each and the scoring wall
(~29 min on full-300) is set almost entirely by the 7 persistently-erroring
instances running to the harness's 1800s default. Optional pass-through to
run_evaluation --timeout; unset preserves the harness default.
* feat(swebench): conc-matched agent workers, 900s scoring timeout, app rename, app-scoped sweep
Pareto-sweep conclusions (11 runs):
- SWEBENCH_AGENT_WORKERS defaults to the config's CONC (else 64): the eval
drives the server at the concurrency its config was tuned for. At conc144
this cut full-run generation 90m -> 51m at identical score (five full
runs: 160-163/300); the old w144 hang risk is contained by the completion
watchdog (three clean w144 runs since).
- SWEBENCH_EVAL_TIMEOUT defaults to 900s: real test runs bill ~33s; only
the persistently-erroring instances touch the ceiling and they gate the
scoring tail.
- swe-rex's hardcoded Modal app name is patched to SWEBENCH_MODAL_APP_NAME
(default infx-evals-swe) so the dashboard shows ours, not the library's.
- The post-generation sweep is now scoped to that app via
Sandbox.list(app_id=...) -- it can no longer touch other apps' sandboxes
in the shared workspace (narrows the concurrent-tenant hazard to
same-app legs only).
Agent-sandbox CPU knob stays unset by default: the sweep measured ~zero
command timeouts at Modal's fractional-core default (1 in 300 on the full
set) -- the agent loop is inference-bound and boosting is pure cost.
* docs: waiver for eval-harness runtime patches (Check 9, PR #1947)
The SWE-bench eval patches three pinned eval-tooling packages at install
time (mini-swe-agent, swe-rex, swebench harness) -- sandbox-lifecycle and
cost fixes measured at ~17x Modal spend reduction. No inference engine or
serving stack is touched; the waiver is filed proactively because the
mechanical shape (heredoc patches in benchmark_lib.sh) matches Check 9's
inline-patch pattern.
* docs: drop eval-tooling waiver -- maintainer call: Check 9 targets engine/serving patches (vendor patchwork), not our own eval-harness tooling; vLLM runs as shipped throughout
* fix(swebench): gate the one un-gated agentic recipe; pin swebench; refresh EVALS.md
Merge-readiness sweep findings:
- BLOCKER: dsv4_fp4_mi355x_vllm.sh arrived via a main-merge (#2109) after the
eval-gating rollout, so it was the only 1 of 23 single-node agentic recipes
without the EVAL_ONLY/maybe_run_eval block. A live config targets it
(configs/amd-master.yaml: dsv4-fp4-mi355x-vllm-agentic), which the generator
marks for eval -- under EVAL_ONLY=true the recipe would fall through to the
throughput replay and produce no score (failing the eval-scores gate).
Appended the standard gating block (matches its sglang sibling).
- swebench install pinned to ==4.1.0: the harness CLI flags and the
_patch_swebench_scoring anchors are verified against 4.1.0; an unpinned
upgrade could drift either.
- EVALS.md refreshed to the shipped behavior: agentic-only default, 0.50 gate,
50-slice/full run sizing, and the current knob set/defaults.
* docs+test: clear merge-readiness-sweep nits (docs truth + EVAL_LIMIT guard test)
Second readiness sweep returned GO (no blockers; mi355x gate fix held).
Clearing the actionable nits:
- EVALS.md: --all-evals no longer says agentic configs are excluded (they're
included and run swebench under evals-only/all-evals; excluded only from the
default non-eval sweep).
- generate_sweep_configs.py: three "(single-shot)" comments corrected to
agentic (generation is agentic-only).
- benchmark-tmpl.yml: eval-limit input description corrected (empty = 50-slice
swebench default, not "full set").
- tests: cover the EVAL_LIMIT positive-integer rejection guard (-5/abc/3.5 ->
fail fast) and the full/0 whole-split sentinels. 53 eval tests pass.
modal left unpinned deliberately: unlike the three source-patched packages, it
is a client to a live hosted service where an exact pin invites client/server
skew.
* refactor(evals): move inline runtime patches to utils/evals/patches/
The lm-eval sitecustomize, mini-swe-agent/swe-rex, and swebench Modal
scorer patches were embedded in benchmark_lib.sh as heredocs. Move each
verbatim into a standalone Python file under utils/evals/patches/ (with
the rationale comments as docstrings) and have the _patch_* shell
helpers invoke them via a BASH_SOURCE-anchored path, matching how
run_lm_eval already anchors task YAMLs.
Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(evals): convert thresholds.json to thresholds.yaml
The repo uses YAML for configuration everywhere else, so move the eval
thresholds config to YAML too. validate_scores.py now parses the config
with yaml.safe_load (JSON is a YAML subset, so legacy JSON configs via
--thresholds still load); on runner hosts without PyYAML, JSON configs
fall back to the stdlib json module and YAML configs fail with an
actionable error instead of silently weakening the gate.
Requested by @cquil11 in PR #1947 review.
Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor(evals): clean runtime patch integration
* style(evals): trim implementation comments
* style(evals): tighten rationale comments
* refactor(evals): call unified eval entrypoint
* docs(evals): explain lm-eval task arguments
* fix(evals): run full SWE-bench Lite by default
* fix(evals): schedule AgentX evals from changelog
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1720917eto work around vLLM schedulerValueErrorin the hybrid KV cache manager's_update_requests_with_invalid_blockspath09663abde) for DSv4-Pro MI355X agentic configlmcache_driventransfer mode, add--supported-transfer-modeflagTest plan
EngineDeadErrororValueErrorin scheduler🤖 Generated with Claude Code