[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估 by adibarra · Pull Request #1947 · SemiAnalysisAI/InferenceX

adibarra · 2026-06-26T23:56:32Z

Summary

Adds SWE-bench Lite as the accuracy evaluation for single-node agentic-coding recipes. Fixed-sequence-length recipes continue to use lm-eval tasks such as GSM8K; run_eval selects the framework from the scenario and still accepts an explicit override.

Execution path

run_eval dispatches agentic recipes to run_swebench_eval.
mini-swe-agent 2.4.5 runs one coding agent per SWE-bench Lite instance against the model's local OpenAI-compatible endpoint.
swe-rex 1.4.0 creates the per-instance execution sandboxes on Modal. Agent concurrency defaults to the recipe concurrency and can be overridden with SWEBENCH_AGENT_WORKERS.
The generated prediction set is scored by the official SWE-bench 4.1.0 harness. CI uses Modal scoring sandboxes; local Docker remains available.
swebench_score.py publishes resolved rate in the existing lm-eval result shape, so artifact collection and validate_scores.py use the same downstream path as the other accuracy evaluations.

The CI default is a 50-instance slice. EVAL_LIMIT=full (or 0) evaluates the complete SWE-bench Lite split. The score gate is exact_match,resolved >= 0.50; the denominator is the submitted instance count for sliced runs and the full set for full runs.

Recipe and workflow integration

Enables eval-only gating for all single-node agentic-coding recipes.
Preserves benchmark-only runs: the agentic accuracy evaluation is entered only by the existing eval-only/all-evals controls.
Threads Modal credentials through the benchmark workflow and bootstraps ~/.modal.toml when CI supplies token environment variables.
Uploads predictions, trajectories, official harness reports, and the normalized result JSON even when generation or scoring fails.
Adds watchdog and timeout controls for agent generation, per-instance scoring, and whole-run scoring so stalled sandboxes do not hold the GPU job indefinitely.

Runtime compatibility patches

The external packages are pinned and patched at runtime because the required fixes are not available in the pinned releases:

mini-swe-agent: terminate the environment on every exit and submit a valid working-tree diff when the agent exhausts its step budget without a submission.
swe-rex: correct Modal sandbox termination and clean up startup failures; use an app name scoped to this evaluation.
SWE-bench Modal scorer: reserve the configured CPU count and terminate each scoring sandbox when its instance completes.
lm-eval: accept reasoning_content when content is empty and avoid adding typed text wrappers unsupported by the TRT endpoint.

The patch implementations live in utils/evals/patches/; benchmark_lib.sh only installs pinned dependencies and invokes those scripts. Every source rewrite is anchor-checked, idempotent, and atomic: an upstream source mismatch fails without writing a partial patch.

Validation

Successful end-to-end workflow runs on the PR branch:

Run	Scope	Result
29061184942	Production-path defaults, 50 instances	32/50 resolved (64%)
29109355974	Post-review merge-readiness validation, 50 instances	36/50 resolved (72%)
29123068609	Full workflow and sandbox lifecycle cost validation, 50 instances	38/50 resolved (76%)
29163378164	App-scoped sweep and concurrency-matched workers, 50 instances	38/50 resolved (76%)

All four workflow runs completed successfully, including generation, official scoring, artifact collection, score validation, and the hardware success-rate job.

github-actions · 2026-06-26T23:56:41Z

Thanks for the contribution! For vLLM & SGLang, please ensure that your recipes is similar to the official vLLM recipes and/or the SGLang cookbook

If it is not, please create a PR first before we can merge your single node PR into the master branch. Let's ensure that the documentation is first class such that the entire ML community can benefit from your hard work! Thank you

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. If re-running failed jobs is attempted, PR authors are responsible for ensuring it passes. See GitHub's docs on re-running failed jobs: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

As a rule of thumb, generally, PR authors should request a review & get a PR approval from the respective companies' CODEOWNERS before requesting a review from core maintainers.

If additional help is needed, PR authors can reach out to core maintainers over Slack.

感谢你的贡献！对于 vLLM 与 SGLang，请确保你的 recipe 与官方 vLLM recipes 和/或 SGLang cookbook 保持一致

如果不一致，请先创建一个 PR，之后我们才能将你的单节点 PR 合并到 master 分支。让我们确保文档保持一流水准，使整个 ML 社区都能从你的辛勤工作中受益！谢谢

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。如果选择重新运行失败的任务，PR 作者有责任确保其最终通过。参见 GitHub 关于重新运行失败任务的文档：https://docs.github.com/en/actions/how-tos/manage-workflow-runs/re-run-workflows-and-jobs#re-running-failed-jobs-in-a-workflow

一般而言，PR 作者应先向相应公司的 CODEOWNERS 请求审阅并获得 PR 批准，然后再请求核心维护者审阅。

如需更多帮助，PR 作者可通过 Slack 联系核心维护者。

…arness scoring, Modal-capable) Add a SWE-bench Lite accuracy eval that generates patches via the lm-eval harness and scores them with the official swebench evaluation harness. - utils/evals/swebench_lite.yaml: lm-eval task config for SWE-bench Lite generation (prompt/doc-to-text, generation kwargs, dataset wiring). - utils/evals/swebench_score.py: post-processing + scoring. Extracts model patches from lm-eval output, feeds them to the swebench harness, and emits a "resolved" rate. Supports running the harness locally or on Modal via SWEBENCH_USE_MODAL (Modal pass-through so scoring can run off-box). - utils/collect_eval_results.py: extract_lm_metrics learns a "resolved" filter branch so the swebench resolved metric is collected alongside the existing lm-eval metrics. - utils/evals/thresholds.json: add the swebench_lite threshold entry. - utils/evals/EVALS.md: document the SWE-bench Lite eval and how scoring works. - benchmarks/benchmark_lib.sh: add run_swebench_eval, _install_swebench_deps, maybe_run_eval, and Modal pass-through. run_eval now picks a per-scenario default framework (agentic-coding -> swebench, fixed-seq-len -> lm-eval); an explicit EVAL_FRAMEWORK env var or --framework arg overrides the default. EVAL_TASKS_DIR selects the task yaml. - utils/evals/test_swebench_eval.py, utils/evals/test_run_eval_dispatch.py: tests for the scorer and the scenario/framework dispatch precedence.

…gentic configs) Wire the SWE-bench Lite eval into the sweep matrix so it runs on agentic coding configs, and route it through e2e-tests. - utils/matrix_logic/generate_sweep_configs.py: add mark_eval_entries and mark_all_eval_entries. For agentic configs these mark exactly one eval entry per (model, runner, framework, precision) group at the highest concurrency, single-node only, so each unique agentic config gets one swebench eval run rather than one per concurrency point. - utils/matrix_logic/test_generate_sweep_configs.py: add test_marks_agentic_entry_for_swebench and update TestMarkAllEvalEntries to cover the agentic marking behavior. - .github/workflows/e2e-tests.yml: add the agentic-eval-config bucket, a test-sweep-agentic-evals job, and make collect-evals depend on it. The AGENTIC_EVAL filter (agentic + no prefill + run-eval) selects the eval entries; the throughput AGENTIC filter (agentic + not run-eval) excludes them so throughput and eval runs don't collide. - benchmarks/single_node/agentic/kimik2.5_fp4_b300.sh: add the eval hook so the recipe triggers the agentic swebench eval.

…1.0) + bootstrap Modal creds from env swebench 4.1.0 exposes --max_workers in both Docker and Modal modes; --parallelism does not exist. Fix run_harness() to emit --max_workers in the Modal branch. Add _ensure_modal_credentials() to benchmark_lib.sh: swebench's credential check only looks for ~/.modal.toml, but CI supplies MODAL_TOKEN_ID/ MODAL_TOKEN_SECRET env vars (GitHub secret). The helper bootstraps the file from the env vars when the file is absent, so the harness check passes. Called in run_swebench_eval() right after _install_swebench_deps, scoring path only. Update the Modal test name and assertions, the run_swebench_eval docstring, and the EVALS.md knobs bullet to document the credential bootstrapping.

Apply the EVAL_ONLY=true if/else gating pattern (already present in kimik2.5_fp4_b300.sh) to the remaining 24 single-node agentic recipes in benchmarks/single_node/agentic/. In eval-only mode each recipe skips the multi-turn agentic replay and calls maybe_run_eval "$PORT" against the live server; run_eval auto-selects swebench for the agentic-coding scenario. The deprecated/ subdirectory was not touched.

…job env GitHub secrets MODAL_TOKEN_ID/MODAL_TOKEN_SECRET are now available; bootstrap into ~/.modal.toml happens in benchmark_lib.sh:_ensure_modal_credentials. SWEBENCH_USE_MODAL is only read by swebench-path functions, so it is inert for lm-eval/gsm8k jobs.

# Conflicts: # benchmarks/single_node/agentic/dsr1_fp4_b200.sh # benchmarks/single_node/agentic/dsr1_fp4_mi355x.sh # benchmarks/single_node/agentic/glm5.1_fp4_mi355x.sh # benchmarks/single_node/agentic/glm5_fp8_b200.sh # benchmarks/single_node/agentic/gptoss_fp4_b200.sh # benchmarks/single_node/agentic/gptoss_fp4_h100.sh # benchmarks/single_node/agentic/gptoss_fp4_h200.sh # benchmarks/single_node/agentic/gptoss_fp4_mi300x.sh # benchmarks/single_node/agentic/gptoss_fp4_mi325x.sh

- Re-sync test-sweep-agentic-evals inputs with main's test-sweep-agentic: offloading -> kv-offloading + kv-offload-backend + total-cpu-dram-gb. - Add EVAL_ONLY/maybe_run_eval tail gating to the agentic recipes AgentX v1.0 added (dsv4_fp4_b200_sglang, dsv4_fp4_b300_sglang, minimaxm3_fp8_h100/ h200/mi300x/mi325x) so eval-only runs skip the replay like the others. - test_run_eval_dispatch: set KV_OFFLOADING=none so the new source-time agentic guard in benchmark_lib.sh is satisfied (dispatch logic unaffected).

…dling Add EVAL_LIMIT env var to run_lm_eval() so --limit N is appended to the lm_eval invocation when set, enabling small smoke runs (e.g. 10 instances) without touching the full ~300-instance swebench suite. Wire the knob through benchmark-tmpl.yml (new eval-limit input + EVAL_LIMIT env) and e2e-tests.yml (both workflow_dispatch and workflow_call inputs; passed through to test-sweep-evals and test-sweep-agentic-evals with: blocks). Document the variable in utils/evals/EVALS.md. Harden _ensure_modal_credentials against b300 slurm/pyxis containers where --export=ALL propagates the HOST's HOME into the container; if HOME is unset, mkdir -p fails, or the directory isn't writable, remap HOME to /tmp/inferencex-modal-home before writing ~/.modal.toml. Remap is scoped to the write path (SWEBENCH_USE_MODAL=true, file absent, tokens present). Tests: functional shim tests for --limit presence/absence; HOME-remap tests covering writable home (no remap), read-only parent (remap + 600 perms), and non-writable existing dir (remap); and a no-op test when SWEBENCH_USE_MODAL=false.

…mpty SWEBENCH_NAMESPACE arg - Add `include_agentic: bool = False` to `mark_eval_entries`; wrap the `ag_sn_groups` agentic-marking block in `if include_agentic:` so that default sweeps no longer set `run-eval: true` on any agentic entry. The e2e-tests.yml AGENTIC filter (`not x.get('run-eval', False)`) then routes all agentic entries to the throughput job, restoring main parity. - Pass `include_agentic=args.evals_only or args.all_evals` in `main()` so --evals-only and --all-evals continue to mark and select agentic entries. - Replace `${SWEBENCH_NAMESPACE+--namespace "$SWEBENCH_NAMESPACE"}` with an `ns_args` array in `run_swebench_eval`; when `SWEBENCH_NAMESPACE=""` the old form word-split to a bare `--namespace` (argparse error); the array form safely expands `--namespace ""` or nothing when unset. - Tests: `test_marks_agentic_entry_for_swebench` updated to pass `include_agentic=True`; new `test_default_mode_does_not_mark_agentic` asserts zero agentic entries marked in default mode; new ns_args unit tests cover unset/empty/value cases plus a static assertion that the old pattern is gone from benchmark_lib.sh.

… KeyErrors on unregistered task-name paths) The pinned lm-eval (0.4.9.2, ref b315ef3) crashes with KeyError: '<task_name>' in pretty_print_task (tasks/__init__.py:681) when --tasks is given a file path to an external YAML whose task: name is not in lm-eval's bundled registry. gsm8k/gpqa_diamond are immune because those names exist in the bundled registry; swebench_lite is not. Fix: in run_lm_eval(), add optional EVAL_INCLUDE_PATH support — when set, injects --include_path "$EVAL_INCLUDE_PATH" just before --tasks; inert when unset (gsm8k/gpqa production invocations are byte-identical). In run_swebench_eval(), switch the generation call from EVAL_TASKS_DIR="$yaml_path" (path form → KeyError) to EVAL_TASKS_DIR="$task_name" (name form) EVAL_INCLUDE_PATH="$(dirname "$yaml_path")" (registers the dir) with save/restore of both vars so EVAL_INCLUDE_PATH does not leak to subsequent lm-eval invocations. The dataset_path-from-YAML derivation (awk over yaml_path) is unchanged — generation and scoring remain in lockstep. Tests: two shim-based dynamic tests (EVAL_INCLUDE_PATH set/unset → flag present/absent in argv; --tasks carries name vs. yaml path) and one static assertion that run_swebench_eval source contains EVAL_INCLUDE_PATH wiring.

…iling newline fail validation) Live probe proved it: MODAL_TOKEN_SECRET secret has a trailing whitespace char; raw auth fails ('Token validation failed'), whitespace-stripped auth succeeds. Strip whitespace/quotes and re-export in _ensure_modal_credentials so both the modal client (env) and the bootstrapped ~/.modal.toml are clean.

…lure - run_swebench_eval: wrap scoring in timeout ${SWEBENCH_SCORE_TIMEOUT:-7200}s. The overnight 300-instance run stalled ~7h in Modal image builds and held the b300 allocation until the slurm wall; a stalled backend now fails fast. - maybe_run_eval: always stage eval artifacts (append_lm_eval_summary) even when the eval fails, then propagate the rc — samples/predictions survive for diagnosis instead of dying in the job sandbox.

…ation input) Agent harnesses (SWE-agent / mini-swe-agent) emit standard predictions.jsonl directly; this bypasses lm-eval samples parsing and feeds the existing Modal scoring + results pipeline unchanged. Groundwork for agentic swebench.

…sandboxes SWEBENCH_GEN_MODE=agentic runs a real agent loop per instance instead of the single-shot prompt: mini-swe-agent (2.4.5) drives the local OpenAI-compatible endpoint; each instance's shell executes in a Modal sandbox (swe-rex[modal], official swebench per-instance images -- no docker needed on the GPU node). preds.json feeds the existing Modal scoring via --predictions-file (which now also accepts the dict-keyed preds.json format directly). - benchmark_lib.sh: _run_swebench_agentic_generation (config overlay, slice via EVAL_LIMIT, workers/step/timeout knobs), _install_swebench_agent_deps (mini-swe-agent==2.4.5 + swe-rex[modal]==1.4.0), gen-mode branch in run_swebench_eval feeding scoring via score_input array. - swebench_score.py: --predictions-file accepts dict preds.json or JSONL. - workflows: swebench-gen-mode input threaded e2e-tests -> benchmark-tmpl env. - tests: shim-driven agentic-generation test + predictions-file format tests. Single-shot remains the default; agentic is the real SWE-bench setting.

Fresh installs print a multi-line version banner on import; take only the last stdout line and validate it is a file. Shim test now emulates the banner.

mini's default startup_timeout=60s is consumed by the cold GB-scale swebench image pull alone ('Runtime did not start within 0s'). Default 900s via SWEBENCH_AGENT_STARTUP_TIMEOUT; command timeout 300s (mini default 60s is too tight for running repo test suites) via SWEBENCH_AGENT_CMD_TIMEOUT.

Trajectories are the primary forensic artifact for agent tuning; they previously died with the job's temp dir. Copy *.traj* flat into the eval output (append_lm_eval_summary flattens *.json* into the workspace root), upload via new globs, and clean up post-upload.

Findings from 10-trajectory deep-dive (first-10 Lite, DSv4): - 3/5 unresolved agents submitted without ever running the failing test - 1 agent had the CORRECT fix on disk at step 31, burned 44 steps fighting an unfixable sandbox C-extension build, and hit the step cap without submitting - CoT leaks into visible content (deepseek_v4 reasoning parser init failure, recipe-side follow-up) -- 'execute over prose' guidance mitigates Replace the static config heredoc with a runtime merger that appends targeted guidance to mini's instance_template: verify-before-submit, build-failure escape hatch, submission discipline, step-budget framing. Single merged config replaces the dual -c chain.

Every agent sandbox was billing a full hour for ~7-minute instances (observed: batches dying at 59m59s on the Modal dashboard). Three leaks: - mini-swe-agent 2.4.5 process_instance() never calls env.stop(), even on success, so every sandbox lives until runtime_timeout (3600s default). - swe-rex 1.4.0 ModalDeployment.stop() has its poll check inverted: it terminates only sandboxes that already exited and skips running ones. - ModalDeployment.start() leaks the sandbox when the runtime never comes alive (the startup-timeout failure mode). Fix: _patch_swebench_agent_cleanup() patches the installed files at dep install (idempotent, anchor-checked against the pinned versions) so sandboxes terminate the moment their instance finishes; a post-generation workspace sweep reaps anything that slips through (crashed workers, outer timeout kills; SWEBENCH_SANDBOX_SWEEP=0 disables for tests); and the merged config now sets runtime_timeout explicitly (SWEBENCH_AGENT_RUNTIME_TIMEOUT, default 3600) as a pure backstop. No agent-visible behavior change: cleanup happens after instance completion, so resolved-rate comparisons across runs stay clean.

… budget exhaustion Run-1/3 findings (50 instances, tuned template): - Metric bug: the harness report's total_instances is the full dataset size (300) even with EVAL_LIMIT=50, so a 32/50 (64%) run was published as 0.107 and nearly tripped the 0.10 threshold gate. parse_resolved now prefers submitted_instances over total_instances (identical for full-split runs). - 6/50 instances hit LimitsExceeded after 75 steps and submitted NOTHING, despite forensics showing fixes can be complete mid-run. patched process_instance now falls back to submitting `git diff` of the working tree when an instance ends abnormally with a live sandbox (requires rc 0 and a `diff --git` prefix so an error string can never become a patch). Empty submissions score zero, so the fallback is strictly >=. - Stage the swebench harness report as swebench_report_<task>.json and upload it; it names resolved/unresolved per instance and was previously left behind on the node.

Run-2/3 verified the sandbox-cleanup patches (applied on the node, sweep found 0 lingering sandboxes) but 0 fallback submissions fired while 6 instances still ended LimitsExceeded with empty patches. Root cause: mini's agent run loop absorbs InterruptAgentFlow (Submitted, LimitsExceeded, ...) and RETURNS normally with an empty submission -- LimitsExceeded never reaches process_instance's except branch, which is where the fallback hook lived (their trajectories carry no traceback/exception_str keys, confirming the normal-return path). Move the primary hook to just after agent.run(): any empty submission with a live sandbox now submits `git diff` of the tree (same rc-0 + "diff --git"-prefix guards). The except-path hook stays for real exceptions.

Two full-300 Modal scorings measured ~$80 each in eval sandboxes alone (vs $0.99-5.91 for image builds -- caching was never the cost driver). Root cause: swebench's run_evaluation_modal.py hardcodes cpu=4 per sandbox; Modal bills reserved cores and the test runs are predominantly single-threaded pytest. Patch the installed file at dep install (idempotent, anchor-checked, numeric-validated) to SWEBENCH_EVAL_SANDBOX_CPU (default 2). Per-instance tests run somewhat slower on fewer cores; scoring parallelism absorbs it.

@cquil11

The repo uses YAML for configuration everywhere else, so move the eval thresholds config to YAML too. validate_scores.py now parses the config with yaml.safe_load (JSON is a YAML subset, so legacy JSON configs via --thresholds still load); on runner hosts without PyYAML, JSON configs fall back to the stdlib json module and YAML configs fail with an actionable error instead of silently weakening the gate. Requested by @cquil11 in PR #1947 review. Co-authored-by: Cameron Quilici <60715037+cquil11@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

# Conflicts: # benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh

… the recipe Finalize the B300 AgentX vLLM search space (dsv4-fp4-b300-vllm-agentic): - Add a TP8 GPU-resident arm at conc [1,2,4,6,8]. - Trim TP4 GPU-resident to conc [1,2,4,6,8,12,16,20]; extend TP4 SimpleCPU lazy-offload down to conc [20,24,28,32,36,40]. - Add an MTP speculative-decoding twin (num_speculative_tokens=3) for every topology (TP8/TP4 GPU-resident, TP4 SimpleCPU, DEP4, DEP8), each mirroring its non-MTP conc-list, routed via the launcher's spec-decoding=mtp suffix to dsv4_fp4_b300_vllm_mtp.sh. MTP script: NUM_SPEC_TOKENS=3 -> TOKENS_PER_SEQ=4, so FULL_DECODE_ONLY cudagraph capture sizes (num_seqs*TOKENS_PER_SEQ) scale to num_seqs*4. Sync the tail with dsv4_fp4_b300_vllm.sh to restore the EVAL_ONLY branch (added upstream by #1947) so MTP configs also run the SWE-bench Lite accuracy eval. Add the perf-changelog entry for dsv4-fp4-b300-vllm-agentic (PR #2258). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tion #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ix schema PR #1947 made process_changelog's eval pass select single-node agentic (SWE-bench) eval rows via mark_eval_entries --evals-only, but ChangelogMatrixEntry.evals still only accepted SingleNodeMatrixEntry, so any changelog entry touching an agentic-coding config now fails validation (missing isl/osl, extra kv-offloading/scenario-type/duration). Accept Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry] in evals and add run-eval/eval-only (default false) to the agentic entry model, with a regression test. 中文：PR #1947 使 process_changelog 的评估生成步骤通过 mark_eval_entries --evals-only 选择单节点 agentic（SWE-bench）评估行，但 ChangelogMatrixEntry.evals 仍只接受 SingleNodeMatrixEntry，导致任何涉及 agentic-coding 配置的变更日志条目都无法通过校验（缺少 isl/osl，多出 kv-offloading/scenario-type/duration 字段）。将 evals 改为接受 Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry]，并为 agentic 条目模型添加 run-eval/eval-only（默认 false）字段，附回归测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…log matrix schema" This reverts commit bdc33bab6a72a0847c26be0dbe4e10bd8fe4a2eb to keep this PR scoped to the B300 LMCache recipe. The PR #1947 schema regression (agentic changelog entries fail matrix validation) needs a standalone fix PR; until that merges, check-changelog on this PR is expected to fail. 中文：回退 bdc33ba，使本 PR 仅保留 B300 LMCache 配方相关改动。PR #1947 造成的 schema 回归（agentic 变更日志条目无法通过矩阵校验）需通过独立 PR 修复；在该修复合并前，本 PR 的 check-changelog 预期会失败。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… sweep job / 修复 agentic SWE-bench 评估调度：新增 agentic_evals 独立分桶与扫描任务 (#2267) * fix: dispatch agentic SWE-bench eval rows via a dedicated bucket and sweep job Since #1947, generate_sweep_configs --evals-only marks the top-concurrency agentic arm of each config as a SWE-bench eval row, but two consumers were never taught about agentic eval rows, so any PR touching a single-node agentic-coding config failed its sweep: 1. Schema: ChangelogMatrixEntry.evals only accepted SingleNodeMatrixEntry (fixed-seq-len), so process_changelog.py's final model_validate rejected the agentic row and check-changelog failed before any GPU work. 2. Dispatch: even with the schema widened, run-sweep.yml's sweep-evals job dispatches every eval row with fixed-seq-len inputs; an agentic row gets empty ISL/OSL/MAX_MODEL_LEN and scenario-type defaults to fixed-seq-len, so the launcher runs the fixed_seq_len benchmark script, which exits on check_env_vars. Fix: route agentic eval rows into a new agentic_evals bucket (mirroring the single_node['agentic'] design) dispatched by a new sweep-agentic-evals job cloned from sweep-agentic's input block plus run-eval/eval-only, so an agentic row can never reach the fixed-seq-len eval dispatch. The agentic model accepts optional run-eval/eval-only carried by eval rows; exclude_none keeps benchmark row output byte-identical. 中文：自 #1947 起，generate_sweep_configs --evals-only 会将每个配置中并发度最高的 agentic 搜索空间条目标记为 SWE-bench 评估行，但下游两个消费方并不认识 agentic 评估行，导致任何涉及单节点 agentic-coding 配置的 PR 扫描失败：其一，schema 层 ChangelogMatrixEntry.evals 仅接受固定序列长度条目，check-changelog 直接报错；其二，调度层 sweep-evals 以固定序列长度的输入派发评估行，ISL/OSL/MAX_MODEL_LEN 为空且 scenario-type 回落为 fixed-seq-len，启动器因此运行错误的基准测试脚本并在 check_env_vars 处退出。本修复新增 agentic_evals 独立分桶（与 single_node['agentic'] 设计一致），由新的 sweep-agentic-evals 任务按 sweep-agentic 的输入派发并附加 run-eval/eval-only，使 agentic 评估行不可能进入固定序列长度的评估调度路径；基准测试行的输出保持逐字节不变。 * test: trim agentic eval bucket tests to essential coverage Collapse fixtures into module constants, merge the accept/reject checks into one test, and drop redundant cases; added test lines go from 212 to 100 with the two core guards kept (agentic rows validate only in agentic_evals; realistic rows split end-to-end through the real schema). 中文：精简 agentic 评估分桶测试——fixture 合并为模块常量，接受/拒绝检查合并为单个测试并移除冗余用例，新增测试行数从 212 降至 100，保留两项核心守卫（agentic 行仅能通过 agentic_evals 校验；真实行形状端到端经过真实 schema 拆分）。 * chore: add changelog entry to validate agentic eval dispatch end-to-end Eval-only, agentic-coding-scoped entry for dsv4-fp4-b300-vllm-agentic so the PR sweep (and the on-push main sweep after merge) runs exactly one job: the DEP8 top-concurrency SWE-bench eval through the new sweep-agentic-evals dispatch, confirming the fix for the eval-only failures first seen on #2258/#2259. 中文：新增 changelog 条目以端到端验证 agentic 评估调度——针对 dsv4-fp4-b300-vllm-agentic 的 evals-only、仅 agentic-coding 场景条目，使 PR 扫描（及合并后 main 分支的 push 触发扫描）只运行一个任务：DEP8 最高并发的 SWE-bench 评估经由新的 sweep-agentic-evals 调度执行，确认对 #2258/#2259 上首次出现的 eval-only 失败的修复。 * ws

… 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支 (#2232) * feat(agentic): add LMCache arm to tuned DSV4 FP4 B300 vLLM AgentX recipe Combine the tuned B300 recipe from PR #2225 (nightly image, sparse DSV4 FlashInfer attention, FULL_DECODE_ONLY CUDA graphs, AMXF4 mega-MoE) with the LMCache 0.5.1 KV-offload backend from PR #2153, mirroring the B200 arm in PR #2231. The lmcache arm keeps the stock PyTorch caching allocator instead of PYTORCH_ALLOC_CONF=expandable_segments:True (expandable-segment cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache MP server, the same failure mode as --enable-cumem-allocator on B200) and runs otherwise identical serving flags, so backends are directly comparable. Adds a standalone dsv4-fp4-b300-vllm-agentic-lmcache config section mirroring the vllm-simple DEP4/DEP8 concurrency ladders and a perf-changelog entry triggering it. 中文：将 PR #2225 的 B300 调优配方（nightly 镜像、稀疏 DSV4 FlashInfer 注意力、FULL_DECODE_ONLY CUDA graph、AMXF4 mega-MoE）与 PR #2153 的 LMCache 0.5.1 KV 卸载后端合并，对应 B200 的 PR #2231。lmcache 分支仅以默认 PyTorch 缓存分配器替代 PYTORCH_ALLOC_CONF=expandable_segments:True （expandable segments 的 cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache MP server，与 B200 上 --enable-cumem-allocator 的失败模式相同），其余 serving 参数与其他分支保持一致，便于卸载后端间直接对比。新增独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置（测试点与 vllm-simple 的 DEP4/DEP8 阶梯对齐）及触发它的 perf-changelog 条目。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fill in perf-changelog pr-link with PR #2232 中文：将 perf-changelog 条目的 pr-link 填写为 PR #2232。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(agentic): tier lmcache GPU mem util and add pure-TP lmcache arms Bring-up sweep on PR #2232 (run 29463061871) showed the lmcache arm has <1 GiB GPU headroom at the recipe's gpu-memory-utilization 0.96 (LMCache MP server GPU worker + DeepGEMM/TileLang JIT driver allocations + no expandable segments): every DEP8 point OOMed deterministically growing the torch pool, and 2/9 DEP4 points failed at the margin (JIT module load driver OOM at startup; cuBLAS workspace failure mid-run). Derate DEP8 to the B200-proven 0.92 and DEP4/pure-TP to 0.94. Also add pure-TP LMCache arms (TP8 and TP4, conc [8, 12, 16]) mirroring the upper end of the parent's GPU-resident TP ladders and the B200 lmcache TP8 ladder, for a direct GPU-cache vs LMCache-offload comparison. 中文：PR #2232 的调试扫描（run 29463061871）显示 lmcache 分支在配方的 gpu-memory-utilization 0.96 下 GPU 余量不足 1 GiB（LMCache MP server 的 GPU worker + DeepGEMM/TileLang JIT 驱动层分配 + 无 expandable segments）：所有 DEP8 测试点在扩展 torch 内存池时必然 OOM，DEP4 有 2/9 测试点在边缘失败（启动时 JIT 模块加载驱动 OOM；运行中 cuBLAS workspace 失败）。将 DEP8 降额至 B200 验证过的 0.92，DEP4 与纯 TP 降额至 0.94。另新增纯 TP 的 LMCache 分支（TP8 与 TP4，并发 [8, 12, 16]），与父配置 GPU 驻留 TP 阶梯的高段及 B200 lmcache TP8 阶梯对齐，便于 GPU 缓存与 LMCache 卸载的直接对比。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema PR #1947 made process_changelog's eval pass select single-node agentic (SWE-bench) eval rows via mark_eval_entries --evals-only, but ChangelogMatrixEntry.evals still only accepted SingleNodeMatrixEntry, so any changelog entry touching an agentic-coding config now fails validation (missing isl/osl, extra kv-offloading/scenario-type/duration). Accept Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry] in evals and add run-eval/eval-only (default false) to the agentic entry model, with a regression test. 中文：PR #1947 使 process_changelog 的评估生成步骤通过 mark_eval_entries --evals-only 选择单节点 agentic（SWE-bench）评估行，但 ChangelogMatrixEntry.evals 仍只接受 SingleNodeMatrixEntry，导致任何涉及 agentic-coding 配置的变更日志条目都无法通过校验（缺少 isl/osl，多出 kv-offloading/scenario-type/duration 字段）。将 evals 改为接受 Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry]，并为 agentic 条目模型添加 run-eval/eval-only（默认 false）字段，附回归测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema" This reverts commit bdc33bab6a72a0847c26be0dbe4e10bd8fe4a2eb to keep this PR scoped to the B300 LMCache recipe. The PR #1947 schema regression (agentic changelog entries fail matrix validation) needs a standalone fix PR; until that merges, check-changelog on this PR is expected to fail. 中文：回退 bdc33ba，使本 PR 仅保留 B300 LMCache 配方相关改动。PR #1947 造成的 schema 回归（agentic 变更日志条目无法通过矩阵校验）需通过独立 PR 修复；在该修复合并前，本 PR 的 check-changelog 预期会失败。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(agentic): fold B300 lmcache arms into the official agentic config Per review: run the LMCache sweep together with the official dsv4-fp4-b300-vllm-agentic config instead of a separate section. The standalone dsv4-fp4-b300-vllm-agentic-lmcache config is removed; the parent now carries one lmcache arm per official arm at a +4 conc offset (TP4 [32, 36, 40, 44]; DEP4 [36, 44, 52, 60, 68, 76]; DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]), all inside the ranges validated in the PR #2232 bring-up sweeps. The changelog entry now triggers only the parent config. 中文：按评审意见，LMCache 与官方 dsv4-fp4-b300-vllm-agentic 配置在同一扫描中运行，不再单列配置段。移除独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置；父配置为每个官方分支各增加一个 +4 并发偏移的 lmcache 分支（TP4 [32, 36, 40, 44]；DEP4 [36, 44, 52, 60, 68, 76]；DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]），全部处于 PR #2232 调试扫描验证过的范围内。变更日志条目现仅触发父配置。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +16 conc offset capped at 208 Per review: offset the DEP8 lmcache arm by +16 conc from the official GPU-resident DEP8 ladder and cap the maximum at 208, giving [80, 112, 128, 144, 160, 176, 192, 208]. TP4/DEP4 keep the +4 offset. 中文：按评审意见，DEP8 lmcache 分支相对官方 GPU 驻留 DEP8 阶梯偏移 +16 并发，最大并发限制为 208，即 [80, 112, 128, 144, 160, 176, 192, 208]。 TP4/DEP4 保持 +4 偏移。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +8 conc offset capped at 208 Correct the DEP8 lmcache offset from +16 to +8 (still capped at 208), giving [72, 104, 120, 136, 152, 168, 184, 200]. 中文：将 DEP8 lmcache 分支的并发偏移从 +16 更正为 +8（最大并发仍限制为 208），即 [72, 104, 120, 136, 152, 168, 184, 200]。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update perf-changelog.yaml --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>

* feat(agentic): split DeepSeek-V4 B300 vLLM recipe Carry the B300-only launcher, config search space, and changelog scope from PR #2202. 中文：拆分 DeepSeek-V4 B300 vLLM AgentX 配方，仅保留 PR #2202 中的 B300 启动器、配置搜索空间和变更日志范围。 * docs(changelog): link B300 replacement PR Point the split B300 changelog entry to PR #2225. 中文：将拆分后的 B300 变更日志条目链接到 PR #2225。 * chore: update conc * update dep8 args Signed-off-by: Jeff Ma <jeffjma@umich.edu> * add more tp configs Signed-off-by: Jeff Ma <jeffjma@umich.edu> * dep8 reduce gpu mem util to 0.92 Signed-off-by: Jeff Ma <jeffjma@umich.edu> * chore: update conc list * docs(changelog): point B300 entry to PR #2241 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LjHXfE1FRN9c5XQqgeTdez * fix(changelog): keep perf-changelog additions-only Restore upstream lines whitespace-for-whitespace and append the B300 entry so the changelog diff contains no deletions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LjHXfE1FRN9c5XQqgeTdez * add mtp * [AgentX] DeepSeek-V4 B300 vLLM: MTP (num_speculative_tokens=3) across the recipe Finalize the B300 AgentX vLLM search space (dsv4-fp4-b300-vllm-agentic): - Add a TP8 GPU-resident arm at conc [1,2,4,6,8]. - Trim TP4 GPU-resident to conc [1,2,4,6,8,12,16,20]; extend TP4 SimpleCPU lazy-offload down to conc [20,24,28,32,36,40]. - Add an MTP speculative-decoding twin (num_speculative_tokens=3) for every topology (TP8/TP4 GPU-resident, TP4 SimpleCPU, DEP4, DEP8), each mirroring its non-MTP conc-list, routed via the launcher's spec-decoding=mtp suffix to dsv4_fp4_b300_vllm_mtp.sh. MTP script: NUM_SPEC_TOKENS=3 -> TOKENS_PER_SEQ=4, so FULL_DECODE_ONLY cudagraph capture sizes (num_seqs*TOKENS_PER_SEQ) scale to num_seqs*4. Sync the tail with dsv4_fp4_b300_vllm.sh to restore the EVAL_ONLY branch (added upstream by #1947) so MTP configs also run the SWE-bench Lite accuracy eval. Add the perf-changelog entry for dsv4-fp4-b300-vllm-agentic (PR #2258). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(changelog): accept single-node agentic eval rows in matrix validation #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] B300 vLLM: split MTP + TP8 into separate config-keys Restructure so the sweep runs only the new work, not the existing aggregate: - Revert dsv4-fp4-b300-vllm-agentic to its main (#2241) search space (TP4 [1..32], SimpleCPU [28..40], DEP4, DEP8) -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b300-vllm-agentic-mtp: MTP twins of the aggregate arms (num_speculative_tokens=3), reshaped TP4 [1..20] / SimpleCPU [20..40]. - Add dsv4-fp4-b300-vllm-agentic-tp8: new TP8 GPU-resident topology at conc [1,2,4,6,8], both non-MTP and MTP. - Point the perf-changelog entry at only the two new keys, so only they sweep. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * align synthetic acceptance length Inject rejection_sample_method=synthetic + synthetic_acceptance_length=2.49 into the dsv4-fp4-b300-vllm MTP speculative-config, matching the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) and the sglang gb300 MTP recipe (SGLANG_SIMULATE_ACC_LEN=2.49). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * reduce GPU memory utilization to 0.95 (except DEP8) TP4, DEP4, and TP8 use gpu-memory-utilization 0.95 (down from 0.96); DEP8 stays at 0.92 for its larger prefill token budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * trim B300 MTP search space: fold TP8 into -mtp, drop -tp8 key, trim DEP4 Remove the dsv4-fp4-b300-vllm-agentic-tp8 config-key entirely; its TP8 GPU-resident MTP arm moves into dsv4-fp4-b300-vllm-agentic-mtp (the non-MTP TP8 baseline is dropped). Also drop conc 64 and 72 from the DEP4 arm (now [32,40,48,56]). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "fix(changelog): accept single-node agentic eval rows in matrix validation" This reverts commit 5887e1a. * MTP synthetic acceptance for throughput, real verification for eval Make the MTP --speculative-config conditional on EVAL_ONLY: throughput points pin synthetic acceptance length 2.49 (dsv4-pro golden AL), while the auto-selected SWE-bench accuracy run (EVAL_ONLY=true) uses real MTP with normal target verification. Synthetic acceptance bypasses verification and corrupts the eval output (0.0000 score). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(agentic): wire B300 MTP context parallelism * fix(agentic): derive B300 MTP GPU count from topology * Update perf-changelog.yaml * fix(changelog): restore PR 2258 entry indentation --------- Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(changelog): accept single-node agentic eval rows in matrix validation #1947 made single-node agentic recipes generate a SWE-bench eval row (run_eval/eval_only + agentic fields) but never widened the changelog matrix schema, so ChangelogMatrixEntry.evals (typed list[SingleNodeMatrixEntry], fixed-seq-len only) rejects every agentic eval row -- breaking check-changelog for any single-node agentic PR. Widen evals to the same Union single_node already uses, and give SingleNodeAgenticMatrixEntry optional run_eval/eval_only (None-default, so benchmark rows are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] DeepSeek-V4 B200 vLLM: add MTP (num_speculative_tokens=3) for every agentic arm Add an MTP speculative-decoding twin (spec-decoding: mtp, num_speculative_tokens=3) for every arm of dsv4-fp4-b200-vllm-agentic (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list, routed via the launcher's spec-decoding=mtp suffix to dsv4_fp4_b200_vllm_mtp.sh. New dsv4_fp4_b200_vllm_mtp.sh forks dsv4_fp4_b200_vllm.sh with only the MTP deltas: --speculative-config {"method":"mtp","num_speculative_tokens":3} and --max-cudagraph-capture-size scaled to MAX_NUM_SEQS*(1+N) tokens so FULL_DECODE_ONLY still covers the largest decode batches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(changelog): add B200 vLLM MTP entry (PR #2259) Document the dsv4-fp4-b200-vllm-agentic MTP twins (num_speculative_tokens=3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [AgentX] B200 vLLM: split MTP into a separate config-key Restructure so the sweep runs only the new MTP work, not the existing aggregate: - Revert dsv4-fp4-b200-vllm-agentic to its main (#2224) search space -- unchanged vs main, so it is not re-run. - Add dsv4-fp4-b200-vllm-agentic-mtp: MTP twins (num_speculative_tokens=3) of the aggregate arms (TP8 GPU-resident, TP8 SimpleCPU, DEP8 SimpleCPU, DEP8 Mooncake), each mirroring its non-MTP conc-list. - Point the perf-changelog entry at only the new key, so only it sweeps. (No separate TP8 key: B200's TP8 arms already exist in the #2224 aggregate.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * align synthetic acceptance length to 2.49 Use MTP synthetic rejection sampling with acceptance length 2.49, the dsv4-pro golden AL (thinking_on, num_speculative_tokens=3) from golden_al_distribution/dsv4_mtp.yaml. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * reduce GPU memory utilization from 0.92 to 0.9 Lower gpu-memory-utilization for the B200 vLLM MTP recipe to 0.9 for extra headroom. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "fix(changelog): accept single-node agentic eval rows in matrix validation" This reverts commit 6efd246. * MTP synthetic acceptance for throughput, real verification for eval Make the B200 MTP --speculative-config conditional on EVAL_ONLY: throughput points pin synthetic acceptance length 2.49 (dsv4-pro golden AL), while the auto-selected SWE-bench accuracy run (EVAL_ONLY=true) uses real MTP with normal target verification. Synthetic acceptance bypasses verification and corrupts the eval output (0.0000 score). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * add missing tool call parser for swe eval Signed-off-by: Jeff Ma <jeffjma@umich.edu> * fix(agentic): preserve B200 MTP PCP GPU count * fix(agentic): derive B200 MTP GPU count from topology --------- Signed-off-by: Jeff Ma <jeffjma@umich.edu> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>

adibarra requested a review from a team June 26, 2026 23:56

github-project-automation Bot added this to InferenceMAX Board Jun 26, 2026

github-code-quality Bot found potential problems Jun 26, 2026

View reviewed changes

Comment thread utils/evals/swebench_score.py Fixed

claude Bot reviewed Jun 27, 2026

View reviewed changes

Comment thread utils/evals/swebench_score.py

Comment thread benchmarks/benchmark_lib.sh

adibarra added 4 commits July 2, 2026 12:13

adibarra force-pushed the feat/swebench-lite-eval branch from ce358e4 to e34035d Compare July 2, 2026 18:44

functionstackx changed the title ~~[WIP] add SWE-bench Lite accuracy eval~~ [WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估 Jul 4, 2026

adibarra added 18 commits July 6, 2026 17:36

fix(evals): mini-swe-agent import banner pollutes config-path capture

9dffaed

Fresh installs print a multi-line version banner on import; take only the last stdout line and validate it is a file. Shim test now emulates the banner.

Merge remote-tracking branch 'origin/main' into feat/swebench-lite-eval

a1df115

cquil11 reviewed Jul 13, 2026

View reviewed changes

Comment thread utils/evals/thresholds.json Outdated

github-actions Bot and others added 3 commits July 13, 2026 21:43

refactor(evals): clean runtime patch integration

f7a46e3

style(evals): trim implementation comments

b819597

Oseltamivir reviewed Jul 14, 2026

View reviewed changes

Comment thread benchmarks/benchmark_lib.sh Outdated

Oseltamivir reviewed Jul 14, 2026

View reviewed changes

Comment thread benchmarks/benchmark_lib.sh Outdated

Oseltamivir reviewed Jul 14, 2026

View reviewed changes

Comment thread benchmarks/benchmark_lib.sh

Oseltamivir reviewed Jul 14, 2026

View reviewed changes

Comment thread benchmarks/benchmark_lib.sh

adibarra added 7 commits July 13, 2026 19:54

style(evals): tighten rationale comments

8a4b470

refactor(evals): call unified eval entrypoint

11e31a2

Merge remote-tracking branch 'origin/main' into feat/swebench-lite-eval

d40079d

# Conflicts: # benchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang.sh

docs(evals): explain lm-eval task arguments

c4a122f

fix(evals): run full SWE-bench Lite by default

636ecfe

fix(evals): schedule AgentX evals from changelog

370f9e3

Merge branch 'main' into feat/swebench-lite-eval

c9fd6b1

adibarra merged commit 2952491 into main Jul 17, 2026
4 checks passed

adibarra deleted the feat/swebench-lite-eval branch July 17, 2026 00:40

github-project-automation Bot moved this to Done in InferenceMAX Board Jul 17, 2026

ruizhang0101 mentioned this pull request Jul 17, 2026

[WIP][AMD][MI355x] dsv4-fp4-mi355x-vllm-agentic, install latest lmcache #2216

Closed

ApostaC mentioned this pull request Jul 17, 2026

[LMCache] Add LMCache arm on tuned DSV4 FP4 B300 vLLM AgentX recipe / 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支 #2232

Merged

10 tasks

This was referenced Jul 18, 2026

Fix agentic SWE-bench eval dispatch: dedicated agentic_evals bucket + sweep job / 修复 agentic SWE-bench 评估调度：新增 agentic_evals 独立分桶与扫描任务 #2267

Merged

Add GLM-5.2 NVFP4 B300 SGLang single-node agentic benchmarks #2268

Merged

Navigation Menu

Uh oh!

[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估 - #1947

[WIP] add SWE-bench Lite accuracy eval / 添加 SWE-bench Lite 准确率评估#1947
adibarra merged 52 commits into
mainfrom
feat/swebench-lite-eval

adibarra commented Jun 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

adibarra commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Execution path

Recipe and workflow integration

Runtime compatibility patches

Validation

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adibarra commented Jun 26, 2026 •

edited

Loading