[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups - #2153
[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups#2153ApostaC wants to merge 19 commits into
Conversation
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
cquil11
added
the
agentx
ChatGPT Codex Connector
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Additional findings (outside current diff — PR may have been updated during review):
-
🟡
perf-changelog.yaml:4720-4727— PR title and description are English-only, but AGENTS.md requires bilingual PR metadata: title format<English title> / <中文标题>and a## 中文说明section in the body mirroring the English summary. Please update the title (e.g. append/ [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache) and add a## 中文说明section to the description before merge.Extended reasoning...
AGENTS.md line 7 states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: ' / <中文标题>'. In the PR/issue body, follow the English content with its Chinese translation (e.g. a '## 中文说明' section mirroring the summary)." This is a repo-wide convention that applies to every PR.
The PR under review is missing both required pieces:
- Title:
[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups— English only, no/ <中文标题>suffix. - Body: two English bullets under
# Summary("Adding LMCache configurations for dsv4-vllm launching script" / "Adding 2 nvidia configs with lmcache agentix") with no## 中文说明section following them.
Step-by-step proof of the violation:
- Open AGENTS.md at line 7 — the bilingual rule is explicit and unambiguous, with the exact title format and the
## 中文说明section pattern spelled out. - Read the PR title as rendered on GitHub — it contains only ASCII characters and no
/separator introducing a Chinese portion. - Read the PR body — the entire description is 3 lines of English (
# Summaryheader plus two bullets); grep for中文说明in the body returns nothing. - Therefore both required bilingual elements are absent, directly violating the rule at AGENTS.md:7.
Impact: This is metadata-only — the benchmark scripts and configs themselves are unaffected, CI will still run, and no runtime behavior changes. But the repo convention exists so that non-English speakers on the team can review and search PR history, and every merged PR sets precedent. The rule is stated as a requirement, not a suggestion.
How to fix (no code change needed):
- Edit the PR title to append a Chinese translation after
/, e.g.[LMCache] Add LMCache configs for dsv4 vllm b200/b300 agentix setups / [LMCache] 为 dsv4 vllm b200/b300 agentix 配置添加 LMCache. - Add a
## 中文说明section to the PR body after the English summary, mirroring the two bullets — for example:## 中文说明 - 为 dsv4-vllm 启动脚本添加 LMCache 配置 - 新增 2 个使用 LMCache agentix 的 NVIDIA 配置
Severity is nit — this is a documentation/metadata policy issue, easy to fix in the PR description with no code impact, but should be addressed before merge since AGENTS.md declares it a requirement.
- Title:
…ntry Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109297629 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29109420492 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29116938835 |
| # The v0.24 images ship fixed bindings; reinstalling there would downgrade. | ||
| VLLM_INSTALLED_VERSION=$(python3 -c "from importlib.metadata import version; print(version('vllm'))") | ||
| if [ "$(printf '%s\n' "$VLLM_INSTALLED_VERSION" 0.24.0 | sort -V | head -n1)" != "0.24.0" ]; then | ||
| agentic_pip_install --quiet --force-reinstall --no-deps \ | ||
| 'nvidia-cutlass-dsl-libs-cu13==4.5.2' |
There was a problem hiding this comment.
generally we don't allow reinstalling kernel libraries
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
thanks for the contribution @ApostaC
can u please update recipes.vllm.ai with ur new configs? https://github.com/vllm-project/recipes
vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes
as in sglang cookbook recipes , they also have support for cpu offloading recipes
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29117029396 |
| agentic_pip_install --quiet --no-cache-dir --no-deps \ | ||
| --force-reinstall "mooncake-transfer-engine-cuda13==$MOONCAKE_VERSION" | ||
| python3 -c "from mooncake.store import MooncakeDistributedStore" >/dev/null | ||
| python3 "$(dirname "$0")/patch_vllm_pr45406.py" |
There was a problem hiding this comment.
generally we wanna avoid patches too
There was a problem hiding this comment.
Yeah, this is what I picked from #2138. Once that is in, I will remove this.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29128619300 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29177997711 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
| # AgentX concurrency counts live session trees, not individual requests. | ||
| # Subagent fan-out can push instantaneous request concurrency above CONC, so | ||
| # leave 2x headroom rather than clipping those bursts at the scheduler. | ||
| MAX_NUM_SEQS=$((2 * CONC)) | ||
|
|
There was a problem hiding this comment.
This is being moved to the top of the script
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29272097721 |
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29272097721 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310150332 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29272097721 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310150332 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310150332 |
Signed-off-by: ApostaC <yihua98@uchicago.edu>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29360027324 |
…sweep Rebased onto main after PR #2224 merged the tuned B200 recipe. The LMCache points now live directly in the official dsv4-fp4-b200-vllm-agentic search space alongside the vllm-simple and Mooncake arms (no standalone config section). LMCache 0.5.1 MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; the lmcache arm drops --enable-cumem-allocator because cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server. Ladder validated in the PR #2231 bring-up sweep: peak ~25.7k total tok/s/GPU at DEP8 conc 72 with 96-98% cache hit. 中文：在 PR #2224 的调优配方合并后 rebase 到 main。LMCache 测试点直接并入 官方 dsv4-fp4-b200-vllm-agentic 搜索空间，与 vllm-simple、Mooncake 分支 并列（不再使用独立配置段）。LMCache 0.5.1 MP server（lmcache_driven 传输模式）+ LMCacheMPConnector（沿用 PR #2153）；lmcache 分支去掉 --enable-cumem-allocator（cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache server）。测试点阶梯已在 PR #2231 调试扫描中验证：DEP8 并发 72 达到峰值约 25.7k total tok/s/GPU，缓存命中率 96–98%。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… 在调优后的 DSV4 FP4 B300 vLLM AgentX 配方上新增 LMCache 分支 (#2232) * feat(agentic): add LMCache arm to tuned DSV4 FP4 B300 vLLM AgentX recipe Combine the tuned B300 recipe from PR #2225 (nightly image, sparse DSV4 FlashInfer attention, FULL_DECODE_ONLY CUDA graphs, AMXF4 mega-MoE) with the LMCache 0.5.1 KV-offload backend from PR #2153, mirroring the B200 arm in PR #2231. The lmcache arm keeps the stock PyTorch caching allocator instead of PYTORCH_ALLOC_CONF=expandable_segments:True (expandable-segment cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache MP server, the same failure mode as --enable-cumem-allocator on B200) and runs otherwise identical serving flags, so backends are directly comparable. Adds a standalone dsv4-fp4-b300-vllm-agentic-lmcache config section mirroring the vllm-simple DEP4/DEP8 concurrency ladders and a perf-changelog entry triggering it. 中文：将 PR #2225 的 B300 调优配方（nightly 镜像、稀疏 DSV4 FlashInfer 注意力、FULL_DECODE_ONLY CUDA graph、AMXF4 mega-MoE）与 PR #2153 的 LMCache 0.5.1 KV 卸载后端合并，对应 B200 的 PR #2231。lmcache 分支仅以 默认 PyTorch 缓存分配器替代 PYTORCH_ALLOC_CONF=expandable_segments:True （expandable segments 的 cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache MP server，与 B200 上 --enable-cumem-allocator 的失败模式相同），其余 serving 参数与其他分支保持一致，便于卸载后端间直接对比。新增独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置（测试点与 vllm-simple 的 DEP4/DEP8 阶梯对齐）及触发它的 perf-changelog 条目。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fill in perf-changelog pr-link with PR #2232 中文：将 perf-changelog 条目的 pr-link 填写为 PR #2232。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(agentic): tier lmcache GPU mem util and add pure-TP lmcache arms Bring-up sweep on PR #2232 (run 29463061871) showed the lmcache arm has <1 GiB GPU headroom at the recipe's gpu-memory-utilization 0.96 (LMCache MP server GPU worker + DeepGEMM/TileLang JIT driver allocations + no expandable segments): every DEP8 point OOMed deterministically growing the torch pool, and 2/9 DEP4 points failed at the margin (JIT module load driver OOM at startup; cuBLAS workspace failure mid-run). Derate DEP8 to the B200-proven 0.92 and DEP4/pure-TP to 0.94. Also add pure-TP LMCache arms (TP8 and TP4, conc [8, 12, 16]) mirroring the upper end of the parent's GPU-resident TP ladders and the B200 lmcache TP8 ladder, for a direct GPU-cache vs LMCache-offload comparison. 中文：PR #2232 的调试扫描（run 29463061871）显示 lmcache 分支在配方的 gpu-memory-utilization 0.96 下 GPU 余量不足 1 GiB（LMCache MP server 的 GPU worker + DeepGEMM/TileLang JIT 驱动层分配 + 无 expandable segments）：所有 DEP8 测试点在扩展 torch 内存池时必然 OOM，DEP4 有 2/9 测试点在边缘失败（启动时 JIT 模块加载驱动 OOM；运行中 cuBLAS workspace 失败）。将 DEP8 降额至 B200 验证过的 0.92，DEP4 与纯 TP 降额至 0.94。 另新增纯 TP 的 LMCache 分支（TP8 与 TP4，并发 [8, 12, 16]），与父配置 GPU 驻留 TP 阶梯的高段及 B200 lmcache TP8 阶梯对齐，便于 GPU 缓存与 LMCache 卸载的直接对比。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema PR #1947 made process_changelog's eval pass select single-node agentic (SWE-bench) eval rows via mark_eval_entries --evals-only, but ChangelogMatrixEntry.evals still only accepted SingleNodeMatrixEntry, so any changelog entry touching an agentic-coding config now fails validation (missing isl/osl, extra kv-offloading/scenario-type/duration). Accept Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry] in evals and add run-eval/eval-only (default false) to the agentic entry model, with a regression test. 中文：PR #1947 使 process_changelog 的评估生成步骤通过 mark_eval_entries --evals-only 选择单节点 agentic（SWE-bench）评估行，但 ChangelogMatrixEntry.evals 仍只接受 SingleNodeMatrixEntry，导致任何涉及 agentic-coding 配置的变更日志条目都无法通过校验（缺少 isl/osl，多出 kv-offloading/scenario-type/duration 字段）。将 evals 改为接受 Union[SingleNodeMatrixEntry, SingleNodeAgenticMatrixEntry]，并为 agentic 条目模型添加 run-eval/eval-only（默认 false）字段，附回归测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Revert "fix(matrix): accept agentic SWE-bench eval rows in the changelog matrix schema" This reverts commit bdc33bab6a72a0847c26be0dbe4e10bd8fe4a2eb to keep this PR scoped to the B300 LMCache recipe. The PR #1947 schema regression (agentic changelog entries fail matrix validation) needs a standalone fix PR; until that merges, check-changelog on this PR is expected to fail. 中文：回退 bdc33ba，使本 PR 仅保留 B300 LMCache 配方相关改动。PR #1947 造成的 schema 回归（agentic 变更日志条目无法通过矩阵校验）需通过独立 PR 修复；在该修复合并前，本 PR 的 check-changelog 预期会失败。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(agentic): fold B300 lmcache arms into the official agentic config Per review: run the LMCache sweep together with the official dsv4-fp4-b300-vllm-agentic config instead of a separate section. The standalone dsv4-fp4-b300-vllm-agentic-lmcache config is removed; the parent now carries one lmcache arm per official arm at a +4 conc offset (TP4 [32, 36, 40, 44]; DEP4 [36, 44, 52, 60, 68, 76]; DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]), all inside the ranges validated in the PR #2232 bring-up sweeps. The changelog entry now triggers only the parent config. 中文：按评审意见，LMCache 与官方 dsv4-fp4-b300-vllm-agentic 配置在同一 扫描中运行，不再单列配置段。移除独立的 dsv4-fp4-b300-vllm-agentic-lmcache 配置；父配置为每个官方分支各增加一个 +4 并发偏移的 lmcache 分支（TP4 [32, 36, 40, 44]；DEP4 [36, 44, 52, 60, 68, 76]；DEP8 [68, 100, 116, 132, 148, 164, 180, 196, 228]），全部处于 PR #2232 调试扫描验证过的范围内。变更日志条目现仅触发 父配置。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +16 conc offset capped at 208 Per review: offset the DEP8 lmcache arm by +16 conc from the official GPU-resident DEP8 ladder and cap the maximum at 208, giving [80, 112, 128, 144, 160, 176, 192, 208]. TP4/DEP4 keep the +4 offset. 中文：按评审意见，DEP8 lmcache 分支相对官方 GPU 驻留 DEP8 阶梯偏移 +16 并发，最大并发限制为 208，即 [80, 112, 128, 144, 160, 176, 192, 208]。 TP4/DEP4 保持 +4 偏移。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(agentic): DEP8 lmcache ladder at +8 conc offset capped at 208 Correct the DEP8 lmcache offset from +16 to +8 (still capped at 208), giving [72, 104, 120, 136, 152, 168, 184, 200]. 中文：将 DEP8 lmcache 分支的并发偏移从 +16 更正为 +8（最大并发仍限制为 208），即 [72, 104, 120, 136, 152, 168, 184, 200]。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Update perf-changelog.yaml --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cameron Quilici <cjquilici@gmail.com>
… / 将 LMCache 分支并入官方 DSV4 FP4 B200 vLLM AgentX 扫描 (#2231) * feat(agentic): add LMCache arm to official DSV4 FP4 B200 vLLM AgentX sweep Rebased onto main after PR #2224 merged the tuned B200 recipe. The LMCache points now live directly in the official dsv4-fp4-b200-vllm-agentic search space alongside the vllm-simple and Mooncake arms (no standalone config section). LMCache 0.5.1 MP server (lmcache_driven transfer mode) + LMCacheMPConnector per PR #2153; the lmcache arm drops --enable-cumem-allocator because cuMem/VMM allocations cannot be CUDA-IPC-exported to the LMCache server. Ladder validated in the PR #2231 bring-up sweep: peak ~25.7k total tok/s/GPU at DEP8 conc 72 with 96-98% cache hit. 中文：在 PR #2224 的调优配方合并后 rebase 到 main。LMCache 测试点直接并入 官方 dsv4-fp4-b200-vllm-agentic 搜索空间，与 vllm-simple、Mooncake 分支 并列（不再使用独立配置段）。LMCache 0.5.1 MP server（lmcache_driven 传输模式）+ LMCacheMPConnector（沿用 PR #2153）；lmcache 分支去掉 --enable-cumem-allocator（cuMem/VMM 分配无法通过 CUDA IPC 导出给 LMCache server）。测试点阶梯已在 PR #2231 调试扫描中验证：DEP8 并发 72 达到峰值约 25.7k total tok/s/GPU，缓存命中率 96–98%。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(agentic): align LMCache DEP8 conc-list with the Mooncake ladder Change the LMCache DEP8 points from the vllm-simple ladder to the Mooncake ladder ([12, 20, 28, 36, 44, 52, 60, 68, 76]) so the two external KV stores are benchmarked at identical concurrency points. TP8 stays on [8, 12, 16] (Mooncake has no TP8 arm). 中文：将 LMCache DEP8 测试点从 vllm-simple 阶梯改为 Mooncake 阶梯 （[12, 20, 28, 36, 44, 52, 60, 68, 76]），使两个外部 KV 存储在完全相同的 并发点上进行基准测试。TP8 保持 [8, 12, 16]（Mooncake 无 TP8 分支）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
|
@ApostaC the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience. This is an automated message. |
|
stale PR. please reopen when ready |
3 participants
Summary