Skip to content

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh #2378

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
seungrokj wants to merge 9 commits into from
Closed

[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache refresh #2378

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
1818c53
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: DEP tuning + LMCache ref…
seungrokj Jul 28, 2026
e10d502
docs(perf-changelog): add dsv4-fp4-mi355x-vllm-agentic entry for #2378
seungrokj Jul 28, 2026
1b1a6f1
Merge branch 'main' into amd/agentx_dsv4_vllm_0728
seungrokj Jul 28, 2026
5237349
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: pin lmcache to v0.5.2 in…
seungrokj Jul 28, 2026
c605e76
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: size max-num-seqs per DP…
seungrokj Jul 28, 2026
4bfe42a
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: derate gpu-mem-util to 0…
seungrokj Jul 28, 2026
2da5c48
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: force lm-eval; switch ar…
seungrokj Jul 28, 2026
a649ac4
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: cut DEP8 max-num-batched…
seungrokj Jul 28, 2026
554c46a
[AMD] [AGENTX] dsv4-fp4-mi355x-vllm-agentic: cap gpu-mem-util at 0.85…
seungrokj Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 52 additions & 21 deletions benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION

GPU_COUNT=$TP
if [[ ! "$GPU_COUNT" =~ ^[1-9][0-9]*$ ]]; then
echo "Error: GPU_COUNT must be a positive integer, got '$GPU_COUNT'" >&2
exit 1
fi
export GPU_COUNT

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
fi
Expand Down Expand Up @@ -109,7 +116,7 @@ case "${KV_OFFLOAD_BACKEND:-}" in
# MI355X nodes have ~2.7 TiB of host DRAM available for offload;
# reserve 2.5 TB for the offload pool (leaves ~200 GB headroom for
# worker RSS / page cache / slurm cgroup).
TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))"
TOTAL_CPU_DRAM_PARTITION_GB="$TOTAL_CPU_DRAM_GB"
# Use vLLM's regular native KV-offload path (OffloadingConnector),
# NOT the SimpleCPUOffloadConnector. The "vllm-native" backend resolves to
# OffloadingConnector by default; setting VLLM_USE_SIMPLE_KV_OFFLOAD=1
Expand All @@ -132,7 +139,7 @@ case "${KV_OFFLOAD_BACKEND:-}" in
# ---- Mooncake config ----------------------------------------------------------
# Embedded mode contributes one segment per GPU rank to a shared
# distributed store, so pre-divide the aggregate host-memory budget.
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / TP))
PER_RANK_GB=$((TOTAL_CPU_DRAM_GB / GPU_COUNT))

#MOONCAKE_VERSION=0.3.11.post1
#apt-get update && apt-get install -y libcurl4 libibverbs1 rdma-core librdmacm1 libnuma1 liburing2
Expand Down Expand Up @@ -276,16 +283,14 @@ EOF

git clone https://github.com/LMCache/LMCache.git
cd LMCache
# https://github.com/LMCache/LMCache/pull/3853
git checkout 9229067cec0b3a63bb8a39368d101db7ac0bc3c1
git checkout v0.5.2
pip install -r requirements/build.txt
pip install grpcio==1.78.0
CXX=hipcc BUILD_WITH_HIP=1 pip install -e . --no-build-isolation
cd ..

python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null

TOTAL_CPU_DRAM_PARTITION_GB="$((TOTAL_CPU_DRAM_GB / (8 / TP)))"
TOTAL_CPU_DRAM_PARTITION_GB="$TOTAL_CPU_DRAM_GB"
# Match the B200 Kimi LMCache setup: keep a 2.5 TB semantic CPU KV
# pool, but let the external MP server own that pool so vLLM does not
# split --kv-offloading-size across TP ranks through the integrated
Expand All @@ -310,7 +315,7 @@ EOF
# object present in L1 but no longer readable. Keep the 2.5 TB pool
# size unchanged and only extend the lookup-to-retrieve lease.
LMCACHE_L1_READ_TTL_SECONDS="${LMCACHE_L1_READ_TTL_SECONDS:-7200}"
LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-256}"
LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-1024}"
LMCACHE_MAX_WORKERS="${LMCACHE_MAX_WORKERS:-$TP}"
export PYTHONHASHSEED="${PYTHONHASHSEED:-0}"
export LMCACHE_BLOCKING_TIMEOUT_SECS=1200
Expand Down Expand Up @@ -351,33 +356,57 @@ EOF
esac
fi

# ---- LLM server config ----------------------------------------------------------
PARALLEL_ARGS=(--tensor-parallel-size "$TP" --data-parallel-size 1)
if [ "$DP_ATTENTION" = "true" ]; then
PARALLEL_ARGS=(--tensor-parallel-size 1 --data-parallel-size "$TP")
fi
if [ "$EP_SIZE" -gt 1 ]; then
PARALLEL_ARGS+=(--enable-expert-parallel)
fi

# DEP8 (TP8 + DP-attention) is a high-concurrency arm tuned separately from the
# smaller DEP arm: larger prefill token budget and long-prefill chunking so
# decode latency stays bounded at high concurrency.
IS_DEP8=false
if [ "$DP_ATTENTION" = "true" ] && [ "$TP" -eq 8 ]; then
IS_DEP8=true
fi

EP_ARGS=()
MODE_ARGS=()
MNBT=8192
if [ "$EP_SIZE" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
MODE_ARGS+=(--enable-ep-weight-filter)
fi
if [ "$DP_ATTENTION" = "true" ]; then
MODE_ARGS+=(--prefill-schedule-interval 8)
if [ "$IS_DEP8" = "true" ]; then
MODE_ARGS+=(
--max-num-batched-tokens $((MNBT * 8))
--long-prefill-token-threshold 16384
)
else
MODE_ARGS+=(--max-num-batched-tokens $MNBT)
fi
fi

# AgentX concurrency counts live session trees, not individual requests.
# Subagent fan-out can push instantaneous request concurrency above CONC, so
# leave 2x headroom rather than clipping those bursts at the scheduler.
#if [ "$DP_ATTENTION" = "true" ]; then
# # The DEP source recipe enforces 2*CONC = DP_WORLD_SIZE*MAX_NUM_SEQS.
# MAX_NUM_SEQS=$((2 * CONC / TP))
#else
# # Preserve the previous TP4 scheduler headroom for agentic fan-out.
# MAX_NUM_SEQS=$((2 * CONC))
#fi
MAX_NUM_SEQS=$((2 * CONC))
CONTEXT_LEN=1048576

echo "Starting vllm server..."
set -x
export VLLM_ROCM_USE_AITER=1
#export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

sleep 180

# https://github.com/vllm-project/vllm/pull/45497
git clone https://gist.github.com/seungrokj/a37ff4d9a52db31752e2d5fa5b192e00
cp a37ff4d9a52db31752e2d5fa5b192e00/gistfile1.txt /usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py

{ set +x; } 2>/dev/null
VLLM_CMD=(
vllm serve "$MODEL_PATH" --served-model-name "$MODEL"
Expand All @@ -387,11 +416,13 @@ VLLM_CMD=(
--async-scheduling
--distributed-executor-backend mp
--kv-cache-dtype fp8
--block-size 256
--max-model-len "$CONTEXT_LEN"
"${PARALLEL_ARGS[@]}"
"${EP_ARGS[@]}"
--gpu-memory-utilization 0.8
"${MODE_ARGS[@]}"
--gpu-memory-utilization 0.95
--moe-backend aiter
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_DECODE_ONLY"}'
--tokenizer-mode deepseek_v4
--tool-call-parser deepseek_v4
--reasoning-parser deepseek_v4
Expand Down
11 changes: 6 additions & 5 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1515,21 +1515,22 @@
# image tag, so bumping sglang is just an image tag bump here. Sweeps
# DP-attention on/off and EP=8.

dsv4-fp4-mi355x-vllm-agentic:
image: vllm/vllm-openai-rocm:nightly-09663abde0f50944a8d5ea30120666024b503faa
image: vllm/vllm-openai-rocm:nightly-49f31d7cee425a6d38f8c5bc76877986daf832ed
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: cluster:mi355x-amds

Check failure on line 1522 in configs/amd-master.yaml

View check run for this annotation

Claude / Claude Code Review

Missing required perf-changelog.yaml entry for image/config bump 

This PR bumps the `dsv4-fp4-mi355x-vllm-agentic` image tag, raises `dram-utilization` (0.60→0.80), and rewrites the search-space in `configs/amd-master.yaml`, but no corresponding entry was appended to `perf-changelog.yaml`. Per AGENTS.md's 'Updating Docker images' section this entry is required — it's the mechanism that triggers the benchmark sweep for the changed config, so without it the PR's own stated test plan ('Full-sweep CI green') will not execute.
Comment on lines 1518 to 1522

@claude claude Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 This PR bumps the dsv4-fp4-mi355x-vllm-agentic image tag, raises dram-utilization (0.60→0.80), and rewrites the search-space in configs/amd-master.yaml, but no corresponding entry was appended to perf-changelog.yaml. Per AGENTS.md's 'Updating Docker images' section this entry is required — it's the mechanism that triggers the benchmark sweep for the changed config, so without it the PR's own stated test plan ('Full-sweep CI green') will not execute.

Extended reasoning...

AGENTS.md's "Updating Docker images" section (around line 152) states explicitly: "Update the image tag in the relevant configs/*-master.yaml and/or benchmarks/*.sh, update any related env vars / config params, and append a perf-changelog.yaml entry (required - triggers benchmarks)." AGENTS.md also documents the mechanism itself (line ~78): perf-changelog.yaml is read chronologically and drives which configs get benchmarked in a sweep; new entries must be appended to the end.

This PR does exactly the class of change that section calls out: it bumps the dsv4-fp4-mi355x-vllm-agentic image (nightly-09663abde... → nightly-49f31d7c...), raises dram-utilization from 0.60 to 0.80, and replaces the search-space with a single new DEP+lmcache arm at conc: 76 (configs/amd-master.yaml:1518-1522). These are precisely "image tag" and "related config params" changes.

The diff only touches two files: benchmarks/single_node/agentic/dsv4_fp4_mi355x_vllm.sh and configs/amd-master.yaml. perf-changelog.yaml is not modified anywhere in this PR, and the tail of that file has no entry referencing this PR or dsv4-fp4-mi355x-vllm-agentic — every other recent recipe-bump PR in the history (e.g. #2327, #2345, #2349, #2341, #2305) appended a corresponding changelog entry for the same kind of change. There is an older entry referencing dsv4-fp4-mi355x-vllm-agentic, but it belongs to the recipe's original creation (PR #2109) — since the changelog is append-only and read chronologically, that stale entry does not re-trigger a sweep for this PR's new image/config changes.

Concrete consequence: per the documented trigger mechanism, the full-sweep CI will not pick up dsv4-fp4-mi355x-vllm-agentic for re-benchmarking as a result of this PR's changes, since nothing was appended to signal that this config needs a fresh run. That directly undermines the PR's own stated Test plan checkbox — "Full-sweep CI green for dsv4-fp4-mi355x-vllm-agentic" — which cannot be exercised through the normal trigger path without the changelog entry. The DEP tuning and LMCache refresh this PR is meant to validate would ship unvalidated by the sweep infrastructure.

Fix: append a perf-changelog.yaml entry at the end of the file for dsv4-fp4-mi355x-vllm-agentic, following the existing schema (config-keys / description / pr-link), describing the image bump, dram-utilization change, and search-space rewrite, so the sweep triggers correctly.

precision: fp4
framework: vllm
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.60
- dram-utilization: 0.80
search-space:
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4, 8, 16, 32, 40, 48] }
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, conc-list: [64, 72], router: { name: vllm-router, version: "0.1.14" } }
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }, conc-list: [32, 40, 48] }
#- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 4, 8, 16, 32, 40, 48] }
#- { tp: 8, ep: 1, dp-attn: true, kv-offloading: none, conc-list: [64, 72], router: { name: vllm-router, version: "0.1.14" } }
#- { tp: 8, ep: 1, dp-attn: false, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }, conc-list: [32, 40, 48] }
- { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }, conc-list: [76] }

Check warning on line 1533 in configs/amd-master.yaml

View check run for this annotation

Claude / Claude Code Review

LMCache version metadata stale in active search-space arm 

The active `dsv4-fp4-mi355x-vllm-agentic` search-space arm still tags `kv-offload-backend.version` as the old LMCache PR-3853 SHA (`9229067cec...`), but `dsv4_fp4_mi355x_vllm.sh` now does `git checkout v0.5.2` instead of that SHA. This is a pre-existing-pattern metadata field that isn't consumed by the launch script, so nothing breaks, but every run of this config will record the wrong LMCache version in benchmark result tracking. Fix: update the yaml `version` to `"v0.5.2"`.

@claude claude Bot Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The active dsv4-fp4-mi355x-vllm-agentic search-space arm still tags kv-offload-backend.version as the old LMCache PR-3853 SHA (9229067cec...), but dsv4_fp4_mi355x_vllm.sh now does git checkout v0.5.2 instead of that SHA. This is a pre-existing-pattern metadata field that isn't consumed by the launch script, so nothing breaks, but every run of this config will record the wrong LMCache version in benchmark result tracking. Fix: update the yaml version to "v0.5.2".

Extended reasoning...

The dsv4-fp4-mi355x-vllm-agentic entry in configs/amd-master.yaml has a single active search-space arm (the other two are commented out) that uses kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" }. That SHA is the old LMCache PR-3853 commit that dsv4_fp4_mi355x_vllm.sh used to check out. This PR's diff to the launch script replaces that with git checkout v0.5.2 (dropping the # https://github.com/LMCache/LMCache/pull/3853 comment and the SHA), but the corresponding yaml metadata for the still-active arm was not updated to match.

The version string is pure provenance metadata, not something that drives behavior: the launch script hardcodes the git ref directly and never reads any LMCache-version env var (compare to VLLM_ROUTER_VERSION, which actually is read from the yaml-equivalent hardcoded value). The yaml's kv-offload-backend dict is serialized via generate_sweep_configs.py, flows through run-sweep.yml's toJson(...) into the KV_OFFLOAD_BACKEND_METADATA env var, and is recorded verbatim by utils/agentic/aggregation/process_agentic_result.py (_validate_kv_offload_env / optional_kv_offload_backend_metadata) into each result's kv_offload_backend field. The only validation performed on that metadata is that name == KV_OFFLOAD_BACKEND (i.e. "lmcache" == "lmcache") — the version is never cross-checked against what's actually installed, and validation.py only special-cases strings starting with image:, so the stale SHA passes through untouched.

Concretely: every future run of the active { tp: 8, ep: 1, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: lmcache, version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" } } arm will build and install LMCache v0.5.2 on the box, but the emitted benchmark result will report kv_offload_backend.version = "9229067cec0b3a63bb8a39368d101db7ac0bc3c1". Anyone looking at result provenance/tracking later (e.g. comparing performance across LMCache versions, or auditing which commit produced a given number) will be misled into thinking the run used the old PR-3853 SHA rather than v0.5.2. The PR description itself confirms the intent to move off that SHA ("LMCache: build v0.5.2 (was the PR-3853 SHA)"), so this is a straightforward oversight rather than a deliberate choice — the two now-commented-out arms in the same file still carry the same stale SHA too, but since they're inactive they don't currently emit misleading results.

This does not cause any functional failure or crash — the sweep will run and pass CI, since nothing validates the version value. The fix is a one-line yaml change: version: "9229067cec0b3a63bb8a39368d101db7ac0bc3c1" → version: "v0.5.2" on the active arm (and optionally the two commented-out arms, for consistency, though those aren't running).


# DSv4-Pro FP4 on MI355X via SGLang. Uses a rocm720 mi35x image built off the
# amd/deepseek_v4 branch in sgl-project/sglang; the SHA is encoded in the
Expand Down