Skip to content

dsv4-fp4-b300-vllm: bump to vllm v0.20.0, deep_gemm_mega_moe MoE #1220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
functionstackx wants to merge 2 commits into from
Closed

dsv4-fp4-b300-vllm: bump to vllm v0.20.0, deep_gemm_mega_moe MoE #1220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fa719b0
dsv4-fp4-b300-vllm: bump to vllm v0.20.0, deep_gemm_mega_moe MoE
functionstackx Apr 29, 2026
6eda44c
perf-changelog: add pr-link for dsv4-fp4-b300-vllm entry (#1220)
functionstackx Apr 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2515,7 +2515,7 @@ dsv4-fp8-h200-vllm:
# field, so dp-attn=true is used as the existing vLLM script switch for DP4
# layouts on 4 allocated GPUs.
dsv4-fp4-b300-vllm:
image: vllm/vllm-openai:deepseekv4-cu130
image: vllm/vllm-openai:v0.20.0-cu130
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: b300
Expand Down
16 changes: 12 additions & 4 deletions benchmarks/single_node/dsv4_fp4_b300_vllm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,15 @@ if [ "${EP_SIZE:-1}" -gt 1 ]; then
EP_ARGS=(--enable-expert-parallel)
fi

# Mega-MoE backend and the lower GMU only kick in on the DP-attn path,
# per the vLLM v0.20.0 DeepSeek-V4-Pro recipe. All configs share the
# FULL_AND_PIECEWISE compilation config.
GMU_ARGS=()
MOE_ARGS=()
if [ "${DP_ATTENTION}" = "true" ]; then
MAX_NUM_BATCHED_TOKENS=2048
GMU_ARGS=(--gpu-memory-utilization 0.85)
MOE_ARGS=(--moe-backend deep_gemm_mega_moe)
else
MAX_NUM_BATCHED_TOKENS=$(( ISL * 2 ))
fi
Expand All @@ -66,15 +73,16 @@ start_gpu_monitor

set -x
vllm serve "$MODEL" --host 0.0.0.0 --port "$PORT" \
"${PARALLEL_ARGS[@]}" \
--pipeline-parallel-size 1 \
--kv-cache-dtype fp8 \
--trust-remote-code \
--kv-cache-dtype fp8 \
--block-size 256 \
--no-enable-prefix-caching \
"${PARALLEL_ARGS[@]}" \
"${EP_ARGS[@]}" \
"${GMU_ARGS[@]}" \
"${MOE_ARGS[@]}" \
--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
--attention_config.use_fp4_indexer_cache True \
--attention_config.use_fp4_indexer_cache=True \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--enable-auto-tool-choice \
Expand Down
9 changes: 9 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1969,3 +1969,12 @@
- "Keeps the three validated 8k/1k points: low-latency 1P/1D TP8 conc=1, mid-curve 1P/1D DEP8 conc=256, and max-tpt 3P/1D DEP8 conc=4096"
- "All three recipes run NATS/etcd on a dedicated infra node and use compute-node local NVMe model weights via /mnt/numa1/models/deepseek-v4-pro/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1163

- config-keys:
- dsv4-fp4-b300-vllm
description:
- "Pin image to vllm/vllm-openai:v0.20.0-cu130 (was floating deepseekv4-cu130 tag); DeepGEMM is preinstalled in this image"
- "Use --attention_config.use_fp4_indexer_cache=True and --compilation-config {\"cudagraph_mode\": \"FULL_AND_PIECEWISE\", \"custom_ops\": [\"all\"]} for all configs"
- "Gate --moe-backend deep_gemm_mega_moe and --gpu-memory-utilization 0.85 on DP_ATTENTION=true per the v0.20.0 recipe"
- "Drop --pipeline-parallel-size 1; keep --no-enable-prefix-caching and --max-cudagraph-capture-size 2048"
Comment on lines +1973 to +1979

@claude claude Bot Apr 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new perf-changelog.yaml entry at lines 1973-1979 is missing the required pr-link field, which the Pydantic ChangelogEntry schema (utils/matrix_logic/validation.py:344) declares as a required str with no default on a model that uses extra='forbid'. When utils/process_changelog.py:144 calls ChangelogEntry.model_validate on the diff-extracted entry, it will raise a ValidationError and break the changelog processing workflow. The PR description acknowledges the gap ("pr-link to be filled in via follow-up commit") — please add pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1220 (or whatever the assigned PR number ends up being) before merging.

Extended reasoning...

What the bug is. The new entry appended to perf-changelog.yaml (lines 1973-1979) only carries config-keys and description keys — it has no pr-link. Every other entry in the file (100+ of them, including the mirrored PR #1204 entry just above at line 1963) has a pr-link, so this is unambiguously the required schema.

Why it breaks. utils/matrix_logic/validation.py:344 declares the field as:

pr_link: str = Field(alias="pr-link")

…with no default value, no Optional marker, on a ChangelogEntry model that sets model_config = ConfigDict(extra="forbid", populate_by_name=True) (line 340). Pydantic treats this as a required field — omitting it raises ValidationError: Field required.

The triggering code path. utils/process_changelog.py extracts the added lines from the PR diff (lines 17-41), parses them as YAML via yaml.safe_load (line 118), and then iterates through the resulting list calling ChangelogEntry.model_validate(entry_data) on each entry (line 144). Because this entry is literally what was added in the diff, it is exactly what gets validated — and it will fail.

Impact. The changelog-processing workflow that consumes this file will error out on this entry, blocking whichever CI step / downstream automation depends on it. The PR description itself acknowledges the gap: "Adds a perf-changelog.yaml entry to trigger the affected configs (pr-link to be filled in via follow-up commit once this PR has a number)." Until that follow-up lands, the entry is invalid.

How to fix. Add the pr-link field to the entry, mirroring the format used by every other entry in the file:

- config-keys:
    - dsv4-fp4-b300-vllm
  description:
    - "Pin image to vllm/vllm-openai:v0.20.0-cu130 ..."
    - ...
  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1220

Step-by-step proof.

  1. The diff appends the new entry; process_changelog.py collects the added lines and runs yaml.safe_load on them, producing a list whose last element is {"config-keys": [...], "description": [...]} — no pr-link key.
  2. The loop at process_changelog.py:144 calls ChangelogEntry.model_validate(entry_data) on this dict.
  3. Pydantic walks the model fields. pr_link: str = Field(alias="pr-link") has no default and is not Optional, so Pydantic looks for either pr_link (because populate_by_name=True) or the alias pr-link in the input. Neither key is present.
  4. Pydantic raises pydantic.ValidationError with type=missing, loc=('pr-link',), message "Field required".
  5. That exception propagates up, aborting the processing run and breaking whichever CI/automation step invokes process_changelog.py for this PR.

pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1220
Loading