Skip to content

perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cquil11 wants to merge 12 commits into from
Closed

perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1035a2c
perf(agentx): refresh dsv4-gb300-dynamo-vllm-agentic-mtp-agg harness
cquil11 Aug 7, 2026
5a00917
perf(agentx): refresh dsv4-gb300-dynamo-vllm-agentic-mtp-disagg harness
cquil11 Aug 7, 2026
2900e6f
Merge remote-tracking branch 'origin/main' into agent/refresh-dsv4-gb…
cquil11 Aug 7, 2026
f7161a3
Merge remote-tracking branch 'origin/main' into agent/refresh-dsv4-gb…
cquil11 Aug 7, 2026
3c4bedb
fix(agentx): collect GB300 vLLM server metrics
cquil11 Aug 7, 2026
d72626a
fix(agentx): collect GB300 vLLM server metrics
cquil11 Aug 7, 2026
8755c2f
fix(agentx): require GB300 vLLM backend metrics
cquil11 Aug 7, 2026
1befac5
fix(agentx): require GB300 vLLM backend metrics
cquil11 Aug 7, 2026
00eb507
perf(agentx): consolidate GB300 vLLM submission refresh
cquil11 Aug 7, 2026
d60c5e1
Merge origin/main into GB300 vLLM combined refresh
cquil11 Aug 7, 2026
8f88208
fix(agentx): use concurrency-safe mmap reads
cquil11 Aug 8, 2026
146534c
Merge remote-tracking branch 'origin/main' into agent/refresh-dsv4-gb…
cquil11 Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ...arks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp4-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ benchmark:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
IS_MULTINODE: "true"
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache"
Expand Down
1 change: 1 addition & 0 deletions ...arks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/agentic/agg-gb300-tp8-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@ benchmark:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
# Keep aggregate workers in the multinode result schema so ingestion uses
# the zero decode-worker count instead of duplicating TP into P and D.
IS_MULTINODE: "true"
Expand Down
1 change: 1 addition & 0 deletions ...-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb300-1p1d-dep4-dep8-c128-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ benchmark:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
IS_MULTINODE: "true"
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
Expand Down
1 change: 1 addition & 0 deletions ...-slurm-recipes/vllm/deepseek-v4/agentic/disagg-gb300-1p1d-dep8-dep8-c384-mtp-agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ benchmark:
INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace"
RESULT_DIR: "/logs/agentic"
PORT: "8000"
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: "vllm:"
IS_MULTINODE: "true"
AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0"
AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID: "true"
Expand Down
10 changes: 10 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5526,6 +5526,16 @@
- "Enable SGLang backend metrics on every aggregate, prefill, and decode engine, and fail before publishing a partial trace artifact if required sglang: metrics are absent."
- "Record the recipes' active HiCache host-DRAM tier and, for disaggregated points, Dynamo router commit 5a638087 in nvidia-master metadata so generated artifacts no longer report kv_offloading=none, allocated_cpu_dram_gb=0, or a null router."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2477

- config-keys:
- dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg
- dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-disagg
scenario-type:
- agentic-coding
description:
- "Refresh submission with up to date AgentX harness."
- "Require vLLM backend metrics in every AgentX artifact."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2518

Check warning on line 5538 in perf-changelog.yaml

View check run for this annotation

Claude / Claude Code Review

PR title/description missing required bilingual (English/Chinese) format 

This PR's title and description are English-only, missing the bilingual '<English title> / <中文标题>' format and mirroring Chinese section (e.g. `## 中文说明`) required by AGENTS.md. Please add a Chinese title suffix and translated summary/validation section.

Check warning on line 5538 in perf-changelog.yaml

View check run for this annotation

Claude / Claude Code Review

perf-changelog.yaml entry inserted mid-file instead of appended at end 

The new perf-changelog.yaml entry for PR #2518 (lines 5530-5539) was appended right after the #2477 entry, but AGENTS.md requires entries to be appended to the actual END of the file — this entry now sits mid-file, ahead of five already-merged entries (#2353, #2404, #2514, #2515, #2427) that landed after it via a merge from main. This happened because the branch was rebased/merged with origin/main after the original append; per AGENTS.md's documented conflict-recipe (checkout origin/main's copy
Comment on lines 5529 to +5538

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 This PR's title and description are English-only, missing the bilingual ' / <中文标题>' format and mirroring Chinese section (e.g. ## 中文说明) required by AGENTS.md. Please add a Chinese title suffix and translated summary/validation section.

Extended reasoning...

AGENTS.md (the single guidance file CLAUDE.md points to) states explicitly and emphatically near the top of the document:

PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary...) ... This applies to every PR and every issue.

PR #2518's title is perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness — English only, with no / <中文标题> suffix. The PR body's ## Summary and ## Validation sections likewise have no mirroring Chinese translation (no ## 中文说明 or equivalent). This is the PR's own metadata, not a pre-existing file in the repo, so the convention is fully applicable and was skipped entirely rather than partially satisfied.

Why this isn't a false positive: the rule is not aspirational boilerplate — it's actively followed elsewhere in this same repo's history. Commit ed5e88b, for example, uses a fully bilingual title: "Qwen3.5 H100 AgentX MTP recipe with golden-AL simulated acceptance / Qwen3.5 H100 AgentX MTP 配方（黄金 AL 模拟接受）", and its corresponding PR #2427 carries a mirrored Chinese body section. That shows the convention is live and enforced by convention/maintainer expectation, not dead text.

Why it doesn't block CI: the automated "signoff PR check" bot referenced in the recipe-reminder comment on this PR keys only on the CODEOWNER checklist phrase ("As a PR reviewer and CODEOWNER, I have reviewed this and have"), which AGENTS.md explicitly carves out as an English-verbatim exception. There is no automated gate for the bilingual title/body rule itself, so this finding is a documentation/metadata gap rather than something that will fail a check or block merge mechanically.

Step-by-step proof:

  1. Open AGENTS.md and note the bolded bilingual mandate near the top of the file (quoted above), which explicitly says "This applies to every PR and every issue."
  2. Look at PR perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518's title: perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness. Search for a / followed by CJK characters — none exists.
  3. Look at the PR body's ## Summary and ## Validation sections — both are English bullet lists with no accompanying ## 中文说明 (or similarly named) section translating them.
  4. Compare against commit ed5e88b / PR [Klaud Cold] qwen3.5-fp8-h100-sglang-agentic-mtp: Qwen3.5 H100 AgentX MTP recipe with golden-AL simulated acceptance / Qwen3.5 H100 AgentX MTP 配方（黄金 AL 模拟接受） #2427, which does include the bilingual title format, confirming the convention is actively used and not merely theoretical.
  5. Conclude: the PR's own title and description fail an explicit, currently-enforced repo convention.

Fix: append a Chinese translation to the PR title (perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness / 刷新 dsv4-fp4-gb300-dynamo-vllm-agentic-mtp 测试框架, or similar idiomatic phrasing) and add a ## 中文说明 section under the body mirroring the Summary/Validation content, using the AGENTS.md-preferred term mappings (基准测试, 配置, etc.) rather than literal machine translation.

Comment on lines 5529 to +5538

@claude claude Bot Aug 8, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The new perf-changelog.yaml entry for PR #2518 (lines 5530-5539) was appended right after the #2477 entry, but AGENTS.md requires entries to be appended to the actual END of the file — this entry now sits mid-file, ahead of five already-merged entries (#2353, #2404, #2514, #2515, #2427) that landed after it via a merge from main. This happened because the branch was rebased/merged with origin/main after the original append; per AGENTS.md's documented conflict-recipe (checkout origin/main's copy of perf-changelog.yaml, then re-append this PR's entry at the tail), the #2518 entry should be moved to follow the #2427 entry at the true end of the file (currently line 5592).

Extended reasoning...

What the bug is: AGENTS.md line 78 states explicitly: perf-changelog.yaml 'is read chronologically (oldest at top, newest at bottom) - new entries MUST be appended to the END, never inserted in the middle or prepended.' The entry this PR adds for dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg/-disagg (PR #2518) is currently at lines 5530-5539, but the file continues for over 60 more lines afterward with entries for PR #2353 (kimik3-fp4-h200-vllm-agentic), #2404, #2514, #2515, and #2427 — the actual last entry in the file, at line 5592.

How it happened: The PR diff hunk (@@ -5526,6 +5526,16 @@) shows the #2518 entry inserted immediately after the pre-existing #2477 entry, with the kimik3-fp4-h200-vllm-agentic (#2353) entry appearing as unchanged context directly below it — meaning #2353 already existed below the insertion point before this PR touched the file. So at the time the branch's own commits were authored, #2477 genuinely was the tail (this part of the append was correct). But the later 'Merge origin/main into GB300 vLLM combined refresh' commit (d60c5e1) brought in the newer #2353/#2404/#2514/#2515/#2427 entries from main, and a straightforward 3-way merge landed them after the #2518 entry the branch had already added — leaving #2518 stranded in the middle instead of at the tail.

Why nothing already prevents this: process_changelog.py (referenced in AGENTS.md's CI notes) only forbids line deletions via git diff to guard against tampering with historical entries — it does not check ordering or position. So this mid-file insertion passes CI and the benchmark-trigger sweep (which diffs the whole file) unaffected. The only guard against this class of bug is the documented human convention, which wasn't followed here.

The fix AGENTS.md prescribes: AGENTS.md line 100 gives the exact recipe for precisely this merge scenario: git checkout origin/main -- perf-changelog.yaml, then re-append the PR's own changelog entry at the tail. That step was skipped — a normal git merge was used instead, which interleaves entries by diff-hunk order rather than re-applying the PR's entry after main's newer entries.

Proof (verified against current file state):

  1. wc -l perf-changelog.yaml → 5592 lines total.
  2. grep -n 'pr-link' perf-changelog.yaml | tail -8 shows, in file order: [Klaud Cold] glm5.2-fp4-b300-sglang-agentic-mtp: GLM-5.2 NVFP4 B300 AgentX with EAGLE MTP and golden-AL synthetic acceptance / 新增 GLM-5.2 NVFP4 B300 SGLang EAGLE MTP AgentX 配置（黄金 AL 合成接受率） #2447 (5517), perf(agentx): refresh Qwen3.5 GB300 recipes with correct metrics #2477 (5528), perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518 (5538), [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353 (5544), [KimiK3][AgentX]: GB200 DSpark and Simple CPU KV offload #2404 (5558), Add Qwen3.5 FP8 B200 AgentX MTP #2514 (5569), Add Qwen3.5 FP8 B300 AgentX MTP #2515 (5580), [Klaud Cold] qwen3.5-fp8-h100-sglang-agentic-mtp: Qwen3.5 H100 AgentX MTP recipe with golden-AL simulated acceptance / Qwen3.5 H100 AgentX MTP 配方（黄金 AL 模拟接受） #2427 (5592).
  3. Git log confirms [KimiK3][AgentX]: H200 KimiK3 Day 0 support #2353, [KimiK3][AgentX]: GB200 DSpark and Simple CPU KV offload #2404, Add Qwen3.5 FP8 B200 AgentX MTP #2514, Add Qwen3.5 FP8 B300 AgentX MTP #2515, and [Klaud Cold] qwen3.5-fp8-h100-sglang-agentic-mtp: Qwen3.5 H100 AgentX MTP recipe with golden-AL simulated acceptance / Qwen3.5 H100 AgentX MTP 配方（黄金 AL 模拟接受） #2427 are all already-merged commits on main (e.g. 9cc21aa Add Qwen3.5 FP8 B300 AgentX MTP (#2515), ed5e88b ... (#2427)), i.e. older changes than this open PR.
  4. Since perf(agentx): refresh dsv4-fp4-gb300-dynamo-vllm-agentic-mtp harness #2518 is still an open/unmerged PR, it should be the newest (bottommost) entry once merged, per the 'newest at bottom' rule — instead it sits 5 entries above the true tail.

Impact: Because CI doesn't check position, this doesn't break the sweep-trigger mechanism or fail any check — it's purely a violation of the documented chronological-append convention that will confuse future readers scanning the file top-to-bottom for history, and make future merges more error-prone if not corrected now.

Suggested fix: Follow the AGENTS.md merge-conflict recipe: git checkout origin/main -- perf-changelog.yaml to restore the canonical (pre-PR) file, then re-append the #2518 entry (config-keys dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg/-disagg) as the new last entry, after the current #2427 entry at line 5592.


- config-keys:
- kimik3-fp4-h200-vllm-agentic
Expand Down
12 changes: 7 additions & 5 deletions runners/launch_gb300-nv.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,11 +217,13 @@ elif [[ "$IS_AGENTIC" == "1" && $FRAMEWORK == "dynamo-sglang" && $MODEL_PREFIX =
cp -rT "$GITHUB_WORKSPACE/benchmarks/multi_node/srt-slurm-recipes/sglang/deepseek-v4/agentic" \
recipes/sglang/deepseek-v4/agentic
elif [[ "$IS_AGENTIC" == "1" ]]; then
# Agentic recipes use NVIDIA/srt-slurm v1.0.36. This is the upstream
# version validated in InferenceX PR #2302 and includes per-node DP,
# matching Dynamo health counts, multi-node TP port handling, and
# Mooncake compatibility. Keep it pinned so sweeps are reproducible.
git clone --branch v1.0.36 --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" || exit 1
# Agentic recipes use NVIDIA/srt-slurm v1.0.38. In addition to per-node
# DP, matching Dynamo health counts, multi-node TP port handling, and
# Mooncake compatibility, this release injects logical worker endpoint
# metadata and AIPERF_SERVER_METRICS_URLS into custom benchmarks. The
# latter is required for complete AgentX server-metrics artifacts.
# Keep the release pinned so sweeps are reproducible.
git clone --branch v1.0.38 --single-branch https://github.com/NVIDIA/srt-slurm.git "$SRT_REPO_DIR" || exit 1
cd "$SRT_REPO_DIR" || exit 1

mkdir -p recipes/vllm/deepseek-v4/agentic || exit 1
Expand Down
2 changes: 1 addition & 1 deletion utils/aiperf
Open in desktop
Submodule aiperf updated from 743fb8 to bcfc23