Repository navigation
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -2614,8 +2614,10 @@ def get_batch_defaults( | |||
| # NOTE(Kuntai): Setting large `max_num_batched_tokens` for A100 reduces | |||
| # throughput, see PR #17885 for more details. | |||
| # So here we do an extra device name check to prevent such regression. | |||
| if device_memory >= 160 * GiB_bytes: | |||
| if device_memory >= 160 * GiB_bytes and not current_platform.is_rocm(): | |||
There was a problem hiding this comment.
Maybe we might want this on mi355? this also turns mi355x to use 8192 instead of 16384. @he-yufeng can you validate the perf of mi355 across 8192 and 16384?
|
Honest constraint up front: I have no AMD hardware available to me here, so I cannot produce the mi355 8192-vs-16384 numbers myself. On what the benchmark needs to capture, because it is not steady-state throughput: the failure this PR guards against is the profile_run activation peak at long context (#53961, MI300X), where KV cache init breaks before serving ever starts. So the decisive mi355 experiment is profile_run (or a plain long-context server init) at max_num_batched_tokens=16384, not a tokens/s comparison. If mi355 initializes cleanly at 16384 there, it deserves to stay in the tier and the numbers you are asking for would show it. Two ways to cut it, your call since you can actually run the card: (a) I narrow the exclusion to MI300X/MI325X by name and leave mi355 in the tier pending your validation, or (b) keep the broad ROCm exclusion until you have mi355 data and I expand it back out then. Happy to push either restructure. |
afriedri verified MI355X is fine at 16384 batched tokens, so the broad ROCm gate is replaced by a name check on the MI300X/MI325X family that actually hits the profile_run KV-init failure in vllm-project#53961. Co-authored-by: Kimi K3 Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
1599ec5 to
e6757f9
Compare
|
@afriedri @tjtanaa the rework you asked for is in e6757f9: the >=160GiB batched-token tier now excludes only MI300X/MI325X by name, MI355X stays at 16384 with a regression test, and the five fake-platform tests pass. @yewentao256 invited a ROCm-reviewer ping on the issue, so cc @Rohan138. Anything else you want changed before this lands? |
|
fyi, closing in favor of #56359 which seems to be the correct fix |
4 participants
Purpose
Fixes #53961. #51726 added a
device_memory >= 160GiBtier targeting B200/B300, but it gates on raw capacity, so MI300X (192GB) and MI325X (256GB) fall into it as well.OPENAI_API_SERVERmax_num_batched_tokenssilently doubles (8192 → 16384), the activation peak duringprofile_rundoubles with it, and DeepSeek-V4-Pro can no longer initialize its KV cache at 1M context on MI300X (ValueError: 23.32 GiB KV cache is needed ... larger than the available 13.41 GiB). The replaced 70GiB branch named MI300x explicitly. Gate the new tier to non-ROCm platforms so AMD devices stay on the branch they were on before #51726. No behavioral change for NVIDIA: H100/H200 stay under 160GB, B200/B300 remain in the top tier.Not a duplicate: no open PR references #53961 or the tier gate; #51726 is the change being corrected, and the issue reporter's repro is ROCm-specific.
Test Plan
Added
test_get_batch_defaults_160gib_tier_excludes_rocm(parametrized: B200 keeps 16384, MI300X and MI325X stay at 8192, H100 unchanged). Local pytest execution is blocked on this machine by a broken torch build in the dev venv (native extension crash at import, unrelated to this diff), so the test runs on CI. Lint/type/syntax checks above were run locally; no GPU-dependent verification was claimed.AI assistance: prepared with Kimi K3 assistance; the fix and test were reviewed line by line before submission.