Skip to content

[ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; make w4a4 use the preshuffle triton+asm by default - #53141

Merged
AndreasKaratzas merged 7 commits into
vllm-project:mainfrom
afriedri:remove_fp4_asm_gemm_env
Aug 28, 2026
Merged

[ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; make w4a4 use the preshuffle triton+asm by default#53141
AndreasKaratzas merged 7 commits into
vllm-project:mainfrom
afriedri:remove_fp4_asm_gemm_env

Conversation

@afriedri

@afriedri afriedri commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

VLLM_ROCM_USE_AITER_FP4_ASM_GEMM defaulted to False, routing MXFP4 dense linears to aiter's Triton gemm_afp4wfp4. That kernel spills (256×256×256 8-warp tile pegged at 256 VGPRs) and is slower than the asm path on every shape measured.

The asm path avoids it entirely — gemm_a4w4, or gemm_afp4wfp4_preshuffled_weight_scales for M≤64 on tuned (N,K). This makes it unconditional on gfx950 and removes the env var.

Results

Llama-3.3-70B MXFP4, TP8, bs=128, 1k/1k, steady state:

Config tok/s TPOT (ms)
Triton gemm_afp4wfp4 (current default) 2708 45.07
asm GEMM 3124 39.25

+15% throughput, −13% TPOT.

AI Assistance (Claude) was used. All changes vetted by Andy.

@afriedri
remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable ; make w…
9716b62 
…4a4 use the preshuffle triton+asm by default

Signed-off-by: Andy Friedrich <afriedri@amd.com>
claude[bot]
claude Bot reviewed Aug 20, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the rocm Related to AMD ROCm label Aug 20, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 20, 2026
@AndreasKaratzas AndreasKaratzas added the verified Run pre-commit for new contributors without triggering other tests label Aug 20, 2026
@AndreasKaratzas

AndreasKaratzas commented Aug 20, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84870 for commit 9716b6254cdd.

@afriedri
nomenclature
de196f3 
Signed-off-by: Andy Friedrich <afriedri@amd.com>
@afriedri
Merge branch 'main' into remove_fp4_asm_gemm_env
b5bc2ba
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

❌ @afriedri, A reviewer with write access must run /ci run, approve the PR, or add the ready label first.

@AndreasKaratzas

AndreasKaratzas commented Aug 20, 2026

Copy link
Copy Markdown
Member

/ci run

@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84896 for commit b5bc2bad46ec.

Rohan138
Rohan138 approved these changes Aug 28, 2026
View reviewed changes

@Rohan138 Rohan138 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

Rohan138
Rohan138 reviewed Aug 28, 2026
View reviewed changes
Comment thread tests/kernels/quantization/test_rocm_mxfp4.py Outdated
"""The ROCm FP4 AITER gates should depend only on the env toggles and the
gfx950 hardware check.

The asm FP4 GEMM no longer has an env toggle -- it is always on for

@Rohan138 Rohan138 Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this comment actually?

@afriedri afriedri Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@AndreasKaratzas

AndreasKaratzas commented Aug 28, 2026

Copy link
Copy Markdown
Member

/ci run

@AndreasKaratzas
AndreasKaratzas enabled auto-merge (squash) August 28, 2026 17:15
@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85997 for commit d5eb768c47a1.

@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 28, 2026
@AndreasKaratzas
AndreasKaratzas merged commit 2f1cba7 into vllm-project:main Aug 28, 2026
112 of 113 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 28, 2026
am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026
@afriedri @AndreasKaratzas
[ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; …
15422ef 
…make w4a4 use the preshuffle triton+asm by default (vllm-project#53141)

Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
mylibrar pushed a commit to tanyuqian/vllm that referenced this pull request Sep 3, 2026
@afriedri @AndreasKaratzas suqisun
[ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; …
176cb92 
…make w4a4 use the preshuffle triton+asm by default (vllm-project#53141)

Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
D-G-Dimitrov pushed a commit to D-G-Dimitrov/vllm that referenced this pull request Sep 4, 2026
@afriedri @AndreasKaratzas @D-G-Dimitrov
[ROCm] remove VLLM_ROCM_USE_AITER_FP4_ASM_GEMM environment variable; …
d142b70 
…make w4a4 use the preshuffle triton+asm by default (vllm-project#53141)

Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
(cherry picked from commit 2f1cba7)
@afriedri afriedri mentioned this pull request Sep 15, 2026
Merged
vrdn-23 added a commit to vrdn-23/vllm that referenced this pull request Oct 8, 2026
@vrdn-23 @claude
Merge origin/main into pydantic-settings refactor: resolve envs.py co…
e959e8a 
…nflicts

Took the branch's vllm/envs.py wholesale: main's legacy TYPE_CHECKING block
and environment_variables dict are already superseded by the pydantic
*Settings models here. Then ported main's semantic delta from the 57
main-side commits that touched vllm/envs.py since 03a8d0b:

  vllm-project#52431 VLLM_USE_MODELSCOPE: description (bool already accepts "1")
  vllm-project#52216 prefix_cache_retention_interval promoted to an arg (see vllm-project#55353)
  vllm-project#48484 + VLLM_REPLICATE_EMBED
  vllm-project#52633 + VLLM_CPU_CI_ENV
  vllm-project#52182 - VLLM_TEST_FORCE_FP8_MARLIN
  vllm-project#52523 VLLM_API_KEY not a compile factor (branch already marked it)
  vllm-project#52998 VLLM_ALLREDUCE_USE_FLASHINFER default -> True
  vllm-project#53040 + VLLM_DISABLE_DSV4_MEGAMOE_SHARED_EXPERT_FUSION
  vllm-project#53053 + VLLM_KIMI_K3_GEMM_AR
  vllm-project#50501 + VLLM_XPU_INC_WNA16_BACKEND (case-sensitive Literal)
  vllm-project#52018 + VLLM_B12X_MOE_FP4_FORCE_A16
  vllm-project#43018 + VLLM_KV_OFFLOAD_MAX_BATCH_DESCRIPTORS
  vllm-project#51718 VLLM_KV_CACHE_LAYOUT Literal widened to 8 layouts + description
  vllm-project#51896 + VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB
  vllm-project#53219 VLLM_ENABLE_COHERE_API: description
  vllm-project#53559 - VLLM_TRITON_ATTN_USE_TD shim (already absent on the branch)
  vllm-project#53561 VLLM_MAX_AUDIO_CLIP_FILESIZE_MB: description
  vllm-project#53378 VLLM_ELASTIC_EP_SCALE_UP_LAUNCH not a compile factor
  vllm-project#53141 VLLM_ROCM_USE_AITER_FP4_ASM_GEMM removed; restored by vllm-project#57055
  vllm-project#54160 + VLLM_ENABLE_HPC_OPS
  vllm-project#53536 + VLLM_XPU_FORCE_N_CONTIG_WEIGHT
  vllm-project#53576 + VLLM_ALLREDUCE_USE_FLASHINFER_PCIE_IPC
  vllm-project#54632 + VLLM_MAX_EMBED_DECODE_BYTES (not a compile factor)
  vllm-project#54579 + VLLM_ENABLE_SCALE_OUT_ENDPOINTS; removed by vllm-project#55176 (net zero)
  vllm-project#53491 VLLM_GPU_SYNC_CHECK: description
  vllm-project#54371 + VLLM_PLE_CPU_OFFLOAD; reworked vllm-project#56512, dropped vllm-project#57937 (net zero)
  vllm-project#48247 VLLM_ROCM_USE_AITER_CUSTOM_AR: description
  vllm-project#55353 - VLLM_MM_HASHER_ALGORITHM (+ _lower_mm_hasher),
         - VLLM_PREFIX_CACHE_RETENTION_INTERVAL
  vllm-project#54985 - VLLM_NIXL_EP_MAX_NUM_RANKS
  vllm-project#56512 VLLM_PLE_CPU_OFFLOAD rework (net zero, see vllm-project#54371)
  vllm-project#55176 - VLLM_ENABLE_SCALE_OUT_ENDPOINTS (net zero, see vllm-project#54579)
  vllm-project#53940 + VLLM_ROCM_USE_AITER_MOE_SITUV2 (case-insensitive Literal)
  vllm-project#41074 + VLLM_SKIP_VERSION_SUFFIX (not a compile factor)
  vllm-project#48599 - envs.validate_environ (moved to Platform.validate_environ,
         which reads the branch's environment_variables shim)
  vllm-project#52136 pydocstyle docstring restyle of legacy helpers (no-op here)
  vllm-project#57212 docstring rejoin of legacy helpers (no-op here)
  vllm-project#56913 + FORCE_COLOR (alias, non-"0" is True, not a compile factor)
  vllm-project#57055 VLLM_ROCM_USE_AITER_FP4_ASM_GEMM restored (net zero, see vllm-project#53141)
  vllm-project#57142 shm buffer name autogen -> new _generate_shm_name() (<=30 chars)
  vllm-project#56685 VLLM_HUMMING_INPUT_QUANT_CONFIG: description
  vllm-project#57937 - VLLM_PLE_CPU_OFFLOAD (net zero, see vllm-project#54371)
  vllm-project#50814 + VLLM_MXFP4_EMULATION_DEQUANT_AT_LOAD
  vllm-project#57428 + VLLM_ENABLE_GEMM_RS, - VLLM_KIMI_K3_GEMM_RS
  vllm-project#51600 - VLLM_XPU_ENABLE_XPU_GRAPH
  vllm-project#58582 + VLLM_TRITON_JIT_WARMUP_NUM_THREADS (not a compile factor)
  vllm-project#55145 + VLLM_XPU_PP_MICROBATCH
  vllm-project#58201 VLLM_ROCM_USE_AITER_MOE_SITUV2 6-way choice,
         - VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4
  vllm-project#58919 + VLLM_MOONCAKE_CONNECTOR_TIMEOUT
  vllm-project#56861 + VLLM_ROCM_AITER_MLA_DCP_VERIFY (case-insensitive Literal)
  vllm-project#53492 + VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA
  vllm-project#57991 VLLM_DEEPEP_V2_ALLOW_HYBRID_MODE default True (branch already
         True) + description
  vllm-project#58819 + VLLM_ROCM_USE_AITER_MOE_A4W4_DSV4 (tri-state, bool(int(v)))
  vllm-project#50300 + VLLM_CHAT_TEMPLATE_RENDER_TIMEOUT
  vllm-project#54024 VLLM_CPU_INT4_W4A8: description
  vllm-project#59781 - VLLM_ENGINE_ITERATION_TIMEOUT_S
  vllm-project#57995 + VLLM_FLASHINFER_MOE_A2A_LOW_PRECISION_COMBINE
  vllm-project#59965 VLLM_ROCM_AITER_MLA_DCP_VERIFY default route (final "auto")

Also VLLM_SSM_CONV_STATE_LAYOUT's description follows vllm-project#51718's layout rename.

Parity: 311 branch fields vs 311 main runtime entries, no difference in
either direction. Every var's value matches main's lambdas in an empty
environment and under 15 override scenarios, except case normalization of
the two lowercased ROCm choices (both callers already call .lower()).

Intentionally not ported: main's new consumer tests that match
env_with_choices' error text (test_invalid_backend_value_raises,
test_dcp_verify_env_rejects_unknown_route, and the ROCm-gated
test_aiter_moe_situv2_rejects_unknown_activation) fail on this branch:
pydantic's ValidationError names the field, not the env var. Invalid values
are still rejected. Tracked as a branch-wide follow-up.

Other conflicts and merge fallout:
- requirements/docs.txt, requirements/test/{cpu,cuda,rocm,xpu}.txt: took
  main's lockfiles and regenerated them with the pip-compile pre-commit
  hooks; they differ from main only by the branch's pydantic-settings lines.
- tests/test_envs_pydantic.py: dropped the mm-hasher test, pointed the
  validate_environ tests at Platform.validate_environ, updated the shm-name
  prefix assertion.
- check-test-tethering (new on main): wired test_envs_pydantic.py into the
  CPU params/env job; allowlisted tests/docs/test_env_vars_gen.py, which
  cannot run from a CI job's tests/ cwd (tests/docs shadows docs/).

Tests: pytest tests/test_envs.py tests/test_envs_pydantic.py
tests/docs/test_env_vars_gen.py -> 55 passed; pre-commit run --files on all
changed files -> clean.

AI assistance (Claude) was used for this merge resolution.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Vinay Damodaran <vrdn@hey.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Rohan138 Rohan138 Rohan138 approved these changes
@claude claude[bot] claude[bot] left review comments
@AndreasKaratzas AndreasKaratzas AndreasKaratzas approved these changes
@tjtanaa tjtanaa Awaiting requested review from tjtanaa tjtanaa is a code owner

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed rocm Related to AMD ROCm verified Run pre-commit for new contributors without triggering other tests

Projects

AMD
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@afriedri @AndreasKaratzas @Rohan138