[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert torch.allclose in test_rocm_skinny_gemms.py by rasmith · Pull Request #34181 · vllm-project/vllm

rasmith · 2026-02-09T23:03:37Z

This PR uses torch.testing.assert_close instead of assert torch.allclose in test_rocm_skinny_gemms.py. It is much easier to see how far off a failure is if the former is used.

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

gemini-code-assist

Code Review

This pull request replaces assert torch.allclose with torch.testing.assert_close in tests/kernels/quantization/test_rocm_skinny_gemms.py. This is a good change as torch.testing.assert_close provides more detailed and helpful error messages when tests fail, which aids in debugging. The changes correctly maintain the original absolute and relative tolerances for all assertions. Overall, this is a solid improvement to the test suite's robustness and developer experience.

tjtanaa · 2026-02-10T02:55:01Z

@rasmith can you share some test results?

rasmith · 2026-02-10T02:59:57Z

@rasmith can you share some test results?

Sure, there are failures though that were caused by this PR, however they are being fixed by this PR:

Here are the test results (but this PR doesn't change the results, this PR just makes it easier to inspect the failures that do occur):

============================================================= short test summary info =============================================================
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype0-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-False-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype0-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-False-True-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype0-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-False-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype0-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[False-True-True-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype0-4-65552-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-False-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype0-4-65552-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-False-True-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype0-4-65552-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-False-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype0-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype0-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype0-4-65552-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype1-4-65536-28672-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype1-4-65536-28688-False] - AssertionError: Tensor-likes are not close!
FAILED kernels/quantization/test_rocm_skinny_gemms.py::test_rocm_wvsplitk_fp8_kernel[True-True-True-0-dtype1-4-65552-28672-False] - AssertionError: Tensor-likes are not close!
====================================== 48 failed, 978 passed, 4608 skipped, 3 warnings in 351.13s (0:05:51) =======================================

mergify · 2026-02-11T04:05:41Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @rasmith.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

…b.com:rasmith/vllm into rasmith_add_assert_close_to_skinny_gemms_test

rasmith · 2026-02-11T05:11:49Z

@rasmith can you share some test results?

@tjtanaa A recent PR got merged that fixed all the failures. I would still like to convert all the assert torch.allclose instances to torch.testing.assert_close.

The tests are now:

1026 passed, 4608 skipped, 3 warnings in 280.09s

rasmith · 2026-02-13T04:39:51Z

@tjtanaa Please take another look?

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: Andrii Skliar <askliar@nvidia.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com> Signed-off-by: EricccYang <yangyang4991@gmail.com>

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

rasmith added 2 commits February 9, 2026 19:44

use assert_close intead of allclose

3ac5efe

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

add atol where needed

669ced7

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

rasmith requested a review from tjtanaa as a code owner February 9, 2026 23:03

mergify bot added rocm Related to AMD ROCm bug Something isn't working labels Feb 9, 2026

github-project-automation bot added this to AMD Feb 9, 2026

github-project-automation bot moved this to Todo in AMD Feb 9, 2026

gemini-code-assist bot reviewed Feb 9, 2026

View reviewed changes

Merge branch 'main' into rasmith_add_assert_close_to_skinny_gemms_test

709cf8a

mergify bot added the needs-rebase label Feb 11, 2026

rasmith added 2 commits February 11, 2026 05:03

merge main

1c914b7

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

Merge branch 'rasmith_add_assert_close_to_skinny_gemms_test' of githu…

ec72b98

…b.com:rasmith/vllm into rasmith_add_assert_close_to_skinny_gemms_test

mergify bot removed the needs-rebase label Feb 11, 2026

rasmith added 2 commits February 11, 2026 11:32

Merge branch 'main' into rasmith_add_assert_close_to_skinny_gemms_test

b203151

Merge branch 'main' into rasmith_add_assert_close_to_skinny_gemms_test

ee2d379

rasmith added 2 commits February 18, 2026 21:44

fix tolerance

c5b6ded

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

0.01 -> 1e-2

704e2ff

Signed-off-by: Randall Smith <Randall.Smith@amd.com>

gshtras approved these changes Feb 18, 2026

View reviewed changes

gshtras enabled auto-merge (squash) February 18, 2026 21:54

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 18, 2026

Merge branch 'main' into rasmith_add_assert_close_to_skinny_gemms_test

241ae2a

gshtras merged commit 2b84ac6 into vllm-project:main Feb 18, 2026
14 of 15 checks passed

github-project-automation bot moved this from Todo to Done in AMD Feb 18, 2026

jmamou pushed a commit to jmamou/vllm that referenced this pull request Feb 23, 2026

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert to…

68fcb58

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

llsj14 pushed a commit to llsj14/vllm that referenced this pull request Mar 1, 2026

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert to…

c23aadf

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

tunglinwood pushed a commit to tunglinwood/vllm that referenced this pull request Mar 4, 2026

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert to…

eec641e

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

Copilot AI pushed a commit to machov/vllm that referenced this pull request Mar 10, 2026

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert to…

4e3f066

…rch.allclose in test_rocm_skinny_gemms.py (vllm-project#34181) Signed-off-by: Randall Smith <Randall.Smith@amd.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert torch.allclose in test_rocm_skinny_gemms.py#34181

[CI][AMD][BugFix] Use torch.testing.assert_close instead of assert torch.allclose in test_rocm_skinny_gemms.py#34181
gshtras merged 10 commits intovllm-project:mainfrom
rasmith:rasmith_add_assert_close_to_skinny_gemms_test

rasmith commented Feb 9, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

tjtanaa commented Feb 10, 2026

Uh oh!

rasmith commented Feb 10, 2026 •

edited

Loading

Uh oh!

mergify bot commented Feb 11, 2026

Uh oh!

rasmith commented Feb 11, 2026

Uh oh!

rasmith commented Feb 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rasmith commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

tjtanaa commented Feb 10, 2026

Uh oh!

rasmith commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Feb 11, 2026

Uh oh!

rasmith commented Feb 11, 2026

Uh oh!

rasmith commented Feb 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rasmith commented Feb 9, 2026 •

edited

Loading

rasmith commented Feb 10, 2026 •

edited

Loading