Conversation
Collaborator
Author
|
/rerun-tests test/registered/unit/layers/attention/test_gdn_flashinfer_alignment.py |
Contributor
|
Results for 🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[by Codex]
Summary
aandbtest views with deterministic finite valuestorch.emptycontents, which can contain NaNs and makeassert_closefail nondeterministicallyWhat failure does this fix?
This fixes the
base-a-test-cpufailure observed while testing #33922:test/registered/unit/layers/attention/test_gdn_flashinfer_alignment.pyTestFlashInferGDNAlignment::test_decode_repairs_read_only_arguments_before_flashinfertorch.testing.assert_close(captured["a"], a)reported 2/8 mismatched elements with a greatest absolute difference ofnanThe same test failed standalone on latest
maininlmsysorg/sglang:dev-cu13on a computelab node. In that run,captured["b"]reported 8/8 mismatched elements with a greatest absolute difference ofnan. Which tensor/elements fail depends on the contents returned by the allocator, but both failures have the same root cause.Additional confirmed CI occurrences
I scanned 600 failed
PR Test Baseworkflow runs after #35758 introduced this test and inspected the logs of their failedbase-a-test-cpuroot jobs. The exact failure was confirmed in three pipeline runs across two unrelated PRs:All three failed in
TestFlashInferGDNAlignment::test_decode_repairs_read_only_arguments_before_flashinferwith the same signature: 2/8 mismatched elements and anandifference at index(0, 0, 2). The recurrence across unrelated PRs and separate workflow runs supports that this is an intermittent main-branch test defect rather than a regression caused by any one target PR.Root cause
_view_with_pointer_mod()intentionally builds misaligned views fromtorch.emptybacking storage. In this test,aandbwere never initialized beforekernel.decode():The alignment-repair path correctly copies these views into aligned scratch buffers. The test then compares the copied tensors against the original uninitialized tensors with
torch.testing.assert_close. If the allocator-provided bit patterns contain NaNs, the comparison fails becauseassert_closedoes not consider corresponding NaNs equal by default.Therefore:
mainwithout that PR;qwen 3.8 rebase), merge commit5f216fc33f315ade378bc7cd295ee88f5322d4d3, which added this test file and the uninitialized setup.Fix
Initialize
aandbwith distinct finite sequences before callingdecode(). This removes allocator-dependent NaNs while preserving the test's intended coverage:assert_closechecks still verify lossless copy behavior.Validation
After rebasing onto
60f881b40c:test_gdn_flashinfer_alignment.py: 10 passedBLACK_NUM_WORKERS=1 SKIP=no-commit-to-branch pre-commit run --all-files --show-diff-on-failure: passedCI States
Latest PR Test (Base): ❌ Run #33467025020
Latest PR Test (Extra): ❌ Run #33467024850
Latest PR Test (AMD ROCm 7.2): ❌ Run #33467024983