Repository navigation
[DCP][Kernel][Perf] Fuse the empty-shard LSE mask into the A2A pack kernel #54889
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 2 commits
9efa2f7
f476f47
370f8a9
db07f2a
1de164b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| # SPDX-FileCopyrightText: Copyright contributors to the vLLM project | ||
| """The empty-shard mask fused into the DCP A2A pack kernel must be bit-exact | ||
| with the eager ``mask_dcp_empty_shards_`` pass it replaces.""" | ||
|
|
||
| import pytest | ||
| import torch | ||
|
|
||
| from vllm.v1.attention.ops.dcp import ( | ||
| _dcp_a2a_lse_pack_dim, | ||
| _dcp_a2a_pack_send, | ||
| mask_dcp_empty_shards_, | ||
| ) | ||
|
|
||
| pytestmark = pytest.mark.skipif( | ||
| not torch.cuda.is_available(), reason="needs a GPU for the Triton pack kernel" | ||
| ) | ||
|
|
||
|
|
||
| def _pack(out, lse, world_size, h_per_rank, head_dim, seq_lens, query_start_loc): | ||
| lse_pack_dim = _dcp_a2a_lse_pack_dim(out.dtype) | ||
| send = torch.zeros( | ||
| (world_size, out.shape[0], h_per_rank, head_dim + lse_pack_dim), | ||
| device=out.device, | ||
| dtype=out.dtype, | ||
| ) | ||
| _dcp_a2a_pack_send( | ||
| out, | ||
| lse, | ||
| send, | ||
| world_size, | ||
| h_per_rank, | ||
| head_dim, | ||
| lse_pack_dim, | ||
| seq_lens=seq_lens, | ||
| query_start_loc=query_start_loc, | ||
| ) | ||
| return send | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) | ||
| @pytest.mark.parametrize("world_size,h_per_rank", [(8, 2), (8, 16), (4, 4), (2, 1)]) | ||
| @pytest.mark.parametrize("tokens_per_req", [1, 2, 3]) | ||
| @pytest.mark.parametrize("num_pad_rows", [0, 2]) | ||
| @pytest.mark.parametrize( | ||
| "seq_lens_list", | ||
| [ | ||
| [16, 16, 16, 16], # no empty shards (the long-context decode case) | ||
| [16, 0, 16, 0], # interior empty shards | ||
| [0, 0, 0, 0], # every shard empty | ||
| [16, 16, 0, 0], # trailing empty shards, i.e. cudagraph padding | ||
| [0, 16, 16, 16], # leading empty shard | ||
| [16] * 52, # the profiled conc-52 decode batch | ||
| [7], # single request | ||
| ], | ||
| ) | ||
| def test_fused_mask_matches_eager( | ||
| dtype, world_size, h_per_rank, tokens_per_req, num_pad_rows, seq_lens_list | ||
| ): | ||
| torch.manual_seed(0) | ||
| device = "cuda" | ||
| head_dim = 512 | ||
| num_seqs = len(seq_lens_list) | ||
| # tokens_per_req > 1 is the MTP / multi-token-verify shape. | ||
| num_rows = num_seqs * tokens_per_req + num_pad_rows | ||
| num_heads = world_size * h_per_rank | ||
|
|
||
| out = torch.randn(num_rows, num_heads, head_dim, device=device, dtype=dtype) | ||
| lse = torch.randn(num_rows, num_heads, device=device, dtype=torch.float32) | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we do we need 32 bit numerics here?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The LSE is fp32 in production — the test is mirroring the real dtype, not choosing one. All of the following predate this PR:
It has to be fp32 for range — LSE is a log of summed exponentials, and this mask writes |
||
| seq_lens = torch.tensor(seq_lens_list, device=device, dtype=torch.int32) | ||
| query_start_loc = torch.arange( | ||
| 0, | ||
| (num_seqs + 1) * tokens_per_req, | ||
| tokens_per_req, | ||
| device=device, | ||
| dtype=torch.int32, | ||
| ) # rows past query_start_loc[-1] are cudagraph padding | ||
|
|
||
| # Reference: mask eagerly, then pack with masking disabled. | ||
| ref_lse = lse.clone() | ||
| mask_dcp_empty_shards_(ref_lse, seq_lens, query_start_loc) | ||
| expected = _pack(out, ref_lse, world_size, h_per_rank, head_dim, None, None) | ||
|
|
||
| # Under test: pack with the mask fused in. | ||
| actual = _pack( | ||
| out, lse.clone(), world_size, h_per_rank, head_dim, seq_lens, query_start_loc | ||
| ) | ||
|
|
||
| # Compare bit patterns, not values: for a 2-element LSE pack the high half | ||
| # of -inf bitcasts to a NaN payload, which never compares equal to itself. | ||
| assert torch.equal(actual.view(torch.int16), expected.view(torch.int16)), ( | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. so is this correct? (i.e. using 32 bit numerics in the beginning and then for the comparison only converting them to 16 bit ones)
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think my comment was misleading, and I have rewritten it in Two different tensors are in play. The LSE is fp32 and stays fp32. The send buffer is in the output dtype (16-bit), and Those slots are not meaningful floats at all; they are an opaque container for half of an fp32. A masked row stores bf16 is the top 16 bits of fp32 (same 8-bit exponent), so the truncated half of Bit equality is also strictly stronger than |
||
| "fused mask is not bit-exact with the eager mask" | ||
| ) | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) | ||
| @pytest.mark.parametrize("seed", range(8)) | ||
| def test_fused_mask_matches_eager_ragged(dtype, seed): | ||
| """Non-uniform per-request query lengths. | ||
|
|
||
| The uniform-stride cases above never exercise an unequal gap between | ||
| consecutive ``query_start_loc`` entries, which is what the in-kernel | ||
| boundary search has to get right. MTP with partial acceptance produces | ||
| exactly this: some requests contribute one row, others several, in the | ||
| same batch. | ||
| """ | ||
| torch.manual_seed(seed) | ||
| device = "cuda" | ||
| world_size, h_per_rank, head_dim = 8, 2, 512 | ||
| num_seqs = 17 | ||
|
|
||
| query_lens = torch.randint(1, 5, (num_seqs,), dtype=torch.int32) | ||
| query_start_loc = torch.zeros(num_seqs + 1, dtype=torch.int32) | ||
| query_start_loc[1:] = torch.cumsum(query_lens, 0) | ||
| # Mix real, empty, and (via padding rows) out-of-range shards. | ||
| seq_lens = torch.randint(0, 2, (num_seqs,), dtype=torch.int32) * 16 | ||
| num_rows = int(query_start_loc[-1]) + 3 | ||
|
|
||
| query_start_loc = query_start_loc.to(device) | ||
| seq_lens = seq_lens.to(device) | ||
| num_heads = world_size * h_per_rank | ||
| out = torch.randn(num_rows, num_heads, head_dim, device=device, dtype=dtype) | ||
| lse = torch.randn(num_rows, num_heads, device=device, dtype=torch.float32) | ||
|
|
||
| ref_lse = lse.clone() | ||
| mask_dcp_empty_shards_(ref_lse, seq_lens, query_start_loc) | ||
| expected = _pack(out, ref_lse, world_size, h_per_rank, head_dim, None, None) | ||
| actual = _pack( | ||
| out, lse.clone(), world_size, h_per_rank, head_dim, seq_lens, query_start_loc | ||
| ) | ||
|
|
||
| assert torch.equal(actual.view(torch.int16), expected.view(torch.int16)), ( | ||
| f"ragged query_start_loc mismatch: query_lens={query_lens.tolist()}, " | ||
| f"seq_lens={seq_lens.tolist()}" | ||
| ) | ||
|
|
||
|
|
||
| def test_mask_disabled_is_unmasked(): | ||
| """Passing no seq_lens/query_start_loc must leave the LSE untouched.""" | ||
| torch.manual_seed(0) | ||
| device = "cuda" | ||
| world_size, h_per_rank, head_dim = 8, 2, 512 | ||
| out = torch.randn( | ||
| 4, world_size * h_per_rank, head_dim, device=device, dtype=torch.bfloat16 | ||
| ) | ||
| lse = torch.randn(4, world_size * h_per_rank, device=device, dtype=torch.float32) | ||
|
|
||
| packed = _pack(out, lse, world_size, h_per_rank, head_dim, None, None) | ||
| assert torch.isfinite(packed.float()).all() | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait does this evaluate to true or false for us? I thought we should be using cuda_alike
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It does evaluate correctly to True, but it is the wrong predicate to express the intent, and I have switched it.
torch.cuda.is_available()returnsTrueon ROCm, because the PyTorch ROCm build maps HIP onto thetorch.cudaAPI. So the file was running on our hardware rather than being skipped. Measured in the ROCm image:The trap is the neighbouring predicate:
current_platform.is_cuda()isFalseon ROCm and would have silently skipped the whole file.Switched to [
current_platform.is_cuda_alike()] which is what the rest of the tree uses for "CUDA or ROCm"No behavioural change; 353 tests still pass on MI355X.