[Misc] Remove cache stream and cache events - #3461
Merged
Merged
Conversation
cadedaniel
approved these changes
Mar 18, 2024
cadedaniel
left a comment
cadedaniel
left a comment
Collaborator
There was a problem hiding this comment.
Could we add a test that verifies correctness of swap ?
zhuohan123
approved these changes
Mar 18, 2024
zhuohan123
left a comment
zhuohan123
left a comment
Member
There was a problem hiding this comment.
LGTM! Please consider adding a test as suggested by @cadedaniel
Collaborator
Author
|
@cadedaniel @zhuohan123 Added the test. |
choiceoh
added a commit
to choiceoh/vllm-dsv4
that referenced
this pull request
Jun 20, 2026
…rp-shuffle) + vllm-project#43014 (MoE permute pre-alloc) Two csrc CUDA kernel perf optimizations applied as source commits so the from-source build (docker/Dockerfile, real nvcc compile) actually recompiles them. These are the ones that the prior overlay-on-prebuilt-binary lineage (dsv4-tiera2/tiera3) could NOT pick up, because that lineage was a Python-source overlay on the Aiden b12x prebuilt binaries and never recompiled csrc. Applied (both verified: git apply --check clean on this eb99b8b/flat tree, on the active hot-path, real perf wins): * vllm#44173 (commit 66c1760) — Warp-shuffle + vectorized silu_and_mul_per_block_quant. Pure kernel-internal change to csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu (no binding / signature change → drop-in). Bit-identical output (verified M=16..65536 in PR), ~2.4-2.6x kernel speedup at prefill scale. On the act_quant fusion pass hot-path (act_quant_fusion.py). NOTE: the originally-cited commit c09ad76 is the kernel's ORIGINAL creation (already present in this tree); 66c1760 is the actual warp-shuffle perf commit. * vllm#43014 (net of 4107002..f90eda5, 6 target files) — MoE permute pre-allocated buffer path. Adds moe_permute_with_scratch + moe_permute_sort_workspace_size C++ ops and the MoEPermuteScratch reused- buffer wrapper, wired into cutlass_moe.py + fused_humming_moe.py. 9-14% moe_permute kernel win. Verified the new ops are actually CALLED (not dead code): moe_permute_unpermute.py:214/:71. Intentionally SKIPPED (adversarially analyzed; would be forced builds): * vllm#43162 (fused qnorm/rope/kv head-pad) — the csrc patch targets a templated switch-dispatch refactor (launchFusedDeepseekV4Templated / kNumHeadsQPadded) ABSENT from this older eb99b8b kernel, and its Python half targets the missing nested vllm/models/deepseek_v4/nvidia/ layout. Both halves fail to apply; landing it needs a full manual re-expression against a structurally different kernel + a non-existent file. (The kernel IS used here via deepseek_v4_attention.py:617, so the intent is relevant — but the patch is anchored to a tree generation we don't have.) * vllm#43554 ("router GEMM PDL") — MISLABELED. The PR is actually [Kernel] Remove NormGateLinear (a -836-line DELETION of the dsv4_norm_router_gemm kernel). Its only PDL content is incidental removal of an env-gate around pre-existing PDL in dsv3_router_gemm; it adds no GB10 PDL optimization. This tree ACTIVELY USES NormGateLinear (deepseek_v4.py self.norm_gate, norm_gate_linear.py:30) — applying it would regress, not optimize. flashinfer vllm-project#3461/vllm-project#3624 remain the JIT overlay (tiera3); this commit adds the vLLM csrc deltas that only a from-source build can realize. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR removes
cache_streamandcache_events. Previously, these objects were introduced for fine-grained overlapping between swapping and model execution. However, the overlapping has been disabled since vLLM introduced CUDA graphs. Therefore, we don't need a separate CUDA stream and CUDA events for swapping.This PR removes such unused objects by launching the swapping ops in the default CUDA stream (same stream used in model execution).