Support Confidential Computing (CC) for Qwen3.5-397B-A17B-FP8 on B300 - #28251
Closed
elvischenv wants to merge 4 commits into
Closed
Support Confidential Computing (CC) for Qwen3.5-397B-A17B-FP8 on B300#28251elvischenv wants to merge 4 commits into
elvischenv wants to merge 4 commits into
Conversation
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
github-actions
Bot
added
documentation
Under Confidential Computing the per-step Device->Host result readback in GenerationBatchResult.copy_to_cpu becomes a synchronous cudaMemcpyAsync that blocks the scheduler thread ~one decode step at issue (host dest is Managed/UVM-backed), so the scheduler can't launch the next CUDA graph and overlap is serialized (~40-87% loss at high concurrency). The copy can't be made async under CC; instead run the still-blocking copy on a dedicated daemon thread (TRT-LLM sgl-project#8463 pattern) so it's non-blocking to the scheduler thread, restoring overlap. - utils/common.py: cached is_confidential_compute() via NVML nvmlSystemGetConfComputeState, fail-safe. - managers/overlap_copy_worker.py: AsyncD2HCopyWorker (daemon thread + queue), with explicit teardown. - managers/scheduler.py: gate enable_cc_async_copy = is_confidential_compute(); submit to the worker (CC) or run inline (non-CC) at the 3 issue sites; logs once at INFO when active. - managers/scheduler_output_processor_mixin.py: wait on copy_ready_cpu under CC at the consume sites. Auto-enabled when CC is detected; off-CC behavior is byte-identical. Ref sgl-project#26469. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cc_off uses FlashInfer's trtllm AR+RMSNorm fusion, but under CC it disables itself: create_allreduce_fusion_workspace hardcodes use_symm_dev_mem=True (multicast), and its cuMulticast preflight fails under CC. The kernel itself is multicast-free (one-shot Lamport; 0 multimem in trtllm_allreduce_fusion.cuh) -- only the workspace allocator wanted multicast. Build the fusion on a multicast-free IPC workspace instead (trtllm_create_ipc_workspace_for_all_reduce_fusion(use_symm_dev_mem=False)), skipping the cuMulticast preflight, and drive trtllm_allreduce_fusion with use_oneshot + kARResidualRMSNorm. Restores kernel parity with cc_off. Enabled by default whenever CC is detected (is_confidential_compute()) -- no env var needed; off-CC behavior is unchanged. Logs once at INFO when the multicast-free workspace initializes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Repo-base summary of the async D2H copy worker and the ungated FlashInfer AR+RMSNorm fusion: problem, change, INFO marker, and CC-auto-enable. Pairs with the flashinfer cc-autotuner-fixed branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port of sgl-project#23351 ("Support piecewise CUDA graph with NSA"), translated across this fork's dsa->nsa rename (sgl-project#25014). - nsa_indexer: add k_cache_and_topk_result / logits_head_gate_pcg custom ops; PCG guards in forward_cuda (metadata fetch, delayed K-cache store, gate, seq_lens_cpu access); optional topk_result in _get_topk_ragged; out_cache_loc param in _store_index_k_cache; CP/loop-path asserts; is_compiling rope guard. - nsa_backend: force use_mha=False under PCG; drop _forward_trtllm squeeze(1). - piecewise_context_manager / model_runner / piecewise_cuda_graph_runner: thread nsa_indexers through the forward context. - model_config: un-gate NSA models from the piecewise disable list. - hadamard / layernorm: wrap in register_custom_op with fake impls. - radix_attention: pass cos_sin_cache/is_neox/llama_4_scaling/topk_indices through the split-op schema. - deepseek_v2: skip moe_output_buffer_ctx while compiling. - server_args: disable PCG under NSA prefill-CP; remove flashinfer-MoE PCG-disable block. - test: add test_pcg_glm5_fp4.py (GLM-5-NVFP4, TP=4, B200). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
elvischenv
changed the title
Collaborator
|
@elvischenv The description looks outdated. could you update it? |
Contributor
Author
|
Superseded by #31447. Closing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #31447.
Targets
release/v0.5.12. Enables Confidential Computing (CC) inference on B300 (validated with Qwen3.5-397B-A17B-FP8, TP4).CC perf fixes — auto-enabled when NVIDIA CC is detected (
is_confidential_compute()), byte-identical off-CC:Also included: port of #23351 (piecewise CUDA graph with NSA) for DSA/NSA models.
See
CC_FIXES.mdfor details.CI States
Latest PR Test (Base): ❌ Missing
run-cilabel -- add it to run CI tests.Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.