[MUSA][4/N] Add common device utilities, distributed backend, and custom op wiring by yeahdongcn · Pull Request #17246 · sgl-project/sglang

yeahdongcn · 2026-01-17T03:20:52Z

Motivation

This PR is the fourth in a series of pull requests (tracked in #16565) to add full support for Moore Threads GPUs, leveraging MUSA (Meta-computing Unified System Architecture) to accelerate LLM inference.

Modifications

python/sglang/multimodal_gen/runtime/layers/custom_op.py: Add forward_musa() method and MUSA platform dispatch in CustomOp
python/sglang/srt/configs/device_config.py: Add "musa" to supported device types
python/sglang/srt/distributed/parallel_state.py: Add MUSA device support in GroupCoordinator and cleanup_dist_env_and_memory()
python/sglang/srt/layers/rotary_embedding.py: Add MUSA platform check to rotary embedding initialization
python/sglang/srt/managers/scheduler_metrics_mixin.py: Add "musa graph" label in scheduler metrics logging
python/sglang/srt/model_executor/model_runner.py: Add MUSA support for cublas/attention backend init, mccl distributed backend, and CUDA graph recapture
python/sglang/srt/utils/common.py:
- Add get_mtgpu_memory_capacity() to query Moore Threads GPU memory via mthreads-gmi with torch.musa fallback
- Add MUSA device handling in get_available_gpu_memory(), broadcast_pyobj(), get_device_memory_capacity(), get_device_name(), get_device(), get_device_count(), get_device_core_count(), and get_device_capability()
- Refactor integrated GPU detection to use props.is_integrated instead of hardcoded SM values (same as [diffusion] fix: fix UMA detection #17113)

Testing Done

Tested in a clean torch_musa container.

Qwen3-0.6B without MUSA Graph enabled works fine on a single MTGPU:

root@worker3218:/ws# python -m sglang.launch_server \
     --model-path /home/dist/Qwen3-0.6B/ \
     --host 0.0.0.0 --trust-remote-code \
     --disable-radix-cache --disable-cuda-graph \
     --tp-size 1 --port 43436
[2026-01-17 11:18:24] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 11:18:24 | warnings | 140111369892992 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 11:18:24] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 11:18:24 | warnings | 140111369892992 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 11:18:24] INFO server_args.py:1657: Attention backend not specified. Use triton backend by default.
2026-01-17 11:18:24 | server_args | 140111369892992 | INFO : Attention backend not specified. Use triton backend by default.
[2026-01-17 11:18:25] server_args=ServerArgs(model_path='/home/dist/Qwen3-0.6B/', tokenizer_path='/home/dist/Qwen3-0.6B/', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=43436, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.836, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=8192, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='musa', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=31621149, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='/home/dist/Qwen3-0.6B/', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='pytorch', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=True, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=8192, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='in-seq-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-01-17 11:18:25] Using default HuggingFace chat template with detected content format: string
[2026-01-17 11:18:32] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 11:18:32 | warnings | 139857866790016 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 11:18:32] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 11:18:32 | warnings | 139857866790016 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 11:18:32] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 11:18:32 | warnings | 139962004276352 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 11:18:33] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 11:18:33 | warnings | 139962004276352 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 11:18:33] Init torch distributed begin.
[2026-01-17 11:18:33] Init torch distributed ends. mem usage=0.00 GB
[2026-01-17 11:18:33] MOE_RUNNER_BACKEND is not initialized, the backend will be automatically selected
[2026-01-17 11:18:36] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)
[2026-01-17 11:18:36] Load weight begin. avail mem=79.26 GB
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.65it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.65it/s]

[2026-01-17 11:18:44] Load weight end. type=Qwen3ForCausalLM, dtype=torch.bfloat16, avail mem=77.69 GB, mem usage=1.57 GB.
[2026-01-17 11:18:44] Using KV cache dtype: torch.bfloat16
[2026-01-17 11:18:44] KV Cache is allocated. #tokens: 605665, K size: 32.35 GB, V size: 32.35 GB
[2026-01-17 11:18:44] Memory pool end. avail mem=12.17 GB
[2026-01-17 11:18:45] max_total_num_tokens=605665, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=12.11 GB
[2026-01-17 11:18:46] INFO:     Started server process [30568]
[2026-01-17 11:18:46] INFO:     Waiting for application startup.
[2026-01-17 11:18:46] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-17 11:18:46] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-17 11:18:46] INFO:     Application startup complete.
[2026-01-17 11:18:46] INFO:     Uvicorn running on http://0.0.0.0:43436 (Press CTRL+C to quit)
[2026-01-17 11:18:47] INFO:     127.0.0.1:38066 - "GET /model_info HTTP/1.1" 200 OK
[2026-01-17 11:18:47] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-17 11:19:02] INFO:     127.0.0.1:38072 - "POST /generate HTTP/1.1" 200 OK
[2026-01-17 11:19:02] The server is fired up and ready to roll!
[2026-01-17 11:19:11] INFO:     192.168.36.243:58224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-17 11:19:11] Prefill batch, #new-seq: 1, #new-token: 559, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-17 11:19:13] Decode batch, #running-req: 1, #token: 592, token usage: 0.00, musa graph: False, gen throughput (token/s): 1.01, #queue-req: 0, 
[2026-01-17 11:19:14] Decode batch, #running-req: 1, #token: 632, token usage: 0.00, musa graph: False, gen throughput (token/s): 23.89, #queue-req: 0, 
[2026-01-17 11:19:16] Decode batch, #running-req: 1, #token: 672, token usage: 0.00, musa graph: False, gen throughput (token/s): 22.96, #queue-req: 0, 
[2026-01-17 11:19:18] Decode batch, #running-req: 1, #token: 712, token usage: 0.00, musa graph: False, gen throughput (token/s): 23.81, #queue-req: 0,

Qwen3-0.6B with MUSA Graph enabled works fine on a single MTGPU:

root@worker3218:/ws# python -m sglang.launch_server      --model-path /home/dist/Qwen3-0.6B/      --host 0.0.0.0 --trust-remote-code      --disable-radix-cache      --tp-size 1 --port 43436
[2026-01-17 10:49:28] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 10:49:28 | warnings | 139637166904448 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 10:49:28] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 10:49:28 | warnings | 139637166904448 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 10:49:28] INFO server_args.py:1657: Attention backend not specified. Use triton backend by default.
2026-01-17 10:49:28 | server_args | 139637166904448 | INFO : Attention backend not specified. Use triton backend by default.
[2026-01-17 10:49:29] server_args=ServerArgs(model_path='/home/dist/Qwen3-0.6B/', tokenizer_path='/home/dist/Qwen3-0.6B/', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=43436, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.836, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=8192, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='musa', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=871398282, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='/home/dist/Qwen3-0.6B/', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='pytorch', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path=None, kt_method='AMXINT4', kt_cpuinfer=None, kt_threadpool_count=2, kt_num_gpu_experts=None, kt_max_deferred_experts_per_token=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=True, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=8192, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='in-seq-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-01-17 10:49:29] Using default HuggingFace chat template with detected content format: string
[2026-01-17 10:49:37] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 10:49:37 | warnings | 139686831490176 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 10:49:37] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

2026-01-17 10:49:37 | warnings | 140592998728832 | WARNING : /ws/python/sglang/srt/layers/quantization/awq.py:76: UserWarning: Only CUDA, HIP and XPU support AWQ currently.
  warnings.warn(f"Only CUDA, HIP and XPU support AWQ currently.")

[2026-01-17 10:49:37] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 10:49:37 | warnings | 139686831490176 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 10:49:37] WARNING warnings.py:109: /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

2026-01-17 10:49:37 | warnings | 140592998728832 | WARNING : /ws/python/sglang/srt/layers/quantization/gguf.py:47: UserWarning: Only CUDA support GGUF quantization currently.
  warnings.warn(f"Only CUDA support GGUF quantization currently.")

[2026-01-17 10:49:38] Init torch distributed begin.
[2026-01-17 10:49:38] Init torch distributed ends. mem usage=0.00 GB
[2026-01-17 10:49:38] MOE_RUNNER_BACKEND is not initialized, the backend will be automatically selected
[2026-01-17 10:49:40] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)
[2026-01-17 10:49:40] Load weight begin. avail mem=79.26 GB
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.65it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.65it/s]

[2026-01-17 10:49:48] Load weight end. type=Qwen3ForCausalLM, dtype=torch.bfloat16, avail mem=77.69 GB, mem usage=1.57 GB.
[2026-01-17 10:49:48] Using KV cache dtype: torch.bfloat16
[2026-01-17 10:49:48] KV Cache is allocated. #tokens: 605665, K size: 32.35 GB, V size: 32.35 GB
[2026-01-17 10:49:48] Memory pool end. avail mem=12.17 GB
[2026-01-17 10:49:49] Capture cuda graph begin. This can take up to several minutes. avail mem=12.11 GB
[2026-01-17 10:49:49] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256]
Capturing batches (bs=80 avail_mem=9.78 GB):  64%|██████████████████████████████████████▎                     | 23/36 [03:06<01Capturing batches (bs=72 avail_mem=9.76 GB):  64%|██████████████████████████████████████▎                     | 23/36 [03:06<01Capturing batches (bs=72 avail_mem=9.76 GB):  67%|████████████████████████████████████████                    | 24/36 [03:13<01Capturing batches (bs=64 avail_mem=9.75 GB):  67%|████████████████████████████████████████                    | 24/36 [03:13<01Capturing batches (bs=64 avail_mem=9.75 GB):  69%|█████████████████████████████████████████▋                  | 25/36 [03:21<01Capturing batches (bs=56 avail_mem=9.73 GB):  69%|█████████████████████████████████████████▋                  | 25/36 [03:21<01Capturing batches (bs=56 avail_mem=9.73 GB):  72%|███████████████████████████████████████████▎                | 26/36 [03:28<01Capturing batches (bs=48 avail_mem=9.71 GB):  72%|███████████████████████████████████████████▎                | 26/36 [03:28<01Capturing batches (bs=1 avail_mem=9.57 GB): 100%|█████████████████████████████████████████████████████████████| 36/36 [04:40<00:00,  7.78s/it]
[2026-01-17 10:54:30] Capture cuda graph end. Time elapsed: 281.08 s. mem usage=2.56 GB. avail mem=9.55 GB.
[2026-01-17 10:54:30] max_total_num_tokens=605665, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=9.55 GB
[2026-01-17 10:54:31] INFO:     Started server process [42939]
[2026-01-17 10:54:31] INFO:     Waiting for application startup.
[2026-01-17 10:54:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-17 10:54:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}
[2026-01-17 10:54:31] INFO:     Application startup complete.
[2026-01-17 10:54:31] INFO:     Uvicorn running on http://0.0.0.0:43436 (Press CTRL+C to quit)
[2026-01-17 10:54:32] INFO:     127.0.0.1:53740 - "GET /model_info HTTP/1.1" 200 OK
[2026-01-17 10:54:32] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-17 10:54:33] INFO:     127.0.0.1:53748 - "POST /generate HTTP/1.1" 200 OK
[2026-01-17 10:54:33] The server is fired up and ready to roll!
[2026-01-17 10:54:51] INFO:     192.168.36.243:52951 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2026-01-17 10:54:51] Prefill batch, #new-seq: 1, #new-token: 474, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2026-01-17 10:54:52] Decode batch, #running-req: 1, #token: 507, token usage: 0.00, musa graph: True, gen throughput (token/s): 0.13, #queue-req: 0, 
[2026-01-17 10:54:53] Decode batch, #running-req: 1, #token: 547, token usage: 0.00, musa graph: True, gen throughput (token/s): 55.17, #queue-req: 0, 
[2026-01-17 10:54:53] Decode batch, #running-req: 1, #token: 587, token usage: 0.00, musa graph: True, gen throughput (token/s): 54.33, #queue-req: 0, 
[2026-01-17 10:54:54] Decode batch, #running-req: 1, #token: 627, token usage: 0.00, musa graph: True, gen throughput (token/s): 47.71, #queue-req: 0, 
[2026-01-17 10:54:55] Decode batch, #running-req: 1, #token: 667, token usage: 0.00, musa graph: True, gen throughput (token/s): 42.31, #queue-req: 0, 
[2026-01-17 10:54:56] Decode batch, #running-req: 1, #token: 707, token usage: 0.00, musa graph: True, gen throughput (token/s): 41.09, #queue-req: 0, 
[2026-01-17 10:54:57] Decode batch, #running-req: 1, #token: 747, token usage: 0.00, musa graph: True, gen throughput (token/s): 40.94, #queue-req: 0,

Accuracy Tests

Benchmarking and Profiling

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-01-17T03:20:55Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

yeahdongcn · 2026-01-17T03:22:48Z

Will rebase once #16959 gets merged, then mark this once ready for review.

…tom op wiring Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

yeahdongcn · 2026-01-24T00:30:53Z

Rebased onto upstream/main.

Kangyan-Zhou · 2026-01-28T19:54:17Z

/tag-and-rerun-ci

yeahdongcn · 2026-01-29T02:52:33Z

/rerun-failed-ci

yeahdongcn · 2026-01-29T07:24:46Z

@Kangyan-Zhou Thanks!

…tom op wiring (sgl-project#17246) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file diffusion SGLang Diffusion labels Jan 17, 2026

yeahdongcn changed the title ~~Xd/musa common~~ [MUSA][4/N] Add common device utilities, distributed backend, and custom op wiring Jan 17, 2026

yeahdongcn mentioned this pull request Jan 17, 2026

[Roadmap][Feature] Support Moore Threads (MUSA) GPU #16565

Open

2 tasks done

yeahdongcn added the mthreads label Jan 19, 2026

yeahdongcn force-pushed the xd/musa_common branch 2 times, most recently from 3e3f879 to 0aaca80 Compare January 23, 2026 08:40

[MUSA][4/N] Add common device utilities, distributed backend, and cus…

d10c877

…tom op wiring Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

yeahdongcn force-pushed the xd/musa_common branch from 0aaca80 to d10c877 Compare January 24, 2026 00:29

yeahdongcn marked this pull request as ready for review January 24, 2026 00:30

yeahdongcn requested review from BBuf, Edwardf0t1, Fridge003, HaiShaw, Ying1123, ch-wan, hnyls2002, ispobock, merrymercy, mickqian, yhyang201 and yizhang2077 as code owners January 24, 2026 00:30

github-actions Bot added the run-ci CI: run the baseline test suite on this PR label Jan 28, 2026

Merge branch 'main' into xd/musa_common

5493dc0

Kangyan-Zhou merged commit d3cdee0 into sgl-project:main Jan 29, 2026
152 of 164 checks passed

sfiisf pushed a commit to sfiisf/sglang that referenced this pull request Feb 5, 2026

[MUSA][4/N] Add common device utilities, distributed backend, and cus…

77b21c8

…tom op wiring (sgl-project#17246) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

0826joyce pushed a commit to 0826joyce/sglang-perf-opt that referenced this pull request May 19, 2026

[MUSA][4/N] Add common device utilities, distributed backend, and cus…

d9afb82

…tom op wiring (sgl-project#17246) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[MUSA][4/N] Add common device utilities, distributed backend, and cus…

e6ad4b0

…tom op wiring (sgl-project#17246) Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>

Navigation Menu

Repository navigation

[MUSA][4/N] Add common device utilities, distributed backend, and custom op wiring - #17246

[MUSA][4/N] Add common device utilities, distributed backend, and custom op wiring#17246
Kangyan-Zhou merged 2 commits into
sgl-project:mainfrom
yeahdongcn:xd/musa_common

yeahdongcn commented Jan 17, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jan 17, 2026

Uh oh!

yeahdongcn commented Jan 17, 2026

Uh oh!

yeahdongcn commented Jan 24, 2026

Uh oh!

Kangyan-Zhou commented Jan 28, 2026

Uh oh!

yeahdongcn commented Jan 29, 2026

Uh oh!

Uh oh!

yeahdongcn commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

yeahdongcn commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Testing Done

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist Bot commented Jan 17, 2026

Uh oh!

yeahdongcn commented Jan 17, 2026

Uh oh!

yeahdongcn commented Jan 24, 2026

Uh oh!

Kangyan-Zhou commented Jan 28, 2026

Uh oh!

yeahdongcn commented Jan 29, 2026

Uh oh!

Uh oh!

yeahdongcn commented Jan 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

yeahdongcn commented Jan 17, 2026 •

edited

Loading