Fix deepseek awq v3

[hnyls2002]

python -m sglang.launch_server --model-path cognitivecomputations/DeepSeek-V3-AWQ --tp-size 8 --trust-remote --disable-mla

[halexan]

After this pr being merged.

Can sglang run this cognitivecomputations/DeepSeek-V3-AWQ?

[chenchunhui97]

After this pr being merged.

Can sglang run this cognitivecomputations/DeepSeek-V3-AWQ?

I am having a try......

[Xu-Chen]

We should also introduce triton fused moe kernel like moe_wna16.
AWQ marlin kernel may be just get 10 token/s on 8*A100.

[hnyls2002]

After this pr being merged.

Can sglang run this cognitivecomputations/DeepSeek-V3-AWQ?

Yes, this PR is exactly for this

[pachinko]

``> > After this pr being merged.

Can sglang run this cognitivecomputations/DeepSeek-V3-AWQ?

Yes, this PR is exactly for this

still have a problem, i am running this model cognitivecomputations/DeepSeek-V3-AWQ

[2025-02-11 14:42:20 TP6] Scheduler hit an exception: Traceback (most recent call last):
  File "/WORK/sglang/python/sglang/srt/managers/scheduler.py", line 1816, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/managers/scheduler.py", line 240, in __init__
    self.tp_worker = TpWorkerClass(
                     ^^^^^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 63, in __init__
    self.worker = TpModelWorker(server_args, gpu_id, tp_rank, dp_rank, nccl_port)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/managers/tp_worker.py", line 68, in __init__
    self.model_runner = ModelRunner(
                        ^^^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/model_executor/model_runner.py", line 186, in __init__
    self.load_model()
  File "/WORK/sglang/python/sglang/srt/model_executor/model_runner.py", line 307, in load_model
    self.model = get_model(
                 ^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/model_loader/__init__.py", line 22, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/WORK/sglang/python/sglang/srt/model_loader/loader.py", line 362, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/WORK/sglang/python/sglang/srt/models/deepseek_v2.py", line 924, in load_weights
    param = params_dict[name]
            ~~~~~~~~~~~^^^^^^
KeyError: 'model.layers.6.mlp.experts.w2_weight'

[2025-02-11 14:42:20] Received sigquit from a child proces. It usually means the child failed.

[halexan]

@pachinko

What is your launch command?

[pachinko]

@halexan

python3 -m sglang.launch_server \
    --model-path /home/model/DeepSeek-R1 \
    --tp 8 \
    --dist-init-addr 10.10.0.1:6000 \
    --nnodes 1 \
    --node-rank 0 \
    --trust-remote-code \
    --disable-radix-cache  \
    --disable-outlines-disk-cache \
    --host 0.0.0.0 \
    --port 40000

[halexan]

We should also introduce triton fused moe kernel like moe_wna16. AWQ marlin kernel may be just get 10 token/s on 8*A100.

So, does this pr still use AWQ marlin kernel?

[pachinko]

@halexan

python3 -m sglang.launch_server \
    --model-path /home/model/DeepSeek-R1 \
    --tp 8 \
    --dist-init-addr 10.10.0.1:6000 \
    --nnodes 1 \
    --node-rank 0 \
    --trust-remote-code \
    --disable-radix-cache  \
    --disable-outlines-disk-cache \
    --host 0.0.0.0 \
    --port 40000

I replaced the config.json with the awq version.

[hnyls2002]

@halexan

python3 -m sglang.launch_server \
    --model-path /home/model/DeepSeek-R1 \
    --tp 8 \
    --dist-init-addr 10.10.0.1:6000 \
    --nnodes 1 \
    --node-rank 0 \
    --trust-remote-code \
    --disable-radix-cache  \
    --disable-outlines-disk-cache \
    --host 0.0.0.0 \
    --port 40000

I replaced the config.json with the awq version.

R1 and MLA are not supported by now, due to some unknown accuracy reasons. You can use V3-AWQ with this command

 python -m sglang.launch_server --model-path cognitivecomputations/DeepSeek-V3-AWQ --tp-size 8 --trust-remote --disable-mla

[chenchunhui97]

After this pr being merged.

Can sglang run this cognitivecomputations/DeepSeek-V3-AWQ?

I succeeded to deploy the model on 8*A800 by building docker image on branch fix-dpsk-v3-awq.

[Xu-Chen]

Could you share some benchmark？

[Zachary-ai-engineer]

We tested V3 AWQ based on the latest code and found that indicators such as tpot were relatively poor. How should we solve this problem?

[halexan]

How about benchmark？@chenchunhui97

[zhyncs]

This fix is a bit tricky, I'll merge it first to unblock the awq usage. Refactoring is on its way.

[luweizheng]

My launch script on 8*A800 80G. This model havs been successfully deployed with vLLM with a smaller context length. But it seems vLLM does not optimize well on MLA now.

python3 -m sglang.launch_server --model-path /path/to/DeepSeek-R1-awq/DeepSeek-R1-awq --tp 8 --host 0.0.0.0 --port 11434 --trust-remote-code

Error:

File "/fs/fast/u20247643/envs/sglang/lib/python3.12/site-packages/sglang/srt/model_loader/__init__.py", line 22, in get_model
    return loader.load_model(
           ^^^^^^^^^^^^^^^^^^
  File "/fs/fast/u20247643/envs/sglang/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 362, in load_model
    model.load_weights(self._get_all_weights(model_config, model))
  File "/fs/fast/u20247643/envs/sglang/lib/python3.12/site-packages/sglang/srt/models/deepseek_v2.py", line 962, in load_weights
    w = ops.awq_dequantize(
        ^^^^^^^^^^^^^^^^^^^
  File "/fs/fast/u20247643/envs/sglang/lib/python3.12/site-packages/vllm/_custom_ops.py", line 222, in awq_dequantize
    return torch.ops._C.awq_dequantize(qweight, scales, zeros, split_k_iters,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fs/fast/u20247643/envs/sglang/lib/python3.12/site-packages/torch/_ops.py", line 1116, in __call__
    return self._op(*args, **(kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected scalar type Half but found BFloat16

@chenchunhui97 @zhyncs Any suggestions?