[Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 #3108

zzc0721 · 2025-02-02T10:52:18Z

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

我在kaggle notebook中测试使用T4x2 GPU搭配turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会遇到代码块无限期卡死的问题
使用的lmdeploy是github中的0.7.0.post2+cu118发行版本

!lmdeploy serve api_server Qwen/Qwen2.5-32B-Instruct-AWQ --tp 2

如果手动指定pytorch backend则不会出现这个问题

!lmdeploy serve api_server Qwen/Qwen2.5-32B-Instruct-AWQ --tp 2 --backend pytorch

使用turbomind backend部署gradio也会出现无限期卡死的情况

!lmdeploy serve gradio Qwen/Qwen2.5-32B-Instruct-AWQ --tp 2 --backend turbomind --share

Fetching 15 files: 100%|████████████████████| 15/15 [00:00<00:00, 108100.62it/s]
2025-02-02 10:48:51,310 - lmdeploy - INFO - async_engine.py:259 - input backend=turbomind, backend_config=TurbomindEngineConfig(dtype='auto', model_format=None, tp=2, session_len=None, max_batch_size=128, cache_max_entry_count=0.8, cache_chunk_size=-1, cache_block_seq_len=64, enable_prefix_caching=False, quant_policy=0, rope_scaling_factor=0.0, use_logn_attn=False, download_dir=None, revision=None, max_prefill_token_num=8192, num_tokens_per_iter=0, max_prefill_iters=1)
2025-02-02 10:48:51,310 - lmdeploy - INFO - async_engine.py:260 - input chat_template_config=None
2025-02-02 10:48:51,315 - lmdeploy - INFO - async_engine.py:269 - updated chat_template_onfig=ChatTemplateConfig(model_name='qwen2d5', system=None, meta_instruction=None, eosys=None, user=None, eoh=None, assistant=None, eoa=None, tool=None, eotool=None, separator=None, capability=None, stop_words=None)
2025-02-02 10:48:53,490 - lmdeploy - INFO - turbomind.py:282 - model_source: hf_model
2025-02-02 10:48:53,630 - lmdeploy - INFO - turbomind.py:190 - turbomind model config:

{
  "model_config": {
    "model_name": "",
    "chat_template": "",
    "model_arch": "Qwen2ForCausalLM",
    "head_num": 40,
    "kv_head_num": 8,
    "hidden_units": 5120,
    "vocab_size": 152064,
    "embedding_size": 152064,
    "num_layer": 64,
    "inter_size": [
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648,
      27648
    ],
    "norm_eps": 1e-06,
    "attn_bias": 1,
    "start_id": 151643,
    "end_id": 151645,
    "size_per_head": 128,
    "group_size": 128,
    "weight_type": "int4",
    "session_len": 32768,
    "tp": 2,
    "model_format": "awq",
    "expert_num": [],
    "expert_inter_size": 0,
    "experts_per_token": 0,
    "moe_shared_gate": false,
    "norm_topk_prob": false,
    "routed_scale": 1.0,
    "topk_group": 1,
    "topk_method": "greedy",
    "moe_group_num": 1,
    "q_lora_rank": 0,
    "kv_lora_rank": 0,
    "qk_rope_dim": 0,
    "v_head_dim": 0,
    "tune_layer_num": 1
  },
  "attention_config": {
    "rotary_embedding": 128,
    "rope_theta": 1000000.0,
    "softmax_scale": 0.0,
    "attention_factor": -1.0,
    "max_position_embeddings": 32768,
    "original_max_position_embeddings": 0,
    "rope_scaling_type": "",
    "rope_scaling_factor": 0.0,
    "use_dynamic_ntk": 0,
    "low_freq_factor": 1.0,
    "high_freq_factor": 1.0,
    "beta_fast": 32.0,
    "beta_slow": 1.0,
    "use_logn_attn": 0,
    "cache_block_seq_len": 64
  },
  "lora_config": {
    "lora_policy": "",
    "lora_r": 0,
    "lora_scale": 0.0,
    "lora_max_wo_r": 0,
    "lora_rank_pattern": "",
    "lora_scale_pattern": ""
  },
  "engine_config": {
    "dtype": "auto",
    "model_format": "awq",
    "tp": 2,
    "session_len": null,
    "max_batch_size": 128,
    "cache_max_entry_count": 0.8,
    "cache_chunk_size": -1,
    "cache_block_seq_len": 64,
    "enable_prefix_caching": false,
    "quant_policy": 0,
    "rope_scaling_factor": 0.0,
    "use_logn_attn": false,
    "download_dir": null,
    "revision": null,
    "max_prefill_token_num": 8192,
    "num_tokens_per_iter": 8192,
    "max_prefill_iters": 4
  }
}
[TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 32768.
[TM][INFO] Model: 
head_num: 40
kv_head_num: 8
size_per_head: 128
num_layer: 64
vocab_size: 152064
attn_bias: 1
max_batch_size: 128
max_prefill_token_num: 8192
max_context_token_num: 32768
num_tokens_per_iter: 8192
max_prefill_iters: 4
session_len: 32768
cache_max_entry_count: 0.8
cache_block_seq_len: 64
cache_chunk_size: -1
enable_prefix_caching: 0
start_id: 151643
tensor_para_size: 2
pipeline_para_size: 1
enable_custom_all_reduce: 0
model_name: 
model_dir: 
quant_policy: 0
group_size: 128
expert_per_token: 0
moe_method: 1

[TM][INFO] TM_FUSE_SILU_ACT=1
2025-02-02 10:48:54,353 - lmdeploy - WARNING - turbomind.py:217 - get 2309 model params
[TM][INFO] [LlamaWeight<T>::prepare] workspace size: 283115520                  

[TM][INFO] [LlamaWeight<T>::prepare] workspace size: 283115520

[TM][WARNING] Devicle 0 peer access Device 1 is not available.
[TM][WARNING] Devicle 1 peer access Device 0 is not available.
[WARNING] gemm_config.in is not found; using default GEMM algo
[WARNING] gemm_config.in is not found; using default GEMM algo
[TM][INFO] [BlockManager] block_size = 8 MB
[TM][INFO] [BlockManager] block_size = 8 MB
[TM][INFO] [BlockManager] max_block_count = 324
[TM][INFO] [BlockManager] chunk_size = 324
[TM][INFO] [BlockManager] max_block_count = 324
[TM][INFO] [BlockManager] chunk_size = 324
[TM][WARNING] No enough blocks for `session_len` (32768), `session_len` truncated to 20736.
[TM][INFO] LlamaBatch<T>::Start()
[TM][INFO] LlamaBatch<T>::Start()
[TM][INFO] [Gemm2] Tuning sequence: 8, 16, 32, 48, 64, 96, 128, 192, 256, 384, 512, 768, 1024, 1536, 2048, 3072, 4096, 6144, 8192
[TM][INFO] [Gemm2] 8
[TM][INFO] [Gemm2] 16
[TM][INFO] [Gemm2] 32
[TM][INFO] [Gemm2] 48
[TM][INFO] [Gemm2] 64
[TM][INFO] [Gemm2] 96
[TM][INFO] [Gemm2] 128
[TM][INFO] [Gemm2] 192
[TM][INFO] [Gemm2] 256
[TM][INFO] [Gemm2] 384
[TM][INFO] [Gemm2] 512
[TM][INFO] [Gemm2] 768
[TM][INFO] [Gemm2] 1024
[TM][INFO] [Gemm2] 1536
[TM][INFO] [Gemm2] 2048
[TM][INFO] [Gemm2] 3072
[TM][INFO] [Gemm2] 4096

Reproduction

!lmdeploy serve api_server Qwen/Qwen2.5-32B-Instruct-AWQ --tp 2

Environment

sys.platform: linux
Python: 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: Tesla T4
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.2, V12.2.140
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
PyTorch: 2.5.1+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.1  (built against CUDA 12.4)
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.20.1+cu121
LMDeploy: 0.7.0.post2+
transformers: 4.47.0
gradio: 5.14.0
fastapi: 0.115.8
pydantic: 2.10.3
triton: 3.1.0
NVIDIA Topology: 
	GPU0	GPU1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	PHB	0-3	0		N/A
GPU1	PHB	 X 	0-3	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

The text was updated successfully, but these errors were encountered:

lvhan028 · 2025-02-02T13:43:13Z

用的是哪个版本呢？

zzc0721 · 2025-02-02T13:49:05Z

lmdeploy是github上面发行的0.7.0.post2+cu118，kaggle中的notebook镜像是stable，这个模型也是官方做的AWQ量化

…

---原始邮件--- 发件人: "Lyu ***@***.***> 发送时间: 2025年2月2日(周日) 晚上9:43 收件人: ***@***.***>; 抄送: ***@***.******@***.***>; 主题: Re: [InternLM/lmdeploy] [Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 (Issue #3108) 用的是哪个版本呢？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

Azure99 · 2025-02-03T09:56:00Z

same issue here

lvhan028 · 2025-02-07T14:08:41Z

从日志中看，是在tuning的时候卡着了。@lzhangzz any clue?

lvhan028 · 2025-02-07T14:10:00Z

#3107 (comment) @zzc0721 @Azure99
可否尝试下这个 comment 里说的方法？

lvhan028 assigned lzhangzz Feb 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 #3108

[Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 #3108

zzc0721 commented Feb 2, 2025 •

edited

Loading

lvhan028 commented Feb 2, 2025

zzc0721 commented Feb 2, 2025 via email

Azure99 commented Feb 3, 2025

lvhan028 commented Feb 7, 2025

lvhan028 commented Feb 7, 2025

[Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 #3108

[Bug] 在Kaggle Notebook中使用turbomind backend推理Qwen/Qwen2.5-32B-Instruct-AWQ会无限期卡死 #3108

Comments

zzc0721 commented Feb 2, 2025 • edited Loading

Checklist

Describe the bug

Reproduction

Environment

Error traceback

lvhan028 commented Feb 2, 2025

zzc0721 commented Feb 2, 2025 via email

Azure99 commented Feb 3, 2025

lvhan028 commented Feb 7, 2025

lvhan028 commented Feb 7, 2025

zzc0721 commented Feb 2, 2025 •

edited

Loading