Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] lmdeploy Turbomind 推理导致jupyter 崩溃 #3128

Open
3 tasks done
xiezhipeng-git opened this issue Feb 10, 2025 · 4 comments
Open
3 tasks done

[Bug] lmdeploy Turbomind 推理导致jupyter 崩溃 #3128

xiezhipeng-git opened this issue Feb 10, 2025 · 4 comments
Assignees
@lzhangzz

Comments

@xiezhipeng-git
Copy link

xiezhipeng-git commented Feb 10, 2025 •
edited
Loading

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

在vscode上python3.12 使用lmdeploy 推理。导致jupyter内核崩溃。不知道是哪里的问题

Image

Reproduction

import os
os.environ["LMDEPLOY_USE_MODELSCOPE"] = "True"
from modelscope import snapshot_download
from modelscope.hub.file_download import model_file_download

MODEL_PATH = snapshot_download('qwen/Qwen2.5-Coder-0.5B-Instruct-GPTQ-Int4')
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
pipe = pipeline(MODEL_PATH,
                backend_config=TurbomindEngineConfig(
                    max_batch_size=32,
                    enable_prefix_caching=True,
                    cache_max_entry_count=0.8,
                    session_len=8192,
                    quant_policy=4,
                ))
gen_config = GenerationConfig(top_p=0.8,
                              top_k=40,
                              temperature=0.8,
                              do_sample=True,
                              max_new_tokens=1024)
response = pipe(['Hi, pls intro yourself', 'Shanghai is'],
                gen_config=gen_config)

print(response)

Environment

(base) root@PS2023STCKYAEB:/mnt/d/my/work/study/ai/kaggle_code/aimo2# lmdeploy check_env
sys.platform: linux
Python: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:27:36) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA GeForce RTX 4090
CUDA_HOME: /usr/local/cuda-12.6
NVCC: Cuda compilation tools, release 12.6, V12.6.77
GCC: gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
PyTorch: 2.5.1+cu124
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.4
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 90.1
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.20.1+cu124
LMDeploy: 0.7.0.post3+7780d7d
transformers: 4.48.2
gradio: 5.13.1
fastapi: 0.115.4
pydantic: 2.8.2
triton: 3.1.0
NVIDIA Topology: 
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X                              N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

21:53:05.031 [trace] Start cell 2 execution @ 1739195585031 (clear output)
21:53:05.031 [debug] Kernel acknowledged execution of cell 2 @ 1739195585031
21:53:08.478 [debug] KernelProcess error 54635: [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:15.223 [trace] Executing silently Code (busy) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a
21:53:15.223 [trace] Executing silently Code (busy) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a
21:53:15.895 [debug] Kernel output 54635: [WARNING] gemm_config.in is not found; using default GEMM algo

21:53:28.864 [trace] Cell 2 completed in 23.833s (start: 1739195585031, end: 1739195608864)
21:53:28.864 [trace] Cell 2 executed successfully
21:53:28.864 [trace] Cell Index:3 sent to kernel
21:53:28.881 [trace] Executing silently Code (completed) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a with 1 output(s)
21:53:28.925 [trace] Start cell 3 execution @ 1739195608925 (clear output)
21:53:28.925 [debug] Kernel acknowledged execution of cell 3 @ 1739195608925
21:53:28.925 [trace] Executing silently Code (completed) = def _VSCODE_getVariable(what_to_get, is_debugging, *args):\n# Query Jupyter server for the info a with 1 output(s)
21:53:28.925 [trace] Cell 3 completed in 0s (start: 1739195608925, end: 1739195608925)
21:53:28.926 [trace] Cell 3 executed successfully
21:53:28.926 [trace] Cell Index:0 sent to kernel
21:53:28.930 [trace] Start cell 0 execution @ 1739195608930 (clear output)
21:53:28.930 [debug] Kernel acknowledged execution of cell 0 @ 1739195608930
21:53:28.931 [trace] Cell 0 completed in 0.001s (start: 1739195608930, end: 1739195608931)
21:53:28.931 [trace] Cell 0 executed successfully
21:53:40.511 [debug] Handle Execution of Cells 4 for /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb
21:53:40.511 [trace] Execute Notebook /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 1
21:53:40.511 [trace] Connect to Kernel /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 2
21:53:40.511 [trace] Connected to Kernel /mnt/d/my/work/study/ai/kaggle_code/aimo2/test/llmdeoloy_test.ipynb. Step 3
21:53:40.511 [trace] executeCell 4. Step 4
21:53:40.511 [trace] Cell Index:4 sent to kernel
21:53:40.514 [trace] Start cell 4 execution @ 1739195620514 (clear output)
21:53:40.514 [debug] Kernel acknowledged execution of cell 4 @ 1739195620514
21:53:47.782 [debug] KernelProcess Exited 54635, Exit Code - null [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:47.782 [error] Disposing session as kernel process died ExitCode: undefined, Reason: [TM][WARNING] [LlamaTritonModel] `max_context_token_num` is not set, default to 8192.

21:53:47.783 [debug] Dispose Kernel process 54635.
21:53:47.787 [debug] Disposed Kernel process 54635.
21:53:47.806 [trace] Cell 4 completed in -1739195620.514s (start: 1739195620514, end: undefined)

改成pytorch 推理后端报下列错误。是需要使用awq量化？

RuntimeError                              Traceback (most recent call last)
Cell In[3], [line 2](vscode-notebook-cell:?execution_count=3&line=2)
      [1](vscode-notebook-cell:?execution_count=3&line=1) from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig, PytorchEngineConfig
----> [2](vscode-notebook-cell:?execution_count=3&line=2) pipe = pipeline(MODEL_PATH,
      [3](vscode-notebook-cell:?execution_count=3&line=3)                 # backend_config=TurbomindEngineConfig(
      [4](vscode-notebook-cell:?execution_count=3&line=4)                 backend_config=PytorchEngineConfig(
      [5](vscode-notebook-cell:?execution_count=3&line=5)                     max_batch_size=32,
      [6](vscode-notebook-cell:?execution_count=3&line=6)                     enable_prefix_caching=True,
      [7](vscode-notebook-cell:?execution_count=3&line=7)                     cache_max_entry_count=0.[8](vscode-notebook-cell:?execution_count=3&line=8),
      8                     session_len=81[9](vscode-notebook-cell:?execution_count=3&line=9)2,
      9                     quant_policy=4,
     [10](vscode-notebook-cell:?execution_count=3&line=10)                 ))

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:83, in pipeline(model_path, backend_config, chat_template_config, log_level, max_log_len, **kwargs)
     [80](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:80) backend = 'pytorch' if type(backend_config) is PytorchEngineConfig else 'turbomind'
     [81](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:81) logger.info(f'Using {backend} engine')
---> [83](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:83) return pipeline_class(model_path,
     [84](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:84)                       backend=backend,
     [85](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:85)                       backend_config=backend_config,
     [86](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:86)                       chat_template_config=chat_template_config,
     [87](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:87)                       max_log_len=max_log_len,
     [88](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/api.py:88)                       **kwargs)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:276, in AsyncEngine.__init__(self, model_path, model_name, backend, backend_config, chat_template_config, max_log_len, **kwargs)
    [274](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:274)     self._build_turbomind(model_path=model_path, backend_config=backend_config, **kwargs)
    [275](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:275) elif backend == 'pytorch':
--> [276](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:276)     self._build_pytorch(model_path=model_path, backend_config=backend_config, **kwargs)
    [277](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:277) else:
    [278](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:278)     raise ValueError(f'unsupported backend {backend}')

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:328, in AsyncEngine._build_pytorch(self, model_path, backend_config, **kwargs)
    [326](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:326) """Innter build method for pytorch backend."""
    [327](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:327) from lmdeploy.pytorch.engine import Engine
--> [328](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:328) self.engine = Engine(model_path=model_path, tokenizer=self.tokenizer, engine_config=backend_config)
    [329](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:329) self.backend_config = self.engine.engine_config
    [330](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/serve/async_engine.py:330) self.hf_tm_cfg = getattr(self.engine.model_config, 'hf_config', None)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:148, in Engine.__init__(self, model_path, tokenizer, engine_config, trust_remote_code)
    [145](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:145) backend_config = _build_backend_config(engine_config)
    [147](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:147) with get_device_manager().context(self.device_context):
--> [148](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:148)     self.model_agent = build_model_agent(model_path,
    [149](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:149)                                          cache_config=cache_config,
    [150](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:150)                                          backend_config=backend_config,
    [151](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:151)                                          trust_remote_code=trust_remote_code,
    [152](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:152)                                          adapters=adapters,
    [153](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:153)                                          tp=self.tp,
    [154](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:154)                                          dtype=engine_config.dtype,
    [155](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:155)                                          custom_module_map=engine_config.custom_module_map)
    [157](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:157) self.input_processor = self.model_agent.get_input_processor()
    [159](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/engine.py:159) cache_config = self.model_agent.cache_config

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:703, in build_model_agent(model_path, cache_config, backend_config, trust_remote_code, adapters, tp, dtype, custom_module_map)
    [701](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:701) model_config.custom_module_map = custom_module_map
    [702](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:702) if tp == 1:
--> [703](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:703)     model_agent = BaseModelAgent(model_path,
    [704](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:704)                                  model_config=model_config,
    [705](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:705)                                  cache_config=cache_config,
    [706](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:706)                                  backend_config=backend_config,
    [707](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:707)                                  adapters=adapters,
    [708](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:708)                                  trust_remote_code=trust_remote_code)
    [709](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:709) else:
    [710](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:710)     model_agent = TPModelAgent(model_path,
    [711](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:711)                                model_config=model_config,
    [712](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:712)                                cache_config=cache_config,
   (...)
    [715](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:715)                                adapters=adapters,
    [716](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:716)                                trust_remote_code=trust_remote_code)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:208, in BaseModelAgent.__init__(self, model_path, model_config, cache_config, backend_config, adapters, trust_remote_code)
    [205](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:205) self.backend_config = backend_config
    [206](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:206) self._adapters = adapters
--> [208](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:208) self.patched_model = self._build_model(model_path, adapters, device=device)
    [210](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:210) _update_cache_config(model_config, cache_config)
    [212](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:212) backend = get_backend()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:229, in BaseModelAgent._build_model(self, model_path, adapters, device)
    [227](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:227)     update_custom_module_map(custom_module_map)
    [228](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:228) logger.info('build model.')
--> [229](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:229) patched_model = build_patched_model(self.model_config, device=device)
    [230](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:230) logger.info('loading weights.')
    [231](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/engine/model_agent.py:231) load_model_weights(patched_model, model_path, device=device)

File ~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    [113](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:113) @functools.wraps(func)
    [114](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:114) def decorate_context(*args, **kwargs):
    [115](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:115)     with ctx_factory():
--> [116](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/torch/utils/_contextlib.py:116)         return func(*args, **kwargs)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:195, in build_patched_model(config, device)
    [193](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:193) model_config = config.hf_config
    [194](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:194) dtype = config.dtype
--> [195](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:195) return build_model_from_hf_config(model_config, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:186, in build_model_from_hf_config(model_config, dtype, device)
    [184](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:184)     device = torch.device('cuda')
    [185](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:185) model_cls = _get_model_class(model_config, module_map)
--> [186](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:186) model = model_cls(model_config, ctx_mgr, dtype=dtype, device=device)
    [187](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/patch.py:187) return model.eval()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:306, in Qwen2ForCausalLM.__init__(self, config, ctx_mgr, dtype, device)
    [304](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:304) self.ctx_mgr = ctx_mgr
    [305](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:305) # build model
--> [306](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:306) self.model = Qwen2Model(config, dtype=dtype, device=device)
    [307](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:307) # build lm_head
    [308](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:308) self.lm_head = build_rowwise_linear(config.hidden_size,
    [309](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:309)                                     config.vocab_size,
    [310](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:310)                                     bias=False,
    [311](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:311)                                     dtype=dtype,
    [312](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:312)                                     device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:219, in Qwen2Model.__init__(self, config, dtype, device)
    [211](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:211) self.embed_tokens = nn.Embedding(config.vocab_size,
    [212](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:212)                                  config.hidden_size,
    [213](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:213)                                  self.padding_idx,
    [214](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:214)                                  dtype=dtype,
    [215](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:215)                                  device=device)
    [217](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:217) # build all decode layers
    [218](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:218) self.layers = nn.ModuleList([
--> [219](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:219)     Qwen2DecoderLayer(config, layer_idx, dtype=dtype, device=device)
    [220](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:220)     for layer_idx in range(config.num_hidden_layers)
    [221](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:221) ])
    [223](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:223) # build norm
    [224](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:224) self.norm = RMSNorm(config.hidden_size, config.rms_norm_eps, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:153, in Qwen2DecoderLayer.__init__(self, config, layer_idx, dtype, device)
    [150](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:150) quantization_config = getattr(config, 'quantization_config', None)
    [152](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:152) # build attention layer
--> [153](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:153) self.self_attn = Qwen2Attention(config, dtype=dtype, device=device)
    [155](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:155) # build MLP
    [156](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:156) self.mlp = Qwen2MLP(config, dtype=dtype, device=device)

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:30, in Qwen2Attention.__init__(self, config, dtype, device)
     [28](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:28) num_replicate_kv_heads = getattr(config, 'num_replicate_key_value_heads', 1)
     [29](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:29) # packed qkv
---> [30](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:30) self.qkv_proj = build_qkv_proj(hidden_size,
     [31](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:31)                                num_q_heads=num_heads,
     [32](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:32)                                num_kv_heads=num_key_value_heads,
     [33](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:33)                                head_size=head_dim,
     [34](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:34)                                bias=True,
     [35](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:35)                                quant_config=quantization_config,
     [36](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:36)                                dtype=dtype,
     [37](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:37)                                device=device,
     [38](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:38)                                num_replicate_kv_heads=num_replicate_kv_heads)
     [40](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:40) # rotary embedding
     [41](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/models/qwen2.py:41) self.apply_rotary_pos_emb = ApplyRotaryEmb()

File ~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1495, in build_qkv_proj(in_features, num_q_heads, num_kv_heads, head_size, head_size_v, bias, quant_config, dtype, device, is_tp, num_replicate_kv_heads)
   [1483](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1483)     return QKVW8A8Linear(in_features=in_features,
   [1484](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1484)                          num_q_heads=num_q_heads,
   [1485](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1485)                          num_kv_heads=num_kv_heads,
   (...)
   [1492](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1492)                          num_replicate_kv_heads=num_replicate_kv_heads,
   [1493](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1493)                          quant_dtype=quant_dtype)
   [1494](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1494) else:
-> [1495](https://vscode-remote+wsl-002bubuntu-002d24-002e04.vscode-resource.vscode-cdn.net/mnt/d/my/work/study/ai/kaggle_code/aimo2/test/~/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/nn/linear.py:1495)     raise RuntimeError(f'Unsupported quant method: {quant_method}')

RuntimeError: Unsupported quant method: gptq

换成awq 和 pytorch 以后得到下列错误提示

loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands

awq 和 Turbomind同样jupyter崩溃
@xiezhipeng-git
Copy link
Author

xiezhipeng-git commented Feb 10, 2025 •
edited
Loading

@lvhan028 试过好几次了。必定崩溃。唯一一次没有崩溃的是没加quant_policy=4, 这个配置
好像是先导致了python崩溃。然后python崩溃导致了jupyter 崩溃

@xiezhipeng-git xiezhipeng-git changed the title [Bug] 推理导致jupyter 崩溃 [Bug] lmdeploy Turbomind 推理导致jupyter 崩溃 Feb 10, 2025
@lvhan028
Copy link
Collaborator

pytorch engine没有支持 gptq 量化算法

@xiezhipeng-git
Copy link
Author

xiezhipeng-git commented Feb 10, 2025 •
edited
Loading

pytorch engine没有支持 gptq 量化算法

换成awq 以后有
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands
@lvhan028
如果切回 TurbomindEngineConfig
还是jupyter崩溃

@lvhan028
Copy link
Collaborator

类似 loc("/root/anaconda3/lib/python3.12/site-packages/lmdeploy/pytorch/kernels/cuda/pagedattention.py":475:11): error: operation scheduled before its operands 可以不用管

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lzhangzz lzhangzz

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lvhan028 @xiezhipeng-git @lzhangzz