OOM when use InternVL2_5-1B-MPO

[BobHo5474]

I followed the installation guide to build mldeploy (0.7.0.post3) from source.
Inference using the PyTorch engine works fine.
However, after quantizing the model to 4-bit using AWQ, I encountered an OOM error when loading the model with the TurboMind engine.
I try to set "session_len=2048" in TurbomindEngineConfig.

[lvhan028]

Can you share the following information?

• running lmdeploy check_env
• the reproducible code

[BobHo5474]

I will get the error when I run lmdeploy check_env because I built lmdeploy on Jetson Orin.

Below is the code,
from lmdeploy import pipeline, TurbomindEngineConfig, PytorchEngineConfig
pipe = pipeline("./InternVL2_5-1B-MPO-4bit/", backend_config=TurbomindEngineConfig(model_format="awq", session_len=2048))

I run lmdeploy lite auto_awq OpenGVLab/InternVL2_5-1B-MPO --work-dir InternVL2_5-1B-MPO-4bit to quantize model.

[lvhan028]

Can you help open INFO log level? Let's check what the log indicates

from lmdeploy import pipeline, TurbomindEngineConfig, PytorchEngineConfig
pipe = pipeline("./InternVL2_5-1B-MPO-4bit/", backend_config=TurbomindEngineConfig(model_format="awq",session_len=2048), log_level='INFO')

[lvhan028]

What's the mem size of jetson orin?

[BobHo5474]

GPU memory size is 16GB, and I uploaded the log file. log.txt