[Bug]: Mistral model loading reporting unsupported operand type(s) #18573
Description
Your current environment
```
vLLM Version : 0.8.5.post1
```
🐛 Describe the bug
If you run the script of
LLM(
model="mistralai/Mistral-7B-Instruct-v0.2",
**vllm_kwargs,
)
or any models based on Mistral, you would encounter the issue
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'`
in vllm/model_executor/models/llama.py line 135.
Solution:
I think VLLM is trying to be efficient in terms of loading Mistral model using their source code from Llama, which is totally fine.
The real issue lies in the line 131 in vllm/model_executor/models/llama.py, where they have this line of
self.head_dim = getattr(config, "head_dim",
self.hidden_size // self.total_num_heads)
The interesting thing here is that MistralConfig does seem to contain the key of head_dim (corresponding to a value of null), which means that it will get the value of NoneType.
Therefore, a simple and quick fix is to replace the line 131 by
if not getattr(config, "head_dim"):
self.head_dim = self.hidden_size // self.total_num_heads
else:
self.head_dim = getattr(config, "head_dim",
self.hidden_size // self.total_num_heads)
Hope this helps!
Official Fix:
The issue has been fixed in a recent update #18432.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.