[Bug]: OOM when serve Gemma3-AWQ

[hahmad2008]

Your current environment

Vllm 0.8.3

🐛 Describe the bug

I couldn't serve gemma3-AWQ model on 24G GPU.

• command1:

VLLM_USE_V1=0  vllm serve gaunernst/gemma-3-27b-it-int4-awq \
 --max-model-len 4096 --gpu-memory-utilization 0.98   --distributed-executor-backend ray  --dtype float16 

• error1:

INFO 04-07 15:52:21 [model_runner.py:1146] Model loading took 17.6110 GiB and 12.273077 seconds

 ValueError: The model's max seq len (4096) is larger than the maximum number of tokens that can be stored in KV cache (3584). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

• command2:

 VLLM_USE_V1=0  vllm serve gaunernst/gemma-3-27b-it-int4-awq \
 --max-model-len 128 --gpu-memory-utilization 0.98   --distributed-executor-backend ray  --dtype float16 

• error2:

INFO 04-07 15:54:56 [worker.py:267] the current vLLM instance can use total_gpu_memory (21.99GiB) x gpu_memory_utilization (0.98) = 21.55GiB

ERROR 04-07 15:55:35 [engine.py:448]   File "/python3.9/site-packages/torch/cuda/graphs.py", line 84, in capture_end
ERROR 04-07 15:55:35 [engine.py:448]     super().capture_end()
ERROR 04-07 15:55:35 [engine.py:448] RuntimeError: CUDA error: out of memory
ERROR 04-07 15:55:35 [engine.py:448] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

Only work when I set context length to 128 and num seq to 2, but the response for any request is NULL!!

 VLLM_USE_V1=0  vllm serve gaunernst/gemma-3-27b-it-int4-awq --max-num-seqs 2  \
 --max-model-len 128 --gpu-memory-utilization 0.9   --distributed-executor-backend ray  --dtype float16 

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You should avoid setting --dtype float16 as it leads to numerical instabilities

[hahmad2008]

@DarkLight1337 what should I use instead?

[DarkLight1337]

You can just remove --dtype, since you're using quantization anyway

[liaspas]

Without specifying dtype, the error becomes:

ValueError: torch.bfloat16 is not supported for quantization method awq. Supported dtypes: [torch.float16]

So it seems awq only works with float16, but in that case the model's output is empty.

[DarkLight1337]

cc @mgoin @WoosukKwon any way to solve this?

[liaspas]

For some extra context, this happens with a T4 GPU. Using A100 seems to work fine without specifying --dtype.

[DarkLight1337]

It might be that there are no suitable kernels for your hardware to use this quantization method. @mgoin correct me if I'm wrong

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!