[Bug]: index out of bound for logits_processors cause vllm.engine.async_llm_engine.AsyncEngineDeadError

[FeiDeng]

Your current environment

use the Vllm with phi3 model "microsoft/Phi-3-mini-128k-instruct"

🐛 Describe the bug

The issue I think is user could give a token id which exceed the vocab size. And then this error cause the vllm.engine.async_llm_engine.AsyncEngineDeadError. The following sample actually not well decribe the issue.

please check this call stack"
0] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/logits_processor.py", line 59, in forward
0] logits = _apply_logits_processors(logits, sampling_metadata)
0] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/logits_processor.py", line 116, in _apply_logits_processors
0] logits_row = logits_processor(past_tokens_ids,
0] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/protocol.py", line 245, in logit_bias_logits_processor
0] logits[int(token_id)] += bias
0] IndexError: index 55434 is out of bounds for dimension 0 with size 32064
0] The above exception was the direct cause of the following exception:
0] Traceback (most recent call last):
0] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 54, in _log_task_completion
0] raise AsyncEngineDeadError(
0] vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.

We may need better error handling. This issue cause some container to error state

[robertgshaw2-redhat]

Is 9857777777 in your vocab size? I don't quite follow what you are trying to do with the process_token function

[FeiDeng]

Thank you for looking at this. 32064 is the vocab size. 9857777777 is a value I randomly give that larger than vocab size. The issue I think is user could give a token id which exceed the vocab size. So we need basiclly a token id check in logit process. (https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/logits_processor.py)

[FeiDeng]

Also this will index error seems raise AsyncEngineDeadError in the and and cause the system in error state. Is there a way we can avoid to get into the error state for vm?

[FeiDeng]

here are some of the call stack:
0] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/logits_processor.py", line 59, in forward
0] logits = _apply_logits_processors(logits, sampling_metadata)
0] File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/logits_processor.py", line 116, in _apply_logits_processors
0] logits_row = logits_processor(past_tokens_ids,
0] File "/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/protocol.py", line 245, in logit_bias_logits_processor
0] logits[int(token_id)] += bias
0] IndexError: index 55434 is out of bounds for dimension 0 with size 32064
0] The above exception was the direct cause of the following exception:
0] Traceback (most recent call last):
0] File "/usr/local/lib/python3.10/dist-packages/vllm/engine/async_llm_engine.py", line 54, in _log_task_completion
0] raise AsyncEngineDeadError(
0] vllm.engine.async_llm_engine.AsyncEngineDeadError: Task finished unexpectedly. This should never happen! Please open an issue on Github. See stack trace above for theactual cause.

[FeiDeng]

I actually add a PR for this. https://github.com/vllm-project/vllm/pull/6927/commits. Please help review