[Bugfix] Fix bad words for Mistral models

[qionghuang6]

When using the bad_words SamplingParam with Mistral models with V1, bad_words compiles incorrectly.
For example, when bad_words=["the"] _bad_words_token_ids is [[1, 3265]] when it should be [[3265]], so the word "the" is never matched.

We fix this by setting add_special_tokens=False for all tokenizers when we generate _bad_words_token_ids

It seems that for non-Mistral models too, we don't want to add the special tokens either, so it's a little unclear why this logic was included, as other models like Llama also include start of sequence tokens.

We can replicate this bug with any Mistral model using the original e2e testing code from #13376

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

Thanks for the bugfix! LGTM but I think the following code needs to be updated as well?

vllm/vllm/logits_process.py

Lines 32 to 37 in 98c89e1

	if isinstance(tokenizer, MistralTokenizer):
	# Mistral tokenizers should not add special tokens
	prompt_token_ids = tokenizer.encode(text=prompt)
	else:
	prompt_token_ids = tokenizer.encode(text=prompt,
	add_special_tokens=False)

[ywang96]

Looks like the issue was caused by the BOS token so this makes sense

vllm/vllm/transformers_utils/tokenizers/mistral.py

Lines 369 to 374 in 98c89e1

	if add_special_tokens is not None:
	return self.tokenizer.encode(text,
	bos=add_special_tokens,
	eos=add_special_tokens)
	else:
	return self.tokenizer.encode(text, bos=True, eos=False)

[qionghuang6]

Thanks for the bugfix! LGTM but I think the following code needs to be updated as well?

vllm/vllm/logits_process.py

Lines 32 to 37 in 98c89e1

	if isinstance(tokenizer, MistralTokenizer):
	# Mistral tokenizers should not add special tokens
	prompt_token_ids = tokenizer.encode(text=prompt)
	else:
	prompt_token_ids = tokenizer.encode(text=prompt,
	add_special_tokens=False)

Ah yes, I only updated it for V1. Just added it now

[ywang96]

LGTM!