Remove incorrect tokenizer info test

[hmellor]

#20575 added, what is now called, /tokenizer_info endpoint.

It added the following serialisation method that is used before the tokenizer info is sent:

vllm/vllm/entrypoints/serve/tokenize/serving.py

Lines 186 to 195 in d95b4be

	def _make_json_serializable(self, obj):
	"""Convert any non-JSON-serializable objects to serializable format."""
	if hasattr(obj, "content"):
	return obj.content
	elif isinstance(obj, dict):
	return {k: self._make_json_serializable(v) for k, v in obj.items()}
	elif isinstance(obj, list):
	return [self._make_json_serializable(item) for item in obj]
	else:
	return obj

In this you can see that if an obj has the attribute content it will only return the content from that object. The values of the the added_tokens_decoder field are AddedToken objects, which do have the attribute content.

This means that, by design, the tokenizer info endpoint was only meant to return the string value of the token. Unfortunately, this is directly contradicted in the test that was added by the same PR

vllm/tests/entrypoints/openai/test_tokenization.py

Lines 304 to 321 in d95b4be

	@pytest.mark.asyncio
	async def test_tokenizer_info_added_tokens_structure(
	server: RemoteOpenAIServer,
	):
	"""Test added_tokens_decoder structure if present."""
	response = requests.get(server.url_for("tokenizer_info"))
	response.raise_for_status()
	result = response.json()
	added_tokens = result.get("added_tokens_decoder")
	if added_tokens:
	for token_id, token_info in added_tokens.items():
	assert isinstance(token_id, str), "Token IDs should be strings"
	assert isinstance(token_info, dict), "Token info should be a dict"
	assert "content" in token_info, "Token info should have content"
	assert "special" in token_info, "Token info should have special flag"
	assert isinstance(token_info["special"], bool), (
	"Special flag should be boolean"
	)

This was not noticed because the model used in the test does not have and added_tokens_decoder for Transformers v4.

This PR opts to simply remove the test so that:

• The existing API (whether or not it was intended) remains unchanged
• The test does not fail for Transformers v5 where added_tokens_decoder are present and the test fails

[gemini-code-assist]

Code Review

This pull request removes an incorrect test for the /tokenizer_info endpoint. The test was asserting an incorrect structure for added_tokens_decoder. While removing the test is a valid fix, I've suggested replacing it with a corrected test that asserts the actual API behavior. This maintains test coverage for this feature.

I am having trouble creating individual review comments. Click here to see my feedback.

tests/entrypoints/openai/test_tokenization.py (304-321)

You've correctly identified that this test is incorrect given the serialization logic. While removing it fixes the immediate issue of a failing test, it also removes test coverage for the added_tokens_decoder field.

A better approach would be to correct the test to reflect the actual behavior of the API. Based on your description, the token_info is serialized to just the string content. The test should be updated to assert this.

This ensures we still have a test for this part of the API response, and it correctly documents the expected behavior.

@pytest.mark.asyncio
async def test_tokenizer_info_added_tokens_structure(
    server: RemoteOpenAIServer,
):
    """Test added_tokens_decoder structure if present."""
    response = requests.get(server.url_for("tokenizer_info"))
    response.raise_for_status()
    result = response.json()
    added_tokens = result.get("added_tokens_decoder")
    if added_tokens:
        for token_id, token_info in added_tokens.items():
            assert isinstance(token_id, str), "Token IDs should be strings"
            assert isinstance(token_info, str), "Token info should be a string"

[hmellor]

I'd ignore Gemini's suggestion. There is little value in testing that a JSON is a dict mapping str to str

[DarkLight1337]

Thanks for the detailed explanation!