Add validation to reject non-text content in system messages

[veeceey]

Summary

Fixes #33925

According to OpenAI API specification, system messages can only contain text content. This PR adds validation to warn on non-text content types (images, audio, video) in system messages.

Problem

vLLM was accepting images and other multimodal content in system messages without any indication, which deviates from the OpenAI API specification. System messages should only support text content.

Changes

• Added check_system_message_content_type model validator in ChatCompletionRequest
• Validator checks if system messages contain non-text content types
• Logs a warning_once with clear message indicating the non-text content type (avoids log spam)
• Handles both explicit type field and inferred types from content keys
• User messages and other roles still accept multimodal content

Test Plan

Added comprehensive tests in tests/entrypoints/openai/test_chat_error.py:

• test_system_message_warns_on_image - Validates that image_url triggers warning (parametrized: explicit and inferred type)
• test_system_message_warns_on_audio - Validates that input_audio triggers warning (parametrized: explicit and inferred type)
• test_system_message_warns_on_video - Validates that video_url triggers warning (parametrized: explicit and inferred type)
• test_system_message_accepts_text - Validates that text content is accepted
• test_system_message_accepts_text_array - Validates that text array format is accepted
• test_user_message_accepts_image - Confirms user messages still accept images

[gemini-code-assist]

Code Review

This pull request adds validation to ensure system messages only contain text content, aligning with the OpenAI API specification. The implementation in ChatCompletionRequest has a potential validation bypass where non-text content parts without an explicit type field are not being rejected. I've suggested a fix for this critical issue. Additionally, I've recommended parameterizing the new tests to cover these implicit type cases for images, audio, and video, ensuring the validation is more robust.

[mergify]

Hi @veeceey, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[qandrew]

overall makes sense to me, can you address the gemini comments?

[veeceey]

Thanks @qandrew and @gemini-code-assist for the thorough review!

I've addressed all the feedback in the latest commit:

Validation bypass fix:

• Extended the validation to infer content type from keys when the explicit type field is missing
• Now detects image_url, image_pil, input_audio, audio_url, video_url, etc. even without explicit type
• Prevents the validation bypass mentioned in the review

Test improvements:

• Parameterized all three multimodal tests (image, audio, video)
• Each test now covers both explicit type ({"type": "image_url", ...}) and implicit type ({"image_url": ...})
• Ensures the validation is robust against both content formats

This should make the validation comprehensive and aligned with how vLLM actually parses multimodal content.

[gemini-code-assist]

Thanks for the update, @veeceey! I appreciate you taking the time to address the feedback so thoroughly. The changes you've implemented, especially extending the validation to infer content types from keys and the comprehensive parameterized tests, look great and effectively resolve the concerns raised. Excellent work!

[mergify]

Hi @veeceey, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy or markdownlint failing?

mypy and markdownlint are run differently in CI. If the failure is related to either of these checks, please use the following commands to run them locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10
# For markdownlint
pre-commit run --hook-stage manual markdownlint

[veeceey]

Hi @qandrew, just to confirm: all the gemini review comments have been addressed (validation bypass fix with type inference, and parameterized tests for image/audio/video). The pre-commit checks are now passing as well after the latest formatting fix commit. Ready for re-review when you get a chance!

[chaunceyjiang]

Thanks @veeceey and @qandrew. I just have one question: does this limitation also apply to some multimodal models?

cc @DarkLight1337 WDYT?

[veeceey]

Good question @chaunceyjiang. This validation follows the OpenAI API specification, which states that system messages should only contain text content. The spec is consistent regardless of whether the model is multimodal or not -- multimodal content (images, audio, video) should be sent in user messages, not system messages.

That said, if there are specific multimodal models in vLLM that use system messages differently from the OpenAI spec, this could be made configurable or the validation could be relaxed for those models. Happy to adjust the approach based on what @DarkLight1337 and the team think is best.

[DarkLight1337]

I am not aware of multimodal models that use multimodal system messages, but some users might craft their own system messages that do that, for whatever reason. So I suggest being less strict about it.

cc @Isotr0py @ywang96

[veeceey]

Thanks for the feedback @DarkLight1337! That's a fair point — I've updated the validation to log a warning instead of raising an error. This way users are informed that they're deviating from the OpenAI API spec, but won't be blocked if they intentionally send multimodal system messages. The latest commit changes VLLMValidationError to logger.warning and updates the tests accordingly.

[veeceey]

Thanks @DarkLight1337! Updated logger.warning to logger.warning_once to avoid log spam as suggested. The validation is intentionally kept as a warning (not an error) to be less strict and accommodate users who may craft multimodal system messages.

[DarkLight1337]

PTAL at the failing entrypoints test

[veeceey]

Fixed in the latest commit. The issue was that warning_once uses @lru_cache internally, so when two parametrized test cases trigger a warning with the same content type (e.g. both produce part_type = "image_url"), the second case hits the cache and doesn't actually emit a log message -- causing the caplog assertion to fail.

Added an autouse fixture that clears the _print_warning_once cache before and after each test, so every test case gets a fresh state.

The language-models-tests-hybrid-1 failure looks unrelated to this PR since our changes only touch protocol.py and test_chat_error.py.

[DarkLight1337]

PTAL at the failing test

[veeceey]

Rebased onto main to pick up the MockVllmConfig / HfRenderer signature changes that landed recently. The entrypoints-integration-api-server-1 failure was due to the branch being ~190 commits behind -- the test mock setup was out of sync with the new VllmConfig wrapper. Should be clean now.

[veeceey]

Rebased onto latest main to pick up recent fixes (including #34516 error response bugfix and #34590). This should resolve the CI failures - the previous rebase wasn't actually pushed. New CI run should be green.

[veeceey]

@DarkLight1337 The only remaining CI failure is entrypoints-integration-responses-api, which is the known flaky SSL test in test_harmony.py::test_function_calling -- the same one that was just fixed in #34624 (mocking the get_weather call to avoid api.open-meteo.com SSL errors in CI). You can see that #34618, which was merged before #34624, has the exact same responses-api failure. This is unrelated to our changes.

entrypoints-integration-api-server-1 is still running but api-server-2 already passed, and all the other entrypoints tests (llm, pooling, unit-tests, v1, openai-api-correctness) are green. Looks like the mock.patch fix resolved those failures.