[Misc] Use helper function to generate dummy messages in OpenAI MM tests#26875
[Misc] Use helper function to generate dummy messages in OpenAI MM tests#26875DarkLight1337 merged 3 commits intovllm-project:mainfrom
Conversation
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
There was a problem hiding this comment.
Code Review
This pull request effectively refactors the multi-modal tests for audio, video, and vision to use helper functions for generating dummy messages, which significantly reduces code duplication. The implementation is clean and improves maintainability. However, I've found a critical issue in tests/entrypoints/openai/test_video.py where a test case was incorrectly refactored, leading to it being broken. Please address this issue.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request refactors multi-modal tests for OpenAI entrypoints by introducing helper functions to generate dummy messages, which successfully reduces code duplication within each test file. The changes are generally good and improve readability. I've pointed out a minor inconsistency in the test data for video tests that could break assertions. I've also suggested a further refactoring to consolidate the new, very similar helper functions into a single generic function to improve long-term maintainability and fully adhere to the DRY principle.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request is a good step towards improving the test suite by refactoring duplicated code for creating dummy messages into helper functions. This has been applied to tests for audio, video, and vision modalities. The changes make the individual tests cleaner and easier to read.
My main feedback is that while this removes duplication within each test file, it introduces very similar helper functions across three different files (test_audio.py, test_video.py, test_vision.py). I've left a comment suggesting a further refactoring to a single, generic helper function in a shared utility file (e.g., tests/utils.py) to eliminate this new duplication and further improve the maintainability of the codebase.
| def dummy_messages_from_audio_url( | ||
| audio_urls: str | list[str], | ||
| content_text: str = "What's happening in this audio?", | ||
| ): | ||
| messages = [ | ||
| if isinstance(audio_urls, str): | ||
| audio_urls = [audio_urls] | ||
|
|
||
| return [ | ||
| { | ||
| "role": "user", | ||
| "content": [ | ||
| {"type": "audio_url", "audio_url": {"url": audio_url}}, | ||
| {"type": "text", "text": "What's happening in this audio?"}, | ||
| *( | ||
| {"type": "audio_url", "audio_url": {"url": audio_url}} | ||
| for audio_url in audio_urls | ||
| ), | ||
| {"type": "text", "text": content_text}, | ||
| ], | ||
| } | ||
| ] |
There was a problem hiding this comment.
While this helper function is a good step towards reducing code duplication within this file, similar helper functions (dummy_messages_from_video_url, dummy_messages_from_image_url) have been added to test_video.py and test_vision.py, which are nearly identical. This introduces a new form of code duplication across test files.
To further improve maintainability, consider creating a single, more generic helper function in a shared location like tests/utils.py. This function could take the media type as an argument.
For example, you could have a function in tests/utils.py:
def dummy_messages_from_media_url(
media_type: str,
media_urls: str | list[str],
content_text: str,
):
if isinstance(media_urls, str):
media_urls = [media_urls]
media_key = f"{media_type}_url"
return [
{
"role": "user",
"content": [
*(
{"type": media_key, media_key: {"url": media_url}}
for media_url in media_urls
),
{"type": "text", "text": content_text},
],
}
]Then, in this file, you could use it like this:
from tests.utils import dummy_messages_from_media_url
# ...
messages = dummy_messages_from_media_url(
"audio",
audio_url,
"What's happening in this audio?"
)This would eliminate the duplicated helper functions and make the test suite easier to maintain. The default text can be passed from each test file.
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: bbartels <benjamin@bartels.dev>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk> Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
…sts (vllm-project#26875) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Purpose
Avoid copy pasting the same message over and over again
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.