Skip to content

[Misc] Use helper function to generate dummy messages in OpenAI MM tests#26875

Merged
DarkLight1337 merged 3 commits intovllm-project:mainfrom
DarkLight1337:dummy-messages
Oct 15, 2025
Merged

[Misc] Use helper function to generate dummy messages in OpenAI MM tests#26875
DarkLight1337 merged 3 commits intovllm-project:mainfrom
DarkLight1337:dummy-messages

Conversation

@DarkLight1337
Copy link
Copy Markdown
Member

@DarkLight1337 DarkLight1337 commented Oct 15, 2025

Purpose

Avoid copy pasting the same message over and over again

Test Plan

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337 DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 15, 2025
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively refactors the multi-modal tests for audio, video, and vision to use helper functions for generating dummy messages, which significantly reduces code duplication. The implementation is clean and improves maintainability. However, I've found a critical issue in tests/entrypoints/openai/test_video.py where a test case was incorrectly refactored, leading to it being broken. Please address this issue.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Copy Markdown
Member Author

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors multi-modal tests for OpenAI entrypoints by introducing helper functions to generate dummy messages, which successfully reduces code duplication within each test file. The changes are generally good and improve readability. I've pointed out a minor inconsistency in the test data for video tests that could break assertions. I've also suggested a further refactoring to consolidate the new, very similar helper functions into a single generic function to improve long-term maintainability and fully adhere to the DRY principle.

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
@DarkLight1337
Copy link
Copy Markdown
Member Author

/gemini review

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a good step towards improving the test suite by refactoring duplicated code for creating dummy messages into helper functions. This has been applied to tests for audio, video, and vision modalities. The changes make the individual tests cleaner and easier to read.

My main feedback is that while this removes duplication within each test file, it introduces very similar helper functions across three different files (test_audio.py, test_video.py, test_vision.py). I've left a comment suggesting a further refactoring to a single, generic helper function in a shared utility file (e.g., tests/utils.py) to eliminate this new duplication and further improve the maintainability of the codebase.

Comment on lines +56 to 74
def dummy_messages_from_audio_url(
audio_urls: str | list[str],
content_text: str = "What's happening in this audio?",
):
messages = [
if isinstance(audio_urls, str):
audio_urls = [audio_urls]

return [
{
"role": "user",
"content": [
{"type": "audio_url", "audio_url": {"url": audio_url}},
{"type": "text", "text": "What's happening in this audio?"},
*(
{"type": "audio_url", "audio_url": {"url": audio_url}}
for audio_url in audio_urls
),
{"type": "text", "text": content_text},
],
}
]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While this helper function is a good step towards reducing code duplication within this file, similar helper functions (dummy_messages_from_video_url, dummy_messages_from_image_url) have been added to test_video.py and test_vision.py, which are nearly identical. This introduces a new form of code duplication across test files.

To further improve maintainability, consider creating a single, more generic helper function in a shared location like tests/utils.py. This function could take the media type as an argument.

For example, you could have a function in tests/utils.py:

def dummy_messages_from_media_url(
    media_type: str,
    media_urls: str | list[str],
    content_text: str,
):
    if isinstance(media_urls, str):
        media_urls = [media_urls]

    media_key = f"{media_type}_url"
    return [
        {
            "role": "user",
            "content": [
                *(
                    {"type": media_key, media_key: {"url": media_url}}
                    for media_url in media_urls
                ),
                {"type": "text", "text": content_text},
            ],
        }
    ]

Then, in this file, you could use it like this:

from tests.utils import dummy_messages_from_media_url

# ...

messages = dummy_messages_from_media_url(
    "audio",
    audio_url,
    "What's happening in this audio?"
)

This would eliminate the duplicated helper functions and make the test suite easier to maintain. The default text can be passed from each test file.

@Isotr0py Isotr0py enabled auto-merge (squash) October 15, 2025 04:57
@DarkLight1337 DarkLight1337 enabled auto-merge (squash) October 15, 2025 04:58
@DarkLight1337 DarkLight1337 merged commit b8a4572 into vllm-project:main Oct 15, 2025
21 checks passed
@DarkLight1337 DarkLight1337 deleted the dummy-messages branch October 15, 2025 07:17
bbartels pushed a commit to bbartels/vllm that referenced this pull request Oct 16, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: bbartels <benjamin@bartels.dev>
albertoperdomo2 pushed a commit to albertoperdomo2/vllm that referenced this pull request Oct 16, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
lywa1998 pushed a commit to lywa1998/vllm that referenced this pull request Oct 20, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
alhridoy pushed a commit to alhridoy/vllm that referenced this pull request Oct 24, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
0xrushi pushed a commit to 0xrushi/vllm that referenced this pull request Oct 26, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: 0xrushi <6279035+0xrushi@users.noreply.github.com>
rtourgeman pushed a commit to rtourgeman/vllm that referenced this pull request Nov 10, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
devpatelio pushed a commit to SumanthRH/vllm that referenced this pull request Nov 29, 2025
…sts (vllm-project#26875)

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants