fix(frontend): always include usage, when configured to do so by max-wittig · Pull Request #20983 · vllm-project/vllm

max-wittig · 2025-07-15T12:12:30Z

Even when stream_options is not specified

This is a follow up from https://github.com/vllm-project/vllm/pull/19695/files where I didn't consider the fact that we also want to include the usage, when no stream options are specified

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

Test Plan

Run vllm with --enable-force-include-usage
Send a normal streaming request to vLLM and observe the usage being included at the end

Test Result

(Optional) Documentation Update

gemini-code-assist

Summary of Changes

Hello @max-wittig, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a fix to ensure that usage statistics are consistently included in the completion API's responses. It addresses a specific scenario where these statistics were omitted when stream_options were not explicitly provided, aligning the behavior with the intended functionality of always reporting usage. This is a follow-up to a previous change to cover this edge case.

Highlights

Completion API: Ensured that usage statistics are always included in the completion response, even when stream_options are not specified, by leveraging the enable_force_include_usage flag.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request correctly ensures that usage information is included in completion streams when stream_options is not specified, by respecting the enable_force_include_usage flag. The change is simple, well-contained, and directly addresses the issue described. The code is clear and I see no issues with it.

github-actions · 2025-07-15T12:50:22Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

max-wittig · 2025-07-17T06:30:23Z

@aarnphm maybe you could take a look. This is a fix for the contribution that I made in #19695. Thank you!

NickLucche

I think you need to edit at least serving_chat and serving_transcriptions too.
Which could make for an opportunity to group the logic into a util function

max-wittig · 2025-07-18T09:57:07Z

@NickLucche Thanks for the review. I will check it!

mergify · 2025-07-18T10:38:07Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @max-wittig.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

max-wittig · 2025-09-12T09:01:28Z

Seems like this now fails without an error message that can be understood, because of #21234. This MR is open since a long time and we keep rebasing and fixing it.

Could you explain how I can see what's wrong or how to fix those bc-lint issues? @zhewenl

/cc @simon-mo

max-wittig · 2025-09-15T07:51:01Z

Seems like bc-lint passes now after rebase.

Ready for re-review @NickLucche

NickLucche · 2025-09-15T10:40:23Z

Re-runnning tests, we might have some issues on CI apologies.

bbartels · 2025-09-15T16:54:32Z

Checks all passed 🟩

max-wittig · 2025-09-17T13:05:11Z

Rebased again. Could somebody please review this one? @NickLucche

NickLucche

I would suggest spinning up a different server with a light model to test out that single feature for now.
Switching fixture to "function" would make the whole test run slower.

NickLucche · 2025-09-17T15:31:02Z

tests/entrypoints/openai/test_transcription_validation.py

+def server(request):
+    if marker := request.node.get_closest_marker("extra_server_args"):
+        SERVER_ARGS.append(marker.args[0])
+


sorry about the delay, I hadn't checked tests thoroughly, I don't think this is actually working as intended.
As the fixture is per-module, it is going to be created once and then re-used for all other tests.
What's happening here is that it's using the first tests args to spawn the server, so it's dependent on the order of tests.

Hi. Thanks for the review! I've tried to add a separate server now, but seems like it cannot start as the primary one is taking all the resources. Any way around this?

(EngineCore_DP0 pid=31416) File "/vllm/vllm/worker/worker_base.py", line 611, in init_device (EngineCore_DP0 pid=31416) self.worker.init_device() # type: ignore (EngineCore_DP0 pid=31416) ^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore_DP0 pid=31416) File "/vllm/vllm/v1/worker/gpu_worker.py", line 181, in init_device (EngineCore_DP0 pid=31416) raise ValueError( (EngineCore_DP0 pid=31416) ValueError: Free memory on device (13.3/139.8 GiB) on startup is less than desired GPU memory utilization (0.9, 125.82 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.

@max-wittig let's try and spawn the module and particularly the newly added function fixture with minimal --gpu-memory-utilization and args.

mergify · 2025-09-19T11:50:21Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @max-wittig.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

NickLucche

Left a few suggestions to try and get these tests running

NickLucche · 2025-09-22T13:13:36Z

tests/entrypoints/openai/test_chat.py

+        "--max-model-len",
+        "2048",
+        "--enforce-eager",
+        "--max-num-seqs",
+        "128",


we're only testing usage so let's spin up the most lightweight server we can.
We can try

--max-model-len 128 --max-num-seqs 1 --gpu-memory-utilization 0.2

I got it working now, but I think I will still have to limit the gpu-memory-utilization of the main server

@NickLucche Hi! Seems like I can't start a second server without running into resource limits. Could you give me any pointers here?

@max-wittig how about we try @pytest.fixture(scope="class") and then group tests into 2 classes, one requirining vanilla server, and the other one requiring the one with flag in this PR?

I've tried it with scope=module now and by putting it in a separate file. Seems cleaner that way in my opinion.

tests/entrypoints/openai/test_chat.py

tests/entrypoints/openai/test_transcription_validation.py

mergify · 2025-10-06T09:25:11Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @max-wittig.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

NickLucche · 2025-10-06T09:26:23Z

I think global format changes have affected this approved PR too https://vllm-dev.slack.com/archives/C07R5Q1Q2BB/p1759663228844749

bbartels · 2025-10-09T11:51:53Z

@max-wittig I appreciate your efforts on this!

max-wittig · 2025-10-09T13:01:19Z

@NickLucche Much nicer formatting now! I rebased again.

mergify · 2025-10-13T11:42:45Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @max-wittig.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Even when stream_options is not specified Signed-off-by: Max Wittig <max.wittig@siemens.com> Signed-off-by: Antoine Auger <antoineauger@users.noreply.github.com>

Signed-off-by: Antoine Auger <antoineauger@users.noreply.github.com>

Signed-off-by: Max Wittig <max.wittig@siemens.com>

max-wittig · 2025-10-13T15:21:37Z

@NickLucche This is ready and passing again!

NickLucche · 2025-10-14T07:17:59Z

Thanks for your patience @max-wittig !

max-wittig · 2025-10-14T07:30:04Z

@NickLucche Thanks for the reviews and the merge!

gemini-code-assist bot reviewed Jul 15, 2025

View reviewed changes

mergify bot added the frontend label Jul 15, 2025

gemini-code-assist bot reviewed Jul 15, 2025

View reviewed changes

max-wittig force-pushed the fix/stream-options-always-usage branch from 658b824 to 9b42f26 Compare July 15, 2025 12:14

max-wittig marked this pull request as ready for review July 15, 2025 14:21

max-wittig requested a review from aarnphm as a code owner July 15, 2025 14:21

NickLucche requested changes Jul 18, 2025

View reviewed changes

max-wittig force-pushed the fix/stream-options-always-usage branch from 9b42f26 to 6f81ce5 Compare July 18, 2025 10:37

mergify bot added the needs-rebase label Jul 18, 2025

max-wittig force-pushed the fix/stream-options-always-usage branch from 6f81ce5 to c6c5ff4 Compare July 18, 2025 10:43

mergify bot removed the needs-rebase label Jul 18, 2025

max-wittig force-pushed the fix/stream-options-always-usage branch 15 times, most recently from c1e0090 to 88021ff Compare July 18, 2025 14:26

max-wittig force-pushed the fix/stream-options-always-usage branch from 0f35c00 to afcfb09 Compare September 12, 2025 08:40

mergify bot removed the needs-rebase label Sep 12, 2025

max-wittig force-pushed the fix/stream-options-always-usage branch 3 times, most recently from b48218c to 1012a74 Compare September 12, 2025 08:55

max-wittig force-pushed the fix/stream-options-always-usage branch from 1012a74 to 3e4ce83 Compare September 15, 2025 07:49

max-wittig force-pushed the fix/stream-options-always-usage branch 2 times, most recently from 6ec1908 to 8f928e7 Compare September 17, 2025 12:54

NickLucche requested changes Sep 17, 2025

View reviewed changes

NickLucche requested changes Sep 22, 2025

View reviewed changes

NickLucche approved these changes Oct 1, 2025

View reviewed changes

max-wittig and others added 3 commits October 13, 2025 16:00

fix(openai): always include usage when server is configured to do so

121ba82

Even when stream_options is not specified Signed-off-by: Max Wittig <max.wittig@siemens.com> Signed-off-by: Antoine Auger <antoineauger@users.noreply.github.com>

test(entrypoints): add extra_server_args marker for server fixture

fda6e8e

Signed-off-by: Antoine Auger <antoineauger@users.noreply.github.com>

refactor(open): use independent server/client

d2f2425

Signed-off-by: Max Wittig <max.wittig@siemens.com>

max-wittig mentioned this pull request Oct 15, 2025

chore: remove unused marker #26890

Merged

5 tasks

max-wittig mentioned this pull request Nov 21, 2025

[Feat][CLI] enforce-include-usage #19695

Merged

4 tasks

Uh oh!

Conversation

max-wittig commented Jul 15, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Essential Elements of an Effective PR Description Checklist

Purpose

Test Plan

Test Result

(Optional) Documentation Update

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

github-actions bot commented Jul 15, 2025

Uh oh!

max-wittig commented Jul 17, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

max-wittig commented Jul 18, 2025

Uh oh!

mergify bot commented Jul 18, 2025

Uh oh!

max-wittig commented Sep 12, 2025

Uh oh!

max-wittig commented Sep 15, 2025

Uh oh!

NickLucche commented Sep 15, 2025

Uh oh!

bbartels commented Sep 15, 2025

Uh oh!

max-wittig commented Sep 17, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented Sep 19, 2025

Uh oh!

NickLucche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Oct 6, 2025

Uh oh!

NickLucche commented Oct 6, 2025

Uh oh!

bbartels commented Oct 9, 2025

Uh oh!

max-wittig commented Oct 9, 2025

Uh oh!

mergify bot commented Oct 13, 2025

Uh oh!

max-wittig commented Oct 13, 2025

Uh oh!

NickLucche commented Oct 14, 2025

max-wittig commented Jul 15, 2025 •

edited by github-actions bot

Loading