[Bugfix] Omit null `annotations` and `usage.prompt_tokens_details` in chat completions by vadiklyutiy · Pull Request #64177 · vllm-project/vllm

vadiklyutiy · 2026-10-28T01:36:33Z

Overview

On a default server, a non-streaming chat completion returns "annotations": null in each message and "prompt_tokens_details": null in usage. The OpenAI spec makes both fields optional but not nullable, so strict schema validators reject the reply. This PR leaves both keys out when they are None. It also types annotations as a list, as the spec and the OpenAI SDK do. Fixes vllm-project/vllm issue 61425.

Another PR, #61431, which fixes the same issue, has had no activity for two weeks.

Claims

Non-streaming chat completions no longer send annotations: null or usage.prompt_tokens_details: null. A field that has a value is still sent.
ChatMessage.annotations is now list[OpenAIAnnotation] | None, the same as the OpenAI spec and SDK. Before, a real value could not be sent in the right shape.
/v1/completions, and any other response that uses UsageInfo, also stops sending prompt_tokens_details: null.

Validation

Setup: 1 x NVIDIA B200. vLLM commit 77154ca6116ae34cbfe314beef1016bde9b63989 (0.32.0rc1.dev100+g77154ca61). Model Qwen/Qwen3-0.6B. Default server flags.

Reproducer:

vllm serve Qwen/Qwen3-0.6B --port 18061

curl -s http://127.0.0.1:18061/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Reply READY."}],"max_tokens":16,"temperature":0}' \
| python3 -c 'import json,sys; d=json.load(sys.stdin); print("CHECK", json.dumps({"annotations": d["choices"][0]["message"].get("annotations","<absent>"), "prompt_tokens_details": d["usage"].get("prompt_tokens_details","<absent>")}))'

Before: 3 of 3 runs on this commit showed the bug. It also showed in 3 of 3 runs with the vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 image.

"message":{"role":"assistant","content":"<think>\nOkay, the user said \"Reply READY.\" I need to respond appropriately","refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":null}
"usage":{"prompt_tokens":11,"total_tokens":27,"completion_tokens":16,"prompt_tokens_details":null,"completion_tokens_details":null}
CHECK {"annotations": null, "prompt_tokens_details": null}

After: 6 of 6 runs showed the fix, each on a fresh server. I made 3 of them, and a separate re-check made the other 3.

"message":{"role":"assistant","content":"<think>\nOkay, the user said \"Reply READY.\" I need to respond appropriately","refusal":null,"audio":null,"function_call":null,"reasoning":null}
"usage":{"prompt_tokens":11,"total_tokens":27,"completion_tokens":16,"completion_tokens_details":null}
CHECK {"annotations": "<absent>", "prompt_tokens_details": "<absent>"}

The model text did not change. Only the two keys changed. One more server start with the fix failed because a GPU test running at the same time held the GPU memory. That run sent no request.

Other checks on a live server with the fix:

/v1/completions, non-streaming: "usage":{"prompt_tokens":1,"total_tokens":5,"completion_tokens":4,"completion_tokens_details":null}.
Streaming chat with include_usage: the last chunk has "usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}. This path already used exclude_none=True before the fix.
With --enable-prompt-tokens-details: CHECK {"annotations": "<absent>", "prompt_tokens_details": {"cached_tokens": 0, "created_cache_tokens": 0, "multimodal_tokens": null}}. The field is still sent when it has a value.

Tests:

python -m pytest -v tests/entrypoints/openai/test_tool_choice_content_none.py tests/entrypoints/openai/completion/test_completion_text_offset.py

With the fix: 10 passed. With the two old protocol files put back: 3 failed, 7 passed. The three failures are the three new tests.
Existing tests with the fix: chat_completion/test_serving_chat.py -k "not TestGPTOSS" 181 passed, completion/test_completion_error.py 46 passed, anthropic/test_anthropic_messages_conversion.py 108 passed, cohere/test_cohere_chat_message.py 9 passed, openai/test_tool_calls_serialization.py 6 passed.
Not run: the TestGPTOSS* classes, which start a real openai/gpt-oss-20b server. Also not run: cohere/test_serving_conversion.py and cohere/test_serving_streaming.py, because the cohere package is not installed in my environment.
pre-commit run --files on the four changed files: all hooks pass.
Both test files already run in Buildkite (.buildkite/test_areas/entrypoints.yaml).

Details

Root cause: the non-streaming chat route returns JSONResponse(content=result.model_dump()) (chat_completion/api_router.py). A plain model_dump() writes every None field as null.

ChatMessage.annotations defaults to None, and nothing in chat completions sets it. It was also typed as one Annotation, but the spec and the SDK use a list.
UsageInfo.prompt_tokens_details is None whenever --enable-prompt-tokens-details is off.

ChatMessage._serialize drops an empty tool_calls, but nothing dropped these two fields. exclude_unset=True would not help either. serving.py assigns usage.prompt_tokens_details after it builds UsageInfo, so the field counts as set even when it is None.

Spec (openai-openapi openapi.yaml 2.3.0): ChatCompletionResponseMessage.annotations is type: array, and CompletionUsage.prompt_tokens_details is type: object. Both are optional and not nullable. refusal, audio and function_call are nullable, so they stay as they are.

Fix: Field(default=None, exclude_if=lambda value: value is None) on the two fields. vLLM already uses this pattern for ResponsesResponse.metrics (responses/protocol.py). It applies to both model_dump() and model_dump_json(), so every route that dumps these models gets the fix. exclude_if needs pydantic 2.11 or later, and vLLM requires 2.12 or later. The fields stay on the models, so code that reads usage.prompt_tokens_details as an attribute (the Anthropic and Cohere layers) does not change.

Options I did not choose:

model_dump(exclude_none=True) in the router. It also drops content: null and logprobs: null, and the spec requires those keys. It would also fix only one router.
[] and {} as defaults. PromptTokenUsageInfo() dumps cached_tokens: null, which the spec does not allow either. An empty details object would also claim details that --enable-prompt-tokens-details is meant to turn on.

Points for review:

usage.completion_tokens_details is still null, and the spec has the same rule for it. I left it out because the issue does not ask for it. Also, test_streaming_reasoning_usage_counts_across_deltas in test_serving_chat.py asserts model_dump()["completion_tokens_details"] == expected_details, where expected_details is None when reasoning details are off. The same change on that field makes this test fail with KeyError. A maintainer should decide whether that test should change.
Typing annotations as a list is a small break for outside Python code that passes one Annotation object to ChatMessage. Nothing in vllm/ sets this field, and that input never gave a reply that the spec or the SDK accepts.
The UsageInfo change also removes prompt_tokens_details: null from completions, pooling, scoring and transcription responses.
Sample outputs in docs/models/pooling_models/classify.md, docs/models/pooling_models/embed.md and examples/features/openai_batch/README.md still show "prompt_tokens_details": null. They are old captured outputs, and I left them unchanged.
The served /openapi.json still marks both fields as nullable, because pydantic does not put exclude_if into the schema.
With --enable-prompt-tokens-details, the details object still has vLLM-only keys set to null (for example multimodal_tokens). That is outside this fix.

Pull Request Checklist

I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).
AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

… chat completions  ## Overview On a default server, a non-streaming chat completion returns `"annotations": null` in each message and `"prompt_tokens_details": null` in `usage`. The OpenAI spec makes both fields optional but not nullable, so strict schema validators reject the reply. This PR leaves both keys out when they are `None`. It also types `annotations` as a list, as the spec and the OpenAI SDK do. Fixes vllm-project/vllm issue 61425. Compared with vllm-project#61431, this fix: - Has a test that pins the `annotations` list type named in the issue. I put the old single-`Annotation` type back into vllm-project#61431's tree and ran the tests of both PRs. Only our `test_chat_completion_response_keeps_annotations_and_prompt_tokens_details` failed. vllm-project#61431's new test also passes on the old code. - Tests the `/v1/completions` path, which uses the same `UsageInfo`, through the real `request_output_to_completion_response`. vllm-project#61431 tests only the chat dump. ## Claims - Non-streaming chat completions no longer send `annotations: null` or `usage.prompt_tokens_details: null`. A field that has a value is still sent. - `ChatMessage.annotations` is now `list[OpenAIAnnotation] | None`, the same as the OpenAI spec and SDK. Before, a real value could not be sent in the right shape. - `/v1/completions`, and any other response that uses `UsageInfo`, also stops sending `prompt_tokens_details: null`. ## Validation Setup: 1 x NVIDIA B200. vLLM commit `77154ca6116ae34cbfe314beef1016bde9b63989` (`0.32.0rc1.dev100+g77154ca61`). Model `Qwen/Qwen3-0.6B`. Default server flags. Reproducer: ```bash vllm serve Qwen/Qwen3-0.6B --port 18061 ``` ```bash curl -s http://127.0.0.1:18061/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Reply READY."}],"max_tokens":16,"temperature":0}' \ | python3 -c 'import json,sys; d=json.load(sys.stdin); print("CHECK", json.dumps({"annotations": d["choices"][0]["message"].get("annotations","<absent>"), "prompt_tokens_details": d["usage"].get("prompt_tokens_details","<absent>")}))' ``` Before: 3 of 3 runs on this commit showed the bug. It also showed in 3 of 3 runs with the `vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989` image. ``` "message":{"role":"assistant","content":"<think>\nOkay, the user said \"Reply READY.\" I need to respond appropriately","refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":null} "usage":{"prompt_tokens":11,"total_tokens":27,"completion_tokens":16,"prompt_tokens_details":null,"completion_tokens_details":null} CHECK {"annotations": null, "prompt_tokens_details": null} ``` After: 6 of 6 runs showed the fix, each on a fresh server. I made 3 of them, and a separate re-check made the other 3. ``` "message":{"role":"assistant","content":"<think>\nOkay, the user said \"Reply READY.\" I need to respond appropriately","refusal":null,"audio":null,"function_call":null,"reasoning":null} "usage":{"prompt_tokens":11,"total_tokens":27,"completion_tokens":16,"completion_tokens_details":null} CHECK {"annotations": "<absent>", "prompt_tokens_details": "<absent>"} ``` The model text did not change. Only the two keys changed. One more server start with the fix failed because a GPU test running at the same time held the GPU memory. That run sent no request. Other checks on a live server with the fix: - `/v1/completions`, non-streaming: `"usage":{"prompt_tokens":1,"total_tokens":5,"completion_tokens":4,"completion_tokens_details":null}`. - Streaming chat with `include_usage`: the last chunk has `"usage":{"prompt_tokens":11,"total_tokens":15,"completion_tokens":4}`. This path already used `exclude_none=True` before the fix. - With `--enable-prompt-tokens-details`: `CHECK {"annotations": "<absent>", "prompt_tokens_details": {"cached_tokens": 0, "created_cache_tokens": 0, "multimodal_tokens": null}}`. The field is still sent when it has a value. Tests: ```bash python -m pytest -v tests/entrypoints/openai/test_tool_choice_content_none.py tests/entrypoints/openai/completion/test_completion_text_offset.py ``` - With the fix: `10 passed`. With the two old protocol files put back: `3 failed, 7 passed`. The three failures are the three new tests. - Existing tests with the fix: `chat_completion/test_serving_chat.py -k "not TestGPTOSS"` 181 passed, `completion/test_completion_error.py` 46 passed, `anthropic/test_anthropic_messages_conversion.py` 108 passed, `cohere/test_cohere_chat_message.py` 9 passed, `openai/test_tool_calls_serialization.py` 6 passed. - Not run: the `TestGPTOSS*` classes, which start a real `openai/gpt-oss-20b` server. Also not run: `cohere/test_serving_conversion.py` and `cohere/test_serving_streaming.py`, because the `cohere` package is not installed in my environment. - `pre-commit run --files` on the four changed files: all hooks pass. - Both test files already run in Buildkite (`.buildkite/test_areas/entrypoints.yaml`). ## Details Root cause: the non-streaming chat route returns `JSONResponse(content=result.model_dump())` (`chat_completion/api_router.py`). A plain `model_dump()` writes every `None` field as `null`. - `ChatMessage.annotations` defaults to `None`, and nothing in chat completions sets it. It was also typed as one `Annotation`, but the spec and the SDK use a list. - `UsageInfo.prompt_tokens_details` is `None` whenever `--enable-prompt-tokens-details` is off. `ChatMessage._serialize` drops an empty `tool_calls`, but nothing dropped these two fields. `exclude_unset=True` would not help either. `serving.py` assigns `usage.prompt_tokens_details` after it builds `UsageInfo`, so the field counts as set even when it is `None`. Spec (openai-openapi `openapi.yaml` 2.3.0): `ChatCompletionResponseMessage.annotations` is `type: array`, and `CompletionUsage.prompt_tokens_details` is `type: object`. Both are optional and not nullable. `refusal`, `audio` and `function_call` are nullable, so they stay as they are. Fix: `Field(default=None, exclude_if=lambda value: value is None)` on the two fields. vLLM already uses this pattern for `ResponsesResponse.metrics` (`responses/protocol.py`). It applies to both `model_dump()` and `model_dump_json()`, so every route that dumps these models gets the fix. `exclude_if` needs pydantic 2.11 or later, and vLLM requires 2.12 or later. The fields stay on the models, so code that reads `usage.prompt_tokens_details` as an attribute (the Anthropic and Cohere layers) does not change. Options I did not choose: - `model_dump(exclude_none=True)` in the router. It also drops `content: null` and `logprobs: null`, and the spec requires those keys. It would also fix only one router. - `[]` and `{}` as defaults. `PromptTokenUsageInfo()` dumps `cached_tokens: null`, which the spec does not allow either. An empty details object would also claim details that `--enable-prompt-tokens-details` is meant to turn on. Points for review: - `usage.completion_tokens_details` is still `null`, and the spec has the same rule for it. I left it out because the issue does not ask for it. Also, `test_streaming_reasoning_usage_counts_across_deltas` in `test_serving_chat.py` asserts `model_dump()["completion_tokens_details"] == expected_details`, where `expected_details` is `None` when reasoning details are off. The same change on that field makes this test fail with `KeyError`. A maintainer should decide whether that test should change. - Typing `annotations` as a list is a small break for outside Python code that passes one `Annotation` object to `ChatMessage`. Nothing in `vllm/` sets this field, and that input never gave a reply that the spec or the SDK accepts. - The `UsageInfo` change also removes `prompt_tokens_details: null` from completions, pooling, scoring and transcription responses. - Sample outputs in `docs/models/pooling_models/classify.md`, `docs/models/pooling_models/embed.md` and `examples/features/openai_batch/README.md` still show `"prompt_tokens_details": null`. They are old captured outputs, and I left them unchanged. - The served `/openapi.json` still marks both fields as nullable, because pydantic does not put `exclude_if` into the schema. - With `--enable-prompt-tokens-details`, the details object still has vLLM-only keys set to `null` (for example `multimodal_tokens`). That is outside this fix. --- <details> <summary> Pull Request Checklist </summary> - [x] I used vLLM's `/pr-checklist` skill. (Mandatory for agents, optional for humans). - [x] AI assistance was used during the creation of this PR. - [ ] **Design Fit:** Minimizes impact on core components, reuses existing functionality, and justifies added complexity. - [ ] **Testing and Validation:** Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI. - [ ] **Code Quality and Style:** Keeps code and comments clear and concise, and updates relevant documentation and examples. - [ ] **Pull Request Contents:** Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide. </details> Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

DarkLight1337 · 2026-10-28T04:32:51Z

/ci run

github-actions · 2026-10-28T04:34:55Z

✅ Triggered Buildkite CI #97868 for commit 344d8ac10963.

Python's `UsageInfo.prompt_tokens_details` is now left out when unset instead of serialized as `null` (vllm-project#64177): the OpenAI spec types it as an object, not nullable. `Usage` is the one wire type every route reports, so chat, completions and generate, streamed or not, omit it when `--enable-prompt-tokens-details` is off. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: richard-for-bugen[bot] <342439978+richard-for-bugen[bot]@users.noreply.github.com>

Python's response `ChatMessage.annotations` is now left out when unset instead of serialized as `null` (vllm-project#64177): the OpenAI spec types it as an array, not nullable. The Rust frontend never produces annotations, so non-streaming chat messages no longer carry the key. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: richard-for-bugen[bot] <342439978+richard-for-bugen[bot]@users.noreply.github.com>

vadiklyutiy requested review from DarkLight1337, chaunceyjiang and robertgshaw2-redhat as code owners October 28, 2026 01:36

claude Bot reviewed Oct 28, 2026

View reviewed changes

vadiklyutiy requested review from AndreasKaratzas, NickLucche, aarnphm and njhill as code owners October 28, 2026 01:36

mergify Bot added frontend tool-calling bug Something isn't working labels Oct 28, 2026

github-project-automation Bot added this to Tool Calling (arpera) and Tool Calling Oct 28, 2026

github-project-automation Bot moved this to Unsorted queue in Tool Calling (arpera) Oct 28, 2026

github-project-automation Bot moved this to Unsorted queue in Tool Calling Oct 28, 2026

DarkLight1337 approved these changes Oct 28, 2026

View reviewed changes

Merge branch 'main' into lyutiydoctor/20261015-015726-vllm-61425

344d8ac

DarkLight1337 enabled auto-merge (squash) October 28, 2026 04:32

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Oct 28, 2026

DarkLight1337 merged commit f60edcc into vllm-project:main Oct 28, 2026
79 checks passed

richard-for-bugen Bot mentioned this pull request Oct 28, 2026

[Rust Frontend] Omit unset usage prompt_tokens_details and message annotations like Python BugenZhao/vllm#73

Open

BugenZhao mentioned this pull request Nov 3, 2026

[Rust Frontend] Omit unset usage prompt_tokens_details and message annotations like Python #65374

Merged

6 tasks done

fank mentioned this pull request Nov 3, 2026

fix(frontend): backport omission of null usage details (vllm#64177) enthus-appdev/vllm-pcai#82

Closed

Navigation Menu

Repository navigation

[Bugfix] Omit null `annotations` and `usage.prompt_tokens_details` in chat completions - #64177

[Bugfix] Omit null `annotations` and `usage.prompt_tokens_details` in chat completions#64177
DarkLight1337 merged 2 commits into
vllm-project:mainfrom
vadiklyutiy:lyutiydoctor/20261015-015726-vllm-61425

vadiklyutiy commented Oct 28, 2026

Uh oh!

claude Bot left a comment

Uh oh!

DarkLight1337 commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vadiklyutiy commented Oct 28, 2026

Overview

Claims

Validation

Details

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

DarkLight1337 commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants