[Bugfix] Fix Responses API harmony streaming: token splitting, missing done events, nested sequence_number

[Pradyun92]

Purpose

Three fixes for Responses API streaming with harmony models (e.g., gpt_oss):

Bug 1: Multi-token RequestOutput loses intermediate channel transitions

With Eagle speculative decoding, RequestOutput objects contain multiple token_ids. In _generate_with_builtin_tools, the entire multi-token output is passed to StreamingHarmonyContext.append_output(), which processes all tokens in a loop but only yields the context once. If the batch crosses channel boundaries (e.g., reasoning → function call), intermediate channel transitions and their content are lost.

Fix: In _generate_with_builtin_tools (engine/serving.py), split multi-token RequestOutput objects into single-token ones for StreamingHarmonyContext before calling append_output(). Each single-token output gets its own yield context, ensuring the harmony parser processes tokens one at a time.

Bug 2: Final message done events not emitted

In _process_harmony_streaming_events, done events (`response.output_text.done`, `response.content_part.done`, `response.output_item.done`) are only emitted when the next message starts (`is_expecting_start()`). The last message never triggers this because the `async for` loop ends first.

Symptom: Streamed content is truncated — e.g., streaming gives `"2 + 2 equals"` but `response.completed` has `"2 + 2 equals 4."`.

Fix: After the `async for` loop in `_process_harmony_streaming_events`, emit done events for the final message.

Bug 3: `sequence_number=-1` in nested response items

Events in `streaming_events.py` are created with placeholder `sequence_number=-1`. `_increment_sequence_number_and_return` fixes the top-level event but not nested items (e.g., `ResponseFunctionToolCall` inside `ResponseOutputItemDoneEvent.item`).

Fix: Also set `sequence_number` on nested `item` attributes.

Not duplicating existing PRs: #36445 is about non-harmony models. No other open PRs target these specific issues.

AI assistance: This PR was developed with AI assistance (Claude). The submitter has reviewed all changes and tested end-to-end.

Test Plan

```bash

Start vLLM with a harmony model + Eagle speculative decoding

python -m vllm.entrypoints.openai.api_server --model

Test streaming text (checks for last-token loss)

curl -X POST http://localhost:8000/v1/responses
-H "Content-Type: application/json"
-d '{"model": "", "stream": true, "input": "What is 2+2? Answer briefly."}'

Test tool call (checks for sequence_number=-1 and channel transitions)

curl -X POST http://localhost:8000/v1/responses
-H "Content-Type: application/json"
-d '{"model": "", "stream": true, "input": "What is the weather in Tokyo?",
"tools": [{"type": "function", "name": "get_weather", "description": "Get weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}]}'
```

Test Result

Bug 1 — Before: With Eagle, multi-token batches crossing channel boundaries lose intermediate content.
Bug 1 — After: Each token processed individually; all channel transitions preserved.

Bug 2 — Before: Streaming gives `"2 + 2 equals"`, completed gives `"2 + 2 equals 4."` — last tokens lost.
Bug 2 — After: Streamed content matches completed content exactly.

Bug 3 — Before: `ResponseFunctionToolCall` nested in `response.output_item.done` has `sequence_number: -1`.
Bug 3 — After: Nested item carries the correct sequence number.

Tested with gpt-oss-120b model across all scenarios.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR
• The test plan
• The test results
• (Optional) Documentation update
• (Optional) Release notes update

[gemini-code-assist]

Code Review

This pull request introduces two important fixes for streaming with harmony models in the Responses API. The first fix correctly addresses an issue where done events were not being emitted for the final message in a stream by adding a post-loop cleanup logic, which is a robust solution. The second fix resolves a bug where nested items in streaming events retained an incorrect placeholder sequence_number, by ensuring the sequence number is propagated to these nested items. The changes are well-targeted, clear, and improve the correctness of the streaming API. I have reviewed the code and found no issues.

[mergify]

Hi @Pradyun92, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @Pradyun92, the pre-commit checks have failed. Please run:

uv pip install pre-commit
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Pradyun92.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork