[Bug]: Streaming=True causes missing or scrambled tokens with GPT-OSS 120B on vLLM v0.11.0

[cxz1418]

Your current environment

| NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 |

• vLLM versions: v0.10.2 and v0.11.0
• Model: GPT-OSS 120B
• Streaming: True
• --enforce-eager: Tested

🐛 Describe the bug

Issue:
When using Streaming=True, some tokens are missing or arrive in scrambled order.

• With Streaming=False, all tokens are generated correctly.
• Using --enforce-eager produces the correct token sequence but significantly slows down generation.

This issue occurs in both v0.10.2 and v0.11.0.

Expected behavior:
Streaming should produce all tokens in the correct order, similar to --enforce-eager, without the performance penalty.

Steps to reproduce:

1. Run GPT-OSS 120B with vLLM v0.11.0 (or v0.10.2)
2. Enable streaming (Streaming=True)
3. Generate text and observe missing or scrambled tokens

Additional notes:
The problem appears to be specific to asynchronous streaming. Using eager execution ensures correct token order but reduces speed.

Command Example

CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve openai/gpt-oss-120b \
   --tensor-parallel-size 4 \
   --enable-expert-parallel \
   --tool-call-parser openai \
   --reasoning-parser openai_gptoss \
   --enable-auto-tool-choice \
   --async-scheduling \
   --max-model-len 131072 \
   --gpu-memory-utilization 0.90 \
   --max-num-seqs 32 \
   --host 0.0.0.0 \
   --max-num-batched-tokens 8192 \
   --port 20003

---

[robertgshaw2-redhat]

Thanks for this report. Do you have any client side that consistent reproduces this?

[njhill]

@cxz1418 I notice that you are specifying --async-scheduling, have you been able to reproduce this without that flag?

[cxz1418]

@robertgshaw2-redhat
Yes, We tested it from multiple clients, but the same issue occurs where some tokens are missing in the output.
@njhill
Disabling --async-scheduling also results in the same behavior.

[cxz1418]

Would streaming work normally for other users?
We are also wondering if updating our drivers might resolve the issue.

[otkeri]

I've also performed numerous tests to address this issue.

It doesn't occur in non-streaming mode. However, it does occur in streaming mode.

So, I tested inference with another LLM engine that supports streaming mode, and token dropouts don't occur in that engine.

[cxz1418]

@robertgshaw2-redhat @njhill
That's really strange…
I'm running experiments on another server, and on A100 40G × 8, the streaming works normally.
| NVIDIA-SMI 560.35.05 Driver Version: 560.35.05 CUDA Version: 12.6 |

Could the different driver be causing this issue?

[danladis]

I have the same problem. Both with and without the --async-scheduling flag. Also with older and the newest version (v0.11.0)

[sebastianhorwege]

@danladis same here

[bbrowning]

I've been unable to reproduce this so far on my DGX Spark node in streaming with CUDA 13.0. Can anyone that has observed this reproduce this with a single request? Does it take a certain level of concurrency? There are a few reports here of tokens missing and/or out of order and some reports of things working fine, all with streaming. If it's consistently reproducible on a specific set of hardware, vLLM versions, and specific request that would all help reproduce and track this down.

[njhill]

What I assume is the same issue seems to occur reliably in the CI of this PR which enables async scheduling: https://buildkite.com/vllm/ci/builds/39507#019a978c-72cb-41db-9021-e25066a66f4c.

However I've been unable to repro at all myself running the same branch/test on a separate H100. It feels like a timing / race condition type thing, where the timing/sequencing could be indirectly affected by the particular CPU/GPU speed and/or use of async scheduling. But that's just a fairly vague guess.

[robertgshaw2-redhat]

What I assume is the same issue seems to occur reliably in the CI of this PR which enables async scheduling: https://buildkite.com/vllm/ci/builds/39507#019a978c-72cb-41db-9021-e25066a66f4c.

However I've been unable to repro at all myself running the same branch/test on a separate H100. It feels like a timing / race condition type thing, where the timing/sequencing could be indirectly affected by the particular CPU/GPU speed and/or use of async scheduling. But that's just a fairly vague guess.

Im struggling a bit with * where * the race condition could be and * why * it only happens with gpt-oss. I would think that if we had some sort of systematic issue that it would pop up for other models too. Maybe something in harmony/reponses api?

To everyone in this thread --- are you using /v1/chat/completions or are you using responses?

[robertgshaw2-redhat]

@danladis @sebastianhorwege @cxz1418

[sebastianhorwege]

For me it's responses api

[njhill]

Yes I assume it's harmony/responses specific