[Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams

[positive666]

Summary

Chat streaming recounts reasoning tokens from the full generated history on every delta, even when continuous usage is disabled. For Qwen's parser with thinking disabled, this falls back to repeatedly decoding and parsing the entire history.

Count once per nonempty choice before final usage and middleware metadata. Preserve per-delta counting for continuous/forced usage, and skip accounting-only work when reasoning usage details are disabled. No changes to parser counting rules, sampling, or model computation.

Results

Historical A/B, pinned source 114abd1c; one L40S, TP1, existing Qwen3.8-27B-FP8 model directory (config: Qwen3_5ForConditionalGeneration). Same environment and default compile/CUDA graphs for both versions. Reverse-order repeat:

Output workload	API CPU seconds: before → after	Reduction	Output tok/s: before → after
8 requests × 2,048 tokens	21.45 → 3.34	84.4%	184.23 → 184.25
1 request × 8,192 tokens	46.81 → 5.87	87.5%	25.177 → 25.182

CPU seconds measure API-process user+system time, not request wall time. No meaningful model-throughput gain was observed. Requests used temperature=0, seed=42 and ignore_eos for fixed lengths; this is serving validation, not a task-accuracy benchmark.

All single-request scenarios matched token IDs, text and usage. Replaying the same 25,344 model tokens through both frontends produced identical SSE and final usage in all five scenarios.

Tests

Branch is merged with current main (1596fa5f). The serving hunk still matches HEAD: per-delta count_reasoning_tokens only when continuous/forced usage is on; otherwise one recount per nonempty choice before final usage.

Linux CPU re-run on GitHub main 1c3ef2ad plus this patch (CUDA_VISIBLE_DEVICES empty, VLLM_TARGET_DEVICE=cpu):

python3 -m pytest tests/entrypoints/openai/chat_completion/test_serving_chat.py \
  -k streaming_reasoning_usage -q --tb=line

• 119 passed, 47 deselected.
• 80 parametrized recount combinations (n, details on/off, continuous/forced/none).
• 39 lifecycle cases (13 × 3 usage modes): empty generator/outputs/finish/prefill, one empty choice, uneven finish, empty final delta, ignore finished choice, exception before/after, finish_reason=error, cancel, and aclose.
• Baseline still over-counts on the 64 excess-recount combinations; this patch reduces those to one call per nonempty choice when continuous usage is off.

Related work

No same-scope open fix. #55216 optimizes membership checks inside reasoning extraction, not usage recount frequency. #44398 is an older reasoning-usage feature with a final-counting precedent; this patch optimizes the already-merged implementation while preserving continuous usage. #54238/#54585/#50240 change parser counting rules, not how often serving recounts. #56129 covers the reasoning field round-trip, not usage accounting.

AI assistance was used for implementation and validation. The submitter reviewed the changes and reran the relevant tests.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[positive666]

Follow-up on current main:

• Merged main (1596fa5f) so the serving hunk applies on HEAD without changing parser counting rules.
• Added in-tree lifecycle coverage for continuous / final / none usage, including empty choices, cancel, aclose, engine exceptions, and finish_reason=error.
• Linux CPU pytest (CUDA_VISIBLE_DEVICES empty) on GitHub main 1c3ef2ad + this patch: tests/entrypoints/openai/chat_completion/test_serving_chat.py -k streaming_reasoning_usage → 119 passed, 47 deselected (80 recount combinations + 39 lifecycle cases).

Happy to rebase again if review wants a different test split.