[bench] Take each request's prompt length from the server

[nvcnvn]

Motivation

RequestFuncOutput.prompt_len is copied from the dataset row
(RequestFuncOutput.init_new). One row is one request for single-turn datasets,
so that is fine. But a multi-turn row is replayed as one request per round, and
wrap_multi_turn_request_func replaces only the prompt — so all N rounds report
the row's single value.

Anything that sums prompt_len is then wrong. Two things do: the
--cache-report hit rate, and input_lens in the JSON output.

On an agentic-trace run of 16 conversations x 10 turns:

--cache-report:  Cache hit rate: 12.4%     (denominator 18,951,900)
engine counters: hit rate       83.8%      (prefill_effective_tokens_total: 2,902,144)

The denominator is 6.5x too large. agentic-trace is worst because its loader
does not tokenize at all — it reads the whole trajectory's token count out of the
trace file (its own comment: "Informational only: multi-turn replay ignores
per-row prompt_len"). generated-shared-prefix is affected too, at
--gsp-num-turns > 1 and under --gsp-fast-prepare, which sets prompt_len = 1.

Modifications

Take the prompt length from the server, which is the only side that knows it:
usage.prompt_tokens on the OpenAI-compatible routes, meta_info.prompt_tokens
on the native one. Both parse sites already read that block for
completion_tokens. Where the server reports nothing, the existing value stays.

A streamed chat response only carries usage when the request sets
stream_options.include_usage (or the server runs with
--stream-response-default-include-usage, default off). So the multi-turn
wrapper now asks for it on each round. It is scoped to the wrapper: single-turn
benchmarks send exactly the same payload as before.

What changes in the output: only --cache-report and input_lens in the
JSON read RequestFuncOutput.prompt_len. The headline Total input tokens and
input/total throughput come from input_requests[i].prompt_len and are
unchanged, so existing benchmark numbers stay comparable. For single-turn rows
the server figure can differ from the dataset's by the chat template's tokens.

Accuracy Tests

test/registered/unit/bench/test_bench_serving_prompt_len.py, CPU CI. The stub
server behaves like a default SGLang server (usage only when requested). The
multi-turn case fails on main, and without the wrapper change, with the bug in
one line:

AssertionError: Lists differ: [9999, 9999, 9999] != [100, 250, 400]

The row's value once per round, instead of each request's own prompt.

Speed Tests and Profiling

N/A

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #36243176795
Latest PR Test (Extra): ✅ Run #36243181038
Latest PR Test (AMD ROCm 10): ❌ Run #36243176715

[nvcnvn]

@kpham-sgl can you take a look? I see you review and work on some related story.

[BBuf]

One edge case in the multi-turn wrapper: could we default include_usage inside stream_options, rather than defaulting the whole object?

If extra_request_body already contains stream_options={}, null, or {"continuous_usage_stats": false}, the current setdefault does not add include_usage. These are valid request configurations, but with the server's default usage setting, no prompt count arrives and the multi-turn length bug remains. Reusing this PR's HTTP fixture, I get [9999, 9999, 9999] instead of [100, 250, 400], while all requests succeed.

This fixes those cases while preserving an explicit include_usage=false:

options = inner_input.extra_request_body.get("stream_options") or {}
options.setdefault("include_usage", True)
inner_input.extra_request_body["stream_options"] = options

Please cover these cases in the multi-turn test as well.

[BBuf]

Could you also simplify test/registered/unit/bench/test_bench_serving_prompt_len.py? It's 253 lines for four cases, with quite a bit of repeated setup and explanation.

Please trim the long docstrings (the bug background is already in the PR description), consolidate the repeated Namespace setup and server startup/cleanup, and use subTest for similar cases where practical. The server helper can also bind directly to port 0 and register cleanup once, including server_close().

Please keep the multi-turn HTTP regression that verifies usage is actually requested and each round gets its own prompt length, along with non-streaming/fallback coverage and the stream_options edge cases above. We should be able to make the test much shorter without losing that coverage.

[nvcnvn]

Thanks @BBuf, both addressed in 707d7d5 — the stream_options edge cases are covered in the multi-turn test.