[Feature]: benchmark_serving.py should support --logprobs

[afeldman-nm]

🚀 The feature, motivation and pitch

The OpenAI API (and by extension vLLM's completions functionality) supports configuring the number of logprobs-per-token to return at the granularity of each request, via the logprobs argument. However, benchmarks/benchmark_serving.py currently does not configure the logprobs argument when generating requests, nor does benchmarks/benchmark_serving.py have a --logprobs CLI argument. This is an issue because it is desirable to benchmark the impact of different --logprobs settings on vLLM performance.

So the issue is that (1) benchmarks/benchmark_serving.py should support a --logprobs argument, and (2) the value of the --logprobs CLI argument should configure the underlying logprobs argument to the completion requests generated during benchmarking.

Alternatives

In tests/utils.py, the function

completions_with_server_args(
    prompts: List[str],
    model_name: str,
    server_cli_args: List[str],
    num_logprobs: Optional[int],
    max_wait_seconds: int = 240,
)

shows how to configure OpenAI API requests with the logprobs argument set.

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.