[Misc][Benchmark] Add support for CustomDataset

[ekagra-ranjan]

Fixes: #14036 (review) and #18509

Currently, online serving benchmark only loads some specified dataset. This PR adds support for CustomDataset which can load arbitrary datasets for benchmarking. It accepts dataset in jsonl format and can be extended to other format since it converts the input data to into a standardized format of list of dictionary in load_data() which then is used in sample().
Example is given in the comment of the function.

Usage

MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
ONLINE_CUSTOM_DATASET="data.jsonl"

time python3 benchmarks/benchmark_serving.py --port 9001 --save-result --save-detailed \
  --backend vllm \
  --model "${MODEL_NAME}" \
  --endpoint /v1/completions \
  --dataset-name custom \
  --dataset-path ${ONLINE_CUSTOM_DATASET} \
  --custom-skip-chat-template \
  --num-prompts 80 \
  --max-concurrency 1 \
  --temperature=0.3 \
  --top-p=0.75 \
  --result-dir "./log/"

example of data.jsonl

{"prompt": "What is the capital of India?"}
{"prompt": "What is the capital of Iran?"}
{"prompt": "What is the capital of China?"}

cc: @ywang96

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

Overall LGTM and I left a comment! Can you also update the benchmark readme to include this dataset? Thanks!

[ekagra-ranjan]

@ywang96 - all done! Pls have another look.

[ywang96]

🚢 @ekagra-ranjan Thank you for the contribution!

[ekagra-ranjan]

There is 1 failure in V1 test which is blocking merge

=========================== short test summary info ============================
--
  | [2025-05-28T21:23:02Z] FAILED v1/engine/test_async_llm.py::test_abort[engine_args0-Hello my name is Robert and-RequestOutputKind.DELTA] - assert not True
  | [2025-05-28T21:23:02Z]  +  where True = has_unfinished_requests()
  | [2025-05-28T21:23:02Z]  +    where has_unfinished_requests = <vllm.v1.engine.output_processor.OutputProcessor object at 0x7ff42ed8d4f0>.has_unfinished_requests
  | [2025-05-28T21:23:02Z]  +      where <vllm.v1.engine.output_processor.OutputProcessor object at 0x7ff42ed8d4f0> = <vllm.v1.engine.async_llm.AsyncLLM object at 0x7ff433c1ce60>.output_processor
  | [2025-05-28T21:23:02Z] ============ 1 failed, 50 passed, 21 warnings in 760.68s (0:12:40) =============
  | [2025-05-28T21:23:05Z] 🚨 Error: The command exited with status 1

I dont think its caused by this PR. Could be related to this. @ywang96 - what are your thoughts?

[ywang96]

@ekagra-ranjan I've merged this branch from main so let's see if it helps!