[Rust][Benchmark] Support HF ShareGPT datasets in multi-turn mode

[xiaguan]

Summary

• allow Rust vllm bench serve to load ShareGPT-format Hugging Face datasets in multi-turn mode
• reuse the existing HF downloader and ShareGPT conversation loader
• apply --hf-output-len and preserve conversation-style --multi-turn-max-turns semantics

Example

vllm bench serve \
  --backend openai-chat \
  --dataset-name hf \
  --dataset-path Inferact/codex_swebenchpro_traces \
  --multi-turn

Inferact/codex_swebenchpro_traces exposes 610 ShareGPT-format conversations through the Hugging Face dataset server. A network dry run loaded one conversation with three turns and 44,103 user-message tokens.

Scope

This supports HF configs whose rows use ShareGPT conversations entries with alternating from / value messages. It intentionally does not add canonical request-trace replay.

Duplicate-work check

I searched open PRs for hf multi-turn benchmark, ShareGPT HuggingFace multi-turn, and Inferact/codex_swebenchpro_traces; no PR implementing this support was found.

Tests

• cargo fmt --all -- --check
• cargo nextest run -p vllm-bench — 149 passed, 10 skipped
• cargo clippy -p vllm-bench --all-targets -- -D warnings
• pre-commit run --files rust/src/bench/README.md rust/src/bench/src/config.rs rust/src/bench/src/multi_turn.rs
• network dry run against Inferact/codex_swebenchpro_traces — 1 conversation, 3 turns, 44,103 user-message tokens

Model evaluation is not applicable because this changes only benchmark dataset loading and does not modify model execution or outputs.

AI assistance was used to inspect the repository, implement the change, and run tests. The human submitter must review and understand every changed line before marking this PR ready.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9144d55c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Make HF cache size-aware for multi-turn runs

When a user does a small HF multi-turn run first, such as --num-prompts 1 --dry-run, this call asks download_hf_dataset to cache only that many rows; the helper's cache key is just dataset/config/split and it returns any existing file before checking the requested row count. A later benchmark of the same HF ShareGPT dataset with more conversations will therefore load the undersized cache and load_sharegpt_multi_turn will oversample the few cached conversations, producing repeated conversations and misleading benchmark results. Please either make the HF cache size-aware or ensure this path fetches enough rows when the existing cache is too small.

Useful? React with 👍 / 👎.

[xiaguan]

Thanks for catching this. This appears to be a pre-existing issue in the HF dataset cache rather than something introduced by this PR: the current single-turn HF path on main already calls download_hf_dataset(..., config.num_prompts) and relies on the same dataset/config/split-only cache key before oversampling the loaded rows. This PR reuses that existing downloader contract for multi-turn loading without changing its cache behavior.

I agree that making the cache size-aware would improve both single-turn and multi-turn HF benchmarks, but that requires changing the shared cache contract and deserves its own focused fix and regression coverage. I would prefer to keep this PR scoped to enabling ShareGPT-format HF datasets in the existing multi-turn pipeline so the behavior change remains small and reviewable.

[github-actions]

✅ @xiaguan, CI is now available for this PR.

• /ci run starts a CI build.
• /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.

[mergify]

Hi @xiaguan, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[xiaguan]

The failing pre-commit check comes from existing repository-wide ShellCheck findings in unrelated shell scripts; this PR does not modify those files.

[mergify]

Hi @xiaguan, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[esmeetu]

/ci retry

[github-actions]

✅ No earlier CI build exists for this PR. Use /ci run first.

[esmeetu]

/ci run

[github-actions]

✅ Triggered Buildkite CI #88720 for commit 8f5fe95dd040.