[Benchmark] Cleanup deprecated nightly benchmark and adjust the docstring for performance benchmark

[KuntaiDu]

Purpose

Cleanup the nightly benchmark suite because of two reasons:

• The codepath of comparing vLLM against other engines is no longer maintained.
• The performance benchmarking CI is now triggered under Pytorch Hub instead of in vLLM's CI pipeline.

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request effectively cleans up the deprecated nightly benchmark suite by removing numerous files and reorganizing the directory structure. The changes align well with the stated purpose. However, I've identified a few inconsistencies in the documentation that were likely overlooked during the refactoring. These include incorrect paths in commands, broken links, and descriptions of functionality that has been removed. I've provided specific comments to help resolve these documentation issues.

[yeqcharlotte]

could you fix markdown precommit?

[yeqcharlotte]

we have b200 too~

not blocking this PR. cc: @huydhn @linzebing -- we need to add some new docs for vllm's continuous profiling and benchmarking runs! especially if these are locally runnable https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks

[linzebing]

@namanlalitnyu : can you take this one?

[namanlalitnyu]

Sure, I'll take this up.

But, I have a small concern with these changes, that we might have to consider. Here, we are migrating the file run-performance-benchmarks.sh from the nightly-benchmarks directory to the performance-benchmarks directory.
In order for our continuous vLLM benchmarking CI to work successfully in Pytorch-infra, I think we might have to update the vllm-benchmark.yml workflow as well, to adapt to these new changes, once they are merged. As, we would be referencing the incorrect directory location at a couple of places there.
This Draft PR (84) is ready and can be merged once these changes are verified from vLLM.

cc: @huydhn @linzebing

[namanlalitnyu]

@yeqcharlotte @linzebing
Added the documentation for vLLM continuous profiling and benchmarking in this PR: 25819. But, I am not able to explicitly add reviewers to it 😅

[KuntaiDu]

Thank you @namanlalitnyu and @yeqcharlotte for checking. Please let me know if you prefer to put the renaming of nightly-benchmarks folder in a follow-up PR.

[KuntaiDu]

Also B200 added.

[yeqcharlotte]

thanks for cleaning this up! please fix some broken links and mkdown lints.

[huydhn]

For the sake of completeness, here is a round of benchmark https://github.com/pytorch/pytorch-integration-testing/actions/runs/18112289981 running with the change from this PR

It looks like we need to tweak the workflows on PyTorch side a bit to handle the missing .buildkite/nightly-benchmarks/tests directory. Also, I don't think we can delete .buildkite/nightly-benchmarks/scripts/run-nightly-benchmarks.sh without moving that script to PyTorch and call the new script here https://github.com/pytorch/pytorch-integration-testing/blob/main/.github/workflows/vllm-benchmark.yml#L295

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @KuntaiDu.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

Documentation preview: https://vllm--25786.org.readthedocs.build/en/25786/

[KuntaiDu]

Thank you for all the feedbacks. I am travelling and will be back soon.

[KuntaiDu]

Now back to work

[KuntaiDu]

@yeqcharlotte I have updated this PR, would be really good if you can take a look again.

[yeqcharlotte]

@KuntaiDu fix the markdown lints?

Error: .buildkite/performance-benchmarks/README.md:8 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
Error: .buildkite/performance-benchmarks/README.md:19 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]
Error: .buildkite/performance-benchmarks/README.md:137 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2]

[yeqcharlotte]

thanks for cleaning this up!

[KuntaiDu]

@namanlalitnyu It would be nice if we could also start the process of changing the pytorch CI correspondingly (like this draft PR (pytorch/pytorch-integration-testing#84))

[namanlalitnyu]

@namanlalitnyu It would be nice if we could also start the process of changing the pytorch CI correspondingly (like this draft PR (pytorch/pytorch-integration-testing#84))
@KuntaiDu thanks for updating and the heads up. Yes, we have trigged the CI pipeline with these changes, and once they pass successfully, we will merge the updates there as well.