Skip to content

[CI] Add scripts for bechmark collection and email distribution.#1307

Merged
hsliuustc0106 merged 10 commits into
vllm-project:mainfrom
congw729:ci/nighly-performance
Mar 3, 2026
Merged

[CI] Add scripts for bechmark collection and email distribution.#1307
hsliuustc0106 merged 10 commits into
vllm-project:mainfrom
congw729:ci/nighly-performance

Conversation

@congw729
Copy link
Copy Markdown
Collaborator

@congw729 congw729 commented Feb 10, 2026 •
edited
Loading

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Nightly collecting performance.
This whole pipeline contains 3 steps: 1. Benchmark collection -> 2. Report generation -> 3. Report sending
This PR contains the codes for step 2 & step 3.

Test Plan

Local tests for separate steps, done.
Test the whole pipeline on Buildkite, done.

Test Result

Can upload and download the artifacts successfully.
image

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

@congw729 congw729 marked this pull request as draft February 10, 2026 10:02
@congw729 congw729 mentioned this pull request Feb 10, 2026
Closed
1 task
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Feb 10, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e30583a621

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread tools/nightly/generate_nightly_perf_excel.py Outdated
Comment thread tools/nightly/generate_nightly_perf_excel.py
@congw729 congw729 changed the title [WIP] Add scripts for json->xlsx convertion codes. [WIP] Add scripts for bechmark data json->xlsx convertion codes. Feb 11, 2026
@congw729 congw729 force-pushed the ci/nighly-performance branch from 6af48c0 to 9b053b5 Compare February 11, 2026 08:09
@congw729 congw729 changed the title [WIP] Add scripts for bechmark data json->xlsx convertion codes. [WIP] Add scripts for bechmark collection and email distribution. Feb 11, 2026
@congw729 congw729 force-pushed the ci/nighly-performance branch 3 times, most recently from 055a959 to a6a4871 Compare February 12, 2026 03:07
@congw729 congw729 changed the title [WIP] Add scripts for bechmark collection and email distribution. Add scripts for bechmark collection and email distribution. Feb 12, 2026
@congw729 congw729 marked this pull request as ready for review February 12, 2026 09:25
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Feb 12, 2026
View reviewed changes
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 250f79a80d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread .buildkite/pipeline.yml Outdated
Comment thread .buildkite/test-nightly.yaml Outdated
@david6666666 david6666666 added the ready label to trigger buildkite CI label Feb 12, 2026
@congw729 congw729 changed the title Add scripts for bechmark collection and email distribution. [CI] Add scripts for bechmark collection and email distribution. Feb 13, 2026
@congw729 congw729 force-pushed the ci/nighly-performance branch 2 times, most recently from 4f8aece to 43e2fdc Compare February 13, 2026 08:29
@Gaohan123 Gaohan123 removed the ready label to trigger buildkite CI label Feb 13, 2026
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@lishunyang12 lishunyang12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR bundles unrelated changes (diffusion sleep rewrite, PID detection rewrite, CI pipeline gutting) with the nightly perf scripts -- should be separate PRs.

lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread .buildkite/pipeline.yml Outdated
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread .buildkite/test-nightly.yaml Outdated
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread .buildkite/test-nightly.yml
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread .buildkite/test-nightly.yml Outdated
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread tools/nightly/generate_nightly_perf_excel.py
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread tools/nightly/generate_nightly_perf_excel.py
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread tools/nightly/send_nightly_perf_email.py
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread vllm_omni/entrypoints/utils.py
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread vllm_omni/diffusion/worker/diffusion_worker.py
lishunyang12
lishunyang12 reviewed Feb 21, 2026
View reviewed changes
Comment thread vllm_omni/entrypoints/omni_stage.py Outdated
@hsliuustc0106 hsliuustc0106 added the ready label to trigger buildkite CI label Feb 24, 2026
@Gaohan123
Copy link
Copy Markdown
Collaborator

Gaohan123 commented Feb 25, 2026

Please resolve CI failure. Thanks.

@congw729
Copy link
Copy Markdown
Collaborator Author

congw729 commented Feb 26, 2026 •
edited
Loading

Please resolve CI failure. Thanks.

The whole pipeline contains 3 steps: 1. Benchmark collection -> 2. Report generation -> 3. Report sending
The failure occurs at step 1 and is not addressed by this PR; it will be fixed in #1468.

@congw729 congw729 force-pushed the ci/nighly-performance branch 6 times, most recently from 2657086 to 738855a Compare February 26, 2026 08:00
@congw729
Copy link
Copy Markdown
Collaborator Author

congw729 commented Feb 26, 2026

I will come back to this PR next week. H100 is in shortage right now.

@congw729 congw729 force-pushed the ci/nighly-performance branch from 738855a to 5a11034 Compare February 27, 2026 08:38
@hsliuustc0106
Copy link
Copy Markdown
Collaborator

hsliuustc0106 commented Mar 2, 2026

the CI is keeping failing

@congw729
Copy link
Copy Markdown
Collaborator Author

congw729 commented Mar 3, 2026 •
edited
Loading

the CI is keeping failing

Yeap, still under testing

congw729 added 9 commits March 3, 2026 14:30
@congw729
Add script for benchmark xlsx file generation & email distribution.
930fa09 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Benchmark Collection & Report Generation & Email Sending
e017b62 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Using tests/benchmarks/test_serve_cli.py to generate benchmark.
08f27f7 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Update
1b561b8 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Updare H100 link.
ddfad5f 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Try
5e97c25 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Modify result-dir get method
cbc3cf5 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Modify datetime.
d719efb 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729
Ready for merge.
398af31 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729 congw729 force-pushed the ci/nighly-performance branch from 2d9133f to 398af31 Compare March 3, 2026 06:36
@congw729
Copy link
Copy Markdown
Collaborator Author

congw729 commented Mar 3, 2026

This PR is ready for merge. @hsliuustc0106

@congw729
Fix AWS link.
50c60d4 
Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
hsliuustc0106
hsliuustc0106 approved these changes Mar 3, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@hsliuustc0106 hsliuustc0106 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@hsliuustc0106 hsliuustc0106 enabled auto-merge (squash) March 3, 2026 06:44
@hsliuustc0106 hsliuustc0106 merged commit e7f2c88 into vllm-project:main Mar 3, 2026
6 of 7 checks passed
yjb767868009 pushed a commit to yjb767868009/vllm-omni that referenced this pull request Mar 11, 2026
@congw729 @yjb767868009
[CI] Add scripts for bechmark collection and email distribution. (vll…
d5ac438 
…m-project#1307)

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
@congw729 congw729 deleted the ci/nighly-performance branch March 19, 2026 02:45
clodaghwalsh17 pushed a commit to clodaghwalsh17/nm-vllm-omni-ent that referenced this pull request May 12, 2026
@congw729 @clodaghwalsh17
[CI] Add scripts for bechmark collection and email distribution. (vll…
76b25e2 
…m-project#1307)

Signed-off-by: Alicia <115451386+congw729@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lishunyang12 lishunyang12 lishunyang12 left review comments

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@hsliuustc0106 hsliuustc0106 hsliuustc0106 approved these changes

Assignees

No one assigned

Labels

ready label to trigger buildkite CI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@congw729 @hsliuustc0106 @Gaohan123 @lishunyang12 @david6666666