Skip to content

Update test case parameter to have the throughput above 8.0 #19994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yaochengji merged 1 commit into from
Jun 24, 2025
Merged

Update test case parameter to have the throughput above 8.0 #19994

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .buildkite/scripts/tpu/config_v6e_1.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ CONTAINER_NAME=vllm-tpu

# vllm config
MODEL=meta-llama/Llama-3.1-8B-Instruct
MAX_NUM_SEQS=512
MAX_NUM_BATCHED_TOKENS=512
MAX_NUM_SEQS=256
MAX_NUM_BATCHED_TOKENS=1024
Comment on lines +7 to +8
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better maintainability, it would be helpful to add a comment explaining the rationale for these specific values. The PR description mentions this is to increase throughput, and capturing that context directly in the configuration file will be valuable for anyone adjusting these parameters in the future.

# Tuned to increase throughput on v6e TPUs by allowing larger token batches.
MAX_NUM_SEQS=256
MAX_NUM_BATCHED_TOKENS=1024

TENSOR_PARALLEL_SIZE=1
MAX_MODEL_LEN=2048
DOWNLOAD_DIR=/mnt/disks/persist
Expand Down