[NVIDIA] feat: adds more configurations for GB200 SGLang DSR1 #335
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
|
|
||
| #!/bin/bash | ||
|
|
||
| set -x | ||
|
|
||
| source "$(dirname "$0")/benchmark_lib.sh" | ||
|
|
||
| check_env_vars CONC_LIST ISL OSL IMAGE SPEC_DECODING MODEL_PATH \ | ||
| PREFILL_NUM_WORKERS PREFILL_TP PREFILL_EP PREFILL_DP_ATTN \ | ||
| DECODE_NUM_WORKERS DECODE_TP DECODE_EP DECODE_DP_ATTN \ | ||
| PREFILL_NODES DECODE_NODES N_ADDITIONAL_FRONTENDS SGL_SLURM_JOBS_PATH # SGL_SLURM_JOBS_PATH FIXME | ||
|
|
||
| # Always clone and setup Dynamo | ||
| echo "Cloning Dynamo repository..." | ||
| git clone https://github.com/ai-dynamo/dynamo.git | ||
| cd dynamo && git checkout ishan/fp48k1k && cd .. # All configs are now tracked in this branch | ||
|
|
||
| cd "$SGL_SLURM_JOBS_PATH" | ||
|
|
||
| # Set up SGL launch script-specific environment variables | ||
| export TIME_LIMIT="04:00:00" | ||
| export MODEL_PATH=$MODEL_PATH | ||
| export CONFIG_DIR=$CONFIG_DIR | ||
| export CONTAINER_IMAGE=$IMAGE | ||
| export GPU_TYPE="gb200-fp4" | ||
|
|
||
| # Launch jobs based on ISL/OSL | ||
| # Replace ' ' in CONC_LIST with 'x' such that the concurrency list is represented | ||
| # by a list of numbers delimted by 'x'. This is because of how the underlying launch script | ||
| # expects the concurrencies. | ||
| bash ./submit_disagg.sh $PREFILL_NODES \ | ||
| $PREFILL_NUM_WORKERS \ | ||
| $DECODE_NODES \ | ||
| $DECODE_NUM_WORKERS \ | ||
| $N_ADDITIONAL_FRONTENDS \ | ||
| $ISL $OSL "${CONC_LIST// /x}" inf \ | ||
| $GPU_TYPE \ | ||
| $SCRIPT_MODE |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,22 +14,16 @@ export SLURM_JOB_NAME="benchmark-dynamo.job" | |
|
|
||
| ### FRAMEWORK_DIFF_IF_STATEMENT #1 - difference in setting up envvars | ||
| if [[ $FRAMEWORK == "dynamo-sglang" ]]; then | ||
| export IMAGE="/mnt/lustre01/artifacts/containers/lmsysorg+sglang+v0.5.5.post2.sqsh" | ||
|
|
||
| if [[ $PRECISION == "fp4" ]]; then | ||
| export MODEL_PATH="/mnt/lustre01/models/deepseek-r1-0528-fp4-v2" | ||
|
Collaborator
There was a problem hiding this comment. for posterity, it would be preferable if this was retrieved from the master config
Collaborator
Author
There was a problem hiding this comment. Thanks for the suggestion! I have addressed them through InferenceMAX/InferenceMAX@35d7555 and InferenceMAX/InferenceMAX@45cc883, which applies to TRTLLM side of code as well. |
||
| else | ||
| export MODEL_PATH="/mnt/lustre01/models/deepseek-r1-0528" | ||
|
|
||
| fi | ||
|
|
||
| export CONFIG_DIR="/mnt/lustre01/artifacts/sglang-configs/1k1k" | ||
| export SGL_SLURM_JOBS_PATH="dynamo/examples/backends/sglang/slurm_jobs" | ||
| else | ||
| SQUASH_FILE="/mnt/lustre01/users/sa-shared/images/$(echo "$IMAGE" | sed 's/[\/:@#]/_/g').sqsh" | ||
| srun --partition=$SLURM_PARTITION --exclusive --time=180 bash -c "enroot import -o $SQUASH_FILE docker://$IMAGE" | ||
|
|
@@ -148,4 +142,4 @@ PY | |
| done | ||
| fi | ||
|
|
||
| echo "All result files processed" | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the PR, is it possible to have the IMAGE inherit from the nvidia-master.yaml instead of hard setting in the launcher script?
kinda like what trtllm dynamo already does?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @functionstackx , thanks for the comment! I have updated the code in InferenceMAX/InferenceMAX@c1024db so that Dynamo+SGLang will also pull the container from nvidia-master.yaml.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
closes https://github.com/InferenceMAX/InferenceMAX/issues/334