Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ deployment:
--seed 1024
--quantization ascend
--max-num-seqs 4
--max-model-len 32768
--max-model-len 36864
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The max-model-len value is duplicated in the second deployment configuration on line 75. To improve maintainability and prevent future inconsistencies where one value is updated but the other is not, consider defining this value as a variable in the env_common section and referencing it in both server_cmd blocks.

For example, you could add MAX_MODEL_LEN: 36864 to env_common and then use --max-model-len $MAX_MODEL_LEN in both deployment commands. This pattern is already used for $SERVER_PORT.

--max-num-batched-tokens 16384
--trust-remote-code
--gpu-memory-utilization 0.9
Expand Down Expand Up @@ -72,7 +72,7 @@ deployment:
--seed 1024
--quantization ascend
--max-num-seqs 4
--max-model-len 32768
--max-model-len 36864
--max-num-batched-tokens 256
--trust-remote-code
--gpu-memory-utilization 0.9
Expand Down
Loading