[docker] feat: update to vllm 0.10.0, mcore 0.13, transformers 4.55.4#3192
[docker] feat: update to vllm 0.10.0, mcore 0.13, transformers 4.55.4#3192vermouth1992 merged 42 commits intoverl-project:mainfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates several key dependencies, including vLLM to 0.10.0, Megatron-Core to 0.13, and Transformers to 4.55.4. The changes primarily involve updating version numbers in Dockerfiles and documentation, and adapting the code to the new APIs of these libraries.
My review found one high-severity issue in the monkey-patching logic for vision models. The current patch for transformers>=4.55.0 is likely ineffective, necessitating a performance-degrading workaround that disables flash attention for vision models when using Ulysses sequence parallelism. I've suggested a more robust patching strategy that should fix the underlying issue and allow for the removal of the workaround.
Other changes appear to be correct and consistent with the goal of updating dependencies.
| return attn_output | ||
|
|
||
|
|
||
| def _ulysses_flash_attention_forward_new_api( |
There was a problem hiding this comment.
Please modify the name to _ulysses_flash_attention_forward_transformers_4_55
f9f3116 to
91361d1
Compare
| max_model_len=self.max_model_len, | ||
| max_num_seqs=config.max_num_seqs, | ||
| load_format=config.load_format, | ||
| load_format="dummy" if self.config.load_format.startswith("dummy") else self.config.load_format, |
There was a problem hiding this comment.
@vermouth1992 I fix vllm as sglang's way of load_format setting.
bc1e915 to
b46789a
Compare
What does this PR do?
[docker] feat: update to vllm 0.10.0, mcore 0.13, transformers 4.55.4
Checklist Before Starting
[{modules}] {type}: {description}(This will be checked by the CI){modules}includefsdp,megatron,sglang,vllm,rollout,trainer,ci,training_utils,recipe,hardware,deployment,ray,worker,single_controller,misc,perf,model,algo,env,tool,ckpt,doc,data,like[megatron, fsdp, doc]{type}is infeat,fix,refactor,chore,test[BREAKING]to the beginning of the title.[BREAKING][fsdp, megatron] feat: dynamic batchingTest
API and Usage Example
# Add code snippet or script demonstrating how to use thisDesign & Code Changes
Checklist Before Submitting
Important
Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.
pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=alwaysci-requestchannel in theverlSlack workspace. (If not accessible, please try the Feishu group (飞书群).)