[BugFix] Fix ascend config check#1092
Conversation
372e466 to
b9a4e95
Compare
324099c to
d1439b7
Compare
|
|
||
| # for V1 Engine, aclgraph doesn't work with deepseek model and only qwen model is well tested. | ||
| if envs.VLLM_USE_V1 and vllm_config.model_config is not None and not enforce_eager: | ||
| if envs.VLLM_USE_V1 and vllm_config.model_config is not None and not enforce_eager and not ascend_config.torchair_graph_config.enabled: |
There was a problem hiding this comment.
this check should be moved to platform.py:check_and_update_config
| @@ -126,7 +129,7 @@ def check_ascend_config(vllm_config, enforce_eager): | |||
| "Torchair graph mode only works with deepseek model.") | |||
There was a problem hiding this comment.
should be:
if ascend_config.torchair_graph_config.enabled:
if envs.VLLM_MLA_DISABLE:
logger.warning(
"Torchair graph mode is still experimental and not supported for V1 without mla currently, "
"it has been disabled automatically.")
ascend_config.torchair_graph_config.enabled = False
elif vllm_config.model_config:
model_type = vllm_config.model_config.hf_config.model_type
if "deepseek" not in model_type:
raise NotImplementedError(
"Torchair graph mode only works with deepseek model.")
ascend_config. ascend_scheduler_config.enabled = False -> ascend_config.torchair_graph_config.enabled = False
if vllm_config.model_config: -> elif vllm_config.model_config:
d1439b7 to
3eefdc9
Compare
|
@NeverRaR Thanks for your review. I've updated the check logic to make it more clear. Please take a look at again. |
| | `torchair_graph_config` | dict | `{}` | The config options for torchair graph mode | | ||
| | `ascend_scheduler_config` | dict | `{}` | The config options for ascend scheduler | | ||
| | `expert_tensor_parallel_size` | str | `1` | Expert tensor parallel size the model to use. | | ||
| | `refresh` | bool | `false` | Whether to refresh global ascend config content. This value is usually used by rlhf case. | |
There was a problem hiding this comment.
Not, this value is only used for rlhf, for verl or someother framework, the case is:
- verl load and update vllm config
- verl start LLM with additional config in external_executor mode
in the fisrt step, the ascend config has been initialized, then in the second step, the additional config will be skipped.
To solve the problem, we should let verl pass refresh, the we can regenerate the config
830e333 to
8cf5adf
Compare
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
8cf5adf to
755edcf
Compare
…llm-project#1092) Merge branch wengang/cherry-pick-1029-1092 of git@code.alipay.com:Theta/vllm-ascend.git into dev-v0.9.0604 https://code.alipay.com/Theta/vllm-ascend/pull_requests/108 Reviewed-by: 子宏 <tanzhiqiang.tzq@antgroup.com> * [Misc] Refactor additional_config (vllm-project#1029) * [BugFix] Fix ascend config check (vllm-project#1092) * [Misc] Update benchmark scripts
Fix the ascend config check logic:
1. refactor check_ascend_config to make it clear:
1. torchair graph should not work with enforce_eager=True
2. aclgraph should not work with torchair graph
3. add refresh config for rlhf case
4. fix a typo in model runner
5. change expert_tensor_parallel_size default to 0 to keep the same as
before
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Fix the ascend config check logic:
1. refactor check_ascend_config to make it clear:
1. torchair graph should not work with enforce_eager=True
2. aclgraph should not work with torchair graph
3. add refresh config for rlhf case
4. fix a typo in model runner
5. change expert_tensor_parallel_size default to 0 to keep the same as
before
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Fix the ascend config check logic: