[Bug]: Mamba2 models (Bamba and Codestral Mamba) fail on RoCM

[tlrmchlsmth]

Your current environment

Via @hackey:

I am using:
ROCM (Dual AMD 7900 xtx)
Ubuntu 24.04

🐛 Describe the bug

See #6479 (comment)

Specifically this part:

registry.py:321]     from vllm.attention.backends.flash_attn import FlashAttentionMetadata ERROR 02-21 11:17:10 registry.py:321]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/flash_attn.py", line 25, in <module> ERROR 02-21 11:17:10 registry.py:321]     from vllm.vllm_flash_attn import (flash_attn_varlen_func, ERROR 02-21 11:17:10 registry.py:321] ImportError: cannot import name 'flash_attn_varlen_func' from 'vllm.vllm_flash_attn' (unknown location) ERROR 02-21 11:17:10 registry.py:321]  Traceback (most recent call last): File "/usr/local/bin/vllm", line 8, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main 

It looks like the problem is caused by importing FlashAttentionMetadata in MambaMixer2, which pulls in vllm_flash_attn, which is unsupported on RoCM.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hackey]

Thanks for the help! I saw the error about flash attention, but I was actually surprised. After all, it seems like fa for rocm is now implemented using triton in vllm. And it seems like mamba can work without fa. At least I was able to compile it correctly from git.

[tlrmchlsmth]

You're right, it's not needed. It's just being imported for this check, since the Mamba mixer looks inside the attention metadata. I'll put up a fix in the next day or so.

vllm/vllm/model_executor/layers/mamba/mamba_mixer2.py

Lines 389 to 391 in 288cc6c

	if (isinstance(attn_metadata,
	(FlashAttentionMetadata, XFormersMetadata,
	PlaceholderAttentionMetadata))

[hackey]

You're right, it's not needed. It's just being imported for this check, since the Mamba mixer looks inside the attention metadata. I'll put up a fix in the next day or so.

vllm/vllm/model_executor/layers/mamba/mamba_mixer2.py

Lines 389 to 391 in 288cc6c

if (isinstance(attn_metadata,
(FlashAttentionMetadata, XFormersMetadata,
PlaceholderAttentionMetadata))

@tlrmchlsmth, hello! Any news on the possibility of launching the codestral mamba model on ROCM?

I removed the check you wrote about. But there are still errors. Apparently, it's not that simple:
`
vllm serve /app/model/Codestral-Mamba-7b --port 8002 --tensor-parallel-size 2 --gpu-memory-utilization 0.95 --max-model-len 32000

INFO 02-25 08:13:09 init.py:207] Automatically detected platform rocm.
INFO 02-25 08:13:09 api_server.py:911] vLLM API server version 0.7.4.dev17+gb2c3fc5d.d20250225
INFO 02-25 08:13:09 api_server.py:912] args: Namespace(subparser='serve', model_tag='/app/model/Codestral-Mamba-7b', config='', host=None, port=8002, uvicorn_log_level='info', allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, enable_reasoning=False, reasoning_parser=None, tool_call_parser=None, tool_parser_plugin='', model='/app/model/Codestral-Mamba-7b', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', max_model_len=32000, guided_decoding_backend='xgrammar', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=2, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', generation_config=None, override_generation_config=None, enable_sleep_mode=False, calculate_kv_scales=False, additional_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function ServeSubcommand.cmd at 0x7b94e9cb65c0>)
INFO 02-25 08:13:09 api_server.py:208] Started engine process with PID 52
INFO 02-25 08:13:12 init.py:207] Automatically detected platform rocm.
ERROR 02-25 08:13:18 registry.py:321] Error in inspecting model architecture 'Mamba2ForCausalLM'
ERROR 02-25 08:13:18 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 528, in _run_in_subprocess
ERROR 02-25 08:13:18 registry.py:321] returned.check_returncode()
ERROR 02-25 08:13:18 registry.py:321] File "/usr/lib/python3.12/subprocess.py", line 504, in check_returncode
ERROR 02-25 08:13:18 registry.py:321] raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 02-25 08:13:18 registry.py:321] subprocess.CalledProcessError: Command '['/usr/bin/python3.12', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 02-25 08:13:18 registry.py:321]
ERROR 02-25 08:13:18 registry.py:321] The above exception was the direct cause of the following exception:
ERROR 02-25 08:13:18 registry.py:321]
ERROR 02-25 08:13:18 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 319, in _try_inspect_model_cls
ERROR 02-25 08:13:18 registry.py:321] return model.inspect_model_cls()
ERROR 02-25 08:13:18 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 290, in inspect_model_cls
ERROR 02-25 08:13:18 registry.py:321] return _run_in_subprocess(
ERROR 02-25 08:13:18 registry.py:321] ^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 531, in _run_in_subprocess
ERROR 02-25 08:13:18 registry.py:321] raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 02-25 08:13:18 registry.py:321] RuntimeError: Error raised in subprocess:
ERROR 02-25 08:13:18 registry.py:321] :128: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 02-25 08:13:18 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:18 registry.py:321] File "", line 198, in _run_module_as_main
ERROR 02-25 08:13:18 registry.py:321] File "", line 88, in _run_code
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 552, in
ERROR 02-25 08:13:18 registry.py:321] _run()
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 545, in _run
ERROR 02-25 08:13:18 registry.py:321] result = fn()
ERROR 02-25 08:13:18 registry.py:321] ^^^^
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 291, in
ERROR 02-25 08:13:18 registry.py:321] lambda: _ModelInfo.from_model_cls(self.load_model_cls()))
ERROR 02-25 08:13:18 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 294, in load_model_cls
ERROR 02-25 08:13:18 registry.py:321] mod = importlib.import_module(self.module_name)
ERROR 02-25 08:13:18 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:18 registry.py:321] File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module
ERROR 02-25 08:13:18 registry.py:321] return _bootstrap._gcd_import(name[level:], package, level)
ERROR 02-25 08:13:18 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:18 registry.py:321] File "", line 1387, in _gcd_import
ERROR 02-25 08:13:18 registry.py:321] File "", line 1360, in _find_and_load
ERROR 02-25 08:13:18 registry.py:321] File "", line 1331, in _find_and_load_unlocked
ERROR 02-25 08:13:18 registry.py:321] File "", line 935, in _load_unlocked
ERROR 02-25 08:13:18 registry.py:321] File "", line 999, in exec_module
ERROR 02-25 08:13:18 registry.py:321] File "", line 488, in _call_with_frames_removed
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mamba2.py", line 15, in
ERROR 02-25 08:13:18 registry.py:321] from vllm.model_executor.layers.mamba.mamba_mixer2 import (
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/mamba/mamba_mixer2.py", line 9, in
ERROR 02-25 08:13:18 registry.py:321] from vllm.attention.backends.flash_attn import FlashAttentionMetadata
ERROR 02-25 08:13:18 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/flash_attn.py", line 25, in
ERROR 02-25 08:13:18 registry.py:321] from vllm.vllm_flash_attn import (flash_attn_varlen_func,
ERROR 02-25 08:13:18 registry.py:321] ImportError: cannot import name 'flash_attn_varlen_func' from 'vllm.vllm_flash_attn' (unknown location)
ERROR 02-25 08:13:18 registry.py:321]
Traceback (most recent call last):
File "/usr/local/bin/vllm", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
args.dispatch_function(args)
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 34, in cmd
uvloop.run(run_server(args))
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
return __asyncio.run(
^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 946, in run_server
async with build_async_engine_client(args) as engine_client:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 138, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 219, in build_async_engine_client_from_engine_args
engine_config = engine_args.create_engine_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1130, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1050, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 370, in init
self.multimodal_config = self._init_multimodal_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 431, in _init_multimodal_config
if ModelRegistry.is_multimodal_model(architectures):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 460, in is_multimodal_model
model_cls, _ = self.inspect_model_cls(architectures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 420, in inspect_model_cls
return self._raise_for_unsupported(architectures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 372, in _raise_for_unsupported
raise ValueError(
ValueError: Model architectures ['Mamba2ForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 02-25 08:13:21 registry.py:321] Error in inspecting model architecture 'Mamba2ForCausalLM'
ERROR 02-25 08:13:21 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 528, in _run_in_subprocess
ERROR 02-25 08:13:21 registry.py:321] returned.check_returncode()
ERROR 02-25 08:13:21 registry.py:321] File "/usr/lib/python3.12/subprocess.py", line 504, in check_returncode
ERROR 02-25 08:13:21 registry.py:321] raise CalledProcessError(self.returncode, self.args, self.stdout,
ERROR 02-25 08:13:21 registry.py:321] subprocess.CalledProcessError: Command '['/usr/bin/python3.12', '-m', 'vllm.model_executor.models.registry']' returned non-zero exit status 1.
ERROR 02-25 08:13:21 registry.py:321]
ERROR 02-25 08:13:21 registry.py:321] The above exception was the direct cause of the following exception:
ERROR 02-25 08:13:21 registry.py:321]
ERROR 02-25 08:13:21 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 319, in _try_inspect_model_cls
ERROR 02-25 08:13:21 registry.py:321] return model.inspect_model_cls()
ERROR 02-25 08:13:21 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 290, in inspect_model_cls
ERROR 02-25 08:13:21 registry.py:321] return _run_in_subprocess(
ERROR 02-25 08:13:21 registry.py:321] ^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 531, in _run_in_subprocess
ERROR 02-25 08:13:21 registry.py:321] raise RuntimeError(f"Error raised in subprocess:\n"
ERROR 02-25 08:13:21 registry.py:321] RuntimeError: Error raised in subprocess:
ERROR 02-25 08:13:21 registry.py:321] :128: RuntimeWarning: 'vllm.model_executor.models.registry' found in sys.modules after import of package 'vllm.model_executor.models', but prior to execution of 'vllm.model_executor.models.registry'; this may result in unpredictable behaviour
ERROR 02-25 08:13:21 registry.py:321] Traceback (most recent call last):
ERROR 02-25 08:13:21 registry.py:321] File "", line 198, in _run_module_as_main
ERROR 02-25 08:13:21 registry.py:321] File "", line 88, in _run_code
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 552, in
ERROR 02-25 08:13:21 registry.py:321] _run()
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 545, in _run
ERROR 02-25 08:13:21 registry.py:321] result = fn()
ERROR 02-25 08:13:21 registry.py:321] ^^^^
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 291, in
ERROR 02-25 08:13:21 registry.py:321] lambda: _ModelInfo.from_model_cls(self.load_model_cls()))
ERROR 02-25 08:13:21 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 294, in load_model_cls
ERROR 02-25 08:13:21 registry.py:321] mod = importlib.import_module(self.module_name)
ERROR 02-25 08:13:21 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 registry.py:321] File "/usr/lib/python3.12/importlib/init.py", line 90, in import_module
ERROR 02-25 08:13:21 registry.py:321] return _bootstrap._gcd_import(name[level:], package, level)
ERROR 02-25 08:13:21 registry.py:321] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 registry.py:321] File "", line 1387, in _gcd_import
ERROR 02-25 08:13:21 registry.py:321] File "", line 1360, in _find_and_load
ERROR 02-25 08:13:21 registry.py:321] File "", line 1331, in _find_and_load_unlocked
ERROR 02-25 08:13:21 registry.py:321] File "", line 935, in _load_unlocked
ERROR 02-25 08:13:21 registry.py:321] File "", line 999, in exec_module
ERROR 02-25 08:13:21 registry.py:321] File "", line 488, in _call_with_frames_removed
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mamba2.py", line 15, in
ERROR 02-25 08:13:21 registry.py:321] from vllm.model_executor.layers.mamba.mamba_mixer2 import (
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/mamba/mamba_mixer2.py", line 9, in
ERROR 02-25 08:13:21 registry.py:321] from vllm.attention.backends.flash_attn import FlashAttentionMetadata
ERROR 02-25 08:13:21 registry.py:321] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/flash_attn.py", line 25, in
ERROR 02-25 08:13:21 registry.py:321] from vllm.vllm_flash_attn import (flash_attn_varlen_func,
ERROR 02-25 08:13:21 registry.py:321] ImportError: cannot import name 'flash_attn_varlen_func' from 'vllm.vllm_flash_attn' (unknown location)
ERROR 02-25 08:13:21 registry.py:321]
ERROR 02-25 08:13:21 engine.py:400] Model architectures ['Mamba2ForCausalLM'] failed to be inspected. Please check the logs for more details.
ERROR 02-25 08:13:21 engine.py:400] Traceback (most recent call last):
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
ERROR 02-25 08:13:21 engine.py:400] engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
ERROR 02-25 08:13:21 engine.py:400] engine_config = engine_args.create_engine_config(usage_context)
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1130, in create_engine_config
ERROR 02-25 08:13:21 engine.py:400] model_config = self.create_model_config()
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1050, in create_model_config
ERROR 02-25 08:13:21 engine.py:400] return ModelConfig(
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 370, in init
ERROR 02-25 08:13:21 engine.py:400] self.multimodal_config = self._init_multimodal_config(
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 431, in _init_multimodal_config
ERROR 02-25 08:13:21 engine.py:400] if ModelRegistry.is_multimodal_model(architectures):
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 460, in is_multimodal_model
ERROR 02-25 08:13:21 engine.py:400] model_cls, _ = self.inspect_model_cls(architectures)
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 420, in inspect_model_cls
ERROR 02-25 08:13:21 engine.py:400] return self._raise_for_unsupported(architectures)
ERROR 02-25 08:13:21 engine.py:400] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-25 08:13:21 engine.py:400] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 372, in _raise_for_unsupported
ERROR 02-25 08:13:21 engine.py:400] raise ValueError(
ERROR 02-25 08:13:21 engine.py:400] ValueError: Model architectures ['Mamba2ForCausalLM'] failed to be inspected. Please check the logs for more details.
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 402, in run_mp_engine
raise e
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 391, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 119, in from_engine_args
engine_config = engine_args.create_engine_config(usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1130, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1050, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 370, in init
self.multimodal_config = self._init_multimodal_config(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 431, in _init_multimodal_config
if ModelRegistry.is_multimodal_model(architectures):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 460, in is_multimodal_model
model_cls, _ = self.inspect_model_cls(architectures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 420, in inspect_model_cls
return self._raise_for_unsupported(architectures)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py", line 372, in _raise_for_unsupported
raise ValueError(
ValueError: Model architectures ['Mamba2ForCausalLM'] failed to be inspected. Please check the logs for more details.
`

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!