[MM] Allow skipping memory profiling for multimodal models.

[ywang96]

Purpose

Memory profiling for multimodal models involves processes input dummy data into features, then encode these features into embeddings and store them. Since most LMMs have a relatively small encoder, sometimes users may want to tune down --gpu-memory-utilization themselves and skip this profiling for faster startup time (especially for RL scenario).

This PR adds --skip-mm-profiling option for users to achieve so.

Test Plan

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct

INFO 08-15 01:42:53 [core.py:199] init engine (profile, create kv cache, warmup model) took 26.52 seconds

Command: vllm serve Qwen/Qwen2.5-VL-7B-Instruct --skip-mm-profiling

INFO 08-15 01:47:42 [core.py:199] init engine (profile, create kv cache, warmup model) took 20.86 seconds

Test Result

(Optional) Documentation Update

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request introduces a --skip-mm-profiling flag to accelerate engine startup for multimodal models by bypassing memory profiling. The implementation correctly propagates this new configuration from the command line to the model runners. However, I've identified a critical issue in both gpu_model_runner.py and tpu_model_runner.py where the code unconditionally accesses model_config.multimodal_config. This will lead to an AttributeError and crash the engine when running non-multimodal models, as multimodal_config will be None. I have provided code suggestions to fix this by ensuring the attribute is only accessed for multimodal models.

[huachenheli]

I think we can make the code more readable:

        if self.skip_mm_profiling:
            logger.info("Skipping memory profiling for multimodal encoder and "
                        "encoder cache.")
        else:
            # self.supports_mm_inputs already checked above.
            ...

[ywang96]

Note that self.skip_mm_profiling is defined to be True if the flag is set and self.supports_mm_inputs=True - this means you will have to check again anyways if self.supports_mm_inputs is True in that else gate.

The reason for this definition is that this particular check here is really for showing the message when users explicitly turned off profiling for a model that takes mm inputs, otherwise we will be showing this message for text-only models that could be a bit confusing imo.

[ywang96]

See my change in fd90599 - probably more readable now?

[huachenheli]

I'm a big fan of "positive flag names". Empirically, enable_xyz=False causes less cognitive overhead compared to disable_xyz=True. Maybe we can set this instead as enable_mm_profiling and default it to True instead?

[ywang96]

We tried that with a few things with positive flag actually but ends up having a bit messy situation IMO (we have --enable-prefix-caching and --no-enable-prefix-caching)

Personally, I actually think vllm serve model_name --skip-mm-profiling is more intuitive than vllm serve model_name --enable-mm-profiling=False or vllm serve model_name --no-enable-mm-profiling, and is more consistent with other negative flags in we have (e.g, disable_sliding_window, skip_tokenizer_init, etc) when we want the positive flag to be the default behavior. What do you think?

[DarkLight1337]

I think in terms of user perspective it is better to use --skip- rather than --no-enable-. But maybe we can adjust the argument parser to support both ways while keeping a positive name for the Python variable

[huachenheli]

Ah okay. If the underlying parser is using the "--no-enable-xyz" style instead of "--enable-xyz=false" (similar to how c++ gflags work) then I guess "skip" is indeed cleaner.

[hmellor]

--arg and --no-arg is actually built in Python behaviour, so we adopted it to he more standard from a Python perspective.

https://docs.python.org/3/library/argparse.html#argparse.BooleanOptionalAction

[ywang96]

--arg and --no-arg is actually built in Python behaviour, so we adopted it to he more standard from a Python perspective.

Yea - I meant more like having default behavior to be an explicit positive flag (instead of just having --disable-prefix-caching) seems a bit weird to me.

[hmellor]

Yeah I agree, I was just providing context :)

[huachenheli]

Do we need a stream synchronize here? The TPU version seems to explicitly doing a sync. Is that not needed for cuda?

[ywang96]

I'm not too familiar with why TPU did that but at least on cuda the two models are running on the same stream (this is by design since we don't want encoder to affect decoder implicitly in any possible way)