Fix LoRA adapter silently failing on Pixtral/Ministral-3 models

[timon0305]

Purpose

Fix LoRA adapters silently having no effect when loaded on Pixtral-based models
(e.g. mistralai/Ministral-3-8B-Instruct-2512).

The root cause is that PixtralForConditionalGeneration is missing the
hf_to_vllm_mapper attribute that other multimodal models (like
Mistral3ForConditionalGeneration) define. Without this mapper, LoRA weight
names from HuggingFace checkpoints (e.g. model.language_model.model.layers.…)
are not translated to vLLM's internal names (e.g. language_model.model.layers.…),
so no LoRA modules match and the adapter is silently ignored.

Changes:

• Add hf_to_vllm_mapper to PixtralForConditionalGeneration (same mapping as
  Mistral3ForConditionalGeneration)
• Add a warning in LoRAModelManager.activate_adapter() when none of the adapter
  weights match any LoRA-eligible module
• Add a warning in WorkerLoRAManager._load_adapter() when loaded LoRA module
  names don't match any model module

Fixes #34591

Test Plan

• Verify that LoRA adapters trained against Mistral3ForConditionalGeneration in
  HuggingFace format correctly apply when served via
  PixtralForConditionalGeneration
• Verify that the new warnings fire when an adapter is loaded but no weights
  match (e.g. by intentionally removing the mapper)

Test Result

Pre-commit hooks all pass (ruff check, ruff format, mypy, typos, etc.)

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results

[gemini-code-assist]

Code Review

The pull request correctly identifies and addresses the silent failure of LoRA adapters on Pixtral models by adding a WeightsMapper and defensive warnings. However, the mapping prefixes in pixtral.py appear to mismatch the actual attribute names used in the model implementation, which will likely prevent the fix from working as intended. Additionally, a redundant and potentially inefficient check was added to the worker manager.

[gemini-code-assist]

This check is redundant because a more precise check is performed in LoRAModelManager.activate_adapter (which is called immediately after loading in add_adapter). Furthermore, calling model.named_modules() to build a set on every adapter load is inefficient for large models like Pixtral (12B). If an early check is desired, it should be performed against self._adapter_manager.modules, which contains the actual LoRA-eligible module names.

[gemini-code-assist]

The target prefixes in the WeightsMapper do not match the actual attribute names defined in PixtralForConditionalGeneration. Specifically, vision_tower should be vision_encoder and multi_modal_projector should be vision_language_adapter. Additionally, based on the PR description, mapping model.language_model. to language_model.model. will result in an extra model. prefix (e.g., language_model.model.model.layers...), as the internal vLLM path is language_model.model.layers....

Suggested change

	hf_to_vllm_mapper = WeightsMapper(
	orig_to_new_prefix={
	"model.language_model.": "language_model.model.",
	"model.vision_tower.": "vision_tower.",
	"model.multi_modal_projector.": "multi_modal_projector.",
	"lm_head.": "language_model.lm_head.",
	}
	)
	hf_to_vllm_mapper = WeightsMapper(
	orig_to_new_prefix={
	"model.language_model.": "language_model.",
	"model.vision_tower.": "vision_encoder.",
	"model.multi_modal_projector.": "vision_language_adapter.",
	"lm_head.": "language_model.lm_head.",
	}
	)

[jeejeelee]

These warnings may be misleading, please delete them

[jeejeelee]

Thank you for your contribution! Since #36963 is a superset of this PR, I'll go ahead and close this one.