[2/N] Harden Transformers modelling backend multi-modal path

[hmellor]

Continues from #51408

Added

• --mm-encoder-only and --limit-mm-per-prompt <modality>=0 now skip weights for this backend, via a _mark_model_components hook around AutoModel.from_config
• Audio encoders can be torch compiled; compile_mm_encoder was hardcoded to the image encoder

Fixed

• _get_prompt_updates returned None against a declared Sequence[PromptUpdate]
• The overridden apply() skipped the placeholder/item count validation the base class performs
• _split_embeddings silently padded or truncated encoder output to fit; it now warns

Changed

• _embed_audio/_embed_vision -> _process_audio_input/_process_image_input; embed_multimodal returns []
• get_max_image_size -> get_image_size_with_most_features returning ImageSize
• Modality detection deduplicated into _get_supported_modalities; dead _is_video_model removed
• Decoder and encoder lookups share one throwaway meta model instead of one each: 3 builds -> 1 per load

Notes

• HF's get_decoder/get_encoder are best-effort hasattr probes over a constructed model, with no class-level equivalent — hence the throwaway meta model
• They report the same class for image and video, so marking and compile decoration both dedupe by class. Otherwise image=0 could skip a tower video needs, and support_torch_compile would apply twice
• Supported modalities must be intersected before marking: get_limit_per_prompt returns 999 for unspecified ones, so all(limit == 0) would never hold
• Emu3 and granite-speech degrade to unmarked; HF can't distinguish their encoder from the model

Tests

• Added test_tower_weights_skipped_when_modality_disabled — nothing in the repo asserted weight skipping for any model
• Added test_marking_skipped_without_tokenizer

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[DarkLight1337]

/ci run

[github-actions]

✅ Triggered Buildkite CI #83134 for commit 7b6380fa80e3.