[yechank-nvidia]

Description

Define one SupportsMultimodal contract for model capabilities, encoder execution, item scheduling, fusion, and model-owned cache access. The executor uses this contract; MultimodalModelMixin provides the existing shared implementation. Independent implementations can opt in with a class marker.

Reuse the existing cache through public accessors, including the existing initialization hook. Accept Nemotron's read-only encoder-group property and align five encoder entry-point annotations with the mixin's Sequence input. Encoder computations, cache ownership, and tensor return values are preserved.

Developed independently on main (9619d5b2ee), without depending on batching #20897 or handoff #20902. Config-only support hooks match encoder-DP #17330 and attention-backend #20802; their defaults declare no support. Batching's encoded Params-view returns will be incorporated after it lands. The model onboarding guide documents these roles.

Validation

• 435 tests passed, covering item scheduling, cache/prefetch, startup memory profiling, decoder/encoder runners, and public API stability. Four tests added.
• mypy 1.19.1: production roots and static conformance of 13 concrete multimodal model types passed. The Protocol/shared-implementation check is included in the configured mypy roots.
• Class-marker checks passed for 17 model classes, including bases; Phi-4-MM retains its input-processor fallback. These checks load no model weights and do not establish new feature support.
• Pre-commit passed with type checking performed in the native container. Full configured mypy reports the same 12 existing sampler-test diagnostics as clean main 9619d5b2ee.
• Review confirmed unchanged encoder/fusion method bodies and no added tensor copies or host synchronization. E2E performance was not measured.

PR Checklist

• Reviewed the applicable PR checklist items.