Support nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16 (and nvidia/C-RADIOv2-H)

[netanel-haber]

Motivation

Support Multimodal nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL-BF16.

NVIDIA Nemotron Nano v2 12B VL model enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q&A and summarization capabilities.

Support its vision encoder: nvidia/CRadioV2-H.

Modifications

• Added python/sglang/srt/models/nano_nemotron_vl.py
• Added python/sglang/srt/configs/radio.py

Accuracy Tests

Reference was VLLM, with EVS turned off, temperature 0.
Both VideoMME and DocVQA run via VLMEvalKit.

• Video: VideoMME results on par with VLLM:
  • VideoMME_sglang.json
  • VideoMME_vllm.json
• Images: DocVQA_Val: 94.329, identical to VLLM.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[yhyang201]

Do you think it would be possible to open a separate PR for the changes made to the following files?

• python/sglang/test/gsm8k_mixin.py

• python/sglang/test/mmmu_vlm_mixin.py

• test/srt/models/test_vlm_models.py

This would help us streamline the review and move forward more quickly. Thanks!

[netanel-haber]

Do you think it would be possible to open a separate PR for the changes made to the following files?

Sure, I'll do so promptly.

[netanel-haber]

#13506 @yhyang201

[yhyang201]

Could you rebase and resolve the conflicts? Then we can run the CI tests — I believe we’ll be able to merge it soon.

[netanel-haber]

Could you rebase and resolve the conflicts? Then we can run the CI tests — I believe we’ll be able to merge it soon.

Done

[yhyang201]

Can you check the cause of the CI error?

config = config_class.from_pretrained(model, revision=revision)
  File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 640, in from_pretrained
    return cls.from_dict(config_dict, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py", line 808, in from_dict
    config = cls(**config_dict)
  File "/public_sglang_ci/runner-l3a-gpu-23/_work/sglang/sglang/python/sglang/srt/configs/nano_nemotron_vl.py", line 56, in __init__
    vision_auto_config = get_class_from_dynamic_module(
  File "/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py", line 604, in get_class_from_dynamic_module
    final_module = get_cached_module_file(
  File "/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py", line 467, in get_cached_module_file
    get_cached_module_file(
  File "/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py", line 427, in get_cached_module_file
    modules_needed = check_imports(resolved_module_file)
  File "/usr/local/lib/python3.10/dist-packages/transformers/dynamic_module_utils.py", line 260, in check_imports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: open_clip. Run `pip install open_clip`

[netanel-haber]

Can you check the cause of the CI error?
...
ImportError: This modeling file requires the following packages that were not found in your environment: open_clip. Run pip install open_clip

The commit I just pushed should entirely prevent that error from occuring.