[VLM] Support cos sin cache for Ernie4.5-VL

[yuan-luo]

Motivation

This PR refactors the rotary positional embedding implementation to expose an explicit cos/sin cache interface and reuse it in the 2D vision RoPE code path. Instead of recomputing frequencies and calling cos() / sin() repeatedly, we precompute and cache the 1D cos/sin values once, then index into this cache for both text RoPE and the 2D grid RoPE used by the vision encoder.
The framework has already been implemented in #15205 , this PR is to enforce this enhancement into more VLMs.

Performance improved TTFT 3%.

Modifications

Accuracy Tests

lmms_eval no drops:

Main:

root@c7e9bb6a6789:/sgl-workspace/sglang# python3 -m lmms_eval --model openai_compatible   --model_args model_version=baidu/ERNIE-4.5-VL-28B-A3B  --tasks mmmu_val   --batch_size 16

| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|Stderr_CLT|Stderr_Clustered|
|--------|------:|------|-----:|--------|---|-----:|---|------|----------|----------------|
|mmmu_val|      0|none  |     0|mmmu_acc|↑  |0.2644|±  |N/A   |N/A       |N/A             |

PR:

➜  bench_script python3 -m lmms_eval --model openai_compatible   --model_args model_version=baidu/ERNIE-4.5-VL-28B-A3B   --tasks mmmu_val   --batch_size 16
2026-03-03 04:14:29 | INFO     | __main__:cli_evaluate:476 - Verbosity set to INFO
2026-03-03 04:14:32 | INFO     | __main__:cli_evaluate_single:565 - Evaluation tracker args: {}
2026-03-03 04:14:32 | INFO     | __main__:cli_evaluate_single:649 - Selected Tasks: ['mmmu_val']
2026-03-03 04:14:32 | INFO     | lmms_eval.evaluator:simple_evaluate:170 - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2026-03-03 04:14:33 | INFO     | lmms_eval.evaluator:evaluate:515 - Running on rank 0 (local rank 0)
2026-03-03 04:14:33 | INFO     | lmms_eval.api.task:build_all_requests:428 - Building contexts for mmmu_val on rank 0...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 900/900 [00:00<00:00, 13835.13it/s]
2026-03-03 04:14:33 | INFO     | lmms_eval.evaluator:evaluate:609 - Running generate_until requests
Model Responding: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍| 897/900 [07:59<00:01,  1.89it/s]2026-03-03 04:22:33 | INFO     | lmms_eval.models.model_utils.gen_metrics:log_metrics:136 - Metric summary - Total elapsed time: 14580.099s, Total gen tokens: 112510, Avg speed: 7.7 tokens/s
Model Responding: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 900/900 [07:59<00:00,  1.88it/s]
Postprocessing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 900/900 [00:00<00:00, 8850.28it/s]
{'Overall-Art and Design': {'num': 120, 'acc': 0.33333}, 'Art': {'num': 30, 'acc': 0.23333}, 'Art_Theory': {'num': 30, 'acc': 0.23333}, 'Design': {'num': 30, 'acc': 0.5}, 'Music': {'num': 30, 'acc': 0.36667}, 'Overall-Business': {'num': 150, 'acc': 0.24}, 'Accounting': {'num': 30, 'acc': 0.23333}, 'Economics': {'num': 30, 'acc': 0.3}, 'Finance': {'num': 30, 'acc': 0.13333}, 'Manage': {'num': 30, 'acc': 0.2}, 'Marketing': {'num': 30, 'acc': 0.33333}, 'Overall-Science': {'num': 150, 'acc': 0.20667}, 'Biology': {'num': 30, 'acc': 0.2}, 'Chemistry': {'num': 30, 'acc': 0.1}, 'Geography': {'num': 30, 'acc': 0.3}, 'Math': {'num': 30, 'acc': 0.2}, 'Physics': {'num': 30, 'acc': 0.23333}, 'Overall-Health and Medicine': {'num': 150, 'acc': 0.27333}, 'Basic_Medical_Science': {'num': 30, 'acc': 0.26667}, 'Clinical_Medicine': {'num': 30, 'acc': 0.23333}, 'Diagnostics_and_Laboratory_Medicine': {'num': 30, 'acc': 0.3}, 'Pharmacy': {'num': 30, 'acc': 0.3}, 'Public_Health': {'num': 30, 'acc': 0.26667}, 'Overall-Humanities and Social Science': {'num': 120, 'acc': 0.26667}, 'History': {'num': 30, 'acc': 0.4}, 'Literature': {'num': 30, 'acc': 0.36667}, 'Sociology': {'num': 30, 'acc': 0.13333}, 'Psychology': {'num': 30, 'acc': 0.16667}, 'Overall-Tech and Engineering': {'num': 210, 'acc': 0.28571}, 'Agriculture': {'num': 30, 'acc': 0.26667}, 'Architecture_and_Engineering': {'num': 30, 'acc': 0.46667}, 'Computer_Science': {'num': 30, 'acc': 0.23333}, 'Electronics': {'num': 30, 'acc': 0.03333}, 'Energy_and_Power': {'num': 30, 'acc': 0.36667}, 'Materials': {'num': 30, 'acc': 0.23333}, 'Mechanical_Engineering': {'num': 30, 'acc': 0.4}, 'Overall': {'num': 900, 'acc': 0.26667}}
fatal: not a git repository (or any of the parent directories): .git
2026-03-03 04:22:33 | INFO     | lmms_eval.loggers.evaluation_tracker:save_results_aggregated:238 - Output path not provided, skipping saving results aggregated
openai_compatible (model_version=baidu/ERNIE-4.5-VL-28B-A3B), gen_kwargs: (), limit: None, offset: 0, num_fewshot: None, batch_size: 16
| Tasks  |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|--------|------:|------|-----:|--------|---|-----:|---|------|
|mmmu_val|      0|none  |     0|mmmu_acc|↑  |0.2667|±  |N/A   |

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[yuan-luo]

/tag-and-rerun-ci

[yuan-luo]

/rerun-failed-ci

[yuan-luo]

/rerun-failed-ci