[Fix] Address remaining issues of supporting MiniCPMV

[mickqian]

Motivation

Address remaining issues of #2785

Modifications

1. Update document of implementing a new vision-llm
2. Add some test for comparing logits output of SGLang and HF
3. Code cleanup

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• [] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling.

[zhaochenyang20]

Great work! Once ready, please ask us to review.

[merrymercy]

also remove these vllm dependency

sglang/python/sglang/srt/layers/attention/vision.py

Lines 8 to 9 in 2584f6d

	from vllm.distributed import parallel_state
	from vllm.distributed import utils as dist_utils

[zhaochenyang20]

@mickqian mick, great work. So we are ended and just need to past the review? Thanks!

[yizhang2077]

@mickqian there are still some failed cases, but it seems they are not related to your pr?

[zhaochenyang20]

@mickqian there are still some failed cases, but it seems they are not related to your pr?

Yeah. CI is wrong possibly due to kernel or largely refactor.

[yizhang2077]

Yeah. CI is wrong possibly due to kernel or largely refactor.

Since test_vision_*.py has been all passed, I think we can approve this pr