Skip to content

Issues: open-compass/VLMEvalKit

Beta
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

95 Open 292 Closed
95 Open 292 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

测试api based model时preprocess_message_with_role()会造成system prompt无限叠加
#977 opened May 4, 2025 by vvirgooo2
1
Adding LiveXiv - a live benchmark
#976 opened May 3, 2025 by NimrodShabtay
torchrun多卡出现问题
#975 opened May 3, 2025 by mantou-cloud
多线程同时读取tsv文件时有线程读取不成功
#973 opened May 1, 2025 by vvirgooo2
Turn Off Saving Images in LMUData
#965 opened Apr 28, 2025 by moured
1
The inference on the AMBER dataset is very slow. WIP
#955 opened Apr 26, 2025 by pspdada
@MaoSong2022
6
OCRBenchV2 Feature Request Extra attention is needed
#949 opened Apr 25, 2025 by JVal123
1
Did not detect the .env file when using torchrun
#943 opened Apr 23, 2025 by Blissy-32
4
很多模型使用vlmevalkit评测不同的数据集时效果都有较大的差距
#941 opened Apr 23, 2025 by lanyuki
3
version of transfomers
#936 opened Apr 22, 2025 by The-Git-Rain
2
support for MV-Math Feature Request Extra attention is needed
#933 opened Apr 18, 2025 by SakurajimaMaiii
[bug] 反馈两个bug，模型和数据集用config.json配置会出现一些问题 WIP
#929 opened Apr 17, 2025 by gulixin0922
1
JanusPro
#928 opened Apr 17, 2025 by 125jx
1
Error in analyzing video datasets with Qwen2.5-VL-3B-Instruct
#927 opened Apr 16, 2025 by DanialTaheri
[Benchmark] Add support for CharXiv benchmark
#921 opened Apr 16, 2025 by MaoSong2022
2 of 3 tasks
Do you support customered models and local models on evaluation of image and video datasets?for example mistral-small-24b. If so, How to configure it?
#914 opened Apr 14, 2025 by 752191455
1
Qwen2.5-VL-3B-Instruct在很多评测数据集的指标与官方榜单差距较大
#907 opened Apr 11, 2025 by LIRENDA621
2
通过LMDeployAPI方式部署Qwen2-VL-7B-Instruct，测精度
#902 opened Apr 9, 2025 by simplew2011
Model load error in Ola model
#900 opened Apr 9, 2025 by zhizhou57
很多评测数据集的post_check函数报错
#898 opened Apr 9, 2025 by LIRENDA621
2
request to support skywork-r1v Feature Request Extra attention is needed
#892 opened Apr 2, 2025 by Orlando-CS
@FangXinyu-0913
1
The evaluation results of Qwen2.5-VL-7B on some benchmarks are inconsistent with that on the leaderboard
#884 opened Mar 28, 2025 by Lemok00
@kennymckormick
3
Running Video-MME with a different number of frames
#876 opened Mar 25, 2025 by Yazan-Almajnouni
@FangXinyu-0913
1
Qwen2.5vl 7b can't reproduce Mathvista_MINI like leaderboard
#874 opened Mar 25, 2025 by alchemistyzz
@FangXinyu-0913
3
Mathverse vision only score different from leaderboard
#869 opened Mar 24, 2025 by DietDietDiet
@FangXinyu-0913
5
ProTip! Find all open issues with in progress development work with linked:pr.