New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问推理VLM模型时，显存是怎么分配的？ #2896

Open

wuxianyess opened this issue Dec 13, 2024 · 1 comment

Assignees

wuxianyess commented Dec 13, 2024 •

edited

Loading

lmdeploy单卡部署了一个14b vlm模型，用openai的api调用，单次上传的图片数量可以到近百张，请问显存是怎么分配的，还是api调用有什么需要注意的？

wuxianyess changed the title ~~请问推理图文模型时，显存是怎么分配的？~~ 请问推理VLM模型时，显存是怎么分配的？

lvhan028 assigned irexyc

Collaborator

irexyc commented Dec 18, 2024

默认提特征时，单次处理一个图片，所以 vision 模型的显存并不会根据图片数量线性增长。

100张图你是指一个请求还是并发请求中图片共100张？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment