Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

如何在实际应用中提升模型效率？ #3922

Open
mzgcz opened this issue Dec 3, 2024 · 1 comment
Open

如何在实际应用中提升模型效率？ #3922

mzgcz opened this issue Dec 3, 2024 · 1 comment
Labels
Question

Comments

@mzgcz
Copy link

mzgcz commented Dec 3, 2024

General Question

在实际应用中要怎样提升在线模型（Streaming）的效率呢？
语言模型可以通过batch size进行批量推理，来提升推理效率；可以使用多实例来应对推理请求并发的情况；可以使用TensorRT来优化推理速度。
请问对于PaddleSpeech在线模型，上面哪些措施是可行的，有没有更好的推荐？
@mzgcz mzgcz added the Question label Dec 3, 2024
@zxcd
Copy link
Collaborator

zxcd commented Dec 5, 2024

目前暂不支持多线程推理，但可以使用多实例

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mzgcz @zxcd