New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

whisper 支持in-flight batch 了吗 #710

Open

wangjxxxhi12 opened this issue Feb 13, 2025 · 2 comments

wangjxxxhi12 commented Feb 13, 2025

看代码好像不支持，但是好像有一个in-flight 的commit

Author

wangjxxxhi12 commented Feb 14, 2025

看起来确实是还没有实现，@yuekaizhang 有计划实现吗

Collaborator

yuekaizhang commented Feb 14, 2025

现在的代码是支持的哦，默认是 python bindings 版本的 infligh batch, cpp 版本的 infligh batch 在这里 https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/docs/whisper.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment