DeepSpeed inference server

[bharatv007]

Is there support for a server like interface for Deepspeed, something on Nvidia triton server or any other serving support?
I could not find much in documentation or code, please point me if there is one.

[jeffra]

Hi @bharatv007, thank you for your question. We have a framework called MII that aims to achieve these goals. We are also actively working on adding additional features for MII, so keep an eye on the repo and our twitter account (https://twitter.com/MSFTDeepSpeed).

https://github.com/microsoft/deepspeed-mii

/cc @tohtana

[bharatv007]

Thank you!