New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add onnx support #580

Open

davanstrien opened this issue May 10, 2022 · 0 comments

Assignees

Labels

Owner

davanstrien commented May 10, 2022 •

edited

Loading

Currently, onnx isn't used for inference of transformer models. Alongside using ray one approach to improve inference throughput again would be to:

download models
do the onnx export from the model
load using onnxrunntime
catch errors for unsupported models
do some initial validation of results?

Hopefully, we get some of this for free with Optimum if we wait a bit: https://github.com/huggingface/optimum

davanstrien added the enhancement label

davanstrien self-assigned this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment