how to serve onnx based models on the web via rest api

[aria3ppp]

is there any least affort way to serve the models on the web? i mean llama.cpp have llama.server tool that deploy a openai api like rest api? is there any option here for onnxruntime?

[tarekziade]

That would be another project that wraps onnxruntime in a web service.

I know Triton has a backend for onnx https://github.com/triton-inference-server/onnxruntime_backend
And I saw this project on Github https://github.com/kibae/onnxruntime-server

But never tried them