Support to load ONNX APIs based on JS runtime #947
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey there 🫡
I've been working withing supabase team on supabase/edge-runtime#368.
Since
edge-runtime
is based on short-live sandboxed workers, the use oftransformers.js
was very expensive due the model caching and session state. At this point the original proposal was to replace all the JS libraries liketransformers.js
to a embedded runtime backend by rust.The Pipeline proposal
After some problems with
enc/dec
models I'd discuss with Supabase team, where @gregnr did the great suggestion of provide a more performative environment while keep using a well tested and maintained library like yours.So we decided to move on it and export our custom
ONNX runtime APIs
to theedge-runtime
environment, from thetransformers.js
side the only changed needed is to priority load theONNX
variable fromglobalThis
. We'd follow the same API ofonnxruntime-custom
package so everything should work as expected.The custom binding proposal
If you would like to test our runtime pls check my tranformers.js integration branch, at this file.
I also recommend to read the DEVELOPERS.md section to get more information about how to run the environment.
But basically just need to: