Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support to load ONNX APIs based on JS runtime #947

Merged
merged 3 commits into from
Sep 30, 2024

Conversation

kallebysantos
Copy link
Contributor

@kallebysantos kallebysantos commented Sep 26, 2024

Hey there 🫡
I've been working withing supabase team on supabase/edge-runtime#368.
Since edge-runtime is based on short-live sandboxed workers, the use of transformers.js was very expensive due the model caching and session state. At this point the original proposal was to replace all the JS libraries like transformers.js to a embedded runtime backend by rust.

The Pipeline proposal

image

After some problems with enc/dec models I'd discuss with Supabase team, where @gregnr did the great suggestion of provide a more performative environment while keep using a well tested and maintained library like yours.

So we decided to move on it and export our custom ONNX runtime APIs to the edge-runtime environment, from the transformers.js side the only changed needed is to priority load the ONNX variable from globalThis. We'd follow the same API of onnxruntime-custom package so everything should work as expected.

The custom binding proposal

image


If you would like to test our runtime pls check my tranformers.js integration branch, at this file.
I also recommend to read the DEVELOPERS.md section to get more information about how to run the environment.
But basically just need to:

$ ./scripts/run_dind.sh

$ curl --request POST \
  --url http://localhost:9998/ort-rust-backend \
  --header 'Content-Type: application/json' \
  --data '{
	"input": [
		"one day I will see the world",
		"I will never forgive u"
	]
}'

- Allows custom JS runtimes to expose their own ONNX API.
@xenova xenova changed the base branch from main to v3 September 27, 2024 13:11
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@xenova
Copy link
Collaborator

xenova commented Sep 27, 2024

Thanks for the PR and additional info! I've rebased to Transformers.js v3 and will ensure this is included in the next release of @huggingface/transformers.

@xenova xenova self-requested a review September 27, 2024 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants