-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cerebras Integration #48
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
im so amazed with the inference speed. Great work cerebras, @henrytwo . Ive made few suggestions and also we dont need the autobuild notebook formatting in this. We can have a seperate PR.
cc @marklysze
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i've tested with the chess tool call example, a few general examples and marketing campaign use cases. the speed of the inference is super impressive. thanks @henrytwo.
Looks good to me.
* Organize some more modules * cleanup model_client
Why are these changes needed?
Add integration for Cerebras, which provides super low-latency, high speed LLM inference. Currently Llama 3.1-8B/70B are supported.
Tool calling examples have been provided in this PR, which also function with streaming. Additionally, token cost calculations have been implemented.
Checks