-
Notifications
You must be signed in to change notification settings - Fork 242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle errors in forward pass, delete cache and cancel all sequences (Survive OOM) #98
Conversation
Code Metrics Report─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code Complexity ─────────────────────────────────────────────────────────────────────────────── Rust 46 16395 1158 702 14535 797 ─────────────────────────────────────────────────────────────────────────────── Total 46 16395 1158 702 14535 797 ─────────────────────────────────────────────────────────────────────────────── Estimated Cost to Develop 48,880 Estimated Schedule Effort 10.144342 months Estimated People Required 3.931182 ─────────────────────────────────────────────────────────────────────────────── Processed 557802 bytes, 0.558 megabytes (SI) ─────────────────────────────────────────────────────────────────────────────── |
Another alternative is to exit the process here https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-core/src/pipeline/mistral.rs#L418 Tokio won't finish the process on panic https://users.rust-lang.org/t/panic-in-tokio-task-does-not-end-the-program-execution/45731 |
Thanks for adding this, I like the idea of just raising an error instead of just |
37f558f
to
f9cdd16
Compare
Thank you! |
I was playing around with the codebase, adding a few dbg! around and seeing what goes on under the hood.
While doing so I stumbled upon an Out of Memory error. I was using a very long prompt, and it won't fit my GPU indeed.
When this happened, the server no longer worked, the process was still alive and GPU memory had not been freed.
This behavior can be simulated with panicking anywhere in the forward function of the pipeline: https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-core/src/pipeline/mistral.rs#L371
I changed the Pipeline trait to return errors on the forward pass instead of panicking. I handled the errors by setting all sequence states to error, deleting the GPU cache and sending an error message to the receiver.
This makes the server continue to work and frees all GPU memory, allowing new requests to go on normally.
@EricLBuehler What are your thoughts?
If this approach is any good, I'll make the errors OpenAI API compatible too.
Also, I noticed the Status Code of the error message was 200, not 4xx or 5xx (but that's a different issue #97)