New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Genny CUDA memory exception #939

Open

lilhoser opened this issue Sep 27, 2024 · 1 comment

lilhoser commented Sep 27, 2024

Describe the bug
Genny using onnxruntimegenai.cuda package fails to load https://huggingface.co/microsoft/Phi-3-mini-128k-instruct-onnx

To Reproduce
Steps to reproduce the behavior:

Run genny with Debug_Cuda configuration selected.
Open C:\Downloads\models\Phi-3-mini-128k-instruct-onnx\cuda\cuda-int4-rtn-block-32 or C:\Downloads\models\Phi-3-mini-128k-instruct-onnx\cuda\cuda-fp16
Type a message, hit enter
See error

Expected behavior

Is this model supported?

Screenshots

Desktop (please complete the following information):

OS: Windows 11

The text was updated successfully, but these errors were encountered:

MaxAkbar commented Sep 28, 2024

I had the same issue, I think it's an issue with the fp16. I can run it on Phi-3.5-mini-instruct-cuda-fp32-onnx.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment