Manually downloading a LLM

[ekuznetsov139]

Software

Desktop Application

Operating System / Platform

Linux

Your Pieces OS Version

Latest

Early Access Program

• Yes, this is related to an Early Access Program feature.

Kindly describe the bug and include as much detail as possible on what you were doing so we can reproduce the bug.

I am attempting to install Pieces with a local LLM. Unfortunately, I have an unstable network connection (which is why I want a local model in the first place.) I am trying to get Mistral-7B.

I found that Pieces wants to download the whole model (4 GB) all at once. If it fails for whatever reason, it restarts from the beginning. This is annoying and it makes downloading virtually impossible, because my network connection does not stay long enough to pull 4GB in one go.

I have figured out where the file is being downloaded to and from, and managed to download it manually with wget. Now the problem is that Pieces refuses to recognize it. Even if I put it in the right place, the Pieces app stil shows the Download button in the UI, and clicking Download truncates the file to zero and starts the download from the beginning.

I tried to edit models.db and change "downloaded" to true in the relevant line, but it has no effect.

Is there anything else I could do?

[hra42]

I don't think you could import a model you downloaded manually. The problem is that pieces doesn't just download the raw model, but rather how the integration works.

@mark-at-pieces might be able to talk about this further.

[pieces-support-bot]

Hi @ekuznetsov139 - Thank you so much for creating this issue. Your issue has been automatically triaged and routed to the proper Pieces team member. Look for a follow-up within the next 24 hours.

Based on the details you provided, you might find these related issues helpful:

• Bug: unable to cancel the download of an LLLM
• Error Encountered with Pieces App - "Something went wrong with processing"
• Onboarding doesn't generate pieces nor demo chats

In the meantime, please check out these helpful resources:

• Existing Issues: https://github.com/pieces-app/support/issues
• Documentation: https://docs.pieces.app
• YouTube Channel: https://www.youtube.com/@getpieces

We appreciate your patience and contribution to making Pieces better!

[brian-at-pieces]

Hey @ekuznetsov139 sorry you're having this issue. The reason we delete the model on a failed download is becuase we're unable to determine whether the partial download was corrupted or not, so we had to fully delete to ensure success.

Nice try on updating the db property, but that won't be enough. Try this:

1. Shutdown Pieces
2. Add an empty "download_success.txt" to the mistral directory in com.pieces.os/llm_chat_models Also check that your directory structure looks like the attached screenshot
3. Restart Pieces

We actually have a new release coming the next day or two that completely overhauls our on-device LLM infrastructure - including the ability to restart a failed download from where it left off. If you can hang on until then your problem will be resolved.

[ekuznetsov139]

Thanks. I have been able to install Mistral-7B as per these instructions. Unfortunately, it did not work: the model showed as "active", but, if I tried to make any query, the app sat thinking for a minute or two and then Pieces OS crashed. For the period of time when it was thinking. I was observing zero GPU or GPU memory usage (I have a mobile 4070 with 8 GB of memory). I have Ubuntu 24 with all the latest updates, vulkan installed, etc. I looked through the logs but did not see anything relevant. I was able to get good responses from cloud LLMs, it was only the local LLM that was malfunctioning.

I tried to download a different model to see if the failure is specific to mistral (I saw an open ticket, issue #112 saying that that specific model had problems), but, before I could test it, Pieces got updated to the newest version (I guess that is the one you were referring to?) Now it can't use local models at all, because, if I go to the UI page, it asks to install ollama, and if I do that, it immediately says "Ollama failed to install".

I tried to uninstall and reinstall pieces-os, it did not help.

In the logs, I see the following:

I can confirm that there is no ollama subfolder in /home/eugene/Documents/com.pieces.os/production.

Do you want a separate ticket for this issue?

[brian-at-pieces]

@ekuznetsov139 really sorry for all the trouble here. I've opened a new issue for the failure to install and I'm currently investigating. In the meantime, if you download Ollama yourself Pieces should be able to utilize it: https://ollama.com/download/linux

I'll get back to you ASAP. Thanks for the patience