ollama: Add needed rocm libraries to rocmPath

[ulrikstrid]

Description of changes

Since version 0.1.29 more ROCm libraries are needed in the environment.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[ulrikstrid]

Result of nixpkgs-review pr 297447 run on x86_64-linux 1

[marcusramberg]

What does this fix? 0.1.29 superficially seems to be working fine already?

[ulrikstrid]

What does this fix? 0.1.29 superficially seems to be working fine already?

It's currently broken on master when building with rocm which you can test running this command:
nix-build --expr '(import ./. {}).ollama.override { acceleration = "rocm"; }'

[abysssol]

@ulrikstrid Thank you for this. It seems strange that ollama lost the ability to detect libraries exposed through buildInputs, but I'm glad it's this simple to get ollama to build again.

Unfortunately, this isn't a complete fix. While ollama seems to build correctly now, it doesn't detect the rocm libraries at runtime. So, it still can't use the gpu, and falls back to cpu mode. I did try adding all the rocm libs to the wrapper script, but that didn't do anything.

This is good progress, and I'm happy to see it. It looks ready to merge to me.

[drupol]

Please update the PR title and use Ollama in lower case. Same for the commit log message.

That typo is the reason why the CI is not triggering the build and tests.

Thanks

[ulrikstrid]

I changed the commit, I'm going to play a bit with the the environment to see if I can make it run on the GPU. If not this should be OK to merge and I can create a followup PR if I figure it out.

[ulrikstrid]

According to this comment we need rocm v6: ollama/ollama#738 (comment)
Since this at least unblocks the build maybe it's worth merging while we wait for #287846 (which looks good to me).

Is that OK by you @drupol ?

[drupol]

Of course ! Thank you !

[dchambers]

I tried using nix-shell to run the previous version of ollama (0.1.28) to get around this issue, like so:

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/af67cfa87bed7fd62812aa888cd134412c9e6d75.tar.gz -p ollama
ollama serve

While this doesn't have the same rocm v6 issue, it doesn't seem to find the rocm libraries either AFAICT. Here's what I see:

HSA_OVERRIDE_GFX_VERSION=10.3.0 ollama serve
time=2024-04-06T22:32:39.267+01:00 level=INFO source=images.go:710 msg="total blobs: 5"
time=2024-04-06T22:32:39.267+01:00 level=INFO source=images.go:717 msg="total unused blobs removed: 0"
time=2024-04-06T22:32:39.268+01:00 level=INFO source=routes.go:1021 msg="Listening on 127.0.0.1:11434 (version 0.1.28)"
time=2024-04-06T22:32:39.268+01:00 level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..."
time=2024-04-06T22:32:39.290+01:00 level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [cpu_avx cpu_avx2 cpu]"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=gpu.go:94 msg="Detecting GPU type"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: []"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=gpu.go:265 msg="Searching for GPU management library librocm_smi64.so"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=gpu.go:311 msg="Discovered GPU libraries: []"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-04-06T22:32:39.290+01:00 level=INFO source=routes.go:1044 msg="no GPU detected"

Might this be because nix-shell doesn't have access to the nixpkgs.config.rocmSupport = true; config that I have in my configuration.nix?

[abysssol]

Might this be because nix-shell doesn't have access to the nixpkgs.config.rocmSupport = true; config that I have in my configuration.nix?

@dchambers yes, it compiled without gpu support. I never added support for reading rocmSupport because I was unable to get it to do anything. I now see that I should have set nixpkgs.config.rocmSupport = true; to test it.

Is there anything stopping you from using the current version of ollama (0.1.30) that's in nixpkgs-unstable? As far as I know, rocm is working correctly now. If 0.1.30 compiled with acceleration = "rocm"; set doesn't use your gpu, please open an issue with details and ping me.

[dchambers]

Thanks so much @abysssol, that did it! 🎉

In case any other NixOS noobs come this way in the future, here's the config I added to my configuration.nix to make this work:

  nixpkgs.overlays = [
    (final: prev: {
      ollama = prev.ollama.override { acceleration = "rocm"; };
    })
  ];

[abysssol]

@dchambers I'm not sure if I'm missing some important context or something, but shouldn't the nixos service be the most convenient way to use ollama?

services.ollama = {
  enable = true;
  acceleration = "rocm";
  # optional: if using stable nixos, you can override the stable, outdated version with the unstable version
  package = unstable.ollama;
};

[dchambers]

@dchambers I'm not sure if I'm missing some important context or something, but shouldn't the nixos service be the most convenient way to use ollama?

Nice, I had no idea you could do that! Thanks again for your help @abysssol 🙇