python3Packages.vllm: 0.3.3->0.5.2

[cfhammill]

Description of changes

Updates vllm. Requires two new python packages, lm-format-enforcer and prometheus-fastapi-instrumentator, this PR adds both.

I was able to get the cuda version to build on a slightly older revision of nixpkgs. Would have needed to rebuild the entire python/pytorch/cuda stack to try against master. Compiling vllm alone takes > 1h so this was slow going. Also have not built against rocm.

@CertainLach you mentioned that this need a special triton? It compiles ok for me with the version I fixed in #325843.

It compiles and can be imported, but since tests have not been enabled (and likely wouldn't work anyway) unsure if this works as expected.

CC @SomeoneSerge @ConnorBaker @samuela

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[CertainLach]

@CertainLach you mentioned that this need a special triton? It compiles ok for me with the version I fixed in #325843.

It was failing at runtime without new triton for me, I still maintain fork of nixpkgs with updated triton because of that.
Not sure if they have fixed this version requirement now, will check it on this PR later

[cfhammill]

I think I've addressed everything, it should be good to go @samuela @SomeoneSerge @CertainLach.

Let me know if you want me to squash the commits.

[SomeoneSerge]

Result of nixpkgs-review pr 328258 --extra-nixpkgs-config '{ cudaSupport = true; cudaCapabilities = [ "7.0" "8.0" "8.6" ]; cudaEnableForwardCompat = false; allowUnfreePredicate = p: builtins.all ( license: license.free || builtins.elem license.shortName [ "CUDA EULA" "cuDNN EULA" "cuTENSOR EULA" "NVidia OptiX EULA" ] ) (if builtins.isList p.meta.license then p.meta.license else [ p.meta.license ]); permittedInsecurePackages = [ "electron-24.8.6" "zotero-6.0.27" ]; }' run on x86_64-linux 1

4 packages marked as broken and skipped:

• python311Packages.vllm
• python311Packages.vllm.dist
• python312Packages.vllm
• python312Packages.vllm.dist

8 packages built:

• python311Packages.lm-format-enforcer
• python311Packages.lm-format-enforcer.dist
• python311Packages.prometheus-fastapi-instrumentator
• python311Packages.prometheus-fastapi-instrumentator.dist
• python312Packages.lm-format-enforcer
• python312Packages.lm-format-enforcer.dist
• python312Packages.prometheus-fastapi-instrumentator
• python312Packages.prometheus-fastapi-instrumentator.dist

[cfhammill]

thanks @SomeoneSerge, any idea why the vllm build is marked broken? Your config should have had cuda enabled

[SomeoneSerge]

┃ 
┃        error: Package ‘python3.12-outlines-0.0.45’ in /home/ss/.cache/nixpkgs-review/pr-328258-1/nixpkgs/pkgs/development/python-modules/outlines/default.nix:56 is marked as broken, refusing to evaluate.
┃ 
┃        a) To temporarily allow broken packages, you can use an environment variable
┃           for a single invocation of the nix tools.
┃ 

broken on master (Ofborg rebases PRs on master before running any checks)

[cfhammill]

outlines was broken in ff7d46d by an untested version bump, and marked as broken a few days ago when someone noticed.

I fixed and tacked it on to this PR.

[cfhammill]

A new version of vllm has been released, seems like a small one so hopefully no need to modify the build other than bump version and hash. Trying now.

[SomeoneSerge]

Result of nixpkgs-review pr 328258 --extra-nixpkgs-config '{ cudaSupport = true; cudaCapabilities = [ "7.0" "8.0" "8.6" ]; cudaEnableForwardCompat = false; allowUnfreePredicate = p: builtins.all ( license: license.free || builtins.elem license.shortName [ "CUDA EULA" "cuDNN EULA" "cuTENSOR EULA" "NVidia OptiX EULA" ] ) (if builtins.isList p.meta.license then p.meta.license else [ p.meta.license ]); permittedInsecurePackages = [ "electron-24.8.6" "zotero-6.0.27" ]; }' run on x86_64-linux 1

8 packages marked as broken and skipped:

• python311Packages.outlines
• python311Packages.outlines.dist
• python311Packages.vllm
• python311Packages.vllm.dist
• python312Packages.outlines
• python312Packages.outlines.dist
• python312Packages.vllm
• python312Packages.vllm.dist

12 packages built:

• python311Packages.lm-format-enforcer
• python311Packages.lm-format-enforcer.dist
• python311Packages.prometheus-fastapi-instrumentator
• python311Packages.prometheus-fastapi-instrumentator.dist
• python311Packages.pyairports
• python311Packages.pyairports.dist
• python312Packages.lm-format-enforcer
• python312Packages.lm-format-enforcer.dist
• python312Packages.prometheus-fastapi-instrumentator
• python312Packages.prometheus-fastapi-instrumentator.dist
• python312Packages.pyairports
• python312Packages.pyairports.dist

[cfhammill]

sorry, removing the broken annotation on outlines (I'm working across two branches [a semi-stable one that builds and works for me and includes a vllm patch for MIG support, and this one] and I'm evidently failing to merge/cherrypick properly, apologies!)

[SomeoneSerge]

evidently failing to merge/cherrypick properly, apologies!)

No worries. It's just

git fetch origin # assuming origin refers to git@github.com:NixOS/nixpkgs
git rebase -i origin/master

Also you might want to look into lazygit

[cfhammill]

Thanks @SomeoneSerge. Turns out this was not the issue I thought, this PR does have the broken annotation removed, not sure how it got reintroduced, checking now.

[bcdarwin]

Made some largely cosmetic suggestions, but please do squash into commits for vllm, outlines, and pyairports.

[cfhammill]

will squash shortly, thanks @bcdarwin

[SomeoneSerge]

Side note: torch in nativeBuildInputs looks wrong

[cfhammill]

yeah, not sure why that was put there.

[bcdarwin]

Maybe one of those 'let the user bring their own torch' ideas a la the current torchmetrics?

[SomeoneSerge]

Maybe one of those 'let the user bring their own torch' ideas a la the current torchmetrics?

I rather meant that vllm likely needs the host platform's torch, not the build platform's

[cfhammill]

ok - I think this torch can be removed, torch is already in propagatedBuildInputs, shouldn't be needed twice.

[SomeoneSerge]

Result of nixpkgs-review pr 328258 --extra-nixpkgs-config '{ cudaSupport = true; cudaCapabilities = [ "7.0" "8.0" "8.6" ]; cudaEnableForwardCompat = false; allowUnfreePredicate = p: builtins.all ( license: license.free || builtins.elem license.shortName [ "CUDA EULA" "cuDNN EULA" "cuTENSOR EULA" "NVidia OptiX EULA" ] ) (if builtins.isList p.meta.license then p.meta.license else [ p.meta.license ]); permittedInsecurePackages = [ "electron-24.8.6" "zotero-6.0.27" ]; }' run on x86_64-linux 1

2 packages failed to build:

• python312Packages.vllm
• python312Packages.vllm.dist

14 packages built:

• python311Packages.lm-format-enforcer
• python311Packages.lm-format-enforcer.dist
• python311Packages.outlines
• python311Packages.outlines.dist
• python311Packages.pyairports
• python311Packages.pyairports.dist
• python311Packages.vllm
• python311Packages.vllm.dist
• python312Packages.lm-format-enforcer
• python312Packages.lm-format-enforcer.dist
• python312Packages.outlines
• python312Packages.outlines.dist
• python312Packages.pyairports
• python312Packages.pyairports.dist

[cfhammill]

@SomeoneSerge do you have the build logs for python312Packages.vllm?

[SomeoneSerge]

I had to re-run nixpkgs-review because logs got GC-ed, but here: https://gist.github.com/SomeoneSerge/6a3344368c146171160a92d6865ffd57

TLDR: upstream marked vllm as only supporting up to 3.11; I highly doubt that was intentional though

[SomeoneSerge]

https://github.com/vllm-project/vllm/blob/af647fb8b3ea9d910f7d1bc104af8986d048a8e2/CMakeLists.txt#L17

I'm tempted to patch this out and either replace the list with the current interpreter's version or replace the whole thing with an unwrapped find_package

[SomeoneSerge]

So uh I'm not a vllm user, you probably know better: is there value in providing python311Packages.vllm but not python3Packages.vllm? I'm content to merge the PR (as is plus the disabled = ... attribute) since it's upstream who decided not to support 3.12

[cfhammill]

I'm using python311 so not having 312 doesn't bother me personally, but it looks like there isn't a good reason it's disabled.
vllm-project/vllm#6877
other than excessively tight version bounds. I'd be ok with either disabling 312 or patching the cmake file to allow 312 and hoping that works.

Stale

[SomeoneSerge]

@cfhammill I just added the stuff I said was optional, idk why; the python 3.12 build... doesn't immediately fail (it's been running for 40' now) so I intend to leave it at that; I'll check on Ofborg tomorrow and probably just merge, but please do run some checks and skim over the extra stuff I pushed that it makes sense. I don't use vllm so I can't reason much about this

[cfhammill]

I'm happy with this! Thanks @SomeoneSerge! Merge when ready.

[SomeoneSerge]

Thanks @cfhammill