torchvision: fix #217878; migrate to cudaPackages by ConnorBaker · Pull Request #218035 · NixOS/nixpkgs

ConnorBaker · 2023-02-24T14:35:31Z

Description of changes

when building with CUDA, we must make sure to use the same C/C++ compiler as NVCC to avoid symbol errors when linking
move to cudaPackages to reduce closure size

Currently, torchvision doesn't even build on master because the default GCC is 12, which is unsupported by NVCC until CUDA 12.0.

This fixes #217878.

Things done

ConnorBaker · 2023-02-24T17:01:28Z

Result of nixpkgs-review pr 218035 run on x86_64-linux 1

2 packages marked as broken and skipped:

python310Packages.bambi
python310Packages.pywick

3 packages failed to build:

python310Packages.elegy
python310Packages.torchinfo
tts

20 packages built:

easyocr (python310Packages.easyocr)
khoj
larynx-train (python310Packages.larynx-train)
python310Packages.arviz
python310Packages.boxx
python310Packages.bpycv
python310Packages.clip
python310Packages.ffcv
python310Packages.grad-cam
python310Packages.pymc
python310Packages.pytorch-lightning
python310Packages.pytorch-metric-learning
python310Packages.pytorch-pfn-extras
python310Packages.sentence-transformers
python310Packages.tensorboardx
python310Packages.timm
python310Packages.torch-tb-profiler
python310Packages.torchvision
python310Packages.trainer
python310Packages.zcs

ConnorBaker · 2023-02-24T23:12:32Z

Result of nixpkgs-review pr 218035 run on x86_64-linux 1

1 package marked as broken and skipped:

python310Packages.pywick

2 packages failed to build:

python310Packages.elegy
python310Packages.torchinfo

21 packages built:

easyocr (python310Packages.easyocr)
khoj
larynx-train (python310Packages.larynx-train)
python310Packages.arviz
python310Packages.boxx
python310Packages.bpycv
python310Packages.clip
python310Packages.ffcv
python310Packages.grad-cam
python310Packages.pymc
python310Packages.pytorch-lightning
python310Packages.pytorch-metric-learning
python310Packages.pytorch-pfn-extras
python310Packages.sentence-transformers
python310Packages.tensorboardx
python310Packages.timm
python310Packages.torch-tb-profiler
python310Packages.torchvision
python310Packages.trainer
python310Packages.zcs
tts

ConnorBaker · 2023-02-24T23:22:01Z

Failed derivations

python3.10-torchinfo-1.64.drv (derivation hash: n4jdw6ybjxcrs5pl4qjv7i2hb5wmamq5)
- https://gist.github.com/19cb88a6514449bcce3161a291e8955c
- Failing test cases unrelated to this PR; doesn't even build on master due to torchvision failing
python3.10-treex-0.6.11.drv (derivation hash: aqr1jrf7wlqywagbg9bkhbdzfpbvbx9p)
- https://gist.github.com/d71f6d2d34cb1ea18f25e0e1f8e602e8
- Failing test cases unrelated to this PR

ConnorBaker · 2023-02-25T02:51:10Z

Running again after rebasing on master

ConnorBaker · 2023-02-25T05:24:50Z

Result of nixpkgs-review pr 218035 run on x86_64-linux 1

1 package marked as broken and skipped:

python310Packages.pywick

2 packages failed to build:

python310Packages.elegy
python310Packages.torchinfo

21 packages built:

easyocr (python310Packages.easyocr)
khoj
larynx-train (python310Packages.larynx-train)
python310Packages.arviz
python310Packages.boxx
python310Packages.bpycv
python310Packages.clip
python310Packages.ffcv
python310Packages.grad-cam
python310Packages.pymc
python310Packages.pytorch-lightning
python310Packages.pytorch-metric-learning
python310Packages.pytorch-pfn-extras
python310Packages.sentence-transformers
python310Packages.tensorboardx
python310Packages.timm
python310Packages.torch-tb-profiler
python310Packages.torchvision
python310Packages.trainer
python310Packages.zcs
tts

ConnorBaker · 2023-02-25T05:44:18Z

Failed derivations

python3.10-torchinfo-1.64.drv (derivation hash: n4jdw6ybjxcrs5pl4qjv7i2hb5wmamq5)
- https://gist.github.com/19cb88a6514449bcce3161a291e8955c
- Failing test cases unrelated to this PR; doesn't even build on master due to torchvision failing
python3.10-treex-0.6.11.drv (derivation hash: zp8514kqpmwsffk8nbagi2i6l1rkw8hc)
- https://gist.github.com/a3651d44bc5aef31be315e4d67b517ea
- Failing test cases unrelated to this PR

ConnorBaker · 2023-02-25T05:46:17Z

cc @NixOS/cuda-maintainers

pkgs/development/python-modules/torchvision/default.nix

SomeoneSerge

✅ Builds and runs (some random snippet)

pkgs/development/python-modules/torchvision/default.nix

FRidh · 2023-02-28T08:28:38Z

For wrapping the compiler, we could probably do something like

  gcc11Stdenv = overrideCC gccStdenv buildPackages.gcc11;

and use wrapCC on top of the cuda compiler. That way we could get a cudaStdenv.

samuela · 2023-02-28T15:31:07Z

Actually, I think we should just upgrade the cudatoolkit_12 gcc to gcc12 to match stdenv: https://cs.github.com/NixOS/nixpkgs/blob/4a6cd14f37b22c9d2f90e3774b8ea9601025ad6b/pkgs/development/compilers/cudatoolkit/versions.toml#L79

It was only ever held back to gcc11 because stdenv was stuck on gcc11. now that stdenv is on gcc12 IIUC that should resolve these issues?

ConnorBaker · 2023-03-09T18:08:03Z

Actually, I think we should just upgrade the cudatoolkit_12 gcc to gcc12 to match stdenv: https://cs.github.com/NixOS/nixpkgs/blob/4a6cd14f37b22c9d2f90e3774b8ea9601025ad6b/pkgs/development/compilers/cudatoolkit/versions.toml#L79

It was only ever held back to gcc11 because stdenv was stuck on gcc11. now that stdenv is on gcc12 IIUC that should resolve these issues?

I'm in support of that upgrade, though it might be problematic because some packages (cough torch cough) have explicit requirements for CUDA versions:

nixpkgs/pkgs/development/python-modules/torch/default.nix

Lines 49 to 50 in 6169ccb

    
           assert !cudaSupport || (let majorIs = lib.versions.major cudatoolkit.version; 
        
                                   in majorIs == "9" || majorIs == "10" || majorIs == "11");

On a different note, is it reasonable to have asserts like that in packages? I don't know how one would be able to circumvent that if they wanted to build against HEAD (which mostly supports CUDA 12) by overriding the src attribute. (That's excluding the fact the attribute set is recursive, which is perhaps why I've been unsuccessful so far in trying to do exactly that and had to write my own derivation.)

samuela · 2023-03-09T18:24:56Z

Do we know that pytorch doesn't support CUDA 12, or is that assert overly cautious? We can always pin pytorch to a CUDA 11.x version until they get up to speed.

ConnorBaker · 2023-03-09T18:31:05Z

Do we know that pytorch doesn't support CUDA 12, or is that assert overly cautious? We can always pin pytorch to a CUDA 11.x version until they get up to speed.

iirc you can build with whatever, and a quick once-over of cpp_extension.py didn't reveal any specific matching on CUDA version to prohibit builds: https://github.com/pytorch/pytorch/blob/fe05266fda4f908130dea7cbac37e9264c0429a2/torch/utils/cpp_extension.py#L382-L420

However, they do have a tracking issue for CUDA 12 support: pytorch/pytorch#91122

In my experience, when built against HEAD it mostly just works, though there are a few outstanding issues.

SomeoneSerge · 2023-03-09T18:36:27Z

I think it's safe to say that we should prefer setting meta.broken or showing warnings to just killing Nix eval. The particular piece of code is legacy

samuela · 2023-03-09T18:36:44Z

Ok, yeah in that case we should wait until pytorch/pytorch#91122 is resolved. Curious what this all has to do with the gcc upgrade though?

ConnorBaker · 2023-03-09T18:40:19Z

Ok, yeah in that case we should wait until pytorch/pytorch#91122 is resolved. Curious what this all has to do with the gcc upgrade though?

That's my bad -- I misinterpreted your comment about cudaPackages_12 and GCC 12 (#218035 (comment)) to mean "we should make that the default." Sorry!

pkgs/development/python-modules/torchvision/default.nix

…ages - when building with CUDA, we must make sure to use the same C/C++ compiler as NVCC to avoid symbol errors when linking - move to cudaPackages to reduce closure size - separate build-time and run-time CUDA dependencies

samuela · 2023-03-09T22:23:33Z

Result of nixpkgs-review pr 218035 run on x86_64-linux 1

1 package marked as broken and skipped:

python310Packages.pywick

2 packages failed to build:

python310Packages.elegy
python310Packages.torchinfo

23 packages built:

easyocr (python310Packages.easyocr)
khoj
larynx-train (python310Packages.larynx-train)
python310Packages.arviz
python310Packages.bambi
python310Packages.boxx
python310Packages.bpycv
python310Packages.clip
python310Packages.fastai
python310Packages.ffcv
python310Packages.grad-cam
python310Packages.pymc
python310Packages.pytorch-lightning
python310Packages.pytorch-metric-learning
python310Packages.pytorch-pfn-extras
python310Packages.sentence-transformers
python310Packages.tensorboardx
python310Packages.timm
python310Packages.torch-tb-profiler
python310Packages.torchvision
python310Packages.trainer
python310Packages.zcs
tts

samuela · 2023-03-09T22:29:29Z

Failures:

error: builder for '/nix/store/9hrvr6cfrjcm41829sanav6r1y2p5fhv-python3.10-torchinfo-1.64.drv' failed with exit code 1;
       last 10 log lines:
       >                 "Failed to run torchinfo. See above stack traces for more details. "
       >                 f"Executed layers up to: {executed_layers}"
       >             ) from e
       > E           RuntimeError: Failed to run torchinfo. See above stack traces for more details. Executed layers up to: []
       >
       > torchinfo/torchinfo.py:303: RuntimeError
       > =========================== short test summary info ============================
       > FAILED tests/exceptions_test.py::test_input_size_half_precision - RuntimeError: Failed to run torchinfo. See above stack traces for more deta...
       > ============ 1 failed, 55 passed, 6 skipped, 2 deselected in 7.04s =============
       > /nix/store/c3f4jdwzn8fm9lp72m91ffw524bakp6v-stdenv-linux/setup: line 1593: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/9hrvr6cfrjcm41829sanav6r1y2p5fhv-python3.10-torchinfo-1.64.drv'.
error: builder for '/nix/store/n06x3d9zs61a2m0cw1nmmxsd42n5n93l-python3.10-treex-0.6.11.drv' failed with exit code 1;
       last 10 log lines:
       >   /build/source/tests/nn/test_sequential.py:38: FutureWarning: jax.tree_leaves is deprecated, and will be removed in a future release. Use jax.tree_util.tree_leaves instead.
       >     leaves = jax.tree_leaves(mlp)
       >
       > -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
       > =========================== short test summary info ============================
       > FAILED tests/test_treex.py::TestTreex::test_is_tree - assert False
       > FAILED tests/test_treex.py::TestTreex::test_list - AssertionError: assert False
       > FAILED tests/test_treex.py::TestTreex::test_treelist - assert False
       > ============ 3 failed, 147 passed, 101 warnings in 98.92s (0:01:38) ============
       > /nix/store/c3f4jdwzn8fm9lp72m91ffw524bakp6v-stdenv-linux/setup: line 1593: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/n06x3d9zs61a2m0cw1nmmxsd42n5n93l-python3.10-treex-0.6.11.drv'.
error: 1 dependencies of derivation '/nix/store/gizrf302d6d0ips7ngigzk4wy6nmvrzi-python3.10-elegy-0.8.6.drv' failed to build
error: 2 dependencies of derivation '/nix/store/l9ljdzkw64025nwhjbc7382pinb4c5w3-review-shell.drv' failed to build

are not related.

samuela · 2023-03-09T22:30:24Z

LGTM thanks @ConnorBaker !

pkgs/development/python-modules/torchvision/default.nix

SomeoneSerge · 2023-03-10T01:29:17Z

(We already had bumped cuda12's gcc in the -ccbin pr iirc)

nixos-discourse · 2023-03-16T12:43:58Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-45/26397/1

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Feb 24, 2023

ofborg bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Feb 24, 2023

ConnorBaker mentioned this pull request Feb 24, 2023

torchvision: Does not use the same compiler as torch when building with CUDA #217913

Closed

ConnorBaker force-pushed the fix/torchvision-set-cuda-compilers branch from fdec768 to a38527c Compare February 24, 2023 20:29

ConnorBaker force-pushed the fix/torchvision-set-cuda-compilers branch from a38527c to d397194 Compare February 25, 2023 02:34

ConnorBaker marked this pull request as ready for review February 25, 2023 05:45

ConnorBaker requested review from FRidh and jonringer as code owners February 25, 2023 05:45

SomeoneSerge added the 6.topic: cuda Parallel computing platform and API label Feb 25, 2023

SomeoneSerge mentioned this pull request Feb 25, 2023

cudaPackages: point nvcc at a compatible -ccbin #218265

Merged

14 tasks

SomeoneSerge reviewed Feb 26, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

SomeoneSerge reviewed Feb 26, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

SomeoneSerge approved these changes Feb 26, 2023

View reviewed changes

SomeoneSerge reviewed Feb 26, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

samuela requested changes Feb 28, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

ConnorBaker self-assigned this Mar 8, 2023

ConnorBaker mentioned this pull request Mar 9, 2023

torch: add cudaCapabilities to passthru #220377

Closed

ConnorBaker force-pushed the fix/torchvision-set-cuda-compilers branch from d397194 to 16dbe78 Compare March 9, 2023 19:27

ConnorBaker requested a review from samuela March 9, 2023 19:29

ConnorBaker mentioned this pull request Mar 9, 2023

Python 3.10 closure contains incompatible versions of CUDA and g++, causing build failures #217878

Closed

ConnorBaker changed the title ~~torchvision: fix compilers when using CUDA; migrate to cudaPackages~~ torchvision: fix #217878; migrate to cudaPackages Mar 9, 2023

samuela requested changes Mar 9, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

pkgs/development/python-modules/torchvision/default.nix Outdated Show resolved Hide resolved

ConnorBaker force-pushed the fix/torchvision-set-cuda-compilers branch from 16dbe78 to ed00c38 Compare March 9, 2023 21:27

samuela approved these changes Mar 9, 2023

View reviewed changes

samuela merged commit 508cf08 into NixOS:master Mar 9, 2023

ConnorBaker commented Mar 9, 2023

View reviewed changes

pkgs/development/python-modules/torchvision/default.nix Show resolved Hide resolved

Uh oh!

Conversation

ConnorBaker commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

ConnorBaker commented Feb 24, 2023

Uh oh!

ConnorBaker commented Feb 24, 2023

Uh oh!

ConnorBaker commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failed derivations

Uh oh!

ConnorBaker commented Feb 25, 2023

Uh oh!

ConnorBaker commented Feb 25, 2023

Uh oh!

ConnorBaker commented Feb 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failed derivations

Uh oh!

ConnorBaker commented Feb 25, 2023

Uh oh!

Uh oh!

Uh oh!

SomeoneSerge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

FRidh commented Feb 28, 2023

Uh oh!

samuela commented Feb 28, 2023

Uh oh!

ConnorBaker commented Mar 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samuela commented Mar 9, 2023

Uh oh!

ConnorBaker commented Mar 9, 2023

Uh oh!

SomeoneSerge commented Mar 9, 2023

Uh oh!

samuela commented Mar 9, 2023

Uh oh!

ConnorBaker commented Mar 9, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samuela commented Mar 9, 2023

Uh oh!

samuela commented Mar 9, 2023

Uh oh!

samuela commented Mar 9, 2023

Uh oh!

Uh oh!

SomeoneSerge commented Mar 10, 2023

Uh oh!

nixos-discourse commented Mar 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ConnorBaker commented Feb 24, 2023 •

edited

Loading

ConnorBaker commented Feb 24, 2023 •

edited

Loading

ConnorBaker commented Feb 25, 2023 •

edited

Loading

ConnorBaker commented Mar 9, 2023 •

edited

Loading