magma: unbreak for cudaPackages_12#283640
Conversation
Hotfix based on the suggestion from NixOS#281656 (comment)
| ] ++ lists.optionals (cudaPackages.cudaAtLeast "12.0.0") [ | ||
| (lib.cmakeBool "USE_FORTRAN" false) |
There was a problem hiding this comment.
any reason not to do
(lib.cmakeBool "USE_FORTRAN" !(cudaPackages.cudaAtLeast "12.0.0"))
or something similar?
There was a problem hiding this comment.
Didn't test how the option affects cuda11 (it's somewhat late in the night...), this way we only specify it for cuda12
ofborg
bot
added
10.rebuild-darwin: 0
|
Is there a downside to taking the approach of specifying the fortran name-mangling convention? Also, is there a downside to disabling Fortran for Magma? |
|
Well |
|
No crazy differences so I think it's good to go as a hot fix. Let's get it merged! ./pr-283640-cuda-11/bin/nix-cuda-test
Seed set to 42
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Missing logger folder: /home/connorbaker/nix-cuda-test/lightning_logs
Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to data/cifar-10-python.tar.gz
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 170498071/170498071 [00:02<00:00, 78881307.89it/s]
Extracting data/cifar-10-python.tar.gz to data
Files already downloaded and verified
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0
1 | model | ViT | 86.3 M
-----------------------------------------------
86.3 M Trainable params
0 Non-trainable params
86.3 M Total params
345.317 Total estimated model params size (MB)
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:28<00:00, 8.80it/s, v_num=0, train_loss=2.350, val_loss=2.330]`Trainer.fit` stopped: `max_epochs=10` reached.
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:30<00:00, 8.62it/s, v_num=0, train_loss=2.350, val_loss=2.330]$ ./pr-283640-cuda-12/bin/nix-cuda-test
Seed set to 42
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Files already downloaded and verified
Files already downloaded and verified
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
| Name | Type | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0
1 | model | ViT | 86.3 M
-----------------------------------------------
86.3 M Trainable params
0 Non-trainable params
86.3 M Total params
345.317 Total estimated model params size (MB)
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:26<00:00, 9.04it/s, v_num=1, train_loss=2.350, val_loss=2.330]`Trainer.fit` stopped: `max_epochs=10` reached.
Epoch 9: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 781/781 [01:28<00:00, 8.86it/s, v_num=1, train_loss=2.350, val_loss=2.330] |
delroth
added
the
12.approvals: 2
|
@ConnorBaker AFAIK the effective linear algebra backend is mediated with the Also, IIUC, magma/cuSOLVER are only utilized for PyTorch functions related to matrix inversion, QR/Cholesky/LU decomposition, and SVD. A ViT training loop will not include any of these AFAIK. tldr, I'm a big fan of running benchmarks but I'm concerned that we may not have benchmarked quite the right thing in this case |
|
Ah, that's a good point. I also like benchmarks, especially when they measure the thing they're supposed to be benchmarking :/ I need to look into adding Magma's test cases as a passthru or something and exposing them separately. In the meantime, I'm going to take the approach of specifying name-mangling as part of a general update to the magma package. |
|
Yeah, sorry to be the parade rain-er here! But not a big deal ultimately... working magma > broken magma |
5 participants
#269639 seems to have broken magma (and torch, and what not); #281656 (comment) suggests a hot fix and it seems to work. CC @samuela @dmayle
Description of changes
Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.