Crash when using normalizers.MacenkoNormalizer(backend='torch')

[RECranston]

Hi there

I'm running into an issue when normalising 2 of my images when running normalizers.MacenkoNormalizer(backend='torch')
I am running the following code:

target_array = np.load(f"{input_dir}{target_patch}_{target_id}.npy")

test_padded_array = np.load(f"{padded_array_input_dir}padded_array_{file_id}.npy")
test_patch1 = np.load(f"{input_dir}patch_0x0_{file_id}.npy")
test_patch2 = np.load(f"{input_dir}patch_0x1_{file_id}.npy")
test_patch3 = np.load(f"{input_dir}patch_1x0_{file_id}.npy")
test_patch4 = np.load(f"{input_dir}patch_1x1_{file_id}.npy")


T = transforms.Compose([
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x*255)
])

normalizer = torchstain.normalizers.MacenkoNormalizer(backend='torch')
normalizer.fit(T(target_array))

t_norm_patch_1x1 = T(test_patch4)


norm_patch_1x1, H_1x1, E_1x1 = normalizer.normalize(I=t_norm_patch_1x1, stains=True)

It's crashing on the last line with the error

Intel MKL ERROR: Parameter 3 was incorrect on entry to SGELSY.
Intel MKL ERROR: Parameter 3 was incorrect on entry to SGELSY.

{
	"name": "RuntimeError",
	"message": "false INTERNAL ASSERT FAILED at \"../aten/src/ATen/native/BatchLinearAlgebra.cpp\":1537, please report a bug to PyTorch. torch.linalg.lstsq: Argument 3 has illegal value. Most certainly there is a bug in the implementation calling the backend library.",
	"stack": "---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[23], line 59
     52 t_norm_patch_1x1 = T(test_patch4)
     55 # normalise
     56 #norm_patch_0x0, H_0x0, E_0x0 = normalizer.normalize(I=t_norm_patch_0x0, stains=True)
     57 #norm_patch_0x1, H_0x1, E_0x1 = normalizer.normalize(I=t_norm_patch_0x1, stains=True)
     58 #norm_patch_1x0, H_1x0, E_1x0 = normalizer.normalize(I=t_norm_patch_1x0, stains=True)
---> 59 norm_patch_1x1, H_1x1, E_1x1 = normalizer.normalize(I=t_norm_patch_1x1, stains=True)

File ~/PBTG/.venv/lib/python3.8/site-packages/torchstain/torch/normalizers/macenko.py:104, in TorchMacenkoNormalizer.normalize(self, I, Io, alpha, beta, stains)
     81 ''' Normalize staining appearence of H&E stained images
     82 
     83 Example use:
   (...)
    100     Macenko et al., ISBI 2009
    101 '''
    102 c, h, w = I.shape
--> 104 HE, C, maxC = self.__compute_matrices(I, Io, alpha, beta)
    106 # normalize stain concentrations
    107 C *= (self.maxCRef / maxC).unsqueeze(-1)

File ~/PBTG/.venv/lib/python3.8/site-packages/torchstain/torch/normalizers/macenko.py:69, in TorchMacenkoNormalizer.__compute_matrices(self, I, Io, alpha, beta)
     65 eigvecs = eigvecs[:, [1, 2]]
     67 HE = self.__find_HE(ODhat, eigvecs, alpha)
---> 69 C = self.__find_concentration(OD, HE)
     70 maxC = torch.stack([percentile(C[0, :], 99), percentile(C[1, :], 99)])
     72 return HE, C, maxC

File ~/PBTG/.venv/lib/python3.8/site-packages/torchstain/torch/normalizers/macenko.py:58, in TorchMacenkoNormalizer.__find_concentration(self, OD, HE)
     55 if not self.updated_lstsq:
     56     return torch.lstsq(Y, HE)[0][:2]
---> 58 return torch.linalg.lstsq(HE, Y)[0]

RuntimeError: false INTERNAL ASSERT FAILED at \"../aten/src/ATen/native/BatchLinearAlgebra.cpp\":1537, please report a bug to PyTorch. torch.linalg.lstsq: Argument 3 has illegal value. Most certainly there is a bug in the implementation calling the backend library."
}"

I've tried it on each patch from this image (0x0,0x1,1x0,1x1) and have get the same error returned. I'm quite new to python so apologies in advance if this is a basic question.

I've also tried this workflow using tensorflow as the backend (as in issue #37 ) but it also crashes.

Thanks in advance for any help.

[andreped]

Hello, @RECranston! :]

I think the owners of this repo have archived it. At least it is no longer being maintained. So I can't merge any fixes if we find a bug.

So, let's hope it is just a pytorch version issue. Which pytorch version are you using? Could you try downgrading it?

[RECranston]

Hi @andreped

Thank you for your speedy reply! :)
Version 2.4.1 I think. I can downgrade within my virtual environment if that is the issue

pip show torch

Name: torch
Version: 2.4.1
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: [email protected]
License: BSD-3
Location: /home/ruth2/PBTG/.venv/lib/python3.8/site-packages
Requires: filelock, fsspec, jinja2, networkx, nvidia-cublas-cu12, nvidia-cuda-cupti-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-runtime-cu12, nvidia-cudnn-cu12, nvidia-cufft-cu12, nvidia-curand-cu12, nvidia-cusolver-cu12, nvidia-cusparse-cu12, nvidia-nccl-cu12, nvidia-nvtx-cu12, sympy, triton, typing-extensions
Required-by: torchvision

Thanks very much for your help

[andreped]

I can downgrade within my virtual environment if that is the issue

At least try that first. As no one has reported this issue before, I assume there is something wrong with your setup.

Then again, if newer pytorch versions are no longer compatible, it would be great if the maintainers updated torchstain as well.

[RECranston]

Hi @andreped

Thanks for your reply. I will give this a go with a few versions and get back to you.

Appreciate your help with this!

[andreped]

@RECranston Also, I assumed you tried using the exact image in the example with the algorithm? If that does not work, there is some local setup issue.

---

EDIT: Lastly, if changing PyTorch version does not fix you issue. Could you try running this exact example in Google CoLab, just to sanity check that there is not something very wrong in the code?

[RECranston]

Hi @andreped

I've tried running the following downgraded versions and I'm still getting the same error.
torch 2.3.0 and torchvision 0.18.1 - fail
torch 2.2.0 and torchvision 0.17.0 - fail
torch 2.1.0 and torchvision 0.16.0 - fail

Strangely I'm not able to run the example in colab as it is unable to find the downloaded files for some reason (despite the right file paths)
I'm managing to run these functions on the majority of my images - it's only failing on 2 cases.

Sorry I realise this is not much help

[andreped]

Oh, the last point there is quite key.

So the tool works fine but it fails on some images. That's due to the numerical instabilities of the algorithm. If there is an image that lacks colours similar to HE, for instance an image with just glass, the algorithm might fail.

To fix, when looping across all image, do a try-except and skip images that fail to convert as these are images you do not care for regardless.

I made a tool for this, which is based on torchstain. That way you can normalize all your patches quite easily:
https://github.com/andreped/fast-stain-normalization

[RECranston]

Hi @andreped

Thanks so much for your help with this. I'll add in some try-except blocks and also check out fast-stain-normalization too!
Really appreciate it 👍

[andreped]

@RECranston As the issue seemed to be on improper usage of the tool on suboptimal images, and not really a bug in the code, then it would be great if you could close this issue :]

Feel free to open a new ticket, if you have other issues! Otherwise, good luck!

[RECranston]

Will do! I've checked out the problem images again and one has a low tissue to background ratio and the other looks like a bad scan so this explanation makes sense. Thanks for your help - closing the issue now :)