nbclient.exceptions.DeadKernelError: Kernel died in ./modules/3d_image_transforms.ipynb #1698
Comments
|
It looks to have died on the second last cell: deformed_data_dict = rand_elastic(data_dict)
print(f"image shape: {deformed_data_dict['image'].shape}")
image, label = deformed_data_dict["image"][0], deformed_data_dict["label"][0]
plt.figure("visualise", (8, 4))
plt.subplot(1, 2, 1)
plt.title("image")
plt.imshow(image[:, :, 5], cmap="gray")
plt.subplot(1, 2, 2)
plt.title("label")
plt.imshow(label[:, :, 5])
plt.show()I can't tell from that if it's the elastic deformation or the matplotlib calls that cause the crash, it could be either if the environment isn't right and some compiled code caused a crash. |
|
Would be worth running with CUDA_LAUNCH_BLOCKING="1" if the test is running pre-processing on the GPU |
I don't think the transforms are run on the GPU. You can see in this tutorial. |
There may be some interaction anyway with the CUDA components in Pytorch so it's worth trying. Are there any other environment variables to set to enhance debug output? We don't have much else to go on since we can't replicate the issue locally. |
|
tried with |
|
Next thing to try is enabling fault handling: https://docs.python.org/3/library/faulthandler.html Add |
|
I traced the error back to the Convolution in the Gaussian filter, which is used by the |
Signed-off-by: YunLiu <[email protected]>
Workaround for #1698 ### Description Skip run papermill for modules/3d_image_transforms.ipynb for now to avoid breaking daily ci, will review this one after update base image. ### Checks <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [ ] Avoid including large-size files in the PR. - [ ] Clean up long text outputs from code cells in the notebook. - [ ] For security purposes, please check the contents and remove any sensitive info such as user names and private key. - [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use relative paths for tutorial repo files (3) put figure and graphs in the `./figure` folder - [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>` --------- Signed-off-by: KumoLiu <[email protected]> Signed-off-by: YunLiu <[email protected]>
The text was updated successfully, but these errors were encountered: