EOFError

[JuliusWang-7]

Hi, Fabian.
I don't know how to figure out this problem. What's more, when I use the command -c, it still continues from epoch 100, even i have trained over 300 epochs. Could you give me some advice, thanks a lot.

[FabianIsensee]

Please delete all the npy files in the nnUNet_preprocessed folder of your task and try again

[FabianIsensee]

(do not delete the npz files!)

And make sure your SSD is not full

[JuliusWang-7]

Thanks. It does work.

[JuliusWang-7]

It still stops from time to time. Although the command -c works now, I have to delete the npy files again and again.

[FabianIsensee]

Are you training in a docker container?

[JuliusWang-7]

Are you training in a docker container?

No, I trained it locally in Linux

[FabianIsensee]

Mhm it appears like something is wrong with the location you store the data at. Is it a local SSD?

[ProfessorHuang]

Similar problem, my training stops from time to time. So, I have to continue the training with command -c manually. (but I trained it in a docker container)

[JuliusWang-7]

Mhm it appears like something is wrong with the location you store the data at. Is it a local SSD?

yes

[FabianIsensee]

Hm that is strange. Have you checked your RAM? Maybe the ram was full and the system killed some background worker

[JuliusWang-7]

Hm that is strange. Have you checked your RAM? Maybe the ram was full and the system killed some background worker

Thanks, maybe something wrong caused by the Ram, because I run four folds at the same time.
But what is strange is when I run 3d_fullres this problem won't happen.
Happy New Year! Wish you have a wonderful year, Fabian.

[FabianIsensee]

Hm you should be able to train multiple folds simultaneously. I do it all the time. The only thing you have to consider is that only one of the trainings can to the extraction of the files (npz -> npy) at once, so if you train multiple folds you need to first just start one fold. Only once this fold is using the GPU you can start the others. If you already have the files extracted from a previous training then you can start all folds simultaneously