Error while exporting rave

[agustinissidoro]

After training rave I try to export the model and get the following. It happens both in a Google Colab environment and in a Anaconda environmente. Does any one know what is going on? Thank you very much!

(rave) C:\Users\agustin\RAVE>python export_rave.py --run /runs/kendrick/rave --cached false --name kendrick
[1470.26] exporting model
[9597.60] loading model from checkpoint
[9597.60] using None
Traceback (most recent call last):
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 309, in _check_seekable
f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\agustin\RAVE\export_rave.py", line 204, in
model = RAVE.load_from_checkpoint(RUN, strict=False).eval()
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\core\saving.py", line 139, in load_from_checkpoint
checkpoint = pl_load(checkpoint_path, map_location=lambda storage, loc: storage)
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\pytorch_lightning\utilities\cloud_io.py", line 42, in load
return torch.load(path_or_url, map_location=map_location)
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 699, in load
with _open_file_like(f, 'rb') as opened_file:
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 236, in _open_file_like
return _open_buffer_reader(name_or_buffer)
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 221, in init
_check_seekable(buffer)
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 312, in _check_seekable
raise_err_msg(["seek", "tell"], e)
File "C:\Users\agustin\anaconda3\envs\rave\lib\site-packages\torch\serialization.py", line 305, in raise_err_msg
raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

[caillonantoine]

Hi :) Your last run version has no checkpoint in it !

[philx01]

So how do we fix this?

[philx01]

When I run train… there seems to be no ckpt file created? How many epochs does it take for the ckpt file to get created?

[agustinissidoro]

If the training is running ok, just be pacient. The number of epochs until a checkpoint varies greatly on the characteristics of the dataset and (I'm just guessing based on observation) your hardware. I have trained networks where I got checkpoints every 10 epochs and others that I got checkpoints every 117 epochs. Hitherto, I haven't managed to train a satisfactory model. I was running on Google Colab because I have very little GPU (2gb). I have trained the model for ~3 hours and didn't get something sounding properly. I guess it takes many many hours on training. I quitted the task until I can have the proper hardware my own, otherwise it is pointless.

[ogbabydiesal]

I have no checkpoints after 1400 epochs, each epoch has 256 steps, is there a checkpoint setting somewhere I could edit to set checkpoints? I tried to export a model and a sustained digital noise.

[ogbabydiesal]

ah was looking in the wrong folder, in my many attempts installing libraries and running with various virtual environments I had created multiple run folders, I see them now.