Export failure due to partially trained model?

[inter-inter]

I had to end a training prematurely because it had been going a week and my GPU was overheating. The final line on the Terminal before termination:

Epoch 5616: 24%|███████████▍ | 131/549 [00:40<02:10, 3.20it/s, v_num=0]

Gleaning from a few posts here that it is still possible to export a model that has not been trained to completion, I tried to export the model. Export failed with the following:

I0628 20:51:27.421437 18944 export.py:379] save model
Traceback (most recent call last):
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\Scripts\rave.exe\__main__.py", line 7, in <module>
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\scripts\main_cli.py", line 34, in main
    app.run(export.main)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\absl\app.py", line 308, in run
    _run_main(main, args)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\absl\app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\scripts\export.py", line 387, in main
    scripted_rave.export_to_ts(os.path.join(FLAGS.run, model_name))
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\nn_tilde\__init__.py", line 168, in export_to_ts
    scripted = torch.jit.script(self)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_script.py", line 1284, in script
    return torch.jit._recursive.create_script_module(
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_recursive.py", line 480, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_recursive.py", line 546, in create_script_module_impl
    create_methods_and_properties_from_stubs(concrete_type, method_stubs, property_stubs)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_recursive.py", line 397, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_recursive.py", line 898, in compile_unbound_method
    create_methods_and_properties_from_stubs(concrete_type, (stub,), ())
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\torch\jit\_recursive.py", line 397, in create_methods_and_properties_from_stubs
    concrete_type._create_methods_and_properties(property_defs, property_rcbs, method_defs, method_rcbs, method_defaults)
RuntimeError:
Unknown type name 'rave.blocks.AdaptiveInstanceNormalization':
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\scripts\export.py", line 176
    def update_adain(self):
        for m in self.modules():
            if isinstance(m, rave.blocks.AdaptiveInstanceNormalization):
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
                m.learn_x.zero_()
                m.learn_y.zero_()
'VariationalScriptedRAVE.update_adain' is being compiled since it was called from 'VariationalScriptedRAVE.encode'
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\scripts\export.py", line 196
    def encode(self, x):
        if self.is_using_adain:
            self.update_adain()
            ~~~~~~~~~~~~~~~~~ <--- HERE

        if self.resampler is not None:
'VariationalScriptedRAVE.encode' is being compiled since it was called from 'VariationalScriptedRAVE.forward'
  File "C:\Users\ADDAMS-MORTICIA\anaconda3\envs\raveworld\lib\site-packages\scripts\export.py", line 231
    def forward(self, x):
        return self.decode(self.encode(x), from_forward=True)
                                       ~ <--- HERE

I am curious if someone can enlighten me as to whether the export failed because of an incompletely trained model or due to some other reason. And if there were a possibility of exporting the model (trained, I believe, to more than 3,000,000 steps), how would I go about it?

Gratefully yours,
Andy

[caillonantoine]

Hi there,

Normally I have already fixed the problem, make sure you have the latest version of the repo (pip install -U acids-rave) and retry exporting :)

[inter-inter]

Thank you so much! After installing the export succeeded, and the model is functioning in Max.

I am still curious: is there an automatic shutoff of the training process or does it simply go on indefinitely until the user manually terminates the process?

I am now moving on to training Prior and am running into an issue with what seems to be a PyTorch incompatibility... I will post again once I have a better handle on the problem. (By the way, is it better to post in the "Issues" tab or the "Discussion" tab for questions like these?)