Train framework without autoencoder not working

[nilsleh]

In your paper you mention that using the autoencoder is optional. I would like to try this for a different domain, however, the code base does not seem to support this out of the box, when using the suggested training pipeline and specifying autoencoder: null in the config file and running python main.py with that config file.

In the trainer.py file:

• Line 732 assumes that autoencoder is used

In gaussian_diffusion.py:

• Line 502 should be moved inside the if block

However, fixing the above two yields:
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 256 but got size 64 for tensor number 1 in the list. in Line 882 of the Unet. So I suppose, the LQ image needs to be upsampled to a correct shape some time before? However, I am not sure about the other missing details, so hoping that you could take a look at it. Thanks in advance.

[nilsleh]

@zsyOAOA sorry to comment again, but would you have any input on this matter? I am in the process of opening a PR with possible fixes, so that the community can try running the method without the autoencoder, but would like to get your thoughts on the above. Thanks in advance.

[zsyOAOA]

My codebase is borrowed from guided diffusion, which is a diffusion on pixel space. You need to take some time going through the whole training process and replace the autoencoder with an identity function.

[nilsleh]

Just replacing the autoencoder with an identity does not work, since the code relies on an instantiated autoencoder and its attributes in numerous places. Also the config files need to change since the image dimension of the "image" and "lq" do not align anymore.