About optim

[xihuai18]

Here is a note from Pytorch document, https://pytorch.org/docs/1.2.0/optim.html#constructing-it.
If you need to move a model to GPU via .cuda(), please do so before constructing optimizers for it. Parameters of a model after .cuda() will be different objects with those before the call.

In general, you should make sure that optimized parameters live in consistent locations when optimizers are constructed and used.

I noted that you can .cuda() or .cpu() after the optimizers are constructed, but the algorithm still work. I can't figure out it.