Allow negative learning rate

[Cauldrath]

This can be used to train away from a group of images you don't want
As this moves the model away from a point instead of towards it, the change in the model is unbounded
So, don't set it too low. -4e-7 seemed to work well.

[kohya-ss]

Thank you for this! However, when the model trains away from the images, I think the model will eventually break down because that is what happens with any broken image.

This was the case when I tried negative loss before.

[Cauldrath]

Thank you for this! However, when the model trains away from the images, I think the model will eventually break down because that is what happens with any broken image.

This was the case when I tried negative loss before.

Yes, I also experienced that when I set the value to be too negative, but didn't have the same problems when I stuck to -4e-7 for 4k images with a batch size of 4 and 4 repeats. (The dataset was built by searching for images with tags for things I didn't want, then removing those tags from the caption files.) I guess it's a question of whether you want to trust the user with the ability to ruin their model if they use the wrong settings, but there's a lot of other ways to do that. Perhaps it needs to be combined with something that limits how far the model can be trained from the original tensor values?

This may also be something that could be handled better with distillation or something like DPO.

[kohya-ss]

That makes sense! I will merge this sooner :)