Add Lion optimizer

[james77777778]

Related to keras-team/keras#18442

EDITED: update learning_rate=0.001

The golden in test_correctness_with_golden is generated by tf.keras.optimizers.Lion(learning_rate=0.001) with following script:

import numpy as np
import tensorflow as tf

optimizer = tf.keras.optimizers.Lion(learning_rate=0.001)

x = tf.Variable(np.ones([10]))
grads = tf.constant(np.arange(0.1, 1.1, 0.1))
first_grads = tf.constant(np.full((10,), 0.01))

optimizer.apply_gradients(zip([first_grads], [x]))
for _ in range(5):
    print(x.numpy())
    optimizer.apply_gradients(zip([grads], [x]))

# outputs
[0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999 0.999]
[0.998 0.998 0.998 0.998 0.998 0.998 0.998 0.998 0.998 0.998]
[0.997 0.997 0.997 0.997 0.997 0.997 0.997 0.997 0.997 0.997]
[0.996 0.996 0.996 0.996 0.996 0.996 0.996 0.996 0.996 0.996]
[0.995 0.995 0.995 0.995 0.995 0.995 0.995 0.995 0.995 0.995]

[fchollet]

Thank you for the PR! Looking great!

For performance reasons, we reimplement our optimizers for torch using torch C++ ops called from Python -- here's an example of such a PR: https://github.com/keras-team/keras-core/pull/534/files

Would you be able to include the torch version of the optimizer in this PR?

[fchollet]

Just noticed this -- the LR here is 1e-4, but it should be 1e-3 like in the other optimizers. This was already an issue in tf.keras, we should fix it here.

[james77777778]

Fixed

[fchollet]

either "between 0 and 1" or "in the [0, 1] range"

[james77777778]

Fixed

[james77777778]

For performance reasons, we reimplement our optimizers for torch using torch C++ ops called from Python -- here's an example of such a PR: https://github.com/keras-team/keras-core/pull/534/files

Would you be able to include the torch version of the optimizer in this PR?

I didn't notice that there is a separate folder for torch's optimizers.
torch_lion.py has been added.

As torch lacks _foreach_sign_ operator, I implemented it by c_t = [c.sign() for c in c_t]
Ref: https://github.com/pytorch/pytorch/blob/6847c965f5a05d5631357e6af4cf759231770b44/torch/optim/rprop.py#L300

[fchollet]

Awesome work! 👍
LGTM