Learning Rate schedules (ReduceLROnPlateau, cosine) support

[kirillbobyrev]

Most Deep Learning frameworks support Learning Rate reduction schedulers (cosine, ReduceLROnPLateau etc), fixed schedule, etc. While not very hard to implement manually, out-of-the-box support would definitely be very nice.

Single schedules are not too complicated, but joint schedules and some extra support is definitely useful.

E.g. PyTorch ReduceLROnPlateau or Jax's Optim schedules.

This is the first time I'm seeing and reading Candle's code, so please let me know if there is better channel/place to ask questions like this.

[EricLBuehler]

@kirillbobyrev, I opened #2225 which refactors the structure to allow this to be implemented.

[kirillbobyrev]

Thanks a lot!