Random notes

[hypnopump]

• Prediction of angular features:

  • See this: https://discord.com/channels/729741769192767510/797547607345201162/874987318069592095

• Types of recurrent layers

  • multiplicativeLSTM: https://florianwilhelm.info/2018/08/multiplicative_LSTM_for_sequence_based_recos/
  • layerNormLSTM: https://github.com/exe1023/LSTM_LN/blob/master/lstm.py
  • multiplicative Integration: https://papers.nips.cc/paper/2016/file/f69e505b08403ad2298b9f262659929a-Paper.pdf
  • other variants: https://github.com/FlorianMuellerklein/death_metal_lstm/blob/master/scripts/models.py
  • Novel Arch: https://arxiv.org/pdf/1911.11033.pdf
  • SRU (https://arxiv.org/abs/1709.02755): https://github.com/bamtercelboo/pytorch_SRU
  • peephole: https://towardsdatascience.com/building-a-lstm-by-hand-on-pytorch-59c02a4ec091
  • uRNN: https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1194/reports/custom/15842215.pdf
  • delayedLSTM: https://arxiv.org/pdf/1909.00021.pdf
  • osiparc: ?

[ClashLuke]

You could also check out uRNN, OSiPaRC and https://arxiv.org/abs/1909.00021

[hypnopump]

oh cool! Added urnn and delayed to the list. Could you point me to an OSiPaRC reference?

[ClashLuke]

OSiPaRC unfortunately doesnt have a paper. It's a more powerful and general uRNN implementation with constant memory consumption.

[hypnopump]

oh, you mean this? https://github.com/HomebrewNLP/HomebrewNLP/blob/b7d1975c31568d0ade81df5e415a5a4fa755ef6b/module.py#L117 ? have you measured execution performance wrt a classic LSTM?

RGN2 models are a bit slow to train already...

[ClashLuke]

Exactly, yeah. I dont recall how it compares to LSTM, but uRNN claims to outperform LSTM and this is basically a more powerful uRNN with RevNet bolted onto it.
I'd expect it to have a worse step time but overall faster convergence, but we definitely need to test that.