This is an attempt to familiarize myself with PyTorch. In this example, the target to generate a sequence of continuous data (sine waves or mix of them) using LSTM
- 16/04/2017: When trying to generate a simple sine wave, the system flats out. It is unclear for me why this happens. The same happens with 2 and 3 sine-wave components.
- 18/04/2017: Thanks to the advice of Sean Robertson - https://discuss.pytorch.org/t/lstm-time-sequence-generation/1916/4 - to reduce the frequency of the sine-waves, I was finally able to generate signals. The 2 and 3 sine-wave components are working well (the 3 is a bit unstable).
- 19/04/2017: The method of teaching the model using only the ground truth is called Teacher Forcing.
- 20/04/2017: After further testing , I found my model works when the sine
wave has a relatively high frequency (1/60 Hz or more). Lower frequency like
(1/180 Hz) doesn't work.
- With a sequence length of 100 timesteps, the model flats out when I use it for generation.
- I tried to increase the sequence length till 500. The model no longer flats
out, but the performance is poor. Probably the dependency is too long for the
model to remember.
- I need a way to be definite about this issue
- Train the model on generation instead of prediction: training the model
on its own output
- Strangely, it doesn't lead to different results. With low frequencies, it doesn't work. With higher frequency, its performance is almost similar to the naive approach (where I train on the ground) truth.
- Try Bengio approach DAD
Scheduled Sampling- Not optimistic though
- Re-package the experiment in order to be able to give it a set of configurations, and it will run them and store their results.