Resume Training

[FrancescoFornasa]

Good morning to everyone.
I was wondering how should I change the code for allowing the training to be done in repeating iterations.
Since i can only train the network during nighttime (due to some intersections on that machine with a colleague of mine), I was thinking on starting the training from a CheckPoint model that i saved.
This should be not so hard to implement, but I am struggling in finding a solution by myself, so any suggestion will be very appreciated.
Thank you all a lot for this amazing work.

[aditya1709]

I tried out a few solutions, nothing seemed to work. It ended up training from the beginning. Have you made any progress on this front?

[kemusa]

I think if you just point the train_dir flag to the directory of your checkpoints it should load your latest checkpoint file from your previous training job.

Also you may have to comment out lines 338 and 339 in train.py as they will delete your checkpoints in that directory before checking for them. Not sure why those lines are there.