Checkpoint stateful handlers and metrics

[amatsukawa]

🚀 Feature

Things that are attached to the Engine might have state that would ideally be checkpointed and restored using as part of the Engine's state_dict.

An example is a Checkpoint handler when it has a score_function. Currently, the priorities the Checkpoint class stores is not saved anywhere. It is not able to recover gracefully from failure without manual intervention to parse the checkpoint path names, and directly setting the internals of the class.

Handlers and Metrics should have state_dict and load_state_dict methods (empty by default), and I think it should be possible for these to automatically make it into/restored from the Engine's state_dict when it's attached to an Engine.

[vfdev-5]

@amatsukawa thanks for FR! Yes, it definitely makes sense for handlers and metrics with internal state 👍

[amatsukawa]

FWIW, I'm completely happy with #1156 and the way things work now.

With complications I didn't think about, eg. some handlers needing to go on the valid engine and handlers needing to run in a specific order (Checkpoint needs to run last after all other stateful handlers) perhaps automatically adding things to the engine's checkpoints is not trivial and might make things harder to reason about.

[H4dr1en]

Bringing here a specific use case that this FR could solve:

Having the RunningAverage metric being able to restore the state would allow to not "forget" previous iterations scores when resuming an experiments. Otherwise the metric can show "peaks" when resuming an experiment, as can be shown in the figure below: