Model Checkpointing for DIET, TED and ResponseSelector

[tttthomasssss]

Proposed changes:

Implemented model checkpointing for DIET. The best DIET model during training will be stored instead of just the last model. The model is evaluated on the basis of evaluate_every_number_of_epochs. Checkpointing is enabled iff the following is set for the DIETClassifier, ResponseSelector and TEDPolicy in the config.yml file:
* checkpoint_model: True
* evaluate_on_number_of_examples > 0

The model is stored to whatever location has been specified with the --out parameter when calling rasa train nlu ...

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[tabergma]

Great start 🚀 Left some comments.

Main concern is that we don't need the MODEL_CHECKPOINT_DIR, I think we can simple store the intermediate best model in a tmp directory. Also this solution should be independent from DIET. All Rasa Models should be able to store the best model, e.g. TEDPolicy and ResponseSelector. I guess you also need the option checkpoint_model over there.

[tttthomasssss]

Thanks very much for the review comments!

Checkpointing for TEDPolicy and the ResponseSelector is yet to be added.

[tttthomasssss]

@tabergma I...ahem...finally managed to get back to this issue. Could you kindly review the latest changes? Checkpointing is now also added to the response_selector (though likely the old version 😱 ) and TEDPolicy.

[tabergma]

Left a few more comments. Thanks for tackling so many already. Once those comments are resolved and all checks pass, I guess we are good to go 💯

[tabergma]

Looks great! Thanks for resolving all my comments 💯

[Ghostvv]

@tttthomasssss is it only for DIET or for TED as well?

[tttthomasssss]

@Ghostvv it covers TED and the ResponseSelector as well

[parangitis]

what parameter/metrics do you use to determine the best model, is it combination of acc and loss or only one of them?