Add tests for no_trainer and fix existing examples

[muellerzr]

New tests for the no_trainer scripts

What does this add?

• Adds in test cases for each of the no_trainer scripts, mocking how the Transformers counterparts work
• Fixes a small variety of bugs inside the no_trainer scripts, discovered while writing these tests
• Introduces the ability to write a json file at the end of training, so that tests can be performed, similar to the Transformers tests

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[muellerzr]

CI failures were fixed by removing:

        if torch_device != "cuda":
            testargs.append("--no_cuda")

from clm, mlm, and ner.

From what I could see they were unused, so I didn't duplicate them from the transformers tests. Let me know if they should be added back in, with special behavior on those tests 😄

[sgugger]

Great work! 😍

[sgugger]

For information, here are the durations:

61.24s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_swag
60.97s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_squad
51.48s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_speech_recognition_seq2seq
44.82s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_swag
40.75s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_squad
32.17s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_squad_seq2seq
27.32s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_ner
26.55s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_speech_recognition_ctc
26.51s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_clm
21.61s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_ner
18.85s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_clm
17.32s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_glue
16.42s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_wav2vec2_pretraining
15.14s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_glue
14.38s call     examples/pytorch/test_accelerate_examples.py::ExamplesTests::test_run_mlm
14.05s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_mlm
3.41s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_audio_classification
1.05s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_run_clm_config_overrides
0.76s call     examples/pytorch/test_pytorch_examples.py::ExamplesTests::test_generation
56 durations < 0.05 secs were omitted

Could the run_swag_no_trainer be made a bit faster? The other ones look okay.

[muellerzr]

Changed checkpointing tests to be by epoch, and also not saving with swag.
Reduced time by almost 40% overall

Here were those times locally for me:

Before

======================================================================================= slowest durations ========================================================================================
15.11s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_swag_no_trainer
9.99s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_ner_no_trainer
9.70s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_squad_no_trainer
7.90s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_clm_no_trainer
6.33s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_glue_no_trainer
4.39s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_mlm_no_trainer

After

======================================================================================= slowest durations ========================================================================================
7.47s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_clm_no_trainer
6.30s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_squad_no_trainer
5.33s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_ner_no_trainer
5.13s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_glue_no_trainer
4.06s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_swag_no_trainer
3.89s call     examples/pytorch/test_accelerate_examples.py::ExamplesTestsNoTrainer::test_run_mlm_no_trainer