[benchmark tool] trainer-benchmark.py

[stas00]

This PR adds a benchmarking tool for HF Trainer args - e.g. compare --fp16 vs --bf16 performance, but can do that for multiple dimensions and it prints automatic tables suitable for instant pasting in Issues, including relative performance. e.g.:

Variation	Train samples per second	Diff %	Train loss
--per_device_train_batch_size 1	7.77	0	1.90
--per_device_train_batch_size 2	15.51	100	2.01
--per_device_train_batch_size 4	29.66	282	2.09
--per_device_train_batch_size 8	61.16	687	2.16
--per_device_train_batch_size 16	115.84	1392	2.25
--per_device_train_batch_size 32	224.96	2797	2.38

This is produced by:

CUDA_VISIBLE_DEVICES=0 python ./scripts/benchmark/trainer-benchmark.py \
--base-cmd \
' examples/pytorch/translation/run_translation.py --model_name_or_path t5-base \
--output_dir output_dir --do_train --label_smoothing 0.1 --logging_strategy no \
--save_strategy no --max_source_length 512 \
--max_target_length 512 --num_train_epochs 1 --overwrite_output_dir \
--source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" \
--source_prefix "translate English to Romanian: " --warmup_steps 50 \
--max_train_samples 5000 --dataloader_num_workers 2 --bf16' \
--target-metric-key train_samples_per_second --repeat-times 1 --variations \
'--per_device_train_batch_size 1|--per_device_train_batch_size 2|--per_device_train_batch_size 4|--per_device_train_batch_size 8|--per_device_train_batch_size 16|--per_device_train_batch_size 32' \
--report-metric-keys train_loss --repeat-times 1 

To add more dimensions simply add another --variations arg, e.g.:

--variations '|--fp16|--bf16' '--tf32 0|--tf32 1' 

will lead to a Cartesian product of each arg with an outcome of:

Variation	Train samples per second	Diff %	Train loss
--tf32 0	272.59	0	2.49
--tf32 1	581.61	113	2.49
--fp16 --tf32 0	643.07	136	2.49
--fp16 --tf32 1	635.24	133	2.49
--bf16 --tf32 0	616.23	126	2.50
--bf16 --tf32 1	612.59	125	2.50

See the doc at the beginning of the script for details. But it has lots of cool features like automatically reporting the hardware/software, preformatting everything for both copy-n-paste into the Issues/docs and also an easy to see console-only 2nd version at the end for when you debug things.

And more practical examples and reports for its use are here: #15026 and #14608

It should be relatively easy to adopt this tool to be used with accelerate or with any other command line tool as long as there is a defined way to get to results, so each tool will need to have its own sub-class if we decided to extend it. I think @siddk suggested he might look into extending it.

But let's see first that you like it and I placed it in a good location. The intention for it to be an internal tool, so I hope any map and such code will be tolerated.

It's possible that we could use it for some basic regression testing. But it'd need to be further extended to support storing results and detecting regressions.

@LysandreJik, @patrickvonplaten, @sgugger, @patil-suraj

[sgugger]

Thanks a lot for working on this. Even if for internal use, we can make a slight effort to make the code more readable so anyone can contribute to it easily :-)
It's going to be very useful to have it!

[patrickvonplaten]

I don't really understand where those numbers are coming from. Are they random?

[stas00]

This is for debugging the output formatting - i.e. still perfecting the look-n-feel of the result reports, so if you want to tweak it these return immediately and with a few re-runs it'll give different outputs to validate that different numbers will be formatted well.

[stas00]

I added a note explain what is it for

[patrickvonplaten]

Do you think it could be possible to also provide a link to where additional metric keys could be found?

[stas00]

do we have that information? it'd be different in different scripts, no?

That's why I documented output_dir/all_results.json so once you run it you will see all the available keys there for the given type of script.

[patrickvonplaten]

do the variations just correspond to the precision? Maybe call it precision_variations then?

[stas00]

no.

Variations are just that - variations - it can be anything. --variations "A|B|C" "D|E|F" will do a Cartesian product to produce A D, A E, ..., C F and run each variation. If it's just --variations "A|B|C" it will just run A, then B, then C.

I was just using precisions as an example. but you can compare say --variations "--bs 4|--bs 6|--bs 8" (we don't have --bs, but you get the idea).

I'm open to suggestions if the name is not intuitive.

[patrickvonplaten]

I see! Sorry I misread it then. But then variations can only correspond to args of TrainingArguments no? So maybe we can say training_args_variations?

[stas00]

The idea is that this tool is going to be expanded to support other programs which don't necessarily have training args, after all it doesn't care which args are varied. @siddk is planning to expand it to support accelerate and down the road others, hence the generic name.

So the mnemonic is "variations to compare" whatever they are.

[patrickvonplaten]

Very cool! Thanks for making this available to everybody. Didn't understand all the command line code in-detail, but looks like a very useful script to have

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.