Add comparison plotting script

[tomaarsen]

Hello!

Pull Request overview

• Add a results comparison plotting script to the scrips folder.

Details

As promised in #268 (comment), this PR adds a potentially useful script for visualising the differences between various outputs of the run_....py (e.g. run_fewshot.py) scripts provided in this repository. I wrote it to help visualise the potential performance differences resulting from #268, and I think it might help others, too.

How to use

Firstly, one or more results must be generated, e.g. via run_fewshot.py:

python scripts/setfit/run_fewshot.py --dataset=sst2 --sample_sizes 2 4 8 16 --batch_size 32

Note that if the parameters between various runs are the same, but the repository code is different, then you must rename the results directory after generating some results. Otherwise, it'll assume the old results are from the exact same setup, and it'll skip generating the new results.

Afterwards, the plotting script can be used, pointing to the various model outputs that you want to compare, e.g.:

python scripts/plot_summary_comparison.py --paths \
    scripts\setfit\results_emotion_main_5_epochs\paraphrase-mpnet-base-v2-CosineSimilarityLoss-logistic_regression-iterations_20-batch_32 \
    scripts\setfit\results_emotion_unique_pairs_True_5_epochs\paraphrase-mpnet-base-v2-CosineSimilarityLoss-logistic_regression-iterations_20-batch_32

This produces outputs like can be seen in #268 (comment) and #268 (comment). One example that used five results paths is this plot:

• Tom Aarsen