enabling accuracy tests for mbxp gsm8k datasets (#178)

[rbogdano]

This change was requested to enable accuracy evaluating on more datasets like GSM8k, MBXP and OpenOrca. To do this test without my change I needed to use evaluation script from vllm-benchmark repository which is confusing.
The dataset that I was using is a combination of three datasets (OpenOrca, mbxp, gsm8k).

This accuracy check was done using mixtral-8x7b. Results below:

{'rouge1': 45.5749, 'rouge2': 23.3549, 'rougeL': 30.4524, 'rougeLsum': 42.5538, 'gsm8k': 73.88, 'mbxp': 60.04, 'gen_len': 4298469, 'gen_num': 15000, 'gen_tok_len': 2805167, 'tokens_per_sample': 187.0, 'performance': 0, 'accuracy': 99.8}

[regisss]

Can you move all these files to a new folder called examples/text-generation/mbxp_evaluation and add a README there explaining quickly how to install and run everything please?

[karol-brejna-i]

We are very close to the release date.
@rbogdano Is there a chance to introduce required changes in the next release (1.18 with Synapse 1.21) or do we need to postpone it?
It looks like it would be good to have this PR merged...

[rbogdano]

Can you move all these files to a new folder called examples/text-generation/mbxp_evaluation and add a README there explaining quickly how to install and run everything please?

fixed

[regisss]

Can't we move all files to the mbxp_evaluation folder?

[kwisniewski98]

@regisss I've moved those files, but now I'll be testing that, will let you know when I finish testing it

[regisss]

@kwisniewski98 I suggest to also add a small README.md file in examples/text-generation/mbxp_evaluation just to know how to use this code

[rbogdano]

We've identified new issues that are blocking this PR.
Currently, we're not generating the JSON with the responses required by evaluation.py. As a result, this PR is incomplete.
The next step would be to implement response generation in run_generation.py, but doing so would take additional time. Properly testing this functionality would also require additional effort and time.
Therefore, I suggest moving this change to next release.

[rbogdano]

I am closing this PR due to lack of functionality in run_generation.py script.

In the same time I would like to introduce fix for this issue in another PR where I implemented this functionality and added results evaluation guide.
#1986