feat(bench): convenient benchmark tool

[doraneko6]

Hi,

Recently I was comparing the running speed of the mlx framework and the llama.cpp framework. I found that the llama.cpp repository provides a tool like llama-bench to allow the community to easily perform fast and multiple benchmarks with different models, seqlen..., while mlx does not seem to have a tool with similar functions.

I also found that there are related issues in the community that mentioned this, such as issue 126 in old repo.

I think a good, convenient benchmarking tool would give the community more motivation to provide more benchmark data on their platforms, similar to what llama.cpp does, we can also maintain detailed data provided by the community on different models on their Apple devices.

Inspired by gauri-nagavkar/mlxlm_bench, I wrote this benchmark suite so that the community can directly perform detailed evaluation similar to llama-bench with the mlx_lm.bench command.

For example, use
mlx_lm.bench -m $HOME/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4,$HOME/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4 -p 1 -n 16,32 -r 2
will generate below information:

Model	Model Load Time (s)	Prompt Tokens	Prompt TPS	Response Tokens	Response TPS	Execution Time (s)	Memory Usage (GB)
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4	0.404	1	134.849	16	184.841	0.094	1.75
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4	0.404	1	132.548	32	178.798	0.187	1.75
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4	0.544	1	108.821	16	139.084	0.124	6.02
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4	0.544	1	110.32	32	134.757	0.247	6.02

Which gives some information like the Prompt TPS, Response TPS and so on.

I hope this module can bring more vitality to the community, because I have observed that there are still many discussions on the benchmark discussion of llama.cpp.

Regards,
B1ACK917

[ivanfioravanti]

I think this is now covered by mlx_lm.benchmark, no?

[awni]

Yes let's close this. It has some nice features that we may want to rebase on the existing benchmark tool (like support for multiple prompt / generation lengths, maybe the output format thing as well). But we can add those on an as-needed basis.

[awni]

PS thanks @doraneko6 for the contribution! If you are interested to add any of the above the PRs would be welcome!