add: mlx_lm.bench for more convenient benchmark (#126)

[doraneko6]

Hi,

Recently I was comparing the running speed of the mlx framework and the llama.cpp framework. I found that the llama.cpp repository provides a tool like llama-bench to allow the community to easily perform fast and multiple benchmarks with different models, seqlen..., while mlx does not seem to have a tool with similar functions.

I also found that there are related issues in the community that mentioned this, such as #126.

I think a good, convenient benchmarking tool would give the community more motivation to provide more benchmark data on their platforms, similar to what llama.cpp does, we can also maintain detailed data provided by the community on different models on their Apple devices.

Inspired by gauri-nagavkar/mlxlm_bench, I wrote this benchmark suite so that the community can directly perform detailed evaluation similar to llama-bench with the mlx_lm.bench command.

For example, use
mlx_lm.bench -m $HOME/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4,$HOME/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4 -p 1 -n 16,32 -r 2
will generate below information:

Model	Model Load Time (s)	Prompt Tokens	Prompt TPS	Response Tokens	Response TPS	Execution Time (s)	Memory Usage (GB)
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4	0.404	1	134.849	16	184.841	0.094	1.75
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-3B-Instruct-Q4	0.404	1	132.548	32	178.798	0.187	1.75
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4	0.544	1	108.821	16	139.084	0.124	6.02
/Users/cyberdz/Files/mlx/Qwen/Qwen2.5-7B-Instruct-Q4	0.544	1	110.32	32	134.757	0.247	6.02

Which gives some information like the Prompt TPS, Response TPS and so on.

I hope this module can bring more vitality to the community, because I have observed that there are still many discussions on the benchmark discussion of llama.cpp.

Regards,
B1ACK917

[awni]

Moved ml-explore/mlx-lm#10