Skip to content

Examples : Add new sweep-bench benchmark #225

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Feb 23, 2025
Merged

Examples : Add new sweep-bench benchmark #225

merged 4 commits into from
Feb 23, 2025

Conversation

@saood06
Copy link
Collaborator

@saood06 saood06 commented Feb 23, 2025

Port of ggml-org/llama.cpp@9488fbf

This is a good tool to benchmark with as requested by #223.

As a very quick demo I generated this, just by running this ( ./llama-sweep-bench -c 2048 -ub 512 -m WizardLM-2-8x22B-IQ4_K_R4.gguf -ctk q8_KV -ctv q8_0 -fa --output-format jsonl and then sweep-bench-plot.py with the output).

performance_comparison_pp

performance_comparison_tg

  • Self-reported review complexity:
    • Low
    • Medium
    • High

ikawrakow
ikawrakow approved these changes Feb 23, 2025
View reviewed changes
Copy link
Owner

@ikawrakow ikawrakow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this - can be very useful.

@saood06 saood06 merged commit 46bf73a into main Feb 23, 2025
Nexesenex added a commit to Nexesenex/ik_llama.cpp.nxs that referenced this pull request Mar 3, 2025
@Nexesenex
Revert "Add new sweep-bench benchmark (ikawrakow#225)"
a1f07f1
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Apr 26, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
e59a5f1 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
@ubergarm
Copy link
Contributor

@saood06 thanks I'm a convert to llama-sweep-bench! It is indeed very useful.

I pushed a branch on my personal mainline llama.cpp fork just to use for testing performance across forks. I don't plan to open a PR to mainline, but just left it up there in case anyone else is using it. I'm guessing ik has something similar as we were comparing the new GLM-4 performance.

Thanks!

ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 3, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
cb5f8f1 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 4, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
2067e45 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 7, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
c73fc5b 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 13, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
5813f60 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 18, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
40544c2 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request May 30, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
6ab18b5 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jun 22, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
b682982 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jun 27, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
00d375c 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jun 28, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
8550185 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jul 2, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
cdaf1a0 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jul 10, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
efd55db 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jul 11, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
69f2230 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Jul 12, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
595a2ad 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 5, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
f1d6a68 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 10, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
5a8c5c9 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 14, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
d50048d 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 17, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
e775216 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 24, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
16588ae 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 25, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
f876771 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 28, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
789cbdf 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Aug 29, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
b62c466 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 2, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
b09aa80 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 5, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
d69c3f1 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 11, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
717ab10 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 22, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
d04d196 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 25, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
6310bb6 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 26, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
049c649 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Sep 28, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
ee52039 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Oct 1, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
7fbdeaf 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Oct 13, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
0293a5f 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Oct 15, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
0673119 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
ubergarm added a commit to ubergarm/llama.cpp that referenced this pull request Oct 16, 2025
@ubergarm
Refactor ik_llama.cpp back to llama.cpp
22dff21 
This is based on saood06's PR ikawrakow/ik_llama.cpp#225
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ikawrakow ikawrakow ikawrakow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@saood06 @ubergarm @ikawrakow @sszymczy