kv : change structures in kv-cells for improve operations asymptotic

[lexasub]

Overview

Change containers for llama-kv-cache for boost lookup and modify
replace std::bitset<LLAMA_MAX_SEQ> -> uint64_t w[N_SEQ_WORDS]
replace std::map<llama_pos, int> seq_pos[LLAMA_MAX_SEQ] -> seq_pos_t seq_pos[LLAMA_MAX_SEQ]

Additional information

May be filled later after discussion

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - write tests, review code

[lexasub]

@ngxson, also may be your review may good)

[ggml-gh-bot]

Hi @lexasub, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[lexasub]

Hi @lexasub, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.
• Large PR: Large changes require prior discussion (e.g. an issue or RFC) and maintainers may not be able to review this PR as-is. Consider splitting it into smaller, focused PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Multiple open PRs from a new contributor: Drafts don't count toward the limit. ) now i have only old draft commits.
Large PR: most diff - in test 1151 lines)

[lexasub]

rtx 5090 - (pci-e-v3) E5-2620 v2. Muse Glimmer q4

-np 2 -c 262144 -fa on

before changes : (in progress)
PP(input) :
126617 tok,
2319 decode,
TG(output) :
1053 tok
65.14 t/s

after changes :
PP(input) :
126617 tok,
2339 decode,
TG(output) :
1258 tok
65.89 t/s

log (2).log

[ngxson]

this change is quite excessive tbh, I doubt if this code path is hot enough to be consider

we can proceed if you can proof if this brings significant end-to-end perf improvement (pp or tg measure)

most of the time is spent on ggml's graph sched IIRC, kv cache management is just too small

[lexasub]

also




(my changes best in this case)

[ngxson]

please use proper llama-bench

[lexasub]

before

after (smaller volatility)

rtx 5090 - (pci-e-v3) E5-2620 v2. Muse Glimmer q4 . gpu is very good for this cpu. cpu very slow
now retrying with nvidia-sm -lgc 2400, ./bin/llama-bench -m ~/Muse-Glimmer-30B-UD-Q4_K_XL.gguf -ngl 99 -fa on
-pg 1024,256 -pg 8192,256 -pg 32768,256 -pg 65536,256 -pg 126613,256
-r 5 --prio 3 --poll 100 --delay 2 -o csv

[ngxson]

I don't see any significant improvements here, the PR's result is already within master's error margin

[lexasub]

okay, i go the deeper, llama-bench on profiling didn't show kv-cells usage( but via llama server i see it in profile), i will notify when kv cache give improvement on llama-bench

in my opinion - llama-bench doesn't test prefill, tokenization. this mr improve prefil!