ggml : persistent expert cache for --n-cpu-moe (RFC #20757)

[e1n00r]

PR withdrawn by author.

[ggml-gh-bot]

Hi @e1n00r, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• AI-generated content: This project does not accept PRs, descriptions or commit messages that are fully or predominantly AI-generated. If you have used AI to assist you in writing code, please make sure to disclose that explicitly.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[am17an]

Please stop submitting such large PRs. No one will review them unless you demonstrate you can understand and maintain the code.

[vlascik]

@am17an the PR is below 1000 lines spread over 11 files, most of it is in tests, and it's well structured. I really don't know what other way you want actual features to get done.

Especially the ones that can get 2x speed ups on MoEs for free.

I mean, if you're too lazy to shepherd this through, at least leave the discussion about it open instead of insta-closing it on a whim.

[am17an]

@vlascik I don't think you have enough experience contributing to the llama.cpp code (AFAIK your experience is 0) to comment on the scope of this change. Discussions can be opened via the discussion page, and all PRs need to follow the contributing guidelines. Moreover, Please note that maintainers reserve the right to make final decisions on PRs.

[vlascik]

@am17an if 1000 line PR is too much of a scope for a project, the project is fubar. is llama project so broken it can't handle 1000 line PRs? state so publicly, so that the community can stop wasting time contributing to a project like that and move on.

You know, I've been watching this for a while, and here are some of the facts, fairly obvious even "without an experience of contributing to llama.cpp code":

1. llama's cpu offloading "strategy" is, let's just say, bad.
2. there have been years of research and dozens of papers written about offload caching for MoEs. with the speedups in range of 2-11x reported too.
3. despite that, there seems to be no movement and no plan on this from llama's side. what are you even doing.

Do you have a different plan for some free-range, hand-crafted version of this PR? Let's hear it.

Or, you know, you could, at the absolute very least, try to not gate-keep meaningful contributions without a reason better than "I'm not feeling it".

[am17an]

Do you have a different plan for some free-range, hand-crafted version of this PR? Let's hear it.

Yes I do actually. See #21067

Also, please be cordial. Consider this your last warning.