Warmup and NUMA changes added, MLA changes updated

[saood06]

This is actually usable at much higher context size, main worked fine under llama-batched-bench but paged out and had worse performance in server, and was limited to ~8k. This is tested in server at ~8k, with TG at 2.18, and is currently launched with 64K, 128K errors out. Tested on dual socket Xeon E5-2690 v3 with 384 GB RAM.

CPU buffer size = 362010.72 MiB
n_ctx      = 64000
CPU NUMA KV buffer size = 313101.56 MiB
KV self size  = 305000.00 MiB, K (f16): 183000.00 MiB, V (f16): 122000.00 MiB
KV self size  = 4289.06 MiB, K^R (f16):  476.56 MiB, c^KV (f16): 3812.50 MiB
NUMA compute buffer size = 32343.01 MiB

• To-do: Fix NUMA to actually toggle depending on if NUMA is enabled or not.
• To-do: Sync RPC to make it functional and add performance and add override model tensor buffers [#11397] ( will have limited practical benefit while old KV cache is still being allocated and not used). Edit: Based on initial tests on llama.cpp this causes performance loss, and also does not function past ~28 layers.
• To-do: Grab any FA implementation if one appears.

[fairydreaming]

Cool! Note that this mmap based KV buffer allocator shall work even without NUMA, I shall probably name it ggml_backend_mmap_buffer_type.

[saood06]

Cool! Note that this mmap based KV buffer allocator shall work even without NUMA, I shall probably name it ggml_backend_mmap_buffer_type.

Makes sense, even though it synergizes with NUMA, it still has benefits without NUMA.

If you end up using this branch, I'd appreciate performance numbers.

I'm going to try and crudely make it so that the old KV cache does not allocate, because I do think if that's done you can offload everything besides the non shared experts with just 24GB of VRAM at ~23k context, assuming it works. I was only able to RPC 29 layers even when I had ample VRAM, 30+ would just silently crash the RPC server ( without the MLA branch, I have yet to test the diff you gave me to make the MLA branch work).

[saood06]

Some runtime numbers showing 30K context.

kv cache rm [p0, end) | timestamp=1738584936 p0=3
kv cache rm [p0, end) | timestamp=1738585206 p0=2051
kv cache rm [p0, end) | timestamp=1738585594 p0=4099
kv cache rm [p0, end) | timestamp=1738586098 p0=6147
kv cache rm [p0, end) | timestamp=1738586716 p0=8195
kv cache rm [p0, end) | timestamp=1738587443 p0=10243
kv cache rm [p0, end) | timestamp=1738588289 p0=12291
kv cache rm [p0, end) | timestamp=1738589245 p0=14339
kv cache rm [p0, end) | timestamp=1738590323 p0=16387
kv cache rm [p0, end) | timestamp=1738591540 p0=18435
kv cache rm [p0, end) | timestamp=1738592866 p0=20483
kv cache rm [p0, end) | timestamp=1738594456 p0=22531
kv cache rm [p0, end) | timestamp=1738596175 p0=24579
prompt eval time     = 12074054.06 ms / 25522 tokens (  473.08 ms per token,     2.11 tokens per second) | timestamp=1738599260 t_prompt_processing=12074054.06 n_prompt_tokens_processed=25522 t_token=473.08416503408824 n_tokens_second=2.113788779905463
generation eval time = 2250383.89 ms /  2088 runs   ( 1077.77 ms per token,     0.93 tokens per second) | timestamp=1738599260 t_token_generation=2250383.888 n_decoded=2088 t_token=1077.7700613026818 n_tokens_second=0.9278416945366968
total time = 14324437.95 ms | timestamp=1738599260  t_prompt_processing=12074054.06 t_token_generation=2250383.888 t_total=14324437.948

At higher context you can see how PP slows down as it gets deeper into the prompt.

Another high context generation, less PP in this one as it cached most of the previous prompt.

kv cache rm [p0, end) | timestamp=1738649710 p0=26931
prompt eval time     =  126917.48 ms /   143 tokens (  887.53 ms per token,     1.13 tokens per second) | timestamp=1738652616 t_prompt_processing=126917.477 n_prompt_tokens_processed=143 t_token=887.5348041958042 n_tokens_second=1.1267163780761258
generation eval time = 2778653.10 ms /  2726 runs   ( 1019.32 ms per token,     0.98 tokens per second) |  timestamp=1738652616 t_token_generation=2778653.096 n_decoded=2726 t_token=1019.3151489361702 n_tokens_second=0.9810508565909877
 total time = 2905570.57 ms | timestamp=1738652616 id_slot=0 id_task=11466 t_prompt_processing=126917.477 t_token_generation=2778653.096 t_total=2905570.573