General discussion (issues / suggestions)

[Luro223]

Hello dear developer of KoboldAI CPP.

I've been using 1.79.1 since release and I have i3-3120M with HD Graphics 4000 and HD 8700M, since 1.79.1 I finally managed to use Vulkan for HD 8700M, it is a bit faster now, but still I can't use dGPU's capabilities, KoboldAI CPP still only uses CPU, it only uses small amount of GPU's memory, also it crashes if I use more than 0 GPU layers on Vulkan.

CLBlast worked earlier with HD Graphics 4000, but with 1.7X it stopped working, AMD Accelerated Parallel Processing never actually worked with CLBlast.

Using --sdvaeauto slightly increases performance, I'll show the results later.
Edit: --sdvaeauto slightly increases performance but in rare cases, I need to test more.

Also is it possible for you to add toggle-able DRY, XTC, Temperature, Top-P, Top-K, either from command-line interface or GUI? I don't use any of these with some models, because, I think, some of them might affect performance, even if not used (0 value), especially with XTC.

[Luro223]

Comparsion of Vulkan and Old CPU with 1K context.
Some performance boost, yet no dGPU utilization.
.....Vulkan, HD8700M.....
Processing Prompt (924 / 924 tokens)
Generating (100 / 100 tokens)
[00:00:00] CtxLimit:1024/1024, Amt:100/100, Init:0.10s, Process:378.21s (409.3ms/T = 2.44T/s), Generate:50.80s (508.0ms/T = 1.97T/s), Total:429.01s (0.23T/s)
Benchmark Completed - v1.79.1 Results:
.....
Flags: NoAVX2=True Threads=4 HighPriority=False Cublas_Args=None Tensor_Split=None BlasThreads=4 BlasBatchSize=-1 FlashAttention=True KvCache=2
Timestamp: 2024-12-18 00:00:00.000000+00:00
Backend: koboldcpp_vulkan_noavx2.dll
Layers: 0
Model: mistral7b-erebus-v3.Q4_K_M
MaxCtx: 1024
GenAmount: 100
.....
ProcessingTime: 378.215s
ProcessingSpeed: 2.44T/s
GenerationTime: 50.795s
GenerationSpeed: 1.97T/s
TotalTime: 429.010s
Output: 1 1 1 1
.....
.....CPU (Old CPU).....
Processing Prompt (924 / 924 tokens)
Generating (100 / 100 tokens)
[00:00:00] CtxLimit:1024/1024, Amt:100/100, Init:0.09s, Process:387.69s (419.6ms/T = 2.38T/s), Generate:53.23s (532.3ms/T = 1.88T/s), Total:440.93s (0.23T/s)
Benchmark Completed - v1.79.1 Results:
.....
Flags: NoAVX2=True Threads=4 HighPriority=False Cublas_Args=None Tensor_Split=None BlasThreads=4 BlasBatchSize=-1 FlashAttention=True KvCache=2
Timestamp: 2024-12-18 00:00:00.000000+00:00
Backend: koboldcpp_noavx2.dll
Layers: 0
Model: mistral7b-erebus-v3.Q4_K_M
MaxCtx: 1024
GenAmount: 100
.....
ProcessingTime: 387.695s
ProcessingSpeed: 2.38T/s
GenerationTime: 53.235s
GenerationSpeed: 1.88T/s
TotalTime: 440.930s
Output: 1 1 1 1
.....

[LostRuins]

CLBlast with 0 layers doesnt work at all?

[Luro223]

CLBlast with 0 layers doesnt work at all?

Yeah, absolutely, neither with HD Graphics 4000 or AMD Accelerated Parallel Processing(Oland), though it worked with versions earlier than 1.7X, but only with HD Graphics 4000.

[Luro223]

CLBlast(Old CPU) with 0 layers(1.79.1)
Loading model: L:\AI-Models\mistral7b-erebus-v3.Q4_K_M.gguf
Traceback (most recent call last):
File "koboldcpp.py", line 5009, in
main(parser.parse_args(),start_server=True)
File "koboldcpp.py", line 4630, in main
loadok = load_model(modelname)
File "koboldcpp.py", line 930, in load_model
ret = handle.load_model(inputs)
OSError: [WinError -1073741795] Windows Error 0xc000001d
[776] Failed to execute script 'koboldcpp' due to unhandled exception!

[Luro223]

Same error with Oland.
If windbg or System Informer is enough to dump error, then I could use one of them and send dmp to you.

[Luro223]

Also if it's true or not, I noticed 1.79.1 is less creative compared to 1.69.1 is it because of DRY or XTC, I don't know, but the outputs are always different compared to 1.69.1, even if I disable DRY and XTC.
For example 1.69.1 gives more creative responses, but 1.79.1 gives more logic responses, even with DRY/XTC 0 and different models. I can reproduce this and show you if you're interested.

[LostRuins]

The 2 versions shouldnt have any difference in creativity. 1.80 is just released, you can try that.

[Luro223]

After a comprehensive testing of 1.80 the same errors, but noticeable performance boost(especially with longer contexts).
1.79.1 VS 1.80 (Vulkan-HD8700M)
Vulkan-HD8700M-1.79.1.txt
Vulkan-HD8700M-1.80.0.txt
CLBlast 1.69.1 VS 1.80.0 (to prove that 1.80.0 still doesn't work) (Also I've noticed 1.7X-1.80 uses koboldcpp_clblast.dll, while 1.69.1 uses koboldcpp_clblast_noavx2.dll, even though 1.80.0 has this library, yet didn't see it being used)
CLBlast (Old CPU)-I3-3120M.txt
CLBlast (Old CPU)-Oland.txt
CLBlast NoAVX2 (Old CPU)-I3-3120M(1.69.1).txt

[Luro223]

Also to clarify all confusions with Creativity, I'll provide all fine-tuned custom settings for UI and saved Character Card, so you'll able to reproduce all the errors related to creativity
UI Settings :
set-005.zip
Test Character Cards (GENERATED RESULTS) :
1.69.1,1.80.zip
Test Character Card (CLEAN TO GENERATE) :
Kirby-clean.zip
Main LLM I used for all this :
L3-8B-Stheno-v3.2-NEO-V1-D_AU-Q4_K_M-imat13 Link

[Luro223]

All the errors is happening when you try to change DRY, even if Mult./Base/A.Len 0, and cannot be disabled either from UI or from command-line arguments, even others like XTC, TOP K and etc.
For example all the settings related to DRY are 0, everything generated will be weird.

[Luro223]

Also GPU Utilization is same 0% with 1.80, only about ~40MB of GPU is used, and 100% of CPU :

Also other presets perform very well too (Old Cpu), and I already showed the comparsion of VULKAN:
1.79.1.txt
1.80.0.txt
(Only 256 context, but with bigger contexts works better).

[Luro223]

Also Vulkan with more than 0 layers crashes :
Vulkan-1Layers.txt

[LostRuins]

It will only use noavx2.dll if you selected "old cpu" option. If you have avx2 support you should not use that!

[Luro223]

It will only use noavx2.dll if you selected "old cpu" option. If you have avx2 support you should not use that!

Then what other variant do I have? CPU (Old CPU) Works, but CLBlast (Old CPU) doesn't, it uses koboldcpp_clblast.dll, even with --noavx2 --nommap --usecpu flags, koboldcpp_clblast_noavx2.dll library is completely unusable.

[Luro223]

If I delete koboldcpp_clblast.dll and rename koboldcpp_clblast_noavx2.dll to koboldcpp_clblast.dll it works surprisingly well(Only used 256 context for testing):
1.80_CLBlast_noavx2-FORCED.txt
1.80_CLBlast_noavx2-FORCED-BENCHMARK.txt(Comparsion between CLblast(Intel OpenCL-HD Graphics 4000) and CPU (Old CPU))
1.80_CLBlast_noavx2-FORCED-BENCHMARK-Intel(R) HD Graphics 4000.txt(Intel(R) HD Graphics 4000 only)
1.80_CLBlast_noavx2-FORCED-BENCHMARK-Intel(R) Core(TM) i3-3120M CPU @ 2.50GHz.txt(i3-3120M only)
(Mostly because I have 2 intel OpenCL devices, one uses directly (called Intel(R) Core(TM) i3-3120M CPU @ 2.50GHz), and one from driver (Intel(R) HD Graphics 4000).
Also Use CPU (Old CPU) is a bit faster than Intel's OpenCL, but OpenCL works faster with longer contexts.
Also Oland with CLblast still doesn't work, gives absolutely identical error as 1.69.1(cl_khr_f16 (not supported)):
1.80_CLBlast_noavx2-FORCED-Oland.txt