cuda: backport upstream revert of "reset cuda context after reading memory size" (#24715) - #238
Conversation
|
I double check in ggml-org it already know this issue the ggml-org pull/24682 they not fix they |
…23935)" (ggml-org#24715) This reverts commit 0f7fada.
|
test [OK] - model reload (7 configs, 6 reloads) for not test - multi-GPU |
|
Confirmed the second commit is byte-identical to upstream's revert ea21e03 (ggml-org#24715), and agree the revert beats the ever_active gate given upstream closed ggml-org#24682 in favor of reverting. Post-merge test on GB10: llama-bench --n-cpu-moe 0,8,16 -ctk turbo4 -ctv turbo3 (2 model reloads) completes with no 'CUDA error: invalid argument'. Nice writeup of the upstream history — made this an easy review. |
Close #237
### Suggested fixOnly reset a device that has never been used. Add an ever_active flag toggml_backend_cuda_device_context , set it at the three sites that do active_count++(buffer alloc, host buffer alloc, backend init), and gate the reset on it:if (ctx->active_count == 0 && !ctx->ever_active) {CUDA_CHECK(cudaDeviceReset());}This preserves the original intent -- avoiding a lazily created context that permanentlyholds VRAM when memory is queried before anything has been allocated -- while neverresetting a context whose kernel state other code still assumes is live.- [x] Before the fix, llama-bench --n-cpu-moe 32,34,36,38,40,42,44 (6 model reloads)aborts with CUDA error: invalid argument right after the first configuration- [x] With the fix, the same sweep completes all 14 rows with no abortWhy a revert rather than a targeted fix
The first commit in this PR (
f3c29b7) gated the reset behind anever_activeflag so adevice that had already been used would never be reset. It works, but upstream has
already settled this differently:
0f7fada56entirely inea21e0395(Revert "cuda: reset cuda context after reading memory size (#23935)" ggml-org/llama.cpp#24715), authored by the sameperson who wrote the original commit
re-applied kernel attributes via a context generation counter -- the same class of fix
as
ever_activeCUDA: ggml_cuda_compute_forward: MUL_MAT failed(the crash reported here)--fitmodel load taking 20x longerThis branch has diverged from upstream at this point: it carries
0f7fada56but not therevert. The second commit (
af8ac64) cherry-picksea21e0395, which supersedes thefirst commit and brings this file back in line with upstream.
llama-bench --n-cpu-moe 32,34,36,38,40,42,44 -ngl 99 -ctk turbo4 -ctv turbo3 -b 4096 -ub 2048aborts withCUDA error: invalid argumentimmediately after the first configuration
with no abort
-DGGML_CUDA=ON, no errors