tensor-split meta backend fixes

[max-krasnyansky]

Overview

Ran into a few issues with the tensor-split meta backend while working on #26501

• It was not propagating buffer_usage mode
  We need to see usage=weights in ggml-hexagon and ggml-opencl
• It was not calling init_tensor for the new tensors it creates with the secondary backend buffer
  This caused some of the tensors to end up not fully initialized, specifically for ggml-hexagon
  (and probably opencl and sycl) the issue was missing tensor->extra which is allocated in init_tensor.
• And the last thing, it was assuming that every single op is supported by its secondary backends which means the CPU fallback didn't work.
  We typically rely on the CPU fallback in ggml-hexagon (at least the embeddings and the lm-head) so it makes sense to allow for those with the tensor-split mode.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, as part of the main PR mentioned above

[max-krasnyansky]

@JohannesGaessler @ggerganov
Please let me know if you have any suggestions/objections. Feel free to push into that branch if you want to update things further.

[max-krasnyansky]

@am17an @JohannesGaessler @gaugarg-nv
Can you guys please ack/nack/update this?
Wanted to unblock my main PR.

[ggerganov]

In which cases does this occur?

[max-krasnyansky]

@ggerganov apologies for the delay (traveling). This was the first thing I ran into when I started testing ggml-hexagon on dual-NPU devices with split-mode tensor. One of the CPU fallbacks was hitting this path but I don't recall which model. I think it was Qwen3.5-4B. I don't have access to the dual-NPU device at the moment so I can't quite reproduce the scenario.
It seems that this path might end up calling ggml_backend_meta_get_split_state on the next->src[] that is mapped to the CPU backend.

                // Skip MIRRORED nodes that don't consume node
                auto skip_unrelated = [&]() {
                    while (id + 1 < cgraph->n_nodes) {
                        ggml_tensor * next = cgraph->nodes[id+1];
                        if (ggml_backend_meta_get_split_state(next, false).axis != GGML_BACKEND_SPLIT_AXIS_MIRRORED) {
                            break;
                        }
                        bool safe = true;
                        for (int s = 0; s < GGML_MAX_SRC; s++) {
                            if (next->src[s] == nullptr) {
                                continue;
                            }
                            if (next->src[s] == node) {
                                safe = false;
                                break;
                            }
        >>>>>>>             if (ggml_backend_meta_get_split_state(next->src[s], false).axis != GGML_BACKEND_SPLIT_AXIS_MIRRORED) {
                                safe = false;
                                break;
                            }

[ggerganov]

I think we should take a look at a repro and see if we can reproduce with Metal/CUDA virtual devices.

[max-krasnyansky]

@ggerganov sorry for the delay.
I can't seem to reproduce the original failure. So I removed this specific change. If I run into it again I'll start a separate PR.

[youyoulyz]

Hi, I'm hitting a crash on latest master (929d47a) when using -sm tensor with a Qwen3.8-27B Q4_K_M model on 2x RTX 3080. It works fine on pr-27342 (5ecbe1a).

Error:
ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
CUDA error: an illegal memory access was encountered
current device: 0, in function ggml_backend_cuda_buffer_init_tensor
cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size)

Preceded by:

common_fit_params: failed to fit params to free device memory: llama_params_fit is not implemented for

SPLIT_MODE_TENSOR, abort

Bisect points to:

d59d455 tensor-split meta backend fixes (#26502)

Repro command:

llama-server \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  -ngl 99 -sm tensor -c 262144 \
  --parallel 2 -fa on -ctk q8_0 -ctv q8_0 --kv-unified

[ggerganov]

I can confirm the crash using virtual CUDA devices.

@youyoulyz Could you confirm it works with #27433?

[markun]

I had the same crash. Moving the init after the view_src / view_offs / data assignments seems to fix it, but I'm not 100% sure if that's the correct fix.

diff --git a/ggml/src/ggml-backend-meta.cpp b/ggml/src/ggml-backend-meta.cpp
index 775ae9926..ef1a2a345 100644
--- a/ggml/src/ggml-backend-meta.cpp
+++ b/ggml/src/ggml-backend-meta.cpp
@@ -1179,12 +1179,6 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m
         ggml_set_name(t_ij, tensor->name);

         t_ij->buffer = simple_buf;
-        if (simple_buf) {
-            // the backend that owns the buffer will set .extra
-            ggml_backend_buffer_init_tensor(simple_buf, t_ij);
-        } else {
-            t_ij->extra = tensor->extra;
-        }

         t_ij->view_src = tensor->view_src;
         t_ij->view_offs = tensor->view_offs;
@@ -1216,6 +1210,12 @@ static enum ggml_status ggml_backend_meta_buffer_init_tensor_impl(ggml_backend_m
             t_ij->data = (char *) ggml_backend_buffer_get_base(simple_buf)
                 + size_t(tensor->data) - size_t(ggml_backend_buffer_get_base(tensor->buffer));
         }
+        if (simple_buf) {
+            // the backend that owns the buffer will set .extra
+            ggml_backend_buffer_init_tensor(simple_buf, t_ij);
+        } else {
+            t_ij->extra = tensor->extra;
+        }
         for (int i = 0; i < GGML_MAX_SRC; i++) {
             t_ij->src[i] = tensor->src[i];
             if (tensor->src[i] == tensor) {

[youyoulyz]

I can confirm the crash using virtual CUDA devices.

@youyoulyz Could you confirm it works with #27433?

@ggerganov Confirmed it is working, details are in #27433

[ggerganov]

@max-krasnyansky Let's revert this for now. I am thinking we need to have -sm tensor tests added to the CI. At least running test-llama-archs and the E2E server workflow. We are almost there now that we support virtual devices.

[max-krasnyansky]

@max-krasnyansky Let's revert this for now. I am thinking we need to have -sm tensor tests added to the CI. At least running test-llama-archs and the E2E server workflow. We are almost there now that we support virtual devices.

Ah. Bummer. Ok. I'll try to find a CUDA setup to test on and resubmit after that.
And yes, tensor-split tests in the CI would be great.

[cdanis]

On my dual RTX A5000s, I was able to reproduce a crash at first, but I can also verify that @markun's patch fixed it for me. After that I ran some performance and perplexity tests and did one agentic session, and all seemed well.

[cdanis]

@max-krasnyansky if you need more testing on CUDA please let me know!

[max-krasnyansky]

@max-krasnyansky if you need more testing on CUDA please let me know!

@cdanis Yes please #27586