From 4c1178686b29c257d8eb9766b0cc6a8d9be84d3a Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 17:02:02 -0700 Subject: [PATCH 01/81] ggml-cuda: add internal AllReduce provider for tensor parallelism MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Introduces a NCCL-free AllReduce implementation for LLAMA_SPLIT_MODE_TENSOR using a single-phase CUDA kernel that pipelines D2H copy, cross-GPU handshake via pinned-memory volatile flags, and the reduction in one kernel launch per GPU. New files: - ggml/src/ggml-cuda/comm.cuh — ggml_cuda_allreduce_provider enum - ggml/src/ggml-cuda/allreduce.cuh — pipeline API declarations - ggml/src/ggml-cuda/allreduce.cu — kernel + pipeline init/dispatch ggml-cuda.cu changes: - ggml_backend_cuda_comm_context gains ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- NOTES-allreduce.md | 311 + ggml/src/ggml-cuda/allreduce.cu | 388 + ggml/src/ggml-cuda/allreduce.cuh | 36 + ggml/src/ggml-cuda/comm.cuh | 22 + ggml/src/ggml-cuda/ggml-cuda.cu | 10896 +++++++++++++++-------------- 5 files changed, 6241 insertions(+), 5412 deletions(-) create mode 100644 NOTES-allreduce.md create mode 100644 ggml/src/ggml-cuda/allreduce.cu create mode 100644 ggml/src/ggml-cuda/allreduce.cuh create mode 100644 ggml/src/ggml-cuda/comm.cuh diff --git a/NOTES-allreduce.md b/NOTES-allreduce.md new file mode 100644 index 00000000000..49734567f76 --- /dev/null +++ b/NOTES-allreduce.md @@ -0,0 +1,311 @@ +# AllReduce Provider Abstraction — Working Notes + +## Context + +Tensor-parallel mode (`LLAMA_SPLIT_MODE_TENSOR = 3`) splits attention and FFN weight +matrices across N GPUs. Each GPU computes a partial result; an AllReduce sums them +before the next layer begins. + +## Where the Reduction Happens + +``` +src/llama-context.cpp — validates SPLIT_MODE_TENSOR requirements (FlashAttn required, no KV quant) +src/llama-model.cpp — llama_meta_device_get_split_state(): assigns split axis per tensor + attn_q/k/v, ffn_up/gate → PARTIAL (needs AllReduce) + output → MIRRORED (no AllReduce) +ggml/src/ggml-backend-meta.cpp — ggml_backend_meta_graph_compute(): drives the subgraph loop + after each PARTIAL subgraph: calls comm_allreduce(), or + falls back to allreduce_fallback() (CPU-based) +ggml/src/ggml-cuda/ggml-cuda.cu — the CUDA-side implementations (NCCL + future internal) +``` + +### Subgraph Execution Loop (ggml-backend-meta.cpp ~line 2023) + +``` +for each subgraph i: + compute subgraph on each GPU in parallel + if i < last_subgraph: + if comm_ctx set: + try comm_allreduce(comm_ctx, last_nodes_per_gpu[]) + if allreduce failed (or no comm_ctx): + allreduce_fallback(i) ← copies to CPU, reduces, copies back +``` + +## Data Structures + +| Struct | File | Purpose | +|--------|------|---------| +| `ggml_backend_cuda_comm_context` | `ggml-cuda.cu` | holds provider enum + NCCL comms (or future internal state) | +| `ggml_backend_meta_context` | `ggml-backend-meta.cpp` | holds `comm_ctx` (opaque) + `comm_allreduce` fn ptr | +| `ggml_cuda_device_info` | `common.cuh` | per-device CC, VRAM, default split ratios | + +## Provider Abstraction Added + +### New file: `ggml/src/ggml-cuda/comm.cuh` + +Defines `enum ggml_cuda_allreduce_provider`: +- `GGML_CUDA_ALLREDUCE_NCCL` — NCCL/RCCL (default when compiled in) +- `GGML_CUDA_ALLREDUCE_INTERNAL` — internal host/CUDA staged reduction (stub for now) + +### Changes to `ggml/src/ggml-cuda/ggml-cuda.cu` + +- `ggml_backend_cuda_comm_context` now always exists; holds `provider` + conditionally `comms`. +- `ggml_cuda_select_allreduce_provider()` — new selection function (see below). +- `ggml_backend_cuda_comm_init()` — constructs context, selects provider, inits NCCL comms or internal state. +- `ggml_backend_cuda_comm_allreduce_tensor()` — dispatches to `_nccl` or `_internal` helper. +- `ggml_backend_cuda_comm_allreduce_nccl()` — extracted from old monolithic function; logic unchanged. +- `ggml_backend_cuda_comm_allreduce_internal()` — stub returning `false` (triggers meta fallback). + +The public interface (`ggml_backend_comm_init` / `_free` / `_allreduce_tensor` proc addresses) is unchanged. + +## Provider Selection Logic (`ggml_cuda_select_allreduce_provider`) + +Priority order: +1. `GGML_CUDA_ALLREDUCE=nccl` env var — force NCCL (warn if not compiled in). +2. `GGML_CUDA_ALLREDUCE=internal` env var — force internal. +3. NCCL when `GGML_USE_NCCL` is defined at compile time. +4. INTERNAL otherwise (with a warning on NVIDIA non-HIP/MUSA builds). + +Future: inspect hardware topology before choosing the default: + +```cpp +// Check if all device pairs have direct NVLink: +int native_atomic; +cudaDeviceGetP2PAttribute(&native_atomic, + cudaDevP2PAttrNativeAtomicSupported, dev_i, dev_j); +// If any pair lacks NVLink, internal may win for small tensors on PCIe. +``` + +## Files Changed + +``` +ggml/src/ggml-cuda/comm.cuh NEW — provider enum +ggml/src/ggml-cuda/ggml-cuda.cu MOD — provider selection, dispatch, NCCL helper extracted, internal stub +``` + +## Files NOT Changed (intentionally) + +- `ggml/src/ggml-backend-meta.cpp` — no changes needed; uses opaque `comm_ctx` + fn ptr already. +- `ggml/include/ggml-backend.h` — public `comm_*` typedef signatures unchanged. +- `include/llama.h` — `llama_split_mode` enum unchanged. + +--- + +## Prototype Analysis: `nccl_injector_prototype/` + +### What the prototype is + +A Windows DLL injected via Microsoft Detours that intercepts NCCL calls and reroutes +AllReduce to a faster internal kernel for the 2-GPU float32 case. We are NOT using the +injection/Detours machinery — we're implementing directly inside llama.cpp. + +### Single-Phase Kernel (what we're using) + +The prototype has two strategies. We only want the **single-phase merged kernel** +(`allreduce_f32_kernel` in `src/kernels.cu`). It merges D2H copy + cross-GPU +synchronization + reduction into one kernel launch per GPU. + +**Execution: 1 block × 256 threads per GPU.** + +``` +Phase A (all 256 threads): vectorized D2H copy, sendbuf → host_mine + - float4 loads (16 bytes/thread/iteration) for the bulk + - scalar tail for remainder if count % 4 != 0 + __threadfence_system() + __syncthreads() ← make D2H visible system-wide + +Phase B (thread 0 only): signal + spin + signal_publish(arrival_mine, 1) ← volatile write + __threadfence_system() + while signal_observe(arrival_other) == 0: ← volatile read, __nanosleep(100) between polls + (optional: log spin count to debug buf every 4096 iters) + __syncthreads() ← broadcast "both D2H done" to all threads + __threadfence_system() ← acquire peer's host_other writes + +Phase C (all 256 threads): reduce + recvbuf[i] = sendbuf[i] + host_other[i] ← float4 vectorized +``` + +**Why it's fast:** the D2H copy and the cross-GPU spin overlap naturally — GPU-0 starts +spinning while GPU-1's 256 threads are still copying their data. No extra kernel launches +or host round-trips. + +### Signal Mechanism + +Three options exist via `SIGNAL_MECHANISM` macro; default (and recommended) is 1: + +```cuda +// Publish: volatile write + system fence +*(volatile int*)p = value; +__threadfence_system(); + +// Observe: volatile read (no fence needed — __threadfence_system() after syncthreads covers it) +return *(const volatile int*)p; +``` + +One int per GPU. Values: 0 = not arrived, 1 = arrived. Reset to 0 before each call. +Single writer per slot (owning GPU), single reader (peer GPU) — no atomics needed. + +### Host-Side Setup (what to port, minus the NCCL hooks) + +**Per-pipeline state to allocate at `comm_init` time:** + +``` +host_buf[N] float* cudaMallocHost, one per GPU, >= max_tensor_bytes +arrival[POOL×N] int* cudaMallocHost, ring buffer, one int slot per GPU per in-flight call +stream[N] cudaStream_t cudaStreamCreateWithFlags(cudaStreamNonBlocking) +ev_pool[N][POOL] cudaEvent_t cudaEventCreateWithFlags(cudaEventDisableTiming) + × 2 events per slot (app = "wait for upstream work", ker = "kernel done") +debug[N×4] int* cudaMallocHost, optional, 4 ints per GPU for spin diagnostics +``` + +**Pool size:** 128 slots in the prototype. Events + arrival slots wrap together; must +sync on `ev_pool[r][slot].ker` before reusing arrival slot (slot ownership check). + +**Kernel dispatch sequence:** + +```cpp +// For each GPU r in parallel: +cudaEventRecord(ev[r].app, upstream_stream[r]); // capture upstream work +cudaStreamWaitEvent(internal_stream[r], ev[r].app); // internal stream waits for it +launch_allreduce_kernel(..., internal_stream[r]); // launch merged kernel +cudaEventRecord(ev[r].ker, internal_stream[r]); // record kernel completion +cudaStreamWaitEvent(upstream_stream[r], ev[r].ker); // upstream waits for kernel +``` + +This inserts the allreduce into the existing CUDA streams without blocking the host. + +**Warmup:** 64 iterations with 32 KB payloads at `comm_init` time. Amortizes +driver overhead and encourages GPU clock boost before real inference begins. + +**Watchdog (optional):** poll arrival + debug values from host every ~20 ms to detect +deadlocks without killing the process. + +### What to Discard (Detours/Injection Overhead) + +| File | Reason to skip | +|------|---------------| +| `src/dllmain.cpp` | DLL entry point, Detours attach/detach | +| `src/launcher.cpp` | Standalone DLL injector executable | +| `src/nccl_types.h` | NCCL function pointer typedefs (not needed when calling directly) | +| `src/hooks.cpp` (partially) | NCCL function wrapping, PendingOp queue, GroupStart/End logic | +| `src/hooks.h` | Hook declarations | + +**Keep from `hooks.cpp`:** +- `AllReducePipeline` struct (minus NCCL-specific fields) +- `init_ar_pipeline()` logic +- `execute_all_reduce_kernel()` dispatch logic (adapted for our stream model) + +**Keep from `kernels.cu`:** +- `allreduce_f32_kernel` exactly as-is (can rename) +- `signal_publish` / `signal_observe` device functions +- `launch_allreduce_f32` wrapper (adapt to our context) + +**Discard from `kernels.cu`:** +- `allreduce_d2h_f32_kernel` — phase 1 of two-phase approach +- `allreduce_reduce_f32_kernel` — phase 2 of two-phase approach + +### Current Limitations & Extension Plan + +**Current prototype only handles:** +- Exactly 2 GPUs +- `float32` data type +- Tensors ≤ 256 KB (64K floats, `AR_KERNEL_THRESHOLD`) +- Sum reduction only + +--- + +## Extension Plan for the Internal Implementation + +### Data Types Beyond float32 + +The prototype's kernel is float32 only. In llama.cpp the allreduce tensors are always +FP32 (the NCCL path already converts larger tensors to BF16 before sending and back +after). We should follow the same pattern: + +**Strategy A — FP32 kernel only (simplest, sufficient for most cases):** +- Tensors ≤ threshold: run internal kernel as FP32 directly (matches prototype) +- Tensors > threshold: convert F32→BF16 on GPU, run BF16 kernel, convert back + - Halves PCIe/pinned-host bandwidth for large tensors + - BF16 kernel is identical structure but with `__nv_bfloat16` / `__nv_bfloat162` + +**Strategy B — templated kernel:** + +```cuda +template +__global__ void allreduce_kernel( + const T* sendbuf, T* recvbuf, + AccT* host_mine, const AccT* host_other, + int count, int* arrival_mine, int* arrival_other, ...) +{ + // D2H: convert T → AccT on the fly (if T != AccT), store AccT to host_mine + // Reduce: read AccT from sendbuf (via on-the-fly upcast) + host_other, write T to recvbuf +} +``` + +Instantiate for: +- `` — FP32 direct (fast, bulk of tensors) +- `` — BF16 tensors, accumulate as FP32 in host_mine +- `` — FP16 tensors, accumulate as FP32 + +The host_mine staging buffer always stores the accumulation type (float), so size is +always `count * sizeof(float)` regardless of tensor type. Simpler than varying buffer types. + +### Tensor Size Beyond 256 KB + +The prototype bails to CPU sync for large tensors. Options: + +**Option 1 — Multi-block kernel (recommended):** +Launch `ceil(count / BLOCK_ELEMENTS)` blocks instead of 1. Each block handles its own +arrival signaling independently (need one arrival int pair per block, or use a shared +atomic). This allows pipelining — later blocks can start D2H while earlier blocks +have already signaled. + +**Option 2 — Chunked sequential:** +Call the single-block kernel in a loop, each call covering `CHUNK_SIZE` elements. +Simple but adds kernel launch overhead. + +**Option 3 — Keep threshold, fall back to NCCL/CPU for large:** +The NCCL path already handles large tensors well (BF16 compressed). Use internal +only for tensors under a tuned threshold where it beats NCCL. This is probably the +right first step — just match or beat NCCL in the size range where NCCL has latency +overhead. + +### More Than 2 GPUs + +The prototype is hardcoded 2-GPU. The single-phase approach generalizes to N GPUs: + +**For N=3 or N=4 (small N), tree or ring approach:** + +**Ring AllReduce (reduce-scatter + all-gather):** +1. Reduce-scatter: each GPU sends to next, keeps accumulated result for its chunk +2. All-gather: each GPU sends its final chunk to all others + +For N=2 the ring degenerates to the simple pairwise protocol already in the prototype. +The arrival mechanism needs one slot per `(gpu, neighbor)` pair. + +**Alternative for small N: star topology** (one GPU is root): +1. All non-root GPUs send to root's host_buf in parallel +2. Root reduces all contributions +3. Root broadcasts to all non-root + +Simpler to implement than ring but root becomes bottleneck for N > 2. + +For the initial implementation: focus on N=2 (covers the most common dual-GPU case), +then extend to N=4 for 4×GPU servers. + +### Size Threshold Tuning + +The prototype uses 256 KB for PCIe 4.0 x16. Our threshold should be determined by +benchmarking; likely different on PCIe 5.0 and definitely different on NVLink. +Expose as `GGML_CUDA_ALLREDUCE_INTERNAL_THRESHOLD` env var (elements, default 65536) +so users can tune without recompiling. + +--- + +## Open Questions + +1. What are the actual tensor shapes/sizes in the allreduce calls during inference? + Need a trace to know what the P50/P95 sizes are. +2. Target GPU topology? NVLink or PCIe? Determines whether internal can beat NCCL. +3. Is BF16 staging acceptable precision-wise, or is FP32 end-to-end required? +4. How many GPUs max? Design differs significantly between N=2 and N≥8. +5. Should we support the watchdog/spin limit for hang detection in production? diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu new file mode 100644 index 00000000000..51bcfacf64e --- /dev/null +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -0,0 +1,388 @@ +#include "allreduce.cuh" +#include "ggml-impl.h" + +#include + +// --------------------------------------------------------------------------- +// Cross-GPU signal mechanism +// +// One int per (slot, rank) pair in pinned host memory: 0 = not arrived, +// 1 = arrived. There is exactly one writer (the owning GPU) and one reader +// (the peer), so we don't need atomics. A volatile store paired with +// __threadfence_system() provides the release ordering that makes the D2H +// writes visible system-wide before the arrival flag is observed. +// +// atomicAdd_system() (mechanism 0 in the prototype) is broken on RTX 5090 +// (hostNativeAtomicSupported = 0), so we use the volatile path throughout. +// --------------------------------------------------------------------------- + +static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { + *(volatile int *)p = 1; + __threadfence_system(); +} + +static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { + return *(const volatile int *)p; +} + +// --------------------------------------------------------------------------- +// Single-phase AllReduce kernel — float32, 2 GPUs +// +// Both GPUs run this kernel simultaneously in independent streams. Each GPU: +// +// Phase 1 (all threads): copy sendbuf → host_mine via float4 loads. +// __threadfence_system() commits writes to host. +// Phase 2 (thread 0): set arrival_mine = 1; spin on arrival_other == 1. +// Phase 3 (all threads): reduce: recvbuf[i] = sendbuf[i] + host_other[i]. +// +// The single-block configuration means __syncthreads() is sufficient for +// intra-block coordination and we can use the cheaper non-cooperative launch. +// 256 threads gives good occupancy while keeping register pressure low. +// --------------------------------------------------------------------------- +static __global__ void ggml_cuda_ar_f32_kernel( + const float * __restrict__ sendbuf, + float * __restrict__ recvbuf, + float * __restrict__ host_mine, + const float * __restrict__ host_other, + int count, + int * arrival_mine, + int * arrival_other) { + + const int tid = threadIdx.x; + const int nt = blockDim.x; + const int count4 = count >> 2; + const int tail = count4 << 2; + + // Phase 1: vectorised D2H copy using float4 (16 bytes per load/store). + { + const float4 * s4 = reinterpret_cast(sendbuf); + float4 * d4 = reinterpret_cast(host_mine); + for (int i = tid; i < count4; i += nt) { + d4[i] = s4[i]; + } + // Scalar tail if count is not a multiple of 4. + if (tid < count - tail) { + host_mine[tail + tid] = sendbuf[tail + tid]; + } + } + + // Commit all host writes before signalling; __syncthreads() ensures + // every thread's stores are in flight before thread 0 writes the flag. + __threadfence_system(); + __syncthreads(); + + // Phase 2: thread 0 signals this GPU's arrival, then spins until the + // peer signals back. The spin uses __nanosleep to yield the SM to + // other work rather than burning cycles in a hot loop. + if (tid == 0) { + ggml_cuda_ar_signal_set(arrival_mine); + while (ggml_cuda_ar_signal_get(arrival_other) == 0) { + __nanosleep(100); + } + } + + // Broadcast "peer has arrived" and acquire the peer's host_other writes. + __syncthreads(); + __threadfence_system(); + + // Phase 3: reduce — each thread handles its slice of the output. + { + const float4 * s4 = reinterpret_cast(sendbuf); + const float4 * o4 = reinterpret_cast(host_other); + float4 * r4 = reinterpret_cast(recvbuf); + for (int i = tid; i < count4; i += nt) { + float4 a = s4[i]; + float4 b = o4[i]; + r4[i] = make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); + } + if (tid < count - tail) { + recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; + } + } +} + +// --------------------------------------------------------------------------- +// Pipeline structure +// --------------------------------------------------------------------------- + +// Number of slots in the event / arrival ring. 128 is well above the actual +// in-flight depth (single digits in practice) while keeping init cost low. +static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; + +// Byte spacing between adjacent arrival ints. Two cache lines (128 bytes) +// ensures the arrival slots for the two GPUs never share a cache line, +// preventing false-sharing stalls on the polling GPU. +static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; + +struct ggml_cuda_ar_event_slot { + cudaEvent_t app = nullptr; // upstream computation complete + cudaEvent_t ker = nullptr; // AllReduce kernel complete +}; + +struct ggml_cuda_ar_pipeline { + int n_devices; + int devices[GGML_CUDA_MAX_DEVICES]; + size_t buf_bytes; // bytes per device in host_buf[] + uint64_t call_count; + + // Per-device resources. + float * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging + cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking kernel streams + ggml_cuda_ar_event_slot * ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] + + // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. + // Index helper: use ggml_cuda_ar_arrival_ptr(). + char * arrival; +}; + +// Return a pointer to the arrival int for (slot, rank). +static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, int rank) { + const size_t offset = ((size_t)slot * p->n_devices + rank) * GGML_CUDA_AR_ARRIVAL_STRIDE; + return reinterpret_cast(p->arrival + offset); +} + +// --------------------------------------------------------------------------- +// Init / free +// --------------------------------------------------------------------------- + +ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( + const int * devices, int n_devices, size_t max_bytes) { + GGML_ASSERT(n_devices >= 2 && n_devices <= GGML_CUDA_MAX_DEVICES); + + auto * p = new ggml_cuda_ar_pipeline{}; + p->n_devices = n_devices; + p->buf_bytes = 0; + p->call_count = 0; + p->arrival = nullptr; + for (int i = 0; i < n_devices; ++i) { + p->devices[i] = devices[i]; + p->host_buf[i] = nullptr; + p->streams[i] = nullptr; + p->ev_pool[i] = nullptr; + } + + // Per-device streams and event pools. + for (int i = 0; i < n_devices; ++i) { + ggml_cuda_set_device(p->devices[i]); + + if (cudaStreamCreateWithFlags(&p->streams[i], cudaStreamNonBlocking) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaStreamCreateWithFlags failed for device %d\n", + __func__, p->devices[i]); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + + p->ev_pool[i] = new ggml_cuda_ar_event_slot[GGML_CUDA_AR_POOL_SIZE](); + for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { + const bool ok = + cudaEventCreateWithFlags(&p->ev_pool[i][s].app, cudaEventDisableTiming) == cudaSuccess && + cudaEventCreateWithFlags(&p->ev_pool[i][s].ker, cudaEventDisableTiming) == cudaSuccess; + if (!ok) { + GGML_LOG_ERROR("%s: cudaEventCreate failed for device %d slot %d\n", + __func__, p->devices[i], s); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + } + } + + // Arrival ring: cache-line padded so each GPU's int is on its own line. + const size_t arrival_bytes = + (size_t)GGML_CUDA_AR_POOL_SIZE * n_devices * GGML_CUDA_AR_ARRIVAL_STRIDE; + if (cudaHostAlloc(reinterpret_cast(&p->arrival), arrival_bytes, + cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for arrival ring failed (%zu bytes)\n", + __func__, arrival_bytes); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + memset(p->arrival, 0, arrival_bytes); + + // Per-device pinned staging buffers. + p->buf_bytes = max_bytes; + for (int i = 0; i < n_devices; ++i) { + if (cudaHostAlloc(reinterpret_cast(&p->host_buf[i]), max_bytes, + cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for staging failed (%zu bytes)\n", + __func__, max_bytes); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + memset(p->host_buf[i], 0, max_bytes); + } + + // Warmup: run the kernel N times at the expected tensor size to pay the + // first-use driver / PCIe / page-mapping cost during model load rather + // than during the first inference step, and to encourage the GPU clock + // governor to boost before timing begins. + // Currently limited to the 2-GPU case. + if (n_devices == 2) { + constexpr int WARMUP_ITERS = 64; + constexpr size_t WARMUP_COUNT = 8192; // 32 KB of fp32 + constexpr size_t WARMUP_BYTES = WARMUP_COUNT * sizeof(float); + + float * dev_buf[2] = {}; + bool warmup_ok = true; + for (int i = 0; i < 2; ++i) { + ggml_cuda_set_device(p->devices[i]); + if (cudaMalloc(reinterpret_cast(&dev_buf[i]), WARMUP_BYTES) != cudaSuccess) { + GGML_LOG_WARN("%s: warmup alloc failed for device %d, skipping\n", + __func__, p->devices[i]); + warmup_ok = false; + break; + } + } + + if (warmup_ok) { + // Reuse slot 0 for every iteration, resetting arrival before each. + for (int iter = 0; iter < WARMUP_ITERS; ++iter) { + for (int r = 0; r < 2; ++r) { + *ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r) = 0; + } + for (int r = 0; r < 2; ++r) { + ggml_cuda_set_device(p->devices[r]); + ggml_cuda_ar_f32_kernel<<streams[r]>>>( + dev_buf[r], dev_buf[r], + p->host_buf[r], + p->host_buf[1 - r], + static_cast(WARMUP_COUNT), + ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r), + ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, 1 - r)); + } + } + for (int i = 0; i < 2; ++i) { + ggml_cuda_set_device(p->devices[i]); + cudaStreamSynchronize(p->streams[i]); + } + GGML_LOG_DEBUG("%s: warmup complete (%d iters x %zu KB)\n", + __func__, WARMUP_ITERS, WARMUP_BYTES >> 10); + } + + for (int i = 0; i < 2; ++i) { + if (dev_buf[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaFree(dev_buf[i]); + } + } + } + + GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " + "%zu KB staging per GPU\n", + __func__, n_devices, max_bytes >> 10); + return p; +} + +void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { + if (!p) { + return; + } + for (int i = 0; i < p->n_devices; ++i) { + if (p->host_buf[i]) { + cudaFreeHost(p->host_buf[i]); + } + if (p->ev_pool[i]) { + ggml_cuda_set_device(p->devices[i]); + for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { + if (p->ev_pool[i][s].app) { cudaEventDestroy(p->ev_pool[i][s].app); } + if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } + } + delete[] p->ev_pool[i]; + } + if (p->streams[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaStreamDestroy(p->streams[i]); + } + } + if (p->arrival) { + cudaFreeHost(p->arrival); + } + delete p; +} + +// --------------------------------------------------------------------------- +// Dispatch +// --------------------------------------------------------------------------- + +bool ggml_cuda_ar_allreduce( + ggml_cuda_ar_pipeline * p, + ggml_backend_t * backends, + ggml_tensor ** tensors) { + GGML_ASSERT(p != nullptr); + + const int n = p->n_devices; + + // Only the 2-GPU path is implemented; fall back for larger communicators. + if (n != 2) { + return false; + } + + // Only FP32 tensors are handled by the kernel; other types need a + // separate implementation. + if (tensors[0]->type != GGML_TYPE_F32) { + return false; + } + + const int64_t ne = ggml_nelements(tensors[0]); + const size_t bytes = (size_t)ne * sizeof(float); + + if (ne == 0) { + return true; + } + + if (bytes > p->buf_bytes) { + // Staging buffers too small; the caller should fall back. + // TODO: reallocate or chunk for larger tensors. + return false; + } + + // Cycle through the event pool. On the second pass through the ring, + // synchronise on the slot's ker event before touching arrival ints — + // the event and arrival pools wrap in lock-step so this guarantees that + // the kernels which last used this slot have finished. + const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); + const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; + p->call_count++; + + if (pool_lapped) { + for (int i = 0; i < n; ++i) { + ggml_cuda_set_device(p->devices[i]); + CUDA_CHECK(cudaEventSynchronize(p->ev_pool[i][slot].ker)); + } + } + + // Reset the arrival ints for this slot before any kernel can read them. + for (int i = 0; i < n; ++i) { + *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; + } + + // Insert the kernel into each GPU's existing compute stream via events: + // record(app, compute_stream) — capture "upstream done" point + // wait(internal_stream, app) — internal stream defers until then + // launch kernel on internal_stream + // record(ker, internal_stream) — capture "kernel done" point + // wait(compute_stream, ker) — compute stream resumes after kernel + for (int i = 0; i < n; ++i) { + const int peer = 1 - i; // valid for n == 2 only + ggml_cuda_set_device(p->devices[i]); + auto * cuda_ctx = static_cast(backends[i]->context); + ggml_cuda_ar_event_slot & ev = p->ev_pool[i][slot]; + + CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); + + ggml_cuda_ar_f32_kernel<<streams[i]>>>( + static_cast(tensors[i]->data), + static_cast(tensors[i]->data), + p->host_buf[i], + p->host_buf[peer], + static_cast(ne), + ggml_cuda_ar_arrival_ptr(p, slot, i), + ggml_cuda_ar_arrival_ptr(p, slot, peer)); + CUDA_CHECK(cudaGetLastError()); + + CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); + } + + return true; +} diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh new file mode 100644 index 00000000000..f14ff4b6175 --- /dev/null +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -0,0 +1,36 @@ +#pragma once + +#include "common.cuh" +#include "ggml-backend-impl.h" + +#include + +// Maximum tensor size (bytes per GPU) handled by the internal kernel path. +// Tensors larger than this are not yet supported and ggml_cuda_ar_allreduce() +// returns false, allowing the caller to fall back to another provider. +static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB + +// Opaque pipeline context — owns all pinned buffers, streams, and events. +struct ggml_cuda_ar_pipeline; + +// Allocate and warm up a pipeline for n_devices GPUs. +// devices[] holds the CUDA device IDs in rank order. +// max_bytes is the staging buffer size per device; must be at least as large +// as the largest tensor that will be reduced. +// Returns nullptr on allocation failure. +ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( + const int * devices, int n_devices, size_t max_bytes); + +// Release all resources owned by the pipeline. +void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); + +// Execute an in-place AllReduce (sum) across tensors[0..n_devices-1]. +// tensors[i] must live on the device managed by backends[i] and be +// contiguous FP32. +// Returns true on success. Returns false when the tensor type or size is +// outside the currently supported range; the caller should fall back to +// another provider (NCCL or the meta-backend CPU reduce). +bool ggml_cuda_ar_allreduce( + ggml_cuda_ar_pipeline * pipeline, + ggml_backend_t * backends, + ggml_tensor ** tensors); diff --git a/ggml/src/ggml-cuda/comm.cuh b/ggml/src/ggml-cuda/comm.cuh new file mode 100644 index 00000000000..ee738d4555c --- /dev/null +++ b/ggml/src/ggml-cuda/comm.cuh @@ -0,0 +1,22 @@ +#pragma once + +// AllReduce provider for multi-GPU tensor parallelism. +// +// The meta backend splits each transformer layer's PARTIAL-axis subgraph across +// N GPUs and requires an AllReduce after each segment to sum the partial results. +// This enum selects which implementation performs that reduction. +// +// The active provider is chosen once at communicator init time by +// ggml_cuda_select_allreduce_provider() and stored in +// ggml_backend_cuda_comm_context::provider. +enum ggml_cuda_allreduce_provider { + // NVIDIA/AMD Collective Communications Library (NCCL/RCCL). + // Optimal on NVLink/NVSwitch topologies; auto-selects the best transport. + // Requires GGML_USE_NCCL at compile time. + GGML_CUDA_ALLREDUCE_NCCL = 0, + + // Internal host/CUDA staged reduction built into llama.cpp. + // Works on any interconnect (PCIe, NVLink) without an external library. + // Can outperform NCCL on PCIe-only systems for latency-sensitive tensor sizes. + GGML_CUDA_ALLREDUCE_INTERNAL = 1, +}; diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 185956317e0..d18ead6cd07 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1,5412 +1,5484 @@ -#include "ggml-cuda.h" -#include "ggml-impl.h" -#include "ggml-backend-impl.h" - -#include "ggml-cuda/common.cuh" -#include "ggml-cuda/acc.cuh" -#include "ggml-cuda/add-id.cuh" -#include "ggml-cuda/arange.cuh" -#include "ggml-cuda/argmax.cuh" -#include "ggml-cuda/argsort.cuh" -#include "ggml-cuda/binbcast.cuh" -#include "ggml-cuda/clamp.cuh" -#include "ggml-cuda/concat.cuh" -#include "ggml-cuda/conv-transpose-1d.cuh" -#include "ggml-cuda/conv2d.cuh" -#include "ggml-cuda/conv2d-dw.cuh" -#include "ggml-cuda/conv2d-transpose.cuh" -#include "ggml-cuda/convert.cuh" -#include "ggml-cuda/count-equal.cuh" -#include "ggml-cuda/cpy.cuh" -#include "ggml-cuda/cross-entropy-loss.cuh" -#include "ggml-cuda/cumsum.cuh" -#include "ggml-cuda/diagmask.cuh" -#include "ggml-cuda/diag.cuh" -#include "ggml-cuda/fattn.cuh" -#include "ggml-cuda/getrows.cuh" -#include "ggml-cuda/im2col.cuh" -#include "ggml-cuda/mmf.cuh" -#include "ggml-cuda/mmq.cuh" -#include "ggml-cuda/mmvf.cuh" -#include "ggml-cuda/mmvq.cuh" -#include "ggml-cuda/norm.cuh" -#include "ggml-cuda/opt-step-adamw.cuh" -#include "ggml-cuda/opt-step-sgd.cuh" -#include "ggml-cuda/out-prod.cuh" -#include "ggml-cuda/pad.cuh" -#include "ggml-cuda/pool2d.cuh" -#include "ggml-cuda/quantize.cuh" -#include "ggml-cuda/rope.cuh" -#include "ggml-cuda/roll.cuh" -#include "ggml-cuda/scale.cuh" -#include "ggml-cuda/softcap.cuh" -#include "ggml-cuda/softmax.cuh" -#include "ggml-cuda/ssm-conv.cuh" -#include "ggml-cuda/ssm-scan.cuh" -#include "ggml-cuda/sum.cuh" -#include "ggml-cuda/sumrows.cuh" -#include "ggml-cuda/top-k.cuh" -#include "ggml-cuda/mean.cuh" -#include "ggml-cuda/tsembd.cuh" -#include "ggml-cuda/topk-moe.cuh" -#include "ggml-cuda/unary.cuh" -#include "ggml-cuda/upscale.cuh" -#include "ggml-cuda/wkv.cuh" -#include "ggml-cuda/gla.cuh" -#include "ggml-cuda/gated_delta_net.cuh" -#include "ggml-cuda/set.cuh" -#include "ggml-cuda/set-rows.cuh" -#include "ggml-cuda/pad_reflect_1d.cuh" -#include "ggml-cuda/solve_tri.cuh" -#include "ggml-cuda/tri.cuh" -#include "ggml-cuda/cumsum.cuh" -#include "ggml-cuda/fill.cuh" -#include "ggml.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size"); - -[[noreturn]] -void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) { - int id = -1; // in case cudaGetDevice fails - (void)cudaGetDevice(&id); - - GGML_LOG_ERROR(GGML_CUDA_NAME " error: %s\n", msg); - GGML_LOG_ERROR(" current device: %d, in function %s at %s:%d\n", id, func, file, line); - GGML_LOG_ERROR(" %s\n", stmt); - // abort with GGML_ABORT to get a stack trace - GGML_ABORT(GGML_CUDA_NAME " error"); -} - -// this is faster on Windows -// probably because the Windows CUDA libraries forget to make this check before invoking the drivers -void ggml_cuda_set_device(int device) { - int current_device; - CUDA_CHECK(cudaGetDevice(¤t_device)); - - if (device == current_device) { - return; - } - - CUDA_CHECK(cudaSetDevice(device)); -} - -int ggml_cuda_get_device() { - int id; - CUDA_CHECK(cudaGetDevice(&id)); - return id; -} - -static cudaError_t ggml_cuda_device_malloc(void ** ptr, size_t size, int device) { - ggml_cuda_set_device(device); - cudaError_t err; - if (getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr) { - err = cudaMallocManaged(ptr, size); -#if defined(GGML_USE_HIP) - if (err == hipSuccess) { - // hipMemAdviseSetCoarseGrain is an optional performance hint; - // ignore errors (e.g. hipErrorInvalidValue on some APU/iGPU configs). - (void)cudaMemAdvise(*ptr, size, hipMemAdviseSetCoarseGrain, device); - (void)hipGetLastError(); // clear any error - } - - // fall back to cudaMalloc if not supported (e.g. on Windows) - if (err == hipErrorNotSupported) { - static bool warned_unsupported = false; - if (!warned_unsupported) { - GGML_LOG_WARN("hipMallocManaged unsupported, falling back to hipMalloc.\n"); - warned_unsupported = true; - } - - err = cudaMalloc(ptr, size); - } -#endif // defined(GGML_USE_HIP) - } else { - err = cudaMalloc(ptr, size); - } - return err; -} - -#if defined(GGML_USE_HIP) -static int ggml_cuda_parse_id(char devName[]) { - // A list of possible Target IDs can be found under the rocclr/clr repo in device.cpp - // these values are not stable so this is susceptible to breakage - // https://github.com/ROCm/clr/blob/amd-staging/rocclr/device/device.cpp - int archMajor = 0x0; - int archMinor = 0x0; - int archNum = GGML_CUDA_CC_OFFSET_AMD; - int archLen = strlen(devName); - char archName[archLen + 1]; - - // strip leading 'gfx' while copying into our buffer - if (archLen > 3) { - strcpy(archName, &devName[3]); - archLen -= 3; - } - - // trim trailing :xnack- or :sramecc- statuses - archLen = strcspn(archName, ":"); - archName[archLen] = '\0'; - - // tease out the version information - if (archLen > 8) { - // versions labeled generic use '-' as delimiter - // strip the trailing "-generic" then iterate through what remains - if ((strstr(archName, "-generic"))) { - archName[archLen - 8] = '\0'; - char * pch; - if ((pch = strtok(archName, "-"))) { - archMajor = (int)strtoul(pch, 0, 16); - if ((pch = strtok(NULL, "-"))) { - archMinor = 0x10 * (int)strtoul(pch, 0, 16); - } - } - } - } else if (archLen >= 3) { - // last two digits should be the minor * 0x10 + stepping - archMinor = (int)strtoul(&archName[archLen - 2], 0, 16); - archName[archLen - 2] = '\0'; - - // only the major version remains - archMajor = (int)strtoul(archName, 0, 16); - } - archNum += archMajor * 0x100; - archNum += archMinor; - return archNum; -} -#endif // defined(GGML_USE_HIP) - -static ggml_cuda_device_info ggml_cuda_init() { - ggml_cuda_device_info info = {}; - - cudaError_t err = cudaGetDeviceCount(&info.device_count); - if (err != cudaSuccess) { - GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err)); - return info; - } - - GGML_ASSERT(info.device_count <= GGML_CUDA_MAX_DEVICES); - - int64_t total_vram = 0; - for (int id = 0; id < info.device_count; ++id) { - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); - total_vram += prop.totalGlobalMem; - } - GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n", - __func__, info.device_count, (size_t)(total_vram / (1024 * 1024))); - total_vram = 0; - - std::vector> turing_devices_without_mma; - for (int id = 0; id < info.device_count; ++id) { - int device_vmm = 0; - -#if defined(GGML_USE_VMM) - CUdevice device; - CU_CHECK(cuDeviceGet(&device, id)); - CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device)); - - if (device_vmm) { - CUmemAllocationProp alloc_prop = {}; - alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; - alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - alloc_prop.location.id = id; - CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED)); - } -#endif // defined(GGML_USE_VMM) - info.devices[id].vmm = !!device_vmm; - - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); - - info.default_tensor_split[id] = total_vram; - total_vram += prop.totalGlobalMem; - info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034) - info.devices[id].nsm = prop.multiProcessorCount; - info.devices[id].smpb = prop.sharedMemPerBlock; - info.devices[id].warp_size = prop.warpSize; - -#ifndef GGML_USE_MUSA - int supports_coop_launch = 0; - CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, id)); - info.devices[id].supports_cooperative_launch = !!supports_coop_launch; -#else - info.devices[id].supports_cooperative_launch = false; -#endif // !(GGML_USE_MUSA) - -#if defined(GGML_USE_HIP) - info.devices[id].smpbo = prop.sharedMemPerBlock; - - info.devices[id].cc = ggml_cuda_parse_id(prop.gcnArchName); - if ((info.devices[id].cc & 0xff00) == 0x0) { - GGML_LOG_WARN("invalid architecture ID received for device %d %s: %s cc %d.%d\n", - id, prop.name, prop.gcnArchName, prop.major, prop.minor); - - // Fallback to prop.major and prop.minor - if (prop.major > 0) { - info.devices[id].cc = GGML_CUDA_CC_OFFSET_AMD + prop.major * 0x100; - info.devices[id].cc += prop.minor * 0x10; - } - } - GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n", - id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff, - device_vmm ? "yes" : "no", prop.warpSize, - (size_t)(prop.totalGlobalMem / (1024 * 1024))); -#elif defined(GGML_USE_MUSA) - // FIXME: Ensure compatibility with varying warp sizes across different MUSA archs. - info.devices[id].warp_size = 32; - info.devices[id].smpbo = prop.sharedMemPerBlockOptin; - info.devices[id].cc = GGML_CUDA_CC_OFFSET_MTHREADS + prop.major * 0x100; - info.devices[id].cc += prop.minor * 0x10; - GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", - id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", - (size_t)(prop.totalGlobalMem / (1024 * 1024))); -#else - info.devices[id].smpbo = prop.sharedMemPerBlockOptin; - info.devices[id].cc = 100*prop.major + 10*prop.minor; - GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", - id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", - (size_t)(prop.totalGlobalMem / (1024 * 1024))); - std::string device_name(prop.name); - if (device_name == "NVIDIA GeForce MX450") { - turing_devices_without_mma.push_back({ id, device_name }); - } else if (device_name == "NVIDIA GeForce MX550") { - turing_devices_without_mma.push_back({ id, device_name }); - } else if (device_name.substr(0, 21) == "NVIDIA GeForce GTX 16") { - turing_devices_without_mma.push_back({ id, device_name }); - } - - // Temporary performance fix: - // Setting device scheduling strategy for iGPUs with cc121 to "spinning" to avoid delays in cuda synchronize calls. - // TODO: Check for future drivers the default scheduling strategy and - // remove this call again when cudaDeviceScheduleSpin is default. - if (prop.major == 12 && prop.minor == 1) { - CUDA_CHECK(cudaSetDevice(id)); - CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin)); - } - -#endif // defined(GGML_USE_HIP) - } - - if (ggml_cuda_highest_compiled_arch(GGML_CUDA_CC_TURING) >= GGML_CUDA_CC_TURING && !turing_devices_without_mma.empty()) { - GGML_LOG_INFO("The following devices will have suboptimal performance due to a lack of tensor cores:\n"); - for (size_t device_pos = 0; device_pos < turing_devices_without_mma.size(); device_pos++) { - GGML_LOG_INFO( - " Device %d: %s\n", turing_devices_without_mma[device_pos].first, turing_devices_without_mma[device_pos].second.c_str()); - } - GGML_LOG_INFO( - "Consider compiling with CMAKE_CUDA_ARCHITECTURES=61-virtual;80-virtual and DGGML_CUDA_FORCE_MMQ to force the use of the Pascal code for Turing.\n"); - } - - for (int id = 0; id < info.device_count; ++id) { - info.default_tensor_split[id] /= total_vram; - } - - // configure logging to stdout - // CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr)); - - if (getenv("GGML_CUDA_P2P") != nullptr) { - for (int id = 0; id < info.device_count; ++id) { - ggml_cuda_set_device(id); - for (int id_other = 0; id_other < info.device_count; ++id_other) { - if (id == id_other) { - continue; - } - int can_access_peer; - CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id, id_other)); - if (can_access_peer) { - CUDA_CHECK(cudaDeviceEnablePeerAccess(id_other, 0)); - } - } - } - } - - return info; -} - -const ggml_cuda_device_info & ggml_cuda_info() { - static ggml_cuda_device_info info = ggml_cuda_init(); - return info; -} - -// #define DEBUG_CUDA_MALLOC - -// buffer pool for cuda (legacy) -struct ggml_cuda_pool_leg : public ggml_cuda_pool { - static const int MAX_BUFFERS = 256; - - int device; - struct ggml_cuda_buffer { - void * ptr = nullptr; - size_t size = 0; - }; - - ggml_cuda_buffer buffer_pool[MAX_BUFFERS] = {}; - size_t pool_size = 0; - - explicit ggml_cuda_pool_leg(int device) : - device(device) { - } - - ~ggml_cuda_pool_leg() { - clear_pool(); - GGML_ASSERT(pool_size == 0); - } - - void clear_pool() { - ggml_cuda_set_device(device); - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer & b = buffer_pool[i]; - if (b.ptr != nullptr) { - CUDA_CHECK(cudaFree(b.ptr)); - pool_size -= b.size; - b.ptr = nullptr; - b.size = 0; - } - } - } - - void * alloc(size_t size, size_t * actual_size) override { -#ifdef DEBUG_CUDA_MALLOC - int nnz = 0; - size_t max_size = 0; -#endif - size_t best_diff = 1ull << 36; - int ibest = -1; - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer& b = buffer_pool[i]; - if (b.ptr != nullptr) { -#ifdef DEBUG_CUDA_MALLOC - ++nnz; - if (b.size > max_size) max_size = b.size; -#endif - if (b.size >= size) { - size_t diff = b.size - size; - if (diff < best_diff) { - best_diff = diff; - ibest = i; - if (!best_diff) { - void * ptr = b.ptr; - *actual_size = b.size; - b.ptr = nullptr; - b.size = 0; - return ptr; - } - } - } - } - } - if (ibest >= 0) { - ggml_cuda_buffer& b = buffer_pool[ibest]; - void * ptr = b.ptr; - *actual_size = b.size; - b.ptr = nullptr; - b.size = 0; - return ptr; - } - void * ptr; - size_t look_ahead_size = (size_t) (1.05 * size); - look_ahead_size = 256 * ((look_ahead_size + 255)/256); - ggml_cuda_set_device(device); - cudaError_t err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); - if (err == cudaErrorMemoryAllocation) { - (void)cudaGetLastError(); - const size_t cached_bytes = pool_size; - GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: alloc of %.2f MiB failed, flushing %.2f MiB of cached buffers and retrying\n", - device, look_ahead_size/1024.0/1024.0, cached_bytes/1024.0/1024.0); - CUDA_CHECK(cudaDeviceSynchronize()); - clear_pool(); - err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); - if (err == cudaSuccess) { - GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: retry succeeded\n", device); - } - } - CUDA_CHECK(err); - *actual_size = look_ahead_size; - pool_size += look_ahead_size; -#ifdef DEBUG_CUDA_MALLOC - GGML_LOG_INFO("%s[%d]: %d buffers, max_size = %u MB, pool_size = %u MB, requested %u MB\n", __func__, device, nnz, - (uint32_t)(max_size / 1024 / 1024), (uint32_t)(pool_size / 1024 / 1024), (uint32_t)(size / 1024 / 1024)); -#endif - return ptr; - } - - void free(void * ptr, size_t size) override { - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer& b = buffer_pool[i]; - if (b.ptr == nullptr) { - b.ptr = ptr; - b.size = size; - return; - } - } - GGML_LOG_DEBUG(GGML_CUDA_NAME " buffer pool full, increase MAX_CUDA_BUFFERS\n"); - ggml_cuda_set_device(device); - CUDA_CHECK(cudaFree(ptr)); - pool_size -= size; - } -}; - -// pool with virtual memory -#if defined(GGML_USE_VMM) -struct ggml_cuda_pool_vmm : public ggml_cuda_pool { - static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB - - int device; - CUdeviceptr pool_addr = 0; - size_t pool_used = 0; - size_t pool_size = 0; - size_t granularity; -#if defined(GGML_USE_HIP) - std::vector> mappings; -#endif - - explicit ggml_cuda_pool_vmm(int device) : - device(device), - granularity(ggml_cuda_info().devices[device].vmm_granularity) { - } - - ~ggml_cuda_pool_vmm() { - if (pool_addr != 0) { -#if defined(GGML_USE_HIP) - // Workaround for https://github.com/ROCm/ROCR-Runtime/issues/285 - for (std::pair & mapping : mappings) { - CU_CHECK(cuMemUnmap(mapping.first, mapping.second)); - } -#else - CU_CHECK(cuMemUnmap(pool_addr, pool_size)); -#endif - CU_CHECK(cuMemAddressFree(pool_addr, CUDA_POOL_VMM_MAX_SIZE)); - } - } - - void * alloc(size_t size, size_t * actual_size) override { - // round up the allocation size to the alignment to ensure that all allocations are aligned for all data types - const size_t alignment = 128; - size = alignment * ((size + alignment - 1) / alignment); - - size_t avail = pool_size - pool_used; - - if (size > avail) { - // round up to the next multiple of the granularity - size_t reserve_size = size - avail; - reserve_size = granularity * ((reserve_size + granularity - 1) / granularity); - - GGML_ASSERT(pool_size + reserve_size <= CUDA_POOL_VMM_MAX_SIZE); - - // allocate more physical memory - CUmemAllocationProp prop = {}; - prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; - prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - prop.location.id = device; - CUmemGenericAllocationHandle handle; - CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0)); - - // reserve virtual address space (if not already reserved) - if (pool_addr == 0) { - CU_CHECK(cuMemAddressReserve(&pool_addr, CUDA_POOL_VMM_MAX_SIZE, 0, 0, 0)); - } - - // map at the end of the pool - CUdeviceptr start_ptr = (CUdeviceptr)((char *)(pool_addr) + pool_size); - CU_CHECK(cuMemMap(start_ptr, reserve_size, 0, handle, 0)); -#if defined(GGML_USE_HIP) - mappings.push_back({start_ptr, reserve_size}); -#endif - - // the memory allocation handle is no longer needed after mapping - CU_CHECK(cuMemRelease(handle)); - - // set access - CUmemAccessDesc access = {}; - access.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - access.location.id = device; - access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; - CU_CHECK(cuMemSetAccess((CUdeviceptr)((char *)(pool_addr) + pool_size), reserve_size, &access, 1)); - - // add to the pool - pool_size += reserve_size; - - //printf("cuda pool[%d]: size increased to %llu MB (reserved %llu MB)\n", - // device, (unsigned long long) (pool_size/1024/1024), - // (unsigned long long) (reserve_size/1024/1024)); - } - - GGML_ASSERT(pool_addr != 0); - - void * ptr = (void *) ((CUdeviceptr)((char *)(pool_addr) + pool_used)); - *actual_size = size; - pool_used += size; - -#ifdef DEBUG_CUDA_MALLOC - printf("cuda pool[%d]: allocated %llu bytes at %llx\n", device, (unsigned long long) size, ptr); -#endif - - return ptr; - } - - void free(void * ptr, size_t size) override { -#ifdef DEBUG_CUDA_MALLOC - printf("cuda pool[%d]: freed %llu bytes at %llx\n", device, (unsigned long long) size, ptr); -#endif - - pool_used -= size; - - // all deallocations must be in reverse order of the allocations - GGML_ASSERT(ptr == (void *) ((char *)(pool_addr) + pool_used)); - } -}; -#endif // defined(GGML_USE_VMM) - -std::unique_ptr ggml_backend_cuda_context::new_pool_for_device(int device, - [[maybe_unused]] int stream_no) { -#if defined(GGML_USE_VMM) - if (ggml_cuda_info().devices[device].vmm) { - return std::unique_ptr(new ggml_cuda_pool_vmm(device)); - } -#endif // defined(GGML_USE_VMM) - return std::unique_ptr(new ggml_cuda_pool_leg(device)); -} - -// destroying a cuBLAS handle while a graph is being captured in a different thread can result in a CUDA error -// this lock is used to ensure that no cuBLAS handle is destroyed while a graph is being captured - -static std::mutex ggml_cuda_lock; -static std::condition_variable ggml_cuda_lock_cv; -static std::atomic ggml_cuda_lock_counter; - -ggml_backend_cuda_context::~ggml_backend_cuda_context() { - std::unique_lock lock(ggml_cuda_lock); - ggml_cuda_lock_cv.wait(lock, []{ return ggml_cuda_lock_counter.load(std::memory_order_relaxed) == 0; }); - - if (copy_event != nullptr) { - CUDA_CHECK(cudaEventDestroy(copy_event)); - } - for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { - for (int j = 0; j < GGML_CUDA_MAX_STREAMS; ++j) { - if (streams[i][j] != nullptr) { - CUDA_CHECK(cudaStreamDestroy(streams[i][j])); - } - } - if (cublas_handles[i] != nullptr) { - CUBLAS_CHECK(cublasDestroy(cublas_handles[i])); - } - } -} - - -// cuda buffer - -struct ggml_backend_cuda_buffer_context { - int device; - void * dev_ptr = nullptr; - std::string name; - - ggml_backend_cuda_buffer_context(int device, void * dev_ptr) : - device(device), dev_ptr(dev_ptr), - name(GGML_CUDA_NAME + std::to_string(device)) { - } - - ~ggml_backend_cuda_buffer_context() { - CUDA_CHECK(cudaFree(dev_ptr)); - } -}; - -static void ggml_backend_cuda_buffer_free_buffer(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - delete ctx; -} - -static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) { - return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer; -} - -static void * ggml_backend_cuda_buffer_get_base(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - return ctx->dev_ptr; -} - -static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - if (tensor->view_src != NULL) { - assert(tensor->view_src->buffer->buft == buffer->buft); - return GGML_STATUS_SUCCESS; - } - - if (ggml_is_quantized(tensor->type) && tensor->view_src == nullptr && ggml_backend_buffer_get_usage(buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE) { - // initialize padding to 0 to avoid possible NaN values - const size_t original_size = ggml_nbytes(tensor); - const size_t padded_size = ggml_backend_buft_get_alloc_size(buffer->buft, tensor); - - if (padded_size > original_size) { - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size)); - } - } - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemsetAsync((char *) tensor->data + offset, value, size, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpy2DAsync( - (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpy2DAsync( - data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) { - if (ggml_backend_buffer_is_cuda(src->buffer)) { - ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context; - ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context; - if (src_ctx->device == dst_ctx->device) { - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread)); - } else { -#ifdef GGML_CUDA_NO_PEER_COPY - return false; -#else - CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_ctx->device, src->data, src_ctx->device, ggml_nbytes(src), cudaStreamPerThread)); -#endif - } - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - return true; - } - return false; - - GGML_UNUSED(buffer); -} - -static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemsetAsync(ctx->dev_ptr, value, buffer->size, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = { - /* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer, - /* .get_base = */ ggml_backend_cuda_buffer_get_base, - /* .init_tensor = */ ggml_backend_cuda_buffer_init_tensor, - /* .memset_tensor = */ ggml_backend_cuda_buffer_memset_tensor, - /* .set_tensor = */ ggml_backend_cuda_buffer_set_tensor, - /* .get_tensor = */ ggml_backend_cuda_buffer_get_tensor, - /* .set_tensor_2d = */ ggml_backend_cuda_buffer_set_tensor_2d, - /* .get_tensor_2d = */ ggml_backend_cuda_buffer_get_tensor_2d, - /* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor, - /* .clear = */ ggml_backend_cuda_buffer_clear, - /* .reset = */ NULL, -}; - -// cuda buffer type -struct ggml_backend_cuda_buffer_type_context { - int device; - std::string name; -}; - -static const char * ggml_backend_cuda_buffer_type_get_name(ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; - - return ctx->name.c_str(); -} - -static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name; -} - -static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; - - ggml_cuda_set_device(buft_ctx->device); - - void * dev_ptr; - cudaError_t err = ggml_cuda_device_malloc(&dev_ptr, size, buft_ctx->device); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - GGML_LOG_ERROR("%s: allocating %.2f MiB on device %d: cudaMalloc failed: %s\n", __func__, size / 1024.0 / 1024.0, buft_ctx->device, cudaGetErrorString(err)); - return nullptr; - } - - ggml_backend_cuda_buffer_context * ctx = new ggml_backend_cuda_buffer_context(buft_ctx->device, dev_ptr); - - return ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size); -} - -static size_t ggml_backend_cuda_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { - return 128; - - GGML_UNUSED(buft); -} - -static size_t ggml_backend_cuda_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { - size_t size = ggml_nbytes(tensor); - int64_t ne0 = tensor->ne[0]; - - if (ggml_is_quantized(tensor->type)) { - if (ne0 % MATRIX_ROW_PADDING != 0) { - GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor)); - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - } - - return size; - - GGML_UNUSED(buft); -} - -static const ggml_backend_buffer_type_i ggml_backend_cuda_buffer_type_interface = { - /* .get_name = */ ggml_backend_cuda_buffer_type_get_name, - /* .alloc_buffer = */ ggml_backend_cuda_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cuda_buffer_type_get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cuda_buffer_type_get_alloc_size, - /* .is_host = */ NULL, -}; - -ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int device) { - static std::mutex mutex; - std::lock_guard lock(mutex); - - if (device >= ggml_backend_cuda_get_device_count()) { - return nullptr; - } - - static ggml_backend_buffer_type ggml_backend_cuda_buffer_types[GGML_CUDA_MAX_DEVICES]; - - static bool ggml_backend_cuda_buffer_type_initialized = false; - - if (!ggml_backend_cuda_buffer_type_initialized) { - for (int i = 0; i < ggml_backend_cuda_get_device_count(); i++) { - ggml_backend_cuda_buffer_types[i] = { - /* .iface = */ ggml_backend_cuda_buffer_type_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), i), - /* .context = */ new ggml_backend_cuda_buffer_type_context{i, GGML_CUDA_NAME + std::to_string(i)}, - }; - } - ggml_backend_cuda_buffer_type_initialized = true; - } - - return &ggml_backend_cuda_buffer_types[device]; -} - -// cuda split buffer - -static int64_t get_row_rounding(const std::array & tensor_split) { - int64_t row_rounding = 0; - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { - continue; - } - - const int cc = ggml_cuda_info().devices[id].cc; - row_rounding = std::max(row_rounding, (int64_t)get_mmq_y_host(cc)); - } - return row_rounding; -} - -static void get_row_split(int64_t * row_low, int64_t * row_high, const ggml_tensor * tensor, const std::array & tensor_split, int id) { - const int64_t nrows = ggml_nrows(tensor); - const int64_t rounding = get_row_rounding(tensor_split); - - *row_low = id == 0 ? 0 : nrows*tensor_split[id]; - *row_low -= *row_low % rounding; - - if (id == ggml_backend_cuda_get_device_count() - 1) { - *row_high = nrows; - } else { - *row_high = nrows*tensor_split[id + 1]; - *row_high -= *row_high % rounding; - } -} - -static size_t ggml_nbytes_split(const struct ggml_tensor * tensor, int nrows_split) { - static_assert(GGML_MAX_DIMS == 4, "GGML_MAX_DIMS is not 4 - update this function"); - - return nrows_split*ggml_row_size(tensor->type, tensor->ne[0]); -} - -struct ggml_backend_cuda_split_buffer_type_context { - int main_device; - std::array tensor_split; - std::string name; -}; - -struct ggml_backend_cuda_split_buffer_context { - ~ggml_backend_cuda_split_buffer_context() { - for (ggml_tensor_extra_gpu * extra : tensor_extras) { - for (int id = 0; id < GGML_CUDA_MAX_DEVICES; ++id) { - for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { - if (extra->events[id][is] != nullptr) { - CUDA_CHECK(cudaEventDestroy(extra->events[id][is])); - } - } - if (extra->data_device[id] != nullptr) { - CUDA_CHECK(cudaFree(extra->data_device[id])); - } - } - delete extra; - } - } - - std::vector tensor_extras; -}; - - -static void ggml_backend_cuda_split_buffer_free_buffer(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; - delete ctx; -} - -static void * ggml_backend_cuda_split_buffer_get_base(ggml_backend_buffer_t buffer) { - // the pointers are stored in the tensor extras, this is just a dummy address and never dereferenced - return (void *)0x1000; - - GGML_UNUSED(buffer); -} - -static enum ggml_status ggml_backend_cuda_split_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { - GGML_ASSERT(tensor->view_src == nullptr); // views of split tensors are not supported - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - - ggml_tensor_extra_gpu * extra = new ggml_tensor_extra_gpu{}; - ctx->tensor_extras.push_back(extra); - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - // FIXME: do not crash if cudaMalloc fails - // currently, init_tensor cannot fail, it needs to be fixed in ggml-backend first - ggml_cuda_set_device(id); - char * buf; - CUDA_CHECK(ggml_cuda_device_malloc((void**)&buf, size, id)); - - // set padding to 0 to avoid possible NaN values - if (size > original_size) { - CUDA_CHECK(cudaMemset(buf + original_size, 0, size - original_size)); - } - - extra->data_device[id] = buf; - - for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { - CUDA_CHECK(cudaEventCreateWithFlags(&extra->events[id][is], cudaEventDisableTiming)); - } - } - tensor->extra = extra; - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_split_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - // split tensors must always be set in their entirety at once - GGML_ASSERT(offset == 0); - GGML_ASSERT(size == ggml_nbytes(tensor)); - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - const size_t nb1 = tensor->nb[1]; - ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - const size_t offset_split = row_low*nb1; - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - const char * buf_host = (const char *)data + offset_split; - CUDA_CHECK(cudaMemcpyAsync(extra->data_device[id], buf_host, original_size, cudaMemcpyHostToDevice, cudaStreamPerThread)); - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - } -} - -static void ggml_backend_cuda_split_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - // split tensors must always be set in their entirety at once - GGML_ASSERT(offset == 0); - GGML_ASSERT(size == ggml_nbytes(tensor)); - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - const size_t nb1 = tensor->nb[1]; - ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - const size_t offset_split = row_low*nb1; - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - char * buf_host = (char *)data + offset_split; - CUDA_CHECK(cudaMemcpyAsync(buf_host, extra->data_device[id], original_size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - } -} - -static void ggml_backend_cuda_split_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { - GGML_UNUSED(buffer); - GGML_UNUSED(value); -} - -static const ggml_backend_buffer_i ggml_backend_cuda_split_buffer_interface = { - /* .free_buffer = */ ggml_backend_cuda_split_buffer_free_buffer, - /* .get_base = */ ggml_backend_cuda_split_buffer_get_base, - /* .init_tensor = */ ggml_backend_cuda_split_buffer_init_tensor, - /* .memset_tensor = */ NULL, - /* .set_tensor = */ ggml_backend_cuda_split_buffer_set_tensor, - /* .get_tensor = */ ggml_backend_cuda_split_buffer_get_tensor, - /* .set_tensor_2d = */ NULL, - /* .get_tensor_2d = */ NULL, - /* .cpy_tensor = */ NULL, - /* .clear = */ ggml_backend_cuda_split_buffer_clear, - /* .reset = */ NULL, -}; - -// cuda split buffer type - -static const char * ggml_backend_cuda_split_buffer_type_get_name(ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; - - return ctx->name.c_str(); -} - -static bool ggml_backend_buft_is_cuda_split(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_split_buffer_type_get_name; -} - -static ggml_backend_buffer_t ggml_backend_cuda_split_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - // since we don't know the exact split after rounding, we cannot allocate the device buffers at this point - // instead, we allocate them for each tensor separately in init_tensor - // however, the size still represents the maximum cumulative size of all the device buffers after the tensors are allocated, - // as returned by get_alloc_size. this limit is enforced during tensor allocation by ggml-alloc, so it must be correct. - ggml_backend_cuda_split_buffer_context * ctx = new ggml_backend_cuda_split_buffer_context(); - - return ggml_backend_buffer_init(buft, ggml_backend_cuda_split_buffer_interface, ctx, size); -} - -static size_t ggml_backend_cuda_split_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { - return 128; - - GGML_UNUSED(buft); -} - -static size_t ggml_backend_cuda_split_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { - ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - size_t total_size = 0; - - const int64_t ne0 = tensor->ne[0]; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - total_size += ggml_nbytes_split(tensor, nrows_split); - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - total_size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - } - - return total_size; -} - -static bool ggml_backend_cuda_split_buffer_type_is_host(ggml_backend_buffer_type_t buft) { - return false; - - GGML_UNUSED(buft); -} - -static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_interface = { - /* .get_name = */ ggml_backend_cuda_split_buffer_type_get_name, - /* .alloc_buffer = */ ggml_backend_cuda_split_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cuda_split_buffer_type_get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cuda_split_buffer_type_get_alloc_size, - /* .is_host = */ ggml_backend_cuda_split_buffer_type_is_host, -}; - -#ifdef GGML_USE_NCCL -struct ggml_backend_cuda_comm_context { - std::vector backends; - std::vector comms; - - ~ggml_backend_cuda_comm_context() { - for (ncclComm_t comm : comms) { - NCCL_CHECK(ncclCommDestroy(comm)); - } - } -}; -#endif // GGML_USE_NCCL - -static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { -#ifdef GGML_USE_NCCL - if (comm_ctx_v == nullptr) { - return; - } - ggml_backend_cuda_comm_context * comm_ctx = (ggml_backend_cuda_comm_context *) comm_ctx_v; - delete comm_ctx; -#else - GGML_UNUSED(comm_ctx_v); -#endif // GGML_USE_NCCL -} - -static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { -#ifdef GGML_USE_NCCL - for (size_t i = 0; i < n_backends; i++) { - if (!ggml_backend_is_cuda(backends[i])) { - return nullptr; - } - } - ggml_backend_cuda_comm_context * ret = new ggml_backend_cuda_comm_context; - std::vector dev_ids; - ret->backends.reserve(n_backends); - dev_ids.reserve(n_backends); - for (size_t i = 0; i < n_backends; i++) { - ret->backends.push_back(backends[i]); - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backends[i]->context; - dev_ids.push_back(cuda_ctx->device); - } - - ret->comms.resize(n_backends); - NCCL_CHECK(ncclCommInitAll(ret->comms.data(), n_backends, dev_ids.data())); - return ret; -#else - // If NCCL is installed it is used by default for optimal performance. - // However, NVIDIA does not distribute NCCL with CUDA so users may be unwittingly missing this package. - // RCCL is disabled by default, users are explicitly opting in. - // Therefore print no warning for RCCL. -#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) - static bool warning_printed = false; - if (!warning_printed) { - GGML_LOG_WARN("%s: NVIDIA Collective Communications Library (NCCL) is unavailable, multi GPU performance will be suboptimal\n", __func__); - warning_printed = true; - } -#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) - GGML_UNUSED_VARS(backends, n_backends); - return nullptr; -#endif // GGML_USE_NCCL -} - -static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { -#ifdef GGML_USE_NCCL - const int64_t ne = ggml_nelements(tensors[0]); - // FIXME the input of llm_graph_context::build_in_out_ids can produce a tensor with 0 elements if n_outputs == 0 - // This then causes a crash in this function - if (ne == 0) { - return true; - } - - GGML_ASSERT(comm_ctx_v != nullptr); - ggml_backend_cuda_comm_context * comm_ctx = (ggml_backend_cuda_comm_context *) comm_ctx_v; - const size_t n_backends = comm_ctx->backends.size(); - - for (size_t i = 0; i < n_backends; ++i) { - GGML_ASSERT(tensors[i] != nullptr); - GGML_ASSERT(ggml_nelements(tensors[i]) == ne); - GGML_ASSERT(ggml_is_contiguously_allocated(tensors[i])); - } - - // For small tensors, simply reduce them as FP32. - // The following heuristic for how "small" a tensor should be is based on RTX 4090s connected via 16x PCIe 4.0. - if ((n_backends <= 2 && ne < 32768) || (n_backends == 3 && ne < 131072) || (n_backends >= 4 && ne < 262144)) { - for (size_t i = 0; i < n_backends; ++i) { - if ((tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - ggml_cuda_set_device(cuda_ctx->device); - CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, ggml_nbytes(tensors[i]), cuda_ctx->stream())); - } - } - NCCL_CHECK(ncclGroupStart()); - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - NCCL_CHECK(ncclAllReduce(tensors[i]->data, tensors[i]->data, ne, ncclFloat, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); - } - NCCL_CHECK(ncclGroupEnd()); - - return true; - } - - // For large tensors it's faster to compress them to BF16 for the reduction: - to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); - to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); - - ggml_cuda_pool_alloc tmp[GGML_CUDA_MAX_DEVICES]; - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - tmp[i].pool = &cuda_ctx->pool(); - tmp[i].alloc(ne); - - ggml_cuda_set_device(cuda_ctx->device); - if (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) { - to_bf16(tensors[i]->data, tmp[i].get(), ne, cuda_ctx->stream()); - } else { - CUDA_CHECK(cudaMemsetAsync(tmp[i].get(), 0, ne * sizeof(nv_bfloat16), cuda_ctx->stream())); - } - CUDA_CHECK(cudaGetLastError()); - } - - NCCL_CHECK(ncclGroupStart()); - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - NCCL_CHECK(ncclAllReduce(tmp[i].get(), tmp[i].get(), ne, ncclBfloat16, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); - } - NCCL_CHECK(ncclGroupEnd()); - - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - - ggml_cuda_set_device(cuda_ctx->device); - to_fp32(tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); - CUDA_CHECK(cudaGetLastError()); - } - - return true; -#else - GGML_UNUSED_VARS(comm_ctx_v, tensors); - return false; -#endif // GGML_USE_NCCL -} - -ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { - static std::mutex mutex; - std::lock_guard lock(mutex); - - static std::map>, struct ggml_backend_buffer_type> buft_map; - - std::array tensor_split_arr = {}; - - bool all_zero = tensor_split == nullptr || std::all_of(tensor_split, tensor_split + GGML_CUDA_MAX_DEVICES, [](float x) { return x == 0.0f; }); - if (all_zero) { - tensor_split_arr = ggml_cuda_info().default_tensor_split; - } else { - float split_sum = 0.0f; - for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { - tensor_split_arr[i] = split_sum; - split_sum += tensor_split[i]; - } - for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { - tensor_split_arr[i] /= split_sum; - } - } - - auto it = buft_map.find({main_device, tensor_split_arr}); - if (it != buft_map.end()) { - return &it->second; - } - auto * ctx = new ggml_backend_cuda_split_buffer_type_context{ - main_device, - tensor_split_arr, - GGML_CUDA_NAME + std::to_string(main_device) + "_Split", - }; - - struct ggml_backend_buffer_type buft { - /* .iface = */ ggml_backend_cuda_split_buffer_type_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), main_device), - /* .context = */ ctx, - }; - - auto result = buft_map.emplace(std::make_pair(main_device, tensor_split_arr), buft); - return &result.first->second; -} - -// host buffer type - -static const char * ggml_backend_cuda_host_buffer_type_name(ggml_backend_buffer_type_t buft) { - return GGML_CUDA_NAME "_Host"; - - GGML_UNUSED(buft); -} - -static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; -} - -static void ggml_backend_cuda_host_buffer_free_buffer(ggml_backend_buffer_t buffer) { - CUDA_CHECK(cudaFreeHost(buffer->context)); -} - -static void * ggml_cuda_host_malloc(size_t size) { - if (getenv("GGML_CUDA_NO_PINNED") != nullptr) { - return nullptr; - } - - void * ptr = nullptr; - cudaError_t err = cudaMallocHost((void **) &ptr, size); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - GGML_LOG_DEBUG("%s: failed to allocate %.2f MiB of pinned memory: %s\n", __func__, - size / 1024.0 / 1024.0, cudaGetErrorString(err)); - return nullptr; - } - - return ptr; -} - -static ggml_backend_buffer_t ggml_backend_cuda_host_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - void * ptr = ggml_cuda_host_malloc(size); - - if (ptr == nullptr) { - // fallback to cpu buffer - return ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size); - } - - ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size); - buffer->buft = buft; - buffer->iface.free_buffer = ggml_backend_cuda_host_buffer_free_buffer; - - return buffer; -} - -ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type() { - static struct ggml_backend_buffer_type ggml_backend_cuda_buffer_type_host = { - /* .iface = */ { - /* .get_name = */ ggml_backend_cuda_host_buffer_type_name, - /* .alloc_buffer = */ ggml_backend_cuda_host_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cpu_buffer_type()->iface.get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cpu_buffer_type()->iface.get_alloc_size, - /* .is_host = */ ggml_backend_cpu_buffer_type()->iface.is_host, - }, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), 0), - /* .context = */ nullptr, - }; - - return &ggml_backend_cuda_buffer_type_host; -} - -//static bool ggml_backend_buffer_is_cuda_host(ggml_backend_buffer_t buffer) { -// return buffer->buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; -//} - -/// kernels - -typedef void (*ggml_cuda_op_mul_mat_t)( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, - const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, - const int64_t src1_padded_row_size, cudaStream_t stream); - -#ifndef GGML_CUDA_PEER_MAX_BATCH_SIZE -#define GGML_CUDA_PEER_MAX_BATCH_SIZE 128 -#endif // GGML_CUDA_PEER_MAX_BATCH_SIZE - -#define MUL_MAT_SRC1_COL_STRIDE 128 - -static cudaError_t ggml_cuda_cpy_tensor_2d( - void * dst, const struct ggml_tensor * src, int64_t i3, int64_t i2, int64_t i1_low, int64_t i1_high, cudaStream_t stream) { - - const char * src_ptr = (const char *) src->data; - char * dst_ptr = (char *) dst; - - const int64_t ne0 = src->ne[0]; - const int64_t nb0 = src->nb[0]; - const int64_t nb1 = src->nb[1]; - const int64_t nb2 = src->nb[2]; - const int64_t nb3 = src->nb[3]; - const enum ggml_type type = src->type; - const int64_t ts = ggml_type_size(type); - const int64_t bs = ggml_blck_size(type); - const int64_t i1_diff = i1_high - i1_low; - - const char * x = src_ptr + i1_low*nb1 + i2*nb2 + i3*nb3; - if (nb0 == ts && nb1 == ts*ne0/bs) { - return cudaMemcpyAsync(dst_ptr, x, i1_diff*nb1, cudaMemcpyDeviceToDevice, stream); - } else if (nb0 == ts) { - return cudaMemcpy2DAsync(dst_ptr, ts*ne0/bs, x, nb1, ts*ne0/bs, i1_diff, cudaMemcpyDeviceToDevice, stream); - } else { - for (int64_t i1 = 0; i1 < i1_diff; i1++) { - const void * rx = (const void *) ((const char *) x + i1*nb1); - void * rd = (void *) (dst_ptr + i1*ts*ne0/bs); - // pretend the row is a matrix with cols=1 - cudaError_t r = cudaMemcpy2DAsync(rd, ts/bs, rx, nb0, ts/bs, ne0, cudaMemcpyDeviceToDevice, stream); - if (r != cudaSuccess) { - return r; - } - } - return cudaSuccess; - } -} - -struct cublas_force_compute_type { - bool fp32 = false; - bool fp16 = false; -}; - -static const cublas_force_compute_type & ggml_cuda_cublas_get_force_compute_type() { - static const cublas_force_compute_type compute_type = [] { - cublas_force_compute_type result; - - const bool ggml_cuda_force_cublas_compute_32f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F") != nullptr; - const bool ggml_cuda_force_cublas_compute_16f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F") != nullptr; - - GGML_ASSERT(ggml_cuda_force_cublas_compute_16f_env == false || ggml_cuda_force_cublas_compute_32f_env == false); - - if (ggml_cuda_force_cublas_compute_32f_env) { - GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F\n"); - result.fp32 = true; - } else if (ggml_cuda_force_cublas_compute_16f_env) { - GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F\n"); - result.fp16 = true; - } - - return result; - }(); - - return compute_type; -} - -static void ggml_cuda_op_mul_mat_cublas( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, - const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, - const int64_t src1_padded_row_size, cudaStream_t stream) { - - GGML_ASSERT(src0_dd_i != nullptr); - GGML_ASSERT(src1_ddf_i != nullptr); - GGML_ASSERT(dst_dd_i != nullptr); - - const int64_t ne00 = src0->ne[0]; - const int64_t ne10 = src1->ne[0]; - - const int64_t ne0 = dst->ne[0]; - - const int64_t row_diff = row_high - row_low; - - int id = ggml_cuda_get_device(); - - // the main device has a larger memory buffer to hold the results from all GPUs - // ldc == nrows of the matrix that cuBLAS writes into - int64_t ldc = id == ctx.device ? ne0 : row_diff; - - const int cc = ggml_cuda_info().devices[id].cc; - - const bool supports_bf16 = GGML_CUDA_CC_IS_NVIDIA(cc) || GGML_CUDA_CC_IS_AMD(cc) || - (GGML_CUDA_CC_IS_MTHREADS(cc) && cc >= GGML_CUDA_CC_QY2); - - const bool use_fp16 = - src0->type != GGML_TYPE_NVFP4 && - (src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && - ggml_is_contiguous(src0) && - row_diff == src0->ne[1] && - dst->op_params[0] == GGML_PREC_DEFAULT; - - if (supports_bf16 && src0->type == GGML_TYPE_BF16 && ggml_is_contiguous(src0) && row_diff == src0->ne[1]) { - ggml_cuda_pool_alloc src1_as_bf16(ctx.pool(id)); - if (src1->type != GGML_TYPE_BF16) { - const to_bf16_cuda_t to_bf16_cuda = ggml_get_to_bf16_cuda(src1->type); - GGML_ASSERT(to_bf16_cuda != nullptr); - size_t ne = src1_ncols*ne10; - src1_as_bf16.alloc(ne); - to_bf16_cuda(src1_ddf_i, src1_as_bf16.get(), ne, stream); - } - const nv_bfloat16 * src1_ptr = src1->type == GGML_TYPE_BF16 ? (const nv_bfloat16 *) src1_ddf_i : src1_as_bf16.get(); - const nv_bfloat16 * src0_ptr = (const nv_bfloat16 *)src0_dd_i; - ggml_cuda_pool_alloc dst_bf16(ctx.pool(id), row_diff*src1_ncols); - - const float alpha_f32 = 1.0f; - const float beta_f32 = 0.0f; - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha_f32, src0_ptr, CUDA_R_16BF, ne00, - src1_ptr, CUDA_R_16BF, ne10, - &beta_f32, dst_bf16.get(), CUDA_R_16BF, ldc, - CUBLAS_COMPUTE_32F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); - to_fp32_cuda(dst_bf16.get(), dst_dd_i, row_diff*src1_ncols, stream); - } else if (fast_fp16_hardware_available(cc) && use_fp16) { - // convert src0 and src1 to fp16, multiply as fp16, convert dst to fp32 - ggml_cuda_pool_alloc src0_as_f16(ctx.pool(id)); - if (src0->type != GGML_TYPE_F16) { - const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src0->type); - GGML_ASSERT(to_fp16_cuda != nullptr); - size_t ne = row_diff*ne00; - src0_as_f16.alloc(ne); - to_fp16_cuda(src0_dd_i, src0_as_f16.get(), ne, stream); - } - const half * src0_ptr = src0->type == GGML_TYPE_F16 ? (const half *) src0_dd_i : src0_as_f16.get(); - - ggml_cuda_pool_alloc src1_as_f16(ctx.pool(id)); - if (src1->type != GGML_TYPE_F16) { - const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type); - GGML_ASSERT(to_fp16_cuda != nullptr); - size_t ne = src1_ncols*ne10; - src1_as_f16.alloc(ne); - to_fp16_cuda(src1_ddf_i, src1_as_f16.get(), ne, stream); - } - const half * src1_ptr = src1->type == GGML_TYPE_F16 ? (const half *) src1_ddf_i : src1_as_f16.get(); - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - - const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); - - if (!force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) - || GGML_CUDA_CC_IS_RDNA4(cc) - || cc == GGML_CUDA_CC_VOLTA - || force_compute_type.fp32)) - { - const float alpha = 1.0f; - const float beta = 0.0f; - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha, src0_ptr, CUDA_R_16F, ne00, - src1_ptr, CUDA_R_16F, ne10, - &beta, dst_dd_i, CUDA_R_32F, ldc, - CUBLAS_COMPUTE_32F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } else { - ggml_cuda_pool_alloc dst_f16(ctx.pool(id), row_diff*src1_ncols); - - const half alpha_f16 = 1.0f; - const half beta_f16 = 0.0f; - - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha_f16, src0_ptr, CUDA_R_16F, ne00, - src1_ptr, CUDA_R_16F, ne10, - &beta_f16, dst_f16.get(), CUDA_R_16F, ldc, - CUBLAS_COMPUTE_16F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_F16); - to_fp32_cuda(dst_f16.get(), dst_dd_i, row_diff*src1_ncols, stream); - } - } else { - ggml_cuda_pool_alloc src0_ddq_as_f32(ctx.pool(id)); - ggml_cuda_pool_alloc src1_ddq_as_f32(ctx.pool(id)); - - if (src0->type != GGML_TYPE_F32) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src0->type); - GGML_ASSERT(to_fp32_cuda != nullptr); - src0_ddq_as_f32.alloc(row_diff*ne00); - to_fp32_cuda(src0_dd_i, src0_ddq_as_f32.get(), row_diff*ne00, stream); - } - if (src1->type != GGML_TYPE_F32) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src1->type); - GGML_ASSERT(to_fp32_cuda != nullptr); - src1_ddq_as_f32.alloc(src1_ncols*ne10); - to_fp32_cuda(src1_ddf_i, src1_ddq_as_f32.get(), src1_ncols*ne10, stream); - } - - const float * src0_ddf_i = src0->type == GGML_TYPE_F32 ? (const float *) src0_dd_i : src0_ddq_as_f32.get(); - const float * src1_ddf1_i = src1->type == GGML_TYPE_F32 ? (const float *) src1_ddf_i : src1_ddq_as_f32.get(); - - const float alpha = 1.0f; - const float beta = 0.0f; - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - CUBLAS_CHECK( - cublasSgemm(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha, src0_ddf_i, ne00, - src1_ddf1_i, ne10, - &beta, dst_dd_i, ldc)); - } - - GGML_UNUSED_VARS(dst, src1_ddq_i, src1_padded_row_size); -} - -static cudaError_t ggml_cuda_Memcpy2DPeerAsync( - void * dst, int dstDevice, size_t dpitch, void * src, int srcDevice, size_t spitch, size_t width, size_t height, cudaStream_t stream) { - -#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) - // cudaMemcpy2DAsync may fail with copies between vmm pools of different devices - cudaMemcpy3DPeerParms p = {}; - p.dstDevice = dstDevice; - p.dstPtr = make_cudaPitchedPtr(dst, dpitch, dpitch, height); - p.srcDevice = srcDevice; - p.srcPtr = make_cudaPitchedPtr(src, spitch, spitch, height); - p.extent = make_cudaExtent(width, height, 1); - return cudaMemcpy3DPeerAsync(&p, stream); -#else - // HIP does not support cudaMemcpy3DPeerAsync or vmm pools - GGML_UNUSED(dstDevice); - GGML_UNUSED(srcDevice); - return cudaMemcpy2DAsync(dst, dpitch, src, spitch, width, height, cudaMemcpyDeviceToDevice, stream); -#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) -} - -static void ggml_cuda_op_mul_mat( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, ggml_cuda_op_mul_mat_t op, - quantize_cuda_t quantize_src1) { - - const int64_t ne00 = src0->ne[0]; - const int64_t ne01 = src0->ne[1]; - const int64_t ne02 = src0->ne[2]; - const int64_t ne03 = src0->ne[3]; - - const int64_t ne10 = src1->ne[0]; - const int64_t ne11 = src1->ne[1]; - const int64_t ne12 = src1->ne[2]; - const int64_t ne13 = src1->ne[3]; - const int64_t nrows1 = ggml_nrows(src1); - - const int64_t ne0 = dst->ne[0]; - const int64_t ne1 = dst->ne[1]; - - // const int64_t nb10 = src1->nb[0]; - const int64_t nb11 = src1->nb[1]; - const int64_t nb12 = src1->nb[2]; - const int64_t nb13 = src1->nb[3]; - - const int64_t nb2 = dst->nb[2]; - const int64_t nb3 = dst->nb[3]; - - ggml_backend_cuda_buffer_context * src1_ctx = (ggml_backend_cuda_buffer_context *) src1->buffer->context; - ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *) dst->buffer->context; - - GGML_ASSERT(src1->type == GGML_TYPE_F32 || (src1->ne[2] == 1 && src1->ne[3] == 1)); - - GGML_ASSERT(ne12 % ne02 == 0); - GGML_ASSERT(ne13 % ne03 == 0); - - const int64_t i02_divisor = ne12 / ne02; - const int64_t i03_divisor = ne13 / ne03; - - const size_t src0_ts = ggml_type_size(src0->type); - const size_t src0_bs = ggml_blck_size(src0->type); - const size_t q8_1_ts = sizeof(block_q8_1); - const size_t q8_1_bs = QK8_1; - - const bool src0_is_contiguous = ggml_is_contiguous(src0); - const bool src1_is_contiguous = ggml_is_contiguous(src1); - - const int64_t src1_padded_col_size = GGML_PAD(ne10, MATRIX_ROW_PADDING); - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); - GGML_ASSERT(!(split && ne02 > 1)); - GGML_ASSERT(!(split && ne03 > 1)); - GGML_ASSERT(!(split && ne02 < ne12)); - GGML_ASSERT(!(split && ne03 < ne13)); - - ggml_tensor_extra_gpu * src0_extra = split ? (ggml_tensor_extra_gpu *) src0->extra : nullptr; - - - std::array tensor_split; - if (split) { - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; - tensor_split = buft_ctx->tensor_split; - } - - struct dev_data { - int cc; - - ggml_cuda_pool_alloc src0_dd_alloc; - ggml_cuda_pool_alloc src1_ddf_alloc; - ggml_cuda_pool_alloc src1_ddq_alloc; - ggml_cuda_pool_alloc dst_dd_alloc; - - char * src0_dd = nullptr; - float * src1_ddf = nullptr; // float - char * src1_ddq = nullptr; // q8_1 - float * dst_dd = nullptr; - - int64_t row_low; - int64_t row_high; - }; - - dev_data dev[GGML_CUDA_MAX_DEVICES]; - - int used_devices = 0; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - dev[id].cc = ggml_cuda_info().devices[id].cc; - - // by default, use all rows - dev[id].row_low = 0; - dev[id].row_high = ne01; - - // for multi GPU, get the row boundaries from tensor split - // and round to mul_mat_q tile sizes - if (split) { - const int64_t rounding = get_row_rounding(tensor_split); - - if (id != 0) { - dev[id].row_low = ne01*tensor_split[id]; - if (dev[id].row_low < ne01) { - dev[id].row_low -= dev[id].row_low % rounding; - } - } - - if (id != ggml_backend_cuda_get_device_count() - 1) { - dev[id].row_high = ne01*tensor_split[id + 1]; - if (dev[id].row_high < ne01) { - dev[id].row_high -= dev[id].row_high % rounding; - } - } - } - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { - continue; - } - - used_devices++; - - const bool src1_on_device = id == src1_ctx->device; - const bool dst_on_device = id == dst_ctx->device; - - ggml_cuda_set_device(id); - cudaStream_t stream = ctx.stream(id, 0); - - if (src0_is_contiguous) { - dev[id].src0_dd = split ? (char *) src0_extra->data_device[id] : (char *) src0->data; - } else { - // If src0 is not contiguous it will be copied to a temporary buffer. - // This buffer needs to be cleared entirely because multiple regions will function as padding. - const size_t nbytes_data = ggml_nbytes(src0); - const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); - dev[id].src0_dd = dev[id].src0_dd_alloc.alloc(ctx.pool(id), nbytes_data + nbytes_padding); - CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd, 0, nbytes_data + nbytes_padding, stream)); - } - - // If src0 is on a temporary compute buffer (partial offloading) there may be some padding that needs to be cleared: - if (ne00 % MATRIX_ROW_PADDING != 0 && ggml_is_quantized(src0->type) && ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && src0->view_src == nullptr) { - GGML_ASSERT(ggml_is_contiguously_allocated(src0)); - GGML_ASSERT(!src0->view_src); - const size_t nbytes_data = ggml_row_size(src0->type, (dev[id].row_high - dev[id].row_low)*ne00); - const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); - CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd + nbytes_data, 0, nbytes_padding, stream)); - } - - if (src1_on_device && src1_is_contiguous) { - dev[id].src1_ddf = (float *) src1->data; - } else { - dev[id].src1_ddf = dev[id].src1_ddf_alloc.alloc(ctx.pool(id), ggml_nelements(src1)); - } - - if (quantize_src1) { - size_t src_1_ddq_size = nrows1*src1_padded_col_size*q8_1_ts/q8_1_bs; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - src_1_ddq_size += get_mmq_x_max_host(dev[id].cc)*sizeof(block_q8_1_mmq); - } - dev[id].src1_ddq = dev[id].src1_ddq_alloc.alloc(ctx.pool(id), src_1_ddq_size); - - if (src1_on_device && src1_is_contiguous) { - quantize_src1( - dev[id].src1_ddf, nullptr, dev[id].src1_ddq, src0->type, ne10, - nb11/sizeof(float), nb12/sizeof(float), nb13/sizeof(float), - src1_padded_col_size, ne11, ne12, ne13, stream); - CUDA_CHECK(cudaGetLastError()); - } - } - - if (dst_on_device) { - dev[id].dst_dd = (float *) dst->data; - } else { - const size_t size_dst_ddf = split ? (dev[id].row_high - dev[id].row_low)*ne1 : ggml_nelements(dst); - dev[id].dst_dd = dev[id].dst_dd_alloc.alloc(ctx.pool(id), size_dst_ddf); - } - } - - // if multiple devices are used they need to wait for the main device - // here an event is recorded that signals that the main device has finished calculating the input data - if (split && used_devices > 1) { - ggml_cuda_set_device(ctx.device); - CUDA_CHECK(cudaEventRecord(src0_extra->events[ctx.device][0], ctx.stream())); - } - - const int64_t src1_col_stride = split && used_devices > 1 ? MUL_MAT_SRC1_COL_STRIDE : ne11; - for (int64_t src1_col_0 = 0; src1_col_0 < ne11; src1_col_0 += src1_col_stride) { - const int64_t is = split ? (src1_col_0/src1_col_stride) % GGML_CUDA_MAX_STREAMS : 0; - const int64_t src1_ncols = src1_col_0 + src1_col_stride > ne11 ? ne11 - src1_col_0 : src1_col_stride; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { - continue; - } - - const bool src1_on_device = id == src1_ctx->device; - const bool dst_on_device = id == dst_ctx->device; - const int64_t row_diff = dev[id].row_high - dev[id].row_low; - - ggml_cuda_set_device(id); - cudaStream_t stream = ctx.stream(id, is); - - // wait for main GPU data if necessary - if (split && (id != ctx.device || is != 0)) { - CUDA_CHECK(cudaStreamWaitEvent(stream, src0_extra->events[ctx.device][0], 0)); - } - - for (int64_t i0 = 0; i0 < ne13*ne12; ++i0) { - const int64_t i03 = i0 / ne12; - const int64_t i02 = i0 % ne12; - - size_t src1_ddq_i_offset = i0*ne11 * src1_padded_col_size*q8_1_ts/q8_1_bs; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - src1_ddq_i_offset += src1_col_0 * sizeof(block_q8_1_mmq); - } else { - src1_ddq_i_offset += src1_col_0 * src1_padded_col_size*q8_1_ts/q8_1_bs; - } - - // for split tensors the data begins at i0 == i0_offset_low - const size_t nbytes_src0_matrix = ne01*ne00*src0_ts / src0_bs; - char * src0_dd_i = dev[id].src0_dd + ((i03/i03_divisor)*ne02 + (i02/i02_divisor)) * nbytes_src0_matrix; - float * src1_ddf_i = dev[id].src1_ddf + (i0*ne11 + src1_col_0) * ne10; - char * src1_ddq_i = dev[id].src1_ddq + src1_ddq_i_offset; - float * dst_dd_i = dev[id].dst_dd + (i0*ne1 + src1_col_0) * (dst_on_device ? ne0 : row_diff); - - // the main device memory buffer can be on VRAM scratch, with space for all partial results - // in that case an offset on dst_ddf_i is needed - if (id == ctx.device) { - dst_dd_i += dev[id].row_low; // offset is 0 if no tensor split - } - - // copy src0, src1 to device if necessary - if (src1_is_contiguous) { - if (id != ctx.device) { - if (quantize_src1) { - char * src1_ddq_i_source = dev[ctx.device].src1_ddq + src1_ddq_i_offset; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - const size_t pitch = ne11*sizeof(block_q8_1_mmq); - const size_t width = src1_ncols*sizeof(block_q8_1_mmq); - const size_t height = src1_padded_col_size/(4*QK8_1); - CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync(src1_ddq_i, id, pitch, src1_ddq_i_source, ctx.device, pitch, width, height, stream)); - } else { - CUDA_CHECK(cudaMemcpyPeerAsync( - src1_ddq_i, id, src1_ddq_i_source, ctx.device, src1_ncols*src1_padded_col_size*q8_1_ts/q8_1_bs, stream)); - } - } else { - float * src1_ddf_i_source = (float *) src1->data; - src1_ddf_i_source += (i0*ne11 + src1_col_0) * ne10; - CUDA_CHECK(cudaMemcpyPeerAsync(src1_ddf_i, id, src1_ddf_i_source, ctx.device, - src1_ncols*ne10*sizeof(float), stream)); - } - } - } else if (src1_on_device && !src1_is_contiguous) { - CUDA_CHECK(ggml_cuda_cpy_tensor_2d( - src1_ddf_i, src1, i03, i02, src1_col_0, src1_col_0+src1_ncols, stream)); - } else { - GGML_ABORT("fatal error"); - } - - if (quantize_src1 && !src1_is_contiguous) { - quantize_src1( - src1_ddf_i, nullptr, src1_ddq_i, src0->type, ne10, ne10, ne11*ne10, ne12*ne11*ne10, - src1_padded_col_size, src1_ncols, 1, 1, stream); - CUDA_CHECK(cudaGetLastError()); - } - - if (src1_col_0 == 0 && !src0_is_contiguous && i03 % i03_divisor == 0 && i02 % i02_divisor == 0) { - CUDA_CHECK(ggml_cuda_cpy_tensor_2d( - src0_dd_i, src0, i03/i03_divisor, i02/i02_divisor, dev[id].row_low, dev[id].row_high, stream)); - } - - // do the computation - op(ctx, src0, src1, dst, src0_dd_i, src1_ddf_i, src1_ddq_i, dst_dd_i, - dev[id].row_low, dev[id].row_high, src1_ncols, src1_padded_col_size, stream); - CUDA_CHECK(cudaGetLastError()); - - // copy dst to host or other device if necessary - if (!dst_on_device) { - void * dst_off_device = dst->data; - if (split) { - // src0 = weight matrix is saved as a transposed matrix for better memory layout. - // dst is NOT transposed. - // The outputs of matrix matrix multiplications can therefore NOT simply be concatenated for >1 GPU. - // Instead they need to be copied to the correct slice in ne0 = dst row index. - // If dst is a vector with ne0 == 1 then you don't have to do this but it still produces correct results. - float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); - GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); - dhf_dst_i += src1_col_0*ne0 + dev[id].row_low; - CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync( - dhf_dst_i, ctx.device, ne0*sizeof(float), dst_dd_i, id, row_diff*sizeof(float), row_diff*sizeof(float), src1_ncols, stream)); - } else { - float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); - GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); - dhf_dst_i += src1_col_0*ne0; - CUDA_CHECK(cudaMemcpyAsync(dhf_dst_i, dst_dd_i, src1_ncols*ne0*sizeof(float), cudaMemcpyDeviceToDevice, stream)); - } - } - - // add event for the main device to wait on until other device is done - if (split && (id != ctx.device || is != 0)) { - CUDA_CHECK(cudaEventRecord(src0_extra->events[id][is], stream)); - } - } - } - } - - // main device waits for all other devices to be finished - if (split && ggml_backend_cuda_get_device_count() > 1) { - int64_t is_max = (ne11 + MUL_MAT_SRC1_COL_STRIDE - 1) / MUL_MAT_SRC1_COL_STRIDE; - is_max = is_max <= GGML_CUDA_MAX_STREAMS ? is_max : GGML_CUDA_MAX_STREAMS; - - ggml_cuda_set_device(ctx.device); - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if (dev[id].row_low == dev[id].row_high) { - continue; - } - for (int64_t is = 0; is < is_max; ++is) { - CUDA_CHECK(cudaStreamWaitEvent(ctx.stream(), src0_extra->events[id][is], 0)); - } - } - } -} - -static __global__ void k_compute_batched_ptrs( - const void * src0_as_f16, const void * src1_as_f16, char * dst, - const void ** ptrs_src, void ** ptrs_dst, - int64_t ne12, int64_t ne13, - int64_t ne23, - size_t nb02, size_t nb03, - size_t nb12, size_t nb13, - size_t nbd2, size_t nbd3, - int64_t r2, int64_t r3) { - const int64_t i13 = blockIdx.x * blockDim.x + threadIdx.x; - const int64_t i12 = blockIdx.y * blockDim.y + threadIdx.y; - - if (i13 >= ne13 || i12 >= ne12) { - return; - } - - const int64_t i03 = i13 / r3; - const int64_t i02 = i12 / r2; - - ptrs_src[0*ne23 + i12 + i13*ne12] = (const char *) src0_as_f16 + i02*nb02 + i03*nb03; - ptrs_src[1*ne23 + i12 + i13*ne12] = (const char *) src1_as_f16 + i12*nb12 + i13*nb13; - ptrs_dst[0*ne23 + i12 + i13*ne12] = ( char *) dst + i12*nbd2 + i13*nbd3; -} - -// Type traits for mapping ggml types to CUDA/cuBLAS types -template -struct batched_mul_mat_traits; - -template<> -struct batched_mul_mat_traits { - using cuda_type = float; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; - static inline const cudaDataType_t data_type = CUDA_R_32F; - static inline const ggml_type ggml_type_val = GGML_TYPE_F32; - static inline const float alpha = 1.0f; - static inline const float beta = 0.0f; - static inline const void* get_alpha() { static const float val = alpha; return &val; } - static inline const void* get_beta() { static const float val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp32_nc_cuda(src_type); } -}; - -template<> -struct batched_mul_mat_traits { - using cuda_type = nv_bfloat16; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; - static inline const cudaDataType_t data_type = CUDA_R_16BF; - static inline const ggml_type ggml_type_val = GGML_TYPE_BF16; - static inline const float alpha = 1.0f; - static inline const float beta = 0.0f; - static inline const void* get_alpha() { static const float val = alpha; return &val; } - static inline const void* get_beta() { static const float val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_bf16_nc_cuda(src_type); } -}; - -template<> -struct batched_mul_mat_traits { - using cuda_type = half; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_16F; - static inline const cudaDataType_t data_type = CUDA_R_16F; - static inline const ggml_type ggml_type_val = GGML_TYPE_F16; - static inline const half alpha = 1.0; - static inline const half beta = 0.0; - static inline const void* get_alpha() { static const half val = alpha; return &val; } - static inline const void* get_beta() { static const half val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp16_nc_cuda(src_type); } -}; - -template -static void ggml_cuda_mul_mat_batched_cublas_impl(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - using traits = batched_mul_mat_traits; - using cuda_t = typename traits::cuda_type; - - GGML_ASSERT(!ggml_is_transposed(src0)); - GGML_ASSERT(!ggml_is_transposed(src1)); - GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft)); - GGML_ASSERT(src0->type == src0_type); - GGML_ASSERT(ggml_is_contiguous(dst)); - - // Byte offsets and tensor dimensions are currently used in an inconsistent way for dst. - // As long as dst is contiguous this does not matter though. - - GGML_TENSOR_BINARY_OP_LOCALS - - const int64_t ne_dst = ggml_nelements(dst); - cudaStream_t main_stream = ctx.stream(); - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(), main_stream)); - - float * dst_ddf = (float *) dst->data; - const size_t ts_src1 = ggml_type_size(src1->type); - GGML_ASSERT(nb10 == ts_src1); - int64_t s11 = nb11 / ts_src1; - int64_t s12 = nb12 / ts_src1; - int64_t s13 = nb13 / ts_src1; - - const cuda_t * src0_ptr = nullptr; - const cuda_t * src1_ptr = nullptr; - - ggml_cuda_pool_alloc src0_alloc(ctx.pool()); - ggml_cuda_pool_alloc src1_alloc(ctx.pool()); - - bool is_src0_cont_2 = ggml_is_contiguous_2(src0); - bool is_src1_cont_2 = ggml_is_contiguous_2(src1); - - // Handle src0 - src0_ptr = (const cuda_t *) src0->data; - - // Handle src1 - convert if necessary - if (src1->type == src0_type) { - src1_ptr = (const cuda_t *) src1->data; - } else { - // Convert src1 to target type using traits conversion functions - const int64_t ne_src1 = ggml_nelements(src1); - src1_alloc.alloc(ne_src1); - - const auto convert_func = traits::get_nc_converter(src1->type); - GGML_ASSERT(convert_func != nullptr); - convert_func(src1->data, src1_alloc.get(), ne10, ne11, ne12, ne13, s11, s12, s13, main_stream); - src1_ptr = src1_alloc.get(); - s11 = ne10; - s12 = ne11*s11; - s13 = ne12*s12; - - is_src1_cont_2 = true; - } - - // Setup destination buffer - ggml_cuda_pool_alloc dst_temp(ctx.pool()); - char * dst_t; - size_t nbd2 = dst->nb[2]; - size_t nbd3 = dst->nb[3]; - - cublasComputeType_t cu_compute_type = traits::compute_type; - cudaDataType_t cu_data_type = traits::data_type; - cudaDataType_t cu_data_type_a = traits::data_type; - cudaDataType_t cu_data_type_b = traits::data_type; - const void * alpha = traits::get_alpha(); - const void * beta = traits::get_beta(); - - const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); - - int id = ggml_cuda_get_device(); - const int cc = ggml_cuda_info().devices[id].cc; - static constexpr bool is_src0_type_f16 = src0_type == GGML_TYPE_F16; - - // bf16 and fp32 are already being computed in fp32 (ensure it using static_assert), - // so checking necessity of forced fp32 only for fp16 src0_type - static_assert(is_src0_type_f16 || traits::compute_type == CUBLAS_COMPUTE_32F); - - const bool need_compute_32f = is_src0_type_f16 && !force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) - || GGML_CUDA_CC_IS_RDNA4(cc) - || cc == GGML_CUDA_CC_VOLTA - || force_compute_type.fp32); - - if (dst->op_params[0] == GGML_PREC_DEFAULT && !need_compute_32f) { - if constexpr (src0_type == GGML_TYPE_F32) { - dst_t = (char *) dst_ddf; // Direct F32 output - } else { - dst_t = (char *) dst_temp.alloc(ne_dst); - nbd2 /= sizeof(float) / sizeof(cuda_t); - nbd3 /= sizeof(float) / sizeof(cuda_t); - } - } else { - dst_t = (char *) dst_ddf; - cu_compute_type = batched_mul_mat_traits::compute_type; - cu_data_type = batched_mul_mat_traits::data_type; - alpha = batched_mul_mat_traits::get_alpha(); - beta = batched_mul_mat_traits::get_beta(); - } - - GGML_ASSERT(ne12 % ne02 == 0); - GGML_ASSERT(ne13 % ne03 == 0); - - // broadcast factors - const int64_t r2 = ne12/ne02; - const int64_t r3 = ne13/ne03; - - if (r2 == 1 && r3 == 1 && is_src0_cont_2 && is_src1_cont_2) { - // with a [0, 2, 1, 3] perm. and ne02==1 the matrix strides need to be determined from dim 3: - const int64_t sma = ne02 == 1 ? nb03/nb00 : nb02/nb00; - const int64_t smb = ne12 == 1 ? s13 : s12; - - // there is no broadcast and src0, src1 are contiguous across dims 2, 3 - // use cublasGemmStridedBatchedEx - CUBLAS_CHECK( - cublasGemmStridedBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, - ne01, ne11, ne10, - alpha, src0_ptr, cu_data_type_a, nb01/nb00, sma, // strideA - src1_ptr, cu_data_type_b, s11, smb, // strideB - beta, dst_t, cu_data_type, ne0, ne1*ne0, // strideC - ne12*ne13, - cu_compute_type, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } else { - // use cublasGemmBatchedEx - const int64_t ne23 = ne12*ne13; - - ggml_cuda_pool_alloc ptrs_src(ctx.pool(), 2*ne23); - ggml_cuda_pool_alloc< void *> ptrs_dst(ctx.pool(), 1*ne23); - - size_t src1_stride_size = sizeof(cuda_t); - - const int threads_x = 16; - const int threads_y = 16; - dim3 block_dims(threads_x, threads_y); - - dim3 grid_dims( - (ne13 + threads_x - 1) / threads_x, - (ne12 + threads_y - 1) / threads_y - ); - k_compute_batched_ptrs<<>>( - src0_ptr, src1_ptr, dst_t, - ptrs_src.get(), ptrs_dst.get(), - ne12, ne13, - ne23, - nb02, nb03, - (src1->type == src0_type) ? nb12 : s12*src1_stride_size, - (src1->type == src0_type) ? nb13 : s13*src1_stride_size, - nbd2, nbd3, - r2, r3); - - CUDA_CHECK(cudaGetLastError()); - - CUBLAS_CHECK( - cublasGemmBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, - ne01, ne11, ne10, - alpha, (const void **) (ptrs_src.get() + 0*ne23), cu_data_type_a, nb01/nb00, - (const void **) (ptrs_src.get() + 1*ne23), cu_data_type_b, s11, - beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne0, - ne23, - cu_compute_type, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } - - // Convert output back to F32 if needed - if (dst->op_params[0] == GGML_PREC_DEFAULT && cu_data_type != CUDA_R_32F) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(traits::ggml_type_val); - to_fp32_cuda(dst_temp.get(), dst_ddf, ne_dst, main_stream); - } -} - -static void ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - GGML_ASSERT(src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16 || src0->type == GGML_TYPE_F32); - - switch (src0->type) { - case GGML_TYPE_F32: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - case GGML_TYPE_BF16: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - case GGML_TYPE_F16: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - default: - GGML_ABORT("Unsupported type"); - } -} - -static bool ggml_cuda_should_fuse_mul_mat(const ggml_tensor * ffn_up, - const ggml_tensor * ffn_gate, - const ggml_tensor * glu, - const ggml_tensor * ffn_up_bias = nullptr, - const ggml_tensor * ffn_gate_bias = nullptr) { - const bool has_bias = ffn_up_bias != nullptr || ffn_gate_bias != nullptr; - - if (has_bias && (!ffn_up_bias || !ffn_gate_bias)) { - return false; - } - - const bool is_mul_mat = ffn_up->op == GGML_OP_MUL_MAT && ffn_gate->op == GGML_OP_MUL_MAT && glu->op == GGML_OP_GLU; - const bool is_mul_mat_id = ffn_up->op == GGML_OP_MUL_MAT_ID && ffn_gate->op == GGML_OP_MUL_MAT_ID && glu->op == GGML_OP_GLU; - - GGML_ASSERT(ffn_up && ffn_gate && glu); - - if (!is_mul_mat && !is_mul_mat_id) { - return false; - } - - const ggml_op expected_bias_op = is_mul_mat ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (has_bias) { - if (ffn_up_bias->op != expected_bias_op || ffn_gate_bias->op != expected_bias_op) { - return false; - } - - if (glu->src[0] != ffn_gate_bias || glu->src[1] != ffn_up_bias) { - return false; - } - - if (expected_bias_op == GGML_OP_ADD) { - const bool up_has_mul = ffn_up_bias->src[0] == ffn_up || ffn_up_bias->src[1] == ffn_up; - const bool gate_has_mul = ffn_gate_bias->src[0] == ffn_gate || ffn_gate_bias->src[1] == ffn_gate; - if (!up_has_mul || !gate_has_mul) { - return false; - } - } else { // GGML_OP_ADD_ID - if (ffn_up_bias->src[0] != ffn_up || ffn_gate_bias->src[0] != ffn_gate) { - return false; - } - if (ffn_up_bias->src[2] != ffn_up->src[2] || ffn_gate_bias->src[2] != ffn_gate->src[2]) { - return false; - } - } - } else { - if (glu->src[0] != ffn_gate && glu->src[1] != ffn_up) { - return false; - } - } - - if (ffn_up->src[0]->type != ffn_gate->src[0]->type || !ggml_are_same_shape(ffn_up->src[0], ffn_gate->src[0]) || - !ggml_are_same_stride(ffn_up->src[0], ffn_gate->src[0])) { - return false; - } - - if (ffn_up->src[1] != ffn_gate->src[1]) { - return false; - } - - if (ffn_up->src[2] && (ffn_up->src[2] != ffn_gate->src[2])) { - return false; - } - - static constexpr std::array valid_glu_ops = { GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU, GGML_GLU_OP_SWIGLU_OAI }; - - if (std::find(valid_glu_ops.begin(), valid_glu_ops.end(), ggml_get_glu_op(glu)) == valid_glu_ops.end()) { - return false; - } - - if (const bool swapped = ggml_get_op_params_i32(glu, 1); swapped) { - return false; - } - - const bool split = ggml_backend_buft_is_cuda_split(ffn_up->src[0]->buffer->buft) || - ggml_backend_buft_is_cuda_split(ffn_gate->src[0]->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - return true; -} - -static bool ggml_cuda_should_fuse_mul_mat_vec_f(const ggml_tensor * tensor) { - ggml_tensor * src0 = tensor->src[0]; - ggml_tensor * src1 = tensor->src[1]; - const ggml_tensor * dst = tensor; - - const bool is_mul_mat_id = tensor->op == GGML_OP_MUL_MAT_ID; - - bool use_mul_mat_vec_f = - (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) && - src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, is_mul_mat_id ? src1->ne[2] : src1->ne[1]); - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || - ggml_backend_buft_is_cuda_split(src1->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - //we only support fusion for ncols_dst = 1 - if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { - return false; - } - - if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { - return false; - } - - - return use_mul_mat_vec_f; -} - -static bool ggml_cuda_should_fuse_mul_mat_vec_q(const ggml_tensor * tensor) { - ggml_tensor * src0 = tensor->src[0]; - ggml_tensor * src1 = tensor->src[1]; - const ggml_tensor * dst = tensor; - - const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && - ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && - src0->view_src; - - bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear && src1->type == GGML_TYPE_F32 && - dst->type == GGML_TYPE_F32 && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; - - // fusion is not universally faster on Pascal - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - if (cc <= GGML_CUDA_CC_PASCAL) { - return false; - } - //we only support fusion for ncols_dst = 1 - if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { - return false; - } - - if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { - return false; - } - - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || - ggml_backend_buft_is_cuda_split(src1->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - return use_mul_mat_vec_q; -} - -static void ggml_cuda_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); - - // If src0 is a temporary compute buffer it may have some padding that needs to be cleared for mul_mat_vec_q or mul_mat_q. - // But if src0 is also a view of another tensor then this cannot be done safely because it may overwrite valid tensor data. - // Therefore, in such cases use cuBLAS. - const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE - && ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && src0->view_src; - - bool use_mul_mat_vec_f = (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - bool use_mul_mat_f = !ggml_is_quantized(src0->type) - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32 - && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; - bool use_mul_mat_q = ggml_is_quantized(src0->type) && !bad_padding_clear - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - - bool any_gpus_with_slow_fp16 = false; - - if (split) { - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; - auto & tensor_split = buft_ctx->tensor_split; - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - // skip devices that are not going to do any work: - if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { - continue; - } - - const int cc = ggml_cuda_info().devices[id].cc; - const int warp_size = ggml_cuda_info().devices[id].warp_size; - use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); - use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); - any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); - } - } else { - const int cc = ggml_cuda_info().devices[ctx.device].cc; - const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size; - use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); - use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); - any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); - } - - // debug helpers - //printf("src0: %8d %8d %8d %8d\n", src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3]); - //printf(" %8d %8d %8d %8d\n", src0->nb[0], src0->nb[1], src0->nb[2], src0->nb[3]); - //printf("src1: %8d %8d %8d %8d\n", src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3]); - //printf(" %8d %8d %8d %8d\n", src1->nb[0], src1->nb[1], src1->nb[2], src1->nb[3]); - //printf("src0 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src0), ggml_is_transposed(src0), ggml_type_name(src0->type), src0->name); - //printf("src1 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src1), ggml_is_transposed(src1), ggml_type_name(src1->type), src1->name); - - //TODO update for generic tensor parallelism - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - bool use_batched_cublas_f16 = src0->type == GGML_TYPE_F16 && (src1->type == GGML_TYPE_F16 || !any_gpus_with_slow_fp16); - bool use_batched_cublas_bf16 = src0->type == GGML_TYPE_BF16 && bf16_mma_hardware_available(cc); - bool use_batched_cublas_f32 = src0->type == GGML_TYPE_F32; - - if (!split && use_mul_mat_vec_f) { - // the custom F16 vector kernel can be used over batched cuBLAS GEMM - // but this is only faster for GPUs without tensor cores or with a thin src0 matrix (particularly KQV in attention) - ggml_cuda_mul_mat_vec_f(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_f) { - ggml_cuda_mul_mat_f(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_vec_q) { - ggml_cuda_mul_mat_vec_q(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_q) { - ggml_cuda_mul_mat_q(ctx, src0, src1, nullptr, dst); - } else if (!split && (use_batched_cublas_f16 || use_batched_cublas_bf16 || use_batched_cublas_f32) - && !ggml_is_transposed(src0) && !ggml_is_transposed(src1) && src1->ne[2]*src1->ne[3] > 1) { - // general KQ + KQV multi-batch without FlashAttention - ggml_cuda_mul_mat_batched_cublas(ctx, src0, src1, dst); - } else if (use_mul_mat_vec_f) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_f, nullptr); - } else if (use_mul_mat_vec_q) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_q, quantize_row_q8_1_cuda); - } else if (use_mul_mat_q) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_q, quantize_mmq_q8_1_cuda); - } else { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_cublas, nullptr); - } -} - -static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor * dst) { - const ggml_tensor * src0 = dst->src[0]; - const ggml_tensor * src1 = dst->src[1]; - const ggml_tensor * ids = dst->src[2]; - - GGML_ASSERT(src1->type == GGML_TYPE_F32); - GGML_ASSERT(dst->type == GGML_TYPE_F32); - GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft) && "mul_mat_id does not support split buffers"); - - GGML_TENSOR_BINARY_OP_LOCALS - - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - - // [TAG_MUL_MAT_ID_CUDA_GRAPHS] - if (src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { - static_assert(MMVQ_MAX_BATCH_SIZE == MMVF_MAX_BATCH_SIZE); - if (ne2 <= MMVQ_MAX_BATCH_SIZE) { - if (ggml_is_quantized(src0->type)) { - const int mmvq_mmid_max = get_mmvq_mmid_max_batch(src0->type, cc); - if (ne2 <= mmvq_mmid_max) { - ggml_cuda_mul_mat_vec_q(ctx, src0, src1, ids, dst); - return; - } - } else { - if (GGML_CUDA_CC_IS_AMD(cc)) { - ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst); - return; - } - } - } - - if (ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) { - ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst); - return; - } - - if (ggml_cuda_should_use_mmf(src0->type, cc, WARP_SIZE, src0->ne, src0->nb, src1->ne[2], /*mul_mat_id=*/true)) { - ggml_cuda_mul_mat_f(ctx, src0, src1, ids, dst); - return; - } - } - - // note: this path should not be reached when recording CUDA graphs, because it requires stream synchronization - // TODO: add asserts to verify this. should work with CUDA, HIP, etc. - cudaStream_t stream = ctx.stream(); - - GGML_ASSERT(nb12 % nb11 == 0); - GGML_ASSERT(nb2 % nb1 == 0); - - const ggml_type type_src1_sorted = (src0->type == GGML_TYPE_F16 && !fast_fp16_hardware_available(cc)) - || ggml_is_quantized(src0->type) ? GGML_TYPE_F32 : src0->type; - const ggml_type type_dst_sorted = GGML_TYPE_F32; - const size_t ts_src1_sorted = ggml_type_size(type_src1_sorted); - const size_t ts_dst_sorted = ggml_type_size(type_dst_sorted); - - const int64_t n_expert_used = ids->ne[0]; - const int64_t ne_get_rows = ne12 * n_expert_used; - - std::vector ids_to_sorted_host; - ids_to_sorted_host.reserve(2*ne_get_rows); - std::vector ids_from_sorted_host(ne_get_rows); - - ggml_cuda_pool_alloc ids_buf_dev(ctx.pool(), 2*ne_get_rows); - - std::vector tokens_per_expert(ne02); - - ggml_cuda_pool_alloc src1_sorted(ctx.pool(), ne12*n_expert_used*ne10*ts_src1_sorted); - ggml_cuda_pool_alloc dst_sorted(ctx.pool(), ne2 *n_expert_used* ne0*ts_dst_sorted); - - std::vector ids_host(ggml_nbytes(ids)); - CUDA_CHECK(cudaMemcpyAsync(ids_host.data(), ids->data, ggml_nbytes(ids), cudaMemcpyDeviceToHost, stream)); - CUDA_CHECK(cudaStreamSynchronize(stream)); - - for (int64_t i02 = 0; i02 < ne02; ++i02) { // expert matrices - for (int64_t i12 = 0; i12 < ne12; ++i12) { // tokens - for (int64_t iex = 0; iex < n_expert_used; ++iex) { - const int32_t expert_to_use = *(const int32_t *)(ids_host.data() + i12*ids->nb[1] + iex*ids->nb[0]); - assert(expert_to_use >= 0 && expert_to_use < ne02); - if (expert_to_use == i02) { - ids_from_sorted_host[i12*n_expert_used + iex] = ids_to_sorted_host.size(); - ids_to_sorted_host.push_back(i12*ne11 + iex % ne11); - tokens_per_expert[i02]++; - break; - } - } - } - } - GGML_ASSERT(ids_to_sorted_host.size() == size_t(ne_get_rows)); - - ids_to_sorted_host.insert(ids_to_sorted_host.end(), ids_from_sorted_host.begin(), ids_from_sorted_host.end()); - - CUDA_CHECK(cudaMemcpyAsync(ids_buf_dev.ptr, ids_to_sorted_host.data(), 2*ne_get_rows*sizeof(int32_t), cudaMemcpyHostToDevice, stream)); - CUDA_CHECK(cudaStreamSynchronize(stream)); - - const int32_t * ids_to_sorted = ids_buf_dev.ptr + 0*ne_get_rows; - const int32_t * ids_from_sorted = ids_buf_dev.ptr + 1*ne_get_rows; - - get_rows_cuda(src1->data, src1->type, ids_to_sorted, src1_sorted.ptr, type_src1_sorted, - ne10, nb11, nb12, nb13, - ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), - ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, stream); - CUDA_CHECK(cudaGetLastError()); - - char * src1_data_cur = (char *) src1_sorted.ptr; - char * dst_data_cur = (char *) dst_sorted.ptr; - for (int64_t i02 = 0; i02 < ne02; ++i02) { - if (tokens_per_expert[i02] == 0) { - continue; - } - - ggml_tensor src0_slice = *src0; - src0_slice.ne[2] = 1; - src0_slice.nb[3] = src0_slice.nb[2]; - src0_slice.op = GGML_OP_VIEW; - src0_slice.view_src = dst->src[0]; // non-const pointer to src0 - src0_slice.data = (char *) src0->data + i02*nb02; - - ggml_tensor src1_slice; - memset(&src1_slice, 0, sizeof(src1_slice)); - src1_slice.buffer = src1->buffer; - src1_slice.type = type_src1_sorted; - src1_slice.ne[0] = ne10; - src1_slice.ne[1] = tokens_per_expert[i02]; - src1_slice.ne[2] = 1; - src1_slice.ne[3] = 1; - src1_slice.nb[0] = ts_src1_sorted; - src1_slice.nb[1] = src1_slice.ne[0] * src1_slice.nb[0]; - src1_slice.nb[2] = src1_slice.ne[1] * src1_slice.nb[1]; - src1_slice.nb[3] = src1_slice.ne[2] * src1_slice.nb[2]; - src1_slice.data = src1_data_cur; - - ggml_tensor dst_slice; - memset(&dst_slice, 0, sizeof(dst_slice)); - dst_slice.buffer = dst->buffer; - dst_slice.type = type_dst_sorted; - dst_slice.ne[0] = ne0; - dst_slice.ne[1] = tokens_per_expert[i02]; - dst_slice.ne[2] = 1; - dst_slice.ne[3] = 1; - dst_slice.nb[0] = ts_dst_sorted; - dst_slice.nb[1] = dst_slice.ne[0] * dst_slice.nb[0]; - dst_slice.nb[2] = dst_slice.ne[1] * dst_slice.nb[1]; - dst_slice.nb[3] = dst_slice.ne[2] * dst_slice.nb[2]; - dst_slice.data = dst_data_cur; - - ggml_cuda_mul_mat(ctx, &src0_slice, &src1_slice, &dst_slice); - CUDA_CHECK(cudaGetLastError()); - - src1_data_cur += src1_slice.nb[2]; - dst_data_cur += dst_slice.nb[2]; - } - - get_rows_cuda(dst_sorted.ptr, type_dst_sorted, ids_from_sorted, dst->data, dst->type, - ne0, ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, - ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), - nb1, nb2, nb3, stream); -} - -static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct ggml_tensor * dst) { - switch (dst->op) { - case GGML_OP_ARGMAX: - ggml_cuda_argmax(ctx, dst); - break; - case GGML_OP_COUNT_EQUAL: - ggml_cuda_count_equal(ctx, dst); - break; - case GGML_OP_REPEAT: - ggml_cuda_op_repeat(ctx, dst); - break; - case GGML_OP_REPEAT_BACK: - ggml_cuda_op_repeat_back(ctx, dst); - break; - case GGML_OP_GET_ROWS: - ggml_cuda_op_get_rows(ctx, dst); - break; - case GGML_OP_GET_ROWS_BACK: - ggml_cuda_op_get_rows_back(ctx, dst); - break; - case GGML_OP_SET_ROWS: - ggml_cuda_op_set_rows(ctx, dst); - break; - case GGML_OP_SET: - ggml_cuda_op_set(ctx, dst); - break; - case GGML_OP_DUP: - ggml_cuda_dup(ctx, dst); - break; - case GGML_OP_CPY: - ggml_cuda_cpy(ctx, dst->src[0], dst->src[1]); - break; - case GGML_OP_CONT: - ggml_cuda_dup(ctx, dst); - break; - case GGML_OP_ADD: - case GGML_OP_ADD1: // TODO: more efficient implementation - ggml_cuda_op_add(ctx, dst); - break; - case GGML_OP_ADD_ID: - ggml_cuda_op_add_id(ctx, dst); - break; - case GGML_OP_SUB: - ggml_cuda_op_sub(ctx, dst); - break; - case GGML_OP_ACC: - ggml_cuda_op_acc(ctx, dst); - break; - case GGML_OP_MUL: - ggml_cuda_op_mul(ctx, dst); - break; - case GGML_OP_DIV: - ggml_cuda_op_div(ctx, dst); - break; - case GGML_OP_UNARY: - switch (ggml_get_unary_op(dst)) { - case GGML_UNARY_OP_ABS: - ggml_cuda_op_abs(ctx, dst); - break; - case GGML_UNARY_OP_SGN: - ggml_cuda_op_sgn(ctx, dst); - break; - case GGML_UNARY_OP_NEG: - ggml_cuda_op_neg(ctx, dst); - break; - case GGML_UNARY_OP_STEP: - ggml_cuda_op_step(ctx, dst); - break; - case GGML_UNARY_OP_GELU: - ggml_cuda_op_gelu(ctx, dst); - break; - case GGML_UNARY_OP_SILU: - ggml_cuda_op_silu(ctx, dst); - break; - case GGML_UNARY_OP_GELU_ERF: - ggml_cuda_op_gelu_erf(ctx, dst); - break; - case GGML_UNARY_OP_GELU_QUICK: - ggml_cuda_op_gelu_quick(ctx, dst); - break; - case GGML_UNARY_OP_TANH: - ggml_cuda_op_tanh(ctx, dst); - break; - case GGML_UNARY_OP_RELU: - ggml_cuda_op_relu(ctx, dst); - break; - case GGML_UNARY_OP_SIGMOID: - ggml_cuda_op_sigmoid(ctx, dst); - break; - case GGML_UNARY_OP_HARDSIGMOID: - ggml_cuda_op_hardsigmoid(ctx, dst); - break; - case GGML_UNARY_OP_HARDSWISH: - ggml_cuda_op_hardswish(ctx, dst); - break; - case GGML_UNARY_OP_EXP: - ggml_cuda_op_exp(ctx, dst); - break; - case GGML_UNARY_OP_ELU: - ggml_cuda_op_elu(ctx, dst); - break; - case GGML_UNARY_OP_XIELU: - ggml_cuda_op_xielu(ctx, dst); - break; - case GGML_UNARY_OP_FLOOR: - ggml_cuda_op_floor(ctx, dst); - break; - case GGML_UNARY_OP_CEIL: - ggml_cuda_op_ceil(ctx, dst); - break; - case GGML_UNARY_OP_ROUND: - ggml_cuda_op_round(ctx, dst); - break; - case GGML_UNARY_OP_TRUNC: - ggml_cuda_op_trunc(ctx, dst); - break; - case GGML_UNARY_OP_EXPM1: - ggml_cuda_op_expm1(ctx, dst); - break; - case GGML_UNARY_OP_SOFTPLUS: - ggml_cuda_op_softplus(ctx, dst); - break; - default: - return false; - } - break; - case GGML_OP_GLU: - switch (ggml_get_glu_op(dst)) { - case GGML_GLU_OP_REGLU: - ggml_cuda_op_reglu(ctx, dst); - break; - case GGML_GLU_OP_GEGLU: - ggml_cuda_op_geglu(ctx, dst); - break; - case GGML_GLU_OP_SWIGLU: - ggml_cuda_op_swiglu(ctx, dst); - break; - case GGML_GLU_OP_SWIGLU_OAI: - ggml_cuda_op_swiglu_oai(ctx, dst); - break; - case GGML_GLU_OP_GEGLU_ERF: - ggml_cuda_op_geglu_erf(ctx, dst); - break; - case GGML_GLU_OP_GEGLU_QUICK: - ggml_cuda_op_geglu_quick(ctx, dst); - break; - default: - return false; - } - break; - case GGML_OP_NORM: - ggml_cuda_op_norm(ctx, dst); - break; - case GGML_OP_GROUP_NORM: - ggml_cuda_op_group_norm(ctx, dst); - break; - case GGML_OP_L2_NORM: - ggml_cuda_op_l2_norm(ctx, dst); - break; - case GGML_OP_CONCAT: - ggml_cuda_op_concat(ctx, dst); - break; - case GGML_OP_UPSCALE: - ggml_cuda_op_upscale(ctx, dst); - break; - case GGML_OP_PAD: - ggml_cuda_op_pad(ctx, dst); - break; - case GGML_OP_PAD_REFLECT_1D: - ggml_cuda_op_pad_reflect_1d(ctx, dst); - break; - case GGML_OP_ARANGE: - ggml_cuda_op_arange(ctx, dst); - break; - case GGML_OP_TIMESTEP_EMBEDDING: - ggml_cuda_op_timestep_embedding(ctx, dst); - break; - case GGML_OP_LEAKY_RELU: - ggml_cuda_op_leaky_relu(ctx, dst); - break; - case GGML_OP_SILU_BACK: - ggml_cuda_op_silu_back(ctx, dst); - break; - case GGML_OP_RMS_NORM: - ggml_cuda_op_rms_norm(ctx, dst); - break; - case GGML_OP_RMS_NORM_BACK: - ggml_cuda_op_rms_norm_back(ctx, dst); - break; - case GGML_OP_MUL_MAT: - ggml_cuda_mul_mat(ctx, dst->src[0], dst->src[1], dst); - break; - case GGML_OP_MUL_MAT_ID: - ggml_cuda_mul_mat_id(ctx, dst); - break; - case GGML_OP_OUT_PROD: - ggml_cuda_out_prod(ctx, dst); - break; - case GGML_OP_SCALE: - ggml_cuda_op_scale(ctx, dst); - break; - case GGML_OP_SQR: - ggml_cuda_op_sqr(ctx, dst); - break; - case GGML_OP_SQRT: - ggml_cuda_op_sqrt(ctx, dst); - break; - case GGML_OP_SIN: - ggml_cuda_op_sin(ctx, dst); - break; - case GGML_OP_COS: - ggml_cuda_op_cos(ctx, dst); - break; - case GGML_OP_CLAMP: - ggml_cuda_op_clamp(ctx, dst); - break; - case GGML_OP_LOG: - ggml_cuda_op_log(ctx, dst); - break; - case GGML_OP_NONE: - case GGML_OP_RESHAPE: - case GGML_OP_VIEW: - case GGML_OP_PERMUTE: - case GGML_OP_TRANSPOSE: - break; - case GGML_OP_DIAG: - ggml_cuda_op_diag(ctx, dst); - break; - case GGML_OP_DIAG_MASK_INF: - ggml_cuda_op_diag_mask_inf(ctx, dst); - break; - case GGML_OP_SOFT_MAX: - ggml_cuda_op_soft_max(ctx, dst); - break; - case GGML_OP_SOFT_MAX_BACK: - ggml_cuda_op_soft_max_back(ctx, dst); - break; - case GGML_OP_ROPE: - ggml_cuda_op_rope(ctx, dst); - break; - case GGML_OP_ROPE_BACK: - ggml_cuda_op_rope_back(ctx, dst); - break; - case GGML_OP_ROLL: - ggml_cuda_op_roll(ctx, dst); - break; - case GGML_OP_IM2COL: - ggml_cuda_op_im2col(ctx, dst); - break; - case GGML_OP_IM2COL_3D: - ggml_cuda_op_im2col_3d(ctx, dst); - break; - case GGML_OP_CONV_2D: - ggml_cuda_op_conv2d(ctx, dst); - break; - case GGML_OP_CONV_2D_DW: - ggml_cuda_op_conv2d_dw(ctx, dst); - break; - case GGML_OP_CONV_TRANSPOSE_2D: - ggml_cuda_conv_2d_transpose_p0(ctx, dst); - break; - case GGML_OP_CONV_TRANSPOSE_1D: - ggml_cuda_op_conv_transpose_1d(ctx,dst); - break; - case GGML_OP_POOL_2D: - ggml_cuda_op_pool2d(ctx, dst); - break; - case GGML_OP_SUM: - ggml_cuda_op_sum(ctx, dst); - break; - case GGML_OP_CUMSUM: - ggml_cuda_op_cumsum(ctx, dst); - break; - case GGML_OP_SUM_ROWS: - ggml_cuda_op_sum_rows(ctx, dst); - break; - case GGML_OP_MEAN: - ggml_cuda_op_mean(ctx, dst); - break; - case GGML_OP_SSM_CONV: - ggml_cuda_op_ssm_conv(ctx, dst); - break; - case GGML_OP_SSM_SCAN: - ggml_cuda_op_ssm_scan(ctx, dst); - break; - case GGML_OP_TOP_K: - ggml_cuda_op_top_k(ctx, dst); - break; - case GGML_OP_ARGSORT: - ggml_cuda_op_argsort(ctx, dst); - break; - case GGML_OP_FLASH_ATTN_EXT: - ggml_cuda_flash_attn_ext(ctx, dst); - break; - case GGML_OP_CROSS_ENTROPY_LOSS: - ggml_cuda_cross_entropy_loss(ctx, dst); - break; - case GGML_OP_TRI: - ggml_cuda_op_tri(ctx, dst); - break; - case GGML_OP_RWKV_WKV6: - ggml_cuda_op_rwkv_wkv6(ctx, dst); - break; - case GGML_OP_GATED_LINEAR_ATTN: - ggml_cuda_op_gated_linear_attn(ctx, dst); - break; - case GGML_OP_GATED_DELTA_NET: - ggml_cuda_op_gated_delta_net(ctx, dst); - break; - case GGML_OP_RWKV_WKV7: - ggml_cuda_op_rwkv_wkv7(ctx, dst); - break; - case GGML_OP_CROSS_ENTROPY_LOSS_BACK: - ggml_cuda_cross_entropy_loss_back(ctx, dst); - break; - case GGML_OP_OPT_STEP_ADAMW: - ggml_cuda_opt_step_adamw(ctx, dst); - break; - case GGML_OP_OPT_STEP_SGD: - ggml_cuda_opt_step_sgd(ctx, dst); - break; - case GGML_OP_SOLVE_TRI: - ggml_cuda_op_solve_tri(ctx, dst); - break; - case GGML_OP_FILL: - ggml_cuda_op_fill(ctx, dst); - break; - default: - return false; - } - - cudaError_t err = cudaGetLastError(); - if (err != cudaSuccess) { - GGML_LOG_ERROR("%s: %s failed\n", __func__, ggml_op_desc(dst)); - CUDA_CHECK(err); - } - - return true; -} - -//////////////////////////////////////////////////////////////////////////////// - -// backend - -static const char * ggml_backend_cuda_get_name(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - return cuda_ctx->name.c_str(); -} - -static void ggml_backend_cuda_free(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - delete cuda_ctx; - delete backend; -} - -static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpy2DAsync( - (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpy2DAsync( - data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream())); -} - -static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) { - ggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer; - ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer; - - if (!ggml_backend_is_cuda(backend_src) || !ggml_backend_is_cuda(backend_dst)) { - return false; - } - - if (!ggml_backend_buffer_is_cuda(buf_src) || !ggml_backend_buffer_is_cuda(buf_dst)) { - return false; - } - - // device -> device copy - ggml_backend_cuda_context * cuda_ctx_src = (ggml_backend_cuda_context *) backend_src->context; - ggml_backend_cuda_context * cuda_ctx_dst = (ggml_backend_cuda_context *) backend_dst->context; - - ggml_backend_cuda_buffer_context * buf_ctx_src = (ggml_backend_cuda_buffer_context *) buf_src->context; - ggml_backend_cuda_buffer_context * buf_ctx_dst = (ggml_backend_cuda_buffer_context *) buf_dst->context; - - if (cuda_ctx_src->device != buf_ctx_src->device || cuda_ctx_dst->device != buf_ctx_dst->device) { -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: backend and buffer devices do not match\n", __func__); -#endif // NDEBUG - return false; - } - - if (backend_src != backend_dst) { - // copy on src stream - if (cuda_ctx_src->device == cuda_ctx_dst->device) { - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); - } else { -#ifdef GGML_CUDA_NO_PEER_COPY - return false; -#else - CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream())); -#endif // GGML_CUDA_NO_PEER_COPY - } - - // record event on src stream after the copy - if (!cuda_ctx_src->copy_event) { - ggml_cuda_set_device(cuda_ctx_src->device); - CUDA_CHECK(cudaEventCreateWithFlags(&cuda_ctx_src->copy_event, cudaEventDisableTiming)); - } - - CUDA_CHECK(cudaEventRecord(cuda_ctx_src->copy_event, cuda_ctx_src->stream())); - - // wait on dst stream for the copy to complete - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx_dst->stream(), cuda_ctx_src->copy_event, 0)); - } else { - // src and dst are on the same backend - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); - } - return true; -} - -static void ggml_backend_cuda_synchronize(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream())); - - GGML_UNUSED(backend); -} - -#ifdef USE_CUDA_GRAPH -static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) { - - bool use_cuda_graph = true; - // Loop over nodes in GGML graph to obtain info needed for CUDA graph - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_tensor * node = cgraph->nodes[i]; - - if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { - continue; - } - - if (node->src[0] && node->src[0]->buffer && ggml_backend_buft_is_cuda_split(node->src[0]->buffer->buft)) { - use_cuda_graph = false; // Split buffers are not supported by CUDA graph capture -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to split buffer\n", __func__); -#endif - } - - // [TAG_MUL_MAT_ID_CUDA_GRAPHS] - if (node->op == GGML_OP_MUL_MAT_ID) { - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - const int mmvq_mmid_max = get_mmvq_mmid_max_batch(node->src[0]->type, cc); - if (!ggml_is_quantized(node->src[0]->type) || node->ne[2] > mmvq_mmid_max) { - // under these conditions, the mul_mat_id operation will need to synchronize the stream, so we cannot use CUDA graphs - // TODO: figure out a way to enable for larger batch sizes, without hurting performance - // ref: https://github.com/ggml-org/llama.cpp/pull/18958 - use_cuda_graph = false; -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to unsupported node type\n", __func__); -#endif - } - } - - if (!use_cuda_graph) { - break; - } - } - - return use_cuda_graph; -} - -static const void * ggml_cuda_graph_get_key(ggml_cgraph * cgraph) { - return cgraph->nodes[0]; -} - -static bool ggml_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph) { - bool res = false; - - const void * graph_key = ggml_cuda_graph_get_key(cgraph); - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - - if (cgraph->uid != 0 && - cgraph->uid == graph->uid) { - GGML_LOG_DEBUG("CUDA Graph id %zu reused\n", cgraph->uid); - GGML_ASSERT((int)graph->node_props.size() == cgraph->n_nodes); - return false; - } - - graph->uid = cgraph->uid; - - // Check if the graph size has changed - if ((int)graph->node_props.size() != cgraph->n_nodes) { - res = true; - graph->node_props.resize(cgraph->n_nodes); - } - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_cuda_graph::node_properties prop = {}; - memcpy(&prop.node, cgraph->nodes[i], sizeof(ggml_tensor)); - - for (int j = 0; j < GGML_MAX_SRC; ++j) { - if (cgraph->nodes[i]->src[j]) { - prop.node_src_data_ptrs[j] = cgraph->nodes[i]->src[j]->data; - memcpy(prop.node_src_ne[j], cgraph->nodes[i]->src[j]->ne, sizeof(prop.node_src_ne[j])); - memcpy(prop.node_src_nb[j], cgraph->nodes[i]->src[j]->nb, sizeof(prop.node_src_nb[j])); - } - } - - if (res || memcmp(&graph->node_props[i], &prop, sizeof(prop)) != 0) { - graph->node_props[i] = prop; - res = true; - } - } - - return res; -} - -static void ggml_cuda_graph_update_executable(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - -#if CUDART_VERSION >= 12000 - cudaGraphExecUpdateResultInfo result_info; - cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &result_info); -#else - cudaGraphNode_t errorNode; - cudaGraphExecUpdateResult result_info; - cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &errorNode, &result_info); -#endif // CUDART_VERSION >= 12000 - - if (stat == cudaErrorGraphExecUpdateFailure) { -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: CUDA graph update failed\n", __func__); -#endif - - // The pre-existing graph exec cannot be updated due to violated constraints - // so instead clear error and re-instantiate - (void)cudaGetLastError(); - CUDA_CHECK(cudaGraphExecDestroy(graph->instance)); - graph->instance = nullptr; - CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); - } else { - GGML_ASSERT(stat == cudaSuccess); - } -} -#endif // USE_CUDA_GRAPH - -static bool ggml_cuda_should_fuse_rope_set_rows(const ggml_tensor * rope, - const ggml_tensor * view, - const ggml_tensor * set_rows) { - - if (rope->op != GGML_OP_ROPE || view->op != GGML_OP_VIEW || set_rows->op != GGML_OP_SET_ROWS) { - return false; - } - // ne3 not tested - if (rope->src[0]->ne[3] != 1) { - return false; - } - - if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16) { - return false; - } - - if (set_rows->src[1]->type != GGML_TYPE_I64) { - return false; - } - - // The view should flatten two dims of rope into one dim - if (!ggml_is_contiguous(view) || view->ne[0] != rope->ne[0] * rope->ne[1]) { - return false; - } - - // Only norm/neox shaders have the fusion code - const int mode = ((const int32_t *) rope->op_params)[2]; - if (mode != GGML_ROPE_TYPE_NORMAL && mode != GGML_ROPE_TYPE_NEOX) { - return false; - } - - return true; -} - -static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) { - args.sigmoid = false; - args.softmax = false; - args.delayed_softmax = false; - args.prob_bias = false; - args.norm = false; - - const int n_nodes = cgraph->n_nodes; - ggml_tensor ** nodes = cgraph->nodes; - - if (nodes[node_idx]->op == GGML_OP_SOFT_MAX) { - args.softmax = true; - } - - if (nodes[node_idx]->op == GGML_OP_UNARY) { - if (ggml_get_unary_op(nodes[node_idx]) != GGML_UNARY_OP_SIGMOID) { - return false; - } - args.sigmoid = true; - } - - if (nodes[node_idx]->op == GGML_OP_ARGSORT) { - args.delayed_softmax = true; - } - - node_idx++; - - if (args.sigmoid || args.softmax) { - // SOFTMAX -> RESHAPE - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - ggml_tensor * probs_reshaped = nodes[node_idx]; - node_idx++; - - if (node_idx >= n_nodes) { - return false; - } - - // src of bias add is the unreshaped probs (-2 instead of -1) - if (nodes[node_idx]->op == GGML_OP_ADD && nodes[node_idx]->src[0] == nodes[node_idx - 2]) { - args.prob_bias = true; - node_idx++; - } - // RESHAPE/ADD -> ARGSORT - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_ARGSORT) { - return false; - } - - if (args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } else if (!args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 2]) { - return false; - } - - node_idx++; - - // ARGSORT-> VIEW - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_GET_ROWS) { - return false; - } - - // GET_ROWS - if (nodes[node_idx]->src[0] != probs_reshaped || nodes[node_idx]->src[1] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - } else if (args.delayed_softmax) { - if (node_idx - 2 < 0) { - return false; - } - ggml_tensor * probs_reshaped = nodes[node_idx - 2]; - - // VIEW->ARGSORT - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - - // GET_ROWS - if (node_idx >= n_nodes || nodes[node_idx]->src[1] != nodes[node_idx - 1] || - nodes[node_idx]->src[0] != probs_reshaped) { - return false; - } - node_idx++; - - static const std::vector remaining_ops = { GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }; - - for (const ggml_op op : remaining_ops) { - if (node_idx >= n_nodes || nodes[node_idx]->op != op || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - } - } - - // At this point we can check for norm + scale. Everything is now at least valid till the norm - if (node_idx >= n_nodes) { - return true; - } - - if (nodes[node_idx]->op == GGML_OP_RESHAPE) { - //check RESHAPE->SUM_ROWS->CLAMP->DIV->RESHAPE - static const std::vector norm_ops = { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP }; - - args.norm = true; - for (const ggml_op op : norm_ops) { - if (nodes[node_idx]->op == op && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { - node_idx++; - } else { - args.norm = false; - return true; - } - } - - // DIV <- CLAMP, RESHAPE - if (nodes[node_idx]->op != GGML_OP_DIV || nodes[node_idx]->src[1] != nodes[node_idx - 1] || - nodes[node_idx]->src[0] != nodes[node_idx - 3]) { - args.norm = false; - return true; - } - node_idx++; - - if (nodes[node_idx]->op != GGML_OP_RESHAPE || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - args.norm = false; - return true; - } - - node_idx++; - } - - if (nodes[node_idx]->op == GGML_OP_SCALE && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { - args.scale = true; - } - - return true; -} - -// returns whether the write (out) nodes overwrite the read nodes in operation -static bool ggml_cuda_check_fusion_memory_ranges(const ggml_cgraph * cgraph, - const int node_idx, - const int node_count, - const int * out_nodes, - const int out_count, - const bool is_topk_moe = false) { - auto nodes_overlap = [&](const ggml_tensor * a, const ggml_tensor * b) { - const int64_t a_start = (int64_t) a->data; - const int64_t a_end = a_start + ggml_backend_buft_get_alloc_size(a->buffer->buft, a); - - const int64_t b_start = (int64_t) b->data; - const int64_t b_end = b_start + ggml_backend_buft_get_alloc_size(b->buffer->buft, b); - - if ((b_start <= a_start && a_start < b_end) || (a_start <= b_start && b_start < a_end)) { - return true; - } - - return false; - }; - - bool is_ok = true; - // exception for topk-moe, as each row is read entirely before writing - if (ggml_nrows(cgraph->nodes[node_idx]) == 1 && is_topk_moe) { - return true; - } - - for (int i = 0; i < out_count; ++i) { - const ggml_tensor * dst = cgraph->nodes[out_nodes[i]]; - - for (int j = node_idx; j < node_idx + node_count; ++j) { - // Loop over all srcs of all nodes in the fusion. If the src overlaps - // the destination and the src is not an intermediate node that's being - // elided, then disable fusion. - - for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { - const ggml_tensor * src = cgraph->nodes[j]->src[src_idx]; - - if (!src || src->op == GGML_OP_NONE) { - continue; - } - - if (nodes_overlap(dst, src)) { - bool found = false; - - for (int k = node_idx; k < j; ++k) { - if (cgraph->nodes[k] == src) { - found = true; - break; - } - } - - if (!found) { - is_ok = false; - break; - } - } - } - } - } - - return is_ok; -} - - -static bool ggml_cuda_can_fuse(const struct ggml_cgraph * cgraph, - int node_idx, - std::initializer_list ops, - std::initializer_list unary_ops) { -#ifndef NDEBUG - const size_t num_unary = std::count(ops.begin(), ops.end(), GGML_OP_UNARY); - GGML_ASSERT(unary_ops.size() == num_unary); -#endif - - const auto is_equal = [](const std::initializer_list & list1, - const std::initializer_list & list2) { - return std::equal(list1.begin(), list1.end(), list2.begin(), list2.end()); - }; - - std::initializer_list mul_mat_bias_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_GLU }; - std::initializer_list mul_mat_id_bias_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_GLU }; - - std::initializer_list mul_mat_id_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_MUL_MAT_ID, GGML_OP_GLU }; - std::initializer_list mul_mat_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT, GGML_OP_GLU }; - - if ((is_equal(mul_mat_bias_glu_ops, ops) || is_equal(mul_mat_id_bias_glu_ops, ops)) && - ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 4 })) { - const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; - const ggml_tensor * ffn_gate_bias = cgraph->nodes[node_idx + 1]; - const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 2]; - const ggml_tensor * ffn_up_bias = cgraph->nodes[node_idx + 3]; - const ggml_tensor * glu = cgraph->nodes[node_idx + 4]; - - if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu, ffn_up_bias, ffn_gate_bias)) { - int out_nodes[] = { node_idx + 4 }; - return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); - } - } - - if ((is_equal(mul_mat_id_glu_ops, ops) || is_equal(mul_mat_glu_ops, ops)) && - ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { - const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; - const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 1]; - const ggml_tensor * glu = cgraph->nodes[node_idx + 2]; - - if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu)) { - int out_nodes[] = { node_idx + 2 }; - return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); - } - } - - std::initializer_list rope_set_rows_ops = { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }; - - if (is_equal(rope_set_rows_ops, ops) && ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { - const ggml_tensor * rope = cgraph->nodes[node_idx]; - const ggml_tensor * view = cgraph->nodes[node_idx + 1]; - const ggml_tensor * set_rows = cgraph->nodes[node_idx + 2]; - - if (ggml_cuda_should_fuse_rope_set_rows(rope, view, set_rows)) { - return true; - } - } - - if (!ggml_can_fuse(cgraph, node_idx, ops)) { - return false; - } - - if ((ops.size() == 2 || ops.size() == 3) && ops.begin()[0] == GGML_OP_RMS_NORM && ops.begin()[1] == GGML_OP_MUL) { - const ggml_tensor *rms_norm = cgraph->nodes[node_idx]; - const ggml_tensor *mul = cgraph->nodes[node_idx+1]; - const ggml_tensor *add = nullptr; - - if (ops.size() == 3 && ops.begin()[2] == GGML_OP_ADD) { - add = cgraph->nodes[node_idx+2]; - } - - GGML_ASSERT(rms_norm->src[0]->type == GGML_TYPE_F32); - GGML_ASSERT(rms_norm->type == GGML_TYPE_F32); - - //rms norm only supports F32 - if (mul->src[0]->type != GGML_TYPE_F32 || - mul->src[1]->type != GGML_TYPE_F32 || - mul->type != GGML_TYPE_F32) { - return false; - } - - if (add && (add->src[0]->type != GGML_TYPE_F32 || - add->src[1]->type != GGML_TYPE_F32 || - add->type != GGML_TYPE_F32) ) { - return false; - } - - //if rms norm is the B operand, then we don't handle broadcast - if (rms_norm == mul->src[1] && !ggml_are_same_shape(mul->src[0], rms_norm)) { - return false; - } - - //rms_norm kernel assumes contiguous rows - if (!ggml_is_contiguous_rows(mul->src[0]) || !ggml_is_contiguous_rows(mul->src[1])) { - return false; - } - - if (add && (!ggml_is_contiguous(add->src[0]) || !ggml_is_contiguous_rows(add->src[1]))) { - return false; - } - - return true; - } - - if (ops.size() == 2 && ops.begin()[0] == GGML_OP_SSM_CONV && ops.begin()[1] == GGML_OP_UNARY - && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_SILU) { - const ggml_tensor * ssm_conv = cgraph->nodes[node_idx]; - const ggml_tensor * silu = cgraph->nodes[node_idx+1]; - - if (ssm_conv->type != GGML_TYPE_F32 || silu->type != GGML_TYPE_F32) { - return false; - } - - return true; - } - - if (ops.size() == 2 && ops.begin()[0] == GGML_OP_UNARY && ops.begin()[1] == GGML_OP_MUL - && unary_ops.size() == 1 && (unary_ops.begin()[0] == GGML_UNARY_OP_SILU || unary_ops.begin()[0] == GGML_UNARY_OP_SIGMOID || unary_ops.begin()[0] == GGML_UNARY_OP_SOFTPLUS)) { - const ggml_tensor * unary = cgraph->nodes[node_idx]; - const ggml_tensor * mul = cgraph->nodes[node_idx+1]; - - if (ggml_get_unary_op(unary) != unary_ops.begin()[0]) { - return false; - } - - if (unary->type != GGML_TYPE_F32 && unary->type != GGML_TYPE_F16) { - return false; - } - - if (unary->type != mul->type) { - return false; - } - - const ggml_tensor * other = (mul->src[0] == unary) ? mul->src[1] : mul->src[0]; - if (other->type != unary->type) { - return false; - } - if (!ggml_is_contiguous_1(other) || !ggml_is_contiguous_1(unary->src[0]) || !ggml_are_same_shape(other, unary)) { - return false; - } - - return true; - } - - if (ops.size() == 3 && ops.begin()[0] == GGML_OP_SCALE && ops.begin()[1] == GGML_OP_UNARY && ops.begin()[2] == GGML_OP_SCALE - && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_TANH) { - const ggml_tensor *scale = cgraph->nodes[node_idx]; - const ggml_tensor *tanh = cgraph->nodes[node_idx+1]; - const ggml_tensor *scale2 = cgraph->nodes[node_idx+2]; - - GGML_ASSERT(scale->src[0]->type == GGML_TYPE_F32); - GGML_ASSERT(scale->type == GGML_TYPE_F32); - - if (ggml_get_unary_op(tanh) != GGML_UNARY_OP_TANH) { - return false; - } - - // Check for bias - if (ggml_get_op_params_f32(scale, 1) != 0.0f || ggml_get_op_params_f32(scale2, 1) != 0.0f) { - return false; - } - - return true; - } - - return false; -} - -static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph, const bool use_cuda_graph, const bool cuda_graph_update_required, const void * graph_key) { - bool graph_evaluated_or_captured = false; - - // flag used to determine whether it is an integrated_gpu - const bool integrated = ggml_cuda_info().devices[cuda_ctx->device].integrated; - - ggml_cuda_stream_context & stream_ctx = cuda_ctx->stream_context(); - bool is_concurrent_event_active = false; - ggml_cuda_concurrent_event * concurrent_event = nullptr; - bool should_launch_concurrent_events = false; - - const auto try_launch_concurrent_event = [&](const ggml_tensor * node) { - if (stream_ctx.concurrent_events.find(node) != stream_ctx.concurrent_events.end()) { - concurrent_event = &stream_ctx.concurrent_events[node]; - - is_concurrent_event_active = true; - - GGML_LOG_DEBUG("Launching %d streams at %s\n", concurrent_event->n_streams, node->name); - - cudaStream_t main_stream = cuda_ctx->stream(); // this should be stream 0 - GGML_ASSERT(cuda_ctx->curr_stream_no == 0); - CUDA_CHECK(cudaEventRecord(concurrent_event->fork_event, main_stream)); - - for (int i = 1; i <= concurrent_event->n_streams; ++i) { - cudaStream_t stream = cuda_ctx->stream(cuda_ctx->device, i); - CUDA_CHECK(cudaStreamWaitEvent(stream, concurrent_event->fork_event)); - } - } - }; - - while (!graph_evaluated_or_captured) { - // Only perform the graph execution if CUDA graphs are not enabled, or we are capturing the graph. - // With the use of CUDA graphs, the execution will be performed by the graph launch. - if (!use_cuda_graph || cuda_graph_update_required) { - [[maybe_unused]] int prev_i = 0; - - if (stream_ctx.concurrent_events.size() > 0) { - should_launch_concurrent_events = true; - for (const auto & [tensor, event] : stream_ctx.concurrent_events) { - should_launch_concurrent_events = should_launch_concurrent_events && event.is_valid(); - } - } - - if (should_launch_concurrent_events) { - // Restore original node order within each concurrent region to enable fusion within streams - - std::unordered_map node_to_idx; - node_to_idx.reserve(cgraph->n_nodes); - for (int i = 0; i < cgraph->n_nodes; ++i) { - node_to_idx[cgraph->nodes[i]] = i; - } - - for (auto & [fork_node, event] : stream_ctx.concurrent_events) { - // Find positions of all nodes from this event in the current graph - std::vector positions; - positions.reserve(event.original_order.size()); - - bool all_found = true; - for (const ggml_tensor * orig_node : event.original_order) { - auto it = node_to_idx.find(orig_node); - if (it != node_to_idx.end()) { - positions.push_back(it->second); - } else { - all_found = false; - break; - } - } - - if (!all_found || positions.size() != event.original_order.size()) { - continue; - } - - // Sort positions to get contiguous range - std::vector sorted_positions = positions; - std::sort(sorted_positions.begin(), sorted_positions.end()); - - bool is_contiguous = true; - for (size_t i = 1; i < sorted_positions.size(); ++i) { - if (sorted_positions[i] != sorted_positions[i-1] + 1) { - is_contiguous = false; - break; - } - } - - if (!is_contiguous) { - continue; - } - - // Restore original order at the sorted positions - int start_pos = sorted_positions[0]; - for (size_t i = 0; i < event.original_order.size(); ++i) { - cgraph->nodes[start_pos + i] = const_cast(event.original_order[i]); - } - } - } else { - stream_ctx.concurrent_events.clear(); - } - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_tensor * node = cgraph->nodes[i]; - if (is_concurrent_event_active) { - GGML_ASSERT(concurrent_event); - - if (node == concurrent_event->join_node) { - cuda_ctx->curr_stream_no = 0; - for (int i = 1; i <= concurrent_event->n_streams; ++i) { - // Wait on join events of forked streams in the main stream - CUDA_CHECK(cudaEventRecord(concurrent_event->join_events[i - 1], - cuda_ctx->stream(cuda_ctx->device, i))); - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), concurrent_event->join_events[i - 1])); - } - - is_concurrent_event_active = false; - concurrent_event = nullptr; - } else { - GGML_ASSERT (concurrent_event->stream_mapping.find(node) != concurrent_event->stream_mapping.end()); - cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; - GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); - } - } else if (i - prev_i > 1) { - //the previous node was fused - const ggml_tensor * prev_node = cgraph->nodes[i - 1]; - try_launch_concurrent_event(prev_node); - - if (is_concurrent_event_active) { - cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; - GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); - } - } - -#ifdef GGML_CUDA_DEBUG - const int nodes_fused = i - prev_i - 1; - if (nodes_fused > 0) { - GGML_LOG_INFO("nodes_fused: %d\n", nodes_fused); - } -#endif - prev_i = i; - - if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { - continue; - } - - if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { - continue; - } - - // start of fusion operations - static bool disable_fusion = (getenv("GGML_CUDA_DISABLE_FUSION") != nullptr); - if (!disable_fusion) { - ggml_cuda_topk_moe_args args; - - if (cgraph->nodes[i]->op == GGML_OP_UNARY || cgraph->nodes[i]->op == GGML_OP_SOFT_MAX || - cgraph->nodes[i]->op == GGML_OP_ARGSORT) { - const bool can_fuse = ggml_cuda_topk_moe_fusion(cgraph, i, args); - - std::vector ops; - - if (can_fuse) { - const ggml_tensor * logits = node->src[0]; - ggml_tensor * weights = nullptr; - ggml_tensor * ids = nullptr; - const ggml_tensor * bias = nullptr; - const ggml_tensor * clamp = nullptr; - const ggml_tensor * scale = nullptr; - - if (!args.delayed_softmax) { - ggml_op gating_op = args.sigmoid ? GGML_OP_UNARY : GGML_OP_SOFT_MAX; - int out_nodes[2]; // nodes which can't be elided - - if (args.prob_bias) { - bias = cgraph->nodes[i + 2]->src[1]; - ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, - GGML_OP_VIEW, GGML_OP_GET_ROWS }); - out_nodes[0] = i + 4; - ids = cgraph->nodes[i + 4]; - } else { - ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, - GGML_OP_GET_ROWS }); - out_nodes[0] = i + 3; - ids = cgraph->nodes[i + 3]; - } - - if (args.norm) { - ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP, - GGML_OP_DIV, GGML_OP_RESHAPE }); - clamp = cgraph->nodes[i + ops.size() - 3]; - } - if (args.scale) { - ops.insert(ops.end(), { GGML_OP_SCALE }); - scale = cgraph->nodes[i + ops.size() - 1]; - } - - weights = cgraph->nodes[i + ops.size() - 1]; - out_nodes[1] = i + ops.size() - 1; - - if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && - ggml_cuda_should_use_topk_moe(node, logits, weights, ids) && - ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { - ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); - i += ops.size() - 1; - continue; - } - } else if (!args.norm && !args.prob_bias) { - //special case gpt-oss, no norm, no bias. - ops.insert(ops.end(), { GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS, - GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }); - weights = cgraph->nodes[i + 5]; - ids = cgraph->nodes[i + 1]; - const ggml_tensor * softmax = cgraph->nodes[i + 4]; - - int out_nodes[2] = { i + 1, i + 5 }; - if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && - ggml_cuda_should_use_topk_moe(softmax, logits, weights, ids) && - ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { - ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); - i += ops.size() - 1; - continue; - } - } - } - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }, {})) { - ggml_tensor * rope = cgraph->nodes[i]; - ggml_tensor * set_rows = cgraph->nodes[i + 2]; - - ggml_cuda_op_rope_fused(*cuda_ctx, rope, set_rows); - i += 2; - continue; - } - - if (node->op == GGML_OP_ADD || node->op == GGML_OP_MUL) { - int n_fuse = 0; - ggml_op ops[8]; - std::fill(ops, ops + 8, node->op); - - for (; n_fuse <= 6; ++n_fuse){ - if (!ggml_can_fuse(cgraph, i + n_fuse, ops + n_fuse, 2)) { - break; - } - if (cgraph->nodes[i + n_fuse] != cgraph->nodes[i + n_fuse + 1]->src[0]) { - break; - } - if (!ggml_are_same_layout(cgraph->nodes[i + n_fuse]->src[1], cgraph->nodes[i + n_fuse + 1]->src[1])) { - break; - } - } - - n_fuse++; - - if (n_fuse > 1) { - ggml_tensor fused_node; - memcpy(&fused_node, node, sizeof(ggml_tensor)); - for (int j = 0; j < n_fuse - 1; ++j) { - fused_node.src[j + 2] = cgraph->nodes[i + j + 1]->src[1]; - } - fused_node.data = cgraph->nodes[i + n_fuse - 1]->data; - if (node->op == GGML_OP_ADD) { - ggml_cuda_op_fused_add(*cuda_ctx, &fused_node, n_fuse); - } else { - ggml_cuda_op_fused_mul(*cuda_ctx, &fused_node, n_fuse); - } - i += n_fuse - 1; - - continue; - } - } - - bool fused_mul_mat_vec = false; - int fused_node_count = 0; - - for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { - const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (ggml_cuda_can_fuse(cgraph, i, { op, bias_op, op, bias_op, GGML_OP_GLU }, {})) { - ggml_tensor * glu = cgraph->nodes[i + 4]; - ggml_tensor * gate_bias_n = glu->src[0]; - ggml_tensor * up_bias_n = glu->src[1]; - - //we don't assume the order for {gate, up}. Instead infer it from the bias tensor - ggml_tensor * gate_n = nullptr; - ggml_tensor * up_n = nullptr; - - if (gate_bias_n->src[0] == cgraph->nodes[i] || gate_bias_n->src[1] == cgraph->nodes[i]) { - gate_n = cgraph->nodes[i]; - up_n = cgraph->nodes[i + 2]; - } else if (gate_bias_n->src[0] == cgraph->nodes[i + 2] || gate_bias_n->src[1] == cgraph->nodes[i + 2]) { - gate_n = cgraph->nodes[i + 2]; - up_n = cgraph->nodes[i]; - } else { - continue; - } - - auto get_bias_tensor = [](const ggml_tensor * bias_node, const ggml_tensor * mul_node, ggml_op op_bias) { - if (op_bias == GGML_OP_ADD) { - if (bias_node->src[0] == mul_node) { - return bias_node->src[1]; - } - if (bias_node->src[1] == mul_node) { - return bias_node->src[0]; - } - return (ggml_tensor *) nullptr; - } - GGML_ASSERT(op_bias == GGML_OP_ADD_ID); - GGML_ASSERT(bias_node->src[0] == mul_node); - return bias_node->src[1]; - }; - - ggml_tensor * up_bias_tensor = get_bias_tensor(up_bias_n, up_n, bias_op); - ggml_tensor * gate_bias_tensor = get_bias_tensor(gate_bias_n, gate_n, bias_op); - - if (!up_bias_tensor || !gate_bias_tensor) { - continue; - } - - // we don't support repeating adds - if (bias_op == GGML_OP_ADD && - (!ggml_are_same_shape(gate_bias_n->src[0], gate_bias_n->src[1]) || - !ggml_are_same_shape(up_bias_n->src[0], up_bias_n->src[1]))) { - continue; - } - - const ggml_tensor * src0 = up_n->src[0]; - const ggml_tensor * src1 = up_n->src[1]; - const ggml_tensor * ids = up_n->src[2]; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(up_n)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate_n->src[0]; - fusion_data.x_bias = up_bias_tensor; - fusion_data.gate_bias = gate_bias_tensor; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 5; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(up_n)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate_n->src[0]; - fusion_data.x_bias = up_bias_tensor; - fusion_data.gate_bias = gate_bias_tensor; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 5; - break; - } - } else if (ggml_cuda_can_fuse(cgraph, i, { op, op, GGML_OP_GLU }, {})) { - ggml_tensor * glu = cgraph->nodes[i + 2]; - ggml_tensor * gate = glu->src[0]; - ggml_tensor * up = glu->src[1]; - - bool ok = (gate == cgraph->nodes[i] && up == cgraph->nodes[i + 1]) - || (gate == cgraph->nodes[i + 1] && up == cgraph->nodes[i]); - - if (!ok) continue; - - const ggml_tensor * src0 = up->src[0]; - const ggml_tensor * src1 = up->src[1]; - const ggml_tensor * ids = up->src[2]; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(up)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate->src[0]; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 3; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(up)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate->src[0]; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 3; - break; - } - } - } - - if (fused_mul_mat_vec) { - i += fused_node_count - 1; - continue; - } - - fused_mul_mat_vec = false; - fused_node_count = 0; - - for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { - const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (!ggml_can_fuse(cgraph, i, { op, bias_op })) { - continue; - } - - ggml_tensor * mm_node = cgraph->nodes[i]; - ggml_tensor * bias_node = cgraph->nodes[i + 1]; - - ggml_tensor * bias_tensor = nullptr; - if (bias_op == GGML_OP_ADD) { - if (bias_node->src[0] == mm_node) { - bias_tensor = bias_node->src[1]; - } else if (bias_node->src[1] == mm_node) { - bias_tensor = bias_node->src[0]; - } else { - continue; - } - } else { - if (bias_node->src[0] != mm_node) { - continue; - } - bias_tensor = bias_node->src[1]; - } - - const ggml_tensor * src0 = mm_node->src[0]; - const ggml_tensor * src1 = mm_node->src[1]; - const ggml_tensor * ids = mm_node->src[2]; - - if (bias_op == GGML_OP_ADD_ID && bias_node->src[2] != ids) { - continue; - } - - if (bias_op == GGML_OP_ADD && !ggml_are_same_shape(bias_node->src[0], bias_node->src[1])) { - continue; - } - - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.x_bias = bias_tensor; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(mm_node)) { - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 2; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(mm_node)) { - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 2; - break; - } - } - - if (fused_mul_mat_vec) { - i += fused_node_count - 1; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL, GGML_OP_ADD}, {})) { - ggml_cuda_op_rms_norm_fused_add(*cuda_ctx, node, cgraph->nodes[i+1], cgraph->nodes[i+2]); - i += 2; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL}, {})) { - ggml_cuda_op_rms_norm_fused(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SSM_CONV, GGML_OP_UNARY }, { GGML_UNARY_OP_SILU })) { - ggml_cuda_op_ssm_conv(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SILU }) || - ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SIGMOID }) || - ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SOFTPLUS })) { - ggml_cuda_op_unary_mul(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SCALE, GGML_OP_UNARY, GGML_OP_SCALE }, { GGML_UNARY_OP_TANH })) { - i += 2; - ggml_cuda_op_softcap(*cuda_ctx, cgraph->nodes[i], node); - continue; - } - } -#ifndef NDEBUG - assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device)); - for (int j = 0; j < GGML_MAX_SRC; j++) { - if (node->src[j] != nullptr) { - assert(node->src[j]->buffer); - assert(node->src[j]->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) || - ggml_backend_buft_is_cuda_split(node->src[j]->buffer->buft) || (integrated && ggml_backend_buft_is_cuda_host(node->src[j]->buffer->buft))); - } - } -#else - GGML_UNUSED(integrated); -#endif // NDEBUG - - bool ok = ggml_cuda_compute_forward(*cuda_ctx, node); - if (!ok) { - GGML_LOG_ERROR("%s: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op)); - } - GGML_ASSERT(ok); - - if (!is_concurrent_event_active) { - try_launch_concurrent_event(node); - } - } - } - -#ifdef USE_CUDA_GRAPH - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (use_cuda_graph && cuda_graph_update_required) { // End CUDA graph capture - if (graph->graph != nullptr) { - CUDA_CHECK(cudaGraphDestroy(graph->graph)); - graph->graph = nullptr; - } - - CUDA_CHECK(cudaStreamEndCapture(cuda_ctx->stream(), &graph->graph)); - graph_evaluated_or_captured = true; // CUDA graph has been captured - - std::lock_guard lock(ggml_cuda_lock); - if (ggml_cuda_lock_counter.fetch_sub(1, std::memory_order_relaxed) == 1) { - ggml_cuda_lock_cv.notify_all(); - } - } else { - graph_evaluated_or_captured = true; // ggml graph has been directly evaluated - } - } - - if (use_cuda_graph) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (graph->instance == nullptr) { // Create executable graph from captured graph. - CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); - } - if (cuda_graph_update_required) { // Update graph executable - ggml_cuda_graph_update_executable(cuda_ctx, graph_key); - } - // Launch graph - CUDA_CHECK(cudaGraphLaunch(graph->instance, cuda_ctx->stream())); -#else - GGML_UNUSED(graph_key); - graph_evaluated_or_captured = true; -#endif // USE_CUDA_GRAPH - } -} - -#ifdef USE_CUDA_GRAPH -static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - - if (graph->graph == nullptr) { - if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_AMPERE) { - if (!graph->disable_due_to_gpu_arch) { - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__); - } - graph->disable_due_to_gpu_arch = true; - } - } - - return graph->is_enabled(); -} -#endif // USE_CUDA_GRAPH - -static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend, ggml_cgraph * cgraph) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - - ggml_cuda_set_device(cuda_ctx->device); - - bool use_cuda_graph = false; - bool cuda_graph_update_required = false; - const void * graph_key = nullptr; - -#ifdef USE_CUDA_GRAPH - graph_key = ggml_cuda_graph_get_key(cgraph); - - ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); - - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (graph->is_enabled()) { - const bool graph_compatible = ggml_cuda_graph_check_compability(cgraph); - if (graph_compatible) { - const bool properties_changed = ggml_cuda_graph_update_required(cuda_ctx, cgraph); - - if (!graph->warmup_complete) { - // Warmup: need at least 2 calls with no property change on the 2nd call - if (!properties_changed) { - graph->warmup_complete = true; - GGML_LOG_DEBUG("%s: CUDA graph warmup complete\n", __func__); - use_cuda_graph = true; - cuda_graph_update_required = true; - } - // else: properties changed or first call - execute directly (use_cuda_graph stays false) - } else { - // Post-warmup: normal CUDA graph operation - if (properties_changed) { - // Properties changed - reset warmup, execute directly until stable again - graph->warmup_complete = false; - GGML_LOG_DEBUG("%s: CUDA graph warmup reset\n", __func__); - } else { - use_cuda_graph = true; - cuda_graph_update_required = graph->instance == nullptr; - } - } - } - } -#endif // USE_CUDA_GRAPH - - if (use_cuda_graph && cuda_graph_update_required) { - // Start CUDA graph capture - { - std::lock_guard lock(ggml_cuda_lock); - ggml_cuda_lock_counter.fetch_add(1, std::memory_order_relaxed); - } - - CUDA_CHECK(cudaStreamBeginCapture(cuda_ctx->stream(), cudaStreamCaptureModeRelaxed)); - } - - ggml_cuda_graph_evaluate_and_capture(cuda_ctx, cgraph, use_cuda_graph, cuda_graph_update_required, graph_key); - - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_event_record(ggml_backend_t backend, ggml_backend_event_t event) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - CUDA_CHECK(cudaEventRecord((cudaEvent_t)event->context, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_event_t event) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - if (ggml_backend_is_cuda(backend)) { - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), (cudaEvent_t)event->context, 0)); - } else { -#if 0 - // untested - auto wait_fn = [](void * user_data) { - ggml_backend_event_t event = (ggml_backend_event_t)user_data; - ggml_backend_event_synchronize(event); - }; - - CUDA_CHECK(cudaLaunchHostFunc(cuda_ctx->stream(), wait_fn, event)); -#endif - GGML_ABORT("fatal error"); - } -} - -static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - -#ifdef USE_CUDA_GRAPH - const void * graph_key = ggml_cuda_graph_get_key(cgraph); - const bool use_cuda_graph = ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); -#else - const bool use_cuda_graph = false; - GGML_UNUSED(cuda_ctx); - GGML_UNUSED(cgraph); -#endif - - static bool enable_graph_optimization = [] { - const char * env = getenv("GGML_CUDA_GRAPH_OPT"); - return env != nullptr && atoi(env) == 1; - }(); - - if (!enable_graph_optimization) { - return; - } - - ggml_cuda_stream_context & stream_context = cuda_ctx->stream_context(); - stream_context.reset(); - - if (!use_cuda_graph || ggml_backend_cuda_get_device_count() != 1) { - return; - } - - // number of out-degrees for a particular node - std::unordered_map fan_out; - // reverse mapping of node to index in the cgraph - std::unordered_map node_indices; - - const auto & is_noop = [](const ggml_tensor * node) -> bool { - return ggml_is_empty(node) || node->op == GGML_OP_NONE || node->op == GGML_OP_RESHAPE || - node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE; - }; - - const auto & depends_on = [](const ggml_tensor * dst, const ggml_tensor * src) -> bool { - for (uint32_t s = 0; s < GGML_MAX_SRC; ++s) { - if (dst->src[s] == src) { - return true; - } - } - // implicit dependency if they view the same tensor - const ggml_tensor * dst2 = dst->view_src ? dst->view_src : dst; - const ggml_tensor * src2 = src->view_src ? src->view_src : src; - if (dst2 == src2) { - return true; - } - return false; - }; - - for (int node_idx = 0; node_idx < cgraph->n_nodes; node_idx++) { - const ggml_tensor * node = cgraph->nodes[node_idx]; - node_indices[node] = node_idx; - - if (is_noop(node)) { - continue; - } - for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { - const ggml_tensor * src = cgraph->nodes[node_idx]->src[src_idx]; - //TODO: check why nrows > 1 fails - if (node && !is_noop(node) && ggml_nrows(node) <= 1) { - fan_out[src] += 1; - } - } - } - - // Target Q, K, V for concurrency - // this is a more general way to find nodes which can be candidates for concurrency (although it has not been tested for anything else): - // 1. find fan-out (fork) nodes where the same input is used at least N times (in QKV, it would be "attn-norm") - // 2. find the join node, where 2 or more of the outputs are required (in QKV, this would "KQ" or "flash-attn") - // 3. account for all branches from the fork to the join - // 4. To extend lifetimes of the tensors, we interleave the branches (see below for more details) - // 5. save the original cgraph and restore it in graph_compute, to enable fusion within streams - // See discussion: https://github.com/ggml-org/llama.cpp/pull/16991#issuecomment-3522620030 - - const int min_fan_out = 3; - const int max_fan_out = 3; - - // store {fork_idx, join_idx} - std::vector> concurrent_node_ranges; - - for (const auto & [root_node, count] : fan_out) { - if (count >= min_fan_out && count <= max_fan_out) { - const int root_node_idx = node_indices[root_node]; - - // only optimize for attn_norm - // TODO: make this more generic - if (!strstr(root_node->name, "attn_norm")) { - continue; - } - - bool is_part_of_event = false; - for (const auto & [start, end] : concurrent_node_ranges) { - if (root_node_idx >= start && root_node_idx <= end) { - is_part_of_event = true; - } - } - - if (is_part_of_event) { - continue; - } - - std::vector> nodes_per_branch; - for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { - const ggml_tensor * node = cgraph->nodes[i]; - if (!is_noop(node) && depends_on(node, root_node)) { - nodes_per_branch.push_back({ node }); - } - } - - GGML_ASSERT(nodes_per_branch.size() == (size_t) count); - - //find the join point - const ggml_tensor * join_node = nullptr; - - const auto & belongs_to_branch = [&](const ggml_tensor * node, - const std::vector & branch) -> bool { - for (const ggml_tensor * n : branch) { - if (depends_on(node, n)) { - return true; - } - } - return false; - }; - - for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { - const ggml_tensor * curr_node = cgraph->nodes[i]; - - int num_joins = 0; - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - if (belongs_to_branch(curr_node, nodes_per_branch[branch_idx])) { - num_joins++; - } - } - - if (num_joins >= 2) { - join_node = curr_node; - break; - } - - bool found_branch = false; - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - std::vector & branch_vec = nodes_per_branch[branch_idx]; - if (belongs_to_branch(curr_node, branch_vec)) { - //continue accumulating - if (std::find(branch_vec.begin(), branch_vec.end(), curr_node) == branch_vec.end()) { - branch_vec.push_back(curr_node); - } - found_branch = true; - } - } - - if (!found_branch && is_noop(curr_node)) { - // we can put it in any branch because it will be ignored - nodes_per_branch[0].push_back({ curr_node }); - } - } - - if (join_node) { - //Create ggml_cuda_concurrent_event - ggml_cuda_concurrent_event concurrent_event(nodes_per_branch.size()); - concurrent_event.join_node = join_node; - - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - for (const ggml_tensor * n : nodes_per_branch[branch_idx]) { - concurrent_event.stream_mapping[n] = branch_idx + 1; - } - } - - int fork_node_idx = node_indices[root_node]; - int join_node_idx = node_indices[join_node]; - - int current_branch_idx = 0; - int current_node_idx = fork_node_idx + 1; - const int n_branches = nodes_per_branch.size(); - - int total_branch_nodes = 0; - for (std::vector branch_nodes : nodes_per_branch) { - total_branch_nodes += branch_nodes.size(); - } - - // there are other nodes in the middle which are unaccounted for - // usually (cpy) nodes, then ignore this fork - if (join_node_idx - fork_node_idx - 1 != total_branch_nodes) { - GGML_LOG_DEBUG( - "Skipping %s because the number of nodes in the middle is not equal to the total number of " - "branch nodes %d != %d\n", - root_node->name, join_node_idx - fork_node_idx - 1, total_branch_nodes); - continue; - } - - // Save the original order of nodes in this region before interleaving - // This is used later to restore grouping for fusion within streams - concurrent_event.original_order.reserve(total_branch_nodes); - for (int i = fork_node_idx + 1; i < join_node_idx; ++i) { - concurrent_event.original_order.push_back(cgraph->nodes[i]); - } - - std::unordered_map & concurrent_events = cuda_ctx->stream_context().concurrent_events; - GGML_ASSERT(concurrent_events.find(root_node) == concurrent_events.end()); - concurrent_events.emplace(root_node, std::move(concurrent_event)); - GGML_LOG_DEBUG("Adding stream at node %s %p\n", root_node->name, root_node); - concurrent_node_ranges.emplace_back(fork_node_idx, join_node_idx); - - // interleave tensors to extend lifetimes so that ggml graph doesn't recycle them - // example transformation: - // [attn-norm, QMul, QNorm, QRope, KMul, KNorm, KRope, VMul, attn] -> - // [attn-norm, QMul, KMul, VMul, QNorm, VNorm, QRope, KRope, attn] - while (current_node_idx < join_node_idx) { - std::vector & branch_nodes = nodes_per_branch[current_branch_idx]; - - bool has_node = false; - for (std::vector branch_node : nodes_per_branch) { - has_node |= branch_node.size() > 0; - } - - GGML_ASSERT(has_node); - - if (branch_nodes.empty()) { - current_branch_idx = (current_branch_idx + 1) % n_branches; - continue; - } - - cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); - current_node_idx++; - branch_nodes.erase(branch_nodes.begin()); - - // append all empty nodes - while (!branch_nodes.empty() && is_noop(branch_nodes.front())) { - cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); - current_node_idx++; - branch_nodes.erase(branch_nodes.begin()); - } - - current_branch_idx = (current_branch_idx + 1) % n_branches; - } - } - } - } -} - -static const ggml_backend_i ggml_backend_cuda_interface = { - /* .get_name = */ ggml_backend_cuda_get_name, - /* .free = */ ggml_backend_cuda_free, - /* .set_tensor_async = */ ggml_backend_cuda_set_tensor_async, - /* .get_tensor_async = */ ggml_backend_cuda_get_tensor_async, - /* .get_tensor_2d_async = */ ggml_backend_cuda_set_tensor_2d_async, - /* .set_tensor_2d_async = */ ggml_backend_cuda_get_tensor_2d_async, - /* .cpy_tensor_async = */ ggml_backend_cuda_cpy_tensor_async, - /* .synchronize = */ ggml_backend_cuda_synchronize, - /* .graph_plan_create = */ NULL, - /* .graph_plan_free = */ NULL, - /* .graph_plan_update = */ NULL, - /* .graph_plan_compute = */ NULL, - /* .graph_compute = */ ggml_backend_cuda_graph_compute, - /* .event_record = */ ggml_backend_cuda_event_record, - /* .event_wait = */ ggml_backend_cuda_event_wait, - /* .graph_optimize = */ ggml_backend_cuda_graph_optimize, -}; - -static ggml_guid_t ggml_backend_cuda_guid() { - static ggml_guid guid = { 0x2c, 0xdd, 0xe8, 0x1c, 0x65, 0xb3, 0x65, 0x73, 0x6a, 0x12, 0x88, 0x61, 0x1c, 0xc9, 0xdc, 0x25 }; - return &guid; -} - -bool ggml_backend_is_cuda(ggml_backend_t backend) { - return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_cuda_guid()); -} - -int ggml_backend_cuda_get_device_count() { - return ggml_cuda_info().device_count; -} - -void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) { - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, device)); - snprintf(description, description_size, "%s", prop.name); -} - -void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) { - ggml_cuda_set_device(device); - - CUDA_CHECK(cudaMemGetInfo(free, total)); -} - -bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) { - if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { - return false; - } - -#if CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) || defined(GGML_USE_HIP) - cudaError_t err = cudaHostRegister(buffer, size, cudaHostRegisterPortable | cudaHostRegisterReadOnly); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - - GGML_LOG_DEBUG("%s: failed to register %.2f MiB of pinned memory: %s\n", __func__, - size / 1024.0 / 1024.0, cudaGetErrorString(err)); - return false; - } - return true; -#else - GGML_UNUSED(buffer); - GGML_UNUSED(size); - return false; -#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) -} - -void ggml_backend_cuda_unregister_host_buffer(void * buffer) { - if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { - return; - } - - cudaError_t err = cudaHostUnregister(buffer); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - } -} - - -// backend device - -struct ggml_backend_cuda_device_context { - int device; - std::string name; - std::string description; - std::string pci_bus_id; - int op_offload_min_batch_size; -}; - -static const char * ggml_backend_cuda_device_get_name(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ctx->name.c_str(); -} - -static const char * ggml_backend_cuda_device_get_description(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ctx->description.c_str(); -} - -#if defined(__linux__) -// Helper function to get available memory from /proc/meminfo for UMA systems -static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_kb, long * free_swap_kb) { - FILE * meminfo_file = nullptr; - // 2KB buffer for reading /proc/meminfo since it does not report size info, should be enough - const size_t BUFFER_SIZE = 2048; - auto file_buffer = std::make_unique(BUFFER_SIZE); - size_t bytes_read = 0; - long huge_tlb_total_pages = -1; - long huge_tlb_free_pages = -1; - long huge_tlb_page_size = -1; - - if (available_memory_kb == nullptr || free_swap_kb == nullptr) { - return false; - } - - meminfo_file = fopen("/proc/meminfo", "r"); - if (meminfo_file == nullptr) { - GGML_LOG_ERROR("%s: failed to open /proc/meminfo\n", __func__); - return false; - } - - // Read file into buffer - bytes_read = fread(file_buffer.get(), 1, BUFFER_SIZE - 1, meminfo_file); - fclose(meminfo_file); - - if (bytes_read == 0) { - GGML_LOG_ERROR("%s: failed to read from /proc/meminfo\n", __func__); - return false; - } - file_buffer[bytes_read] = '\0'; - - *available_memory_kb = -1; - *free_swap_kb = -1; - - // Parse the file buffer line by line - char * line = file_buffer.get(); - char * line_next; - while (line < file_buffer.get() + bytes_read) { - // Find the end of the current line - line_next = strchr(line, '\n'); - if (line_next != nullptr) { - *line_next = '\0'; - line_next++; - } else { - line_next = file_buffer.get() + bytes_read; - } - - long value; - if (sscanf(line, "MemAvailable: %ld kB", &value) == 1) { - *available_memory_kb = value; - } else if (sscanf(line, "SwapFree: %ld kB", &value) == 1) { - *free_swap_kb = value; - } else if (sscanf(line, "HugePages_Total: %ld", &value) == 1) { - huge_tlb_total_pages = value; - } else if (sscanf(line, "HugePages_Free: %ld", &value) == 1) { - huge_tlb_free_pages = value; - } else if (sscanf(line, "Hugepagesize: %ld kB", &value) == 1) { - huge_tlb_page_size = value; - } - - line = line_next; - } - - if (huge_tlb_total_pages != 0 && huge_tlb_total_pages != -1) { - *available_memory_kb = huge_tlb_free_pages * huge_tlb_page_size; - - // Hugetlbfs pages are not swappable. - *free_swap_kb = 0; - } - - GGML_LOG_DEBUG("%s: final available_memory_kb: %ld\n", __func__, *available_memory_kb); - return true; -} -#endif // defined(__linux__) - -static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemGetInfo(free, total)); - -// ref: https://github.com/ggml-org/llama.cpp/pull/17368 -#if defined(__linux__) - // Check if this is a UMA (Unified Memory Architecture) system - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device)); - - // Check if UMA is explicitly enabled via environment variable - bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr; - bool is_uma = prop.integrated > 0 || uma_env; - - if (is_uma) { - // For UMA systems (like DGX Spark), use system memory info - long available_memory_kb = 0; - long free_swap_kb = 0; - - if (ggml_backend_cuda_get_available_uma_memory(&available_memory_kb, &free_swap_kb) && available_memory_kb > 0) { - *free = (size_t)available_memory_kb * 1024; - } else { - GGML_LOG_ERROR("%s: /proc/meminfo reading failed, using cudaMemGetInfo\n", __func__); - } - } -#endif // defined(__linux__) - -} - -static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) { - GGML_UNUSED(dev); - return GGML_BACKEND_DEVICE_TYPE_GPU; -} - -static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_backend_dev_props * props) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - - props->name = ggml_backend_cuda_device_get_name(dev); - props->description = ggml_backend_cuda_device_get_description(dev); - props->type = ggml_backend_cuda_device_get_type(dev); - props->device_id = ctx->pci_bus_id.empty() ? nullptr : ctx->pci_bus_id.c_str(); - ggml_backend_cuda_device_get_memory(dev, &props->memory_free, &props->memory_total); - - bool host_buffer = getenv("GGML_CUDA_NO_PINNED") == nullptr; -#ifdef GGML_CUDA_NO_PEER_COPY - bool events = false; -#else - bool events = true; -#endif - - props->caps = { - /* .async = */ true, - /* .host_buffer = */ host_buffer, - /* .buffer_from_host_ptr = */ false, - /* .events = */ events, - }; -} - -static ggml_backend_t ggml_backend_cuda_device_init_backend(ggml_backend_dev_t dev, const char * params) { - GGML_UNUSED(params); - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ggml_backend_cuda_init(ctx->device); -} - -static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_buffer_type(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ggml_backend_cuda_buffer_type(ctx->device); -} - -static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_host_buffer_type(ggml_backend_dev_t dev) { - GGML_UNUSED(dev); - return ggml_backend_cuda_host_buffer_type(); -} - -// TODO: move these functions here -static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - - // split buffers can only be used with GGML_OP_MUL_MAT - if (op->op != GGML_OP_MUL_MAT) { - for (int i = 0; i < GGML_MAX_SRC; i++) { - if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda_split(op->src[i]->buffer->buft)) { - return false; - } - } - } - - // check if all the sources are allocated on this device - for (int i = 0; i < GGML_MAX_SRC; i++) { - if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda(op->src[i]->buffer->buft)) { - ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)op->src[i]->buffer->buft->context; - if (buft_ctx->device != dev_ctx->device) { - return false; - } - } - } - - switch (op->op) { - case GGML_OP_UNARY: - switch (ggml_get_unary_op(op)) { - case GGML_UNARY_OP_ABS: - case GGML_UNARY_OP_SGN: - case GGML_UNARY_OP_NEG: - case GGML_UNARY_OP_STEP: - case GGML_UNARY_OP_GELU: - case GGML_UNARY_OP_SILU: - case GGML_UNARY_OP_RELU: - case GGML_UNARY_OP_SIGMOID: - case GGML_UNARY_OP_HARDSIGMOID: - case GGML_UNARY_OP_HARDSWISH: - case GGML_UNARY_OP_GELU_ERF: - case GGML_UNARY_OP_GELU_QUICK: - case GGML_UNARY_OP_TANH: - case GGML_UNARY_OP_EXP: - case GGML_UNARY_OP_EXPM1: - case GGML_UNARY_OP_SOFTPLUS: - case GGML_UNARY_OP_ELU: - case GGML_UNARY_OP_XIELU: - case GGML_UNARY_OP_FLOOR: - case GGML_UNARY_OP_CEIL: - case GGML_UNARY_OP_ROUND: - case GGML_UNARY_OP_TRUNC: - // TODO: should become: - //return ggml_is_contiguous_rows(op->src[0]); - return ggml_is_contiguous(op->src[0]); - default: - return false; - } - break; - case GGML_OP_GLU: - switch (ggml_get_glu_op(op)) { - case GGML_GLU_OP_REGLU: - case GGML_GLU_OP_GEGLU: - case GGML_GLU_OP_SWIGLU: - case GGML_GLU_OP_SWIGLU_OAI: - case GGML_GLU_OP_GEGLU_ERF: - case GGML_GLU_OP_GEGLU_QUICK: - return ggml_is_contiguous_1(op->src[0]); - default: - return false; - } - break; - case GGML_OP_MUL_MAT: - case GGML_OP_MUL_MAT_ID: - { - struct ggml_tensor * a = op->src[0]; - struct ggml_tensor * b = op->src[1]; - if (a->buffer && ggml_backend_buft_is_cuda_split(a->buffer->buft)) { - if (a->ne[2] > 1 || a->ne[3] > 1) { - return false; - } - // for small weight matrices the active device can end up without any rows, don't use row split in those cases - // this avoids some edge cases (and the performance would not be good anyways) - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) a->buffer->buft->context; - int64_t row_low; - int64_t row_high; - get_row_split(&row_low, &row_high, a, buft_ctx->tensor_split, dev_ctx->device); - if (row_low == row_high) { - return false; - } - } - if (b->type == GGML_TYPE_F16 && a->type != GGML_TYPE_F16) { - return false; - } -#ifdef GGML_USE_MUSA - const int cc = ggml_cuda_info().devices[dev_ctx->device].cc; - if (b->ne[2]*b->ne[3] > 1 && !ggml_is_transposed(a) && !ggml_is_transposed(b)) { - if (GGML_CUDA_CC_IS_QY1(cc) && op->op == GGML_OP_MUL_MAT && - a->type == GGML_TYPE_F16 && b->type == GGML_TYPE_F16) { - return false; - } - if (GGML_CUDA_CC_IS_QY2(cc) && op->op == GGML_OP_MUL_MAT_ID && - a->type == GGML_TYPE_Q2_K && b->type == GGML_TYPE_F32) { - return false; - } - } -#endif // GGML_USE_MUSA - switch (a->type) { - case GGML_TYPE_F32: - case GGML_TYPE_F16: - case GGML_TYPE_Q1_0: - case GGML_TYPE_Q4_0: - case GGML_TYPE_Q4_1: - case GGML_TYPE_Q5_0: - case GGML_TYPE_Q5_1: - case GGML_TYPE_Q8_0: - case GGML_TYPE_MXFP4: - case GGML_TYPE_NVFP4: - case GGML_TYPE_Q2_K: - case GGML_TYPE_Q3_K: - case GGML_TYPE_Q4_K: - case GGML_TYPE_Q5_K: - case GGML_TYPE_Q6_K: - case GGML_TYPE_Q8_K: - case GGML_TYPE_IQ1_M: - case GGML_TYPE_IQ1_S: - case GGML_TYPE_IQ2_S: - case GGML_TYPE_IQ2_XS: - case GGML_TYPE_IQ2_XXS: - case GGML_TYPE_IQ3_S: - case GGML_TYPE_IQ3_XXS: - case GGML_TYPE_IQ4_NL: - case GGML_TYPE_IQ4_XS: - case GGML_TYPE_BF16: - return true; - default: - return false; - } - } break; - case GGML_OP_OUT_PROD: - return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32; - case GGML_OP_GET_ROWS: - { - switch (op->src[0]->type) { - case GGML_TYPE_F16: - case GGML_TYPE_F32: - case GGML_TYPE_BF16: - case GGML_TYPE_I32: - case GGML_TYPE_Q1_0: - case GGML_TYPE_Q4_0: - case GGML_TYPE_Q4_1: - case GGML_TYPE_Q5_0: - case GGML_TYPE_Q5_1: - case GGML_TYPE_Q8_0: - return true; - default: - return false; - } - } break; - case GGML_OP_GET_ROWS_BACK: - { - return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->ne[2] == 1 && op->ne[3] == 1; - } break; - case GGML_OP_SET_ROWS: - { - return (op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_BF16 || - op->type == GGML_TYPE_Q4_0 || op->type == GGML_TYPE_Q4_1 || op->type == GGML_TYPE_Q5_0 || - op->type == GGML_TYPE_Q5_1 || op->type == GGML_TYPE_Q8_0 || op->type == GGML_TYPE_IQ4_NL) && - op->src[0]->type == GGML_TYPE_F32 && - (op->src[1]->type == GGML_TYPE_I64 || op->src[1]->type == GGML_TYPE_I32); - } break; - case GGML_OP_SET: - { - const ggml_type t = op->type; - return (t == GGML_TYPE_F32 || t == GGML_TYPE_I32) && - t == op->src[0]->type && - t == op->src[1]->type; - } break; - case GGML_OP_CPY: - { - ggml_type src0_type = op->src[0]->type; - ggml_type src1_type = op->src[1]->type; - if ((src0_type == GGML_TYPE_F32 || src0_type == GGML_TYPE_BF16 || src0_type == GGML_TYPE_F16) && - (src1_type == GGML_TYPE_F32 || src1_type == GGML_TYPE_BF16 || src1_type == GGML_TYPE_F16) - ) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q8_0) { - return true; - } - if (src0_type == GGML_TYPE_Q8_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_0) { - return true; - } - if (src0_type == GGML_TYPE_Q4_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_1) { - return true; - } - if (src0_type == GGML_TYPE_Q4_1 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_0) { - return true; - } - if (src0_type == GGML_TYPE_Q5_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_1) { - return true; - } - if (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_IQ4_NL) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) { - return true; - } - if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_I32) { - return true; - } - if (src0_type == src1_type && ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1])) { - return true; - } - return false; - } break; - case GGML_OP_DUP: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_ARGMAX: - case GGML_OP_COUNT_EQUAL: - { - return true; - } break; - case GGML_OP_REPEAT: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_REPEAT_BACK: - return op->type == GGML_TYPE_F32 && (op->src[0]->ne[2]*op->src[0]->ne[3]) <= (1 << 15); - case GGML_OP_CONCAT: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_CONV_TRANSPOSE_1D: - { - ggml_type src0_type = op->src[0]->type; - ggml_type src1_type = op->src[1]->type; - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_F32) { - return true; - } - return false; - } break; - case GGML_OP_SILU_BACK: - return ggml_is_contiguous(op->src[0]) && op->src[0]->type == GGML_TYPE_F32; - break; - case GGML_OP_NORM: - case GGML_OP_RMS_NORM: - case GGML_OP_L2_NORM: - return true; - case GGML_OP_RMS_NORM_BACK: - return ggml_is_contiguous(op->src[0]); - break; - case GGML_OP_NONE: - case GGML_OP_RESHAPE: - case GGML_OP_VIEW: - case GGML_OP_PERMUTE: - case GGML_OP_TRANSPOSE: - case GGML_OP_ADD: - case GGML_OP_ADD_ID: - case GGML_OP_ADD1: - case GGML_OP_SUB: - case GGML_OP_MUL: - case GGML_OP_DIV: - case GGML_OP_SCALE: - case GGML_OP_SQR: - case GGML_OP_SQRT: - case GGML_OP_SIN: - case GGML_OP_COS: - case GGML_OP_CLAMP: - case GGML_OP_LOG: - return true; - case GGML_OP_SSM_SCAN: { - if (op->src[3]->ne[0] == 1) { - // Mamba2 - // (kernel only supports (d_state == 128 || d_state == 256) && d_head % 16 == 0) - return (op->src[0]->ne[0] == 128 || op->src[0]->ne[0] == 256) && op->src[0]->ne[1] % 16 == 0; - } else { - // Mamba - // (kernel only supports d_state == 16, d_head == 1, n_head % 128 == 0, n_group == 1) - return op->src[0]->ne[0] == 16 && op->src[0]->ne[1] == 1 && op->src[0]->ne[2] % 128 == 0 && op->src[4]->ne[1] == 1; - } - } - case GGML_OP_SSM_CONV: { - // assumes d_inner % threads == 0 - return op->src[0]->ne[1] % 128 == 0; - } - case GGML_OP_CONT: - return true; - case GGML_OP_DIAG_MASK_INF: - return true; - case GGML_OP_SOFT_MAX: - return true; - case GGML_OP_SOFT_MAX_BACK: { - float max_bias = 0.0f; - memcpy(&max_bias, (const float *) op->op_params + 1, sizeof(float)); - return max_bias == 0.0f; - } - case GGML_OP_ROLL: - if(op->src[0]->type == GGML_TYPE_F32) { - return true; - } - return false; - case GGML_OP_ROPE: - case GGML_OP_ROPE_BACK: { - return op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]); - } - case GGML_OP_IM2COL: - case GGML_OP_IM2COL_3D: - case GGML_OP_CONV_2D: - case GGML_OP_CONV_2D_DW: - case GGML_OP_CONV_TRANSPOSE_2D: - case GGML_OP_POOL_2D: - return true; - case GGML_OP_ACC: - // TODO: extend support like so: - //return ggml_is_contiguous_rows(op->src[0]) && ggml_is_contiguous_rows(op->src[1]); - return ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1]); - case GGML_OP_SUM: - return ggml_is_contiguous_rows(op->src[0]); - case GGML_OP_TOP_K: - case GGML_OP_ARGSORT: -#ifndef GGML_CUDA_USE_CUB - return op->src[0]->ne[0] <= 1024; -#else - return true; -#endif - case GGML_OP_SUM_ROWS: - case GGML_OP_MEAN: - case GGML_OP_GROUP_NORM: - return ggml_is_contiguous(op->src[0]); - case GGML_OP_PAD: - return true; - case GGML_OP_UPSCALE: - case GGML_OP_PAD_REFLECT_1D: - case GGML_OP_ARANGE: - case GGML_OP_TIMESTEP_EMBEDDING: - case GGML_OP_LEAKY_RELU: - case GGML_OP_RWKV_WKV6: - case GGML_OP_GATED_LINEAR_ATTN: - case GGML_OP_RWKV_WKV7: - return true; - case GGML_OP_GATED_DELTA_NET: - //TODO: enable once MUSA compiler is solved https://github.com/ggml-org/llama.cpp/pull/19504#issuecomment-4018634327 -#ifdef GGML_USE_MUSA - return false; -#else - return true; -#endif // GGML_USE_MUSA - case GGML_OP_FLASH_ATTN_EXT: - return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op); - case GGML_OP_CROSS_ENTROPY_LOSS: - case GGML_OP_CROSS_ENTROPY_LOSS_BACK: - case GGML_OP_OPT_STEP_ADAMW: - case GGML_OP_OPT_STEP_SGD: - case GGML_OP_FILL: - case GGML_OP_CUMSUM: - case GGML_OP_TRI: - case GGML_OP_DIAG: - case GGML_OP_SOLVE_TRI: - return true; - - default: - return false; - } -} - -static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated; - return (((ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft))); -} - -static int64_t get_op_batch_size(const ggml_tensor * op) { - switch (op->op) { - case GGML_OP_GET_ROWS: - return 0; - case GGML_OP_MUL_MAT: - return op->ne[1]; - case GGML_OP_MUL_MAT_ID: - case GGML_OP_ROPE: - case GGML_OP_ROPE_BACK: - return op->ne[2]; - default: - return ggml_nrows(op); - } -} - -static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - - return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size; -} - -static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_t dev) { -#ifdef GGML_CUDA_NO_PEER_COPY - return nullptr; -#else - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *)dev->context; - - ggml_cuda_set_device(dev_ctx->device); - - cudaEvent_t event; - CUDA_CHECK(cudaEventCreateWithFlags(&event, cudaEventDisableTiming)); - - return new ggml_backend_event { - /* .device = */ dev, - /* .context = */ event, - }; -#endif -} - -static void ggml_backend_cuda_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) { - GGML_UNUSED(dev); - - CUDA_CHECK(cudaEventDestroy((cudaEvent_t)event->context)); - delete event; -} - -static void ggml_backend_cuda_device_event_synchronize(ggml_backend_dev_t dev, ggml_backend_event_t event) { - GGML_UNUSED(dev); - CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context)); -} - -static const ggml_backend_device_i ggml_backend_cuda_device_interface = { - /* .get_name = */ ggml_backend_cuda_device_get_name, - /* .get_description = */ ggml_backend_cuda_device_get_description, - /* .get_memory = */ ggml_backend_cuda_device_get_memory, - /* .get_type = */ ggml_backend_cuda_device_get_type, - /* .get_props = */ ggml_backend_cuda_device_get_props, - /* .init_backend = */ ggml_backend_cuda_device_init_backend, - /* .get_buffer_type = */ ggml_backend_cuda_device_get_buffer_type, - /* .get_host_buffer_type = */ ggml_backend_cuda_device_get_host_buffer_type, - /* .buffer_from_host_ptr = */ NULL, - /* .supports_op = */ ggml_backend_cuda_device_supports_op, - /* .supports_buft = */ ggml_backend_cuda_device_supports_buft, - /* .offload_op = */ ggml_backend_cuda_device_offload_op, - /* .event_new = */ ggml_backend_cuda_device_event_new, - /* .event_free = */ ggml_backend_cuda_device_event_free, - /* .event_synchronize = */ ggml_backend_cuda_device_event_synchronize, -}; - -// backend reg - -struct ggml_backend_cuda_reg_context { - std::vector devices; -}; - -static const char * ggml_backend_cuda_reg_get_name(ggml_backend_reg_t reg) { - GGML_UNUSED(reg); - return GGML_CUDA_NAME; -} - -static size_t ggml_backend_cuda_reg_get_device_count(ggml_backend_reg_t reg) { - ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; - return ctx->devices.size(); -} - -static ggml_backend_dev_t ggml_backend_cuda_reg_get_device(ggml_backend_reg_t reg, size_t index) { - ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; - GGML_ASSERT(index < ctx->devices.size()); - return ctx->devices[index]; -} - -static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t reg) { - static std::vector features = []() { - std::vector features; - #define _STRINGIFY(...) #__VA_ARGS__ - #define STRINGIFY(...) _STRINGIFY(__VA_ARGS__) - - #ifdef __CUDA_ARCH_LIST__ - features.push_back({ "ARCHS", STRINGIFY(__CUDA_ARCH_LIST__) }); - #endif - - #ifdef GGML_CUDA_FORCE_MMQ - features.push_back({ "FORCE_MMQ", "1" }); - #endif - - #ifdef GGML_CUDA_FORCE_CUBLAS - features.push_back({ "FORCE_CUBLAS", "1" }); - #endif - - #ifndef GGML_USE_VMM - features.push_back({ "NO_VMM", "1" }); - #endif - - #ifdef GGML_CUDA_NO_PEER_COPY - features.push_back({ "NO_PEER_COPY", "1" }); - #endif - - #ifdef GGML_CUDA_USE_GRAPHS - features.push_back({ "USE_GRAPHS", "1" }); - #endif - - #ifdef GGML_CUDA_PEER_MAX_BATCH_SIZE - features.push_back({ "PEER_MAX_BATCH_SIZE", STRINGIFY(GGML_CUDA_PEER_MAX_BATCH_SIZE) }); - #endif - - #ifdef GGML_CUDA_FA_ALL_QUANTS - features.push_back({ "FA_ALL_QUANTS", "1" }); - #endif - - { - const auto & info = ggml_cuda_info(); - for (int id = 0; id < info.device_count; ++id) { - if (blackwell_mma_available(info.devices[id].cc)) { - features.push_back({ "BLACKWELL_NATIVE_FP4", "1"}); - break; - } - } - } - - #undef _STRINGIFY - #undef STRINGIFY - - features.push_back({ nullptr, nullptr }); - - return features; - }(); - - return features.data(); - - GGML_UNUSED(reg); -} - -static void * ggml_backend_cuda_reg_get_proc_address(ggml_backend_reg_t reg, const char * name) { - GGML_UNUSED(reg); - if (strcmp(name, "ggml_backend_comm_init") == 0) { - return (void *)ggml_backend_cuda_comm_init; - } - if (strcmp(name, "ggml_backend_comm_free") == 0) { - return (void *)ggml_backend_cuda_comm_free; - } - if (strcmp(name, "ggml_backend_comm_allreduce_tensor") == 0) { - return (void *)ggml_backend_cuda_comm_allreduce_tensor; - } - if (strcmp(name, "ggml_backend_split_buffer_type") == 0) { - return (void *)ggml_backend_cuda_split_buffer_type; - } - if (strcmp(name, "ggml_backend_register_host_buffer") == 0) { - return (void *)ggml_backend_cuda_register_host_buffer; - } - if (strcmp(name, "ggml_backend_unregister_host_buffer") == 0) { - return (void *)ggml_backend_cuda_unregister_host_buffer; - } - if (strcmp(name, "ggml_backend_get_features") == 0) { - return (void *)ggml_backend_cuda_get_features; - } - return nullptr; -} - -static const ggml_backend_reg_i ggml_backend_cuda_reg_interface = { - /* .get_name = */ ggml_backend_cuda_reg_get_name, - /* .get_device_count = */ ggml_backend_cuda_reg_get_device_count, - /* .get_device = */ ggml_backend_cuda_reg_get_device, - /* .get_proc_address = */ ggml_backend_cuda_reg_get_proc_address, -}; - -// backend registry -ggml_backend_reg_t ggml_backend_cuda_reg() { - static ggml_backend_reg reg; - static bool initialized = false; - - { - static std::mutex mutex; - std::lock_guard lock(mutex); - if (!initialized) { - ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context; - const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32; - - for (int i = 0; i < ggml_cuda_info().device_count; i++) { - ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context; - dev_ctx->device = i; - dev_ctx->name = GGML_CUDA_NAME + std::to_string(i); - - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, i)); - dev_ctx->description = prop.name; - - char pci_bus_id[16] = {}; - snprintf(pci_bus_id, sizeof(pci_bus_id), "%04x:%02x:%02x.0", prop.pciDomainID, prop.pciBusID, prop.pciDeviceID); - dev_ctx->pci_bus_id = pci_bus_id; - dev_ctx->op_offload_min_batch_size = min_batch_size; - - ggml_backend_dev_t dev = new ggml_backend_device { - /* .iface = */ ggml_backend_cuda_device_interface, - /* .reg = */ ®, - /* .context = */ dev_ctx - }; - ctx->devices.push_back(dev); - } - - reg = ggml_backend_reg { - /* .api_version = */ GGML_BACKEND_API_VERSION, - /* .iface = */ ggml_backend_cuda_reg_interface, - /* .context = */ ctx - }; - } - - initialized = true; - } - - return ® -} - -ggml_backend_t ggml_backend_cuda_init(int device) { - if (device < 0 || device >= ggml_backend_cuda_get_device_count()) { - GGML_LOG_ERROR("%s: invalid device %d\n", __func__, device); - return nullptr; - } - - ggml_backend_cuda_context * ctx = new ggml_backend_cuda_context(device); - if (ctx == nullptr) { - GGML_LOG_ERROR("%s: failed to allocate context\n", __func__); - return nullptr; - } - - ggml_backend_t cuda_backend = new ggml_backend { - /* .guid = */ ggml_backend_cuda_guid(), - /* .iface = */ ggml_backend_cuda_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device), - /* .context = */ ctx, - }; - - return cuda_backend; -} - -GGML_BACKEND_DL_IMPL(ggml_backend_cuda_reg) +#include "ggml-cuda.h" +#include "ggml-impl.h" +#include "ggml-backend-impl.h" + +#include "ggml-cuda/allreduce.cuh" +#include "ggml-cuda/comm.cuh" +#include "ggml-cuda/common.cuh" +#include "ggml-cuda/acc.cuh" +#include "ggml-cuda/add-id.cuh" +#include "ggml-cuda/arange.cuh" +#include "ggml-cuda/argmax.cuh" +#include "ggml-cuda/argsort.cuh" +#include "ggml-cuda/binbcast.cuh" +#include "ggml-cuda/clamp.cuh" +#include "ggml-cuda/concat.cuh" +#include "ggml-cuda/conv-transpose-1d.cuh" +#include "ggml-cuda/conv2d.cuh" +#include "ggml-cuda/conv2d-dw.cuh" +#include "ggml-cuda/conv2d-transpose.cuh" +#include "ggml-cuda/convert.cuh" +#include "ggml-cuda/count-equal.cuh" +#include "ggml-cuda/cpy.cuh" +#include "ggml-cuda/cross-entropy-loss.cuh" +#include "ggml-cuda/cumsum.cuh" +#include "ggml-cuda/diagmask.cuh" +#include "ggml-cuda/diag.cuh" +#include "ggml-cuda/fattn.cuh" +#include "ggml-cuda/getrows.cuh" +#include "ggml-cuda/im2col.cuh" +#include "ggml-cuda/mmf.cuh" +#include "ggml-cuda/mmq.cuh" +#include "ggml-cuda/mmvf.cuh" +#include "ggml-cuda/mmvq.cuh" +#include "ggml-cuda/norm.cuh" +#include "ggml-cuda/opt-step-adamw.cuh" +#include "ggml-cuda/opt-step-sgd.cuh" +#include "ggml-cuda/out-prod.cuh" +#include "ggml-cuda/pad.cuh" +#include "ggml-cuda/pool2d.cuh" +#include "ggml-cuda/quantize.cuh" +#include "ggml-cuda/rope.cuh" +#include "ggml-cuda/roll.cuh" +#include "ggml-cuda/scale.cuh" +#include "ggml-cuda/softcap.cuh" +#include "ggml-cuda/softmax.cuh" +#include "ggml-cuda/ssm-conv.cuh" +#include "ggml-cuda/ssm-scan.cuh" +#include "ggml-cuda/sum.cuh" +#include "ggml-cuda/sumrows.cuh" +#include "ggml-cuda/top-k.cuh" +#include "ggml-cuda/mean.cuh" +#include "ggml-cuda/tsembd.cuh" +#include "ggml-cuda/topk-moe.cuh" +#include "ggml-cuda/unary.cuh" +#include "ggml-cuda/upscale.cuh" +#include "ggml-cuda/wkv.cuh" +#include "ggml-cuda/gla.cuh" +#include "ggml-cuda/gated_delta_net.cuh" +#include "ggml-cuda/set.cuh" +#include "ggml-cuda/set-rows.cuh" +#include "ggml-cuda/pad_reflect_1d.cuh" +#include "ggml-cuda/solve_tri.cuh" +#include "ggml-cuda/tri.cuh" +#include "ggml-cuda/cumsum.cuh" +#include "ggml-cuda/fill.cuh" +#include "ggml.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size"); + +[[noreturn]] +void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) { + int id = -1; // in case cudaGetDevice fails + (void)cudaGetDevice(&id); + + GGML_LOG_ERROR(GGML_CUDA_NAME " error: %s\n", msg); + GGML_LOG_ERROR(" current device: %d, in function %s at %s:%d\n", id, func, file, line); + GGML_LOG_ERROR(" %s\n", stmt); + // abort with GGML_ABORT to get a stack trace + GGML_ABORT(GGML_CUDA_NAME " error"); +} + +// this is faster on Windows +// probably because the Windows CUDA libraries forget to make this check before invoking the drivers +void ggml_cuda_set_device(int device) { + int current_device; + CUDA_CHECK(cudaGetDevice(¤t_device)); + + if (device == current_device) { + return; + } + + CUDA_CHECK(cudaSetDevice(device)); +} + +int ggml_cuda_get_device() { + int id; + CUDA_CHECK(cudaGetDevice(&id)); + return id; +} + +static cudaError_t ggml_cuda_device_malloc(void ** ptr, size_t size, int device) { + ggml_cuda_set_device(device); + cudaError_t err; + if (getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr) { + err = cudaMallocManaged(ptr, size); +#if defined(GGML_USE_HIP) + if (err == hipSuccess) { + // hipMemAdviseSetCoarseGrain is an optional performance hint; + // ignore errors (e.g. hipErrorInvalidValue on some APU/iGPU configs). + (void)cudaMemAdvise(*ptr, size, hipMemAdviseSetCoarseGrain, device); + (void)hipGetLastError(); // clear any error + } + + // fall back to cudaMalloc if not supported (e.g. on Windows) + if (err == hipErrorNotSupported) { + static bool warned_unsupported = false; + if (!warned_unsupported) { + GGML_LOG_WARN("hipMallocManaged unsupported, falling back to hipMalloc.\n"); + warned_unsupported = true; + } + + err = cudaMalloc(ptr, size); + } +#endif // defined(GGML_USE_HIP) + } else { + err = cudaMalloc(ptr, size); + } + return err; +} + +#if defined(GGML_USE_HIP) +static int ggml_cuda_parse_id(char devName[]) { + // A list of possible Target IDs can be found under the rocclr/clr repo in device.cpp + // these values are not stable so this is susceptible to breakage + // https://github.com/ROCm/clr/blob/amd-staging/rocclr/device/device.cpp + int archMajor = 0x0; + int archMinor = 0x0; + int archNum = GGML_CUDA_CC_OFFSET_AMD; + int archLen = strlen(devName); + char archName[archLen + 1]; + + // strip leading 'gfx' while copying into our buffer + if (archLen > 3) { + strcpy(archName, &devName[3]); + archLen -= 3; + } + + // trim trailing :xnack- or :sramecc- statuses + archLen = strcspn(archName, ":"); + archName[archLen] = '\0'; + + // tease out the version information + if (archLen > 8) { + // versions labeled generic use '-' as delimiter + // strip the trailing "-generic" then iterate through what remains + if ((strstr(archName, "-generic"))) { + archName[archLen - 8] = '\0'; + char * pch; + if ((pch = strtok(archName, "-"))) { + archMajor = (int)strtoul(pch, 0, 16); + if ((pch = strtok(NULL, "-"))) { + archMinor = 0x10 * (int)strtoul(pch, 0, 16); + } + } + } + } else if (archLen >= 3) { + // last two digits should be the minor * 0x10 + stepping + archMinor = (int)strtoul(&archName[archLen - 2], 0, 16); + archName[archLen - 2] = '\0'; + + // only the major version remains + archMajor = (int)strtoul(archName, 0, 16); + } + archNum += archMajor * 0x100; + archNum += archMinor; + return archNum; +} +#endif // defined(GGML_USE_HIP) + +static ggml_cuda_device_info ggml_cuda_init() { + ggml_cuda_device_info info = {}; + + cudaError_t err = cudaGetDeviceCount(&info.device_count); + if (err != cudaSuccess) { + GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err)); + return info; + } + + GGML_ASSERT(info.device_count <= GGML_CUDA_MAX_DEVICES); + + int64_t total_vram = 0; + for (int id = 0; id < info.device_count; ++id) { + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); + total_vram += prop.totalGlobalMem; + } + GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n", + __func__, info.device_count, (size_t)(total_vram / (1024 * 1024))); + total_vram = 0; + + std::vector> turing_devices_without_mma; + for (int id = 0; id < info.device_count; ++id) { + int device_vmm = 0; + +#if defined(GGML_USE_VMM) + CUdevice device; + CU_CHECK(cuDeviceGet(&device, id)); + CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device)); + + if (device_vmm) { + CUmemAllocationProp alloc_prop = {}; + alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; + alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + alloc_prop.location.id = id; + CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED)); + } +#endif // defined(GGML_USE_VMM) + info.devices[id].vmm = !!device_vmm; + + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); + + info.default_tensor_split[id] = total_vram; + total_vram += prop.totalGlobalMem; + info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034) + info.devices[id].nsm = prop.multiProcessorCount; + info.devices[id].smpb = prop.sharedMemPerBlock; + info.devices[id].warp_size = prop.warpSize; + +#ifndef GGML_USE_MUSA + int supports_coop_launch = 0; + CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, id)); + info.devices[id].supports_cooperative_launch = !!supports_coop_launch; +#else + info.devices[id].supports_cooperative_launch = false; +#endif // !(GGML_USE_MUSA) + +#if defined(GGML_USE_HIP) + info.devices[id].smpbo = prop.sharedMemPerBlock; + + info.devices[id].cc = ggml_cuda_parse_id(prop.gcnArchName); + if ((info.devices[id].cc & 0xff00) == 0x0) { + GGML_LOG_WARN("invalid architecture ID received for device %d %s: %s cc %d.%d\n", + id, prop.name, prop.gcnArchName, prop.major, prop.minor); + + // Fallback to prop.major and prop.minor + if (prop.major > 0) { + info.devices[id].cc = GGML_CUDA_CC_OFFSET_AMD + prop.major * 0x100; + info.devices[id].cc += prop.minor * 0x10; + } + } + GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n", + id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff, + device_vmm ? "yes" : "no", prop.warpSize, + (size_t)(prop.totalGlobalMem / (1024 * 1024))); +#elif defined(GGML_USE_MUSA) + // FIXME: Ensure compatibility with varying warp sizes across different MUSA archs. + info.devices[id].warp_size = 32; + info.devices[id].smpbo = prop.sharedMemPerBlockOptin; + info.devices[id].cc = GGML_CUDA_CC_OFFSET_MTHREADS + prop.major * 0x100; + info.devices[id].cc += prop.minor * 0x10; + GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", + id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", + (size_t)(prop.totalGlobalMem / (1024 * 1024))); +#else + info.devices[id].smpbo = prop.sharedMemPerBlockOptin; + info.devices[id].cc = 100*prop.major + 10*prop.minor; + GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", + id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", + (size_t)(prop.totalGlobalMem / (1024 * 1024))); + std::string device_name(prop.name); + if (device_name == "NVIDIA GeForce MX450") { + turing_devices_without_mma.push_back({ id, device_name }); + } else if (device_name == "NVIDIA GeForce MX550") { + turing_devices_without_mma.push_back({ id, device_name }); + } else if (device_name.substr(0, 21) == "NVIDIA GeForce GTX 16") { + turing_devices_without_mma.push_back({ id, device_name }); + } + + // Temporary performance fix: + // Setting device scheduling strategy for iGPUs with cc121 to "spinning" to avoid delays in cuda synchronize calls. + // TODO: Check for future drivers the default scheduling strategy and + // remove this call again when cudaDeviceScheduleSpin is default. + if (prop.major == 12 && prop.minor == 1) { + CUDA_CHECK(cudaSetDevice(id)); + CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin)); + } + +#endif // defined(GGML_USE_HIP) + } + + if (ggml_cuda_highest_compiled_arch(GGML_CUDA_CC_TURING) >= GGML_CUDA_CC_TURING && !turing_devices_without_mma.empty()) { + GGML_LOG_INFO("The following devices will have suboptimal performance due to a lack of tensor cores:\n"); + for (size_t device_pos = 0; device_pos < turing_devices_without_mma.size(); device_pos++) { + GGML_LOG_INFO( + " Device %d: %s\n", turing_devices_without_mma[device_pos].first, turing_devices_without_mma[device_pos].second.c_str()); + } + GGML_LOG_INFO( + "Consider compiling with CMAKE_CUDA_ARCHITECTURES=61-virtual;80-virtual and DGGML_CUDA_FORCE_MMQ to force the use of the Pascal code for Turing.\n"); + } + + for (int id = 0; id < info.device_count; ++id) { + info.default_tensor_split[id] /= total_vram; + } + + // configure logging to stdout + // CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr)); + + if (getenv("GGML_CUDA_P2P") != nullptr) { + for (int id = 0; id < info.device_count; ++id) { + ggml_cuda_set_device(id); + for (int id_other = 0; id_other < info.device_count; ++id_other) { + if (id == id_other) { + continue; + } + int can_access_peer; + CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id, id_other)); + if (can_access_peer) { + CUDA_CHECK(cudaDeviceEnablePeerAccess(id_other, 0)); + } + } + } + } + + return info; +} + +const ggml_cuda_device_info & ggml_cuda_info() { + static ggml_cuda_device_info info = ggml_cuda_init(); + return info; +} + +// #define DEBUG_CUDA_MALLOC + +// buffer pool for cuda (legacy) +struct ggml_cuda_pool_leg : public ggml_cuda_pool { + static const int MAX_BUFFERS = 256; + + int device; + struct ggml_cuda_buffer { + void * ptr = nullptr; + size_t size = 0; + }; + + ggml_cuda_buffer buffer_pool[MAX_BUFFERS] = {}; + size_t pool_size = 0; + + explicit ggml_cuda_pool_leg(int device) : + device(device) { + } + + ~ggml_cuda_pool_leg() { + clear_pool(); + GGML_ASSERT(pool_size == 0); + } + + void clear_pool() { + ggml_cuda_set_device(device); + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer & b = buffer_pool[i]; + if (b.ptr != nullptr) { + CUDA_CHECK(cudaFree(b.ptr)); + pool_size -= b.size; + b.ptr = nullptr; + b.size = 0; + } + } + } + + void * alloc(size_t size, size_t * actual_size) override { +#ifdef DEBUG_CUDA_MALLOC + int nnz = 0; + size_t max_size = 0; +#endif + size_t best_diff = 1ull << 36; + int ibest = -1; + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer& b = buffer_pool[i]; + if (b.ptr != nullptr) { +#ifdef DEBUG_CUDA_MALLOC + ++nnz; + if (b.size > max_size) max_size = b.size; +#endif + if (b.size >= size) { + size_t diff = b.size - size; + if (diff < best_diff) { + best_diff = diff; + ibest = i; + if (!best_diff) { + void * ptr = b.ptr; + *actual_size = b.size; + b.ptr = nullptr; + b.size = 0; + return ptr; + } + } + } + } + } + if (ibest >= 0) { + ggml_cuda_buffer& b = buffer_pool[ibest]; + void * ptr = b.ptr; + *actual_size = b.size; + b.ptr = nullptr; + b.size = 0; + return ptr; + } + void * ptr; + size_t look_ahead_size = (size_t) (1.05 * size); + look_ahead_size = 256 * ((look_ahead_size + 255)/256); + ggml_cuda_set_device(device); + cudaError_t err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); + if (err == cudaErrorMemoryAllocation) { + (void)cudaGetLastError(); + const size_t cached_bytes = pool_size; + GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: alloc of %.2f MiB failed, flushing %.2f MiB of cached buffers and retrying\n", + device, look_ahead_size/1024.0/1024.0, cached_bytes/1024.0/1024.0); + CUDA_CHECK(cudaDeviceSynchronize()); + clear_pool(); + err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); + if (err == cudaSuccess) { + GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: retry succeeded\n", device); + } + } + CUDA_CHECK(err); + *actual_size = look_ahead_size; + pool_size += look_ahead_size; +#ifdef DEBUG_CUDA_MALLOC + GGML_LOG_INFO("%s[%d]: %d buffers, max_size = %u MB, pool_size = %u MB, requested %u MB\n", __func__, device, nnz, + (uint32_t)(max_size / 1024 / 1024), (uint32_t)(pool_size / 1024 / 1024), (uint32_t)(size / 1024 / 1024)); +#endif + return ptr; + } + + void free(void * ptr, size_t size) override { + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer& b = buffer_pool[i]; + if (b.ptr == nullptr) { + b.ptr = ptr; + b.size = size; + return; + } + } + GGML_LOG_DEBUG(GGML_CUDA_NAME " buffer pool full, increase MAX_CUDA_BUFFERS\n"); + ggml_cuda_set_device(device); + CUDA_CHECK(cudaFree(ptr)); + pool_size -= size; + } +}; + +// pool with virtual memory +#if defined(GGML_USE_VMM) +struct ggml_cuda_pool_vmm : public ggml_cuda_pool { + static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB + + int device; + CUdeviceptr pool_addr = 0; + size_t pool_used = 0; + size_t pool_size = 0; + size_t granularity; +#if defined(GGML_USE_HIP) + std::vector> mappings; +#endif + + explicit ggml_cuda_pool_vmm(int device) : + device(device), + granularity(ggml_cuda_info().devices[device].vmm_granularity) { + } + + ~ggml_cuda_pool_vmm() { + if (pool_addr != 0) { +#if defined(GGML_USE_HIP) + // Workaround for https://github.com/ROCm/ROCR-Runtime/issues/285 + for (std::pair & mapping : mappings) { + CU_CHECK(cuMemUnmap(mapping.first, mapping.second)); + } +#else + CU_CHECK(cuMemUnmap(pool_addr, pool_size)); +#endif + CU_CHECK(cuMemAddressFree(pool_addr, CUDA_POOL_VMM_MAX_SIZE)); + } + } + + void * alloc(size_t size, size_t * actual_size) override { + // round up the allocation size to the alignment to ensure that all allocations are aligned for all data types + const size_t alignment = 128; + size = alignment * ((size + alignment - 1) / alignment); + + size_t avail = pool_size - pool_used; + + if (size > avail) { + // round up to the next multiple of the granularity + size_t reserve_size = size - avail; + reserve_size = granularity * ((reserve_size + granularity - 1) / granularity); + + GGML_ASSERT(pool_size + reserve_size <= CUDA_POOL_VMM_MAX_SIZE); + + // allocate more physical memory + CUmemAllocationProp prop = {}; + prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; + prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + prop.location.id = device; + CUmemGenericAllocationHandle handle; + CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0)); + + // reserve virtual address space (if not already reserved) + if (pool_addr == 0) { + CU_CHECK(cuMemAddressReserve(&pool_addr, CUDA_POOL_VMM_MAX_SIZE, 0, 0, 0)); + } + + // map at the end of the pool + CUdeviceptr start_ptr = (CUdeviceptr)((char *)(pool_addr) + pool_size); + CU_CHECK(cuMemMap(start_ptr, reserve_size, 0, handle, 0)); +#if defined(GGML_USE_HIP) + mappings.push_back({start_ptr, reserve_size}); +#endif + + // the memory allocation handle is no longer needed after mapping + CU_CHECK(cuMemRelease(handle)); + + // set access + CUmemAccessDesc access = {}; + access.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + access.location.id = device; + access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; + CU_CHECK(cuMemSetAccess((CUdeviceptr)((char *)(pool_addr) + pool_size), reserve_size, &access, 1)); + + // add to the pool + pool_size += reserve_size; + + //printf("cuda pool[%d]: size increased to %llu MB (reserved %llu MB)\n", + // device, (unsigned long long) (pool_size/1024/1024), + // (unsigned long long) (reserve_size/1024/1024)); + } + + GGML_ASSERT(pool_addr != 0); + + void * ptr = (void *) ((CUdeviceptr)((char *)(pool_addr) + pool_used)); + *actual_size = size; + pool_used += size; + +#ifdef DEBUG_CUDA_MALLOC + printf("cuda pool[%d]: allocated %llu bytes at %llx\n", device, (unsigned long long) size, ptr); +#endif + + return ptr; + } + + void free(void * ptr, size_t size) override { +#ifdef DEBUG_CUDA_MALLOC + printf("cuda pool[%d]: freed %llu bytes at %llx\n", device, (unsigned long long) size, ptr); +#endif + + pool_used -= size; + + // all deallocations must be in reverse order of the allocations + GGML_ASSERT(ptr == (void *) ((char *)(pool_addr) + pool_used)); + } +}; +#endif // defined(GGML_USE_VMM) + +std::unique_ptr ggml_backend_cuda_context::new_pool_for_device(int device, + [[maybe_unused]] int stream_no) { +#if defined(GGML_USE_VMM) + if (ggml_cuda_info().devices[device].vmm) { + return std::unique_ptr(new ggml_cuda_pool_vmm(device)); + } +#endif // defined(GGML_USE_VMM) + return std::unique_ptr(new ggml_cuda_pool_leg(device)); +} + +// destroying a cuBLAS handle while a graph is being captured in a different thread can result in a CUDA error +// this lock is used to ensure that no cuBLAS handle is destroyed while a graph is being captured + +static std::mutex ggml_cuda_lock; +static std::condition_variable ggml_cuda_lock_cv; +static std::atomic ggml_cuda_lock_counter; + +ggml_backend_cuda_context::~ggml_backend_cuda_context() { + std::unique_lock lock(ggml_cuda_lock); + ggml_cuda_lock_cv.wait(lock, []{ return ggml_cuda_lock_counter.load(std::memory_order_relaxed) == 0; }); + + if (copy_event != nullptr) { + CUDA_CHECK(cudaEventDestroy(copy_event)); + } + for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { + for (int j = 0; j < GGML_CUDA_MAX_STREAMS; ++j) { + if (streams[i][j] != nullptr) { + CUDA_CHECK(cudaStreamDestroy(streams[i][j])); + } + } + if (cublas_handles[i] != nullptr) { + CUBLAS_CHECK(cublasDestroy(cublas_handles[i])); + } + } +} + + +// cuda buffer + +struct ggml_backend_cuda_buffer_context { + int device; + void * dev_ptr = nullptr; + std::string name; + + ggml_backend_cuda_buffer_context(int device, void * dev_ptr) : + device(device), dev_ptr(dev_ptr), + name(GGML_CUDA_NAME + std::to_string(device)) { + } + + ~ggml_backend_cuda_buffer_context() { + CUDA_CHECK(cudaFree(dev_ptr)); + } +}; + +static void ggml_backend_cuda_buffer_free_buffer(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + delete ctx; +} + +static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) { + return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer; +} + +static void * ggml_backend_cuda_buffer_get_base(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + return ctx->dev_ptr; +} + +static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + if (tensor->view_src != NULL) { + assert(tensor->view_src->buffer->buft == buffer->buft); + return GGML_STATUS_SUCCESS; + } + + if (ggml_is_quantized(tensor->type) && tensor->view_src == nullptr && ggml_backend_buffer_get_usage(buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE) { + // initialize padding to 0 to avoid possible NaN values + const size_t original_size = ggml_nbytes(tensor); + const size_t padded_size = ggml_backend_buft_get_alloc_size(buffer->buft, tensor); + + if (padded_size > original_size) { + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size)); + } + } + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemsetAsync((char *) tensor->data + offset, value, size, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpy2DAsync( + (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpy2DAsync( + data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) { + if (ggml_backend_buffer_is_cuda(src->buffer)) { + ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context; + ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context; + if (src_ctx->device == dst_ctx->device) { + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread)); + } else { +#ifdef GGML_CUDA_NO_PEER_COPY + return false; +#else + CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_ctx->device, src->data, src_ctx->device, ggml_nbytes(src), cudaStreamPerThread)); +#endif + } + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + return true; + } + return false; + + GGML_UNUSED(buffer); +} + +static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemsetAsync(ctx->dev_ptr, value, buffer->size, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = { + /* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer, + /* .get_base = */ ggml_backend_cuda_buffer_get_base, + /* .init_tensor = */ ggml_backend_cuda_buffer_init_tensor, + /* .memset_tensor = */ ggml_backend_cuda_buffer_memset_tensor, + /* .set_tensor = */ ggml_backend_cuda_buffer_set_tensor, + /* .get_tensor = */ ggml_backend_cuda_buffer_get_tensor, + /* .set_tensor_2d = */ ggml_backend_cuda_buffer_set_tensor_2d, + /* .get_tensor_2d = */ ggml_backend_cuda_buffer_get_tensor_2d, + /* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor, + /* .clear = */ ggml_backend_cuda_buffer_clear, + /* .reset = */ NULL, +}; + +// cuda buffer type +struct ggml_backend_cuda_buffer_type_context { + int device; + std::string name; +}; + +static const char * ggml_backend_cuda_buffer_type_get_name(ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; + + return ctx->name.c_str(); +} + +static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name; +} + +static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; + + ggml_cuda_set_device(buft_ctx->device); + + void * dev_ptr; + cudaError_t err = ggml_cuda_device_malloc(&dev_ptr, size, buft_ctx->device); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + GGML_LOG_ERROR("%s: allocating %.2f MiB on device %d: cudaMalloc failed: %s\n", __func__, size / 1024.0 / 1024.0, buft_ctx->device, cudaGetErrorString(err)); + return nullptr; + } + + ggml_backend_cuda_buffer_context * ctx = new ggml_backend_cuda_buffer_context(buft_ctx->device, dev_ptr); + + return ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size); +} + +static size_t ggml_backend_cuda_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { + return 128; + + GGML_UNUSED(buft); +} + +static size_t ggml_backend_cuda_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { + size_t size = ggml_nbytes(tensor); + int64_t ne0 = tensor->ne[0]; + + if (ggml_is_quantized(tensor->type)) { + if (ne0 % MATRIX_ROW_PADDING != 0) { + GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor)); + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + } + + return size; + + GGML_UNUSED(buft); +} + +static const ggml_backend_buffer_type_i ggml_backend_cuda_buffer_type_interface = { + /* .get_name = */ ggml_backend_cuda_buffer_type_get_name, + /* .alloc_buffer = */ ggml_backend_cuda_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cuda_buffer_type_get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cuda_buffer_type_get_alloc_size, + /* .is_host = */ NULL, +}; + +ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int device) { + static std::mutex mutex; + std::lock_guard lock(mutex); + + if (device >= ggml_backend_cuda_get_device_count()) { + return nullptr; + } + + static ggml_backend_buffer_type ggml_backend_cuda_buffer_types[GGML_CUDA_MAX_DEVICES]; + + static bool ggml_backend_cuda_buffer_type_initialized = false; + + if (!ggml_backend_cuda_buffer_type_initialized) { + for (int i = 0; i < ggml_backend_cuda_get_device_count(); i++) { + ggml_backend_cuda_buffer_types[i] = { + /* .iface = */ ggml_backend_cuda_buffer_type_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), i), + /* .context = */ new ggml_backend_cuda_buffer_type_context{i, GGML_CUDA_NAME + std::to_string(i)}, + }; + } + ggml_backend_cuda_buffer_type_initialized = true; + } + + return &ggml_backend_cuda_buffer_types[device]; +} + +// cuda split buffer + +static int64_t get_row_rounding(const std::array & tensor_split) { + int64_t row_rounding = 0; + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { + continue; + } + + const int cc = ggml_cuda_info().devices[id].cc; + row_rounding = std::max(row_rounding, (int64_t)get_mmq_y_host(cc)); + } + return row_rounding; +} + +static void get_row_split(int64_t * row_low, int64_t * row_high, const ggml_tensor * tensor, const std::array & tensor_split, int id) { + const int64_t nrows = ggml_nrows(tensor); + const int64_t rounding = get_row_rounding(tensor_split); + + *row_low = id == 0 ? 0 : nrows*tensor_split[id]; + *row_low -= *row_low % rounding; + + if (id == ggml_backend_cuda_get_device_count() - 1) { + *row_high = nrows; + } else { + *row_high = nrows*tensor_split[id + 1]; + *row_high -= *row_high % rounding; + } +} + +static size_t ggml_nbytes_split(const struct ggml_tensor * tensor, int nrows_split) { + static_assert(GGML_MAX_DIMS == 4, "GGML_MAX_DIMS is not 4 - update this function"); + + return nrows_split*ggml_row_size(tensor->type, tensor->ne[0]); +} + +struct ggml_backend_cuda_split_buffer_type_context { + int main_device; + std::array tensor_split; + std::string name; +}; + +struct ggml_backend_cuda_split_buffer_context { + ~ggml_backend_cuda_split_buffer_context() { + for (ggml_tensor_extra_gpu * extra : tensor_extras) { + for (int id = 0; id < GGML_CUDA_MAX_DEVICES; ++id) { + for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { + if (extra->events[id][is] != nullptr) { + CUDA_CHECK(cudaEventDestroy(extra->events[id][is])); + } + } + if (extra->data_device[id] != nullptr) { + CUDA_CHECK(cudaFree(extra->data_device[id])); + } + } + delete extra; + } + } + + std::vector tensor_extras; +}; + + +static void ggml_backend_cuda_split_buffer_free_buffer(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; + delete ctx; +} + +static void * ggml_backend_cuda_split_buffer_get_base(ggml_backend_buffer_t buffer) { + // the pointers are stored in the tensor extras, this is just a dummy address and never dereferenced + return (void *)0x1000; + + GGML_UNUSED(buffer); +} + +static enum ggml_status ggml_backend_cuda_split_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { + GGML_ASSERT(tensor->view_src == nullptr); // views of split tensors are not supported + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + + ggml_tensor_extra_gpu * extra = new ggml_tensor_extra_gpu{}; + ctx->tensor_extras.push_back(extra); + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + // FIXME: do not crash if cudaMalloc fails + // currently, init_tensor cannot fail, it needs to be fixed in ggml-backend first + ggml_cuda_set_device(id); + char * buf; + CUDA_CHECK(ggml_cuda_device_malloc((void**)&buf, size, id)); + + // set padding to 0 to avoid possible NaN values + if (size > original_size) { + CUDA_CHECK(cudaMemset(buf + original_size, 0, size - original_size)); + } + + extra->data_device[id] = buf; + + for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { + CUDA_CHECK(cudaEventCreateWithFlags(&extra->events[id][is], cudaEventDisableTiming)); + } + } + tensor->extra = extra; + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_split_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + // split tensors must always be set in their entirety at once + GGML_ASSERT(offset == 0); + GGML_ASSERT(size == ggml_nbytes(tensor)); + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + const size_t nb1 = tensor->nb[1]; + ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + const size_t offset_split = row_low*nb1; + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + const char * buf_host = (const char *)data + offset_split; + CUDA_CHECK(cudaMemcpyAsync(extra->data_device[id], buf_host, original_size, cudaMemcpyHostToDevice, cudaStreamPerThread)); + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + } +} + +static void ggml_backend_cuda_split_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + // split tensors must always be set in their entirety at once + GGML_ASSERT(offset == 0); + GGML_ASSERT(size == ggml_nbytes(tensor)); + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + const size_t nb1 = tensor->nb[1]; + ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + const size_t offset_split = row_low*nb1; + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + char * buf_host = (char *)data + offset_split; + CUDA_CHECK(cudaMemcpyAsync(buf_host, extra->data_device[id], original_size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + } +} + +static void ggml_backend_cuda_split_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { + GGML_UNUSED(buffer); + GGML_UNUSED(value); +} + +static const ggml_backend_buffer_i ggml_backend_cuda_split_buffer_interface = { + /* .free_buffer = */ ggml_backend_cuda_split_buffer_free_buffer, + /* .get_base = */ ggml_backend_cuda_split_buffer_get_base, + /* .init_tensor = */ ggml_backend_cuda_split_buffer_init_tensor, + /* .memset_tensor = */ NULL, + /* .set_tensor = */ ggml_backend_cuda_split_buffer_set_tensor, + /* .get_tensor = */ ggml_backend_cuda_split_buffer_get_tensor, + /* .set_tensor_2d = */ NULL, + /* .get_tensor_2d = */ NULL, + /* .cpy_tensor = */ NULL, + /* .clear = */ ggml_backend_cuda_split_buffer_clear, + /* .reset = */ NULL, +}; + +// cuda split buffer type + +static const char * ggml_backend_cuda_split_buffer_type_get_name(ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; + + return ctx->name.c_str(); +} + +static bool ggml_backend_buft_is_cuda_split(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_split_buffer_type_get_name; +} + +static ggml_backend_buffer_t ggml_backend_cuda_split_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + // since we don't know the exact split after rounding, we cannot allocate the device buffers at this point + // instead, we allocate them for each tensor separately in init_tensor + // however, the size still represents the maximum cumulative size of all the device buffers after the tensors are allocated, + // as returned by get_alloc_size. this limit is enforced during tensor allocation by ggml-alloc, so it must be correct. + ggml_backend_cuda_split_buffer_context * ctx = new ggml_backend_cuda_split_buffer_context(); + + return ggml_backend_buffer_init(buft, ggml_backend_cuda_split_buffer_interface, ctx, size); +} + +static size_t ggml_backend_cuda_split_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { + return 128; + + GGML_UNUSED(buft); +} + +static size_t ggml_backend_cuda_split_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { + ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + size_t total_size = 0; + + const int64_t ne0 = tensor->ne[0]; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + total_size += ggml_nbytes_split(tensor, nrows_split); + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + total_size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + } + + return total_size; +} + +static bool ggml_backend_cuda_split_buffer_type_is_host(ggml_backend_buffer_type_t buft) { + return false; + + GGML_UNUSED(buft); +} + +static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_interface = { + /* .get_name = */ ggml_backend_cuda_split_buffer_type_get_name, + /* .alloc_buffer = */ ggml_backend_cuda_split_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cuda_split_buffer_type_get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cuda_split_buffer_type_get_alloc_size, + /* .is_host = */ ggml_backend_cuda_split_buffer_type_is_host, +}; + +// Communication context for multi-GPU AllReduce during tensor parallelism. +// Created once per meta backend instance; provider is fixed at init time. +struct ggml_backend_cuda_comm_context { + ggml_cuda_allreduce_provider provider; + std::vector backends; + +#ifdef GGML_USE_NCCL + std::vector comms; // valid when provider == GGML_CUDA_ALLREDUCE_NCCL +#endif + + ggml_cuda_ar_pipeline * ar_pipeline = nullptr; // valid when provider == GGML_CUDA_ALLREDUCE_INTERNAL + + ~ggml_backend_cuda_comm_context() { +#ifdef GGML_USE_NCCL + if (provider == GGML_CUDA_ALLREDUCE_NCCL) { + for (ncclComm_t comm : comms) { + NCCL_CHECK(ncclCommDestroy(comm)); + } + } +#endif + ggml_cuda_ar_pipeline_free(ar_pipeline); + } +}; + +// Select an AllReduce provider for the given set of CUDA device IDs. +// +// Priority: +// 1. GGML_CUDA_ALLREDUCE env var ("nccl" or "internal") — explicit override. +// 2. NCCL when compiled in (GGML_USE_NCCL defined). +// 3. Internal otherwise. +// +// Future: inspect NVLink topology via cudaDeviceGetP2PAttribute() with +// cudaDevP2PAttrNativeAtomicSupported to prefer INTERNAL on PCIe-only systems +// where host-staged reduction can beat NCCL for small tensors. +static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( + const std::vector & device_ids) { + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + if (env != nullptr) { + if (strcmp(env, "internal") == 0) { + return GGML_CUDA_ALLREDUCE_INTERNAL; + } + if (strcmp(env, "nccl") == 0) { +#ifdef GGML_USE_NCCL + return GGML_CUDA_ALLREDUCE_NCCL; +#else + GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in, using internal provider\n", __func__); + return GGML_CUDA_ALLREDUCE_INTERNAL; +#endif + } + GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); + } + +#ifdef GGML_USE_NCCL + GGML_UNUSED(device_ids); + return GGML_CUDA_ALLREDUCE_NCCL; +#else + GGML_UNUSED(device_ids); + return GGML_CUDA_ALLREDUCE_INTERNAL; +#endif +} + +static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { + if (comm_ctx_v == nullptr) { + return; + } + delete static_cast(comm_ctx_v); +} + +static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { + for (size_t i = 0; i < n_backends; i++) { + if (!ggml_backend_is_cuda(backends[i])) { + return nullptr; + } + } + + std::vector dev_ids; + dev_ids.reserve(n_backends); + for (size_t i = 0; i < n_backends; i++) { + dev_ids.push_back(static_cast(backends[i]->context)->device); + } + + const ggml_cuda_allreduce_provider provider = ggml_cuda_select_allreduce_provider(dev_ids); + + auto * ret = new ggml_backend_cuda_comm_context; + ret->provider = provider; + ret->backends.assign(backends, backends + n_backends); + + switch (provider) { + case GGML_CUDA_ALLREDUCE_NCCL: { +#ifdef GGML_USE_NCCL + ret->comms.resize(n_backends); + NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); +#else + // Unreachable: ggml_cuda_select_allreduce_provider() only returns + // GGML_CUDA_ALLREDUCE_NCCL when GGML_USE_NCCL is defined. + GGML_ABORT("NCCL provider selected but NCCL not compiled in"); +#endif + } break; + + case GGML_CUDA_ALLREDUCE_INTERNAL: { + ret->ar_pipeline = ggml_cuda_ar_pipeline_init( + dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); + if (ret->ar_pipeline == nullptr) { + GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); + delete ret; + return nullptr; + } + } break; + } + + return ret; +} + +#ifdef GGML_USE_NCCL +// AllReduce via NCCL. Reduces as FP32 for small tensors and BF16 for large +// tensors (bandwidth-bound), then converts back to FP32. +static bool ggml_backend_cuda_comm_allreduce_nccl( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + const int64_t ne = ggml_nelements(tensors[0]); + // FIXME the input of llm_graph_context::build_in_out_ids can produce a tensor with 0 elements if n_outputs == 0 + // This then causes a crash in this function + if (ne == 0) { + return true; + } + + const size_t n_backends = comm_ctx->backends.size(); + + for (size_t i = 0; i < n_backends; ++i) { + GGML_ASSERT(tensors[i] != nullptr); + GGML_ASSERT(ggml_nelements(tensors[i]) == ne); + GGML_ASSERT(ggml_is_contiguously_allocated(tensors[i])); + } + + // For small tensors, simply reduce them as FP32. + // The following heuristic for how "small" a tensor should be is based on RTX 4090s connected via 16x PCIe 4.0. + if ((n_backends <= 2 && ne < 32768) || (n_backends == 3 && ne < 131072) || (n_backends >= 4 && ne < 262144)) { + for (size_t i = 0; i < n_backends; ++i) { + if ((tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + ggml_cuda_set_device(cuda_ctx->device); + CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, ggml_nbytes(tensors[i]), cuda_ctx->stream())); + } + } + NCCL_CHECK(ncclGroupStart()); + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + NCCL_CHECK(ncclAllReduce(tensors[i]->data, tensors[i]->data, ne, ncclFloat, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); + } + NCCL_CHECK(ncclGroupEnd()); + return true; + } + + // For large tensors it's faster to compress them to BF16 for the reduction: + to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); + to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); + + ggml_cuda_pool_alloc tmp[GGML_CUDA_MAX_DEVICES]; + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + tmp[i].pool = &cuda_ctx->pool(); + tmp[i].alloc(ne); + + ggml_cuda_set_device(cuda_ctx->device); + if (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) { + to_bf16(tensors[i]->data, tmp[i].get(), ne, cuda_ctx->stream()); + } else { + CUDA_CHECK(cudaMemsetAsync(tmp[i].get(), 0, ne * sizeof(nv_bfloat16), cuda_ctx->stream())); + } + CUDA_CHECK(cudaGetLastError()); + } + + NCCL_CHECK(ncclGroupStart()); + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + NCCL_CHECK(ncclAllReduce(tmp[i].get(), tmp[i].get(), ne, ncclBfloat16, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); + } + NCCL_CHECK(ncclGroupEnd()); + + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + + ggml_cuda_set_device(cuda_ctx->device); + to_fp32(tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); + CUDA_CHECK(cudaGetLastError()); + } + + return true; +} +#endif // GGML_USE_NCCL + +static bool ggml_backend_cuda_comm_allreduce_internal( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); +} + +static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { + if (comm_ctx_v == nullptr) { + return false; + } + auto * comm_ctx = static_cast(comm_ctx_v); + switch (comm_ctx->provider) { +#ifdef GGML_USE_NCCL + case GGML_CUDA_ALLREDUCE_NCCL: + return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); +#endif + case GGML_CUDA_ALLREDUCE_INTERNAL: + return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors); + default: + return false; + } +} + +ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { + static std::mutex mutex; + std::lock_guard lock(mutex); + + static std::map>, struct ggml_backend_buffer_type> buft_map; + + std::array tensor_split_arr = {}; + + bool all_zero = tensor_split == nullptr || std::all_of(tensor_split, tensor_split + GGML_CUDA_MAX_DEVICES, [](float x) { return x == 0.0f; }); + if (all_zero) { + tensor_split_arr = ggml_cuda_info().default_tensor_split; + } else { + float split_sum = 0.0f; + for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { + tensor_split_arr[i] = split_sum; + split_sum += tensor_split[i]; + } + for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { + tensor_split_arr[i] /= split_sum; + } + } + + auto it = buft_map.find({main_device, tensor_split_arr}); + if (it != buft_map.end()) { + return &it->second; + } + auto * ctx = new ggml_backend_cuda_split_buffer_type_context{ + main_device, + tensor_split_arr, + GGML_CUDA_NAME + std::to_string(main_device) + "_Split", + }; + + struct ggml_backend_buffer_type buft { + /* .iface = */ ggml_backend_cuda_split_buffer_type_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), main_device), + /* .context = */ ctx, + }; + + auto result = buft_map.emplace(std::make_pair(main_device, tensor_split_arr), buft); + return &result.first->second; +} + +// host buffer type + +static const char * ggml_backend_cuda_host_buffer_type_name(ggml_backend_buffer_type_t buft) { + return GGML_CUDA_NAME "_Host"; + + GGML_UNUSED(buft); +} + +static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; +} + +static void ggml_backend_cuda_host_buffer_free_buffer(ggml_backend_buffer_t buffer) { + CUDA_CHECK(cudaFreeHost(buffer->context)); +} + +static void * ggml_cuda_host_malloc(size_t size) { + if (getenv("GGML_CUDA_NO_PINNED") != nullptr) { + return nullptr; + } + + void * ptr = nullptr; + cudaError_t err = cudaMallocHost((void **) &ptr, size); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + GGML_LOG_DEBUG("%s: failed to allocate %.2f MiB of pinned memory: %s\n", __func__, + size / 1024.0 / 1024.0, cudaGetErrorString(err)); + return nullptr; + } + + return ptr; +} + +static ggml_backend_buffer_t ggml_backend_cuda_host_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + void * ptr = ggml_cuda_host_malloc(size); + + if (ptr == nullptr) { + // fallback to cpu buffer + return ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size); + } + + ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size); + buffer->buft = buft; + buffer->iface.free_buffer = ggml_backend_cuda_host_buffer_free_buffer; + + return buffer; +} + +ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type() { + static struct ggml_backend_buffer_type ggml_backend_cuda_buffer_type_host = { + /* .iface = */ { + /* .get_name = */ ggml_backend_cuda_host_buffer_type_name, + /* .alloc_buffer = */ ggml_backend_cuda_host_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cpu_buffer_type()->iface.get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cpu_buffer_type()->iface.get_alloc_size, + /* .is_host = */ ggml_backend_cpu_buffer_type()->iface.is_host, + }, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), 0), + /* .context = */ nullptr, + }; + + return &ggml_backend_cuda_buffer_type_host; +} + +//static bool ggml_backend_buffer_is_cuda_host(ggml_backend_buffer_t buffer) { +// return buffer->buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; +//} + +/// kernels + +typedef void (*ggml_cuda_op_mul_mat_t)( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, + const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, + const int64_t src1_padded_row_size, cudaStream_t stream); + +#ifndef GGML_CUDA_PEER_MAX_BATCH_SIZE +#define GGML_CUDA_PEER_MAX_BATCH_SIZE 128 +#endif // GGML_CUDA_PEER_MAX_BATCH_SIZE + +#define MUL_MAT_SRC1_COL_STRIDE 128 + +static cudaError_t ggml_cuda_cpy_tensor_2d( + void * dst, const struct ggml_tensor * src, int64_t i3, int64_t i2, int64_t i1_low, int64_t i1_high, cudaStream_t stream) { + + const char * src_ptr = (const char *) src->data; + char * dst_ptr = (char *) dst; + + const int64_t ne0 = src->ne[0]; + const int64_t nb0 = src->nb[0]; + const int64_t nb1 = src->nb[1]; + const int64_t nb2 = src->nb[2]; + const int64_t nb3 = src->nb[3]; + const enum ggml_type type = src->type; + const int64_t ts = ggml_type_size(type); + const int64_t bs = ggml_blck_size(type); + const int64_t i1_diff = i1_high - i1_low; + + const char * x = src_ptr + i1_low*nb1 + i2*nb2 + i3*nb3; + if (nb0 == ts && nb1 == ts*ne0/bs) { + return cudaMemcpyAsync(dst_ptr, x, i1_diff*nb1, cudaMemcpyDeviceToDevice, stream); + } else if (nb0 == ts) { + return cudaMemcpy2DAsync(dst_ptr, ts*ne0/bs, x, nb1, ts*ne0/bs, i1_diff, cudaMemcpyDeviceToDevice, stream); + } else { + for (int64_t i1 = 0; i1 < i1_diff; i1++) { + const void * rx = (const void *) ((const char *) x + i1*nb1); + void * rd = (void *) (dst_ptr + i1*ts*ne0/bs); + // pretend the row is a matrix with cols=1 + cudaError_t r = cudaMemcpy2DAsync(rd, ts/bs, rx, nb0, ts/bs, ne0, cudaMemcpyDeviceToDevice, stream); + if (r != cudaSuccess) { + return r; + } + } + return cudaSuccess; + } +} + +struct cublas_force_compute_type { + bool fp32 = false; + bool fp16 = false; +}; + +static const cublas_force_compute_type & ggml_cuda_cublas_get_force_compute_type() { + static const cublas_force_compute_type compute_type = [] { + cublas_force_compute_type result; + + const bool ggml_cuda_force_cublas_compute_32f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F") != nullptr; + const bool ggml_cuda_force_cublas_compute_16f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F") != nullptr; + + GGML_ASSERT(ggml_cuda_force_cublas_compute_16f_env == false || ggml_cuda_force_cublas_compute_32f_env == false); + + if (ggml_cuda_force_cublas_compute_32f_env) { + GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F\n"); + result.fp32 = true; + } else if (ggml_cuda_force_cublas_compute_16f_env) { + GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F\n"); + result.fp16 = true; + } + + return result; + }(); + + return compute_type; +} + +static void ggml_cuda_op_mul_mat_cublas( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, + const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, + const int64_t src1_padded_row_size, cudaStream_t stream) { + + GGML_ASSERT(src0_dd_i != nullptr); + GGML_ASSERT(src1_ddf_i != nullptr); + GGML_ASSERT(dst_dd_i != nullptr); + + const int64_t ne00 = src0->ne[0]; + const int64_t ne10 = src1->ne[0]; + + const int64_t ne0 = dst->ne[0]; + + const int64_t row_diff = row_high - row_low; + + int id = ggml_cuda_get_device(); + + // the main device has a larger memory buffer to hold the results from all GPUs + // ldc == nrows of the matrix that cuBLAS writes into + int64_t ldc = id == ctx.device ? ne0 : row_diff; + + const int cc = ggml_cuda_info().devices[id].cc; + + const bool supports_bf16 = GGML_CUDA_CC_IS_NVIDIA(cc) || GGML_CUDA_CC_IS_AMD(cc) || + (GGML_CUDA_CC_IS_MTHREADS(cc) && cc >= GGML_CUDA_CC_QY2); + + const bool use_fp16 = + src0->type != GGML_TYPE_NVFP4 && + (src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && + ggml_is_contiguous(src0) && + row_diff == src0->ne[1] && + dst->op_params[0] == GGML_PREC_DEFAULT; + + if (supports_bf16 && src0->type == GGML_TYPE_BF16 && ggml_is_contiguous(src0) && row_diff == src0->ne[1]) { + ggml_cuda_pool_alloc src1_as_bf16(ctx.pool(id)); + if (src1->type != GGML_TYPE_BF16) { + const to_bf16_cuda_t to_bf16_cuda = ggml_get_to_bf16_cuda(src1->type); + GGML_ASSERT(to_bf16_cuda != nullptr); + size_t ne = src1_ncols*ne10; + src1_as_bf16.alloc(ne); + to_bf16_cuda(src1_ddf_i, src1_as_bf16.get(), ne, stream); + } + const nv_bfloat16 * src1_ptr = src1->type == GGML_TYPE_BF16 ? (const nv_bfloat16 *) src1_ddf_i : src1_as_bf16.get(); + const nv_bfloat16 * src0_ptr = (const nv_bfloat16 *)src0_dd_i; + ggml_cuda_pool_alloc dst_bf16(ctx.pool(id), row_diff*src1_ncols); + + const float alpha_f32 = 1.0f; + const float beta_f32 = 0.0f; + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha_f32, src0_ptr, CUDA_R_16BF, ne00, + src1_ptr, CUDA_R_16BF, ne10, + &beta_f32, dst_bf16.get(), CUDA_R_16BF, ldc, + CUBLAS_COMPUTE_32F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); + to_fp32_cuda(dst_bf16.get(), dst_dd_i, row_diff*src1_ncols, stream); + } else if (fast_fp16_hardware_available(cc) && use_fp16) { + // convert src0 and src1 to fp16, multiply as fp16, convert dst to fp32 + ggml_cuda_pool_alloc src0_as_f16(ctx.pool(id)); + if (src0->type != GGML_TYPE_F16) { + const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src0->type); + GGML_ASSERT(to_fp16_cuda != nullptr); + size_t ne = row_diff*ne00; + src0_as_f16.alloc(ne); + to_fp16_cuda(src0_dd_i, src0_as_f16.get(), ne, stream); + } + const half * src0_ptr = src0->type == GGML_TYPE_F16 ? (const half *) src0_dd_i : src0_as_f16.get(); + + ggml_cuda_pool_alloc src1_as_f16(ctx.pool(id)); + if (src1->type != GGML_TYPE_F16) { + const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type); + GGML_ASSERT(to_fp16_cuda != nullptr); + size_t ne = src1_ncols*ne10; + src1_as_f16.alloc(ne); + to_fp16_cuda(src1_ddf_i, src1_as_f16.get(), ne, stream); + } + const half * src1_ptr = src1->type == GGML_TYPE_F16 ? (const half *) src1_ddf_i : src1_as_f16.get(); + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + + const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); + + if (!force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) + || GGML_CUDA_CC_IS_RDNA4(cc) + || cc == GGML_CUDA_CC_VOLTA + || force_compute_type.fp32)) + { + const float alpha = 1.0f; + const float beta = 0.0f; + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha, src0_ptr, CUDA_R_16F, ne00, + src1_ptr, CUDA_R_16F, ne10, + &beta, dst_dd_i, CUDA_R_32F, ldc, + CUBLAS_COMPUTE_32F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } else { + ggml_cuda_pool_alloc dst_f16(ctx.pool(id), row_diff*src1_ncols); + + const half alpha_f16 = 1.0f; + const half beta_f16 = 0.0f; + + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha_f16, src0_ptr, CUDA_R_16F, ne00, + src1_ptr, CUDA_R_16F, ne10, + &beta_f16, dst_f16.get(), CUDA_R_16F, ldc, + CUBLAS_COMPUTE_16F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_F16); + to_fp32_cuda(dst_f16.get(), dst_dd_i, row_diff*src1_ncols, stream); + } + } else { + ggml_cuda_pool_alloc src0_ddq_as_f32(ctx.pool(id)); + ggml_cuda_pool_alloc src1_ddq_as_f32(ctx.pool(id)); + + if (src0->type != GGML_TYPE_F32) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src0->type); + GGML_ASSERT(to_fp32_cuda != nullptr); + src0_ddq_as_f32.alloc(row_diff*ne00); + to_fp32_cuda(src0_dd_i, src0_ddq_as_f32.get(), row_diff*ne00, stream); + } + if (src1->type != GGML_TYPE_F32) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src1->type); + GGML_ASSERT(to_fp32_cuda != nullptr); + src1_ddq_as_f32.alloc(src1_ncols*ne10); + to_fp32_cuda(src1_ddf_i, src1_ddq_as_f32.get(), src1_ncols*ne10, stream); + } + + const float * src0_ddf_i = src0->type == GGML_TYPE_F32 ? (const float *) src0_dd_i : src0_ddq_as_f32.get(); + const float * src1_ddf1_i = src1->type == GGML_TYPE_F32 ? (const float *) src1_ddf_i : src1_ddq_as_f32.get(); + + const float alpha = 1.0f; + const float beta = 0.0f; + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + CUBLAS_CHECK( + cublasSgemm(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha, src0_ddf_i, ne00, + src1_ddf1_i, ne10, + &beta, dst_dd_i, ldc)); + } + + GGML_UNUSED_VARS(dst, src1_ddq_i, src1_padded_row_size); +} + +static cudaError_t ggml_cuda_Memcpy2DPeerAsync( + void * dst, int dstDevice, size_t dpitch, void * src, int srcDevice, size_t spitch, size_t width, size_t height, cudaStream_t stream) { + +#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) + // cudaMemcpy2DAsync may fail with copies between vmm pools of different devices + cudaMemcpy3DPeerParms p = {}; + p.dstDevice = dstDevice; + p.dstPtr = make_cudaPitchedPtr(dst, dpitch, dpitch, height); + p.srcDevice = srcDevice; + p.srcPtr = make_cudaPitchedPtr(src, spitch, spitch, height); + p.extent = make_cudaExtent(width, height, 1); + return cudaMemcpy3DPeerAsync(&p, stream); +#else + // HIP does not support cudaMemcpy3DPeerAsync or vmm pools + GGML_UNUSED(dstDevice); + GGML_UNUSED(srcDevice); + return cudaMemcpy2DAsync(dst, dpitch, src, spitch, width, height, cudaMemcpyDeviceToDevice, stream); +#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) +} + +static void ggml_cuda_op_mul_mat( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, ggml_cuda_op_mul_mat_t op, + quantize_cuda_t quantize_src1) { + + const int64_t ne00 = src0->ne[0]; + const int64_t ne01 = src0->ne[1]; + const int64_t ne02 = src0->ne[2]; + const int64_t ne03 = src0->ne[3]; + + const int64_t ne10 = src1->ne[0]; + const int64_t ne11 = src1->ne[1]; + const int64_t ne12 = src1->ne[2]; + const int64_t ne13 = src1->ne[3]; + const int64_t nrows1 = ggml_nrows(src1); + + const int64_t ne0 = dst->ne[0]; + const int64_t ne1 = dst->ne[1]; + + // const int64_t nb10 = src1->nb[0]; + const int64_t nb11 = src1->nb[1]; + const int64_t nb12 = src1->nb[2]; + const int64_t nb13 = src1->nb[3]; + + const int64_t nb2 = dst->nb[2]; + const int64_t nb3 = dst->nb[3]; + + ggml_backend_cuda_buffer_context * src1_ctx = (ggml_backend_cuda_buffer_context *) src1->buffer->context; + ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *) dst->buffer->context; + + GGML_ASSERT(src1->type == GGML_TYPE_F32 || (src1->ne[2] == 1 && src1->ne[3] == 1)); + + GGML_ASSERT(ne12 % ne02 == 0); + GGML_ASSERT(ne13 % ne03 == 0); + + const int64_t i02_divisor = ne12 / ne02; + const int64_t i03_divisor = ne13 / ne03; + + const size_t src0_ts = ggml_type_size(src0->type); + const size_t src0_bs = ggml_blck_size(src0->type); + const size_t q8_1_ts = sizeof(block_q8_1); + const size_t q8_1_bs = QK8_1; + + const bool src0_is_contiguous = ggml_is_contiguous(src0); + const bool src1_is_contiguous = ggml_is_contiguous(src1); + + const int64_t src1_padded_col_size = GGML_PAD(ne10, MATRIX_ROW_PADDING); + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); + GGML_ASSERT(!(split && ne02 > 1)); + GGML_ASSERT(!(split && ne03 > 1)); + GGML_ASSERT(!(split && ne02 < ne12)); + GGML_ASSERT(!(split && ne03 < ne13)); + + ggml_tensor_extra_gpu * src0_extra = split ? (ggml_tensor_extra_gpu *) src0->extra : nullptr; + + + std::array tensor_split; + if (split) { + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; + tensor_split = buft_ctx->tensor_split; + } + + struct dev_data { + int cc; + + ggml_cuda_pool_alloc src0_dd_alloc; + ggml_cuda_pool_alloc src1_ddf_alloc; + ggml_cuda_pool_alloc src1_ddq_alloc; + ggml_cuda_pool_alloc dst_dd_alloc; + + char * src0_dd = nullptr; + float * src1_ddf = nullptr; // float + char * src1_ddq = nullptr; // q8_1 + float * dst_dd = nullptr; + + int64_t row_low; + int64_t row_high; + }; + + dev_data dev[GGML_CUDA_MAX_DEVICES]; + + int used_devices = 0; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + dev[id].cc = ggml_cuda_info().devices[id].cc; + + // by default, use all rows + dev[id].row_low = 0; + dev[id].row_high = ne01; + + // for multi GPU, get the row boundaries from tensor split + // and round to mul_mat_q tile sizes + if (split) { + const int64_t rounding = get_row_rounding(tensor_split); + + if (id != 0) { + dev[id].row_low = ne01*tensor_split[id]; + if (dev[id].row_low < ne01) { + dev[id].row_low -= dev[id].row_low % rounding; + } + } + + if (id != ggml_backend_cuda_get_device_count() - 1) { + dev[id].row_high = ne01*tensor_split[id + 1]; + if (dev[id].row_high < ne01) { + dev[id].row_high -= dev[id].row_high % rounding; + } + } + } + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { + continue; + } + + used_devices++; + + const bool src1_on_device = id == src1_ctx->device; + const bool dst_on_device = id == dst_ctx->device; + + ggml_cuda_set_device(id); + cudaStream_t stream = ctx.stream(id, 0); + + if (src0_is_contiguous) { + dev[id].src0_dd = split ? (char *) src0_extra->data_device[id] : (char *) src0->data; + } else { + // If src0 is not contiguous it will be copied to a temporary buffer. + // This buffer needs to be cleared entirely because multiple regions will function as padding. + const size_t nbytes_data = ggml_nbytes(src0); + const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); + dev[id].src0_dd = dev[id].src0_dd_alloc.alloc(ctx.pool(id), nbytes_data + nbytes_padding); + CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd, 0, nbytes_data + nbytes_padding, stream)); + } + + // If src0 is on a temporary compute buffer (partial offloading) there may be some padding that needs to be cleared: + if (ne00 % MATRIX_ROW_PADDING != 0 && ggml_is_quantized(src0->type) && ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && src0->view_src == nullptr) { + GGML_ASSERT(ggml_is_contiguously_allocated(src0)); + GGML_ASSERT(!src0->view_src); + const size_t nbytes_data = ggml_row_size(src0->type, (dev[id].row_high - dev[id].row_low)*ne00); + const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); + CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd + nbytes_data, 0, nbytes_padding, stream)); + } + + if (src1_on_device && src1_is_contiguous) { + dev[id].src1_ddf = (float *) src1->data; + } else { + dev[id].src1_ddf = dev[id].src1_ddf_alloc.alloc(ctx.pool(id), ggml_nelements(src1)); + } + + if (quantize_src1) { + size_t src_1_ddq_size = nrows1*src1_padded_col_size*q8_1_ts/q8_1_bs; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + src_1_ddq_size += get_mmq_x_max_host(dev[id].cc)*sizeof(block_q8_1_mmq); + } + dev[id].src1_ddq = dev[id].src1_ddq_alloc.alloc(ctx.pool(id), src_1_ddq_size); + + if (src1_on_device && src1_is_contiguous) { + quantize_src1( + dev[id].src1_ddf, nullptr, dev[id].src1_ddq, src0->type, ne10, + nb11/sizeof(float), nb12/sizeof(float), nb13/sizeof(float), + src1_padded_col_size, ne11, ne12, ne13, stream); + CUDA_CHECK(cudaGetLastError()); + } + } + + if (dst_on_device) { + dev[id].dst_dd = (float *) dst->data; + } else { + const size_t size_dst_ddf = split ? (dev[id].row_high - dev[id].row_low)*ne1 : ggml_nelements(dst); + dev[id].dst_dd = dev[id].dst_dd_alloc.alloc(ctx.pool(id), size_dst_ddf); + } + } + + // if multiple devices are used they need to wait for the main device + // here an event is recorded that signals that the main device has finished calculating the input data + if (split && used_devices > 1) { + ggml_cuda_set_device(ctx.device); + CUDA_CHECK(cudaEventRecord(src0_extra->events[ctx.device][0], ctx.stream())); + } + + const int64_t src1_col_stride = split && used_devices > 1 ? MUL_MAT_SRC1_COL_STRIDE : ne11; + for (int64_t src1_col_0 = 0; src1_col_0 < ne11; src1_col_0 += src1_col_stride) { + const int64_t is = split ? (src1_col_0/src1_col_stride) % GGML_CUDA_MAX_STREAMS : 0; + const int64_t src1_ncols = src1_col_0 + src1_col_stride > ne11 ? ne11 - src1_col_0 : src1_col_stride; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { + continue; + } + + const bool src1_on_device = id == src1_ctx->device; + const bool dst_on_device = id == dst_ctx->device; + const int64_t row_diff = dev[id].row_high - dev[id].row_low; + + ggml_cuda_set_device(id); + cudaStream_t stream = ctx.stream(id, is); + + // wait for main GPU data if necessary + if (split && (id != ctx.device || is != 0)) { + CUDA_CHECK(cudaStreamWaitEvent(stream, src0_extra->events[ctx.device][0], 0)); + } + + for (int64_t i0 = 0; i0 < ne13*ne12; ++i0) { + const int64_t i03 = i0 / ne12; + const int64_t i02 = i0 % ne12; + + size_t src1_ddq_i_offset = i0*ne11 * src1_padded_col_size*q8_1_ts/q8_1_bs; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + src1_ddq_i_offset += src1_col_0 * sizeof(block_q8_1_mmq); + } else { + src1_ddq_i_offset += src1_col_0 * src1_padded_col_size*q8_1_ts/q8_1_bs; + } + + // for split tensors the data begins at i0 == i0_offset_low + const size_t nbytes_src0_matrix = ne01*ne00*src0_ts / src0_bs; + char * src0_dd_i = dev[id].src0_dd + ((i03/i03_divisor)*ne02 + (i02/i02_divisor)) * nbytes_src0_matrix; + float * src1_ddf_i = dev[id].src1_ddf + (i0*ne11 + src1_col_0) * ne10; + char * src1_ddq_i = dev[id].src1_ddq + src1_ddq_i_offset; + float * dst_dd_i = dev[id].dst_dd + (i0*ne1 + src1_col_0) * (dst_on_device ? ne0 : row_diff); + + // the main device memory buffer can be on VRAM scratch, with space for all partial results + // in that case an offset on dst_ddf_i is needed + if (id == ctx.device) { + dst_dd_i += dev[id].row_low; // offset is 0 if no tensor split + } + + // copy src0, src1 to device if necessary + if (src1_is_contiguous) { + if (id != ctx.device) { + if (quantize_src1) { + char * src1_ddq_i_source = dev[ctx.device].src1_ddq + src1_ddq_i_offset; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + const size_t pitch = ne11*sizeof(block_q8_1_mmq); + const size_t width = src1_ncols*sizeof(block_q8_1_mmq); + const size_t height = src1_padded_col_size/(4*QK8_1); + CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync(src1_ddq_i, id, pitch, src1_ddq_i_source, ctx.device, pitch, width, height, stream)); + } else { + CUDA_CHECK(cudaMemcpyPeerAsync( + src1_ddq_i, id, src1_ddq_i_source, ctx.device, src1_ncols*src1_padded_col_size*q8_1_ts/q8_1_bs, stream)); + } + } else { + float * src1_ddf_i_source = (float *) src1->data; + src1_ddf_i_source += (i0*ne11 + src1_col_0) * ne10; + CUDA_CHECK(cudaMemcpyPeerAsync(src1_ddf_i, id, src1_ddf_i_source, ctx.device, + src1_ncols*ne10*sizeof(float), stream)); + } + } + } else if (src1_on_device && !src1_is_contiguous) { + CUDA_CHECK(ggml_cuda_cpy_tensor_2d( + src1_ddf_i, src1, i03, i02, src1_col_0, src1_col_0+src1_ncols, stream)); + } else { + GGML_ABORT("fatal error"); + } + + if (quantize_src1 && !src1_is_contiguous) { + quantize_src1( + src1_ddf_i, nullptr, src1_ddq_i, src0->type, ne10, ne10, ne11*ne10, ne12*ne11*ne10, + src1_padded_col_size, src1_ncols, 1, 1, stream); + CUDA_CHECK(cudaGetLastError()); + } + + if (src1_col_0 == 0 && !src0_is_contiguous && i03 % i03_divisor == 0 && i02 % i02_divisor == 0) { + CUDA_CHECK(ggml_cuda_cpy_tensor_2d( + src0_dd_i, src0, i03/i03_divisor, i02/i02_divisor, dev[id].row_low, dev[id].row_high, stream)); + } + + // do the computation + op(ctx, src0, src1, dst, src0_dd_i, src1_ddf_i, src1_ddq_i, dst_dd_i, + dev[id].row_low, dev[id].row_high, src1_ncols, src1_padded_col_size, stream); + CUDA_CHECK(cudaGetLastError()); + + // copy dst to host or other device if necessary + if (!dst_on_device) { + void * dst_off_device = dst->data; + if (split) { + // src0 = weight matrix is saved as a transposed matrix for better memory layout. + // dst is NOT transposed. + // The outputs of matrix matrix multiplications can therefore NOT simply be concatenated for >1 GPU. + // Instead they need to be copied to the correct slice in ne0 = dst row index. + // If dst is a vector with ne0 == 1 then you don't have to do this but it still produces correct results. + float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); + GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); + dhf_dst_i += src1_col_0*ne0 + dev[id].row_low; + CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync( + dhf_dst_i, ctx.device, ne0*sizeof(float), dst_dd_i, id, row_diff*sizeof(float), row_diff*sizeof(float), src1_ncols, stream)); + } else { + float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); + GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); + dhf_dst_i += src1_col_0*ne0; + CUDA_CHECK(cudaMemcpyAsync(dhf_dst_i, dst_dd_i, src1_ncols*ne0*sizeof(float), cudaMemcpyDeviceToDevice, stream)); + } + } + + // add event for the main device to wait on until other device is done + if (split && (id != ctx.device || is != 0)) { + CUDA_CHECK(cudaEventRecord(src0_extra->events[id][is], stream)); + } + } + } + } + + // main device waits for all other devices to be finished + if (split && ggml_backend_cuda_get_device_count() > 1) { + int64_t is_max = (ne11 + MUL_MAT_SRC1_COL_STRIDE - 1) / MUL_MAT_SRC1_COL_STRIDE; + is_max = is_max <= GGML_CUDA_MAX_STREAMS ? is_max : GGML_CUDA_MAX_STREAMS; + + ggml_cuda_set_device(ctx.device); + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if (dev[id].row_low == dev[id].row_high) { + continue; + } + for (int64_t is = 0; is < is_max; ++is) { + CUDA_CHECK(cudaStreamWaitEvent(ctx.stream(), src0_extra->events[id][is], 0)); + } + } + } +} + +static __global__ void k_compute_batched_ptrs( + const void * src0_as_f16, const void * src1_as_f16, char * dst, + const void ** ptrs_src, void ** ptrs_dst, + int64_t ne12, int64_t ne13, + int64_t ne23, + size_t nb02, size_t nb03, + size_t nb12, size_t nb13, + size_t nbd2, size_t nbd3, + int64_t r2, int64_t r3) { + const int64_t i13 = blockIdx.x * blockDim.x + threadIdx.x; + const int64_t i12 = blockIdx.y * blockDim.y + threadIdx.y; + + if (i13 >= ne13 || i12 >= ne12) { + return; + } + + const int64_t i03 = i13 / r3; + const int64_t i02 = i12 / r2; + + ptrs_src[0*ne23 + i12 + i13*ne12] = (const char *) src0_as_f16 + i02*nb02 + i03*nb03; + ptrs_src[1*ne23 + i12 + i13*ne12] = (const char *) src1_as_f16 + i12*nb12 + i13*nb13; + ptrs_dst[0*ne23 + i12 + i13*ne12] = ( char *) dst + i12*nbd2 + i13*nbd3; +} + +// Type traits for mapping ggml types to CUDA/cuBLAS types +template +struct batched_mul_mat_traits; + +template<> +struct batched_mul_mat_traits { + using cuda_type = float; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; + static inline const cudaDataType_t data_type = CUDA_R_32F; + static inline const ggml_type ggml_type_val = GGML_TYPE_F32; + static inline const float alpha = 1.0f; + static inline const float beta = 0.0f; + static inline const void* get_alpha() { static const float val = alpha; return &val; } + static inline const void* get_beta() { static const float val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp32_nc_cuda(src_type); } +}; + +template<> +struct batched_mul_mat_traits { + using cuda_type = nv_bfloat16; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; + static inline const cudaDataType_t data_type = CUDA_R_16BF; + static inline const ggml_type ggml_type_val = GGML_TYPE_BF16; + static inline const float alpha = 1.0f; + static inline const float beta = 0.0f; + static inline const void* get_alpha() { static const float val = alpha; return &val; } + static inline const void* get_beta() { static const float val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_bf16_nc_cuda(src_type); } +}; + +template<> +struct batched_mul_mat_traits { + using cuda_type = half; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_16F; + static inline const cudaDataType_t data_type = CUDA_R_16F; + static inline const ggml_type ggml_type_val = GGML_TYPE_F16; + static inline const half alpha = 1.0; + static inline const half beta = 0.0; + static inline const void* get_alpha() { static const half val = alpha; return &val; } + static inline const void* get_beta() { static const half val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp16_nc_cuda(src_type); } +}; + +template +static void ggml_cuda_mul_mat_batched_cublas_impl(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + using traits = batched_mul_mat_traits; + using cuda_t = typename traits::cuda_type; + + GGML_ASSERT(!ggml_is_transposed(src0)); + GGML_ASSERT(!ggml_is_transposed(src1)); + GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft)); + GGML_ASSERT(src0->type == src0_type); + GGML_ASSERT(ggml_is_contiguous(dst)); + + // Byte offsets and tensor dimensions are currently used in an inconsistent way for dst. + // As long as dst is contiguous this does not matter though. + + GGML_TENSOR_BINARY_OP_LOCALS + + const int64_t ne_dst = ggml_nelements(dst); + cudaStream_t main_stream = ctx.stream(); + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(), main_stream)); + + float * dst_ddf = (float *) dst->data; + const size_t ts_src1 = ggml_type_size(src1->type); + GGML_ASSERT(nb10 == ts_src1); + int64_t s11 = nb11 / ts_src1; + int64_t s12 = nb12 / ts_src1; + int64_t s13 = nb13 / ts_src1; + + const cuda_t * src0_ptr = nullptr; + const cuda_t * src1_ptr = nullptr; + + ggml_cuda_pool_alloc src0_alloc(ctx.pool()); + ggml_cuda_pool_alloc src1_alloc(ctx.pool()); + + bool is_src0_cont_2 = ggml_is_contiguous_2(src0); + bool is_src1_cont_2 = ggml_is_contiguous_2(src1); + + // Handle src0 + src0_ptr = (const cuda_t *) src0->data; + + // Handle src1 - convert if necessary + if (src1->type == src0_type) { + src1_ptr = (const cuda_t *) src1->data; + } else { + // Convert src1 to target type using traits conversion functions + const int64_t ne_src1 = ggml_nelements(src1); + src1_alloc.alloc(ne_src1); + + const auto convert_func = traits::get_nc_converter(src1->type); + GGML_ASSERT(convert_func != nullptr); + convert_func(src1->data, src1_alloc.get(), ne10, ne11, ne12, ne13, s11, s12, s13, main_stream); + src1_ptr = src1_alloc.get(); + s11 = ne10; + s12 = ne11*s11; + s13 = ne12*s12; + + is_src1_cont_2 = true; + } + + // Setup destination buffer + ggml_cuda_pool_alloc dst_temp(ctx.pool()); + char * dst_t; + size_t nbd2 = dst->nb[2]; + size_t nbd3 = dst->nb[3]; + + cublasComputeType_t cu_compute_type = traits::compute_type; + cudaDataType_t cu_data_type = traits::data_type; + cudaDataType_t cu_data_type_a = traits::data_type; + cudaDataType_t cu_data_type_b = traits::data_type; + const void * alpha = traits::get_alpha(); + const void * beta = traits::get_beta(); + + const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); + + int id = ggml_cuda_get_device(); + const int cc = ggml_cuda_info().devices[id].cc; + static constexpr bool is_src0_type_f16 = src0_type == GGML_TYPE_F16; + + // bf16 and fp32 are already being computed in fp32 (ensure it using static_assert), + // so checking necessity of forced fp32 only for fp16 src0_type + static_assert(is_src0_type_f16 || traits::compute_type == CUBLAS_COMPUTE_32F); + + const bool need_compute_32f = is_src0_type_f16 && !force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) + || GGML_CUDA_CC_IS_RDNA4(cc) + || cc == GGML_CUDA_CC_VOLTA + || force_compute_type.fp32); + + if (dst->op_params[0] == GGML_PREC_DEFAULT && !need_compute_32f) { + if constexpr (src0_type == GGML_TYPE_F32) { + dst_t = (char *) dst_ddf; // Direct F32 output + } else { + dst_t = (char *) dst_temp.alloc(ne_dst); + nbd2 /= sizeof(float) / sizeof(cuda_t); + nbd3 /= sizeof(float) / sizeof(cuda_t); + } + } else { + dst_t = (char *) dst_ddf; + cu_compute_type = batched_mul_mat_traits::compute_type; + cu_data_type = batched_mul_mat_traits::data_type; + alpha = batched_mul_mat_traits::get_alpha(); + beta = batched_mul_mat_traits::get_beta(); + } + + GGML_ASSERT(ne12 % ne02 == 0); + GGML_ASSERT(ne13 % ne03 == 0); + + // broadcast factors + const int64_t r2 = ne12/ne02; + const int64_t r3 = ne13/ne03; + + if (r2 == 1 && r3 == 1 && is_src0_cont_2 && is_src1_cont_2) { + // with a [0, 2, 1, 3] perm. and ne02==1 the matrix strides need to be determined from dim 3: + const int64_t sma = ne02 == 1 ? nb03/nb00 : nb02/nb00; + const int64_t smb = ne12 == 1 ? s13 : s12; + + // there is no broadcast and src0, src1 are contiguous across dims 2, 3 + // use cublasGemmStridedBatchedEx + CUBLAS_CHECK( + cublasGemmStridedBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, + ne01, ne11, ne10, + alpha, src0_ptr, cu_data_type_a, nb01/nb00, sma, // strideA + src1_ptr, cu_data_type_b, s11, smb, // strideB + beta, dst_t, cu_data_type, ne0, ne1*ne0, // strideC + ne12*ne13, + cu_compute_type, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } else { + // use cublasGemmBatchedEx + const int64_t ne23 = ne12*ne13; + + ggml_cuda_pool_alloc ptrs_src(ctx.pool(), 2*ne23); + ggml_cuda_pool_alloc< void *> ptrs_dst(ctx.pool(), 1*ne23); + + size_t src1_stride_size = sizeof(cuda_t); + + const int threads_x = 16; + const int threads_y = 16; + dim3 block_dims(threads_x, threads_y); + + dim3 grid_dims( + (ne13 + threads_x - 1) / threads_x, + (ne12 + threads_y - 1) / threads_y + ); + k_compute_batched_ptrs<<>>( + src0_ptr, src1_ptr, dst_t, + ptrs_src.get(), ptrs_dst.get(), + ne12, ne13, + ne23, + nb02, nb03, + (src1->type == src0_type) ? nb12 : s12*src1_stride_size, + (src1->type == src0_type) ? nb13 : s13*src1_stride_size, + nbd2, nbd3, + r2, r3); + + CUDA_CHECK(cudaGetLastError()); + + CUBLAS_CHECK( + cublasGemmBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, + ne01, ne11, ne10, + alpha, (const void **) (ptrs_src.get() + 0*ne23), cu_data_type_a, nb01/nb00, + (const void **) (ptrs_src.get() + 1*ne23), cu_data_type_b, s11, + beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne0, + ne23, + cu_compute_type, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } + + // Convert output back to F32 if needed + if (dst->op_params[0] == GGML_PREC_DEFAULT && cu_data_type != CUDA_R_32F) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(traits::ggml_type_val); + to_fp32_cuda(dst_temp.get(), dst_ddf, ne_dst, main_stream); + } +} + +static void ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + GGML_ASSERT(src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16 || src0->type == GGML_TYPE_F32); + + switch (src0->type) { + case GGML_TYPE_F32: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + case GGML_TYPE_BF16: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + case GGML_TYPE_F16: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + default: + GGML_ABORT("Unsupported type"); + } +} + +static bool ggml_cuda_should_fuse_mul_mat(const ggml_tensor * ffn_up, + const ggml_tensor * ffn_gate, + const ggml_tensor * glu, + const ggml_tensor * ffn_up_bias = nullptr, + const ggml_tensor * ffn_gate_bias = nullptr) { + const bool has_bias = ffn_up_bias != nullptr || ffn_gate_bias != nullptr; + + if (has_bias && (!ffn_up_bias || !ffn_gate_bias)) { + return false; + } + + const bool is_mul_mat = ffn_up->op == GGML_OP_MUL_MAT && ffn_gate->op == GGML_OP_MUL_MAT && glu->op == GGML_OP_GLU; + const bool is_mul_mat_id = ffn_up->op == GGML_OP_MUL_MAT_ID && ffn_gate->op == GGML_OP_MUL_MAT_ID && glu->op == GGML_OP_GLU; + + GGML_ASSERT(ffn_up && ffn_gate && glu); + + if (!is_mul_mat && !is_mul_mat_id) { + return false; + } + + const ggml_op expected_bias_op = is_mul_mat ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (has_bias) { + if (ffn_up_bias->op != expected_bias_op || ffn_gate_bias->op != expected_bias_op) { + return false; + } + + if (glu->src[0] != ffn_gate_bias || glu->src[1] != ffn_up_bias) { + return false; + } + + if (expected_bias_op == GGML_OP_ADD) { + const bool up_has_mul = ffn_up_bias->src[0] == ffn_up || ffn_up_bias->src[1] == ffn_up; + const bool gate_has_mul = ffn_gate_bias->src[0] == ffn_gate || ffn_gate_bias->src[1] == ffn_gate; + if (!up_has_mul || !gate_has_mul) { + return false; + } + } else { // GGML_OP_ADD_ID + if (ffn_up_bias->src[0] != ffn_up || ffn_gate_bias->src[0] != ffn_gate) { + return false; + } + if (ffn_up_bias->src[2] != ffn_up->src[2] || ffn_gate_bias->src[2] != ffn_gate->src[2]) { + return false; + } + } + } else { + if (glu->src[0] != ffn_gate && glu->src[1] != ffn_up) { + return false; + } + } + + if (ffn_up->src[0]->type != ffn_gate->src[0]->type || !ggml_are_same_shape(ffn_up->src[0], ffn_gate->src[0]) || + !ggml_are_same_stride(ffn_up->src[0], ffn_gate->src[0])) { + return false; + } + + if (ffn_up->src[1] != ffn_gate->src[1]) { + return false; + } + + if (ffn_up->src[2] && (ffn_up->src[2] != ffn_gate->src[2])) { + return false; + } + + static constexpr std::array valid_glu_ops = { GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU, GGML_GLU_OP_SWIGLU_OAI }; + + if (std::find(valid_glu_ops.begin(), valid_glu_ops.end(), ggml_get_glu_op(glu)) == valid_glu_ops.end()) { + return false; + } + + if (const bool swapped = ggml_get_op_params_i32(glu, 1); swapped) { + return false; + } + + const bool split = ggml_backend_buft_is_cuda_split(ffn_up->src[0]->buffer->buft) || + ggml_backend_buft_is_cuda_split(ffn_gate->src[0]->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + return true; +} + +static bool ggml_cuda_should_fuse_mul_mat_vec_f(const ggml_tensor * tensor) { + ggml_tensor * src0 = tensor->src[0]; + ggml_tensor * src1 = tensor->src[1]; + const ggml_tensor * dst = tensor; + + const bool is_mul_mat_id = tensor->op == GGML_OP_MUL_MAT_ID; + + bool use_mul_mat_vec_f = + (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) && + src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, is_mul_mat_id ? src1->ne[2] : src1->ne[1]); + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || + ggml_backend_buft_is_cuda_split(src1->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + //we only support fusion for ncols_dst = 1 + if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { + return false; + } + + if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { + return false; + } + + + return use_mul_mat_vec_f; +} + +static bool ggml_cuda_should_fuse_mul_mat_vec_q(const ggml_tensor * tensor) { + ggml_tensor * src0 = tensor->src[0]; + ggml_tensor * src1 = tensor->src[1]; + const ggml_tensor * dst = tensor; + + const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && + ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && + src0->view_src; + + bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear && src1->type == GGML_TYPE_F32 && + dst->type == GGML_TYPE_F32 && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; + + // fusion is not universally faster on Pascal + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + if (cc <= GGML_CUDA_CC_PASCAL) { + return false; + } + //we only support fusion for ncols_dst = 1 + if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { + return false; + } + + if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { + return false; + } + + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || + ggml_backend_buft_is_cuda_split(src1->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + return use_mul_mat_vec_q; +} + +static void ggml_cuda_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); + + // If src0 is a temporary compute buffer it may have some padding that needs to be cleared for mul_mat_vec_q or mul_mat_q. + // But if src0 is also a view of another tensor then this cannot be done safely because it may overwrite valid tensor data. + // Therefore, in such cases use cuBLAS. + const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE + && ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && src0->view_src; + + bool use_mul_mat_vec_f = (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + bool use_mul_mat_f = !ggml_is_quantized(src0->type) + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32 + && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; + bool use_mul_mat_q = ggml_is_quantized(src0->type) && !bad_padding_clear + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + + bool any_gpus_with_slow_fp16 = false; + + if (split) { + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; + auto & tensor_split = buft_ctx->tensor_split; + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + // skip devices that are not going to do any work: + if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { + continue; + } + + const int cc = ggml_cuda_info().devices[id].cc; + const int warp_size = ggml_cuda_info().devices[id].warp_size; + use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); + use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); + any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); + } + } else { + const int cc = ggml_cuda_info().devices[ctx.device].cc; + const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size; + use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); + use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); + any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); + } + + // debug helpers + //printf("src0: %8d %8d %8d %8d\n", src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3]); + //printf(" %8d %8d %8d %8d\n", src0->nb[0], src0->nb[1], src0->nb[2], src0->nb[3]); + //printf("src1: %8d %8d %8d %8d\n", src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3]); + //printf(" %8d %8d %8d %8d\n", src1->nb[0], src1->nb[1], src1->nb[2], src1->nb[3]); + //printf("src0 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src0), ggml_is_transposed(src0), ggml_type_name(src0->type), src0->name); + //printf("src1 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src1), ggml_is_transposed(src1), ggml_type_name(src1->type), src1->name); + + //TODO update for generic tensor parallelism + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + bool use_batched_cublas_f16 = src0->type == GGML_TYPE_F16 && (src1->type == GGML_TYPE_F16 || !any_gpus_with_slow_fp16); + bool use_batched_cublas_bf16 = src0->type == GGML_TYPE_BF16 && bf16_mma_hardware_available(cc); + bool use_batched_cublas_f32 = src0->type == GGML_TYPE_F32; + + if (!split && use_mul_mat_vec_f) { + // the custom F16 vector kernel can be used over batched cuBLAS GEMM + // but this is only faster for GPUs without tensor cores or with a thin src0 matrix (particularly KQV in attention) + ggml_cuda_mul_mat_vec_f(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_f) { + ggml_cuda_mul_mat_f(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_vec_q) { + ggml_cuda_mul_mat_vec_q(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_q) { + ggml_cuda_mul_mat_q(ctx, src0, src1, nullptr, dst); + } else if (!split && (use_batched_cublas_f16 || use_batched_cublas_bf16 || use_batched_cublas_f32) + && !ggml_is_transposed(src0) && !ggml_is_transposed(src1) && src1->ne[2]*src1->ne[3] > 1) { + // general KQ + KQV multi-batch without FlashAttention + ggml_cuda_mul_mat_batched_cublas(ctx, src0, src1, dst); + } else if (use_mul_mat_vec_f) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_f, nullptr); + } else if (use_mul_mat_vec_q) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_q, quantize_row_q8_1_cuda); + } else if (use_mul_mat_q) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_q, quantize_mmq_q8_1_cuda); + } else { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_cublas, nullptr); + } +} + +static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor * dst) { + const ggml_tensor * src0 = dst->src[0]; + const ggml_tensor * src1 = dst->src[1]; + const ggml_tensor * ids = dst->src[2]; + + GGML_ASSERT(src1->type == GGML_TYPE_F32); + GGML_ASSERT(dst->type == GGML_TYPE_F32); + GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft) && "mul_mat_id does not support split buffers"); + + GGML_TENSOR_BINARY_OP_LOCALS + + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + + // [TAG_MUL_MAT_ID_CUDA_GRAPHS] + if (src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { + static_assert(MMVQ_MAX_BATCH_SIZE == MMVF_MAX_BATCH_SIZE); + if (ne2 <= MMVQ_MAX_BATCH_SIZE) { + if (ggml_is_quantized(src0->type)) { + const int mmvq_mmid_max = get_mmvq_mmid_max_batch(src0->type, cc); + if (ne2 <= mmvq_mmid_max) { + ggml_cuda_mul_mat_vec_q(ctx, src0, src1, ids, dst); + return; + } + } else { + if (GGML_CUDA_CC_IS_AMD(cc)) { + ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst); + return; + } + } + } + + if (ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) { + ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst); + return; + } + + if (ggml_cuda_should_use_mmf(src0->type, cc, WARP_SIZE, src0->ne, src0->nb, src1->ne[2], /*mul_mat_id=*/true)) { + ggml_cuda_mul_mat_f(ctx, src0, src1, ids, dst); + return; + } + } + + // note: this path should not be reached when recording CUDA graphs, because it requires stream synchronization + // TODO: add asserts to verify this. should work with CUDA, HIP, etc. + cudaStream_t stream = ctx.stream(); + + GGML_ASSERT(nb12 % nb11 == 0); + GGML_ASSERT(nb2 % nb1 == 0); + + const ggml_type type_src1_sorted = (src0->type == GGML_TYPE_F16 && !fast_fp16_hardware_available(cc)) + || ggml_is_quantized(src0->type) ? GGML_TYPE_F32 : src0->type; + const ggml_type type_dst_sorted = GGML_TYPE_F32; + const size_t ts_src1_sorted = ggml_type_size(type_src1_sorted); + const size_t ts_dst_sorted = ggml_type_size(type_dst_sorted); + + const int64_t n_expert_used = ids->ne[0]; + const int64_t ne_get_rows = ne12 * n_expert_used; + + std::vector ids_to_sorted_host; + ids_to_sorted_host.reserve(2*ne_get_rows); + std::vector ids_from_sorted_host(ne_get_rows); + + ggml_cuda_pool_alloc ids_buf_dev(ctx.pool(), 2*ne_get_rows); + + std::vector tokens_per_expert(ne02); + + ggml_cuda_pool_alloc src1_sorted(ctx.pool(), ne12*n_expert_used*ne10*ts_src1_sorted); + ggml_cuda_pool_alloc dst_sorted(ctx.pool(), ne2 *n_expert_used* ne0*ts_dst_sorted); + + std::vector ids_host(ggml_nbytes(ids)); + CUDA_CHECK(cudaMemcpyAsync(ids_host.data(), ids->data, ggml_nbytes(ids), cudaMemcpyDeviceToHost, stream)); + CUDA_CHECK(cudaStreamSynchronize(stream)); + + for (int64_t i02 = 0; i02 < ne02; ++i02) { // expert matrices + for (int64_t i12 = 0; i12 < ne12; ++i12) { // tokens + for (int64_t iex = 0; iex < n_expert_used; ++iex) { + const int32_t expert_to_use = *(const int32_t *)(ids_host.data() + i12*ids->nb[1] + iex*ids->nb[0]); + assert(expert_to_use >= 0 && expert_to_use < ne02); + if (expert_to_use == i02) { + ids_from_sorted_host[i12*n_expert_used + iex] = ids_to_sorted_host.size(); + ids_to_sorted_host.push_back(i12*ne11 + iex % ne11); + tokens_per_expert[i02]++; + break; + } + } + } + } + GGML_ASSERT(ids_to_sorted_host.size() == size_t(ne_get_rows)); + + ids_to_sorted_host.insert(ids_to_sorted_host.end(), ids_from_sorted_host.begin(), ids_from_sorted_host.end()); + + CUDA_CHECK(cudaMemcpyAsync(ids_buf_dev.ptr, ids_to_sorted_host.data(), 2*ne_get_rows*sizeof(int32_t), cudaMemcpyHostToDevice, stream)); + CUDA_CHECK(cudaStreamSynchronize(stream)); + + const int32_t * ids_to_sorted = ids_buf_dev.ptr + 0*ne_get_rows; + const int32_t * ids_from_sorted = ids_buf_dev.ptr + 1*ne_get_rows; + + get_rows_cuda(src1->data, src1->type, ids_to_sorted, src1_sorted.ptr, type_src1_sorted, + ne10, nb11, nb12, nb13, + ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), + ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, stream); + CUDA_CHECK(cudaGetLastError()); + + char * src1_data_cur = (char *) src1_sorted.ptr; + char * dst_data_cur = (char *) dst_sorted.ptr; + for (int64_t i02 = 0; i02 < ne02; ++i02) { + if (tokens_per_expert[i02] == 0) { + continue; + } + + ggml_tensor src0_slice = *src0; + src0_slice.ne[2] = 1; + src0_slice.nb[3] = src0_slice.nb[2]; + src0_slice.op = GGML_OP_VIEW; + src0_slice.view_src = dst->src[0]; // non-const pointer to src0 + src0_slice.data = (char *) src0->data + i02*nb02; + + ggml_tensor src1_slice; + memset(&src1_slice, 0, sizeof(src1_slice)); + src1_slice.buffer = src1->buffer; + src1_slice.type = type_src1_sorted; + src1_slice.ne[0] = ne10; + src1_slice.ne[1] = tokens_per_expert[i02]; + src1_slice.ne[2] = 1; + src1_slice.ne[3] = 1; + src1_slice.nb[0] = ts_src1_sorted; + src1_slice.nb[1] = src1_slice.ne[0] * src1_slice.nb[0]; + src1_slice.nb[2] = src1_slice.ne[1] * src1_slice.nb[1]; + src1_slice.nb[3] = src1_slice.ne[2] * src1_slice.nb[2]; + src1_slice.data = src1_data_cur; + + ggml_tensor dst_slice; + memset(&dst_slice, 0, sizeof(dst_slice)); + dst_slice.buffer = dst->buffer; + dst_slice.type = type_dst_sorted; + dst_slice.ne[0] = ne0; + dst_slice.ne[1] = tokens_per_expert[i02]; + dst_slice.ne[2] = 1; + dst_slice.ne[3] = 1; + dst_slice.nb[0] = ts_dst_sorted; + dst_slice.nb[1] = dst_slice.ne[0] * dst_slice.nb[0]; + dst_slice.nb[2] = dst_slice.ne[1] * dst_slice.nb[1]; + dst_slice.nb[3] = dst_slice.ne[2] * dst_slice.nb[2]; + dst_slice.data = dst_data_cur; + + ggml_cuda_mul_mat(ctx, &src0_slice, &src1_slice, &dst_slice); + CUDA_CHECK(cudaGetLastError()); + + src1_data_cur += src1_slice.nb[2]; + dst_data_cur += dst_slice.nb[2]; + } + + get_rows_cuda(dst_sorted.ptr, type_dst_sorted, ids_from_sorted, dst->data, dst->type, + ne0, ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, + ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), + nb1, nb2, nb3, stream); +} + +static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct ggml_tensor * dst) { + switch (dst->op) { + case GGML_OP_ARGMAX: + ggml_cuda_argmax(ctx, dst); + break; + case GGML_OP_COUNT_EQUAL: + ggml_cuda_count_equal(ctx, dst); + break; + case GGML_OP_REPEAT: + ggml_cuda_op_repeat(ctx, dst); + break; + case GGML_OP_REPEAT_BACK: + ggml_cuda_op_repeat_back(ctx, dst); + break; + case GGML_OP_GET_ROWS: + ggml_cuda_op_get_rows(ctx, dst); + break; + case GGML_OP_GET_ROWS_BACK: + ggml_cuda_op_get_rows_back(ctx, dst); + break; + case GGML_OP_SET_ROWS: + ggml_cuda_op_set_rows(ctx, dst); + break; + case GGML_OP_SET: + ggml_cuda_op_set(ctx, dst); + break; + case GGML_OP_DUP: + ggml_cuda_dup(ctx, dst); + break; + case GGML_OP_CPY: + ggml_cuda_cpy(ctx, dst->src[0], dst->src[1]); + break; + case GGML_OP_CONT: + ggml_cuda_dup(ctx, dst); + break; + case GGML_OP_ADD: + case GGML_OP_ADD1: // TODO: more efficient implementation + ggml_cuda_op_add(ctx, dst); + break; + case GGML_OP_ADD_ID: + ggml_cuda_op_add_id(ctx, dst); + break; + case GGML_OP_SUB: + ggml_cuda_op_sub(ctx, dst); + break; + case GGML_OP_ACC: + ggml_cuda_op_acc(ctx, dst); + break; + case GGML_OP_MUL: + ggml_cuda_op_mul(ctx, dst); + break; + case GGML_OP_DIV: + ggml_cuda_op_div(ctx, dst); + break; + case GGML_OP_UNARY: + switch (ggml_get_unary_op(dst)) { + case GGML_UNARY_OP_ABS: + ggml_cuda_op_abs(ctx, dst); + break; + case GGML_UNARY_OP_SGN: + ggml_cuda_op_sgn(ctx, dst); + break; + case GGML_UNARY_OP_NEG: + ggml_cuda_op_neg(ctx, dst); + break; + case GGML_UNARY_OP_STEP: + ggml_cuda_op_step(ctx, dst); + break; + case GGML_UNARY_OP_GELU: + ggml_cuda_op_gelu(ctx, dst); + break; + case GGML_UNARY_OP_SILU: + ggml_cuda_op_silu(ctx, dst); + break; + case GGML_UNARY_OP_GELU_ERF: + ggml_cuda_op_gelu_erf(ctx, dst); + break; + case GGML_UNARY_OP_GELU_QUICK: + ggml_cuda_op_gelu_quick(ctx, dst); + break; + case GGML_UNARY_OP_TANH: + ggml_cuda_op_tanh(ctx, dst); + break; + case GGML_UNARY_OP_RELU: + ggml_cuda_op_relu(ctx, dst); + break; + case GGML_UNARY_OP_SIGMOID: + ggml_cuda_op_sigmoid(ctx, dst); + break; + case GGML_UNARY_OP_HARDSIGMOID: + ggml_cuda_op_hardsigmoid(ctx, dst); + break; + case GGML_UNARY_OP_HARDSWISH: + ggml_cuda_op_hardswish(ctx, dst); + break; + case GGML_UNARY_OP_EXP: + ggml_cuda_op_exp(ctx, dst); + break; + case GGML_UNARY_OP_ELU: + ggml_cuda_op_elu(ctx, dst); + break; + case GGML_UNARY_OP_XIELU: + ggml_cuda_op_xielu(ctx, dst); + break; + case GGML_UNARY_OP_FLOOR: + ggml_cuda_op_floor(ctx, dst); + break; + case GGML_UNARY_OP_CEIL: + ggml_cuda_op_ceil(ctx, dst); + break; + case GGML_UNARY_OP_ROUND: + ggml_cuda_op_round(ctx, dst); + break; + case GGML_UNARY_OP_TRUNC: + ggml_cuda_op_trunc(ctx, dst); + break; + case GGML_UNARY_OP_EXPM1: + ggml_cuda_op_expm1(ctx, dst); + break; + case GGML_UNARY_OP_SOFTPLUS: + ggml_cuda_op_softplus(ctx, dst); + break; + default: + return false; + } + break; + case GGML_OP_GLU: + switch (ggml_get_glu_op(dst)) { + case GGML_GLU_OP_REGLU: + ggml_cuda_op_reglu(ctx, dst); + break; + case GGML_GLU_OP_GEGLU: + ggml_cuda_op_geglu(ctx, dst); + break; + case GGML_GLU_OP_SWIGLU: + ggml_cuda_op_swiglu(ctx, dst); + break; + case GGML_GLU_OP_SWIGLU_OAI: + ggml_cuda_op_swiglu_oai(ctx, dst); + break; + case GGML_GLU_OP_GEGLU_ERF: + ggml_cuda_op_geglu_erf(ctx, dst); + break; + case GGML_GLU_OP_GEGLU_QUICK: + ggml_cuda_op_geglu_quick(ctx, dst); + break; + default: + return false; + } + break; + case GGML_OP_NORM: + ggml_cuda_op_norm(ctx, dst); + break; + case GGML_OP_GROUP_NORM: + ggml_cuda_op_group_norm(ctx, dst); + break; + case GGML_OP_L2_NORM: + ggml_cuda_op_l2_norm(ctx, dst); + break; + case GGML_OP_CONCAT: + ggml_cuda_op_concat(ctx, dst); + break; + case GGML_OP_UPSCALE: + ggml_cuda_op_upscale(ctx, dst); + break; + case GGML_OP_PAD: + ggml_cuda_op_pad(ctx, dst); + break; + case GGML_OP_PAD_REFLECT_1D: + ggml_cuda_op_pad_reflect_1d(ctx, dst); + break; + case GGML_OP_ARANGE: + ggml_cuda_op_arange(ctx, dst); + break; + case GGML_OP_TIMESTEP_EMBEDDING: + ggml_cuda_op_timestep_embedding(ctx, dst); + break; + case GGML_OP_LEAKY_RELU: + ggml_cuda_op_leaky_relu(ctx, dst); + break; + case GGML_OP_SILU_BACK: + ggml_cuda_op_silu_back(ctx, dst); + break; + case GGML_OP_RMS_NORM: + ggml_cuda_op_rms_norm(ctx, dst); + break; + case GGML_OP_RMS_NORM_BACK: + ggml_cuda_op_rms_norm_back(ctx, dst); + break; + case GGML_OP_MUL_MAT: + ggml_cuda_mul_mat(ctx, dst->src[0], dst->src[1], dst); + break; + case GGML_OP_MUL_MAT_ID: + ggml_cuda_mul_mat_id(ctx, dst); + break; + case GGML_OP_OUT_PROD: + ggml_cuda_out_prod(ctx, dst); + break; + case GGML_OP_SCALE: + ggml_cuda_op_scale(ctx, dst); + break; + case GGML_OP_SQR: + ggml_cuda_op_sqr(ctx, dst); + break; + case GGML_OP_SQRT: + ggml_cuda_op_sqrt(ctx, dst); + break; + case GGML_OP_SIN: + ggml_cuda_op_sin(ctx, dst); + break; + case GGML_OP_COS: + ggml_cuda_op_cos(ctx, dst); + break; + case GGML_OP_CLAMP: + ggml_cuda_op_clamp(ctx, dst); + break; + case GGML_OP_LOG: + ggml_cuda_op_log(ctx, dst); + break; + case GGML_OP_NONE: + case GGML_OP_RESHAPE: + case GGML_OP_VIEW: + case GGML_OP_PERMUTE: + case GGML_OP_TRANSPOSE: + break; + case GGML_OP_DIAG: + ggml_cuda_op_diag(ctx, dst); + break; + case GGML_OP_DIAG_MASK_INF: + ggml_cuda_op_diag_mask_inf(ctx, dst); + break; + case GGML_OP_SOFT_MAX: + ggml_cuda_op_soft_max(ctx, dst); + break; + case GGML_OP_SOFT_MAX_BACK: + ggml_cuda_op_soft_max_back(ctx, dst); + break; + case GGML_OP_ROPE: + ggml_cuda_op_rope(ctx, dst); + break; + case GGML_OP_ROPE_BACK: + ggml_cuda_op_rope_back(ctx, dst); + break; + case GGML_OP_ROLL: + ggml_cuda_op_roll(ctx, dst); + break; + case GGML_OP_IM2COL: + ggml_cuda_op_im2col(ctx, dst); + break; + case GGML_OP_IM2COL_3D: + ggml_cuda_op_im2col_3d(ctx, dst); + break; + case GGML_OP_CONV_2D: + ggml_cuda_op_conv2d(ctx, dst); + break; + case GGML_OP_CONV_2D_DW: + ggml_cuda_op_conv2d_dw(ctx, dst); + break; + case GGML_OP_CONV_TRANSPOSE_2D: + ggml_cuda_conv_2d_transpose_p0(ctx, dst); + break; + case GGML_OP_CONV_TRANSPOSE_1D: + ggml_cuda_op_conv_transpose_1d(ctx,dst); + break; + case GGML_OP_POOL_2D: + ggml_cuda_op_pool2d(ctx, dst); + break; + case GGML_OP_SUM: + ggml_cuda_op_sum(ctx, dst); + break; + case GGML_OP_CUMSUM: + ggml_cuda_op_cumsum(ctx, dst); + break; + case GGML_OP_SUM_ROWS: + ggml_cuda_op_sum_rows(ctx, dst); + break; + case GGML_OP_MEAN: + ggml_cuda_op_mean(ctx, dst); + break; + case GGML_OP_SSM_CONV: + ggml_cuda_op_ssm_conv(ctx, dst); + break; + case GGML_OP_SSM_SCAN: + ggml_cuda_op_ssm_scan(ctx, dst); + break; + case GGML_OP_TOP_K: + ggml_cuda_op_top_k(ctx, dst); + break; + case GGML_OP_ARGSORT: + ggml_cuda_op_argsort(ctx, dst); + break; + case GGML_OP_FLASH_ATTN_EXT: + ggml_cuda_flash_attn_ext(ctx, dst); + break; + case GGML_OP_CROSS_ENTROPY_LOSS: + ggml_cuda_cross_entropy_loss(ctx, dst); + break; + case GGML_OP_TRI: + ggml_cuda_op_tri(ctx, dst); + break; + case GGML_OP_RWKV_WKV6: + ggml_cuda_op_rwkv_wkv6(ctx, dst); + break; + case GGML_OP_GATED_LINEAR_ATTN: + ggml_cuda_op_gated_linear_attn(ctx, dst); + break; + case GGML_OP_GATED_DELTA_NET: + ggml_cuda_op_gated_delta_net(ctx, dst); + break; + case GGML_OP_RWKV_WKV7: + ggml_cuda_op_rwkv_wkv7(ctx, dst); + break; + case GGML_OP_CROSS_ENTROPY_LOSS_BACK: + ggml_cuda_cross_entropy_loss_back(ctx, dst); + break; + case GGML_OP_OPT_STEP_ADAMW: + ggml_cuda_opt_step_adamw(ctx, dst); + break; + case GGML_OP_OPT_STEP_SGD: + ggml_cuda_opt_step_sgd(ctx, dst); + break; + case GGML_OP_SOLVE_TRI: + ggml_cuda_op_solve_tri(ctx, dst); + break; + case GGML_OP_FILL: + ggml_cuda_op_fill(ctx, dst); + break; + default: + return false; + } + + cudaError_t err = cudaGetLastError(); + if (err != cudaSuccess) { + GGML_LOG_ERROR("%s: %s failed\n", __func__, ggml_op_desc(dst)); + CUDA_CHECK(err); + } + + return true; +} + +//////////////////////////////////////////////////////////////////////////////// + +// backend + +static const char * ggml_backend_cuda_get_name(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + return cuda_ctx->name.c_str(); +} + +static void ggml_backend_cuda_free(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + delete cuda_ctx; + delete backend; +} + +static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpy2DAsync( + (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpy2DAsync( + data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream())); +} + +static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) { + ggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer; + ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer; + + if (!ggml_backend_is_cuda(backend_src) || !ggml_backend_is_cuda(backend_dst)) { + return false; + } + + if (!ggml_backend_buffer_is_cuda(buf_src) || !ggml_backend_buffer_is_cuda(buf_dst)) { + return false; + } + + // device -> device copy + ggml_backend_cuda_context * cuda_ctx_src = (ggml_backend_cuda_context *) backend_src->context; + ggml_backend_cuda_context * cuda_ctx_dst = (ggml_backend_cuda_context *) backend_dst->context; + + ggml_backend_cuda_buffer_context * buf_ctx_src = (ggml_backend_cuda_buffer_context *) buf_src->context; + ggml_backend_cuda_buffer_context * buf_ctx_dst = (ggml_backend_cuda_buffer_context *) buf_dst->context; + + if (cuda_ctx_src->device != buf_ctx_src->device || cuda_ctx_dst->device != buf_ctx_dst->device) { +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: backend and buffer devices do not match\n", __func__); +#endif // NDEBUG + return false; + } + + if (backend_src != backend_dst) { + // copy on src stream + if (cuda_ctx_src->device == cuda_ctx_dst->device) { + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); + } else { +#ifdef GGML_CUDA_NO_PEER_COPY + return false; +#else + CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream())); +#endif // GGML_CUDA_NO_PEER_COPY + } + + // record event on src stream after the copy + if (!cuda_ctx_src->copy_event) { + ggml_cuda_set_device(cuda_ctx_src->device); + CUDA_CHECK(cudaEventCreateWithFlags(&cuda_ctx_src->copy_event, cudaEventDisableTiming)); + } + + CUDA_CHECK(cudaEventRecord(cuda_ctx_src->copy_event, cuda_ctx_src->stream())); + + // wait on dst stream for the copy to complete + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx_dst->stream(), cuda_ctx_src->copy_event, 0)); + } else { + // src and dst are on the same backend + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); + } + return true; +} + +static void ggml_backend_cuda_synchronize(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream())); + + GGML_UNUSED(backend); +} + +#ifdef USE_CUDA_GRAPH +static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) { + + bool use_cuda_graph = true; + // Loop over nodes in GGML graph to obtain info needed for CUDA graph + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_tensor * node = cgraph->nodes[i]; + + if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { + continue; + } + + if (node->src[0] && node->src[0]->buffer && ggml_backend_buft_is_cuda_split(node->src[0]->buffer->buft)) { + use_cuda_graph = false; // Split buffers are not supported by CUDA graph capture +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to split buffer\n", __func__); +#endif + } + + // [TAG_MUL_MAT_ID_CUDA_GRAPHS] + if (node->op == GGML_OP_MUL_MAT_ID) { + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + const int mmvq_mmid_max = get_mmvq_mmid_max_batch(node->src[0]->type, cc); + if (!ggml_is_quantized(node->src[0]->type) || node->ne[2] > mmvq_mmid_max) { + // under these conditions, the mul_mat_id operation will need to synchronize the stream, so we cannot use CUDA graphs + // TODO: figure out a way to enable for larger batch sizes, without hurting performance + // ref: https://github.com/ggml-org/llama.cpp/pull/18958 + use_cuda_graph = false; +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to unsupported node type\n", __func__); +#endif + } + } + + if (!use_cuda_graph) { + break; + } + } + + return use_cuda_graph; +} + +static const void * ggml_cuda_graph_get_key(ggml_cgraph * cgraph) { + return cgraph->nodes[0]; +} + +static bool ggml_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph) { + bool res = false; + + const void * graph_key = ggml_cuda_graph_get_key(cgraph); + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + + if (cgraph->uid != 0 && + cgraph->uid == graph->uid) { + GGML_LOG_DEBUG("CUDA Graph id %zu reused\n", cgraph->uid); + GGML_ASSERT((int)graph->node_props.size() == cgraph->n_nodes); + return false; + } + + graph->uid = cgraph->uid; + + // Check if the graph size has changed + if ((int)graph->node_props.size() != cgraph->n_nodes) { + res = true; + graph->node_props.resize(cgraph->n_nodes); + } + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_cuda_graph::node_properties prop = {}; + memcpy(&prop.node, cgraph->nodes[i], sizeof(ggml_tensor)); + + for (int j = 0; j < GGML_MAX_SRC; ++j) { + if (cgraph->nodes[i]->src[j]) { + prop.node_src_data_ptrs[j] = cgraph->nodes[i]->src[j]->data; + memcpy(prop.node_src_ne[j], cgraph->nodes[i]->src[j]->ne, sizeof(prop.node_src_ne[j])); + memcpy(prop.node_src_nb[j], cgraph->nodes[i]->src[j]->nb, sizeof(prop.node_src_nb[j])); + } + } + + if (res || memcmp(&graph->node_props[i], &prop, sizeof(prop)) != 0) { + graph->node_props[i] = prop; + res = true; + } + } + + return res; +} + +static void ggml_cuda_graph_update_executable(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + +#if CUDART_VERSION >= 12000 + cudaGraphExecUpdateResultInfo result_info; + cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &result_info); +#else + cudaGraphNode_t errorNode; + cudaGraphExecUpdateResult result_info; + cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &errorNode, &result_info); +#endif // CUDART_VERSION >= 12000 + + if (stat == cudaErrorGraphExecUpdateFailure) { +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: CUDA graph update failed\n", __func__); +#endif + + // The pre-existing graph exec cannot be updated due to violated constraints + // so instead clear error and re-instantiate + (void)cudaGetLastError(); + CUDA_CHECK(cudaGraphExecDestroy(graph->instance)); + graph->instance = nullptr; + CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); + } else { + GGML_ASSERT(stat == cudaSuccess); + } +} +#endif // USE_CUDA_GRAPH + +static bool ggml_cuda_should_fuse_rope_set_rows(const ggml_tensor * rope, + const ggml_tensor * view, + const ggml_tensor * set_rows) { + + if (rope->op != GGML_OP_ROPE || view->op != GGML_OP_VIEW || set_rows->op != GGML_OP_SET_ROWS) { + return false; + } + // ne3 not tested + if (rope->src[0]->ne[3] != 1) { + return false; + } + + if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16) { + return false; + } + + if (set_rows->src[1]->type != GGML_TYPE_I64) { + return false; + } + + // The view should flatten two dims of rope into one dim + if (!ggml_is_contiguous(view) || view->ne[0] != rope->ne[0] * rope->ne[1]) { + return false; + } + + // Only norm/neox shaders have the fusion code + const int mode = ((const int32_t *) rope->op_params)[2]; + if (mode != GGML_ROPE_TYPE_NORMAL && mode != GGML_ROPE_TYPE_NEOX) { + return false; + } + + return true; +} + +static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) { + args.sigmoid = false; + args.softmax = false; + args.delayed_softmax = false; + args.prob_bias = false; + args.norm = false; + + const int n_nodes = cgraph->n_nodes; + ggml_tensor ** nodes = cgraph->nodes; + + if (nodes[node_idx]->op == GGML_OP_SOFT_MAX) { + args.softmax = true; + } + + if (nodes[node_idx]->op == GGML_OP_UNARY) { + if (ggml_get_unary_op(nodes[node_idx]) != GGML_UNARY_OP_SIGMOID) { + return false; + } + args.sigmoid = true; + } + + if (nodes[node_idx]->op == GGML_OP_ARGSORT) { + args.delayed_softmax = true; + } + + node_idx++; + + if (args.sigmoid || args.softmax) { + // SOFTMAX -> RESHAPE + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + ggml_tensor * probs_reshaped = nodes[node_idx]; + node_idx++; + + if (node_idx >= n_nodes) { + return false; + } + + // src of bias add is the unreshaped probs (-2 instead of -1) + if (nodes[node_idx]->op == GGML_OP_ADD && nodes[node_idx]->src[0] == nodes[node_idx - 2]) { + args.prob_bias = true; + node_idx++; + } + // RESHAPE/ADD -> ARGSORT + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_ARGSORT) { + return false; + } + + if (args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } else if (!args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 2]) { + return false; + } + + node_idx++; + + // ARGSORT-> VIEW + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_GET_ROWS) { + return false; + } + + // GET_ROWS + if (nodes[node_idx]->src[0] != probs_reshaped || nodes[node_idx]->src[1] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + } else if (args.delayed_softmax) { + if (node_idx - 2 < 0) { + return false; + } + ggml_tensor * probs_reshaped = nodes[node_idx - 2]; + + // VIEW->ARGSORT + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + + // GET_ROWS + if (node_idx >= n_nodes || nodes[node_idx]->src[1] != nodes[node_idx - 1] || + nodes[node_idx]->src[0] != probs_reshaped) { + return false; + } + node_idx++; + + static const std::vector remaining_ops = { GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }; + + for (const ggml_op op : remaining_ops) { + if (node_idx >= n_nodes || nodes[node_idx]->op != op || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + } + } + + // At this point we can check for norm + scale. Everything is now at least valid till the norm + if (node_idx >= n_nodes) { + return true; + } + + if (nodes[node_idx]->op == GGML_OP_RESHAPE) { + //check RESHAPE->SUM_ROWS->CLAMP->DIV->RESHAPE + static const std::vector norm_ops = { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP }; + + args.norm = true; + for (const ggml_op op : norm_ops) { + if (nodes[node_idx]->op == op && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { + node_idx++; + } else { + args.norm = false; + return true; + } + } + + // DIV <- CLAMP, RESHAPE + if (nodes[node_idx]->op != GGML_OP_DIV || nodes[node_idx]->src[1] != nodes[node_idx - 1] || + nodes[node_idx]->src[0] != nodes[node_idx - 3]) { + args.norm = false; + return true; + } + node_idx++; + + if (nodes[node_idx]->op != GGML_OP_RESHAPE || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + args.norm = false; + return true; + } + + node_idx++; + } + + if (nodes[node_idx]->op == GGML_OP_SCALE && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { + args.scale = true; + } + + return true; +} + +// returns whether the write (out) nodes overwrite the read nodes in operation +static bool ggml_cuda_check_fusion_memory_ranges(const ggml_cgraph * cgraph, + const int node_idx, + const int node_count, + const int * out_nodes, + const int out_count, + const bool is_topk_moe = false) { + auto nodes_overlap = [&](const ggml_tensor * a, const ggml_tensor * b) { + const int64_t a_start = (int64_t) a->data; + const int64_t a_end = a_start + ggml_backend_buft_get_alloc_size(a->buffer->buft, a); + + const int64_t b_start = (int64_t) b->data; + const int64_t b_end = b_start + ggml_backend_buft_get_alloc_size(b->buffer->buft, b); + + if ((b_start <= a_start && a_start < b_end) || (a_start <= b_start && b_start < a_end)) { + return true; + } + + return false; + }; + + bool is_ok = true; + // exception for topk-moe, as each row is read entirely before writing + if (ggml_nrows(cgraph->nodes[node_idx]) == 1 && is_topk_moe) { + return true; + } + + for (int i = 0; i < out_count; ++i) { + const ggml_tensor * dst = cgraph->nodes[out_nodes[i]]; + + for (int j = node_idx; j < node_idx + node_count; ++j) { + // Loop over all srcs of all nodes in the fusion. If the src overlaps + // the destination and the src is not an intermediate node that's being + // elided, then disable fusion. + + for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { + const ggml_tensor * src = cgraph->nodes[j]->src[src_idx]; + + if (!src || src->op == GGML_OP_NONE) { + continue; + } + + if (nodes_overlap(dst, src)) { + bool found = false; + + for (int k = node_idx; k < j; ++k) { + if (cgraph->nodes[k] == src) { + found = true; + break; + } + } + + if (!found) { + is_ok = false; + break; + } + } + } + } + } + + return is_ok; +} + + +static bool ggml_cuda_can_fuse(const struct ggml_cgraph * cgraph, + int node_idx, + std::initializer_list ops, + std::initializer_list unary_ops) { +#ifndef NDEBUG + const size_t num_unary = std::count(ops.begin(), ops.end(), GGML_OP_UNARY); + GGML_ASSERT(unary_ops.size() == num_unary); +#endif + + const auto is_equal = [](const std::initializer_list & list1, + const std::initializer_list & list2) { + return std::equal(list1.begin(), list1.end(), list2.begin(), list2.end()); + }; + + std::initializer_list mul_mat_bias_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_GLU }; + std::initializer_list mul_mat_id_bias_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_GLU }; + + std::initializer_list mul_mat_id_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_MUL_MAT_ID, GGML_OP_GLU }; + std::initializer_list mul_mat_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT, GGML_OP_GLU }; + + if ((is_equal(mul_mat_bias_glu_ops, ops) || is_equal(mul_mat_id_bias_glu_ops, ops)) && + ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 4 })) { + const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; + const ggml_tensor * ffn_gate_bias = cgraph->nodes[node_idx + 1]; + const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 2]; + const ggml_tensor * ffn_up_bias = cgraph->nodes[node_idx + 3]; + const ggml_tensor * glu = cgraph->nodes[node_idx + 4]; + + if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu, ffn_up_bias, ffn_gate_bias)) { + int out_nodes[] = { node_idx + 4 }; + return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); + } + } + + if ((is_equal(mul_mat_id_glu_ops, ops) || is_equal(mul_mat_glu_ops, ops)) && + ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { + const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; + const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 1]; + const ggml_tensor * glu = cgraph->nodes[node_idx + 2]; + + if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu)) { + int out_nodes[] = { node_idx + 2 }; + return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); + } + } + + std::initializer_list rope_set_rows_ops = { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }; + + if (is_equal(rope_set_rows_ops, ops) && ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { + const ggml_tensor * rope = cgraph->nodes[node_idx]; + const ggml_tensor * view = cgraph->nodes[node_idx + 1]; + const ggml_tensor * set_rows = cgraph->nodes[node_idx + 2]; + + if (ggml_cuda_should_fuse_rope_set_rows(rope, view, set_rows)) { + return true; + } + } + + if (!ggml_can_fuse(cgraph, node_idx, ops)) { + return false; + } + + if ((ops.size() == 2 || ops.size() == 3) && ops.begin()[0] == GGML_OP_RMS_NORM && ops.begin()[1] == GGML_OP_MUL) { + const ggml_tensor *rms_norm = cgraph->nodes[node_idx]; + const ggml_tensor *mul = cgraph->nodes[node_idx+1]; + const ggml_tensor *add = nullptr; + + if (ops.size() == 3 && ops.begin()[2] == GGML_OP_ADD) { + add = cgraph->nodes[node_idx+2]; + } + + GGML_ASSERT(rms_norm->src[0]->type == GGML_TYPE_F32); + GGML_ASSERT(rms_norm->type == GGML_TYPE_F32); + + //rms norm only supports F32 + if (mul->src[0]->type != GGML_TYPE_F32 || + mul->src[1]->type != GGML_TYPE_F32 || + mul->type != GGML_TYPE_F32) { + return false; + } + + if (add && (add->src[0]->type != GGML_TYPE_F32 || + add->src[1]->type != GGML_TYPE_F32 || + add->type != GGML_TYPE_F32) ) { + return false; + } + + //if rms norm is the B operand, then we don't handle broadcast + if (rms_norm == mul->src[1] && !ggml_are_same_shape(mul->src[0], rms_norm)) { + return false; + } + + //rms_norm kernel assumes contiguous rows + if (!ggml_is_contiguous_rows(mul->src[0]) || !ggml_is_contiguous_rows(mul->src[1])) { + return false; + } + + if (add && (!ggml_is_contiguous(add->src[0]) || !ggml_is_contiguous_rows(add->src[1]))) { + return false; + } + + return true; + } + + if (ops.size() == 2 && ops.begin()[0] == GGML_OP_SSM_CONV && ops.begin()[1] == GGML_OP_UNARY + && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_SILU) { + const ggml_tensor * ssm_conv = cgraph->nodes[node_idx]; + const ggml_tensor * silu = cgraph->nodes[node_idx+1]; + + if (ssm_conv->type != GGML_TYPE_F32 || silu->type != GGML_TYPE_F32) { + return false; + } + + return true; + } + + if (ops.size() == 2 && ops.begin()[0] == GGML_OP_UNARY && ops.begin()[1] == GGML_OP_MUL + && unary_ops.size() == 1 && (unary_ops.begin()[0] == GGML_UNARY_OP_SILU || unary_ops.begin()[0] == GGML_UNARY_OP_SIGMOID || unary_ops.begin()[0] == GGML_UNARY_OP_SOFTPLUS)) { + const ggml_tensor * unary = cgraph->nodes[node_idx]; + const ggml_tensor * mul = cgraph->nodes[node_idx+1]; + + if (ggml_get_unary_op(unary) != unary_ops.begin()[0]) { + return false; + } + + if (unary->type != GGML_TYPE_F32 && unary->type != GGML_TYPE_F16) { + return false; + } + + if (unary->type != mul->type) { + return false; + } + + const ggml_tensor * other = (mul->src[0] == unary) ? mul->src[1] : mul->src[0]; + if (other->type != unary->type) { + return false; + } + if (!ggml_is_contiguous_1(other) || !ggml_is_contiguous_1(unary->src[0]) || !ggml_are_same_shape(other, unary)) { + return false; + } + + return true; + } + + if (ops.size() == 3 && ops.begin()[0] == GGML_OP_SCALE && ops.begin()[1] == GGML_OP_UNARY && ops.begin()[2] == GGML_OP_SCALE + && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_TANH) { + const ggml_tensor *scale = cgraph->nodes[node_idx]; + const ggml_tensor *tanh = cgraph->nodes[node_idx+1]; + const ggml_tensor *scale2 = cgraph->nodes[node_idx+2]; + + GGML_ASSERT(scale->src[0]->type == GGML_TYPE_F32); + GGML_ASSERT(scale->type == GGML_TYPE_F32); + + if (ggml_get_unary_op(tanh) != GGML_UNARY_OP_TANH) { + return false; + } + + // Check for bias + if (ggml_get_op_params_f32(scale, 1) != 0.0f || ggml_get_op_params_f32(scale2, 1) != 0.0f) { + return false; + } + + return true; + } + + return false; +} + +static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph, const bool use_cuda_graph, const bool cuda_graph_update_required, const void * graph_key) { + bool graph_evaluated_or_captured = false; + + // flag used to determine whether it is an integrated_gpu + const bool integrated = ggml_cuda_info().devices[cuda_ctx->device].integrated; + + ggml_cuda_stream_context & stream_ctx = cuda_ctx->stream_context(); + bool is_concurrent_event_active = false; + ggml_cuda_concurrent_event * concurrent_event = nullptr; + bool should_launch_concurrent_events = false; + + const auto try_launch_concurrent_event = [&](const ggml_tensor * node) { + if (stream_ctx.concurrent_events.find(node) != stream_ctx.concurrent_events.end()) { + concurrent_event = &stream_ctx.concurrent_events[node]; + + is_concurrent_event_active = true; + + GGML_LOG_DEBUG("Launching %d streams at %s\n", concurrent_event->n_streams, node->name); + + cudaStream_t main_stream = cuda_ctx->stream(); // this should be stream 0 + GGML_ASSERT(cuda_ctx->curr_stream_no == 0); + CUDA_CHECK(cudaEventRecord(concurrent_event->fork_event, main_stream)); + + for (int i = 1; i <= concurrent_event->n_streams; ++i) { + cudaStream_t stream = cuda_ctx->stream(cuda_ctx->device, i); + CUDA_CHECK(cudaStreamWaitEvent(stream, concurrent_event->fork_event)); + } + } + }; + + while (!graph_evaluated_or_captured) { + // Only perform the graph execution if CUDA graphs are not enabled, or we are capturing the graph. + // With the use of CUDA graphs, the execution will be performed by the graph launch. + if (!use_cuda_graph || cuda_graph_update_required) { + [[maybe_unused]] int prev_i = 0; + + if (stream_ctx.concurrent_events.size() > 0) { + should_launch_concurrent_events = true; + for (const auto & [tensor, event] : stream_ctx.concurrent_events) { + should_launch_concurrent_events = should_launch_concurrent_events && event.is_valid(); + } + } + + if (should_launch_concurrent_events) { + // Restore original node order within each concurrent region to enable fusion within streams + + std::unordered_map node_to_idx; + node_to_idx.reserve(cgraph->n_nodes); + for (int i = 0; i < cgraph->n_nodes; ++i) { + node_to_idx[cgraph->nodes[i]] = i; + } + + for (auto & [fork_node, event] : stream_ctx.concurrent_events) { + // Find positions of all nodes from this event in the current graph + std::vector positions; + positions.reserve(event.original_order.size()); + + bool all_found = true; + for (const ggml_tensor * orig_node : event.original_order) { + auto it = node_to_idx.find(orig_node); + if (it != node_to_idx.end()) { + positions.push_back(it->second); + } else { + all_found = false; + break; + } + } + + if (!all_found || positions.size() != event.original_order.size()) { + continue; + } + + // Sort positions to get contiguous range + std::vector sorted_positions = positions; + std::sort(sorted_positions.begin(), sorted_positions.end()); + + bool is_contiguous = true; + for (size_t i = 1; i < sorted_positions.size(); ++i) { + if (sorted_positions[i] != sorted_positions[i-1] + 1) { + is_contiguous = false; + break; + } + } + + if (!is_contiguous) { + continue; + } + + // Restore original order at the sorted positions + int start_pos = sorted_positions[0]; + for (size_t i = 0; i < event.original_order.size(); ++i) { + cgraph->nodes[start_pos + i] = const_cast(event.original_order[i]); + } + } + } else { + stream_ctx.concurrent_events.clear(); + } + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_tensor * node = cgraph->nodes[i]; + if (is_concurrent_event_active) { + GGML_ASSERT(concurrent_event); + + if (node == concurrent_event->join_node) { + cuda_ctx->curr_stream_no = 0; + for (int i = 1; i <= concurrent_event->n_streams; ++i) { + // Wait on join events of forked streams in the main stream + CUDA_CHECK(cudaEventRecord(concurrent_event->join_events[i - 1], + cuda_ctx->stream(cuda_ctx->device, i))); + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), concurrent_event->join_events[i - 1])); + } + + is_concurrent_event_active = false; + concurrent_event = nullptr; + } else { + GGML_ASSERT (concurrent_event->stream_mapping.find(node) != concurrent_event->stream_mapping.end()); + cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; + GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); + } + } else if (i - prev_i > 1) { + //the previous node was fused + const ggml_tensor * prev_node = cgraph->nodes[i - 1]; + try_launch_concurrent_event(prev_node); + + if (is_concurrent_event_active) { + cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; + GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); + } + } + +#ifdef GGML_CUDA_DEBUG + const int nodes_fused = i - prev_i - 1; + if (nodes_fused > 0) { + GGML_LOG_INFO("nodes_fused: %d\n", nodes_fused); + } +#endif + prev_i = i; + + if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { + continue; + } + + if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { + continue; + } + + // start of fusion operations + static bool disable_fusion = (getenv("GGML_CUDA_DISABLE_FUSION") != nullptr); + if (!disable_fusion) { + ggml_cuda_topk_moe_args args; + + if (cgraph->nodes[i]->op == GGML_OP_UNARY || cgraph->nodes[i]->op == GGML_OP_SOFT_MAX || + cgraph->nodes[i]->op == GGML_OP_ARGSORT) { + const bool can_fuse = ggml_cuda_topk_moe_fusion(cgraph, i, args); + + std::vector ops; + + if (can_fuse) { + const ggml_tensor * logits = node->src[0]; + ggml_tensor * weights = nullptr; + ggml_tensor * ids = nullptr; + const ggml_tensor * bias = nullptr; + const ggml_tensor * clamp = nullptr; + const ggml_tensor * scale = nullptr; + + if (!args.delayed_softmax) { + ggml_op gating_op = args.sigmoid ? GGML_OP_UNARY : GGML_OP_SOFT_MAX; + int out_nodes[2]; // nodes which can't be elided + + if (args.prob_bias) { + bias = cgraph->nodes[i + 2]->src[1]; + ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, + GGML_OP_VIEW, GGML_OP_GET_ROWS }); + out_nodes[0] = i + 4; + ids = cgraph->nodes[i + 4]; + } else { + ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, + GGML_OP_GET_ROWS }); + out_nodes[0] = i + 3; + ids = cgraph->nodes[i + 3]; + } + + if (args.norm) { + ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP, + GGML_OP_DIV, GGML_OP_RESHAPE }); + clamp = cgraph->nodes[i + ops.size() - 3]; + } + if (args.scale) { + ops.insert(ops.end(), { GGML_OP_SCALE }); + scale = cgraph->nodes[i + ops.size() - 1]; + } + + weights = cgraph->nodes[i + ops.size() - 1]; + out_nodes[1] = i + ops.size() - 1; + + if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && + ggml_cuda_should_use_topk_moe(node, logits, weights, ids) && + ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { + ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); + i += ops.size() - 1; + continue; + } + } else if (!args.norm && !args.prob_bias) { + //special case gpt-oss, no norm, no bias. + ops.insert(ops.end(), { GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS, + GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }); + weights = cgraph->nodes[i + 5]; + ids = cgraph->nodes[i + 1]; + const ggml_tensor * softmax = cgraph->nodes[i + 4]; + + int out_nodes[2] = { i + 1, i + 5 }; + if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && + ggml_cuda_should_use_topk_moe(softmax, logits, weights, ids) && + ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { + ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); + i += ops.size() - 1; + continue; + } + } + } + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }, {})) { + ggml_tensor * rope = cgraph->nodes[i]; + ggml_tensor * set_rows = cgraph->nodes[i + 2]; + + ggml_cuda_op_rope_fused(*cuda_ctx, rope, set_rows); + i += 2; + continue; + } + + if (node->op == GGML_OP_ADD || node->op == GGML_OP_MUL) { + int n_fuse = 0; + ggml_op ops[8]; + std::fill(ops, ops + 8, node->op); + + for (; n_fuse <= 6; ++n_fuse){ + if (!ggml_can_fuse(cgraph, i + n_fuse, ops + n_fuse, 2)) { + break; + } + if (cgraph->nodes[i + n_fuse] != cgraph->nodes[i + n_fuse + 1]->src[0]) { + break; + } + if (!ggml_are_same_layout(cgraph->nodes[i + n_fuse]->src[1], cgraph->nodes[i + n_fuse + 1]->src[1])) { + break; + } + } + + n_fuse++; + + if (n_fuse > 1) { + ggml_tensor fused_node; + memcpy(&fused_node, node, sizeof(ggml_tensor)); + for (int j = 0; j < n_fuse - 1; ++j) { + fused_node.src[j + 2] = cgraph->nodes[i + j + 1]->src[1]; + } + fused_node.data = cgraph->nodes[i + n_fuse - 1]->data; + if (node->op == GGML_OP_ADD) { + ggml_cuda_op_fused_add(*cuda_ctx, &fused_node, n_fuse); + } else { + ggml_cuda_op_fused_mul(*cuda_ctx, &fused_node, n_fuse); + } + i += n_fuse - 1; + + continue; + } + } + + bool fused_mul_mat_vec = false; + int fused_node_count = 0; + + for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { + const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (ggml_cuda_can_fuse(cgraph, i, { op, bias_op, op, bias_op, GGML_OP_GLU }, {})) { + ggml_tensor * glu = cgraph->nodes[i + 4]; + ggml_tensor * gate_bias_n = glu->src[0]; + ggml_tensor * up_bias_n = glu->src[1]; + + //we don't assume the order for {gate, up}. Instead infer it from the bias tensor + ggml_tensor * gate_n = nullptr; + ggml_tensor * up_n = nullptr; + + if (gate_bias_n->src[0] == cgraph->nodes[i] || gate_bias_n->src[1] == cgraph->nodes[i]) { + gate_n = cgraph->nodes[i]; + up_n = cgraph->nodes[i + 2]; + } else if (gate_bias_n->src[0] == cgraph->nodes[i + 2] || gate_bias_n->src[1] == cgraph->nodes[i + 2]) { + gate_n = cgraph->nodes[i + 2]; + up_n = cgraph->nodes[i]; + } else { + continue; + } + + auto get_bias_tensor = [](const ggml_tensor * bias_node, const ggml_tensor * mul_node, ggml_op op_bias) { + if (op_bias == GGML_OP_ADD) { + if (bias_node->src[0] == mul_node) { + return bias_node->src[1]; + } + if (bias_node->src[1] == mul_node) { + return bias_node->src[0]; + } + return (ggml_tensor *) nullptr; + } + GGML_ASSERT(op_bias == GGML_OP_ADD_ID); + GGML_ASSERT(bias_node->src[0] == mul_node); + return bias_node->src[1]; + }; + + ggml_tensor * up_bias_tensor = get_bias_tensor(up_bias_n, up_n, bias_op); + ggml_tensor * gate_bias_tensor = get_bias_tensor(gate_bias_n, gate_n, bias_op); + + if (!up_bias_tensor || !gate_bias_tensor) { + continue; + } + + // we don't support repeating adds + if (bias_op == GGML_OP_ADD && + (!ggml_are_same_shape(gate_bias_n->src[0], gate_bias_n->src[1]) || + !ggml_are_same_shape(up_bias_n->src[0], up_bias_n->src[1]))) { + continue; + } + + const ggml_tensor * src0 = up_n->src[0]; + const ggml_tensor * src1 = up_n->src[1]; + const ggml_tensor * ids = up_n->src[2]; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(up_n)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate_n->src[0]; + fusion_data.x_bias = up_bias_tensor; + fusion_data.gate_bias = gate_bias_tensor; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 5; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(up_n)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate_n->src[0]; + fusion_data.x_bias = up_bias_tensor; + fusion_data.gate_bias = gate_bias_tensor; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 5; + break; + } + } else if (ggml_cuda_can_fuse(cgraph, i, { op, op, GGML_OP_GLU }, {})) { + ggml_tensor * glu = cgraph->nodes[i + 2]; + ggml_tensor * gate = glu->src[0]; + ggml_tensor * up = glu->src[1]; + + bool ok = (gate == cgraph->nodes[i] && up == cgraph->nodes[i + 1]) + || (gate == cgraph->nodes[i + 1] && up == cgraph->nodes[i]); + + if (!ok) continue; + + const ggml_tensor * src0 = up->src[0]; + const ggml_tensor * src1 = up->src[1]; + const ggml_tensor * ids = up->src[2]; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(up)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate->src[0]; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 3; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(up)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate->src[0]; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 3; + break; + } + } + } + + if (fused_mul_mat_vec) { + i += fused_node_count - 1; + continue; + } + + fused_mul_mat_vec = false; + fused_node_count = 0; + + for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { + const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (!ggml_can_fuse(cgraph, i, { op, bias_op })) { + continue; + } + + ggml_tensor * mm_node = cgraph->nodes[i]; + ggml_tensor * bias_node = cgraph->nodes[i + 1]; + + ggml_tensor * bias_tensor = nullptr; + if (bias_op == GGML_OP_ADD) { + if (bias_node->src[0] == mm_node) { + bias_tensor = bias_node->src[1]; + } else if (bias_node->src[1] == mm_node) { + bias_tensor = bias_node->src[0]; + } else { + continue; + } + } else { + if (bias_node->src[0] != mm_node) { + continue; + } + bias_tensor = bias_node->src[1]; + } + + const ggml_tensor * src0 = mm_node->src[0]; + const ggml_tensor * src1 = mm_node->src[1]; + const ggml_tensor * ids = mm_node->src[2]; + + if (bias_op == GGML_OP_ADD_ID && bias_node->src[2] != ids) { + continue; + } + + if (bias_op == GGML_OP_ADD && !ggml_are_same_shape(bias_node->src[0], bias_node->src[1])) { + continue; + } + + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.x_bias = bias_tensor; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(mm_node)) { + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 2; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(mm_node)) { + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 2; + break; + } + } + + if (fused_mul_mat_vec) { + i += fused_node_count - 1; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL, GGML_OP_ADD}, {})) { + ggml_cuda_op_rms_norm_fused_add(*cuda_ctx, node, cgraph->nodes[i+1], cgraph->nodes[i+2]); + i += 2; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL}, {})) { + ggml_cuda_op_rms_norm_fused(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SSM_CONV, GGML_OP_UNARY }, { GGML_UNARY_OP_SILU })) { + ggml_cuda_op_ssm_conv(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SILU }) || + ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SIGMOID }) || + ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SOFTPLUS })) { + ggml_cuda_op_unary_mul(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SCALE, GGML_OP_UNARY, GGML_OP_SCALE }, { GGML_UNARY_OP_TANH })) { + i += 2; + ggml_cuda_op_softcap(*cuda_ctx, cgraph->nodes[i], node); + continue; + } + } +#ifndef NDEBUG + assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device)); + for (int j = 0; j < GGML_MAX_SRC; j++) { + if (node->src[j] != nullptr) { + assert(node->src[j]->buffer); + assert(node->src[j]->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) || + ggml_backend_buft_is_cuda_split(node->src[j]->buffer->buft) || (integrated && ggml_backend_buft_is_cuda_host(node->src[j]->buffer->buft))); + } + } +#else + GGML_UNUSED(integrated); +#endif // NDEBUG + + bool ok = ggml_cuda_compute_forward(*cuda_ctx, node); + if (!ok) { + GGML_LOG_ERROR("%s: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op)); + } + GGML_ASSERT(ok); + + if (!is_concurrent_event_active) { + try_launch_concurrent_event(node); + } + } + } + +#ifdef USE_CUDA_GRAPH + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (use_cuda_graph && cuda_graph_update_required) { // End CUDA graph capture + if (graph->graph != nullptr) { + CUDA_CHECK(cudaGraphDestroy(graph->graph)); + graph->graph = nullptr; + } + + CUDA_CHECK(cudaStreamEndCapture(cuda_ctx->stream(), &graph->graph)); + graph_evaluated_or_captured = true; // CUDA graph has been captured + + std::lock_guard lock(ggml_cuda_lock); + if (ggml_cuda_lock_counter.fetch_sub(1, std::memory_order_relaxed) == 1) { + ggml_cuda_lock_cv.notify_all(); + } + } else { + graph_evaluated_or_captured = true; // ggml graph has been directly evaluated + } + } + + if (use_cuda_graph) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (graph->instance == nullptr) { // Create executable graph from captured graph. + CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); + } + if (cuda_graph_update_required) { // Update graph executable + ggml_cuda_graph_update_executable(cuda_ctx, graph_key); + } + // Launch graph + CUDA_CHECK(cudaGraphLaunch(graph->instance, cuda_ctx->stream())); +#else + GGML_UNUSED(graph_key); + graph_evaluated_or_captured = true; +#endif // USE_CUDA_GRAPH + } +} + +#ifdef USE_CUDA_GRAPH +static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + + if (graph->graph == nullptr) { + if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_AMPERE) { + if (!graph->disable_due_to_gpu_arch) { + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__); + } + graph->disable_due_to_gpu_arch = true; + } + } + + return graph->is_enabled(); +} +#endif // USE_CUDA_GRAPH + +static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend, ggml_cgraph * cgraph) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + + ggml_cuda_set_device(cuda_ctx->device); + + bool use_cuda_graph = false; + bool cuda_graph_update_required = false; + const void * graph_key = nullptr; + +#ifdef USE_CUDA_GRAPH + graph_key = ggml_cuda_graph_get_key(cgraph); + + ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); + + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (graph->is_enabled()) { + const bool graph_compatible = ggml_cuda_graph_check_compability(cgraph); + if (graph_compatible) { + const bool properties_changed = ggml_cuda_graph_update_required(cuda_ctx, cgraph); + + if (!graph->warmup_complete) { + // Warmup: need at least 2 calls with no property change on the 2nd call + if (!properties_changed) { + graph->warmup_complete = true; + GGML_LOG_DEBUG("%s: CUDA graph warmup complete\n", __func__); + use_cuda_graph = true; + cuda_graph_update_required = true; + } + // else: properties changed or first call - execute directly (use_cuda_graph stays false) + } else { + // Post-warmup: normal CUDA graph operation + if (properties_changed) { + // Properties changed - reset warmup, execute directly until stable again + graph->warmup_complete = false; + GGML_LOG_DEBUG("%s: CUDA graph warmup reset\n", __func__); + } else { + use_cuda_graph = true; + cuda_graph_update_required = graph->instance == nullptr; + } + } + } + } +#endif // USE_CUDA_GRAPH + + if (use_cuda_graph && cuda_graph_update_required) { + // Start CUDA graph capture + { + std::lock_guard lock(ggml_cuda_lock); + ggml_cuda_lock_counter.fetch_add(1, std::memory_order_relaxed); + } + + CUDA_CHECK(cudaStreamBeginCapture(cuda_ctx->stream(), cudaStreamCaptureModeRelaxed)); + } + + ggml_cuda_graph_evaluate_and_capture(cuda_ctx, cgraph, use_cuda_graph, cuda_graph_update_required, graph_key); + + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_event_record(ggml_backend_t backend, ggml_backend_event_t event) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + CUDA_CHECK(cudaEventRecord((cudaEvent_t)event->context, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_event_t event) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + if (ggml_backend_is_cuda(backend)) { + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), (cudaEvent_t)event->context, 0)); + } else { +#if 0 + // untested + auto wait_fn = [](void * user_data) { + ggml_backend_event_t event = (ggml_backend_event_t)user_data; + ggml_backend_event_synchronize(event); + }; + + CUDA_CHECK(cudaLaunchHostFunc(cuda_ctx->stream(), wait_fn, event)); +#endif + GGML_ABORT("fatal error"); + } +} + +static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + +#ifdef USE_CUDA_GRAPH + const void * graph_key = ggml_cuda_graph_get_key(cgraph); + const bool use_cuda_graph = ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); +#else + const bool use_cuda_graph = false; + GGML_UNUSED(cuda_ctx); + GGML_UNUSED(cgraph); +#endif + + static bool enable_graph_optimization = [] { + const char * env = getenv("GGML_CUDA_GRAPH_OPT"); + return env != nullptr && atoi(env) == 1; + }(); + + if (!enable_graph_optimization) { + return; + } + + ggml_cuda_stream_context & stream_context = cuda_ctx->stream_context(); + stream_context.reset(); + + if (!use_cuda_graph || ggml_backend_cuda_get_device_count() != 1) { + return; + } + + // number of out-degrees for a particular node + std::unordered_map fan_out; + // reverse mapping of node to index in the cgraph + std::unordered_map node_indices; + + const auto & is_noop = [](const ggml_tensor * node) -> bool { + return ggml_is_empty(node) || node->op == GGML_OP_NONE || node->op == GGML_OP_RESHAPE || + node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE; + }; + + const auto & depends_on = [](const ggml_tensor * dst, const ggml_tensor * src) -> bool { + for (uint32_t s = 0; s < GGML_MAX_SRC; ++s) { + if (dst->src[s] == src) { + return true; + } + } + // implicit dependency if they view the same tensor + const ggml_tensor * dst2 = dst->view_src ? dst->view_src : dst; + const ggml_tensor * src2 = src->view_src ? src->view_src : src; + if (dst2 == src2) { + return true; + } + return false; + }; + + for (int node_idx = 0; node_idx < cgraph->n_nodes; node_idx++) { + const ggml_tensor * node = cgraph->nodes[node_idx]; + node_indices[node] = node_idx; + + if (is_noop(node)) { + continue; + } + for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { + const ggml_tensor * src = cgraph->nodes[node_idx]->src[src_idx]; + //TODO: check why nrows > 1 fails + if (node && !is_noop(node) && ggml_nrows(node) <= 1) { + fan_out[src] += 1; + } + } + } + + // Target Q, K, V for concurrency + // this is a more general way to find nodes which can be candidates for concurrency (although it has not been tested for anything else): + // 1. find fan-out (fork) nodes where the same input is used at least N times (in QKV, it would be "attn-norm") + // 2. find the join node, where 2 or more of the outputs are required (in QKV, this would "KQ" or "flash-attn") + // 3. account for all branches from the fork to the join + // 4. To extend lifetimes of the tensors, we interleave the branches (see below for more details) + // 5. save the original cgraph and restore it in graph_compute, to enable fusion within streams + // See discussion: https://github.com/ggml-org/llama.cpp/pull/16991#issuecomment-3522620030 + + const int min_fan_out = 3; + const int max_fan_out = 3; + + // store {fork_idx, join_idx} + std::vector> concurrent_node_ranges; + + for (const auto & [root_node, count] : fan_out) { + if (count >= min_fan_out && count <= max_fan_out) { + const int root_node_idx = node_indices[root_node]; + + // only optimize for attn_norm + // TODO: make this more generic + if (!strstr(root_node->name, "attn_norm")) { + continue; + } + + bool is_part_of_event = false; + for (const auto & [start, end] : concurrent_node_ranges) { + if (root_node_idx >= start && root_node_idx <= end) { + is_part_of_event = true; + } + } + + if (is_part_of_event) { + continue; + } + + std::vector> nodes_per_branch; + for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { + const ggml_tensor * node = cgraph->nodes[i]; + if (!is_noop(node) && depends_on(node, root_node)) { + nodes_per_branch.push_back({ node }); + } + } + + GGML_ASSERT(nodes_per_branch.size() == (size_t) count); + + //find the join point + const ggml_tensor * join_node = nullptr; + + const auto & belongs_to_branch = [&](const ggml_tensor * node, + const std::vector & branch) -> bool { + for (const ggml_tensor * n : branch) { + if (depends_on(node, n)) { + return true; + } + } + return false; + }; + + for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { + const ggml_tensor * curr_node = cgraph->nodes[i]; + + int num_joins = 0; + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + if (belongs_to_branch(curr_node, nodes_per_branch[branch_idx])) { + num_joins++; + } + } + + if (num_joins >= 2) { + join_node = curr_node; + break; + } + + bool found_branch = false; + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + std::vector & branch_vec = nodes_per_branch[branch_idx]; + if (belongs_to_branch(curr_node, branch_vec)) { + //continue accumulating + if (std::find(branch_vec.begin(), branch_vec.end(), curr_node) == branch_vec.end()) { + branch_vec.push_back(curr_node); + } + found_branch = true; + } + } + + if (!found_branch && is_noop(curr_node)) { + // we can put it in any branch because it will be ignored + nodes_per_branch[0].push_back({ curr_node }); + } + } + + if (join_node) { + //Create ggml_cuda_concurrent_event + ggml_cuda_concurrent_event concurrent_event(nodes_per_branch.size()); + concurrent_event.join_node = join_node; + + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + for (const ggml_tensor * n : nodes_per_branch[branch_idx]) { + concurrent_event.stream_mapping[n] = branch_idx + 1; + } + } + + int fork_node_idx = node_indices[root_node]; + int join_node_idx = node_indices[join_node]; + + int current_branch_idx = 0; + int current_node_idx = fork_node_idx + 1; + const int n_branches = nodes_per_branch.size(); + + int total_branch_nodes = 0; + for (std::vector branch_nodes : nodes_per_branch) { + total_branch_nodes += branch_nodes.size(); + } + + // there are other nodes in the middle which are unaccounted for + // usually (cpy) nodes, then ignore this fork + if (join_node_idx - fork_node_idx - 1 != total_branch_nodes) { + GGML_LOG_DEBUG( + "Skipping %s because the number of nodes in the middle is not equal to the total number of " + "branch nodes %d != %d\n", + root_node->name, join_node_idx - fork_node_idx - 1, total_branch_nodes); + continue; + } + + // Save the original order of nodes in this region before interleaving + // This is used later to restore grouping for fusion within streams + concurrent_event.original_order.reserve(total_branch_nodes); + for (int i = fork_node_idx + 1; i < join_node_idx; ++i) { + concurrent_event.original_order.push_back(cgraph->nodes[i]); + } + + std::unordered_map & concurrent_events = cuda_ctx->stream_context().concurrent_events; + GGML_ASSERT(concurrent_events.find(root_node) == concurrent_events.end()); + concurrent_events.emplace(root_node, std::move(concurrent_event)); + GGML_LOG_DEBUG("Adding stream at node %s %p\n", root_node->name, root_node); + concurrent_node_ranges.emplace_back(fork_node_idx, join_node_idx); + + // interleave tensors to extend lifetimes so that ggml graph doesn't recycle them + // example transformation: + // [attn-norm, QMul, QNorm, QRope, KMul, KNorm, KRope, VMul, attn] -> + // [attn-norm, QMul, KMul, VMul, QNorm, VNorm, QRope, KRope, attn] + while (current_node_idx < join_node_idx) { + std::vector & branch_nodes = nodes_per_branch[current_branch_idx]; + + bool has_node = false; + for (std::vector branch_node : nodes_per_branch) { + has_node |= branch_node.size() > 0; + } + + GGML_ASSERT(has_node); + + if (branch_nodes.empty()) { + current_branch_idx = (current_branch_idx + 1) % n_branches; + continue; + } + + cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); + current_node_idx++; + branch_nodes.erase(branch_nodes.begin()); + + // append all empty nodes + while (!branch_nodes.empty() && is_noop(branch_nodes.front())) { + cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); + current_node_idx++; + branch_nodes.erase(branch_nodes.begin()); + } + + current_branch_idx = (current_branch_idx + 1) % n_branches; + } + } + } + } +} + +static const ggml_backend_i ggml_backend_cuda_interface = { + /* .get_name = */ ggml_backend_cuda_get_name, + /* .free = */ ggml_backend_cuda_free, + /* .set_tensor_async = */ ggml_backend_cuda_set_tensor_async, + /* .get_tensor_async = */ ggml_backend_cuda_get_tensor_async, + /* .get_tensor_2d_async = */ ggml_backend_cuda_set_tensor_2d_async, + /* .set_tensor_2d_async = */ ggml_backend_cuda_get_tensor_2d_async, + /* .cpy_tensor_async = */ ggml_backend_cuda_cpy_tensor_async, + /* .synchronize = */ ggml_backend_cuda_synchronize, + /* .graph_plan_create = */ NULL, + /* .graph_plan_free = */ NULL, + /* .graph_plan_update = */ NULL, + /* .graph_plan_compute = */ NULL, + /* .graph_compute = */ ggml_backend_cuda_graph_compute, + /* .event_record = */ ggml_backend_cuda_event_record, + /* .event_wait = */ ggml_backend_cuda_event_wait, + /* .graph_optimize = */ ggml_backend_cuda_graph_optimize, +}; + +static ggml_guid_t ggml_backend_cuda_guid() { + static ggml_guid guid = { 0x2c, 0xdd, 0xe8, 0x1c, 0x65, 0xb3, 0x65, 0x73, 0x6a, 0x12, 0x88, 0x61, 0x1c, 0xc9, 0xdc, 0x25 }; + return &guid; +} + +bool ggml_backend_is_cuda(ggml_backend_t backend) { + return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_cuda_guid()); +} + +int ggml_backend_cuda_get_device_count() { + return ggml_cuda_info().device_count; +} + +void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) { + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, device)); + snprintf(description, description_size, "%s", prop.name); +} + +void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) { + ggml_cuda_set_device(device); + + CUDA_CHECK(cudaMemGetInfo(free, total)); +} + +bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) { + if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { + return false; + } + +#if CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) || defined(GGML_USE_HIP) + cudaError_t err = cudaHostRegister(buffer, size, cudaHostRegisterPortable | cudaHostRegisterReadOnly); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + + GGML_LOG_DEBUG("%s: failed to register %.2f MiB of pinned memory: %s\n", __func__, + size / 1024.0 / 1024.0, cudaGetErrorString(err)); + return false; + } + return true; +#else + GGML_UNUSED(buffer); + GGML_UNUSED(size); + return false; +#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) +} + +void ggml_backend_cuda_unregister_host_buffer(void * buffer) { + if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { + return; + } + + cudaError_t err = cudaHostUnregister(buffer); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + } +} + + +// backend device + +struct ggml_backend_cuda_device_context { + int device; + std::string name; + std::string description; + std::string pci_bus_id; + int op_offload_min_batch_size; +}; + +static const char * ggml_backend_cuda_device_get_name(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ctx->name.c_str(); +} + +static const char * ggml_backend_cuda_device_get_description(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ctx->description.c_str(); +} + +#if defined(__linux__) +// Helper function to get available memory from /proc/meminfo for UMA systems +static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_kb, long * free_swap_kb) { + FILE * meminfo_file = nullptr; + // 2KB buffer for reading /proc/meminfo since it does not report size info, should be enough + const size_t BUFFER_SIZE = 2048; + auto file_buffer = std::make_unique(BUFFER_SIZE); + size_t bytes_read = 0; + long huge_tlb_total_pages = -1; + long huge_tlb_free_pages = -1; + long huge_tlb_page_size = -1; + + if (available_memory_kb == nullptr || free_swap_kb == nullptr) { + return false; + } + + meminfo_file = fopen("/proc/meminfo", "r"); + if (meminfo_file == nullptr) { + GGML_LOG_ERROR("%s: failed to open /proc/meminfo\n", __func__); + return false; + } + + // Read file into buffer + bytes_read = fread(file_buffer.get(), 1, BUFFER_SIZE - 1, meminfo_file); + fclose(meminfo_file); + + if (bytes_read == 0) { + GGML_LOG_ERROR("%s: failed to read from /proc/meminfo\n", __func__); + return false; + } + file_buffer[bytes_read] = '\0'; + + *available_memory_kb = -1; + *free_swap_kb = -1; + + // Parse the file buffer line by line + char * line = file_buffer.get(); + char * line_next; + while (line < file_buffer.get() + bytes_read) { + // Find the end of the current line + line_next = strchr(line, '\n'); + if (line_next != nullptr) { + *line_next = '\0'; + line_next++; + } else { + line_next = file_buffer.get() + bytes_read; + } + + long value; + if (sscanf(line, "MemAvailable: %ld kB", &value) == 1) { + *available_memory_kb = value; + } else if (sscanf(line, "SwapFree: %ld kB", &value) == 1) { + *free_swap_kb = value; + } else if (sscanf(line, "HugePages_Total: %ld", &value) == 1) { + huge_tlb_total_pages = value; + } else if (sscanf(line, "HugePages_Free: %ld", &value) == 1) { + huge_tlb_free_pages = value; + } else if (sscanf(line, "Hugepagesize: %ld kB", &value) == 1) { + huge_tlb_page_size = value; + } + + line = line_next; + } + + if (huge_tlb_total_pages != 0 && huge_tlb_total_pages != -1) { + *available_memory_kb = huge_tlb_free_pages * huge_tlb_page_size; + + // Hugetlbfs pages are not swappable. + *free_swap_kb = 0; + } + + GGML_LOG_DEBUG("%s: final available_memory_kb: %ld\n", __func__, *available_memory_kb); + return true; +} +#endif // defined(__linux__) + +static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemGetInfo(free, total)); + +// ref: https://github.com/ggml-org/llama.cpp/pull/17368 +#if defined(__linux__) + // Check if this is a UMA (Unified Memory Architecture) system + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device)); + + // Check if UMA is explicitly enabled via environment variable + bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr; + bool is_uma = prop.integrated > 0 || uma_env; + + if (is_uma) { + // For UMA systems (like DGX Spark), use system memory info + long available_memory_kb = 0; + long free_swap_kb = 0; + + if (ggml_backend_cuda_get_available_uma_memory(&available_memory_kb, &free_swap_kb) && available_memory_kb > 0) { + *free = (size_t)available_memory_kb * 1024; + } else { + GGML_LOG_ERROR("%s: /proc/meminfo reading failed, using cudaMemGetInfo\n", __func__); + } + } +#endif // defined(__linux__) + +} + +static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) { + GGML_UNUSED(dev); + return GGML_BACKEND_DEVICE_TYPE_GPU; +} + +static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_backend_dev_props * props) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + + props->name = ggml_backend_cuda_device_get_name(dev); + props->description = ggml_backend_cuda_device_get_description(dev); + props->type = ggml_backend_cuda_device_get_type(dev); + props->device_id = ctx->pci_bus_id.empty() ? nullptr : ctx->pci_bus_id.c_str(); + ggml_backend_cuda_device_get_memory(dev, &props->memory_free, &props->memory_total); + + bool host_buffer = getenv("GGML_CUDA_NO_PINNED") == nullptr; +#ifdef GGML_CUDA_NO_PEER_COPY + bool events = false; +#else + bool events = true; +#endif + + props->caps = { + /* .async = */ true, + /* .host_buffer = */ host_buffer, + /* .buffer_from_host_ptr = */ false, + /* .events = */ events, + }; +} + +static ggml_backend_t ggml_backend_cuda_device_init_backend(ggml_backend_dev_t dev, const char * params) { + GGML_UNUSED(params); + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ggml_backend_cuda_init(ctx->device); +} + +static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_buffer_type(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ggml_backend_cuda_buffer_type(ctx->device); +} + +static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_host_buffer_type(ggml_backend_dev_t dev) { + GGML_UNUSED(dev); + return ggml_backend_cuda_host_buffer_type(); +} + +// TODO: move these functions here +static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + + // split buffers can only be used with GGML_OP_MUL_MAT + if (op->op != GGML_OP_MUL_MAT) { + for (int i = 0; i < GGML_MAX_SRC; i++) { + if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda_split(op->src[i]->buffer->buft)) { + return false; + } + } + } + + // check if all the sources are allocated on this device + for (int i = 0; i < GGML_MAX_SRC; i++) { + if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda(op->src[i]->buffer->buft)) { + ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)op->src[i]->buffer->buft->context; + if (buft_ctx->device != dev_ctx->device) { + return false; + } + } + } + + switch (op->op) { + case GGML_OP_UNARY: + switch (ggml_get_unary_op(op)) { + case GGML_UNARY_OP_ABS: + case GGML_UNARY_OP_SGN: + case GGML_UNARY_OP_NEG: + case GGML_UNARY_OP_STEP: + case GGML_UNARY_OP_GELU: + case GGML_UNARY_OP_SILU: + case GGML_UNARY_OP_RELU: + case GGML_UNARY_OP_SIGMOID: + case GGML_UNARY_OP_HARDSIGMOID: + case GGML_UNARY_OP_HARDSWISH: + case GGML_UNARY_OP_GELU_ERF: + case GGML_UNARY_OP_GELU_QUICK: + case GGML_UNARY_OP_TANH: + case GGML_UNARY_OP_EXP: + case GGML_UNARY_OP_EXPM1: + case GGML_UNARY_OP_SOFTPLUS: + case GGML_UNARY_OP_ELU: + case GGML_UNARY_OP_XIELU: + case GGML_UNARY_OP_FLOOR: + case GGML_UNARY_OP_CEIL: + case GGML_UNARY_OP_ROUND: + case GGML_UNARY_OP_TRUNC: + // TODO: should become: + //return ggml_is_contiguous_rows(op->src[0]); + return ggml_is_contiguous(op->src[0]); + default: + return false; + } + break; + case GGML_OP_GLU: + switch (ggml_get_glu_op(op)) { + case GGML_GLU_OP_REGLU: + case GGML_GLU_OP_GEGLU: + case GGML_GLU_OP_SWIGLU: + case GGML_GLU_OP_SWIGLU_OAI: + case GGML_GLU_OP_GEGLU_ERF: + case GGML_GLU_OP_GEGLU_QUICK: + return ggml_is_contiguous_1(op->src[0]); + default: + return false; + } + break; + case GGML_OP_MUL_MAT: + case GGML_OP_MUL_MAT_ID: + { + struct ggml_tensor * a = op->src[0]; + struct ggml_tensor * b = op->src[1]; + if (a->buffer && ggml_backend_buft_is_cuda_split(a->buffer->buft)) { + if (a->ne[2] > 1 || a->ne[3] > 1) { + return false; + } + // for small weight matrices the active device can end up without any rows, don't use row split in those cases + // this avoids some edge cases (and the performance would not be good anyways) + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) a->buffer->buft->context; + int64_t row_low; + int64_t row_high; + get_row_split(&row_low, &row_high, a, buft_ctx->tensor_split, dev_ctx->device); + if (row_low == row_high) { + return false; + } + } + if (b->type == GGML_TYPE_F16 && a->type != GGML_TYPE_F16) { + return false; + } +#ifdef GGML_USE_MUSA + const int cc = ggml_cuda_info().devices[dev_ctx->device].cc; + if (b->ne[2]*b->ne[3] > 1 && !ggml_is_transposed(a) && !ggml_is_transposed(b)) { + if (GGML_CUDA_CC_IS_QY1(cc) && op->op == GGML_OP_MUL_MAT && + a->type == GGML_TYPE_F16 && b->type == GGML_TYPE_F16) { + return false; + } + if (GGML_CUDA_CC_IS_QY2(cc) && op->op == GGML_OP_MUL_MAT_ID && + a->type == GGML_TYPE_Q2_K && b->type == GGML_TYPE_F32) { + return false; + } + } +#endif // GGML_USE_MUSA + switch (a->type) { + case GGML_TYPE_F32: + case GGML_TYPE_F16: + case GGML_TYPE_Q1_0: + case GGML_TYPE_Q4_0: + case GGML_TYPE_Q4_1: + case GGML_TYPE_Q5_0: + case GGML_TYPE_Q5_1: + case GGML_TYPE_Q8_0: + case GGML_TYPE_MXFP4: + case GGML_TYPE_NVFP4: + case GGML_TYPE_Q2_K: + case GGML_TYPE_Q3_K: + case GGML_TYPE_Q4_K: + case GGML_TYPE_Q5_K: + case GGML_TYPE_Q6_K: + case GGML_TYPE_Q8_K: + case GGML_TYPE_IQ1_M: + case GGML_TYPE_IQ1_S: + case GGML_TYPE_IQ2_S: + case GGML_TYPE_IQ2_XS: + case GGML_TYPE_IQ2_XXS: + case GGML_TYPE_IQ3_S: + case GGML_TYPE_IQ3_XXS: + case GGML_TYPE_IQ4_NL: + case GGML_TYPE_IQ4_XS: + case GGML_TYPE_BF16: + return true; + default: + return false; + } + } break; + case GGML_OP_OUT_PROD: + return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32; + case GGML_OP_GET_ROWS: + { + switch (op->src[0]->type) { + case GGML_TYPE_F16: + case GGML_TYPE_F32: + case GGML_TYPE_BF16: + case GGML_TYPE_I32: + case GGML_TYPE_Q1_0: + case GGML_TYPE_Q4_0: + case GGML_TYPE_Q4_1: + case GGML_TYPE_Q5_0: + case GGML_TYPE_Q5_1: + case GGML_TYPE_Q8_0: + return true; + default: + return false; + } + } break; + case GGML_OP_GET_ROWS_BACK: + { + return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->ne[2] == 1 && op->ne[3] == 1; + } break; + case GGML_OP_SET_ROWS: + { + return (op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_BF16 || + op->type == GGML_TYPE_Q4_0 || op->type == GGML_TYPE_Q4_1 || op->type == GGML_TYPE_Q5_0 || + op->type == GGML_TYPE_Q5_1 || op->type == GGML_TYPE_Q8_0 || op->type == GGML_TYPE_IQ4_NL) && + op->src[0]->type == GGML_TYPE_F32 && + (op->src[1]->type == GGML_TYPE_I64 || op->src[1]->type == GGML_TYPE_I32); + } break; + case GGML_OP_SET: + { + const ggml_type t = op->type; + return (t == GGML_TYPE_F32 || t == GGML_TYPE_I32) && + t == op->src[0]->type && + t == op->src[1]->type; + } break; + case GGML_OP_CPY: + { + ggml_type src0_type = op->src[0]->type; + ggml_type src1_type = op->src[1]->type; + if ((src0_type == GGML_TYPE_F32 || src0_type == GGML_TYPE_BF16 || src0_type == GGML_TYPE_F16) && + (src1_type == GGML_TYPE_F32 || src1_type == GGML_TYPE_BF16 || src1_type == GGML_TYPE_F16) + ) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q8_0) { + return true; + } + if (src0_type == GGML_TYPE_Q8_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_0) { + return true; + } + if (src0_type == GGML_TYPE_Q4_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_1) { + return true; + } + if (src0_type == GGML_TYPE_Q4_1 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_0) { + return true; + } + if (src0_type == GGML_TYPE_Q5_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_1) { + return true; + } + if (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_IQ4_NL) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) { + return true; + } + if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_I32) { + return true; + } + if (src0_type == src1_type && ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1])) { + return true; + } + return false; + } break; + case GGML_OP_DUP: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_ARGMAX: + case GGML_OP_COUNT_EQUAL: + { + return true; + } break; + case GGML_OP_REPEAT: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_REPEAT_BACK: + return op->type == GGML_TYPE_F32 && (op->src[0]->ne[2]*op->src[0]->ne[3]) <= (1 << 15); + case GGML_OP_CONCAT: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_CONV_TRANSPOSE_1D: + { + ggml_type src0_type = op->src[0]->type; + ggml_type src1_type = op->src[1]->type; + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_F32) { + return true; + } + return false; + } break; + case GGML_OP_SILU_BACK: + return ggml_is_contiguous(op->src[0]) && op->src[0]->type == GGML_TYPE_F32; + break; + case GGML_OP_NORM: + case GGML_OP_RMS_NORM: + case GGML_OP_L2_NORM: + return true; + case GGML_OP_RMS_NORM_BACK: + return ggml_is_contiguous(op->src[0]); + break; + case GGML_OP_NONE: + case GGML_OP_RESHAPE: + case GGML_OP_VIEW: + case GGML_OP_PERMUTE: + case GGML_OP_TRANSPOSE: + case GGML_OP_ADD: + case GGML_OP_ADD_ID: + case GGML_OP_ADD1: + case GGML_OP_SUB: + case GGML_OP_MUL: + case GGML_OP_DIV: + case GGML_OP_SCALE: + case GGML_OP_SQR: + case GGML_OP_SQRT: + case GGML_OP_SIN: + case GGML_OP_COS: + case GGML_OP_CLAMP: + case GGML_OP_LOG: + return true; + case GGML_OP_SSM_SCAN: { + if (op->src[3]->ne[0] == 1) { + // Mamba2 + // (kernel only supports (d_state == 128 || d_state == 256) && d_head % 16 == 0) + return (op->src[0]->ne[0] == 128 || op->src[0]->ne[0] == 256) && op->src[0]->ne[1] % 16 == 0; + } else { + // Mamba + // (kernel only supports d_state == 16, d_head == 1, n_head % 128 == 0, n_group == 1) + return op->src[0]->ne[0] == 16 && op->src[0]->ne[1] == 1 && op->src[0]->ne[2] % 128 == 0 && op->src[4]->ne[1] == 1; + } + } + case GGML_OP_SSM_CONV: { + // assumes d_inner % threads == 0 + return op->src[0]->ne[1] % 128 == 0; + } + case GGML_OP_CONT: + return true; + case GGML_OP_DIAG_MASK_INF: + return true; + case GGML_OP_SOFT_MAX: + return true; + case GGML_OP_SOFT_MAX_BACK: { + float max_bias = 0.0f; + memcpy(&max_bias, (const float *) op->op_params + 1, sizeof(float)); + return max_bias == 0.0f; + } + case GGML_OP_ROLL: + if(op->src[0]->type == GGML_TYPE_F32) { + return true; + } + return false; + case GGML_OP_ROPE: + case GGML_OP_ROPE_BACK: { + return op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]); + } + case GGML_OP_IM2COL: + case GGML_OP_IM2COL_3D: + case GGML_OP_CONV_2D: + case GGML_OP_CONV_2D_DW: + case GGML_OP_CONV_TRANSPOSE_2D: + case GGML_OP_POOL_2D: + return true; + case GGML_OP_ACC: + // TODO: extend support like so: + //return ggml_is_contiguous_rows(op->src[0]) && ggml_is_contiguous_rows(op->src[1]); + return ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1]); + case GGML_OP_SUM: + return ggml_is_contiguous_rows(op->src[0]); + case GGML_OP_TOP_K: + case GGML_OP_ARGSORT: +#ifndef GGML_CUDA_USE_CUB + return op->src[0]->ne[0] <= 1024; +#else + return true; +#endif + case GGML_OP_SUM_ROWS: + case GGML_OP_MEAN: + case GGML_OP_GROUP_NORM: + return ggml_is_contiguous(op->src[0]); + case GGML_OP_PAD: + return true; + case GGML_OP_UPSCALE: + case GGML_OP_PAD_REFLECT_1D: + case GGML_OP_ARANGE: + case GGML_OP_TIMESTEP_EMBEDDING: + case GGML_OP_LEAKY_RELU: + case GGML_OP_RWKV_WKV6: + case GGML_OP_GATED_LINEAR_ATTN: + case GGML_OP_RWKV_WKV7: + return true; + case GGML_OP_GATED_DELTA_NET: + //TODO: enable once MUSA compiler is solved https://github.com/ggml-org/llama.cpp/pull/19504#issuecomment-4018634327 +#ifdef GGML_USE_MUSA + return false; +#else + return true; +#endif // GGML_USE_MUSA + case GGML_OP_FLASH_ATTN_EXT: + return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op); + case GGML_OP_CROSS_ENTROPY_LOSS: + case GGML_OP_CROSS_ENTROPY_LOSS_BACK: + case GGML_OP_OPT_STEP_ADAMW: + case GGML_OP_OPT_STEP_SGD: + case GGML_OP_FILL: + case GGML_OP_CUMSUM: + case GGML_OP_TRI: + case GGML_OP_DIAG: + case GGML_OP_SOLVE_TRI: + return true; + + default: + return false; + } +} + +static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated; + return (((ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft))); +} + +static int64_t get_op_batch_size(const ggml_tensor * op) { + switch (op->op) { + case GGML_OP_GET_ROWS: + return 0; + case GGML_OP_MUL_MAT: + return op->ne[1]; + case GGML_OP_MUL_MAT_ID: + case GGML_OP_ROPE: + case GGML_OP_ROPE_BACK: + return op->ne[2]; + default: + return ggml_nrows(op); + } +} + +static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + + return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size; +} + +static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_t dev) { +#ifdef GGML_CUDA_NO_PEER_COPY + return nullptr; +#else + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *)dev->context; + + ggml_cuda_set_device(dev_ctx->device); + + cudaEvent_t event; + CUDA_CHECK(cudaEventCreateWithFlags(&event, cudaEventDisableTiming)); + + return new ggml_backend_event { + /* .device = */ dev, + /* .context = */ event, + }; +#endif +} + +static void ggml_backend_cuda_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) { + GGML_UNUSED(dev); + + CUDA_CHECK(cudaEventDestroy((cudaEvent_t)event->context)); + delete event; +} + +static void ggml_backend_cuda_device_event_synchronize(ggml_backend_dev_t dev, ggml_backend_event_t event) { + GGML_UNUSED(dev); + CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context)); +} + +static const ggml_backend_device_i ggml_backend_cuda_device_interface = { + /* .get_name = */ ggml_backend_cuda_device_get_name, + /* .get_description = */ ggml_backend_cuda_device_get_description, + /* .get_memory = */ ggml_backend_cuda_device_get_memory, + /* .get_type = */ ggml_backend_cuda_device_get_type, + /* .get_props = */ ggml_backend_cuda_device_get_props, + /* .init_backend = */ ggml_backend_cuda_device_init_backend, + /* .get_buffer_type = */ ggml_backend_cuda_device_get_buffer_type, + /* .get_host_buffer_type = */ ggml_backend_cuda_device_get_host_buffer_type, + /* .buffer_from_host_ptr = */ NULL, + /* .supports_op = */ ggml_backend_cuda_device_supports_op, + /* .supports_buft = */ ggml_backend_cuda_device_supports_buft, + /* .offload_op = */ ggml_backend_cuda_device_offload_op, + /* .event_new = */ ggml_backend_cuda_device_event_new, + /* .event_free = */ ggml_backend_cuda_device_event_free, + /* .event_synchronize = */ ggml_backend_cuda_device_event_synchronize, +}; + +// backend reg + +struct ggml_backend_cuda_reg_context { + std::vector devices; +}; + +static const char * ggml_backend_cuda_reg_get_name(ggml_backend_reg_t reg) { + GGML_UNUSED(reg); + return GGML_CUDA_NAME; +} + +static size_t ggml_backend_cuda_reg_get_device_count(ggml_backend_reg_t reg) { + ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; + return ctx->devices.size(); +} + +static ggml_backend_dev_t ggml_backend_cuda_reg_get_device(ggml_backend_reg_t reg, size_t index) { + ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; + GGML_ASSERT(index < ctx->devices.size()); + return ctx->devices[index]; +} + +static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t reg) { + static std::vector features = []() { + std::vector features; + #define _STRINGIFY(...) #__VA_ARGS__ + #define STRINGIFY(...) _STRINGIFY(__VA_ARGS__) + + #ifdef __CUDA_ARCH_LIST__ + features.push_back({ "ARCHS", STRINGIFY(__CUDA_ARCH_LIST__) }); + #endif + + #ifdef GGML_CUDA_FORCE_MMQ + features.push_back({ "FORCE_MMQ", "1" }); + #endif + + #ifdef GGML_CUDA_FORCE_CUBLAS + features.push_back({ "FORCE_CUBLAS", "1" }); + #endif + + #ifndef GGML_USE_VMM + features.push_back({ "NO_VMM", "1" }); + #endif + + #ifdef GGML_CUDA_NO_PEER_COPY + features.push_back({ "NO_PEER_COPY", "1" }); + #endif + + #ifdef GGML_CUDA_USE_GRAPHS + features.push_back({ "USE_GRAPHS", "1" }); + #endif + + #ifdef GGML_CUDA_PEER_MAX_BATCH_SIZE + features.push_back({ "PEER_MAX_BATCH_SIZE", STRINGIFY(GGML_CUDA_PEER_MAX_BATCH_SIZE) }); + #endif + + #ifdef GGML_CUDA_FA_ALL_QUANTS + features.push_back({ "FA_ALL_QUANTS", "1" }); + #endif + + { + const auto & info = ggml_cuda_info(); + for (int id = 0; id < info.device_count; ++id) { + if (blackwell_mma_available(info.devices[id].cc)) { + features.push_back({ "BLACKWELL_NATIVE_FP4", "1"}); + break; + } + } + } + + #undef _STRINGIFY + #undef STRINGIFY + + features.push_back({ nullptr, nullptr }); + + return features; + }(); + + return features.data(); + + GGML_UNUSED(reg); +} + +static void * ggml_backend_cuda_reg_get_proc_address(ggml_backend_reg_t reg, const char * name) { + GGML_UNUSED(reg); + if (strcmp(name, "ggml_backend_comm_init") == 0) { + return (void *)ggml_backend_cuda_comm_init; + } + if (strcmp(name, "ggml_backend_comm_free") == 0) { + return (void *)ggml_backend_cuda_comm_free; + } + if (strcmp(name, "ggml_backend_comm_allreduce_tensor") == 0) { + return (void *)ggml_backend_cuda_comm_allreduce_tensor; + } + if (strcmp(name, "ggml_backend_split_buffer_type") == 0) { + return (void *)ggml_backend_cuda_split_buffer_type; + } + if (strcmp(name, "ggml_backend_register_host_buffer") == 0) { + return (void *)ggml_backend_cuda_register_host_buffer; + } + if (strcmp(name, "ggml_backend_unregister_host_buffer") == 0) { + return (void *)ggml_backend_cuda_unregister_host_buffer; + } + if (strcmp(name, "ggml_backend_get_features") == 0) { + return (void *)ggml_backend_cuda_get_features; + } + return nullptr; +} + +static const ggml_backend_reg_i ggml_backend_cuda_reg_interface = { + /* .get_name = */ ggml_backend_cuda_reg_get_name, + /* .get_device_count = */ ggml_backend_cuda_reg_get_device_count, + /* .get_device = */ ggml_backend_cuda_reg_get_device, + /* .get_proc_address = */ ggml_backend_cuda_reg_get_proc_address, +}; + +// backend registry +ggml_backend_reg_t ggml_backend_cuda_reg() { + static ggml_backend_reg reg; + static bool initialized = false; + + { + static std::mutex mutex; + std::lock_guard lock(mutex); + if (!initialized) { + ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context; + const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32; + + for (int i = 0; i < ggml_cuda_info().device_count; i++) { + ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context; + dev_ctx->device = i; + dev_ctx->name = GGML_CUDA_NAME + std::to_string(i); + + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, i)); + dev_ctx->description = prop.name; + + char pci_bus_id[16] = {}; + snprintf(pci_bus_id, sizeof(pci_bus_id), "%04x:%02x:%02x.0", prop.pciDomainID, prop.pciBusID, prop.pciDeviceID); + dev_ctx->pci_bus_id = pci_bus_id; + dev_ctx->op_offload_min_batch_size = min_batch_size; + + ggml_backend_dev_t dev = new ggml_backend_device { + /* .iface = */ ggml_backend_cuda_device_interface, + /* .reg = */ ®, + /* .context = */ dev_ctx + }; + ctx->devices.push_back(dev); + } + + reg = ggml_backend_reg { + /* .api_version = */ GGML_BACKEND_API_VERSION, + /* .iface = */ ggml_backend_cuda_reg_interface, + /* .context = */ ctx + }; + } + + initialized = true; + } + + return ® +} + +ggml_backend_t ggml_backend_cuda_init(int device) { + if (device < 0 || device >= ggml_backend_cuda_get_device_count()) { + GGML_LOG_ERROR("%s: invalid device %d\n", __func__, device); + return nullptr; + } + + ggml_backend_cuda_context * ctx = new ggml_backend_cuda_context(device); + if (ctx == nullptr) { + GGML_LOG_ERROR("%s: failed to allocate context\n", __func__); + return nullptr; + } + + ggml_backend_t cuda_backend = new ggml_backend { + /* .guid = */ ggml_backend_cuda_guid(), + /* .iface = */ ggml_backend_cuda_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device), + /* .context = */ ctx, + }; + + return cuda_backend; +} + +GGML_BACKEND_DL_IMPL(ggml_backend_cuda_reg) From 52af99d3377e4e28260c3a26e03d7d39dfc4aa96 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 17:56:20 -0700 Subject: [PATCH 02/81] llama-bench: add --allreduce flag to select AllReduce provider Adds --allreduce to llama-bench (and via the shared field pattern, consistent with other multi-value flags). Useful for isolating hangs or regressions in tensor-parallel mode: pass --allreduce nccl to force NCCL and bypass the internal provider. Also fixes ggml_cuda_select_allreduce_provider() to treat an empty GGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when llama-bench sets it to "" for the "auto" case). Co-Authored-By: Claude Sonnet 4.6 xt gains ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/src/ggml-cuda/ggml-cuda.cu | 2 +- tools/llama-bench/llama-bench.cpp | 4902 +++++++++++++++-------------- 2 files changed, 2472 insertions(+), 2432 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index d18ead6cd07..1e00cbeb6d7 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1177,7 +1177,7 @@ struct ggml_backend_cuda_comm_context { static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( const std::vector & device_ids) { const char * env = getenv("GGML_CUDA_ALLREDUCE"); - if (env != nullptr) { + if (env != nullptr && env[0] != '\0') { if (strcmp(env, "internal") == 0) { return GGML_CUDA_ALLREDUCE_INTERNAL; } diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index e21a80e697b..65ad8199f6e 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -1,2431 +1,2471 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "build-info.h" -#include "common.h" -#include "download.h" -#include "fit.h" -#include "ggml.h" -#include "llama.h" - -#ifdef _WIN32 -# define WIN32_LEAN_AND_MEAN -# ifndef NOMINMAX -# define NOMINMAX -# endif -# include -#endif - -// utils -static uint64_t get_time_ns() { - using clock = std::chrono::high_resolution_clock; - return std::chrono::nanoseconds(clock::now().time_since_epoch()).count(); -} - -static bool tensor_buft_override_equal(const llama_model_tensor_buft_override& a, const llama_model_tensor_buft_override& b) { - if (a.pattern != b.pattern) { - // cString comparison that may be null - if (a.pattern == nullptr || b.pattern == nullptr) { - return false; - } - if (strcmp(a.pattern, b.pattern) != 0) { - return false; - } - } - if (a.buft != b.buft) { - return false; - } - return true; -} - -static bool vec_tensor_buft_override_equal(const std::vector& a, const std::vector& b) { - if (a.size() != b.size()) { - return false; - } - for (size_t i = 0; i < a.size(); i++) { - if (!tensor_buft_override_equal(a[i], b[i])) { - return false; - } - } - return true; -} - -static bool vec_vec_tensor_buft_override_equal(const std::vector>& a, const std::vector>& b) { - if (a.size() != b.size()) { - return false; - } - for (size_t i = 0; i < a.size(); i++) { - if (!vec_tensor_buft_override_equal(a[i], b[i])) { - return false; - } - } - return true; -} - -template static std::string join(const std::vector & values, const std::string & delim) { - std::ostringstream str; - for (size_t i = 0; i < values.size(); i++) { - str << values[i]; - if (i < values.size() - 1) { - str << delim; - } - } - return str.str(); -} - -template static std::vector transform_to_str(const std::vector & values, F f) { - std::vector str_values; - std::transform(values.begin(), values.end(), std::back_inserter(str_values), f); - return str_values; -} - -template static T avg(const std::vector & v) { - if (v.empty()) { - return 0; - } - T sum = std::accumulate(v.begin(), v.end(), T(0)); - return sum / (T) v.size(); -} - -template static T stdev(const std::vector & v) { - if (v.size() <= 1) { - return 0; - } - T mean = avg(v); - T sq_sum = std::inner_product(v.begin(), v.end(), v.begin(), T(0)); - T stdev = std::sqrt(sq_sum / (T) (v.size() - 1) - mean * mean * (T) v.size() / (T) (v.size() - 1)); - return stdev; -} - -static std::string get_cpu_info() { - std::vector cpu_list; - for (size_t i = 0; i < ggml_backend_dev_count(); i++) { - auto * dev = ggml_backend_dev_get(i); - auto dev_type = ggml_backend_dev_type(dev); - if (dev_type == GGML_BACKEND_DEVICE_TYPE_CPU || dev_type == GGML_BACKEND_DEVICE_TYPE_ACCEL) { - cpu_list.push_back(ggml_backend_dev_description(dev)); - } - } - return join(cpu_list, ", "); -} - -static std::string get_gpu_info() { - std::vector gpu_list; - for (size_t i = 0; i < ggml_backend_dev_count(); i++) { - auto * dev = ggml_backend_dev_get(i); - auto dev_type = ggml_backend_dev_type(dev); - if (dev_type == GGML_BACKEND_DEVICE_TYPE_GPU || dev_type == GGML_BACKEND_DEVICE_TYPE_IGPU) { - gpu_list.push_back(ggml_backend_dev_description(dev)); - } - } - return join(gpu_list, ", "); -} - -static std::vector parse_devices_arg(const std::string & value) { - std::vector devices; - std::string trimmed = string_strip(value); - if (trimmed.empty()) { - throw std::invalid_argument("no devices specified"); - } - if (trimmed == "auto") { - return devices; - } - - auto dev_names = string_split(trimmed, '/'); - if (dev_names.size() == 1 && string_strip(dev_names[0]) == "none") { - devices.push_back(nullptr); - return devices; - } - - for (auto & name : dev_names) { - std::string dev_name = string_strip(name); - if (dev_name.empty()) { - throw std::invalid_argument("invalid device specification"); - } - auto * dev = ggml_backend_dev_by_name(dev_name.c_str()); - if (!dev || ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU) { - throw std::invalid_argument(string_format("invalid device: %s", dev_name.c_str())); - } - devices.push_back(dev); - } - - devices.push_back(nullptr); - return devices; -} - -static void register_rpc_server_list(const std::string & servers) { - auto rpc_servers = string_split(servers, ','); - if (rpc_servers.empty()) { - throw std::invalid_argument("no RPC servers specified"); - } - - auto * rpc_reg = ggml_backend_reg_by_name("RPC"); - if (!rpc_reg) { - throw std::invalid_argument("failed to find RPC backend"); - } - - using add_rpc_server_fn = ggml_backend_reg_t (*)(const char * endpoint); - auto * ggml_backend_rpc_add_server_fn = (add_rpc_server_fn) ggml_backend_reg_get_proc_address(rpc_reg, "ggml_backend_rpc_add_server"); - if (!ggml_backend_rpc_add_server_fn) { - throw std::invalid_argument("failed to find RPC add server function"); - } - for (const auto & server : rpc_servers) { - auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); - ggml_backend_register(reg); - } -} - -static std::string devices_to_string(const std::vector & devices) { - if (devices.empty()) { - return "auto"; - } - - if (devices.size() == 1 && devices[0] == nullptr) { - return "none"; - } - - std::vector names; - for (auto * dev : devices) { - if (dev == nullptr) { - break; - } - names.push_back(ggml_backend_dev_name(dev)); - } - - return join(names, "/"); -} - -// command line params -enum output_formats { NONE, CSV, JSON, JSONL, MARKDOWN, SQL }; - -static const char * output_format_str(output_formats format) { - switch (format) { - case NONE: - return "none"; - case CSV: - return "csv"; - case JSON: - return "json"; - case JSONL: - return "jsonl"; - case MARKDOWN: - return "md"; - case SQL: - return "sql"; - default: - GGML_ABORT("invalid output format"); - } -} - -static bool output_format_from_str(const std::string & s, output_formats & format) { - if (s == "none") { - format = NONE; - } else if (s == "csv") { - format = CSV; - } else if (s == "json") { - format = JSON; - } else if (s == "jsonl") { - format = JSONL; - } else if (s == "md") { - format = MARKDOWN; - } else if (s == "sql") { - format = SQL; - } else { - return false; - } - return true; -} - -static const char * split_mode_str(llama_split_mode mode) { - switch (mode) { - case LLAMA_SPLIT_MODE_NONE: - return "none"; - case LLAMA_SPLIT_MODE_LAYER: - return "layer"; - case LLAMA_SPLIT_MODE_ROW: - return "row"; - case LLAMA_SPLIT_MODE_TENSOR: - return "tensor"; - default: - GGML_ABORT("invalid split mode"); - } -} - -static std::string pair_str(const std::pair & p) { - static char buf[32]; - snprintf(buf, sizeof(buf), "%d,%d", p.first, p.second); - return buf; -} - -static std::vector parse_int_range(const std::string & s) { - // first[-last[(+|*)step]] - std::regex range_regex(R"(^(\d+)(?:-(\d+)(?:([\+|\*])(\d+))?)?(?:,|$))"); - - std::smatch match; - std::string::const_iterator search_start(s.cbegin()); - std::vector result; - while (std::regex_search(search_start, s.cend(), match, range_regex)) { - int first = std::stoi(match[1]); - int last = match[2].matched ? std::stoi(match[2]) : first; - char op = match[3].matched ? match[3].str()[0] : '+'; - int step = match[4].matched ? std::stoi(match[4]) : 1; - - for (int i = first; i <= last;) { - result.push_back(i); - - int prev_i = i; - - if (op == '+') { - i += step; - } else if (op == '*') { - i *= step; - } else { - throw std::invalid_argument("invalid range format"); - } - - if (i <= prev_i) { - throw std::invalid_argument("invalid range"); - } - } - search_start = match.suffix().first; - } - - if (search_start != s.cend()) { - throw std::invalid_argument("invalid range format"); - } - - return result; -} - -struct cmd_params { - std::vector model; - std::vector hf_repo; - std::vector hf_file; - std::string hf_token; - std::vector n_prompt; - std::vector n_gen; - std::vector> n_pg; - std::vector n_depth; - std::vector n_batch; - std::vector n_ubatch; - std::vector type_k; - std::vector type_v; - std::vector n_threads; - std::vector cpu_mask; - std::vector cpu_strict; - std::vector poll; - std::vector n_gpu_layers; - std::vector n_cpu_moe; - std::vector split_mode; - std::vector main_gpu; - std::vector no_kv_offload; - std::vector flash_attn; - std::vector> devices; - std::vector> tensor_split; - std::vector> tensor_buft_overrides; - std::vector use_mmap; - std::vector use_direct_io; - std::vector embeddings; - std::vector no_op_offload; - std::vector no_host; - std::vector fit_params_target; - std::vector fit_params_min_ctx; - ggml_numa_strategy numa; - int reps; - ggml_sched_priority prio; - int delay; - bool verbose; - bool progress; - bool no_warmup; - output_formats output_format; - output_formats output_format_stderr; -}; - -static const cmd_params cmd_params_defaults = { - /* model */ { "models/7B/ggml-model-q4_0.gguf" }, - /* hf_repo */ {}, - /* hf_file */ {}, - /* hf_token */ "", - /* n_prompt */ { 512 }, - /* n_gen */ { 128 }, - /* n_pg */ {}, - /* n_depth */ { 0 }, - /* n_batch */ { 2048 }, - /* n_ubatch */ { 512 }, - /* type_k */ { GGML_TYPE_F16 }, - /* type_v */ { GGML_TYPE_F16 }, - /* n_threads */ { cpu_get_num_math() }, - /* cpu_mask */ { "0x0" }, - /* cpu_strict */ { false }, - /* poll */ { 50 }, - /* n_gpu_layers */ { 99 }, - /* n_cpu_moe */ { 0 }, - /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, - /* main_gpu */ { 0 }, - /* no_kv_offload */ { false }, - /* flash_attn */ { false }, - /* devices */ { {} }, - /* tensor_split */ { std::vector(llama_max_devices(), 0.0f) }, - /* tensor_buft_overrides*/ { std::vector{ { nullptr, nullptr } } }, - /* use_mmap */ { true }, - /* use_direct_io */ { false }, - /* embeddings */ { false }, - /* no_op_offload */ { false }, - /* no_host */ { false }, - /* fit_params_target */ { 0 }, - /* fit_params_min_ctx */ { 0 }, - /* numa */ GGML_NUMA_STRATEGY_DISABLED, - /* reps */ 5, - /* prio */ GGML_SCHED_PRIO_NORMAL, - /* delay */ 0, - /* verbose */ false, - /* progress */ false, - /* no_warmup */ false, - /* output_format */ MARKDOWN, - /* output_format_stderr */ NONE, -}; - -static void print_usage(int /* argc */, char ** argv) { - printf("usage: %s [options]\n", argv[0]); - printf("\n"); - printf("options:\n"); - printf(" -h, --help\n"); - printf(" --numa numa mode (default: disabled)\n"); - printf(" -r, --repetitions number of times to repeat each test (default: %d)\n", cmd_params_defaults.reps); - printf(" --prio <-1|0|1|2|3> process/thread priority (default: %d)\n", cmd_params_defaults.prio); - printf(" --delay <0...N> (seconds) delay between each test (default: %d)\n", cmd_params_defaults.delay); - printf(" -o, --output output format printed to stdout (default: %s)\n", output_format_str(cmd_params_defaults.output_format)); - printf(" -oe, --output-err output format printed to stderr (default: %s)\n", output_format_str(cmd_params_defaults.output_format_stderr)); - printf(" --list-devices list available devices and exit\n"); - printf(" -v, --verbose verbose output\n"); - printf(" --progress print test progress indicators\n"); - printf(" --no-warmup skip warmup runs before benchmarking\n"); - printf(" -fitt, --fit-target fit model to device memory with this margin per device in MiB (default: off)\n"); - printf(" -fitc, --fit-ctx minimum ctx size for --fit-target (default: 4096)\n"); - if (llama_supports_rpc()) { - printf(" -rpc, --rpc register RPC devices (comma separated)\n"); - } - printf("\n"); - printf("test parameters:\n"); - printf(" -m, --model (default: %s)\n", join(cmd_params_defaults.model, ",").c_str()); - printf(" -hf, -hfr, --hf-repo /[:quant] Hugging Face model repository; quant is optional, case-insensitive\n"); - printf(" default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.\n"); - printf(" example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M\n"); - printf(" (default: unused)\n"); - printf(" -hff, --hf-file Hugging Face model file. If specified, it will override the quant in --hf-repo\n"); - printf(" (default: unused)\n"); - printf(" -hft, --hf-token Hugging Face access token\n"); - printf(" (default: value from HF_TOKEN environment variable)\n"); - printf(" -p, --n-prompt (default: %s)\n", join(cmd_params_defaults.n_prompt, ",").c_str()); - printf(" -n, --n-gen (default: %s)\n", join(cmd_params_defaults.n_gen, ",").c_str()); - printf(" -pg (default: %s)\n", join(transform_to_str(cmd_params_defaults.n_pg, pair_str), ",").c_str()); - printf(" -d, --n-depth (default: %s)\n", join(cmd_params_defaults.n_depth, ",").c_str()); - printf(" -b, --batch-size (default: %s)\n", join(cmd_params_defaults.n_batch, ",").c_str()); - printf(" -ub, --ubatch-size (default: %s)\n", join(cmd_params_defaults.n_ubatch, ",").c_str()); - printf(" -ctk, --cache-type-k (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_k, ggml_type_name), ",").c_str()); - printf(" -ctv, --cache-type-v (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_v, ggml_type_name), ",").c_str()); - printf(" -t, --threads (default: %s)\n", join(cmd_params_defaults.n_threads, ",").c_str()); - printf(" -C, --cpu-mask (default: %s)\n", join(cmd_params_defaults.cpu_mask, ",").c_str()); - printf(" --cpu-strict <0|1> (default: %s)\n", join(cmd_params_defaults.cpu_strict, ",").c_str()); - printf(" --poll <0...100> (default: %s)\n", join(cmd_params_defaults.poll, ",").c_str()); - printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); - printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); - printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); - printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); - printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); - printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); - printf(" -dev, --device (default: auto)\n"); - printf(" -mmp, --mmap <0|1> (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str()); - printf(" -dio, --direct-io <0|1> (default: %s)\n", join(cmd_params_defaults.use_direct_io, ",").c_str()); - printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str()); - printf(" -ts, --tensor-split (default: 0)\n"); - printf(" -ot --override-tensor =;...\n"); - printf(" (default: disabled)\n"); - printf(" -nopo, --no-op-offload <0|1> (default: 0)\n"); - printf(" --no-host <0|1> (default: %s)\n", join(cmd_params_defaults.no_host, ",").c_str()); - printf("\n"); - printf( - "Multiple values can be given for each parameter by separating them with ','\n" - "or by specifying the parameter multiple times. Ranges can be given as\n" - "'first-last' or 'first-last+step' or 'first-last*mult'.\n"); -} - -static ggml_type ggml_type_from_name(const std::string & s) { - if (s == "f16") { - return GGML_TYPE_F16; - } - if (s == "bf16") { - return GGML_TYPE_BF16; - } - if (s == "q8_0") { - return GGML_TYPE_Q8_0; - } - if (s == "q4_0") { - return GGML_TYPE_Q4_0; - } - if (s == "q4_1") { - return GGML_TYPE_Q4_1; - } - if (s == "q5_0") { - return GGML_TYPE_Q5_0; - } - if (s == "q5_1") { - return GGML_TYPE_Q5_1; - } - if (s == "iq4_nl") { - return GGML_TYPE_IQ4_NL; - } - - return GGML_TYPE_COUNT; -} - -static cmd_params parse_cmd_params(int argc, char ** argv) { - cmd_params params; - std::string arg; - bool invalid_param = false; - const std::string arg_prefix = "--"; - const char split_delim = ','; - - params.verbose = cmd_params_defaults.verbose; - params.output_format = cmd_params_defaults.output_format; - params.output_format_stderr = cmd_params_defaults.output_format_stderr; - params.reps = cmd_params_defaults.reps; - params.numa = cmd_params_defaults.numa; - params.prio = cmd_params_defaults.prio; - params.delay = cmd_params_defaults.delay; - params.progress = cmd_params_defaults.progress; - params.no_warmup = cmd_params_defaults.no_warmup; - - if (const char * env = getenv("HF_TOKEN")) { - params.hf_token = env; - } - - for (int i = 1; i < argc; i++) { - arg = argv[i]; - if (arg.compare(0, arg_prefix.size(), arg_prefix) == 0) { - std::replace(arg.begin(), arg.end(), '_', '-'); - } - - try { - if (arg == "-h" || arg == "--help") { - print_usage(argc, argv); - exit(0); - } else if (arg == "-m" || arg == "--model") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.model.insert(params.model.end(), p.begin(), p.end()); - } else if (arg == "-hf" || arg == "-hfr" || arg == "--hf-repo") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.hf_repo.insert(params.hf_repo.end(), p.begin(), p.end()); - } else if (arg == "-hff" || arg == "--hf-file") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.hf_file.insert(params.hf_file.end(), p.begin(), p.end()); - } else if (arg == "-hft" || arg == "--hf-token") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.hf_token = argv[i]; - } else if (arg == "-p" || arg == "--n-prompt") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_prompt.insert(params.n_prompt.end(), p.begin(), p.end()); - } else if (arg == "-n" || arg == "--n-gen") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_gen.insert(params.n_gen.end(), p.begin(), p.end()); - } else if (arg == "-pg") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], ','); - if (p.size() != 2) { - invalid_param = true; - break; - } - params.n_pg.push_back({ std::stoi(p[0]), std::stoi(p[1]) }); - } else if (arg == "-d" || arg == "--n-depth") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_depth.insert(params.n_depth.end(), p.begin(), p.end()); - } else if (arg == "-b" || arg == "--batch-size") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_batch.insert(params.n_batch.end(), p.begin(), p.end()); - } else if (arg == "-ub" || arg == "--ubatch-size") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_ubatch.insert(params.n_ubatch.end(), p.begin(), p.end()); - } else if (arg == "-ctk" || arg == "--cache-type-k") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector types; - for (const auto & t : p) { - ggml_type gt = ggml_type_from_name(t); - if (gt == GGML_TYPE_COUNT) { - invalid_param = true; - break; - } - types.push_back(gt); - } - if (invalid_param) { - break; - } - params.type_k.insert(params.type_k.end(), types.begin(), types.end()); - } else if (arg == "-ctv" || arg == "--cache-type-v") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector types; - for (const auto & t : p) { - ggml_type gt = ggml_type_from_name(t); - if (gt == GGML_TYPE_COUNT) { - invalid_param = true; - break; - } - types.push_back(gt); - } - if (invalid_param) { - break; - } - params.type_v.insert(params.type_v.end(), types.begin(), types.end()); - } else if (arg == "-dev" || arg == "--device") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto combos = string_split(argv[i], split_delim); - for (const auto & combo : combos) { - try { - params.devices.push_back(parse_devices_arg(combo)); - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } - if (invalid_param) { - break; - } - } else if (arg == "--list-devices") { - std::vector devices; - for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { - auto * dev = ggml_backend_dev_get(i); - if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) { - devices.push_back(dev); - } - } - printf("Available devices:\n"); - if (devices.empty()) { - printf(" (none)\n"); - } - for (auto * dev : devices) { - size_t free, total; - ggml_backend_dev_memory(dev, &free, &total); - printf(" %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / 1024 / 1024, free / 1024 / 1024); - } - exit(0); - } else if (arg == "-t" || arg == "--threads") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_threads.insert(params.n_threads.end(), p.begin(), p.end()); - } else if (arg == "-C" || arg == "--cpu-mask") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.cpu_mask.insert(params.cpu_mask.end(), p.begin(), p.end()); - } else if (arg == "--cpu-strict") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.cpu_strict.insert(params.cpu_strict.end(), p.begin(), p.end()); - } else if (arg == "--poll") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.poll.insert(params.poll.end(), p.begin(), p.end()); - } else if (arg == "-ngl" || arg == "--n-gpu-layers") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_gpu_layers.insert(params.n_gpu_layers.end(), p.begin(), p.end()); - } else if (arg == "-ncmoe" || arg == "--n-cpu-moe") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_cpu_moe.insert(params.n_cpu_moe.end(), p.begin(), p.end()); - } else if (llama_supports_rpc() && (arg == "-rpc" || arg == "--rpc")) { - if (++i >= argc) { - invalid_param = true; - break; - } - try { - register_rpc_server_list(argv[i]); - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } else if (arg == "-sm" || arg == "--split-mode") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector modes; - for (const auto & m : p) { - llama_split_mode mode; - if (m == "none") { - mode = LLAMA_SPLIT_MODE_NONE; - } else if (m == "layer") { - mode = LLAMA_SPLIT_MODE_LAYER; - } else if (m == "row") { - mode = LLAMA_SPLIT_MODE_ROW; - } else if (m == "tensor") { - mode = LLAMA_SPLIT_MODE_TENSOR; - } else { - invalid_param = true; - break; - } - modes.push_back(mode); - } - if (invalid_param) { - break; - } - params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); - } else if (arg == "-mg" || arg == "--main-gpu") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.main_gpu = parse_int_range(argv[i]); - } else if (arg == "-nkvo" || arg == "--no-kv-offload") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_kv_offload.insert(params.no_kv_offload.end(), p.begin(), p.end()); - } else if (arg == "--numa") { - if (++i >= argc) { - invalid_param = true; - break; - } - std::string value(argv[i]); - if (value == "distribute" || value == "") { - params.numa = GGML_NUMA_STRATEGY_DISTRIBUTE; - } else if (value == "isolate") { - params.numa = GGML_NUMA_STRATEGY_ISOLATE; - } else if (value == "numactl") { - params.numa = GGML_NUMA_STRATEGY_NUMACTL; - } else { - invalid_param = true; - break; - } - } else if (arg == "-fa" || arg == "--flash-attn") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.flash_attn.insert(params.flash_attn.end(), p.begin(), p.end()); - } else if (arg == "-mmp" || arg == "--mmap") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.use_mmap.insert(params.use_mmap.end(), p.begin(), p.end()); - } else if (arg == "-dio" || arg == "--direct-io") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.use_direct_io.insert(params.use_direct_io.end(), p.begin(), p.end()); - } else if (arg == "-embd" || arg == "--embeddings") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.embeddings.insert(params.embeddings.end(), p.begin(), p.end()); - } else if (arg == "-nopo" || arg == "--no-op-offload") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_op_offload.insert(params.no_op_offload.end(), p.begin(), p.end()); - } else if (arg == "--no-host") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_host.insert(params.no_host.end(), p.begin(), p.end()); - } else if (arg == "-ts" || arg == "--tensor-split") { - if (++i >= argc) { - invalid_param = true; - break; - } - for (auto ts : string_split(argv[i], split_delim)) { - // split string by ; and / - const std::regex regex{ R"([;/]+)" }; - std::sregex_token_iterator it{ ts.begin(), ts.end(), regex, -1 }; - std::vector split_arg{ it, {} }; - GGML_ASSERT(split_arg.size() <= llama_max_devices()); - - std::vector tensor_split(llama_max_devices()); - for (size_t i = 0; i < llama_max_devices(); ++i) { - if (i < split_arg.size()) { - tensor_split[i] = std::stof(split_arg[i]); - } else { - tensor_split[i] = 0.0f; - } - } - params.tensor_split.push_back(tensor_split); - } - } else if (arg == "-ot" || arg == "--override-tensor") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto * value = argv[i]; - /* static */ std::map buft_list; - if (buft_list.empty()) { - // enumerate all the devices and add their buffer types to the list - for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { - auto * dev = ggml_backend_dev_get(i); - auto * buft = ggml_backend_dev_buffer_type(dev); - if (buft) { - buft_list[ggml_backend_buft_name(buft)] = buft; - } - } - } - auto override_group_span_len = std::strcspn(value, ","); - bool last_group = false; - do { - if (override_group_span_len == 0) { - // Adds an empty override-tensors for an empty span - params.tensor_buft_overrides.push_back({{}}); - if (value[override_group_span_len] == '\0') { - value = &value[override_group_span_len]; - last_group = true; - } else { - value = &value[override_group_span_len + 1]; - override_group_span_len = std::strcspn(value, ","); - } - continue; - } - // Stamps null terminators into the argv - // value for this option to avoid the - // memory leak present in the implementation - // over in arg.cpp. Acceptable because we - // only parse these args once in this program. - auto * override_group = value; - if (value[override_group_span_len] == '\0') { - value = &value[override_group_span_len]; - last_group = true; - } else { - value[override_group_span_len] = '\0'; - value = &value[override_group_span_len + 1]; - } - std::vector group_tensor_buft_overrides{}; - auto override_span_len = std::strcspn(override_group, ";"); - while (override_span_len > 0) { - auto * override = override_group; - if (override_group[override_span_len] != '\0') { - override_group[override_span_len] = '\0'; - override_group = &override_group[override_span_len + 1]; - } else { - override_group = &override_group[override_span_len]; - } - auto tensor_name_span_len = std::strcspn(override, "="); - if (tensor_name_span_len >= override_span_len) { - invalid_param = true; - break; - } - override[tensor_name_span_len] = '\0'; - auto * tensor_name = override; - auto * buffer_type = &override[tensor_name_span_len + 1]; - if (buft_list.find(buffer_type) == buft_list.end()) { - printf("error: unrecognized buffer type '%s'\n", buffer_type); - printf("Available buffer types:\n"); - for (const auto & it : buft_list) { - printf(" %s\n", ggml_backend_buft_name(it.second)); - } - invalid_param = true; - break; - } - group_tensor_buft_overrides.push_back({tensor_name, buft_list.at(buffer_type)}); - override_span_len = std::strcspn(override_group, ";"); - } - if (invalid_param) { - break; - } - group_tensor_buft_overrides.push_back({nullptr,nullptr}); - params.tensor_buft_overrides.push_back(group_tensor_buft_overrides); - override_group_span_len = std::strcspn(value, ","); - } while (!last_group); - } else if (arg == "-r" || arg == "--repetitions") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.reps = std::stoi(argv[i]); - } else if (arg == "--prio") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.prio = (enum ggml_sched_priority) std::stoi(argv[i]); - } else if (arg == "--delay") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.delay = std::stoi(argv[i]); - } else if (arg == "-o" || arg == "--output") { - if (++i >= argc) { - invalid_param = true; - break; - } - invalid_param = !output_format_from_str(argv[i], params.output_format); - } else if (arg == "-oe" || arg == "--output-err") { - if (++i >= argc) { - invalid_param = true; - break; - } - invalid_param = !output_format_from_str(argv[i], params.output_format_stderr); - } else if (arg == "-v" || arg == "--verbose") { - params.verbose = true; - } else if (arg == "--progress") { - params.progress = true; - } else if (arg == "--no-warmup") { - params.no_warmup = true; - } else if (arg == "-fitt" || arg == "--fit-target") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - params.fit_params_target.push_back(std::stoull(v)); - } - } else if (arg == "-fitc" || arg == "--fit-ctx") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - params.fit_params_min_ctx.push_back(std::stoul(v)); - } - } else { - invalid_param = true; - break; - } - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } - - if (invalid_param) { - fprintf(stderr, "error: invalid parameter for argument: %s\n", arg.c_str()); - print_usage(argc, argv); - exit(1); - } - - if (!params.hf_repo.empty()) { - for (size_t i = 0; i < params.hf_repo.size(); i++) { - common_params_model model; - - if (params.hf_file.empty() || params.hf_file[i].empty()) { - model.hf_repo = params.hf_repo[i]; - } else { - model.hf_repo = params.hf_repo[i]; - model.hf_file = params.hf_file[i]; - } - - common_download_opts opts; - opts.bearer_token = params.hf_token; - auto download_result = common_download_model(model, opts); - if (download_result.model_path.empty()) { - fprintf(stderr, "error: failed to download model from HuggingFace\n"); - exit(1); - } - - params.model.push_back(download_result.model_path); - } - } - - // set defaults - if (params.model.empty()) { - params.model = cmd_params_defaults.model; - } - if (params.n_prompt.empty()) { - params.n_prompt = cmd_params_defaults.n_prompt; - } - if (params.n_gen.empty()) { - params.n_gen = cmd_params_defaults.n_gen; - } - if (params.n_pg.empty()) { - params.n_pg = cmd_params_defaults.n_pg; - } - if (params.n_depth.empty()) { - params.n_depth = cmd_params_defaults.n_depth; - } - if (params.n_batch.empty()) { - params.n_batch = cmd_params_defaults.n_batch; - } - if (params.n_ubatch.empty()) { - params.n_ubatch = cmd_params_defaults.n_ubatch; - } - if (params.type_k.empty()) { - params.type_k = cmd_params_defaults.type_k; - } - if (params.type_v.empty()) { - params.type_v = cmd_params_defaults.type_v; - } - if (params.n_gpu_layers.empty()) { - params.n_gpu_layers = cmd_params_defaults.n_gpu_layers; - } - if (params.n_cpu_moe.empty()) { - params.n_cpu_moe = cmd_params_defaults.n_cpu_moe; - } - if (params.split_mode.empty()) { - params.split_mode = cmd_params_defaults.split_mode; - } - if (params.main_gpu.empty()) { - params.main_gpu = cmd_params_defaults.main_gpu; - } - if (params.no_kv_offload.empty()) { - params.no_kv_offload = cmd_params_defaults.no_kv_offload; - } - if (params.flash_attn.empty()) { - params.flash_attn = cmd_params_defaults.flash_attn; - } - if (params.devices.empty()) { - params.devices = cmd_params_defaults.devices; - } - if (params.tensor_split.empty()) { - params.tensor_split = cmd_params_defaults.tensor_split; - } - if (params.tensor_buft_overrides.empty()) { - params.tensor_buft_overrides = cmd_params_defaults.tensor_buft_overrides; - } - if (params.use_mmap.empty()) { - params.use_mmap = cmd_params_defaults.use_mmap; - } - if (params.use_direct_io.empty()) { - params.use_direct_io = cmd_params_defaults.use_direct_io; - } - if (params.embeddings.empty()) { - params.embeddings = cmd_params_defaults.embeddings; - } - if (params.no_op_offload.empty()) { - params.no_op_offload = cmd_params_defaults.no_op_offload; - } - if (params.no_host.empty()) { - params.no_host = cmd_params_defaults.no_host; - } - if (params.n_threads.empty()) { - params.n_threads = cmd_params_defaults.n_threads; - } - if (params.cpu_mask.empty()) { - params.cpu_mask = cmd_params_defaults.cpu_mask; - } - if (params.cpu_strict.empty()) { - params.cpu_strict = cmd_params_defaults.cpu_strict; - } - if (params.poll.empty()) { - params.poll = cmd_params_defaults.poll; - } - if (params.fit_params_target.empty()) { - params.fit_params_target = cmd_params_defaults.fit_params_target; - } - if (params.fit_params_min_ctx.empty()) { - params.fit_params_min_ctx = cmd_params_defaults.fit_params_min_ctx; - } - - return params; -} - -struct cmd_params_instance { - std::string model; - int n_prompt; - int n_gen; - int n_depth; - int n_batch; - int n_ubatch; - ggml_type type_k; - ggml_type type_v; - int n_threads; - std::string cpu_mask; - bool cpu_strict; - int poll; - int n_gpu_layers; - int n_cpu_moe; - llama_split_mode split_mode; - int main_gpu; - bool no_kv_offload; - bool flash_attn; - std::vector devices; - std::vector tensor_split; - std::vector tensor_buft_overrides; - bool use_mmap; - bool use_direct_io; - bool embeddings; - bool no_op_offload; - bool no_host; - size_t fit_target; - uint32_t fit_min_ctx; - - llama_model_params to_llama_mparams() const { - llama_model_params mparams = llama_model_default_params(); - - mparams.n_gpu_layers = n_gpu_layers; - if (!devices.empty()) { - mparams.devices = const_cast(devices.data()); - } - mparams.split_mode = split_mode; - mparams.main_gpu = main_gpu; - mparams.tensor_split = tensor_split.data(); - mparams.use_mmap = use_mmap; - mparams.use_direct_io = use_direct_io; - mparams.no_host = no_host; - - if (n_cpu_moe <= 0) { - if (tensor_buft_overrides.empty()) { - mparams.tensor_buft_overrides = nullptr; - } else { - GGML_ASSERT(tensor_buft_overrides.back().pattern == nullptr && - "Tensor buffer overrides not terminated with empty pattern"); - mparams.tensor_buft_overrides = tensor_buft_overrides.data(); - } - } else { - static std::vector merged; - static std::vector patterns; - - merged.clear(); - patterns.clear(); - - auto first = tensor_buft_overrides.begin(); - auto last = tensor_buft_overrides.end(); - if (first != last && (last - 1)->pattern == nullptr) { - --last; - } - merged.insert(merged.end(), first, last); - - patterns.reserve((size_t) n_cpu_moe); - merged.reserve(merged.size() + (size_t) n_cpu_moe + 1); - - for (int i = 0; i < n_cpu_moe; ++i) { - patterns.push_back(llm_ffn_exps_block_regex(i)); - merged.push_back({ patterns.back().c_str(), - ggml_backend_cpu_buffer_type() }); - } - - merged.push_back({ nullptr, nullptr }); - - mparams.tensor_buft_overrides = merged.data(); - } - - return mparams; - } - - bool equal_mparams(const cmd_params_instance & other) const { - return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && - split_mode == other.split_mode && - main_gpu == other.main_gpu && tensor_split == other.tensor_split && - use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && - devices == other.devices && - no_host == other.no_host && - vec_tensor_buft_override_equal(tensor_buft_overrides, other.tensor_buft_overrides); - } - - llama_context_params to_llama_cparams() const { - llama_context_params cparams = llama_context_default_params(); - - cparams.n_ctx = n_prompt + n_gen + n_depth; - cparams.n_batch = n_batch; - cparams.n_ubatch = n_ubatch; - cparams.type_k = type_k; - cparams.type_v = type_v; - cparams.offload_kqv = !no_kv_offload; - cparams.flash_attn_type = flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED : LLAMA_FLASH_ATTN_TYPE_DISABLED; - cparams.embeddings = embeddings; - cparams.op_offload = !no_op_offload; - cparams.swa_full = false; - - return cparams; - } -}; - -static std::vector get_cmd_params_instances(const cmd_params & params) { - std::vector instances; - - // this ordering minimizes the number of times that each model needs to be reloaded - // clang-format off - for (const auto & m : params.model) - for (const auto & fpt : params.fit_params_target) - for (const auto & fpc : params.fit_params_min_ctx) - for (const auto & nl : params.n_gpu_layers) - for (const auto & ncmoe : params.n_cpu_moe) - for (const auto & sm : params.split_mode) - for (const auto & mg : params.main_gpu) - for (const auto & devs : params.devices) - for (const auto & ts : params.tensor_split) - for (const auto & ot : params.tensor_buft_overrides) - for (const auto & mmp : params.use_mmap) - for (const auto & dio : params.use_direct_io) - for (const auto & noh : params.no_host) - for (const auto & embd : params.embeddings) - for (const auto & nopo : params.no_op_offload) - for (const auto & nb : params.n_batch) - for (const auto & nub : params.n_ubatch) - for (const auto & tk : params.type_k) - for (const auto & tv : params.type_v) - for (const auto & nkvo : params.no_kv_offload) - for (const auto & fa : params.flash_attn) - for (const auto & nt : params.n_threads) - for (const auto & cm : params.cpu_mask) - for (const auto & cs : params.cpu_strict) - for (const auto & nd : params.n_depth) - for (const auto & pl : params.poll) { - for (const auto & n_prompt : params.n_prompt) { - if (n_prompt == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ n_prompt, - /* .n_gen = */ 0, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - - for (const auto & n_gen : params.n_gen) { - if (n_gen == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ 0, - /* .n_gen = */ n_gen, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - - for (const auto & n_pg : params.n_pg) { - if (n_pg.first == 0 && n_pg.second == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ n_pg.first, - /* .n_gen = */ n_pg.second, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - } - // clang-format on - - return instances; -} - -struct test { - static const std::string build_commit; - static const int build_number; - const std::string cpu_info; - const std::string gpu_info; - std::string model_filename; - std::string model_type; - uint64_t model_size; - uint64_t model_n_params; - int n_batch; - int n_ubatch; - int n_threads; - std::string cpu_mask; - bool cpu_strict; - int poll; - ggml_type type_k; - ggml_type type_v; - int n_gpu_layers; - int n_cpu_moe; - llama_split_mode split_mode; - int main_gpu; - bool no_kv_offload; - bool flash_attn; - std::vector devices; - std::vector tensor_split; - std::vector tensor_buft_overrides; - bool use_mmap; - bool use_direct_io; - bool embeddings; - bool no_op_offload; - bool no_host; - size_t fit_target; - uint32_t fit_min_ctx; - int n_prompt; - int n_gen; - int n_depth; - std::string test_time; - std::vector samples_ns; - - test(const cmd_params_instance & inst, const llama_model * lmodel, const llama_context * ctx) : - cpu_info(get_cpu_info()), - gpu_info(get_gpu_info()) { - - model_filename = inst.model; - char buf[128]; - llama_model_desc(lmodel, buf, sizeof(buf)); - model_type = buf; - model_size = llama_model_size(lmodel); - model_n_params = llama_model_n_params(lmodel); - n_batch = inst.n_batch; - n_ubatch = inst.n_ubatch; - n_threads = inst.n_threads; - cpu_mask = inst.cpu_mask; - cpu_strict = inst.cpu_strict; - poll = inst.poll; - type_k = inst.type_k; - type_v = inst.type_v; - n_gpu_layers = inst.n_gpu_layers; - n_cpu_moe = inst.n_cpu_moe; - split_mode = inst.split_mode; - main_gpu = inst.main_gpu; - no_kv_offload = inst.no_kv_offload; - flash_attn = inst.flash_attn; - devices = inst.devices; - tensor_split = inst.tensor_split; - tensor_buft_overrides = inst.tensor_buft_overrides; - use_mmap = inst.use_mmap; - use_direct_io = inst.use_direct_io; - embeddings = inst.embeddings; - no_op_offload = inst.no_op_offload; - no_host = inst.no_host; - fit_target = inst.fit_target; - fit_min_ctx = inst.fit_min_ctx; - n_prompt = inst.n_prompt; - n_gen = inst.n_gen; - n_depth = inst.n_depth; - // RFC 3339 date-time format - time_t t = time(NULL); - std::strftime(buf, sizeof(buf), "%FT%TZ", gmtime(&t)); - test_time = buf; - - (void) ctx; - } - - uint64_t avg_ns() const { return ::avg(samples_ns); } - - uint64_t stdev_ns() const { return ::stdev(samples_ns); } - - std::vector get_ts() const { - int n_tokens = n_prompt + n_gen; - std::vector ts; - std::transform(samples_ns.begin(), samples_ns.end(), std::back_inserter(ts), - [n_tokens](uint64_t t) { return 1e9 * n_tokens / t; }); - return ts; - } - - double avg_ts() const { return ::avg(get_ts()); } - - double stdev_ts() const { return ::stdev(get_ts()); } - - static std::string get_backend() { - std::vector backends; - bool rpc_used = false; - for (size_t i = 0; i < ggml_backend_reg_count(); i++) { - auto * reg = ggml_backend_reg_get(i); - std::string name = ggml_backend_reg_name(reg); - if (string_starts_with(name, "RPC")) { - if (ggml_backend_reg_dev_count(reg) > 0) { - rpc_used = true; - } - } else { - if (name != "CPU") { - backends.push_back(ggml_backend_reg_name(reg)); - } - } - } - if (rpc_used) { - backends.push_back("RPC"); - } - return backends.empty() ? "CPU" : join(backends, ","); - } - - static const std::vector & get_fields() { - static const std::vector fields = { - "build_commit", "build_number", "cpu_info", "gpu_info", "backends", - "model_filename", "model_type", "model_size", "model_n_params", "n_batch", - "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", - "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", - "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", - "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", - "no_op_offload", "no_host", "fit_target", "fit_min_ctx", - "n_prompt", "n_gen", "n_depth", - "test_time", "avg_ns", "stddev_ns", "avg_ts", "stddev_ts" - }; - return fields; - } - - enum field_type { STRING, BOOL, INT, FLOAT }; - - static field_type get_field_type(const std::string & field) { - if (field == "build_number" || field == "n_batch" || field == "n_ubatch" || field == "n_threads" || - field == "poll" || field == "model_size" || field == "model_n_params" || field == "n_gpu_layers" || - field == "main_gpu" || field == "n_prompt" || field == "n_gen" || field == "n_depth" || field == "avg_ns" || - field == "stddev_ns" || field == "no_op_offload" || field == "n_cpu_moe" || - field == "fit_target" || field == "fit_min_ctx") { - return INT; - } - if (field == "f16_kv" || field == "no_kv_offload" || field == "cpu_strict" || field == "flash_attn" || - field == "use_mmap" || field == "use_direct_io" || field == "embeddings" || field == "no_host") { - return BOOL; - } - if (field == "avg_ts" || field == "stddev_ts") { - return FLOAT; - } - return STRING; - } - - std::vector get_values() const { - std::string tensor_split_str; - std::string tensor_buft_overrides_str; - int max_nonzero = 0; - for (size_t i = 0; i < llama_max_devices(); i++) { - if (tensor_split[i] > 0) { - max_nonzero = i; - } - } - for (int i = 0; i <= max_nonzero; i++) { - char buf[32]; - snprintf(buf, sizeof(buf), "%.2f", tensor_split[i]); - tensor_split_str += buf; - if (i < max_nonzero) { - tensor_split_str += "/"; - } - } - if (tensor_buft_overrides.size() == 1) { - // Last element of tensor_buft_overrides is always a null pattern - // so if it is only one element long, it must be a null pattern. - GGML_ASSERT(tensor_buft_overrides[0].pattern == nullptr); - tensor_buft_overrides_str += "none"; - } else { - for (size_t i = 0; i < tensor_buft_overrides.size()-1; i++) { - // Last element of tensor_buft_overrides is always a null pattern - if (tensor_buft_overrides[i].pattern == nullptr) { - tensor_buft_overrides_str += "none"; - } else { - tensor_buft_overrides_str += tensor_buft_overrides[i].pattern; - tensor_buft_overrides_str += "="; - tensor_buft_overrides_str += ggml_backend_buft_name(tensor_buft_overrides[i].buft); - } - if (i + 2 < tensor_buft_overrides.size()) { - tensor_buft_overrides_str += ";"; - } - } - } - std::vector values = { build_commit, - std::to_string(build_number), - cpu_info, - gpu_info, - get_backend(), - model_filename, - model_type, - std::to_string(model_size), - std::to_string(model_n_params), - std::to_string(n_batch), - std::to_string(n_ubatch), - std::to_string(n_threads), - cpu_mask, - std::to_string(cpu_strict), - std::to_string(poll), - ggml_type_name(type_k), - ggml_type_name(type_v), - std::to_string(n_gpu_layers), - std::to_string(n_cpu_moe), - split_mode_str(split_mode), - std::to_string(main_gpu), - std::to_string(no_kv_offload), - std::to_string(flash_attn), - devices_to_string(devices), - tensor_split_str, - tensor_buft_overrides_str, - std::to_string(use_mmap), - std::to_string(use_direct_io), - std::to_string(embeddings), - std::to_string(no_op_offload), - std::to_string(no_host), - std::to_string(fit_target), - std::to_string(fit_min_ctx), - std::to_string(n_prompt), - std::to_string(n_gen), - std::to_string(n_depth), - test_time, - std::to_string(avg_ns()), - std::to_string(stdev_ns()), - std::to_string(avg_ts()), - std::to_string(stdev_ts()) }; - return values; - } - - std::map get_map() const { - std::map map; - auto fields = get_fields(); - auto values = get_values(); - std::transform(fields.begin(), fields.end(), values.begin(), std::inserter(map, map.end()), - std::make_pair); - return map; - } -}; - -const std::string test::build_commit = llama_commit(); -const int test::build_number = llama_build_number(); - -struct printer { - virtual ~printer() {} - - FILE * fout; - - virtual void print_header(const cmd_params & params) { (void) params; } - - virtual void print_test(const test & t) = 0; - - virtual void print_footer() {} -}; - -struct csv_printer : public printer { - static std::string escape_csv(const std::string & field) { - std::string escaped = "\""; - for (auto c : field) { - if (c == '"') { - escaped += "\""; - } - escaped += c; - } - escaped += "\""; - return escaped; - } - - void print_header(const cmd_params & params) override { - std::vector fields = test::get_fields(); - fprintf(fout, "%s\n", join(fields, ",").c_str()); - (void) params; - } - - void print_test(const test & t) override { - std::vector values = t.get_values(); - std::transform(values.begin(), values.end(), values.begin(), escape_csv); - fprintf(fout, "%s\n", join(values, ",").c_str()); - } -}; - -static std::string escape_json(const std::string & value) { - std::string escaped; - for (auto c : value) { - if (c == '"') { - escaped += "\\\""; - } else if (c == '\\') { - escaped += "\\\\"; - } else if (c <= 0x1f) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", c); - escaped += buf; - } else { - escaped += c; - } - } - return escaped; -} - -static std::string format_json_value(const std::string & field, const std::string & value) { - switch (test::get_field_type(field)) { - case test::STRING: - return "\"" + escape_json(value) + "\""; - case test::BOOL: - return value == "0" ? "false" : "true"; - default: - return value; - } -} - -struct json_printer : public printer { - bool first = true; - - void print_header(const cmd_params & params) override { - fprintf(fout, "[\n"); - (void) params; - } - - void print_fields(const std::vector & fields, const std::vector & values) { - assert(fields.size() == values.size()); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, " \"%s\": %s,\n", fields.at(i).c_str(), - format_json_value(fields.at(i), values.at(i)).c_str()); - } - } - - void print_test(const test & t) override { - if (first) { - first = false; - } else { - fprintf(fout, ",\n"); - } - fprintf(fout, " {\n"); - print_fields(test::get_fields(), t.get_values()); - fprintf(fout, " \"samples_ns\": [ %s ],\n", join(t.samples_ns, ", ").c_str()); - fprintf(fout, " \"samples_ts\": [ %s ]\n", join(t.get_ts(), ", ").c_str()); - fprintf(fout, " }"); - fflush(fout); - } - - void print_footer() override { fprintf(fout, "\n]\n"); } -}; - -struct jsonl_printer : public printer { - void print_fields(const std::vector & fields, const std::vector & values) { - assert(fields.size() == values.size()); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, "\"%s\": %s, ", fields.at(i).c_str(), format_json_value(fields.at(i), values.at(i)).c_str()); - } - } - - void print_test(const test & t) override { - fprintf(fout, "{"); - print_fields(test::get_fields(), t.get_values()); - fprintf(fout, "\"samples_ns\": [ %s ],", join(t.samples_ns, ", ").c_str()); - fprintf(fout, "\"samples_ts\": [ %s ]", join(t.get_ts(), ", ").c_str()); - fprintf(fout, "}\n"); - fflush(fout); - } -}; - -struct markdown_printer : public printer { - std::vector fields; - - static int get_field_width(const std::string & field) { - if (field == "model") { - return -30; - } - if (field == "t/s") { - return 20; - } - if (field == "size" || field == "params") { - return 10; - } - if (field == "n_gpu_layers") { - return 3; - } - if (field == "n_threads") { - return 7; - } - if (field == "n_batch") { - return 7; - } - if (field == "n_ubatch") { - return 8; - } - if (field == "type_k" || field == "type_v") { - return 6; - } - if (field == "split_mode") { - return 6; - } - if (field == "flash_attn") { - return 2; - } - if (field == "devices") { - return -12; - } - if (field == "use_mmap") { - return 4; - } - if (field == "use_direct_io") { - return 3; - } - if (field == "test") { - return 15; - } - if (field == "no_op_offload") { - return 4; - } - if (field == "no_host") { - return 4; - } - - int width = std::max((int) field.length(), 10); - - if (test::get_field_type(field) == test::STRING) { - return -width; - } - return width; - } - - static std::string get_field_display_name(const std::string & field) { - if (field == "n_gpu_layers") { - return "ngl"; - } - if (field == "split_mode") { - return "sm"; - } - if (field == "n_threads") { - return "threads"; - } - if (field == "no_kv_offload") { - return "nkvo"; - } - if (field == "flash_attn") { - return "fa"; - } - if (field == "use_mmap") { - return "mmap"; - } - if (field == "use_direct_io") { - return "dio"; - } - if (field == "embeddings") { - return "embd"; - } - if (field == "no_op_offload") { - return "nopo"; - } - if (field == "no_host") { - return "noh"; - } - if (field == "devices") { - return "dev"; - } - if (field == "tensor_split") { - return "ts"; - } - if (field == "tensor_buft_overrides") { - return "ot"; - } - if (field == "fit_target") { - return "fitt"; - } - if (field == "fit_min_ctx") { - return "fitc"; - } - return field; - } - - void print_header(const cmd_params & params) override { - // select fields to print - fields.emplace_back("model"); - fields.emplace_back("size"); - fields.emplace_back("params"); - fields.emplace_back("backend"); - bool is_cpu_backend = test::get_backend().find("CPU") != std::string::npos || - test::get_backend().find("BLAS") != std::string::npos || - test::get_backend().find("ZenDNN") != std::string::npos; - if (!is_cpu_backend) { - fields.emplace_back("n_gpu_layers"); - } - if (params.n_cpu_moe.size() > 1 || params.n_cpu_moe != cmd_params_defaults.n_cpu_moe) { - fields.emplace_back("n_cpu_moe"); - } - if (params.n_threads.size() > 1 || params.n_threads != cmd_params_defaults.n_threads || is_cpu_backend) { - fields.emplace_back("n_threads"); - } - if (params.cpu_mask.size() > 1 || params.cpu_mask != cmd_params_defaults.cpu_mask) { - fields.emplace_back("cpu_mask"); - } - if (params.cpu_strict.size() > 1 || params.cpu_strict != cmd_params_defaults.cpu_strict) { - fields.emplace_back("cpu_strict"); - } - if (params.poll.size() > 1 || params.poll != cmd_params_defaults.poll) { - fields.emplace_back("poll"); - } - if (params.n_batch.size() > 1 || params.n_batch != cmd_params_defaults.n_batch) { - fields.emplace_back("n_batch"); - } - if (params.n_ubatch.size() > 1 || params.n_ubatch != cmd_params_defaults.n_ubatch) { - fields.emplace_back("n_ubatch"); - } - if (params.type_k.size() > 1 || params.type_k != cmd_params_defaults.type_k) { - fields.emplace_back("type_k"); - } - if (params.type_v.size() > 1 || params.type_v != cmd_params_defaults.type_v) { - fields.emplace_back("type_v"); - } - if (params.main_gpu.size() > 1 || params.main_gpu != cmd_params_defaults.main_gpu) { - fields.emplace_back("main_gpu"); - } - if (params.split_mode.size() > 1 || params.split_mode != cmd_params_defaults.split_mode) { - fields.emplace_back("split_mode"); - } - if (params.no_kv_offload.size() > 1 || params.no_kv_offload != cmd_params_defaults.no_kv_offload) { - fields.emplace_back("no_kv_offload"); - } - if (params.flash_attn.size() > 1 || params.flash_attn != cmd_params_defaults.flash_attn) { - fields.emplace_back("flash_attn"); - } - if (params.devices.size() > 1 || params.devices != cmd_params_defaults.devices) { - fields.emplace_back("devices"); - } - if (params.tensor_split.size() > 1 || params.tensor_split != cmd_params_defaults.tensor_split) { - fields.emplace_back("tensor_split"); - } - if (params.tensor_buft_overrides.size() > 1 || !vec_vec_tensor_buft_override_equal(params.tensor_buft_overrides, cmd_params_defaults.tensor_buft_overrides)) { - fields.emplace_back("tensor_buft_overrides"); - } - if (params.use_mmap.size() > 1 || params.use_mmap != cmd_params_defaults.use_mmap) { - fields.emplace_back("use_mmap"); - } - if (params.use_direct_io.size() > 1 || params.use_direct_io != cmd_params_defaults.use_direct_io) { - fields.emplace_back("use_direct_io"); - } - if (params.embeddings.size() > 1 || params.embeddings != cmd_params_defaults.embeddings) { - fields.emplace_back("embeddings"); - } - if (params.no_op_offload.size() > 1 || params.no_op_offload != cmd_params_defaults.no_op_offload) { - fields.emplace_back("no_op_offload"); - } - if (params.no_host.size() > 1 || params.no_host != cmd_params_defaults.no_host) { - fields.emplace_back("no_host"); - } - if (params.fit_params_target.size() > 1 || params.fit_params_target != cmd_params_defaults.fit_params_target) { - fields.emplace_back("fit_target"); - } - if (params.fit_params_min_ctx.size() > 1 || params.fit_params_min_ctx != cmd_params_defaults.fit_params_min_ctx) { - fields.emplace_back("fit_min_ctx"); - } - fields.emplace_back("test"); - fields.emplace_back("t/s"); - - fprintf(fout, "|"); - for (const auto & field : fields) { - fprintf(fout, " %*s |", get_field_width(field), get_field_display_name(field).c_str()); - } - fprintf(fout, "\n"); - fprintf(fout, "|"); - for (const auto & field : fields) { - int width = get_field_width(field); - fprintf(fout, " %s%s |", std::string(std::abs(width) - 1, '-').c_str(), width > 0 ? ":" : "-"); - } - fprintf(fout, "\n"); - } - - void print_test(const test & t) override { - std::map vmap = t.get_map(); - - fprintf(fout, "|"); - for (const auto & field : fields) { - std::string value; - char buf[128]; - if (field == "model") { - value = t.model_type; - } else if (field == "size") { - if (t.model_size < 1024 * 1024 * 1024) { - snprintf(buf, sizeof(buf), "%.2f MiB", t.model_size / 1024.0 / 1024.0); - } else { - snprintf(buf, sizeof(buf), "%.2f GiB", t.model_size / 1024.0 / 1024.0 / 1024.0); - } - value = buf; - } else if (field == "params") { - if (t.model_n_params < 1000 * 1000 * 1000) { - snprintf(buf, sizeof(buf), "%.2f M", t.model_n_params / 1e6); - } else { - snprintf(buf, sizeof(buf), "%.2f B", t.model_n_params / 1e9); - } - value = buf; - } else if (field == "backend") { - value = test::get_backend(); - } else if (field == "test") { - if (t.n_prompt > 0 && t.n_gen == 0) { - snprintf(buf, sizeof(buf), "pp%d", t.n_prompt); - } else if (t.n_gen > 0 && t.n_prompt == 0) { - snprintf(buf, sizeof(buf), "tg%d", t.n_gen); - } else { - snprintf(buf, sizeof(buf), "pp%d+tg%d", t.n_prompt, t.n_gen); - } - if (t.n_depth > 0) { - int len = strlen(buf); - snprintf(buf + len, sizeof(buf) - len, " @ d%d", t.n_depth); - } - value = buf; - } else if (field == "t/s") { - snprintf(buf, sizeof(buf), "%.2f ± %.2f", t.avg_ts(), t.stdev_ts()); - value = buf; - } else if (vmap.find(field) != vmap.end()) { - value = vmap.at(field); - } else { - assert(false); - exit(1); - } - - int width = get_field_width(field); - if (field == "t/s") { - // HACK: the utf-8 character is 2 bytes - width += 1; - } - fprintf(fout, " %*s |", width, value.c_str()); - } - fprintf(fout, "\n"); - } - - void print_footer() override { - fprintf(fout, "\nbuild: %s (%d)\n", test::build_commit.c_str(), test::build_number); - } -}; - -struct sql_printer : public printer { - static std::string get_sql_field_type(const std::string & field) { - switch (test::get_field_type(field)) { - case test::STRING: - return "TEXT"; - case test::BOOL: - case test::INT: - return "INTEGER"; - case test::FLOAT: - return "REAL"; - default: - assert(false); - exit(1); - } - } - - void print_header(const cmd_params & params) override { - std::vector fields = test::get_fields(); - fprintf(fout, "CREATE TABLE IF NOT EXISTS llama_bench (\n"); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, " %s %s%s\n", fields.at(i).c_str(), get_sql_field_type(fields.at(i)).c_str(), - i < fields.size() - 1 ? "," : ""); - } - fprintf(fout, ");\n"); - fprintf(fout, "\n"); - (void) params; - } - - void print_test(const test & t) override { - fprintf(fout, "INSERT INTO llama_bench (%s) ", join(test::get_fields(), ", ").c_str()); - fprintf(fout, "VALUES ("); - std::vector values = t.get_values(); - for (size_t i = 0; i < values.size(); i++) { - fprintf(fout, "'%s'%s", values.at(i).c_str(), i < values.size() - 1 ? ", " : ""); - } - fprintf(fout, ");\n"); - } -}; - -struct ctx_state { - int depth = 0; // in tokens - - std::vector buf; // the llama_context state buffer -}; - -static bool test_prompt(llama_context * ctx, int n_prompt, int n_batch, int n_threads) { - llama_set_n_threads(ctx, n_threads, n_threads); - - const llama_model * model = llama_get_model(ctx); - const llama_vocab * vocab = llama_model_get_vocab(model); - const int32_t n_vocab = llama_vocab_n_tokens(vocab); - - std::vector tokens(n_batch); - - int n_processed = 0; - - while (n_processed < n_prompt) { - int n_tokens = std::min(n_prompt - n_processed, n_batch); - tokens[0] = n_processed == 0 && llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; - for (int i = 1; i < n_tokens; i++) { - tokens[i] = std::rand() % n_vocab; - } - int res = llama_decode(ctx, llama_batch_get_one(tokens.data(), n_tokens)); - if (res != 0) { - fprintf(stderr, "%s: failed to decode prompt batch, res = %d\n", __func__, res); - return false; - } - n_processed += n_tokens; - } - - llama_synchronize(ctx); - return true; -} - -static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { - llama_set_n_threads(ctx, n_threads, n_threads); - - const llama_model * model = llama_get_model(ctx); - const llama_vocab * vocab = llama_model_get_vocab(model); - const int32_t n_vocab = llama_vocab_n_tokens(vocab); - - llama_token token = llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; - - for (int i = 0; i < n_gen; i++) { - int res = llama_decode(ctx, llama_batch_get_one(&token, 1)); - if (res != 0) { - fprintf(stderr, "%s: failed to decode generation batch, res = %d\n", __func__, res); - return false; - } - llama_synchronize(ctx); - token = std::rand() % n_vocab; - } - return true; -} - -static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { - (void) level; - (void) text; - (void) user_data; -} - -static std::unique_ptr create_printer(output_formats format) { - switch (format) { - case NONE: - return nullptr; - case CSV: - return std::unique_ptr(new csv_printer()); - case JSON: - return std::unique_ptr(new json_printer()); - case JSONL: - return std::unique_ptr(new jsonl_printer()); - case MARKDOWN: - return std::unique_ptr(new markdown_printer()); - case SQL: - return std::unique_ptr(new sql_printer()); - } - GGML_ABORT("fatal error"); -} - -int main(int argc, char ** argv) { - std::setlocale(LC_NUMERIC, "C"); - // try to set locale for unicode characters in markdown - std::setlocale(LC_CTYPE, ".UTF-8"); - -#if !defined(NDEBUG) - fprintf(stderr, "warning: asserts enabled, performance may be affected\n"); -#endif - -#if (defined(_MSC_VER) && defined(_DEBUG)) || (!defined(_MSC_VER) && !defined(__OPTIMIZE__)) - fprintf(stderr, "warning: debug build, performance may be affected\n"); -#endif - -#if defined(__SANITIZE_ADDRESS__) || defined(__SANITIZE_THREAD__) - fprintf(stderr, "warning: sanitizer enabled, performance may be affected\n"); -#endif - - // initialize backends - ggml_backend_load_all(); - - cmd_params params = parse_cmd_params(argc, argv); - - auto * cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU); - if (!cpu_dev) { - fprintf(stderr, "%s: error: CPU backend is not loaded\n", __func__); - return 1; - } - auto * cpu_reg = ggml_backend_dev_backend_reg(cpu_dev); - auto * ggml_threadpool_new_fn = (decltype(ggml_threadpool_new) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_new"); - auto * ggml_threadpool_free_fn = (decltype(ggml_threadpool_free) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_free"); - - // initialize llama.cpp - if (!params.verbose) { - llama_log_set(llama_null_log_callback, NULL); - } - llama_backend_init(); - llama_numa_init(params.numa); - - if (!set_process_priority(params.prio)) { - fprintf(stderr, "%s: error: failed to set process priority\n", __func__); - return 1; - } - - // initialize printer - std::unique_ptr p = create_printer(params.output_format); - std::unique_ptr p_err = create_printer(params.output_format_stderr); - - if (p) { - p->fout = stdout; - p->print_header(params); - } - - if (p_err) { - p_err->fout = stderr; - p_err->print_header(params); - } - - std::vector params_instances = get_cmd_params_instances(params); - - llama_model * lmodel = nullptr; - const cmd_params_instance * prev_inst = nullptr; - - // store the llama_context state at the previous depth that we performed a test - // ref: https://github.com/ggml-org/llama.cpp/pull/16944#issuecomment-3478151721 - ctx_state cstate; - - int params_idx = 0; - auto params_count = params_instances.size(); - for (const auto & inst : params_instances) { - params_idx++; - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: starting\n", params_idx, params_count); - } - auto mparams = inst.to_llama_mparams(); - auto cparams = inst.to_llama_cparams(); - - bool do_fit = inst.fit_target != cmd_params_defaults.fit_params_target[0] || - inst.fit_min_ctx != cmd_params_defaults.fit_params_min_ctx[0]; - - std::vector fit_tensor_split(llama_max_devices(), 0.0f); - std::vector fit_overrides(llama_max_tensor_buft_overrides(), {nullptr, nullptr}); - - if (do_fit) { - // free the previous model so fit sees full free VRAM - if (lmodel) { - llama_model_free(lmodel); - lmodel = nullptr; - prev_inst = nullptr; - } - - // use default n_gpu_layers and n_ctx so common_fit_params can adjust them - mparams.n_gpu_layers = llama_model_default_params().n_gpu_layers; - mparams.tensor_split = fit_tensor_split.data(); - mparams.tensor_buft_overrides = fit_overrides.data(); - cparams.n_ctx = 0; - - std::vector margins(llama_max_devices(), inst.fit_target * 1024 * 1024); - - uint32_t n_ctx_needed = inst.n_prompt + inst.n_gen + inst.n_depth; - cparams.n_ctx = std::max(cparams.n_ctx, n_ctx_needed); - - common_fit_params(inst.model.c_str(), &mparams, &cparams, - fit_tensor_split.data(), - fit_overrides.data(), - margins.data(), - inst.fit_min_ctx, - params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); - } - - // keep the same model between tests when possible - if (!lmodel || !prev_inst || !inst.equal_mparams(*prev_inst)) { - if (lmodel) { - llama_model_free(lmodel); - } - - lmodel = llama_model_load_from_file(inst.model.c_str(), mparams); - if (lmodel == NULL) { - fprintf(stderr, "%s: error: failed to load model '%s'\n", __func__, inst.model.c_str()); - return 1; - } - prev_inst = &inst; - } - - llama_context * ctx = llama_init_from_model(lmodel, cparams); - if (ctx == NULL) { - fprintf(stderr, "%s: error: failed to create context with model '%s'\n", __func__, inst.model.c_str()); - llama_model_free(lmodel); - return 1; - } - - test t(inst, lmodel, ctx); - - llama_memory_clear(llama_get_memory(ctx), false); - - // cool off before the test - if (params.delay) { - std::this_thread::sleep_for(std::chrono::seconds(params.delay)); - } - - struct ggml_threadpool_params tpp = ggml_threadpool_params_default(t.n_threads); - if (!parse_cpu_mask(t.cpu_mask, tpp.cpumask)) { - fprintf(stderr, "%s: failed to parse cpu-mask: %s\n", __func__, t.cpu_mask.c_str()); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - tpp.strict_cpu = t.cpu_strict; - tpp.poll = t.poll; - tpp.prio = params.prio; - - struct ggml_threadpool * threadpool = ggml_threadpool_new_fn(&tpp); - if (!threadpool) { - fprintf(stderr, "%s: threadpool create failed : n_threads %d\n", __func__, tpp.n_threads); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - - llama_attach_threadpool(ctx, threadpool, NULL); - - // warmup run - if (!params.no_warmup) { - if (t.n_prompt > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup prompt run\n", params_idx, params_count); - } - //test_prompt(ctx, std::min(t.n_batch, std::min(t.n_prompt, 32)), 0, t.n_batch, t.n_threads); - bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run prompt warmup\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - if (t.n_gen > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup generation run\n", params_idx, params_count); - } - bool res = test_gen(ctx, 1, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run gen warmup\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - } - - for (int i = 0; i < params.reps; i++) { - llama_memory_clear(llama_get_memory(ctx), false); - - if (t.n_depth > 0) { - bool is_cached = t.n_depth == cstate.depth; - - if (is_cached) { - // if previously we have computed at this depth, just restore the state - const size_t ret = llama_state_seq_set_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); - if (ret == 0) { - // if the old state is incompatible with the current context - reprocess from scratch - is_cached = false; - } - } - - if (!is_cached) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_prompt(ctx, t.n_depth, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run depth\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - - // store the context state for reuse in later runs - cstate.depth = t.n_depth; - cstate.buf.resize(llama_state_seq_get_size(ctx, 0)); - llama_state_seq_get_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); - } else { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d (cached)\n", params_idx, params_count, - i + 1, params.reps); - } - } - } - - uint64_t t_start = get_time_ns(); - - if (t.n_prompt > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: prompt run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run prompt\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - if (t.n_gen > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: generation run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_gen(ctx, t.n_gen, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run gen\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - - uint64_t t_ns = get_time_ns() - t_start; - t.samples_ns.push_back(t_ns); - } - - if (p) { - p->print_test(t); - fflush(p->fout); - } - - if (p_err) { - p_err->print_test(t); - fflush(p_err->fout); - } - - llama_perf_context_print(ctx); - - llama_free(ctx); - - ggml_threadpool_free_fn(threadpool); - } - - llama_model_free(lmodel); - - if (p) { - p->print_footer(); - } - - if (p_err) { - p_err->print_footer(); - } - - llama_backend_free(); - - return 0; -} +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "build-info.h" +#include "common.h" +#include "download.h" +#include "fit.h" +#include "ggml.h" +#include "llama.h" + +#ifdef _WIN32 +# define WIN32_LEAN_AND_MEAN +# ifndef NOMINMAX +# define NOMINMAX +# endif +# include +#endif + +// utils +static uint64_t get_time_ns() { + using clock = std::chrono::high_resolution_clock; + return std::chrono::nanoseconds(clock::now().time_since_epoch()).count(); +} + +static bool tensor_buft_override_equal(const llama_model_tensor_buft_override& a, const llama_model_tensor_buft_override& b) { + if (a.pattern != b.pattern) { + // cString comparison that may be null + if (a.pattern == nullptr || b.pattern == nullptr) { + return false; + } + if (strcmp(a.pattern, b.pattern) != 0) { + return false; + } + } + if (a.buft != b.buft) { + return false; + } + return true; +} + +static bool vec_tensor_buft_override_equal(const std::vector& a, const std::vector& b) { + if (a.size() != b.size()) { + return false; + } + for (size_t i = 0; i < a.size(); i++) { + if (!tensor_buft_override_equal(a[i], b[i])) { + return false; + } + } + return true; +} + +static bool vec_vec_tensor_buft_override_equal(const std::vector>& a, const std::vector>& b) { + if (a.size() != b.size()) { + return false; + } + for (size_t i = 0; i < a.size(); i++) { + if (!vec_tensor_buft_override_equal(a[i], b[i])) { + return false; + } + } + return true; +} + +template static std::string join(const std::vector & values, const std::string & delim) { + std::ostringstream str; + for (size_t i = 0; i < values.size(); i++) { + str << values[i]; + if (i < values.size() - 1) { + str << delim; + } + } + return str.str(); +} + +template static std::vector transform_to_str(const std::vector & values, F f) { + std::vector str_values; + std::transform(values.begin(), values.end(), std::back_inserter(str_values), f); + return str_values; +} + +template static T avg(const std::vector & v) { + if (v.empty()) { + return 0; + } + T sum = std::accumulate(v.begin(), v.end(), T(0)); + return sum / (T) v.size(); +} + +template static T stdev(const std::vector & v) { + if (v.size() <= 1) { + return 0; + } + T mean = avg(v); + T sq_sum = std::inner_product(v.begin(), v.end(), v.begin(), T(0)); + T stdev = std::sqrt(sq_sum / (T) (v.size() - 1) - mean * mean * (T) v.size() / (T) (v.size() - 1)); + return stdev; +} + +static std::string get_cpu_info() { + std::vector cpu_list; + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + auto * dev = ggml_backend_dev_get(i); + auto dev_type = ggml_backend_dev_type(dev); + if (dev_type == GGML_BACKEND_DEVICE_TYPE_CPU || dev_type == GGML_BACKEND_DEVICE_TYPE_ACCEL) { + cpu_list.push_back(ggml_backend_dev_description(dev)); + } + } + return join(cpu_list, ", "); +} + +static std::string get_gpu_info() { + std::vector gpu_list; + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + auto * dev = ggml_backend_dev_get(i); + auto dev_type = ggml_backend_dev_type(dev); + if (dev_type == GGML_BACKEND_DEVICE_TYPE_GPU || dev_type == GGML_BACKEND_DEVICE_TYPE_IGPU) { + gpu_list.push_back(ggml_backend_dev_description(dev)); + } + } + return join(gpu_list, ", "); +} + +static std::vector parse_devices_arg(const std::string & value) { + std::vector devices; + std::string trimmed = string_strip(value); + if (trimmed.empty()) { + throw std::invalid_argument("no devices specified"); + } + if (trimmed == "auto") { + return devices; + } + + auto dev_names = string_split(trimmed, '/'); + if (dev_names.size() == 1 && string_strip(dev_names[0]) == "none") { + devices.push_back(nullptr); + return devices; + } + + for (auto & name : dev_names) { + std::string dev_name = string_strip(name); + if (dev_name.empty()) { + throw std::invalid_argument("invalid device specification"); + } + auto * dev = ggml_backend_dev_by_name(dev_name.c_str()); + if (!dev || ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU) { + throw std::invalid_argument(string_format("invalid device: %s", dev_name.c_str())); + } + devices.push_back(dev); + } + + devices.push_back(nullptr); + return devices; +} + +static void register_rpc_server_list(const std::string & servers) { + auto rpc_servers = string_split(servers, ','); + if (rpc_servers.empty()) { + throw std::invalid_argument("no RPC servers specified"); + } + + auto * rpc_reg = ggml_backend_reg_by_name("RPC"); + if (!rpc_reg) { + throw std::invalid_argument("failed to find RPC backend"); + } + + using add_rpc_server_fn = ggml_backend_reg_t (*)(const char * endpoint); + auto * ggml_backend_rpc_add_server_fn = (add_rpc_server_fn) ggml_backend_reg_get_proc_address(rpc_reg, "ggml_backend_rpc_add_server"); + if (!ggml_backend_rpc_add_server_fn) { + throw std::invalid_argument("failed to find RPC add server function"); + } + for (const auto & server : rpc_servers) { + auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); + ggml_backend_register(reg); + } +} + +static std::string devices_to_string(const std::vector & devices) { + if (devices.empty()) { + return "auto"; + } + + if (devices.size() == 1 && devices[0] == nullptr) { + return "none"; + } + + std::vector names; + for (auto * dev : devices) { + if (dev == nullptr) { + break; + } + names.push_back(ggml_backend_dev_name(dev)); + } + + return join(names, "/"); +} + +// command line params +enum output_formats { NONE, CSV, JSON, JSONL, MARKDOWN, SQL }; + +static const char * output_format_str(output_formats format) { + switch (format) { + case NONE: + return "none"; + case CSV: + return "csv"; + case JSON: + return "json"; + case JSONL: + return "jsonl"; + case MARKDOWN: + return "md"; + case SQL: + return "sql"; + default: + GGML_ABORT("invalid output format"); + } +} + +static bool output_format_from_str(const std::string & s, output_formats & format) { + if (s == "none") { + format = NONE; + } else if (s == "csv") { + format = CSV; + } else if (s == "json") { + format = JSON; + } else if (s == "jsonl") { + format = JSONL; + } else if (s == "md") { + format = MARKDOWN; + } else if (s == "sql") { + format = SQL; + } else { + return false; + } + return true; +} + +static const char * split_mode_str(llama_split_mode mode) { + switch (mode) { + case LLAMA_SPLIT_MODE_NONE: + return "none"; + case LLAMA_SPLIT_MODE_LAYER: + return "layer"; + case LLAMA_SPLIT_MODE_ROW: + return "row"; + case LLAMA_SPLIT_MODE_TENSOR: + return "tensor"; + default: + GGML_ABORT("invalid split mode"); + } +} + +static std::string pair_str(const std::pair & p) { + static char buf[32]; + snprintf(buf, sizeof(buf), "%d,%d", p.first, p.second); + return buf; +} + +static std::vector parse_int_range(const std::string & s) { + // first[-last[(+|*)step]] + std::regex range_regex(R"(^(\d+)(?:-(\d+)(?:([\+|\*])(\d+))?)?(?:,|$))"); + + std::smatch match; + std::string::const_iterator search_start(s.cbegin()); + std::vector result; + while (std::regex_search(search_start, s.cend(), match, range_regex)) { + int first = std::stoi(match[1]); + int last = match[2].matched ? std::stoi(match[2]) : first; + char op = match[3].matched ? match[3].str()[0] : '+'; + int step = match[4].matched ? std::stoi(match[4]) : 1; + + for (int i = first; i <= last;) { + result.push_back(i); + + int prev_i = i; + + if (op == '+') { + i += step; + } else if (op == '*') { + i *= step; + } else { + throw std::invalid_argument("invalid range format"); + } + + if (i <= prev_i) { + throw std::invalid_argument("invalid range"); + } + } + search_start = match.suffix().first; + } + + if (search_start != s.cend()) { + throw std::invalid_argument("invalid range format"); + } + + return result; +} + +struct cmd_params { + std::vector model; + std::vector hf_repo; + std::vector hf_file; + std::string hf_token; + std::vector n_prompt; + std::vector n_gen; + std::vector> n_pg; + std::vector n_depth; + std::vector n_batch; + std::vector n_ubatch; + std::vector type_k; + std::vector type_v; + std::vector n_threads; + std::vector cpu_mask; + std::vector cpu_strict; + std::vector poll; + std::vector n_gpu_layers; + std::vector n_cpu_moe; + std::vector split_mode; + std::vector allreduce; + std::vector main_gpu; + std::vector no_kv_offload; + std::vector flash_attn; + std::vector> devices; + std::vector> tensor_split; + std::vector> tensor_buft_overrides; + std::vector use_mmap; + std::vector use_direct_io; + std::vector embeddings; + std::vector no_op_offload; + std::vector no_host; + std::vector fit_params_target; + std::vector fit_params_min_ctx; + ggml_numa_strategy numa; + int reps; + ggml_sched_priority prio; + int delay; + bool verbose; + bool progress; + bool no_warmup; + output_formats output_format; + output_formats output_format_stderr; +}; + +static const cmd_params cmd_params_defaults = { + /* model */ { "models/7B/ggml-model-q4_0.gguf" }, + /* hf_repo */ {}, + /* hf_file */ {}, + /* hf_token */ "", + /* n_prompt */ { 512 }, + /* n_gen */ { 128 }, + /* n_pg */ {}, + /* n_depth */ { 0 }, + /* n_batch */ { 2048 }, + /* n_ubatch */ { 512 }, + /* type_k */ { GGML_TYPE_F16 }, + /* type_v */ { GGML_TYPE_F16 }, + /* n_threads */ { cpu_get_num_math() }, + /* cpu_mask */ { "0x0" }, + /* cpu_strict */ { false }, + /* poll */ { 50 }, + /* n_gpu_layers */ { 99 }, + /* n_cpu_moe */ { 0 }, + /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, + /* allreduce */ { "auto" }, + /* main_gpu */ { 0 }, + /* no_kv_offload */ { false }, + /* flash_attn */ { false }, + /* devices */ { {} }, + /* tensor_split */ { std::vector(llama_max_devices(), 0.0f) }, + /* tensor_buft_overrides*/ { std::vector{ { nullptr, nullptr } } }, + /* use_mmap */ { true }, + /* use_direct_io */ { false }, + /* embeddings */ { false }, + /* no_op_offload */ { false }, + /* no_host */ { false }, + /* fit_params_target */ { 0 }, + /* fit_params_min_ctx */ { 0 }, + /* numa */ GGML_NUMA_STRATEGY_DISABLED, + /* reps */ 5, + /* prio */ GGML_SCHED_PRIO_NORMAL, + /* delay */ 0, + /* verbose */ false, + /* progress */ false, + /* no_warmup */ false, + /* output_format */ MARKDOWN, + /* output_format_stderr */ NONE, +}; + +static void print_usage(int /* argc */, char ** argv) { + printf("usage: %s [options]\n", argv[0]); + printf("\n"); + printf("options:\n"); + printf(" -h, --help\n"); + printf(" --numa numa mode (default: disabled)\n"); + printf(" -r, --repetitions number of times to repeat each test (default: %d)\n", cmd_params_defaults.reps); + printf(" --prio <-1|0|1|2|3> process/thread priority (default: %d)\n", cmd_params_defaults.prio); + printf(" --delay <0...N> (seconds) delay between each test (default: %d)\n", cmd_params_defaults.delay); + printf(" -o, --output output format printed to stdout (default: %s)\n", output_format_str(cmd_params_defaults.output_format)); + printf(" -oe, --output-err output format printed to stderr (default: %s)\n", output_format_str(cmd_params_defaults.output_format_stderr)); + printf(" --list-devices list available devices and exit\n"); + printf(" -v, --verbose verbose output\n"); + printf(" --progress print test progress indicators\n"); + printf(" --no-warmup skip warmup runs before benchmarking\n"); + printf(" -fitt, --fit-target fit model to device memory with this margin per device in MiB (default: off)\n"); + printf(" -fitc, --fit-ctx minimum ctx size for --fit-target (default: 4096)\n"); + if (llama_supports_rpc()) { + printf(" -rpc, --rpc register RPC devices (comma separated)\n"); + } + printf("\n"); + printf("test parameters:\n"); + printf(" -m, --model (default: %s)\n", join(cmd_params_defaults.model, ",").c_str()); + printf(" -hf, -hfr, --hf-repo /[:quant] Hugging Face model repository; quant is optional, case-insensitive\n"); + printf(" default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.\n"); + printf(" example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M\n"); + printf(" (default: unused)\n"); + printf(" -hff, --hf-file Hugging Face model file. If specified, it will override the quant in --hf-repo\n"); + printf(" (default: unused)\n"); + printf(" -hft, --hf-token Hugging Face access token\n"); + printf(" (default: value from HF_TOKEN environment variable)\n"); + printf(" -p, --n-prompt (default: %s)\n", join(cmd_params_defaults.n_prompt, ",").c_str()); + printf(" -n, --n-gen (default: %s)\n", join(cmd_params_defaults.n_gen, ",").c_str()); + printf(" -pg (default: %s)\n", join(transform_to_str(cmd_params_defaults.n_pg, pair_str), ",").c_str()); + printf(" -d, --n-depth (default: %s)\n", join(cmd_params_defaults.n_depth, ",").c_str()); + printf(" -b, --batch-size (default: %s)\n", join(cmd_params_defaults.n_batch, ",").c_str()); + printf(" -ub, --ubatch-size (default: %s)\n", join(cmd_params_defaults.n_ubatch, ",").c_str()); + printf(" -ctk, --cache-type-k (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_k, ggml_type_name), ",").c_str()); + printf(" -ctv, --cache-type-v (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_v, ggml_type_name), ",").c_str()); + printf(" -t, --threads (default: %s)\n", join(cmd_params_defaults.n_threads, ",").c_str()); + printf(" -C, --cpu-mask (default: %s)\n", join(cmd_params_defaults.cpu_mask, ",").c_str()); + printf(" --cpu-strict <0|1> (default: %s)\n", join(cmd_params_defaults.cpu_strict, ",").c_str()); + printf(" --poll <0...100> (default: %s)\n", join(cmd_params_defaults.poll, ",").c_str()); + printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); + printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); + printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); + printf(" --allreduce allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.allreduce, ",").c_str()); + printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); + printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); + printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); + printf(" -dev, --device (default: auto)\n"); + printf(" -mmp, --mmap <0|1> (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str()); + printf(" -dio, --direct-io <0|1> (default: %s)\n", join(cmd_params_defaults.use_direct_io, ",").c_str()); + printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str()); + printf(" -ts, --tensor-split (default: 0)\n"); + printf(" -ot --override-tensor =;...\n"); + printf(" (default: disabled)\n"); + printf(" -nopo, --no-op-offload <0|1> (default: 0)\n"); + printf(" --no-host <0|1> (default: %s)\n", join(cmd_params_defaults.no_host, ",").c_str()); + printf("\n"); + printf( + "Multiple values can be given for each parameter by separating them with ','\n" + "or by specifying the parameter multiple times. Ranges can be given as\n" + "'first-last' or 'first-last+step' or 'first-last*mult'.\n"); +} + +static ggml_type ggml_type_from_name(const std::string & s) { + if (s == "f16") { + return GGML_TYPE_F16; + } + if (s == "bf16") { + return GGML_TYPE_BF16; + } + if (s == "q8_0") { + return GGML_TYPE_Q8_0; + } + if (s == "q4_0") { + return GGML_TYPE_Q4_0; + } + if (s == "q4_1") { + return GGML_TYPE_Q4_1; + } + if (s == "q5_0") { + return GGML_TYPE_Q5_0; + } + if (s == "q5_1") { + return GGML_TYPE_Q5_1; + } + if (s == "iq4_nl") { + return GGML_TYPE_IQ4_NL; + } + + return GGML_TYPE_COUNT; +} + +static cmd_params parse_cmd_params(int argc, char ** argv) { + cmd_params params; + std::string arg; + bool invalid_param = false; + const std::string arg_prefix = "--"; + const char split_delim = ','; + + params.verbose = cmd_params_defaults.verbose; + params.output_format = cmd_params_defaults.output_format; + params.output_format_stderr = cmd_params_defaults.output_format_stderr; + params.reps = cmd_params_defaults.reps; + params.numa = cmd_params_defaults.numa; + params.prio = cmd_params_defaults.prio; + params.delay = cmd_params_defaults.delay; + params.progress = cmd_params_defaults.progress; + params.no_warmup = cmd_params_defaults.no_warmup; + + if (const char * env = getenv("HF_TOKEN")) { + params.hf_token = env; + } + + for (int i = 1; i < argc; i++) { + arg = argv[i]; + if (arg.compare(0, arg_prefix.size(), arg_prefix) == 0) { + std::replace(arg.begin(), arg.end(), '_', '-'); + } + + try { + if (arg == "-h" || arg == "--help") { + print_usage(argc, argv); + exit(0); + } else if (arg == "-m" || arg == "--model") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.model.insert(params.model.end(), p.begin(), p.end()); + } else if (arg == "-hf" || arg == "-hfr" || arg == "--hf-repo") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.hf_repo.insert(params.hf_repo.end(), p.begin(), p.end()); + } else if (arg == "-hff" || arg == "--hf-file") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.hf_file.insert(params.hf_file.end(), p.begin(), p.end()); + } else if (arg == "-hft" || arg == "--hf-token") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.hf_token = argv[i]; + } else if (arg == "-p" || arg == "--n-prompt") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_prompt.insert(params.n_prompt.end(), p.begin(), p.end()); + } else if (arg == "-n" || arg == "--n-gen") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_gen.insert(params.n_gen.end(), p.begin(), p.end()); + } else if (arg == "-pg") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], ','); + if (p.size() != 2) { + invalid_param = true; + break; + } + params.n_pg.push_back({ std::stoi(p[0]), std::stoi(p[1]) }); + } else if (arg == "-d" || arg == "--n-depth") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_depth.insert(params.n_depth.end(), p.begin(), p.end()); + } else if (arg == "-b" || arg == "--batch-size") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_batch.insert(params.n_batch.end(), p.begin(), p.end()); + } else if (arg == "-ub" || arg == "--ubatch-size") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_ubatch.insert(params.n_ubatch.end(), p.begin(), p.end()); + } else if (arg == "-ctk" || arg == "--cache-type-k") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector types; + for (const auto & t : p) { + ggml_type gt = ggml_type_from_name(t); + if (gt == GGML_TYPE_COUNT) { + invalid_param = true; + break; + } + types.push_back(gt); + } + if (invalid_param) { + break; + } + params.type_k.insert(params.type_k.end(), types.begin(), types.end()); + } else if (arg == "-ctv" || arg == "--cache-type-v") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector types; + for (const auto & t : p) { + ggml_type gt = ggml_type_from_name(t); + if (gt == GGML_TYPE_COUNT) { + invalid_param = true; + break; + } + types.push_back(gt); + } + if (invalid_param) { + break; + } + params.type_v.insert(params.type_v.end(), types.begin(), types.end()); + } else if (arg == "-dev" || arg == "--device") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto combos = string_split(argv[i], split_delim); + for (const auto & combo : combos) { + try { + params.devices.push_back(parse_devices_arg(combo)); + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } + if (invalid_param) { + break; + } + } else if (arg == "--list-devices") { + std::vector devices; + for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { + auto * dev = ggml_backend_dev_get(i); + if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) { + devices.push_back(dev); + } + } + printf("Available devices:\n"); + if (devices.empty()) { + printf(" (none)\n"); + } + for (auto * dev : devices) { + size_t free, total; + ggml_backend_dev_memory(dev, &free, &total); + printf(" %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / 1024 / 1024, free / 1024 / 1024); + } + exit(0); + } else if (arg == "-t" || arg == "--threads") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_threads.insert(params.n_threads.end(), p.begin(), p.end()); + } else if (arg == "-C" || arg == "--cpu-mask") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.cpu_mask.insert(params.cpu_mask.end(), p.begin(), p.end()); + } else if (arg == "--cpu-strict") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.cpu_strict.insert(params.cpu_strict.end(), p.begin(), p.end()); + } else if (arg == "--poll") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.poll.insert(params.poll.end(), p.begin(), p.end()); + } else if (arg == "-ngl" || arg == "--n-gpu-layers") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_gpu_layers.insert(params.n_gpu_layers.end(), p.begin(), p.end()); + } else if (arg == "-ncmoe" || arg == "--n-cpu-moe") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_cpu_moe.insert(params.n_cpu_moe.end(), p.begin(), p.end()); + } else if (llama_supports_rpc() && (arg == "-rpc" || arg == "--rpc")) { + if (++i >= argc) { + invalid_param = true; + break; + } + try { + register_rpc_server_list(argv[i]); + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } else if (arg == "-sm" || arg == "--split-mode") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector modes; + for (const auto & m : p) { + llama_split_mode mode; + if (m == "none") { + mode = LLAMA_SPLIT_MODE_NONE; + } else if (m == "layer") { + mode = LLAMA_SPLIT_MODE_LAYER; + } else if (m == "row") { + mode = LLAMA_SPLIT_MODE_ROW; + } else if (m == "tensor") { + mode = LLAMA_SPLIT_MODE_TENSOR; + } else { + invalid_param = true; + break; + } + modes.push_back(mode); + } + if (invalid_param) { + break; + } + params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); + } else if (arg == "--allreduce") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + if (v != "auto" && v != "nccl" && v != "internal") { + invalid_param = true; + break; + } + } + if (invalid_param) { + break; + } + params.allreduce.insert(params.allreduce.end(), p.begin(), p.end()); + } else if (arg == "-mg" || arg == "--main-gpu") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.main_gpu = parse_int_range(argv[i]); + } else if (arg == "-nkvo" || arg == "--no-kv-offload") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_kv_offload.insert(params.no_kv_offload.end(), p.begin(), p.end()); + } else if (arg == "--numa") { + if (++i >= argc) { + invalid_param = true; + break; + } + std::string value(argv[i]); + if (value == "distribute" || value == "") { + params.numa = GGML_NUMA_STRATEGY_DISTRIBUTE; + } else if (value == "isolate") { + params.numa = GGML_NUMA_STRATEGY_ISOLATE; + } else if (value == "numactl") { + params.numa = GGML_NUMA_STRATEGY_NUMACTL; + } else { + invalid_param = true; + break; + } + } else if (arg == "-fa" || arg == "--flash-attn") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.flash_attn.insert(params.flash_attn.end(), p.begin(), p.end()); + } else if (arg == "-mmp" || arg == "--mmap") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.use_mmap.insert(params.use_mmap.end(), p.begin(), p.end()); + } else if (arg == "-dio" || arg == "--direct-io") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.use_direct_io.insert(params.use_direct_io.end(), p.begin(), p.end()); + } else if (arg == "-embd" || arg == "--embeddings") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.embeddings.insert(params.embeddings.end(), p.begin(), p.end()); + } else if (arg == "-nopo" || arg == "--no-op-offload") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_op_offload.insert(params.no_op_offload.end(), p.begin(), p.end()); + } else if (arg == "--no-host") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_host.insert(params.no_host.end(), p.begin(), p.end()); + } else if (arg == "-ts" || arg == "--tensor-split") { + if (++i >= argc) { + invalid_param = true; + break; + } + for (auto ts : string_split(argv[i], split_delim)) { + // split string by ; and / + const std::regex regex{ R"([;/]+)" }; + std::sregex_token_iterator it{ ts.begin(), ts.end(), regex, -1 }; + std::vector split_arg{ it, {} }; + GGML_ASSERT(split_arg.size() <= llama_max_devices()); + + std::vector tensor_split(llama_max_devices()); + for (size_t i = 0; i < llama_max_devices(); ++i) { + if (i < split_arg.size()) { + tensor_split[i] = std::stof(split_arg[i]); + } else { + tensor_split[i] = 0.0f; + } + } + params.tensor_split.push_back(tensor_split); + } + } else if (arg == "-ot" || arg == "--override-tensor") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto * value = argv[i]; + /* static */ std::map buft_list; + if (buft_list.empty()) { + // enumerate all the devices and add their buffer types to the list + for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { + auto * dev = ggml_backend_dev_get(i); + auto * buft = ggml_backend_dev_buffer_type(dev); + if (buft) { + buft_list[ggml_backend_buft_name(buft)] = buft; + } + } + } + auto override_group_span_len = std::strcspn(value, ","); + bool last_group = false; + do { + if (override_group_span_len == 0) { + // Adds an empty override-tensors for an empty span + params.tensor_buft_overrides.push_back({{}}); + if (value[override_group_span_len] == '\0') { + value = &value[override_group_span_len]; + last_group = true; + } else { + value = &value[override_group_span_len + 1]; + override_group_span_len = std::strcspn(value, ","); + } + continue; + } + // Stamps null terminators into the argv + // value for this option to avoid the + // memory leak present in the implementation + // over in arg.cpp. Acceptable because we + // only parse these args once in this program. + auto * override_group = value; + if (value[override_group_span_len] == '\0') { + value = &value[override_group_span_len]; + last_group = true; + } else { + value[override_group_span_len] = '\0'; + value = &value[override_group_span_len + 1]; + } + std::vector group_tensor_buft_overrides{}; + auto override_span_len = std::strcspn(override_group, ";"); + while (override_span_len > 0) { + auto * override = override_group; + if (override_group[override_span_len] != '\0') { + override_group[override_span_len] = '\0'; + override_group = &override_group[override_span_len + 1]; + } else { + override_group = &override_group[override_span_len]; + } + auto tensor_name_span_len = std::strcspn(override, "="); + if (tensor_name_span_len >= override_span_len) { + invalid_param = true; + break; + } + override[tensor_name_span_len] = '\0'; + auto * tensor_name = override; + auto * buffer_type = &override[tensor_name_span_len + 1]; + if (buft_list.find(buffer_type) == buft_list.end()) { + printf("error: unrecognized buffer type '%s'\n", buffer_type); + printf("Available buffer types:\n"); + for (const auto & it : buft_list) { + printf(" %s\n", ggml_backend_buft_name(it.second)); + } + invalid_param = true; + break; + } + group_tensor_buft_overrides.push_back({tensor_name, buft_list.at(buffer_type)}); + override_span_len = std::strcspn(override_group, ";"); + } + if (invalid_param) { + break; + } + group_tensor_buft_overrides.push_back({nullptr,nullptr}); + params.tensor_buft_overrides.push_back(group_tensor_buft_overrides); + override_group_span_len = std::strcspn(value, ","); + } while (!last_group); + } else if (arg == "-r" || arg == "--repetitions") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.reps = std::stoi(argv[i]); + } else if (arg == "--prio") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.prio = (enum ggml_sched_priority) std::stoi(argv[i]); + } else if (arg == "--delay") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.delay = std::stoi(argv[i]); + } else if (arg == "-o" || arg == "--output") { + if (++i >= argc) { + invalid_param = true; + break; + } + invalid_param = !output_format_from_str(argv[i], params.output_format); + } else if (arg == "-oe" || arg == "--output-err") { + if (++i >= argc) { + invalid_param = true; + break; + } + invalid_param = !output_format_from_str(argv[i], params.output_format_stderr); + } else if (arg == "-v" || arg == "--verbose") { + params.verbose = true; + } else if (arg == "--progress") { + params.progress = true; + } else if (arg == "--no-warmup") { + params.no_warmup = true; + } else if (arg == "-fitt" || arg == "--fit-target") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + params.fit_params_target.push_back(std::stoull(v)); + } + } else if (arg == "-fitc" || arg == "--fit-ctx") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + params.fit_params_min_ctx.push_back(std::stoul(v)); + } + } else { + invalid_param = true; + break; + } + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } + + if (invalid_param) { + fprintf(stderr, "error: invalid parameter for argument: %s\n", arg.c_str()); + print_usage(argc, argv); + exit(1); + } + + if (!params.hf_repo.empty()) { + for (size_t i = 0; i < params.hf_repo.size(); i++) { + common_params_model model; + + if (params.hf_file.empty() || params.hf_file[i].empty()) { + model.hf_repo = params.hf_repo[i]; + } else { + model.hf_repo = params.hf_repo[i]; + model.hf_file = params.hf_file[i]; + } + + common_download_opts opts; + opts.bearer_token = params.hf_token; + auto download_result = common_download_model(model, opts); + if (download_result.model_path.empty()) { + fprintf(stderr, "error: failed to download model from HuggingFace\n"); + exit(1); + } + + params.model.push_back(download_result.model_path); + } + } + + // set defaults + if (params.model.empty()) { + params.model = cmd_params_defaults.model; + } + if (params.n_prompt.empty()) { + params.n_prompt = cmd_params_defaults.n_prompt; + } + if (params.n_gen.empty()) { + params.n_gen = cmd_params_defaults.n_gen; + } + if (params.n_pg.empty()) { + params.n_pg = cmd_params_defaults.n_pg; + } + if (params.n_depth.empty()) { + params.n_depth = cmd_params_defaults.n_depth; + } + if (params.n_batch.empty()) { + params.n_batch = cmd_params_defaults.n_batch; + } + if (params.n_ubatch.empty()) { + params.n_ubatch = cmd_params_defaults.n_ubatch; + } + if (params.type_k.empty()) { + params.type_k = cmd_params_defaults.type_k; + } + if (params.type_v.empty()) { + params.type_v = cmd_params_defaults.type_v; + } + if (params.n_gpu_layers.empty()) { + params.n_gpu_layers = cmd_params_defaults.n_gpu_layers; + } + if (params.n_cpu_moe.empty()) { + params.n_cpu_moe = cmd_params_defaults.n_cpu_moe; + } + if (params.split_mode.empty()) { + params.split_mode = cmd_params_defaults.split_mode; + } + if (params.allreduce.empty()) { + params.allreduce = cmd_params_defaults.allreduce; + } + if (params.main_gpu.empty()) { + params.main_gpu = cmd_params_defaults.main_gpu; + } + if (params.no_kv_offload.empty()) { + params.no_kv_offload = cmd_params_defaults.no_kv_offload; + } + if (params.flash_attn.empty()) { + params.flash_attn = cmd_params_defaults.flash_attn; + } + if (params.devices.empty()) { + params.devices = cmd_params_defaults.devices; + } + if (params.tensor_split.empty()) { + params.tensor_split = cmd_params_defaults.tensor_split; + } + if (params.tensor_buft_overrides.empty()) { + params.tensor_buft_overrides = cmd_params_defaults.tensor_buft_overrides; + } + if (params.use_mmap.empty()) { + params.use_mmap = cmd_params_defaults.use_mmap; + } + if (params.use_direct_io.empty()) { + params.use_direct_io = cmd_params_defaults.use_direct_io; + } + if (params.embeddings.empty()) { + params.embeddings = cmd_params_defaults.embeddings; + } + if (params.no_op_offload.empty()) { + params.no_op_offload = cmd_params_defaults.no_op_offload; + } + if (params.no_host.empty()) { + params.no_host = cmd_params_defaults.no_host; + } + if (params.n_threads.empty()) { + params.n_threads = cmd_params_defaults.n_threads; + } + if (params.cpu_mask.empty()) { + params.cpu_mask = cmd_params_defaults.cpu_mask; + } + if (params.cpu_strict.empty()) { + params.cpu_strict = cmd_params_defaults.cpu_strict; + } + if (params.poll.empty()) { + params.poll = cmd_params_defaults.poll; + } + if (params.fit_params_target.empty()) { + params.fit_params_target = cmd_params_defaults.fit_params_target; + } + if (params.fit_params_min_ctx.empty()) { + params.fit_params_min_ctx = cmd_params_defaults.fit_params_min_ctx; + } + + return params; +} + +struct cmd_params_instance { + std::string model; + int n_prompt; + int n_gen; + int n_depth; + int n_batch; + int n_ubatch; + ggml_type type_k; + ggml_type type_v; + int n_threads; + std::string cpu_mask; + bool cpu_strict; + int poll; + int n_gpu_layers; + int n_cpu_moe; + llama_split_mode split_mode; + std::string allreduce; + int main_gpu; + bool no_kv_offload; + bool flash_attn; + std::vector devices; + std::vector tensor_split; + std::vector tensor_buft_overrides; + bool use_mmap; + bool use_direct_io; + bool embeddings; + bool no_op_offload; + bool no_host; + size_t fit_target; + uint32_t fit_min_ctx; + + llama_model_params to_llama_mparams() const { + llama_model_params mparams = llama_model_default_params(); + + mparams.n_gpu_layers = n_gpu_layers; + if (!devices.empty()) { + mparams.devices = const_cast(devices.data()); + } + mparams.split_mode = split_mode; + mparams.main_gpu = main_gpu; + mparams.tensor_split = tensor_split.data(); + mparams.use_mmap = use_mmap; + mparams.use_direct_io = use_direct_io; + mparams.no_host = no_host; + + if (n_cpu_moe <= 0) { + if (tensor_buft_overrides.empty()) { + mparams.tensor_buft_overrides = nullptr; + } else { + GGML_ASSERT(tensor_buft_overrides.back().pattern == nullptr && + "Tensor buffer overrides not terminated with empty pattern"); + mparams.tensor_buft_overrides = tensor_buft_overrides.data(); + } + } else { + static std::vector merged; + static std::vector patterns; + + merged.clear(); + patterns.clear(); + + auto first = tensor_buft_overrides.begin(); + auto last = tensor_buft_overrides.end(); + if (first != last && (last - 1)->pattern == nullptr) { + --last; + } + merged.insert(merged.end(), first, last); + + patterns.reserve((size_t) n_cpu_moe); + merged.reserve(merged.size() + (size_t) n_cpu_moe + 1); + + for (int i = 0; i < n_cpu_moe; ++i) { + patterns.push_back(llm_ffn_exps_block_regex(i)); + merged.push_back({ patterns.back().c_str(), + ggml_backend_cpu_buffer_type() }); + } + + merged.push_back({ nullptr, nullptr }); + + mparams.tensor_buft_overrides = merged.data(); + } + + return mparams; + } + + bool equal_mparams(const cmd_params_instance & other) const { + return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && + split_mode == other.split_mode && allreduce == other.allreduce && + main_gpu == other.main_gpu && tensor_split == other.tensor_split && + use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && + devices == other.devices && + no_host == other.no_host && + vec_tensor_buft_override_equal(tensor_buft_overrides, other.tensor_buft_overrides); + } + + llama_context_params to_llama_cparams() const { + llama_context_params cparams = llama_context_default_params(); + + cparams.n_ctx = n_prompt + n_gen + n_depth; + cparams.n_batch = n_batch; + cparams.n_ubatch = n_ubatch; + cparams.type_k = type_k; + cparams.type_v = type_v; + cparams.offload_kqv = !no_kv_offload; + cparams.flash_attn_type = flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED : LLAMA_FLASH_ATTN_TYPE_DISABLED; + cparams.embeddings = embeddings; + cparams.op_offload = !no_op_offload; + cparams.swa_full = false; + + return cparams; + } +}; + +static std::vector get_cmd_params_instances(const cmd_params & params) { + std::vector instances; + + // this ordering minimizes the number of times that each model needs to be reloaded + // clang-format off + for (const auto & m : params.model) + for (const auto & fpt : params.fit_params_target) + for (const auto & fpc : params.fit_params_min_ctx) + for (const auto & nl : params.n_gpu_layers) + for (const auto & ncmoe : params.n_cpu_moe) + for (const auto & sm : params.split_mode) + for (const auto & ar : params.allreduce) + for (const auto & mg : params.main_gpu) + for (const auto & devs : params.devices) + for (const auto & ts : params.tensor_split) + for (const auto & ot : params.tensor_buft_overrides) + for (const auto & mmp : params.use_mmap) + for (const auto & dio : params.use_direct_io) + for (const auto & noh : params.no_host) + for (const auto & embd : params.embeddings) + for (const auto & nopo : params.no_op_offload) + for (const auto & nb : params.n_batch) + for (const auto & nub : params.n_ubatch) + for (const auto & tk : params.type_k) + for (const auto & tv : params.type_v) + for (const auto & nkvo : params.no_kv_offload) + for (const auto & fa : params.flash_attn) + for (const auto & nt : params.n_threads) + for (const auto & cm : params.cpu_mask) + for (const auto & cs : params.cpu_strict) + for (const auto & nd : params.n_depth) + for (const auto & pl : params.poll) { + for (const auto & n_prompt : params.n_prompt) { + if (n_prompt == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ n_prompt, + /* .n_gen = */ 0, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .allreduce = */ ar, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + + for (const auto & n_gen : params.n_gen) { + if (n_gen == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ 0, + /* .n_gen = */ n_gen, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .allreduce = */ ar, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + + for (const auto & n_pg : params.n_pg) { + if (n_pg.first == 0 && n_pg.second == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ n_pg.first, + /* .n_gen = */ n_pg.second, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .allreduce = */ ar, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + } + // clang-format on + + return instances; +} + +struct test { + static const std::string build_commit; + static const int build_number; + const std::string cpu_info; + const std::string gpu_info; + std::string model_filename; + std::string model_type; + uint64_t model_size; + uint64_t model_n_params; + int n_batch; + int n_ubatch; + int n_threads; + std::string cpu_mask; + bool cpu_strict; + int poll; + ggml_type type_k; + ggml_type type_v; + int n_gpu_layers; + int n_cpu_moe; + llama_split_mode split_mode; + std::string allreduce; + int main_gpu; + bool no_kv_offload; + bool flash_attn; + std::vector devices; + std::vector tensor_split; + std::vector tensor_buft_overrides; + bool use_mmap; + bool use_direct_io; + bool embeddings; + bool no_op_offload; + bool no_host; + size_t fit_target; + uint32_t fit_min_ctx; + int n_prompt; + int n_gen; + int n_depth; + std::string test_time; + std::vector samples_ns; + + test(const cmd_params_instance & inst, const llama_model * lmodel, const llama_context * ctx) : + cpu_info(get_cpu_info()), + gpu_info(get_gpu_info()) { + + model_filename = inst.model; + char buf[128]; + llama_model_desc(lmodel, buf, sizeof(buf)); + model_type = buf; + model_size = llama_model_size(lmodel); + model_n_params = llama_model_n_params(lmodel); + n_batch = inst.n_batch; + n_ubatch = inst.n_ubatch; + n_threads = inst.n_threads; + cpu_mask = inst.cpu_mask; + cpu_strict = inst.cpu_strict; + poll = inst.poll; + type_k = inst.type_k; + type_v = inst.type_v; + n_gpu_layers = inst.n_gpu_layers; + n_cpu_moe = inst.n_cpu_moe; + split_mode = inst.split_mode; + allreduce = inst.allreduce; + main_gpu = inst.main_gpu; + no_kv_offload = inst.no_kv_offload; + flash_attn = inst.flash_attn; + devices = inst.devices; + tensor_split = inst.tensor_split; + tensor_buft_overrides = inst.tensor_buft_overrides; + use_mmap = inst.use_mmap; + use_direct_io = inst.use_direct_io; + embeddings = inst.embeddings; + no_op_offload = inst.no_op_offload; + no_host = inst.no_host; + fit_target = inst.fit_target; + fit_min_ctx = inst.fit_min_ctx; + n_prompt = inst.n_prompt; + n_gen = inst.n_gen; + n_depth = inst.n_depth; + // RFC 3339 date-time format + time_t t = time(NULL); + std::strftime(buf, sizeof(buf), "%FT%TZ", gmtime(&t)); + test_time = buf; + + (void) ctx; + } + + uint64_t avg_ns() const { return ::avg(samples_ns); } + + uint64_t stdev_ns() const { return ::stdev(samples_ns); } + + std::vector get_ts() const { + int n_tokens = n_prompt + n_gen; + std::vector ts; + std::transform(samples_ns.begin(), samples_ns.end(), std::back_inserter(ts), + [n_tokens](uint64_t t) { return 1e9 * n_tokens / t; }); + return ts; + } + + double avg_ts() const { return ::avg(get_ts()); } + + double stdev_ts() const { return ::stdev(get_ts()); } + + static std::string get_backend() { + std::vector backends; + bool rpc_used = false; + for (size_t i = 0; i < ggml_backend_reg_count(); i++) { + auto * reg = ggml_backend_reg_get(i); + std::string name = ggml_backend_reg_name(reg); + if (string_starts_with(name, "RPC")) { + if (ggml_backend_reg_dev_count(reg) > 0) { + rpc_used = true; + } + } else { + if (name != "CPU") { + backends.push_back(ggml_backend_reg_name(reg)); + } + } + } + if (rpc_used) { + backends.push_back("RPC"); + } + return backends.empty() ? "CPU" : join(backends, ","); + } + + static const std::vector & get_fields() { + static const std::vector fields = { + "build_commit", "build_number", "cpu_info", "gpu_info", "backends", + "model_filename", "model_type", "model_size", "model_n_params", "n_batch", + "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", + "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", + "allreduce", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", + "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", + "no_op_offload", "no_host", "fit_target", "fit_min_ctx", + "n_prompt", "n_gen", "n_depth", + "test_time", "avg_ns", "stddev_ns", "avg_ts", "stddev_ts" + }; + return fields; + } + + enum field_type { STRING, BOOL, INT, FLOAT }; + + static field_type get_field_type(const std::string & field) { + if (field == "build_number" || field == "n_batch" || field == "n_ubatch" || field == "n_threads" || + field == "poll" || field == "model_size" || field == "model_n_params" || field == "n_gpu_layers" || + field == "main_gpu" || field == "n_prompt" || field == "n_gen" || field == "n_depth" || field == "avg_ns" || + field == "stddev_ns" || field == "no_op_offload" || field == "n_cpu_moe" || + field == "fit_target" || field == "fit_min_ctx") { + return INT; + } + if (field == "f16_kv" || field == "no_kv_offload" || field == "cpu_strict" || field == "flash_attn" || + field == "use_mmap" || field == "use_direct_io" || field == "embeddings" || field == "no_host") { + return BOOL; + } + if (field == "avg_ts" || field == "stddev_ts") { + return FLOAT; + } + return STRING; + } + + std::vector get_values() const { + std::string tensor_split_str; + std::string tensor_buft_overrides_str; + int max_nonzero = 0; + for (size_t i = 0; i < llama_max_devices(); i++) { + if (tensor_split[i] > 0) { + max_nonzero = i; + } + } + for (int i = 0; i <= max_nonzero; i++) { + char buf[32]; + snprintf(buf, sizeof(buf), "%.2f", tensor_split[i]); + tensor_split_str += buf; + if (i < max_nonzero) { + tensor_split_str += "/"; + } + } + if (tensor_buft_overrides.size() == 1) { + // Last element of tensor_buft_overrides is always a null pattern + // so if it is only one element long, it must be a null pattern. + GGML_ASSERT(tensor_buft_overrides[0].pattern == nullptr); + tensor_buft_overrides_str += "none"; + } else { + for (size_t i = 0; i < tensor_buft_overrides.size()-1; i++) { + // Last element of tensor_buft_overrides is always a null pattern + if (tensor_buft_overrides[i].pattern == nullptr) { + tensor_buft_overrides_str += "none"; + } else { + tensor_buft_overrides_str += tensor_buft_overrides[i].pattern; + tensor_buft_overrides_str += "="; + tensor_buft_overrides_str += ggml_backend_buft_name(tensor_buft_overrides[i].buft); + } + if (i + 2 < tensor_buft_overrides.size()) { + tensor_buft_overrides_str += ";"; + } + } + } + std::vector values = { build_commit, + std::to_string(build_number), + cpu_info, + gpu_info, + get_backend(), + model_filename, + model_type, + std::to_string(model_size), + std::to_string(model_n_params), + std::to_string(n_batch), + std::to_string(n_ubatch), + std::to_string(n_threads), + cpu_mask, + std::to_string(cpu_strict), + std::to_string(poll), + ggml_type_name(type_k), + ggml_type_name(type_v), + std::to_string(n_gpu_layers), + std::to_string(n_cpu_moe), + split_mode_str(split_mode), + allreduce, + std::to_string(main_gpu), + std::to_string(no_kv_offload), + std::to_string(flash_attn), + devices_to_string(devices), + tensor_split_str, + tensor_buft_overrides_str, + std::to_string(use_mmap), + std::to_string(use_direct_io), + std::to_string(embeddings), + std::to_string(no_op_offload), + std::to_string(no_host), + std::to_string(fit_target), + std::to_string(fit_min_ctx), + std::to_string(n_prompt), + std::to_string(n_gen), + std::to_string(n_depth), + test_time, + std::to_string(avg_ns()), + std::to_string(stdev_ns()), + std::to_string(avg_ts()), + std::to_string(stdev_ts()) }; + return values; + } + + std::map get_map() const { + std::map map; + auto fields = get_fields(); + auto values = get_values(); + std::transform(fields.begin(), fields.end(), values.begin(), std::inserter(map, map.end()), + std::make_pair); + return map; + } +}; + +const std::string test::build_commit = llama_commit(); +const int test::build_number = llama_build_number(); + +struct printer { + virtual ~printer() {} + + FILE * fout; + + virtual void print_header(const cmd_params & params) { (void) params; } + + virtual void print_test(const test & t) = 0; + + virtual void print_footer() {} +}; + +struct csv_printer : public printer { + static std::string escape_csv(const std::string & field) { + std::string escaped = "\""; + for (auto c : field) { + if (c == '"') { + escaped += "\""; + } + escaped += c; + } + escaped += "\""; + return escaped; + } + + void print_header(const cmd_params & params) override { + std::vector fields = test::get_fields(); + fprintf(fout, "%s\n", join(fields, ",").c_str()); + (void) params; + } + + void print_test(const test & t) override { + std::vector values = t.get_values(); + std::transform(values.begin(), values.end(), values.begin(), escape_csv); + fprintf(fout, "%s\n", join(values, ",").c_str()); + } +}; + +static std::string escape_json(const std::string & value) { + std::string escaped; + for (auto c : value) { + if (c == '"') { + escaped += "\\\""; + } else if (c == '\\') { + escaped += "\\\\"; + } else if (c <= 0x1f) { + char buf[8]; + snprintf(buf, sizeof(buf), "\\u%04x", c); + escaped += buf; + } else { + escaped += c; + } + } + return escaped; +} + +static std::string format_json_value(const std::string & field, const std::string & value) { + switch (test::get_field_type(field)) { + case test::STRING: + return "\"" + escape_json(value) + "\""; + case test::BOOL: + return value == "0" ? "false" : "true"; + default: + return value; + } +} + +struct json_printer : public printer { + bool first = true; + + void print_header(const cmd_params & params) override { + fprintf(fout, "[\n"); + (void) params; + } + + void print_fields(const std::vector & fields, const std::vector & values) { + assert(fields.size() == values.size()); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, " \"%s\": %s,\n", fields.at(i).c_str(), + format_json_value(fields.at(i), values.at(i)).c_str()); + } + } + + void print_test(const test & t) override { + if (first) { + first = false; + } else { + fprintf(fout, ",\n"); + } + fprintf(fout, " {\n"); + print_fields(test::get_fields(), t.get_values()); + fprintf(fout, " \"samples_ns\": [ %s ],\n", join(t.samples_ns, ", ").c_str()); + fprintf(fout, " \"samples_ts\": [ %s ]\n", join(t.get_ts(), ", ").c_str()); + fprintf(fout, " }"); + fflush(fout); + } + + void print_footer() override { fprintf(fout, "\n]\n"); } +}; + +struct jsonl_printer : public printer { + void print_fields(const std::vector & fields, const std::vector & values) { + assert(fields.size() == values.size()); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, "\"%s\": %s, ", fields.at(i).c_str(), format_json_value(fields.at(i), values.at(i)).c_str()); + } + } + + void print_test(const test & t) override { + fprintf(fout, "{"); + print_fields(test::get_fields(), t.get_values()); + fprintf(fout, "\"samples_ns\": [ %s ],", join(t.samples_ns, ", ").c_str()); + fprintf(fout, "\"samples_ts\": [ %s ]", join(t.get_ts(), ", ").c_str()); + fprintf(fout, "}\n"); + fflush(fout); + } +}; + +struct markdown_printer : public printer { + std::vector fields; + + static int get_field_width(const std::string & field) { + if (field == "model") { + return -30; + } + if (field == "t/s") { + return 20; + } + if (field == "size" || field == "params") { + return 10; + } + if (field == "n_gpu_layers") { + return 3; + } + if (field == "n_threads") { + return 7; + } + if (field == "n_batch") { + return 7; + } + if (field == "n_ubatch") { + return 8; + } + if (field == "type_k" || field == "type_v") { + return 6; + } + if (field == "split_mode") { + return 6; + } + if (field == "flash_attn") { + return 2; + } + if (field == "devices") { + return -12; + } + if (field == "use_mmap") { + return 4; + } + if (field == "use_direct_io") { + return 3; + } + if (field == "test") { + return 15; + } + if (field == "no_op_offload") { + return 4; + } + if (field == "no_host") { + return 4; + } + + int width = std::max((int) field.length(), 10); + + if (test::get_field_type(field) == test::STRING) { + return -width; + } + return width; + } + + static std::string get_field_display_name(const std::string & field) { + if (field == "n_gpu_layers") { + return "ngl"; + } + if (field == "split_mode") { + return "sm"; + } + if (field == "n_threads") { + return "threads"; + } + if (field == "no_kv_offload") { + return "nkvo"; + } + if (field == "flash_attn") { + return "fa"; + } + if (field == "use_mmap") { + return "mmap"; + } + if (field == "use_direct_io") { + return "dio"; + } + if (field == "embeddings") { + return "embd"; + } + if (field == "no_op_offload") { + return "nopo"; + } + if (field == "no_host") { + return "noh"; + } + if (field == "devices") { + return "dev"; + } + if (field == "tensor_split") { + return "ts"; + } + if (field == "tensor_buft_overrides") { + return "ot"; + } + if (field == "fit_target") { + return "fitt"; + } + if (field == "fit_min_ctx") { + return "fitc"; + } + return field; + } + + void print_header(const cmd_params & params) override { + // select fields to print + fields.emplace_back("model"); + fields.emplace_back("size"); + fields.emplace_back("params"); + fields.emplace_back("backend"); + bool is_cpu_backend = test::get_backend().find("CPU") != std::string::npos || + test::get_backend().find("BLAS") != std::string::npos || + test::get_backend().find("ZenDNN") != std::string::npos; + if (!is_cpu_backend) { + fields.emplace_back("n_gpu_layers"); + } + if (params.n_cpu_moe.size() > 1 || params.n_cpu_moe != cmd_params_defaults.n_cpu_moe) { + fields.emplace_back("n_cpu_moe"); + } + if (params.n_threads.size() > 1 || params.n_threads != cmd_params_defaults.n_threads || is_cpu_backend) { + fields.emplace_back("n_threads"); + } + if (params.cpu_mask.size() > 1 || params.cpu_mask != cmd_params_defaults.cpu_mask) { + fields.emplace_back("cpu_mask"); + } + if (params.cpu_strict.size() > 1 || params.cpu_strict != cmd_params_defaults.cpu_strict) { + fields.emplace_back("cpu_strict"); + } + if (params.poll.size() > 1 || params.poll != cmd_params_defaults.poll) { + fields.emplace_back("poll"); + } + if (params.n_batch.size() > 1 || params.n_batch != cmd_params_defaults.n_batch) { + fields.emplace_back("n_batch"); + } + if (params.n_ubatch.size() > 1 || params.n_ubatch != cmd_params_defaults.n_ubatch) { + fields.emplace_back("n_ubatch"); + } + if (params.type_k.size() > 1 || params.type_k != cmd_params_defaults.type_k) { + fields.emplace_back("type_k"); + } + if (params.type_v.size() > 1 || params.type_v != cmd_params_defaults.type_v) { + fields.emplace_back("type_v"); + } + if (params.main_gpu.size() > 1 || params.main_gpu != cmd_params_defaults.main_gpu) { + fields.emplace_back("main_gpu"); + } + if (params.split_mode.size() > 1 || params.split_mode != cmd_params_defaults.split_mode) { + fields.emplace_back("split_mode"); + } + if (params.no_kv_offload.size() > 1 || params.no_kv_offload != cmd_params_defaults.no_kv_offload) { + fields.emplace_back("no_kv_offload"); + } + if (params.flash_attn.size() > 1 || params.flash_attn != cmd_params_defaults.flash_attn) { + fields.emplace_back("flash_attn"); + } + if (params.devices.size() > 1 || params.devices != cmd_params_defaults.devices) { + fields.emplace_back("devices"); + } + if (params.tensor_split.size() > 1 || params.tensor_split != cmd_params_defaults.tensor_split) { + fields.emplace_back("tensor_split"); + } + if (params.tensor_buft_overrides.size() > 1 || !vec_vec_tensor_buft_override_equal(params.tensor_buft_overrides, cmd_params_defaults.tensor_buft_overrides)) { + fields.emplace_back("tensor_buft_overrides"); + } + if (params.use_mmap.size() > 1 || params.use_mmap != cmd_params_defaults.use_mmap) { + fields.emplace_back("use_mmap"); + } + if (params.use_direct_io.size() > 1 || params.use_direct_io != cmd_params_defaults.use_direct_io) { + fields.emplace_back("use_direct_io"); + } + if (params.embeddings.size() > 1 || params.embeddings != cmd_params_defaults.embeddings) { + fields.emplace_back("embeddings"); + } + if (params.no_op_offload.size() > 1 || params.no_op_offload != cmd_params_defaults.no_op_offload) { + fields.emplace_back("no_op_offload"); + } + if (params.no_host.size() > 1 || params.no_host != cmd_params_defaults.no_host) { + fields.emplace_back("no_host"); + } + if (params.fit_params_target.size() > 1 || params.fit_params_target != cmd_params_defaults.fit_params_target) { + fields.emplace_back("fit_target"); + } + if (params.fit_params_min_ctx.size() > 1 || params.fit_params_min_ctx != cmd_params_defaults.fit_params_min_ctx) { + fields.emplace_back("fit_min_ctx"); + } + fields.emplace_back("test"); + fields.emplace_back("t/s"); + + fprintf(fout, "|"); + for (const auto & field : fields) { + fprintf(fout, " %*s |", get_field_width(field), get_field_display_name(field).c_str()); + } + fprintf(fout, "\n"); + fprintf(fout, "|"); + for (const auto & field : fields) { + int width = get_field_width(field); + fprintf(fout, " %s%s |", std::string(std::abs(width) - 1, '-').c_str(), width > 0 ? ":" : "-"); + } + fprintf(fout, "\n"); + } + + void print_test(const test & t) override { + std::map vmap = t.get_map(); + + fprintf(fout, "|"); + for (const auto & field : fields) { + std::string value; + char buf[128]; + if (field == "model") { + value = t.model_type; + } else if (field == "size") { + if (t.model_size < 1024 * 1024 * 1024) { + snprintf(buf, sizeof(buf), "%.2f MiB", t.model_size / 1024.0 / 1024.0); + } else { + snprintf(buf, sizeof(buf), "%.2f GiB", t.model_size / 1024.0 / 1024.0 / 1024.0); + } + value = buf; + } else if (field == "params") { + if (t.model_n_params < 1000 * 1000 * 1000) { + snprintf(buf, sizeof(buf), "%.2f M", t.model_n_params / 1e6); + } else { + snprintf(buf, sizeof(buf), "%.2f B", t.model_n_params / 1e9); + } + value = buf; + } else if (field == "backend") { + value = test::get_backend(); + } else if (field == "test") { + if (t.n_prompt > 0 && t.n_gen == 0) { + snprintf(buf, sizeof(buf), "pp%d", t.n_prompt); + } else if (t.n_gen > 0 && t.n_prompt == 0) { + snprintf(buf, sizeof(buf), "tg%d", t.n_gen); + } else { + snprintf(buf, sizeof(buf), "pp%d+tg%d", t.n_prompt, t.n_gen); + } + if (t.n_depth > 0) { + int len = strlen(buf); + snprintf(buf + len, sizeof(buf) - len, " @ d%d", t.n_depth); + } + value = buf; + } else if (field == "t/s") { + snprintf(buf, sizeof(buf), "%.2f ± %.2f", t.avg_ts(), t.stdev_ts()); + value = buf; + } else if (vmap.find(field) != vmap.end()) { + value = vmap.at(field); + } else { + assert(false); + exit(1); + } + + int width = get_field_width(field); + if (field == "t/s") { + // HACK: the utf-8 character is 2 bytes + width += 1; + } + fprintf(fout, " %*s |", width, value.c_str()); + } + fprintf(fout, "\n"); + } + + void print_footer() override { + fprintf(fout, "\nbuild: %s (%d)\n", test::build_commit.c_str(), test::build_number); + } +}; + +struct sql_printer : public printer { + static std::string get_sql_field_type(const std::string & field) { + switch (test::get_field_type(field)) { + case test::STRING: + return "TEXT"; + case test::BOOL: + case test::INT: + return "INTEGER"; + case test::FLOAT: + return "REAL"; + default: + assert(false); + exit(1); + } + } + + void print_header(const cmd_params & params) override { + std::vector fields = test::get_fields(); + fprintf(fout, "CREATE TABLE IF NOT EXISTS llama_bench (\n"); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, " %s %s%s\n", fields.at(i).c_str(), get_sql_field_type(fields.at(i)).c_str(), + i < fields.size() - 1 ? "," : ""); + } + fprintf(fout, ");\n"); + fprintf(fout, "\n"); + (void) params; + } + + void print_test(const test & t) override { + fprintf(fout, "INSERT INTO llama_bench (%s) ", join(test::get_fields(), ", ").c_str()); + fprintf(fout, "VALUES ("); + std::vector values = t.get_values(); + for (size_t i = 0; i < values.size(); i++) { + fprintf(fout, "'%s'%s", values.at(i).c_str(), i < values.size() - 1 ? ", " : ""); + } + fprintf(fout, ");\n"); + } +}; + +struct ctx_state { + int depth = 0; // in tokens + + std::vector buf; // the llama_context state buffer +}; + +static bool test_prompt(llama_context * ctx, int n_prompt, int n_batch, int n_threads) { + llama_set_n_threads(ctx, n_threads, n_threads); + + const llama_model * model = llama_get_model(ctx); + const llama_vocab * vocab = llama_model_get_vocab(model); + const int32_t n_vocab = llama_vocab_n_tokens(vocab); + + std::vector tokens(n_batch); + + int n_processed = 0; + + while (n_processed < n_prompt) { + int n_tokens = std::min(n_prompt - n_processed, n_batch); + tokens[0] = n_processed == 0 && llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; + for (int i = 1; i < n_tokens; i++) { + tokens[i] = std::rand() % n_vocab; + } + int res = llama_decode(ctx, llama_batch_get_one(tokens.data(), n_tokens)); + if (res != 0) { + fprintf(stderr, "%s: failed to decode prompt batch, res = %d\n", __func__, res); + return false; + } + n_processed += n_tokens; + } + + llama_synchronize(ctx); + return true; +} + +static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { + llama_set_n_threads(ctx, n_threads, n_threads); + + const llama_model * model = llama_get_model(ctx); + const llama_vocab * vocab = llama_model_get_vocab(model); + const int32_t n_vocab = llama_vocab_n_tokens(vocab); + + llama_token token = llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; + + for (int i = 0; i < n_gen; i++) { + int res = llama_decode(ctx, llama_batch_get_one(&token, 1)); + if (res != 0) { + fprintf(stderr, "%s: failed to decode generation batch, res = %d\n", __func__, res); + return false; + } + llama_synchronize(ctx); + token = std::rand() % n_vocab; + } + return true; +} + +static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { + (void) level; + (void) text; + (void) user_data; +} + +static std::unique_ptr create_printer(output_formats format) { + switch (format) { + case NONE: + return nullptr; + case CSV: + return std::unique_ptr(new csv_printer()); + case JSON: + return std::unique_ptr(new json_printer()); + case JSONL: + return std::unique_ptr(new jsonl_printer()); + case MARKDOWN: + return std::unique_ptr(new markdown_printer()); + case SQL: + return std::unique_ptr(new sql_printer()); + } + GGML_ABORT("fatal error"); +} + +int main(int argc, char ** argv) { + std::setlocale(LC_NUMERIC, "C"); + // try to set locale for unicode characters in markdown + std::setlocale(LC_CTYPE, ".UTF-8"); + +#if !defined(NDEBUG) + fprintf(stderr, "warning: asserts enabled, performance may be affected\n"); +#endif + +#if (defined(_MSC_VER) && defined(_DEBUG)) || (!defined(_MSC_VER) && !defined(__OPTIMIZE__)) + fprintf(stderr, "warning: debug build, performance may be affected\n"); +#endif + +#if defined(__SANITIZE_ADDRESS__) || defined(__SANITIZE_THREAD__) + fprintf(stderr, "warning: sanitizer enabled, performance may be affected\n"); +#endif + + // initialize backends + ggml_backend_load_all(); + + cmd_params params = parse_cmd_params(argc, argv); + + auto * cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU); + if (!cpu_dev) { + fprintf(stderr, "%s: error: CPU backend is not loaded\n", __func__); + return 1; + } + auto * cpu_reg = ggml_backend_dev_backend_reg(cpu_dev); + auto * ggml_threadpool_new_fn = (decltype(ggml_threadpool_new) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_new"); + auto * ggml_threadpool_free_fn = (decltype(ggml_threadpool_free) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_free"); + + // initialize llama.cpp + if (!params.verbose) { + llama_log_set(llama_null_log_callback, NULL); + } + llama_backend_init(); + llama_numa_init(params.numa); + + if (!set_process_priority(params.prio)) { + fprintf(stderr, "%s: error: failed to set process priority\n", __func__); + return 1; + } + + // initialize printer + std::unique_ptr p = create_printer(params.output_format); + std::unique_ptr p_err = create_printer(params.output_format_stderr); + + if (p) { + p->fout = stdout; + p->print_header(params); + } + + if (p_err) { + p_err->fout = stderr; + p_err->print_header(params); + } + + std::vector params_instances = get_cmd_params_instances(params); + + llama_model * lmodel = nullptr; + const cmd_params_instance * prev_inst = nullptr; + + // store the llama_context state at the previous depth that we performed a test + // ref: https://github.com/ggml-org/llama.cpp/pull/16944#issuecomment-3478151721 + ctx_state cstate; + + int params_idx = 0; + auto params_count = params_instances.size(); + for (const auto & inst : params_instances) { + params_idx++; + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: starting\n", params_idx, params_count); + } + auto mparams = inst.to_llama_mparams(); + auto cparams = inst.to_llama_cparams(); + + bool do_fit = inst.fit_target != cmd_params_defaults.fit_params_target[0] || + inst.fit_min_ctx != cmd_params_defaults.fit_params_min_ctx[0]; + + std::vector fit_tensor_split(llama_max_devices(), 0.0f); + std::vector fit_overrides(llama_max_tensor_buft_overrides(), {nullptr, nullptr}); + + if (do_fit) { + // free the previous model so fit sees full free VRAM + if (lmodel) { + llama_model_free(lmodel); + lmodel = nullptr; + prev_inst = nullptr; + } + + // use default n_gpu_layers and n_ctx so common_fit_params can adjust them + mparams.n_gpu_layers = llama_model_default_params().n_gpu_layers; + mparams.tensor_split = fit_tensor_split.data(); + mparams.tensor_buft_overrides = fit_overrides.data(); + cparams.n_ctx = 0; + + std::vector margins(llama_max_devices(), inst.fit_target * 1024 * 1024); + + uint32_t n_ctx_needed = inst.n_prompt + inst.n_gen + inst.n_depth; + cparams.n_ctx = std::max(cparams.n_ctx, n_ctx_needed); + + common_fit_params(inst.model.c_str(), &mparams, &cparams, + fit_tensor_split.data(), + fit_overrides.data(), + margins.data(), + inst.fit_min_ctx, + params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); + } + + // set allreduce provider env var before model load (comm_init reads it) + { + const char * ar_val = (inst.allreduce == "auto") ? "" : inst.allreduce.c_str(); +#ifdef _WIN32 + _putenv_s("GGML_CUDA_ALLREDUCE", ar_val); +#else + setenv("GGML_CUDA_ALLREDUCE", ar_val, 1); +#endif + } + + // keep the same model between tests when possible + if (!lmodel || !prev_inst || !inst.equal_mparams(*prev_inst)) { + if (lmodel) { + llama_model_free(lmodel); + } + + lmodel = llama_model_load_from_file(inst.model.c_str(), mparams); + if (lmodel == NULL) { + fprintf(stderr, "%s: error: failed to load model '%s'\n", __func__, inst.model.c_str()); + return 1; + } + prev_inst = &inst; + } + + llama_context * ctx = llama_init_from_model(lmodel, cparams); + if (ctx == NULL) { + fprintf(stderr, "%s: error: failed to create context with model '%s'\n", __func__, inst.model.c_str()); + llama_model_free(lmodel); + return 1; + } + + test t(inst, lmodel, ctx); + + llama_memory_clear(llama_get_memory(ctx), false); + + // cool off before the test + if (params.delay) { + std::this_thread::sleep_for(std::chrono::seconds(params.delay)); + } + + struct ggml_threadpool_params tpp = ggml_threadpool_params_default(t.n_threads); + if (!parse_cpu_mask(t.cpu_mask, tpp.cpumask)) { + fprintf(stderr, "%s: failed to parse cpu-mask: %s\n", __func__, t.cpu_mask.c_str()); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + tpp.strict_cpu = t.cpu_strict; + tpp.poll = t.poll; + tpp.prio = params.prio; + + struct ggml_threadpool * threadpool = ggml_threadpool_new_fn(&tpp); + if (!threadpool) { + fprintf(stderr, "%s: threadpool create failed : n_threads %d\n", __func__, tpp.n_threads); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + + llama_attach_threadpool(ctx, threadpool, NULL); + + // warmup run + if (!params.no_warmup) { + if (t.n_prompt > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup prompt run\n", params_idx, params_count); + } + //test_prompt(ctx, std::min(t.n_batch, std::min(t.n_prompt, 32)), 0, t.n_batch, t.n_threads); + bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run prompt warmup\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + if (t.n_gen > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup generation run\n", params_idx, params_count); + } + bool res = test_gen(ctx, 1, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run gen warmup\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + } + + for (int i = 0; i < params.reps; i++) { + llama_memory_clear(llama_get_memory(ctx), false); + + if (t.n_depth > 0) { + bool is_cached = t.n_depth == cstate.depth; + + if (is_cached) { + // if previously we have computed at this depth, just restore the state + const size_t ret = llama_state_seq_set_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); + if (ret == 0) { + // if the old state is incompatible with the current context - reprocess from scratch + is_cached = false; + } + } + + if (!is_cached) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_prompt(ctx, t.n_depth, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run depth\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + + // store the context state for reuse in later runs + cstate.depth = t.n_depth; + cstate.buf.resize(llama_state_seq_get_size(ctx, 0)); + llama_state_seq_get_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); + } else { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d (cached)\n", params_idx, params_count, + i + 1, params.reps); + } + } + } + + uint64_t t_start = get_time_ns(); + + if (t.n_prompt > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: prompt run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run prompt\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + if (t.n_gen > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: generation run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_gen(ctx, t.n_gen, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run gen\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + + uint64_t t_ns = get_time_ns() - t_start; + t.samples_ns.push_back(t_ns); + } + + if (p) { + p->print_test(t); + fflush(p->fout); + } + + if (p_err) { + p_err->print_test(t); + fflush(p_err->fout); + } + + llama_perf_context_print(ctx); + + llama_free(ctx); + + ggml_threadpool_free_fn(threadpool); + } + + llama_model_free(lmodel); + + if (p) { + p->print_footer(); + } + + if (p_err) { + p_err->print_footer(); + } + + llama_backend_free(); + + return 0; +} From 8c0a7912823a3cd5386afa7327a1620878b95377 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 18:03:32 -0700 Subject: [PATCH 03/81] llama-bench: rename --allreduce to --reduction-provider / -rp Co-Authored-By: Claude Sonnet 4.6 via the shared field pattern, consistent with other multi-value flags). Useful for isolating hangs or regressions in tensor-parallel mode: pass --allreduce nccl to force NCCL and bypass the internal provider. Also fixes ggml_cuda_select_allreduce_provider() to treat an empty GGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when llama-bench sets it to "" for the "auto" case). Co-Authored-By: Claude Sonnet 4.6 xt gains ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- tools/llama-bench/llama-bench.cpp | 38 +++++++++++++++---------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index 65ad8199f6e..37763d5929e 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -335,7 +335,7 @@ struct cmd_params { std::vector n_gpu_layers; std::vector n_cpu_moe; std::vector split_mode; - std::vector allreduce; + std::vector reduction_provider; std::vector main_gpu; std::vector no_kv_offload; std::vector flash_attn; @@ -380,7 +380,7 @@ static const cmd_params cmd_params_defaults = { /* n_gpu_layers */ { 99 }, /* n_cpu_moe */ { 0 }, /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, - /* allreduce */ { "auto" }, + /* reduction_provider */ { "auto" }, /* main_gpu */ { 0 }, /* no_kv_offload */ { false }, /* flash_attn */ { false }, @@ -451,7 +451,7 @@ static void print_usage(int /* argc */, char ** argv) { printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); - printf(" --allreduce allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.allreduce, ",").c_str()); + printf(" -rp, --reduction-provider allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.reduction_provider, ",").c_str()); printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); @@ -762,7 +762,7 @@ static cmd_params parse_cmd_params(int argc, char ** argv) { break; } params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); - } else if (arg == "--allreduce") { + } else if (arg == "-rp" || arg == "--reduction-provider") { if (++i >= argc) { invalid_param = true; break; @@ -777,7 +777,7 @@ static cmd_params parse_cmd_params(int argc, char ** argv) { if (invalid_param) { break; } - params.allreduce.insert(params.allreduce.end(), p.begin(), p.end()); + params.reduction_provider.insert(params.reduction_provider.end(), p.begin(), p.end()); } else if (arg == "-mg" || arg == "--main-gpu") { if (++i >= argc) { invalid_param = true; @@ -1084,8 +1084,8 @@ static cmd_params parse_cmd_params(int argc, char ** argv) { if (params.split_mode.empty()) { params.split_mode = cmd_params_defaults.split_mode; } - if (params.allreduce.empty()) { - params.allreduce = cmd_params_defaults.allreduce; + if (params.reduction_provider.empty()) { + params.reduction_provider = cmd_params_defaults.reduction_provider; } if (params.main_gpu.empty()) { params.main_gpu = cmd_params_defaults.main_gpu; @@ -1158,7 +1158,7 @@ struct cmd_params_instance { int n_gpu_layers; int n_cpu_moe; llama_split_mode split_mode; - std::string allreduce; + std::string reduction_provider; int main_gpu; bool no_kv_offload; bool flash_attn; @@ -1228,7 +1228,7 @@ struct cmd_params_instance { bool equal_mparams(const cmd_params_instance & other) const { return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && - split_mode == other.split_mode && allreduce == other.allreduce && + split_mode == other.split_mode && reduction_provider == other.reduction_provider && main_gpu == other.main_gpu && tensor_split == other.tensor_split && use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && devices == other.devices && @@ -1265,7 +1265,7 @@ static std::vector get_cmd_params_instances(const cmd_param for (const auto & nl : params.n_gpu_layers) for (const auto & ncmoe : params.n_cpu_moe) for (const auto & sm : params.split_mode) - for (const auto & ar : params.allreduce) + for (const auto & rp : params.reduction_provider) for (const auto & mg : params.main_gpu) for (const auto & devs : params.devices) for (const auto & ts : params.tensor_split) @@ -1306,7 +1306,7 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .allreduce = */ ar, + /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1344,7 +1344,7 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .allreduce = */ ar, + /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1382,7 +1382,7 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .allreduce = */ ar, + /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1425,7 +1425,7 @@ struct test { int n_gpu_layers; int n_cpu_moe; llama_split_mode split_mode; - std::string allreduce; + std::string reduction_provider; int main_gpu; bool no_kv_offload; bool flash_attn; @@ -1466,7 +1466,7 @@ struct test { n_gpu_layers = inst.n_gpu_layers; n_cpu_moe = inst.n_cpu_moe; split_mode = inst.split_mode; - allreduce = inst.allreduce; + reduction_provider = inst.reduction_provider; main_gpu = inst.main_gpu; no_kv_offload = inst.no_kv_offload; flash_attn = inst.flash_attn; @@ -1535,7 +1535,7 @@ struct test { "model_filename", "model_type", "model_size", "model_n_params", "n_batch", "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", - "allreduce", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", + "reduction_provider", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", "no_op_offload", "no_host", "fit_target", "fit_min_ctx", "n_prompt", "n_gen", "n_depth", @@ -1621,7 +1621,7 @@ struct test { std::to_string(n_gpu_layers), std::to_string(n_cpu_moe), split_mode_str(split_mode), - allreduce, + reduction_provider, std::to_string(main_gpu), std::to_string(no_kv_offload), std::to_string(flash_attn), @@ -2275,9 +2275,9 @@ int main(int argc, char ** argv) { params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); } - // set allreduce provider env var before model load (comm_init reads it) + // set reduction provider env var before model load (comm_init reads it) { - const char * ar_val = (inst.allreduce == "auto") ? "" : inst.allreduce.c_str(); + const char * ar_val = (inst.reduction_provider == "auto") ? "" : inst.reduction_provider.c_str(); #ifdef _WIN32 _putenv_s("GGML_CUDA_ALLREDUCE", ar_val); #else From 10c475787856c130f01a6dd6839cf45bd7c2795c Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 18:19:47 -0700 Subject: [PATCH 04/81] llama-bench: pass WARN/ERROR log messages through in non-verbose mode The null log callback was silently dropping all messages. WARN and ERROR should always be visible since they indicate legitimate issues (e.g. a requested reduction provider not being available). Co-Authored-By: Claude Sonnet 4.6 vider. Also fixes ggml_cuda_select_allreduce_provider() to treat an empty GGML_CUDA_ALLREDUCE env var the same as unset (avoids spurious warning when llama-bench sets it to "" for the "auto" case). Co-Authored-By: Claude Sonnet 4.6 xt gains ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- tools/llama-bench/llama-bench.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index 37763d5929e..d57c564958c 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -2143,9 +2143,10 @@ static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { } static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { - (void) level; - (void) text; (void) user_data; + if (level >= GGML_LOG_LEVEL_WARN) { + fputs(text, stderr); + } } static std::unique_ptr create_printer(output_formats format) { From 2c1a1dbf9394903543460e655f0b1cb5ee918ddd Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 19:01:11 -0700 Subject: [PATCH 05/81] cmake: improve NCCL detection for source-tree builds, add static/dynamic switch FindNCCL.cmake now searches the cmake source-build layout used by the Windows NCCL port (cmake/lib/Release for static, cmake/src/Release for dynamic import lib) and also checks src/include for the generated nccl.h header. New option GGML_CUDA_NCCL_STATIC (default OFF) selects static vs dynamic linking and controls which paths and library names are searched. Co-Authored-By: Claude Sonnet 4.6 for the "auto" case). Co-Authored-By: Claude Sonnet 4.6 xt gains ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/CMakeLists.txt | 1009 +++++++++++++++-------------- ggml/cmake/FindNCCL.cmake | 112 +++- ggml/src/ggml-cuda/CMakeLists.txt | 539 +++++++-------- 3 files changed, 852 insertions(+), 808 deletions(-) diff --git a/ggml/CMakeLists.txt b/ggml/CMakeLists.txt index 2effd587b41..785ce7d64f5 100644 --- a/ggml/CMakeLists.txt +++ b/ggml/CMakeLists.txt @@ -1,504 +1,505 @@ -cmake_minimum_required(VERSION 3.14...3.28) # for add_link_options and implicit target directories. - -project("ggml" C CXX ASM) - -### GGML Version -set(GGML_VERSION_MAJOR 0) -set(GGML_VERSION_MINOR 10) -set(GGML_VERSION_PATCH 0) -set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}") - -list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/") - -find_program(GIT_EXE NAMES git git.exe NO_CMAKE_FIND_ROOT_PATH) -if(GIT_EXE) - # Get current git commit hash - execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD - WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} - OUTPUT_VARIABLE GGML_BUILD_COMMIT - OUTPUT_STRIP_TRAILING_WHITESPACE - ERROR_QUIET - ) - - # Check if the working directory is dirty (i.e., has uncommitted changes) - execute_process(COMMAND ${GIT_EXE} diff-index --quiet HEAD -- . - WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} - RESULT_VARIABLE GGML_GIT_DIRTY - ERROR_QUIET - ) -endif() - -set(GGML_VERSION "${GGML_VERSION_BASE}") - -if(NOT GGML_BUILD_COMMIT) - set(GGML_BUILD_COMMIT "unknown") -endif() - -# Build the commit string with optional dirty flag -if(DEFINED GGML_GIT_DIRTY AND GGML_GIT_DIRTY EQUAL 1) - set(GGML_BUILD_COMMIT "${GGML_BUILD_COMMIT}-dirty") -endif() - -include(CheckIncludeFileCXX) - -set(CMAKE_EXPORT_COMPILE_COMMANDS ON) - -if (NOT XCODE AND NOT MSVC AND NOT CMAKE_BUILD_TYPE) - set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE) - set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo") -endif() - -if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) - set(GGML_STANDALONE ON) - - set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) - - # configure project version - # TODO -else() - set(GGML_STANDALONE OFF) - - if (NOT CMAKE_RUNTIME_OUTPUT_DIRECTORY) - set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) - endif() -endif() - -if (EMSCRIPTEN) - set(BUILD_SHARED_LIBS_DEFAULT OFF) - - option(GGML_WASM_SINGLE_FILE "ggml: embed WASM inside the generated ggml.js" ON) -else() - if (MINGW) - set(BUILD_SHARED_LIBS_DEFAULT OFF) - else() - set(BUILD_SHARED_LIBS_DEFAULT ON) - endif() -endif() - -# remove the lib prefix on win32 mingw -if (WIN32) - set(CMAKE_STATIC_LIBRARY_PREFIX "") - set(CMAKE_SHARED_LIBRARY_PREFIX "") - set(CMAKE_SHARED_MODULE_PREFIX "") -endif() - -option(BUILD_SHARED_LIBS "ggml: build shared libraries" ${BUILD_SHARED_LIBS_DEFAULT}) -option(GGML_BACKEND_DL "ggml: build backends as dynamic libraries (requires BUILD_SHARED_LIBS)" OFF) -set(GGML_BACKEND_DIR "" CACHE PATH "ggml: directory to load dynamic backends from (requires GGML_BACKEND_DL") - -# -# option list -# - -# TODO: mark all options as advanced when not GGML_STANDALONE - -if (APPLE) - set(GGML_METAL_DEFAULT ON) - set(GGML_BLAS_DEFAULT ON) - set(GGML_BLAS_VENDOR_DEFAULT "Apple") -else() - set(GGML_METAL_DEFAULT OFF) - set(GGML_BLAS_DEFAULT OFF) - set(GGML_BLAS_VENDOR_DEFAULT "Generic") -endif() - -if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) - message(STATUS "Setting GGML_NATIVE_DEFAULT to OFF") - set(GGML_NATIVE_DEFAULT OFF) -else() - set(GGML_NATIVE_DEFAULT ON) -endif() - -# defaults -if (NOT GGML_LLAMAFILE_DEFAULT) - set(GGML_LLAMAFILE_DEFAULT OFF) -endif() - -if (NOT GGML_CUDA_GRAPHS_DEFAULT) - set(GGML_CUDA_GRAPHS_DEFAULT OFF) -endif() - -# general -option(GGML_STATIC "ggml: static link libraries" OFF) -option(GGML_NATIVE "ggml: optimize the build for the current system" ${GGML_NATIVE_DEFAULT}) -option(GGML_LTO "ggml: enable link time optimization" OFF) -option(GGML_CCACHE "ggml: use ccache if available" ON) - -# debug -option(GGML_ALL_WARNINGS "ggml: enable all compiler warnings" ON) -option(GGML_ALL_WARNINGS_3RD_PARTY "ggml: enable all compiler warnings in 3rd party libs" OFF) -option(GGML_GPROF "ggml: enable gprof" OFF) - -# build -option(GGML_FATAL_WARNINGS "ggml: enable -Werror flag" OFF) - -# sanitizers -option(GGML_SANITIZE_THREAD "ggml: enable thread sanitizer" OFF) -option(GGML_SANITIZE_ADDRESS "ggml: enable address sanitizer" OFF) -option(GGML_SANITIZE_UNDEFINED "ggml: enable undefined sanitizer" OFF) - -# instruction set specific -if (GGML_NATIVE OR NOT GGML_NATIVE_DEFAULT) - set(INS_ENB OFF) -else() - set(INS_ENB ON) -endif() - -message(DEBUG "GGML_NATIVE : ${GGML_NATIVE}") -message(DEBUG "GGML_NATIVE_DEFAULT : ${GGML_NATIVE_DEFAULT}") -message(DEBUG "INS_ENB : ${INS_ENB}") - -option(GGML_CPU_HBM "ggml: use memkind for CPU HBM" OFF) -option(GGML_CPU_REPACK "ggml: use runtime weight conversion of Q4_0 to Q4_X_X" ON) -option(GGML_CPU_KLEIDIAI "ggml: use KleidiAI optimized kernels if applicable" OFF) -option(GGML_SSE42 "ggml: enable SSE 4.2" ${INS_ENB}) -option(GGML_AVX "ggml: enable AVX" ${INS_ENB}) -option(GGML_AVX_VNNI "ggml: enable AVX-VNNI" OFF) -option(GGML_AVX2 "ggml: enable AVX2" ${INS_ENB}) -option(GGML_BMI2 "ggml: enable BMI2" ${INS_ENB}) -option(GGML_AVX512 "ggml: enable AVX512F" OFF) -option(GGML_AVX512_VBMI "ggml: enable AVX512-VBMI" OFF) -option(GGML_AVX512_VNNI "ggml: enable AVX512-VNNI" OFF) -option(GGML_AVX512_BF16 "ggml: enable AVX512-BF16" OFF) -if (NOT MSVC) - # in MSVC F16C and FMA is implied with AVX2/AVX512 - option(GGML_FMA "ggml: enable FMA" ${INS_ENB}) - option(GGML_F16C "ggml: enable F16C" ${INS_ENB}) - # MSVC does not seem to support AMX - option(GGML_AMX_TILE "ggml: enable AMX-TILE" OFF) - option(GGML_AMX_INT8 "ggml: enable AMX-INT8" OFF) - option(GGML_AMX_BF16 "ggml: enable AMX-BF16" OFF) -endif() -option(GGML_LASX "ggml: enable lasx" ON) -option(GGML_LSX "ggml: enable lsx" ON) -option(GGML_RVV "ggml: enable rvv" ON) -option(GGML_RV_ZFH "ggml: enable riscv zfh" ON) -option(GGML_RV_ZVFH "ggml: enable riscv zvfh" ON) -option(GGML_RV_ZICBOP "ggml: enable riscv zicbop" ON) -option(GGML_RV_ZIHINTPAUSE "ggml: enable riscv zihintpause" ON) -option(GGML_RV_ZVFBFWMA "ggml: enable riscv zvfbfwma" OFF) -option(GGML_XTHEADVECTOR "ggml: enable xtheadvector" OFF) -option(GGML_VXE "ggml: enable vxe" ${GGML_NATIVE}) - -option(GGML_CPU_ALL_VARIANTS "ggml: build all variants of the CPU backend (requires GGML_BACKEND_DL)" OFF) -set(GGML_CPU_ARM_ARCH "" CACHE STRING "ggml: CPU architecture for ARM") -set(GGML_CPU_POWERPC_CPUTYPE "" CACHE STRING "ggml: CPU type for PowerPC") - -# ggml core -set(GGML_SCHED_MAX_COPIES "4" CACHE STRING "ggml: max input copies for pipeline parallelism") -option(GGML_CPU "ggml: enable CPU backend" ON) -option(GGML_SCHED_NO_REALLOC "ggml: disallow reallocations in ggml-alloc (for debugging)" OFF) - -# 3rd party libs / backends -option(GGML_ACCELERATE "ggml: enable Accelerate framework" ON) -option(GGML_BLAS "ggml: use BLAS" ${GGML_BLAS_DEFAULT}) -set(GGML_BLAS_VENDOR ${GGML_BLAS_VENDOR_DEFAULT} CACHE STRING - "ggml: BLAS library vendor") -option(GGML_LLAMAFILE "ggml: use LLAMAFILE" ${GGML_LLAMAFILE_DEFAULT}) - -option(GGML_CUDA "ggml: use CUDA" OFF) -option(GGML_MUSA "ggml: use MUSA" OFF) -option(GGML_CUDA_FORCE_MMQ "ggml: use mmq kernels instead of cuBLAS" OFF) -option(GGML_CUDA_FORCE_CUBLAS "ggml: always use cuBLAS instead of mmq kernels" OFF) -set (GGML_CUDA_PEER_MAX_BATCH_SIZE "128" CACHE STRING - "ggml: max. batch size for using peer access") -option(GGML_CUDA_NO_PEER_COPY "ggml: do not use peer to peer copies" OFF) -option(GGML_CUDA_NO_VMM "ggml: do not try to use CUDA VMM" OFF) -option(GGML_CUDA_FA "ggml: compile ggml FlashAttention CUDA kernels" ON) -option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashAttention" OFF) -option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) -option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) -set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING - "ggml: cuda link binary compression mode; requires cuda 12.8+") -set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") - -option(GGML_HIP "ggml: use HIP" OFF) -option(GGML_HIP_GRAPHS "ggml: use HIP graph, experimental, slow" OFF) -option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF) -option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON) -option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF) -option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON) -option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF) -option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF) -option(GGML_MUSA_MUDNN_COPY "ggml: enable muDNN for accelerated copy" OFF) -option(GGML_VULKAN "ggml: use Vulkan" OFF) -option(GGML_VULKAN_CHECK_RESULTS "ggml: run Vulkan op checks" OFF) -option(GGML_VULKAN_DEBUG "ggml: enable Vulkan debug output" OFF) -option(GGML_VULKAN_MEMORY_DEBUG "ggml: enable Vulkan memory debug output" OFF) -option(GGML_VULKAN_SHADER_DEBUG_INFO "ggml: enable Vulkan shader debug info" OFF) -option(GGML_VULKAN_VALIDATE "ggml: enable Vulkan validation" OFF) -option(GGML_VULKAN_RUN_TESTS "ggml: run Vulkan tests" OFF) -option(GGML_WEBGPU "ggml: use WebGPU" OFF) -option(GGML_WEBGPU_DEBUG "ggml: enable WebGPU debug output" OFF) -option(GGML_WEBGPU_CPU_PROFILE "ggml: enable WebGPU profiling (CPU)" OFF) -option(GGML_WEBGPU_GPU_PROFILE "ggml: enable WebGPU profiling (GPU)" OFF) -option(GGML_WEBGPU_JSPI "ggml: use JSPI for WebGPU" ON) -option(GGML_ZDNN "ggml: use zDNN" OFF) -option(GGML_VIRTGPU "ggml: use the VirtGPU/Virglrenderer API Remoting frontend" OFF) -option(GGML_VIRTGPU_BACKEND "ggml: build the VirtGPU/Virglrenderer API Remoting backend" OFF) -option(GGML_METAL "ggml: use Metal" ${GGML_METAL_DEFAULT}) -option(GGML_METAL_NDEBUG "ggml: disable Metal debugging" OFF) -option(GGML_METAL_SHADER_DEBUG "ggml: compile Metal with -fno-fast-math" OFF) -option(GGML_METAL_EMBED_LIBRARY "ggml: embed Metal library" ${GGML_METAL}) -set (GGML_METAL_MACOSX_VERSION_MIN "" CACHE STRING - "ggml: metal minimum macOS version") -set (GGML_METAL_STD "" CACHE STRING "ggml: metal standard version (-std flag)") -option(GGML_OPENMP "ggml: use OpenMP" ON) -option(GGML_RPC "ggml: use RPC" OFF) -option(GGML_SYCL "ggml: use SYCL" OFF) -option(GGML_SYCL_F16 "ggml: use 16 bit floats for sycl calculations" OFF) -option(GGML_SYCL_GRAPH "ggml: enable graphs in the SYCL backend" ON) -option(GGML_SYCL_HOST_MEM_FALLBACK "ggml: allow host memory fallback in SYCL reorder (requires kernel 6.8+)" ON) -option(GGML_SYCL_DNN "ggml: enable oneDNN in the SYCL backend" ON) -set (GGML_SYCL_TARGET "INTEL" CACHE STRING - "ggml: sycl target device") -set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING - "ggml: sycl device architecture") - -option(GGML_OPENVINO "ggml: use OPENVINO" OFF) - -option(GGML_OPENCL "ggml: use OpenCL" OFF) -option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF) -option(GGML_OPENCL_EMBED_KERNELS "ggml: embed kernels" ON) -option(GGML_OPENCL_USE_ADRENO_KERNELS "ggml: use optimized kernels for Adreno" ON) -set (GGML_OPENCL_TARGET_VERSION "300" CACHE STRING - "ggml: OpenCL API version to target") - -option(GGML_HEXAGON "ggml: enable Hexagon backend" OFF) -set(GGML_HEXAGON_FP32_QUANTIZE_GROUP_SIZE 128 CACHE STRING "ggml: quantize group size (32, 64, or 128)") - -# toolchain for vulkan-shaders-gen -set (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN "" CACHE FILEPATH "ggml: toolchain file for vulkan-shaders-gen") - -option(GGML_ZENDNN "ggml: use ZenDNN" OFF) -option(ZENDNN_ROOT "ggml: path to ZenDNN installation" "") - -# extra artifacts -option(GGML_BUILD_TESTS "ggml: build tests" ${GGML_STANDALONE}) -option(GGML_BUILD_EXAMPLES "ggml: build examples" ${GGML_STANDALONE}) - -# -# dependencies -# - -set(CMAKE_C_STANDARD 11) -set(CMAKE_C_STANDARD_REQUIRED true) - -set(CMAKE_CXX_STANDARD 17) -set(CMAKE_CXX_STANDARD_REQUIRED true) - -set(THREADS_PREFER_PTHREAD_FLAG ON) - -find_package(Threads REQUIRED) - -include(GNUInstallDirs) - -# -# build the library -# - -add_subdirectory(src) - -# -# tests and examples -# - -if (GGML_BUILD_TESTS) - enable_testing() - add_subdirectory(tests) -endif () - -if (GGML_BUILD_EXAMPLES) - add_subdirectory(examples) -endif () - -# -# install -# - -include(CMakePackageConfigHelpers) - -# all public headers -set(GGML_PUBLIC_HEADERS - include/ggml.h - include/ggml-cpu.h - include/ggml-alloc.h - include/ggml-backend.h - include/ggml-blas.h - include/ggml-cann.h - include/ggml-cpp.h - include/ggml-cuda.h - include/ggml-opt.h - include/ggml-metal.h - include/ggml-rpc.h - include/ggml-virtgpu.h - include/ggml-sycl.h - include/ggml-vulkan.h - include/ggml-webgpu.h - include/ggml-zendnn.h - include/ggml-openvino.h - include/gguf.h) - -set_target_properties(ggml PROPERTIES PUBLIC_HEADER "${GGML_PUBLIC_HEADERS}") -#if (GGML_METAL) -# set_target_properties(ggml PROPERTIES RESOURCE "${CMAKE_CURRENT_SOURCE_DIR}/src/ggml-metal.metal") -#endif() -install(TARGETS ggml LIBRARY PUBLIC_HEADER) -install(TARGETS ggml-base LIBRARY) - -if (GGML_STANDALONE) - configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ggml.pc.in - ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc - @ONLY) - - install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc - DESTINATION share/pkgconfig) -endif() - -# -# Create CMake package -# - - - -# Capture variables prefixed with GGML_. - -set(variable_set_statements -" -####### Expanded from @GGML_VARIABLES_EXPANED@ by configure_package_config_file() ####### -####### Any changes to this file will be overwritten by the next CMake run ####### - -") - -set(GGML_SHARED_LIB ${BUILD_SHARED_LIBS}) - -get_cmake_property(all_variables VARIABLES) -foreach(variable_name IN LISTS all_variables) - if(variable_name MATCHES "^GGML_") - string(REPLACE ";" "\\;" - variable_value "${${variable_name}}") - - set(variable_set_statements - "${variable_set_statements}set(${variable_name} \"${variable_value}\")\n") - endif() -endforeach() - -set(GGML_VARIABLES_EXPANDED ${variable_set_statements}) - -# Create the CMake package and set install location. - -set(GGML_INSTALL_VERSION ${GGML_VERSION}) -set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header files") -set(GGML_LIB_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR} CACHE PATH "Location of library files") -set(GGML_BIN_INSTALL_DIR ${CMAKE_INSTALL_BINDIR} CACHE PATH "Location of binary files") - -configure_package_config_file( - ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in - ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake - INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml - PATH_VARS GGML_INCLUDE_INSTALL_DIR - GGML_LIB_INSTALL_DIR - GGML_BIN_INSTALL_DIR) - -write_basic_package_version_file( - ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake - VERSION ${GGML_INSTALL_VERSION} - COMPATIBILITY SameMajorVersion) - -target_compile_definitions(ggml-base PRIVATE - GGML_VERSION="${GGML_INSTALL_VERSION}" - GGML_COMMIT="${GGML_BUILD_COMMIT}" -) -message(STATUS "ggml version: ${GGML_INSTALL_VERSION}") -message(STATUS "ggml commit: ${GGML_BUILD_COMMIT}") - -install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake - ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake - DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml) - -if (MSVC) - set(MSVC_WARNING_FLAGS - /wd4005 # Macro redefinition - /wd4244 # Conversion from one type to another type, possible loss of data - /wd4267 # Conversion from 'size_t' to a smaller type, possible loss of data - /wd4305 # Conversion from 'type1' to 'type2', possible loss of data - /wd4566 # Conversion from 'char' to 'wchar_t', possible loss of data - /wd4996 # Disable POSIX deprecation warnings - /wd4702 # Unreachable code warnings - ) - set(MSVC_COMPILE_OPTIONS - "$<$:/utf-8>" - "$<$:/utf-8>" - ) - function(configure_msvc_target target_name) - if(TARGET ${target_name}) - target_compile_options(${target_name} PRIVATE ${MSVC_WARNING_FLAGS}) - target_compile_options(${target_name} PRIVATE ${MSVC_COMPILE_OPTIONS}) - endif() - endfunction() - - configure_msvc_target(ggml-base) - configure_msvc_target(ggml) - configure_msvc_target(ggml-cpu) - configure_msvc_target(ggml-cpu-x64) - configure_msvc_target(ggml-cpu-sse42) - configure_msvc_target(ggml-cpu-sandybridge) - # __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512 - # skipping ggml-cpu-ivybridge - # skipping ggml-cpu-piledriver - configure_msvc_target(ggml-cpu-haswell) - configure_msvc_target(ggml-cpu-skylakex) - configure_msvc_target(ggml-cpu-cannonlake) - configure_msvc_target(ggml-cpu-cascadelake) - configure_msvc_target(ggml-cpu-icelake) - # MSVC 2022 doesn't support BF16 intrinsics without `/arch:AVX10.1` ?! - # https://learn.microsoft.com/en-us/cpp/intrinsics/x64-amd64-intrinsics-list?view=msvc-170 - # https://learn.microsoft.com/en-us/cpp/build/reference/arch-x64?view=msvc-170 - # skipping ggml-cpu-cooperlake - # skipping ggml-cpu-zen4 - configure_msvc_target(ggml-cpu-alderlake) - # MSVC doesn't support AMX - # skipping ggml-cpu-sapphirerapids - - if (GGML_BUILD_EXAMPLES) - configure_msvc_target(common-ggml) - configure_msvc_target(common) - - configure_msvc_target(mnist-common) - configure_msvc_target(mnist-eval) - configure_msvc_target(mnist-train) - - configure_msvc_target(gpt-2-ctx) - configure_msvc_target(gpt-2-alloc) - configure_msvc_target(gpt-2-backend) - configure_msvc_target(gpt-2-sched) - configure_msvc_target(gpt-2-quantize) - configure_msvc_target(gpt-2-batched) - - configure_msvc_target(gpt-j) - configure_msvc_target(gpt-j-quantize) - - configure_msvc_target(magika) - configure_msvc_target(yolov3-tiny) - configure_msvc_target(sam) - - configure_msvc_target(simple-ctx) - configure_msvc_target(simple-backend) - endif() - - if (GGML_BUILD_TESTS) - configure_msvc_target(test-mul-mat) - configure_msvc_target(test-arange) - configure_msvc_target(test-backend-ops) - configure_msvc_target(test-cont) - configure_msvc_target(test-conv-transpose) - configure_msvc_target(test-conv-transpose-1d) - configure_msvc_target(test-conv1d) - configure_msvc_target(test-conv2d) - configure_msvc_target(test-conv2d-dw) - configure_msvc_target(test-customop) - configure_msvc_target(test-dup) - configure_msvc_target(test-opt) - configure_msvc_target(test-pool) - endif () -endif() +cmake_minimum_required(VERSION 3.14...3.28) # for add_link_options and implicit target directories. + +project("ggml" C CXX ASM) + +### GGML Version +set(GGML_VERSION_MAJOR 0) +set(GGML_VERSION_MINOR 10) +set(GGML_VERSION_PATCH 0) +set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}") + +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/") + +find_program(GIT_EXE NAMES git git.exe NO_CMAKE_FIND_ROOT_PATH) +if(GIT_EXE) + # Get current git commit hash + execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + OUTPUT_VARIABLE GGML_BUILD_COMMIT + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET + ) + + # Check if the working directory is dirty (i.e., has uncommitted changes) + execute_process(COMMAND ${GIT_EXE} diff-index --quiet HEAD -- . + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + RESULT_VARIABLE GGML_GIT_DIRTY + ERROR_QUIET + ) +endif() + +set(GGML_VERSION "${GGML_VERSION_BASE}") + +if(NOT GGML_BUILD_COMMIT) + set(GGML_BUILD_COMMIT "unknown") +endif() + +# Build the commit string with optional dirty flag +if(DEFINED GGML_GIT_DIRTY AND GGML_GIT_DIRTY EQUAL 1) + set(GGML_BUILD_COMMIT "${GGML_BUILD_COMMIT}-dirty") +endif() + +include(CheckIncludeFileCXX) + +set(CMAKE_EXPORT_COMPILE_COMMANDS ON) + +if (NOT XCODE AND NOT MSVC AND NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE) + set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo") +endif() + +if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) + set(GGML_STANDALONE ON) + + set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) + + # configure project version + # TODO +else() + set(GGML_STANDALONE OFF) + + if (NOT CMAKE_RUNTIME_OUTPUT_DIRECTORY) + set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) + endif() +endif() + +if (EMSCRIPTEN) + set(BUILD_SHARED_LIBS_DEFAULT OFF) + + option(GGML_WASM_SINGLE_FILE "ggml: embed WASM inside the generated ggml.js" ON) +else() + if (MINGW) + set(BUILD_SHARED_LIBS_DEFAULT OFF) + else() + set(BUILD_SHARED_LIBS_DEFAULT ON) + endif() +endif() + +# remove the lib prefix on win32 mingw +if (WIN32) + set(CMAKE_STATIC_LIBRARY_PREFIX "") + set(CMAKE_SHARED_LIBRARY_PREFIX "") + set(CMAKE_SHARED_MODULE_PREFIX "") +endif() + +option(BUILD_SHARED_LIBS "ggml: build shared libraries" ${BUILD_SHARED_LIBS_DEFAULT}) +option(GGML_BACKEND_DL "ggml: build backends as dynamic libraries (requires BUILD_SHARED_LIBS)" OFF) +set(GGML_BACKEND_DIR "" CACHE PATH "ggml: directory to load dynamic backends from (requires GGML_BACKEND_DL") + +# +# option list +# + +# TODO: mark all options as advanced when not GGML_STANDALONE + +if (APPLE) + set(GGML_METAL_DEFAULT ON) + set(GGML_BLAS_DEFAULT ON) + set(GGML_BLAS_VENDOR_DEFAULT "Apple") +else() + set(GGML_METAL_DEFAULT OFF) + set(GGML_BLAS_DEFAULT OFF) + set(GGML_BLAS_VENDOR_DEFAULT "Generic") +endif() + +if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) + message(STATUS "Setting GGML_NATIVE_DEFAULT to OFF") + set(GGML_NATIVE_DEFAULT OFF) +else() + set(GGML_NATIVE_DEFAULT ON) +endif() + +# defaults +if (NOT GGML_LLAMAFILE_DEFAULT) + set(GGML_LLAMAFILE_DEFAULT OFF) +endif() + +if (NOT GGML_CUDA_GRAPHS_DEFAULT) + set(GGML_CUDA_GRAPHS_DEFAULT OFF) +endif() + +# general +option(GGML_STATIC "ggml: static link libraries" OFF) +option(GGML_NATIVE "ggml: optimize the build for the current system" ${GGML_NATIVE_DEFAULT}) +option(GGML_LTO "ggml: enable link time optimization" OFF) +option(GGML_CCACHE "ggml: use ccache if available" ON) + +# debug +option(GGML_ALL_WARNINGS "ggml: enable all compiler warnings" ON) +option(GGML_ALL_WARNINGS_3RD_PARTY "ggml: enable all compiler warnings in 3rd party libs" OFF) +option(GGML_GPROF "ggml: enable gprof" OFF) + +# build +option(GGML_FATAL_WARNINGS "ggml: enable -Werror flag" OFF) + +# sanitizers +option(GGML_SANITIZE_THREAD "ggml: enable thread sanitizer" OFF) +option(GGML_SANITIZE_ADDRESS "ggml: enable address sanitizer" OFF) +option(GGML_SANITIZE_UNDEFINED "ggml: enable undefined sanitizer" OFF) + +# instruction set specific +if (GGML_NATIVE OR NOT GGML_NATIVE_DEFAULT) + set(INS_ENB OFF) +else() + set(INS_ENB ON) +endif() + +message(DEBUG "GGML_NATIVE : ${GGML_NATIVE}") +message(DEBUG "GGML_NATIVE_DEFAULT : ${GGML_NATIVE_DEFAULT}") +message(DEBUG "INS_ENB : ${INS_ENB}") + +option(GGML_CPU_HBM "ggml: use memkind for CPU HBM" OFF) +option(GGML_CPU_REPACK "ggml: use runtime weight conversion of Q4_0 to Q4_X_X" ON) +option(GGML_CPU_KLEIDIAI "ggml: use KleidiAI optimized kernels if applicable" OFF) +option(GGML_SSE42 "ggml: enable SSE 4.2" ${INS_ENB}) +option(GGML_AVX "ggml: enable AVX" ${INS_ENB}) +option(GGML_AVX_VNNI "ggml: enable AVX-VNNI" OFF) +option(GGML_AVX2 "ggml: enable AVX2" ${INS_ENB}) +option(GGML_BMI2 "ggml: enable BMI2" ${INS_ENB}) +option(GGML_AVX512 "ggml: enable AVX512F" OFF) +option(GGML_AVX512_VBMI "ggml: enable AVX512-VBMI" OFF) +option(GGML_AVX512_VNNI "ggml: enable AVX512-VNNI" OFF) +option(GGML_AVX512_BF16 "ggml: enable AVX512-BF16" OFF) +if (NOT MSVC) + # in MSVC F16C and FMA is implied with AVX2/AVX512 + option(GGML_FMA "ggml: enable FMA" ${INS_ENB}) + option(GGML_F16C "ggml: enable F16C" ${INS_ENB}) + # MSVC does not seem to support AMX + option(GGML_AMX_TILE "ggml: enable AMX-TILE" OFF) + option(GGML_AMX_INT8 "ggml: enable AMX-INT8" OFF) + option(GGML_AMX_BF16 "ggml: enable AMX-BF16" OFF) +endif() +option(GGML_LASX "ggml: enable lasx" ON) +option(GGML_LSX "ggml: enable lsx" ON) +option(GGML_RVV "ggml: enable rvv" ON) +option(GGML_RV_ZFH "ggml: enable riscv zfh" ON) +option(GGML_RV_ZVFH "ggml: enable riscv zvfh" ON) +option(GGML_RV_ZICBOP "ggml: enable riscv zicbop" ON) +option(GGML_RV_ZIHINTPAUSE "ggml: enable riscv zihintpause" ON) +option(GGML_RV_ZVFBFWMA "ggml: enable riscv zvfbfwma" OFF) +option(GGML_XTHEADVECTOR "ggml: enable xtheadvector" OFF) +option(GGML_VXE "ggml: enable vxe" ${GGML_NATIVE}) + +option(GGML_CPU_ALL_VARIANTS "ggml: build all variants of the CPU backend (requires GGML_BACKEND_DL)" OFF) +set(GGML_CPU_ARM_ARCH "" CACHE STRING "ggml: CPU architecture for ARM") +set(GGML_CPU_POWERPC_CPUTYPE "" CACHE STRING "ggml: CPU type for PowerPC") + +# ggml core +set(GGML_SCHED_MAX_COPIES "4" CACHE STRING "ggml: max input copies for pipeline parallelism") +option(GGML_CPU "ggml: enable CPU backend" ON) +option(GGML_SCHED_NO_REALLOC "ggml: disallow reallocations in ggml-alloc (for debugging)" OFF) + +# 3rd party libs / backends +option(GGML_ACCELERATE "ggml: enable Accelerate framework" ON) +option(GGML_BLAS "ggml: use BLAS" ${GGML_BLAS_DEFAULT}) +set(GGML_BLAS_VENDOR ${GGML_BLAS_VENDOR_DEFAULT} CACHE STRING + "ggml: BLAS library vendor") +option(GGML_LLAMAFILE "ggml: use LLAMAFILE" ${GGML_LLAMAFILE_DEFAULT}) + +option(GGML_CUDA "ggml: use CUDA" OFF) +option(GGML_MUSA "ggml: use MUSA" OFF) +option(GGML_CUDA_FORCE_MMQ "ggml: use mmq kernels instead of cuBLAS" OFF) +option(GGML_CUDA_FORCE_CUBLAS "ggml: always use cuBLAS instead of mmq kernels" OFF) +set (GGML_CUDA_PEER_MAX_BATCH_SIZE "128" CACHE STRING + "ggml: max. batch size for using peer access") +option(GGML_CUDA_NO_PEER_COPY "ggml: do not use peer to peer copies" OFF) +option(GGML_CUDA_NO_VMM "ggml: do not try to use CUDA VMM" OFF) +option(GGML_CUDA_FA "ggml: compile ggml FlashAttention CUDA kernels" ON) +option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashAttention" OFF) +option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) +option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) +option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) +set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING + "ggml: cuda link binary compression mode; requires cuda 12.8+") +set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") + +option(GGML_HIP "ggml: use HIP" OFF) +option(GGML_HIP_GRAPHS "ggml: use HIP graph, experimental, slow" OFF) +option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF) +option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON) +option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF) +option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON) +option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF) +option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF) +option(GGML_MUSA_MUDNN_COPY "ggml: enable muDNN for accelerated copy" OFF) +option(GGML_VULKAN "ggml: use Vulkan" OFF) +option(GGML_VULKAN_CHECK_RESULTS "ggml: run Vulkan op checks" OFF) +option(GGML_VULKAN_DEBUG "ggml: enable Vulkan debug output" OFF) +option(GGML_VULKAN_MEMORY_DEBUG "ggml: enable Vulkan memory debug output" OFF) +option(GGML_VULKAN_SHADER_DEBUG_INFO "ggml: enable Vulkan shader debug info" OFF) +option(GGML_VULKAN_VALIDATE "ggml: enable Vulkan validation" OFF) +option(GGML_VULKAN_RUN_TESTS "ggml: run Vulkan tests" OFF) +option(GGML_WEBGPU "ggml: use WebGPU" OFF) +option(GGML_WEBGPU_DEBUG "ggml: enable WebGPU debug output" OFF) +option(GGML_WEBGPU_CPU_PROFILE "ggml: enable WebGPU profiling (CPU)" OFF) +option(GGML_WEBGPU_GPU_PROFILE "ggml: enable WebGPU profiling (GPU)" OFF) +option(GGML_WEBGPU_JSPI "ggml: use JSPI for WebGPU" ON) +option(GGML_ZDNN "ggml: use zDNN" OFF) +option(GGML_VIRTGPU "ggml: use the VirtGPU/Virglrenderer API Remoting frontend" OFF) +option(GGML_VIRTGPU_BACKEND "ggml: build the VirtGPU/Virglrenderer API Remoting backend" OFF) +option(GGML_METAL "ggml: use Metal" ${GGML_METAL_DEFAULT}) +option(GGML_METAL_NDEBUG "ggml: disable Metal debugging" OFF) +option(GGML_METAL_SHADER_DEBUG "ggml: compile Metal with -fno-fast-math" OFF) +option(GGML_METAL_EMBED_LIBRARY "ggml: embed Metal library" ${GGML_METAL}) +set (GGML_METAL_MACOSX_VERSION_MIN "" CACHE STRING + "ggml: metal minimum macOS version") +set (GGML_METAL_STD "" CACHE STRING "ggml: metal standard version (-std flag)") +option(GGML_OPENMP "ggml: use OpenMP" ON) +option(GGML_RPC "ggml: use RPC" OFF) +option(GGML_SYCL "ggml: use SYCL" OFF) +option(GGML_SYCL_F16 "ggml: use 16 bit floats for sycl calculations" OFF) +option(GGML_SYCL_GRAPH "ggml: enable graphs in the SYCL backend" ON) +option(GGML_SYCL_HOST_MEM_FALLBACK "ggml: allow host memory fallback in SYCL reorder (requires kernel 6.8+)" ON) +option(GGML_SYCL_DNN "ggml: enable oneDNN in the SYCL backend" ON) +set (GGML_SYCL_TARGET "INTEL" CACHE STRING + "ggml: sycl target device") +set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING + "ggml: sycl device architecture") + +option(GGML_OPENVINO "ggml: use OPENVINO" OFF) + +option(GGML_OPENCL "ggml: use OpenCL" OFF) +option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF) +option(GGML_OPENCL_EMBED_KERNELS "ggml: embed kernels" ON) +option(GGML_OPENCL_USE_ADRENO_KERNELS "ggml: use optimized kernels for Adreno" ON) +set (GGML_OPENCL_TARGET_VERSION "300" CACHE STRING + "ggml: OpenCL API version to target") + +option(GGML_HEXAGON "ggml: enable Hexagon backend" OFF) +set(GGML_HEXAGON_FP32_QUANTIZE_GROUP_SIZE 128 CACHE STRING "ggml: quantize group size (32, 64, or 128)") + +# toolchain for vulkan-shaders-gen +set (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN "" CACHE FILEPATH "ggml: toolchain file for vulkan-shaders-gen") + +option(GGML_ZENDNN "ggml: use ZenDNN" OFF) +option(ZENDNN_ROOT "ggml: path to ZenDNN installation" "") + +# extra artifacts +option(GGML_BUILD_TESTS "ggml: build tests" ${GGML_STANDALONE}) +option(GGML_BUILD_EXAMPLES "ggml: build examples" ${GGML_STANDALONE}) + +# +# dependencies +# + +set(CMAKE_C_STANDARD 11) +set(CMAKE_C_STANDARD_REQUIRED true) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED true) + +set(THREADS_PREFER_PTHREAD_FLAG ON) + +find_package(Threads REQUIRED) + +include(GNUInstallDirs) + +# +# build the library +# + +add_subdirectory(src) + +# +# tests and examples +# + +if (GGML_BUILD_TESTS) + enable_testing() + add_subdirectory(tests) +endif () + +if (GGML_BUILD_EXAMPLES) + add_subdirectory(examples) +endif () + +# +# install +# + +include(CMakePackageConfigHelpers) + +# all public headers +set(GGML_PUBLIC_HEADERS + include/ggml.h + include/ggml-cpu.h + include/ggml-alloc.h + include/ggml-backend.h + include/ggml-blas.h + include/ggml-cann.h + include/ggml-cpp.h + include/ggml-cuda.h + include/ggml-opt.h + include/ggml-metal.h + include/ggml-rpc.h + include/ggml-virtgpu.h + include/ggml-sycl.h + include/ggml-vulkan.h + include/ggml-webgpu.h + include/ggml-zendnn.h + include/ggml-openvino.h + include/gguf.h) + +set_target_properties(ggml PROPERTIES PUBLIC_HEADER "${GGML_PUBLIC_HEADERS}") +#if (GGML_METAL) +# set_target_properties(ggml PROPERTIES RESOURCE "${CMAKE_CURRENT_SOURCE_DIR}/src/ggml-metal.metal") +#endif() +install(TARGETS ggml LIBRARY PUBLIC_HEADER) +install(TARGETS ggml-base LIBRARY) + +if (GGML_STANDALONE) + configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ggml.pc.in + ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc + @ONLY) + + install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc + DESTINATION share/pkgconfig) +endif() + +# +# Create CMake package +# + + + +# Capture variables prefixed with GGML_. + +set(variable_set_statements +" +####### Expanded from @GGML_VARIABLES_EXPANED@ by configure_package_config_file() ####### +####### Any changes to this file will be overwritten by the next CMake run ####### + +") + +set(GGML_SHARED_LIB ${BUILD_SHARED_LIBS}) + +get_cmake_property(all_variables VARIABLES) +foreach(variable_name IN LISTS all_variables) + if(variable_name MATCHES "^GGML_") + string(REPLACE ";" "\\;" + variable_value "${${variable_name}}") + + set(variable_set_statements + "${variable_set_statements}set(${variable_name} \"${variable_value}\")\n") + endif() +endforeach() + +set(GGML_VARIABLES_EXPANDED ${variable_set_statements}) + +# Create the CMake package and set install location. + +set(GGML_INSTALL_VERSION ${GGML_VERSION}) +set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header files") +set(GGML_LIB_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR} CACHE PATH "Location of library files") +set(GGML_BIN_INSTALL_DIR ${CMAKE_INSTALL_BINDIR} CACHE PATH "Location of binary files") + +configure_package_config_file( + ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in + ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml + PATH_VARS GGML_INCLUDE_INSTALL_DIR + GGML_LIB_INSTALL_DIR + GGML_BIN_INSTALL_DIR) + +write_basic_package_version_file( + ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake + VERSION ${GGML_INSTALL_VERSION} + COMPATIBILITY SameMajorVersion) + +target_compile_definitions(ggml-base PRIVATE + GGML_VERSION="${GGML_INSTALL_VERSION}" + GGML_COMMIT="${GGML_BUILD_COMMIT}" +) +message(STATUS "ggml version: ${GGML_INSTALL_VERSION}") +message(STATUS "ggml commit: ${GGML_BUILD_COMMIT}") + +install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake + ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml) + +if (MSVC) + set(MSVC_WARNING_FLAGS + /wd4005 # Macro redefinition + /wd4244 # Conversion from one type to another type, possible loss of data + /wd4267 # Conversion from 'size_t' to a smaller type, possible loss of data + /wd4305 # Conversion from 'type1' to 'type2', possible loss of data + /wd4566 # Conversion from 'char' to 'wchar_t', possible loss of data + /wd4996 # Disable POSIX deprecation warnings + /wd4702 # Unreachable code warnings + ) + set(MSVC_COMPILE_OPTIONS + "$<$:/utf-8>" + "$<$:/utf-8>" + ) + function(configure_msvc_target target_name) + if(TARGET ${target_name}) + target_compile_options(${target_name} PRIVATE ${MSVC_WARNING_FLAGS}) + target_compile_options(${target_name} PRIVATE ${MSVC_COMPILE_OPTIONS}) + endif() + endfunction() + + configure_msvc_target(ggml-base) + configure_msvc_target(ggml) + configure_msvc_target(ggml-cpu) + configure_msvc_target(ggml-cpu-x64) + configure_msvc_target(ggml-cpu-sse42) + configure_msvc_target(ggml-cpu-sandybridge) + # __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512 + # skipping ggml-cpu-ivybridge + # skipping ggml-cpu-piledriver + configure_msvc_target(ggml-cpu-haswell) + configure_msvc_target(ggml-cpu-skylakex) + configure_msvc_target(ggml-cpu-cannonlake) + configure_msvc_target(ggml-cpu-cascadelake) + configure_msvc_target(ggml-cpu-icelake) + # MSVC 2022 doesn't support BF16 intrinsics without `/arch:AVX10.1` ?! + # https://learn.microsoft.com/en-us/cpp/intrinsics/x64-amd64-intrinsics-list?view=msvc-170 + # https://learn.microsoft.com/en-us/cpp/build/reference/arch-x64?view=msvc-170 + # skipping ggml-cpu-cooperlake + # skipping ggml-cpu-zen4 + configure_msvc_target(ggml-cpu-alderlake) + # MSVC doesn't support AMX + # skipping ggml-cpu-sapphirerapids + + if (GGML_BUILD_EXAMPLES) + configure_msvc_target(common-ggml) + configure_msvc_target(common) + + configure_msvc_target(mnist-common) + configure_msvc_target(mnist-eval) + configure_msvc_target(mnist-train) + + configure_msvc_target(gpt-2-ctx) + configure_msvc_target(gpt-2-alloc) + configure_msvc_target(gpt-2-backend) + configure_msvc_target(gpt-2-sched) + configure_msvc_target(gpt-2-quantize) + configure_msvc_target(gpt-2-batched) + + configure_msvc_target(gpt-j) + configure_msvc_target(gpt-j-quantize) + + configure_msvc_target(magika) + configure_msvc_target(yolov3-tiny) + configure_msvc_target(sam) + + configure_msvc_target(simple-ctx) + configure_msvc_target(simple-backend) + endif() + + if (GGML_BUILD_TESTS) + configure_msvc_target(test-mul-mat) + configure_msvc_target(test-arange) + configure_msvc_target(test-backend-ops) + configure_msvc_target(test-cont) + configure_msvc_target(test-conv-transpose) + configure_msvc_target(test-conv-transpose-1d) + configure_msvc_target(test-conv1d) + configure_msvc_target(test-conv2d) + configure_msvc_target(test-conv2d-dw) + configure_msvc_target(test-customop) + configure_msvc_target(test-dup) + configure_msvc_target(test-opt) + configure_msvc_target(test-pool) + endif () +endif() diff --git a/ggml/cmake/FindNCCL.cmake b/ggml/cmake/FindNCCL.cmake index 67511e2d56a..9cba6e882c0 100644 --- a/ggml/cmake/FindNCCL.cmake +++ b/ggml/cmake/FindNCCL.cmake @@ -1,36 +1,76 @@ -# cmake/FindNCCL.cmake - -# NVIDIA does not distribute CMake files with NCCl, therefore use this file to find it instead. - -find_path(NCCL_INCLUDE_DIR - NAMES nccl.h - HINTS ${NCCL_ROOT} $ENV{NCCL_ROOT} $ENV{CUDA_HOME} /usr/local/cuda - PATH_SUFFIXES include -) - -find_library(NCCL_LIBRARY - NAMES nccl - HINTS ${NCCL_ROOT} $ENV{NCCL_ROOT} $ENV{CUDA_HOME} /usr/local/cuda - PATH_SUFFIXES lib lib64 -) - -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(NCCL - DEFAULT_MSG - NCCL_LIBRARY NCCL_INCLUDE_DIR -) - -if(NCCL_FOUND) - set(NCCL_LIBRARIES ${NCCL_LIBRARY}) - set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR}) - - if(NOT TARGET NCCL::NCCL) - add_library(NCCL::NCCL UNKNOWN IMPORTED) - set_target_properties(NCCL::NCCL PROPERTIES - IMPORTED_LOCATION "${NCCL_LIBRARY}" - INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIR}" - ) - endif() -endif() - -mark_as_advanced(NCCL_INCLUDE_DIR NCCL_LIBRARY) +# cmake/FindNCCL.cmake + +# NVIDIA does not distribute CMake files with NCCL, therefore use this file to find it instead. +# +# Inputs: +# NCCL_ROOT — root of an NCCL installation or source build tree +# NCCL_STATIC — if ON, prefer the static library and search cmake/lib/Release (or Debug); +# if OFF (default), prefer the shared/import library and search cmake/src/Release + +if(NCCL_STATIC) + # cmake source-build layout: cmake/lib//nccl_static.lib (or nccl.lib) + set(_nccl_lib_names nccl_static nccl) + set(_nccl_extra_lib_hints + "${NCCL_ROOT}/cmake/lib/Release" + "${NCCL_ROOT}/cmake/lib/Debug" + "${NCCL_ROOT}/cmake/lib" + ) +else() + # cmake source-build layout: cmake/src//nccl.lib (import lib for nccl.dll) + set(_nccl_lib_names nccl) + set(_nccl_extra_lib_hints + "${NCCL_ROOT}/cmake/src/Release" + "${NCCL_ROOT}/cmake/src/Debug" + "${NCCL_ROOT}/cmake/src" + ) +endif() + +find_path(NCCL_INCLUDE_DIR + NAMES nccl.h + HINTS + ${NCCL_ROOT} + "${NCCL_ROOT}/cmake/src/Release" + "${NCCL_ROOT}/cmake/src/Debug" + "${NCCL_ROOT}/cmake/src" + "${NCCL_ROOT}/cmake" + $ENV{NCCL_ROOT} + $ENV{CUDA_HOME} + /usr/local/cuda + PATH_SUFFIXES include src/include +) + +find_library(NCCL_LIBRARY + NAMES ${_nccl_lib_names} + HINTS + ${_nccl_extra_lib_hints} + ${NCCL_ROOT} + $ENV{NCCL_ROOT} + $ENV{CUDA_HOME} + /usr/local/cuda + PATH_SUFFIXES lib lib64 +) + +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(NCCL + DEFAULT_MSG + NCCL_LIBRARY NCCL_INCLUDE_DIR +) + +if(NCCL_FOUND) + set(NCCL_LIBRARIES ${NCCL_LIBRARY}) + set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR}) + + if(NOT TARGET NCCL::NCCL) + if(NCCL_STATIC) + add_library(NCCL::NCCL STATIC IMPORTED) + else() + add_library(NCCL::NCCL UNKNOWN IMPORTED) + endif() + set_target_properties(NCCL::NCCL PROPERTIES + IMPORTED_LOCATION "${NCCL_LIBRARY}" + INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIR}" + ) + endif() +endif() + +mark_as_advanced(NCCL_INCLUDE_DIR NCCL_LIBRARY) diff --git a/ggml/src/ggml-cuda/CMakeLists.txt b/ggml/src/ggml-cuda/CMakeLists.txt index b54d4a6b107..e00afda9c92 100644 --- a/ggml/src/ggml-cuda/CMakeLists.txt +++ b/ggml/src/ggml-cuda/CMakeLists.txt @@ -1,268 +1,271 @@ -cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES - -find_package(CUDAToolkit) - -if (CUDAToolkit_FOUND) - message(STATUS "CUDA Toolkit found") - - if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES) - # native == GPUs available at build time - # 50 == Maxwell, lowest CUDA 12 standard - # 60 == P100, FP16 CUDA intrinsics - # 61 == Pascal, __dp4a instruction (per-byte integer dot product) - # 70 == V100, FP16 tensor cores - # 75 == Turing, int8 tensor cores - # 80 == Ampere, asynchronous data loading, faster tensor core instructions - # 86 == RTX 3000, needs CUDA v11.1 - # 89 == RTX 4000, needs CUDA v11.8 - # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores - # - # XX-virtual == compile CUDA code as PTX, do JIT compilation to binary code on first run - # XX-real == compile CUDA code as device code for this specific architecture - # no suffix == compile as both PTX and device code - # - # The default behavior for a non-native is to build virtual architectures as needed to cover all features needed - # for best performance and to also build real architectures for the most commonly used GPUs. - if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24") - set(CMAKE_CUDA_ARCHITECTURES "native") - else() - if (CUDAToolkit_VERSION VERSION_LESS "13") - list(APPEND CMAKE_CUDA_ARCHITECTURES 50-virtual 61-virtual 70-virtual) - endif () - - list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real) - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8") - list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real) - endif() - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") - # The CUDA architecture 120f-virtual would in principle work for Blackwell support - # but the newly added "f" suffix conflicted with a preexising regex for validating CUDA architectures in CMake. - # So either a recent CMake version or one with the backported fix is needed. - # The following versions should work: - # - CMake >= v3.31.8 && CMake < v4.0.0 - # - CMake >= v4.0.2 - # This is NOT documented in the CMake release notes, - # check Modules/Internal/CMakeCUDAArchitecturesValidate.cmake in the CMake git repository instead. - # However, the architectures 120a-real and 121a-real should work with basically any CMake version and - # until the release of e.g. Rubin there is no benefit to shipping virtual architectures for Blackwell. - list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real) - endif() - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9") - list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real) - endif() - endif() - endif() - - enable_language(CUDA) - - # TODO: Remove once CCCL 3.2 has been released and bundled with CUDA Toolkit - if (GGML_CUDA_CUB_3DOT2) - include(FetchContent) - - FetchContent_Declare( - CCCL - GIT_REPOSITORY https://github.com/nvidia/cccl.git - GIT_TAG v3.2.0 - GIT_SHALLOW TRUE - ) - - FetchContent_MakeAvailable(CCCL) - endif() - - # Replace any plain 12X CUDA architectures with their "architecture-specific" equivalents 12Xa. - # 12X is forwards-compatible, 12Xa is not. - # Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa. - # But while 12X vs. 12Xa can be checked in device code there is (to my knowledge) no easy way to do the same check in host code. - # So for now just replace all instances of 12X with 12Xa, this should be fine until Rubin is released. - foreach(ARCHS IN ITEMS CMAKE_CUDA_ARCHITECTURES CMAKE_CUDA_ARCHITECTURES_NATIVE) - set(FIXED_ARCHS "") - foreach(ARCH IN LISTS ${ARCHS}) - if (ARCH MATCHES "^12[0-9](-real|-virtual)?$") - string(REGEX REPLACE "^(12[0-9])((-real|-virtual)?)$" "\\1a\\2" FIXED_ARCH ${ARCH}) - message(STATUS "Replacing ${ARCH} in ${ARCHS} with ${FIXED_ARCH}") - list(APPEND FIXED_ARCHS "${FIXED_ARCH}") - else() - list(APPEND FIXED_ARCHS "${ARCH}") - endif() - endforeach() - set(${ARCHS} ${FIXED_ARCHS}) - endforeach() - - # If we try to compile a "native" build it will use the 12X architectures and fail. - # So we should instead use the native architectures as determined by CMake after replacing 12X with 12Xa. - # But if at the time of the build no GPUs are connected at all CMAKE_CUDA_ARCHITECTURES will contain garbage that we should not use. - if (CMAKE_CUDA_ARCHITECTURES STREQUAL "native" AND CMAKE_CUDA_ARCHITECTURES_NATIVE MATCHES "^[0-9]+(a|f)?(-real|-virtual)?(;[0-9]+(a|f)?(-real|-virtual)?|;)*$") - set(CMAKE_CUDA_ARCHITECTURES ${CMAKE_CUDA_ARCHITECTURES_NATIVE}) - endif() - message(STATUS "Using CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES} CMAKE_CUDA_ARCHITECTURES_NATIVE=${CMAKE_CUDA_ARCHITECTURES_NATIVE}") - - file(GLOB GGML_HEADERS_CUDA "*.cuh") - list(APPEND GGML_HEADERS_CUDA "../../include/ggml-cuda.h") - - file(GLOB GGML_SOURCES_CUDA "*.cu") - file(GLOB SRCS "template-instances/fattn-tile*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/fattn-mma*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/mmq*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/mmf*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - - if (GGML_CUDA_FA_ALL_QUANTS) - file(GLOB SRCS "template-instances/fattn-vec*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - add_compile_definitions(GGML_CUDA_FA_ALL_QUANTS) - else() - list(APPEND GGML_SOURCES_CUDA - template-instances/fattn-vec-instance-f16-f16.cu - template-instances/fattn-vec-instance-q4_0-q4_0.cu - template-instances/fattn-vec-instance-q8_0-q8_0.cu - template-instances/fattn-vec-instance-bf16-bf16.cu) - endif() - - ggml_add_backend_library(ggml-cuda - ${GGML_HEADERS_CUDA} - ${GGML_SOURCES_CUDA} - ) - - add_compile_definitions(GGML_CUDA_PEER_MAX_BATCH_SIZE=${GGML_CUDA_PEER_MAX_BATCH_SIZE}) - - if (GGML_CUDA_GRAPHS) - add_compile_definitions(GGML_CUDA_USE_GRAPHS) - endif() - - if (GGML_CUDA_FORCE_MMQ) - add_compile_definitions(GGML_CUDA_FORCE_MMQ) - endif() - - if (GGML_CUDA_FORCE_CUBLAS) - add_compile_definitions(GGML_CUDA_FORCE_CUBLAS) - endif() - - if (GGML_CUDA_NO_VMM) - add_compile_definitions(GGML_CUDA_NO_VMM) - endif() - - if (NOT GGML_CUDA_FA) - add_compile_definitions(GGML_CUDA_NO_FA) - endif() - - if (GGML_CUDA_NO_PEER_COPY) - add_compile_definitions(GGML_CUDA_NO_PEER_COPY) - endif() - - if (GGML_STATIC) - if (WIN32) - # As of 12.3.1 CUDA Toolkit for Windows does not offer a static cublas library - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas) - else () - if (GGML_CUDA_CUB_3DOT2) - target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) - endif() - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "10.1") - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static CUDA::cublasLt_static) - else() - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static) - endif() - endif() - else() - if (GGML_CUDA_CUB_3DOT2) - target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) - endif() - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart CUDA::cublas) - endif() - - if (GGML_CUDA_NO_VMM) - # No VMM requested, no need to link directly with the cuda driver lib (libcuda.so) - else() - target_link_libraries(ggml-cuda PRIVATE CUDA::cuda_driver) - endif() - - if (GGML_CUDA_NCCL) - find_package(NCCL) - if (NCCL_FOUND) - add_compile_definitions(GGML_USE_NCCL) - target_link_libraries(ggml-cuda PRIVATE NCCL::NCCL) - else() - message(STATUS "Warning: NCCL not found, performance for multiple CUDA GPUs will be suboptimal") - endif() - endif() - - set(CUDA_CXX_FLAGS "") - - set(CUDA_FLAGS -use_fast_math -extended-lambda) - - if (GGML_CUDA_DEBUG) - list(APPEND CUDA_FLAGS -lineinfo) - add_compile_definitions(GGML_CUDA_DEBUG) - endif() - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") - # Options are: - # - none (not recommended) - # - speed (nvcc's default) - # - balance - # - size - list(APPEND CUDA_FLAGS -compress-mode=${GGML_CUDA_COMPRESSION_MODE}) - endif() - - if (GGML_FATAL_WARNINGS) - list(APPEND CUDA_FLAGS -Werror all-warnings) - endif() - - if (GGML_ALL_WARNINGS AND NOT MSVC) - set(NVCC_CMD ${CMAKE_CUDA_COMPILER} .c) - if (NOT CMAKE_CUDA_HOST_COMPILER STREQUAL "") - list(APPEND NVCC_CMD -ccbin ${CMAKE_CUDA_HOST_COMPILER}) - endif() - - execute_process( - COMMAND ${NVCC_CMD} -Xcompiler --version - OUTPUT_VARIABLE CUDA_CCFULLVER - ERROR_QUIET - ) - - if (NOT CUDA_CCFULLVER MATCHES clang) - set(CUDA_CCID "GNU") - execute_process( - COMMAND ${NVCC_CMD} -Xcompiler "-dumpfullversion -dumpversion" - OUTPUT_VARIABLE CUDA_CCVER - ERROR_QUIET - OUTPUT_STRIP_TRAILING_WHITESPACE - ) - else() - if (CUDA_CCFULLVER MATCHES Apple) - set(CUDA_CCID "AppleClang") - else() - set(CUDA_CCID "Clang") - endif() - string(REGEX REPLACE "^.* version ([0-9.]*).*$" "\\1" CUDA_CCVER ${CUDA_CCFULLVER}) - endif() - - message(STATUS "CUDA host compiler is ${CUDA_CCID} ${CUDA_CCVER}") - - ggml_get_flags(${CUDA_CCID} ${CUDA_CCVER}) - list(APPEND CUDA_CXX_FLAGS ${CXX_FLAGS} ${GF_CXX_FLAGS}) # This is passed to -Xcompiler later - endif() - - if (NOT MSVC) - list(APPEND CUDA_CXX_FLAGS -Wno-pedantic) - else() - # CCCL 3.2 onwards will require a cpp-standard-compliant preprocessor for MSVC - # https://github.com/NVIDIA/cccl/pull/6827 - list(APPEND CUDA_CXX_FLAGS /Zc:preprocessor) - endif() - - list(JOIN CUDA_CXX_FLAGS " " CUDA_CXX_FLAGS_JOINED) # pass host compiler flags as a single argument - - if (NOT CUDA_CXX_FLAGS_JOINED STREQUAL "") - list(APPEND CUDA_FLAGS -Xcompiler ${CUDA_CXX_FLAGS_JOINED}) - endif() - - target_compile_options(ggml-cuda PRIVATE "$<$:${CUDA_FLAGS}>") -else() - message(FATAL_ERROR "CUDA Toolkit not found") -endif() +cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES + +find_package(CUDAToolkit) + +if (CUDAToolkit_FOUND) + message(STATUS "CUDA Toolkit found") + + if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES) + # native == GPUs available at build time + # 50 == Maxwell, lowest CUDA 12 standard + # 60 == P100, FP16 CUDA intrinsics + # 61 == Pascal, __dp4a instruction (per-byte integer dot product) + # 70 == V100, FP16 tensor cores + # 75 == Turing, int8 tensor cores + # 80 == Ampere, asynchronous data loading, faster tensor core instructions + # 86 == RTX 3000, needs CUDA v11.1 + # 89 == RTX 4000, needs CUDA v11.8 + # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores + # + # XX-virtual == compile CUDA code as PTX, do JIT compilation to binary code on first run + # XX-real == compile CUDA code as device code for this specific architecture + # no suffix == compile as both PTX and device code + # + # The default behavior for a non-native is to build virtual architectures as needed to cover all features needed + # for best performance and to also build real architectures for the most commonly used GPUs. + if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24") + set(CMAKE_CUDA_ARCHITECTURES "native") + else() + if (CUDAToolkit_VERSION VERSION_LESS "13") + list(APPEND CMAKE_CUDA_ARCHITECTURES 50-virtual 61-virtual 70-virtual) + endif () + + list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real) + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8") + list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real) + endif() + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") + # The CUDA architecture 120f-virtual would in principle work for Blackwell support + # but the newly added "f" suffix conflicted with a preexising regex for validating CUDA architectures in CMake. + # So either a recent CMake version or one with the backported fix is needed. + # The following versions should work: + # - CMake >= v3.31.8 && CMake < v4.0.0 + # - CMake >= v4.0.2 + # This is NOT documented in the CMake release notes, + # check Modules/Internal/CMakeCUDAArchitecturesValidate.cmake in the CMake git repository instead. + # However, the architectures 120a-real and 121a-real should work with basically any CMake version and + # until the release of e.g. Rubin there is no benefit to shipping virtual architectures for Blackwell. + list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real) + endif() + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9") + list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real) + endif() + endif() + endif() + + enable_language(CUDA) + + # TODO: Remove once CCCL 3.2 has been released and bundled with CUDA Toolkit + if (GGML_CUDA_CUB_3DOT2) + include(FetchContent) + + FetchContent_Declare( + CCCL + GIT_REPOSITORY https://github.com/nvidia/cccl.git + GIT_TAG v3.2.0 + GIT_SHALLOW TRUE + ) + + FetchContent_MakeAvailable(CCCL) + endif() + + # Replace any plain 12X CUDA architectures with their "architecture-specific" equivalents 12Xa. + # 12X is forwards-compatible, 12Xa is not. + # Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa. + # But while 12X vs. 12Xa can be checked in device code there is (to my knowledge) no easy way to do the same check in host code. + # So for now just replace all instances of 12X with 12Xa, this should be fine until Rubin is released. + foreach(ARCHS IN ITEMS CMAKE_CUDA_ARCHITECTURES CMAKE_CUDA_ARCHITECTURES_NATIVE) + set(FIXED_ARCHS "") + foreach(ARCH IN LISTS ${ARCHS}) + if (ARCH MATCHES "^12[0-9](-real|-virtual)?$") + string(REGEX REPLACE "^(12[0-9])((-real|-virtual)?)$" "\\1a\\2" FIXED_ARCH ${ARCH}) + message(STATUS "Replacing ${ARCH} in ${ARCHS} with ${FIXED_ARCH}") + list(APPEND FIXED_ARCHS "${FIXED_ARCH}") + else() + list(APPEND FIXED_ARCHS "${ARCH}") + endif() + endforeach() + set(${ARCHS} ${FIXED_ARCHS}) + endforeach() + + # If we try to compile a "native" build it will use the 12X architectures and fail. + # So we should instead use the native architectures as determined by CMake after replacing 12X with 12Xa. + # But if at the time of the build no GPUs are connected at all CMAKE_CUDA_ARCHITECTURES will contain garbage that we should not use. + if (CMAKE_CUDA_ARCHITECTURES STREQUAL "native" AND CMAKE_CUDA_ARCHITECTURES_NATIVE MATCHES "^[0-9]+(a|f)?(-real|-virtual)?(;[0-9]+(a|f)?(-real|-virtual)?|;)*$") + set(CMAKE_CUDA_ARCHITECTURES ${CMAKE_CUDA_ARCHITECTURES_NATIVE}) + endif() + message(STATUS "Using CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES} CMAKE_CUDA_ARCHITECTURES_NATIVE=${CMAKE_CUDA_ARCHITECTURES_NATIVE}") + + file(GLOB GGML_HEADERS_CUDA "*.cuh") + list(APPEND GGML_HEADERS_CUDA "../../include/ggml-cuda.h") + + file(GLOB GGML_SOURCES_CUDA "*.cu") + file(GLOB SRCS "template-instances/fattn-tile*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/fattn-mma*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/mmq*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/mmf*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + + if (GGML_CUDA_FA_ALL_QUANTS) + file(GLOB SRCS "template-instances/fattn-vec*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + add_compile_definitions(GGML_CUDA_FA_ALL_QUANTS) + else() + list(APPEND GGML_SOURCES_CUDA + template-instances/fattn-vec-instance-f16-f16.cu + template-instances/fattn-vec-instance-q4_0-q4_0.cu + template-instances/fattn-vec-instance-q8_0-q8_0.cu + template-instances/fattn-vec-instance-bf16-bf16.cu) + endif() + + ggml_add_backend_library(ggml-cuda + ${GGML_HEADERS_CUDA} + ${GGML_SOURCES_CUDA} + ) + + add_compile_definitions(GGML_CUDA_PEER_MAX_BATCH_SIZE=${GGML_CUDA_PEER_MAX_BATCH_SIZE}) + + if (GGML_CUDA_GRAPHS) + add_compile_definitions(GGML_CUDA_USE_GRAPHS) + endif() + + if (GGML_CUDA_FORCE_MMQ) + add_compile_definitions(GGML_CUDA_FORCE_MMQ) + endif() + + if (GGML_CUDA_FORCE_CUBLAS) + add_compile_definitions(GGML_CUDA_FORCE_CUBLAS) + endif() + + if (GGML_CUDA_NO_VMM) + add_compile_definitions(GGML_CUDA_NO_VMM) + endif() + + if (NOT GGML_CUDA_FA) + add_compile_definitions(GGML_CUDA_NO_FA) + endif() + + if (GGML_CUDA_NO_PEER_COPY) + add_compile_definitions(GGML_CUDA_NO_PEER_COPY) + endif() + + if (GGML_STATIC) + if (WIN32) + # As of 12.3.1 CUDA Toolkit for Windows does not offer a static cublas library + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas) + else () + if (GGML_CUDA_CUB_3DOT2) + target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) + endif() + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "10.1") + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static CUDA::cublasLt_static) + else() + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static) + endif() + endif() + else() + if (GGML_CUDA_CUB_3DOT2) + target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) + endif() + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart CUDA::cublas) + endif() + + if (GGML_CUDA_NO_VMM) + # No VMM requested, no need to link directly with the cuda driver lib (libcuda.so) + else() + target_link_libraries(ggml-cuda PRIVATE CUDA::cuda_driver) + endif() + + if (GGML_CUDA_NCCL) + if (GGML_CUDA_NCCL_STATIC) + set(NCCL_STATIC ON) + endif() + find_package(NCCL) + if (NCCL_FOUND) + add_compile_definitions(GGML_USE_NCCL) + target_link_libraries(ggml-cuda PRIVATE NCCL::NCCL) + else() + message(STATUS "Warning: NCCL not found, performance for multiple CUDA GPUs will be suboptimal") + endif() + endif() + + set(CUDA_CXX_FLAGS "") + + set(CUDA_FLAGS -use_fast_math -extended-lambda) + + if (GGML_CUDA_DEBUG) + list(APPEND CUDA_FLAGS -lineinfo) + add_compile_definitions(GGML_CUDA_DEBUG) + endif() + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") + # Options are: + # - none (not recommended) + # - speed (nvcc's default) + # - balance + # - size + list(APPEND CUDA_FLAGS -compress-mode=${GGML_CUDA_COMPRESSION_MODE}) + endif() + + if (GGML_FATAL_WARNINGS) + list(APPEND CUDA_FLAGS -Werror all-warnings) + endif() + + if (GGML_ALL_WARNINGS AND NOT MSVC) + set(NVCC_CMD ${CMAKE_CUDA_COMPILER} .c) + if (NOT CMAKE_CUDA_HOST_COMPILER STREQUAL "") + list(APPEND NVCC_CMD -ccbin ${CMAKE_CUDA_HOST_COMPILER}) + endif() + + execute_process( + COMMAND ${NVCC_CMD} -Xcompiler --version + OUTPUT_VARIABLE CUDA_CCFULLVER + ERROR_QUIET + ) + + if (NOT CUDA_CCFULLVER MATCHES clang) + set(CUDA_CCID "GNU") + execute_process( + COMMAND ${NVCC_CMD} -Xcompiler "-dumpfullversion -dumpversion" + OUTPUT_VARIABLE CUDA_CCVER + ERROR_QUIET + OUTPUT_STRIP_TRAILING_WHITESPACE + ) + else() + if (CUDA_CCFULLVER MATCHES Apple) + set(CUDA_CCID "AppleClang") + else() + set(CUDA_CCID "Clang") + endif() + string(REGEX REPLACE "^.* version ([0-9.]*).*$" "\\1" CUDA_CCVER ${CUDA_CCFULLVER}) + endif() + + message(STATUS "CUDA host compiler is ${CUDA_CCID} ${CUDA_CCVER}") + + ggml_get_flags(${CUDA_CCID} ${CUDA_CCVER}) + list(APPEND CUDA_CXX_FLAGS ${CXX_FLAGS} ${GF_CXX_FLAGS}) # This is passed to -Xcompiler later + endif() + + if (NOT MSVC) + list(APPEND CUDA_CXX_FLAGS -Wno-pedantic) + else() + # CCCL 3.2 onwards will require a cpp-standard-compliant preprocessor for MSVC + # https://github.com/NVIDIA/cccl/pull/6827 + list(APPEND CUDA_CXX_FLAGS /Zc:preprocessor) + endif() + + list(JOIN CUDA_CXX_FLAGS " " CUDA_CXX_FLAGS_JOINED) # pass host compiler flags as a single argument + + if (NOT CUDA_CXX_FLAGS_JOINED STREQUAL "") + list(APPEND CUDA_FLAGS -Xcompiler ${CUDA_CXX_FLAGS_JOINED}) + endif() + + target_compile_options(ggml-cuda PRIVATE "$<$:${CUDA_FLAGS}>") +else() + message(FATAL_ERROR "CUDA Toolkit not found") +endif() From c5207a4e375b631f8feffd3b9fb5b16858ae3433 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 19:50:17 -0700 Subject: [PATCH 06/81] ggml-cuda: add AllReduce hang watchdog (GGML_CUDA_AR_WATCHDOG) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When compiled with -DGGML_CUDA_AR_WATCHDOG=ON, uses a debug kernel variant that writes per-GPU spin diagnostics to pinned host memory. A host-side blocking poll (cudaEventQuery + volatile reads) detects hangs and logs WARN with the last observed arrival counters and spin counts, controlled by GGML_CUDA_AR_WATCHDOG (ms timeout) and GGML_CUDA_AR_MAX_SPIN (kernel bailout) env vars at runtime. Zero overhead on the production path — all debug code is behind #ifdef. Co-Authored-By: Claude Sonnet 4.6 ar_pipeline field - Provider selection via GGML_CUDA_ALLREDUCE env var ("nccl" / "internal") - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/CMakeLists.txt | 1 + ggml/src/ggml-cuda/CMakeLists.txt | 4 + ggml/src/ggml-cuda/allreduce.cu | 329 +++++++++++++++++++++++++++--- 3 files changed, 302 insertions(+), 32 deletions(-) diff --git a/ggml/CMakeLists.txt b/ggml/CMakeLists.txt index 785ce7d64f5..bbc7800ab89 100644 --- a/ggml/CMakeLists.txt +++ b/ggml/CMakeLists.txt @@ -209,6 +209,7 @@ option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashA option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) +option(GGML_CUDA_AR_WATCHDOG "ggml: enable internal AllReduce hang watchdog (debug)" OFF) set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING "ggml: cuda link binary compression mode; requires cuda 12.8+") set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") diff --git a/ggml/src/ggml-cuda/CMakeLists.txt b/ggml/src/ggml-cuda/CMakeLists.txt index e00afda9c92..3286a6840aa 100644 --- a/ggml/src/ggml-cuda/CMakeLists.txt +++ b/ggml/src/ggml-cuda/CMakeLists.txt @@ -194,6 +194,10 @@ if (CUDAToolkit_FOUND) endif() endif() + if (GGML_CUDA_AR_WATCHDOG) + add_compile_definitions(GGML_CUDA_AR_WATCHDOG) + endif() + set(CUDA_CXX_FLAGS "") set(CUDA_FLAGS -use_fast_math -extended-lambda) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 51bcfacf64e..eb239f71d43 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -1,8 +1,14 @@ #include "allreduce.cuh" #include "ggml-impl.h" +#include #include +#ifdef GGML_CUDA_AR_WATCHDOG +#include +#include +#endif + // --------------------------------------------------------------------------- // Cross-GPU signal mechanism // @@ -12,8 +18,8 @@ // __threadfence_system() provides the release ordering that makes the D2H // writes visible system-wide before the arrival flag is observed. // -// atomicAdd_system() (mechanism 0 in the prototype) is broken on RTX 5090 -// (hostNativeAtomicSupported = 0), so we use the volatile path throughout. +// atomicAdd_system() is broken on RTX 5090 (hostNativeAtomicSupported = 0), +// so we use the volatile path throughout. // --------------------------------------------------------------------------- static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { @@ -26,7 +32,7 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { } // --------------------------------------------------------------------------- -// Single-phase AllReduce kernel — float32, 2 GPUs +// Single-phase AllReduce kernel — float32, 2 GPUs (production) // // Both GPUs run this kernel simultaneously in independent streams. Each GPU: // @@ -60,20 +66,16 @@ static __global__ void ggml_cuda_ar_f32_kernel( for (int i = tid; i < count4; i += nt) { d4[i] = s4[i]; } - // Scalar tail if count is not a multiple of 4. if (tid < count - tail) { host_mine[tail + tid] = sendbuf[tail + tid]; } } - // Commit all host writes before signalling; __syncthreads() ensures - // every thread's stores are in flight before thread 0 writes the flag. + // Commit all host writes before signalling. __threadfence_system(); __syncthreads(); - // Phase 2: thread 0 signals this GPU's arrival, then spins until the - // peer signals back. The spin uses __nanosleep to yield the SM to - // other work rather than burning cycles in a hot loop. + // Phase 2: thread 0 signals arrival, then spins for the peer. if (tid == 0) { ggml_cuda_ar_signal_set(arrival_mine); while (ggml_cuda_ar_signal_get(arrival_other) == 0) { @@ -81,11 +83,120 @@ static __global__ void ggml_cuda_ar_f32_kernel( } } - // Broadcast "peer has arrived" and acquire the peer's host_other writes. + // Broadcast "peer has arrived" and acquire peer's host_other writes. + __syncthreads(); + __threadfence_system(); + + // Phase 3: reduce. + { + const float4 * s4 = reinterpret_cast(sendbuf); + const float4 * o4 = reinterpret_cast(host_other); + float4 * r4 = reinterpret_cast(recvbuf); + for (int i = tid; i < count4; i += nt) { + float4 a = s4[i]; + float4 b = o4[i]; + r4[i] = make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); + } + if (tid < count - tail) { + recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; + } + } +} + +// --------------------------------------------------------------------------- +// Watchdog debug variant — compiled only when GGML_CUDA_AR_WATCHDOG is defined. +// +// Adds three extra parameters to the kernel and instruments Phase 2: +// +// debug[0] spin iteration count, updated every ~4096 iterations +// (-1 written on max_spin bailout as a sentinel) +// debug[1] last value of arrival_other observed during the spin +// debug[2] readback of arrival_mine immediately after signal_set; +// should always be 1 — if 0, the write did not reach host memory +// debug[3] reserved (always 0) +// +// max_spin if > 0, the spin bails out after this many iterations and the +// kernel logs via printf before proceeding to Phase 3 with stale +// data. Output will be numerically wrong, but the kernel exits +// rather than hanging, which is useful for post-mortem logging. +// +// rank this GPU's rank within the communicator (for printf output) +// --------------------------------------------------------------------------- +#ifdef GGML_CUDA_AR_WATCHDOG +static __global__ void ggml_cuda_ar_f32_kernel_dbg( + const float * __restrict__ sendbuf, + float * __restrict__ recvbuf, + float * __restrict__ host_mine, + const float * __restrict__ host_other, + int count, + int * arrival_mine, + int * arrival_other, + int * debug, + int max_spin, + int rank) { + + const int tid = threadIdx.x; + const int nt = blockDim.x; + const int count4 = count >> 2; + const int tail = count4 << 2; + + // Phase 1: D2H copy (identical to production kernel). + { + const float4 * s4 = reinterpret_cast(sendbuf); + float4 * d4 = reinterpret_cast(host_mine); + for (int i = tid; i < count4; i += nt) { + d4[i] = s4[i]; + } + if (tid < count - tail) { + host_mine[tail + tid] = sendbuf[tail + tid]; + } + } + + __threadfence_system(); + __syncthreads(); + + // Phase 2: signal + instrumented spin. + if (tid == 0) { + ggml_cuda_ar_signal_set(arrival_mine); + + // Readback: can this GPU see its own signal? If writeback == 0 the + // write did not reach host-visible memory, which would explain why + // the peer never observes arrival. + int writeback = ggml_cuda_ar_signal_get(arrival_mine); + debug[2] = writeback; + + int spin = 0; + int last = 0; + while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { + ++spin; + // Periodically expose progress so the host watchdog can read it. + if ((spin & 0xFFF) == 0) { + debug[0] = spin; + debug[1] = last; + } + if (max_spin > 0 && spin >= max_spin) { + debug[0] = -1; // bailout sentinel + debug[1] = last; + printf("ggml_cuda_ar: rank=%d BAILOUT after %d spins; " + "writeback_mine=%d arrival_other=%d " + "(pm=%p po=%p)\n", + rank, spin, writeback, last, + (void *)arrival_mine, (void *)arrival_other); + break; + } + __nanosleep(100); + } + // Write final spin count, unless we already wrote the bailout sentinel. + if (debug[0] != -1) { + debug[0] = spin; + debug[1] = last; + } + } + + // Phase 3: reduce (proceeds even after bailout; output will be wrong). __syncthreads(); __threadfence_system(); - // Phase 3: reduce — each thread handles its slice of the output. { const float4 * s4 = reinterpret_cast(sendbuf); const float4 * o4 = reinterpret_cast(host_other); @@ -100,6 +211,7 @@ static __global__ void ggml_cuda_ar_f32_kernel( } } } +#endif // GGML_CUDA_AR_WATCHDOG // --------------------------------------------------------------------------- // Pipeline structure @@ -109,11 +221,23 @@ static __global__ void ggml_cuda_ar_f32_kernel( // in-flight depth (single digits in practice) while keeping init cost low. static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; -// Byte spacing between adjacent arrival ints. Two cache lines (128 bytes) +// Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) // ensures the arrival slots for the two GPUs never share a cache line, // preventing false-sharing stalls on the polling GPU. static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; +#ifdef GGML_CUDA_AR_WATCHDOG +// Ints per device in the debug buffer. Layout (index → meaning): +// 0 spin count, updated every ~4096 iterations (-1 = bailed out) +// 1 last arrival_other value observed during the spin +// 2 readback of arrival_mine after signal_set (should always be 1) +// 3 reserved +static constexpr int GGML_CUDA_AR_DEBUG_INTS = 4; + +// Host poll interval for the blocking watchdog loop. +static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 20; +#endif + struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete cudaEvent_t ker = nullptr; // AllReduce kernel complete @@ -126,13 +250,21 @@ struct ggml_cuda_ar_pipeline { uint64_t call_count; // Per-device resources. - float * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging - cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking kernel streams - ggml_cuda_ar_event_slot * ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] + float * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging + cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking + ggml_cuda_ar_event_slot *ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. - // Index helper: use ggml_cuda_ar_arrival_ptr(). + // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; + +#ifdef GGML_CUDA_AR_WATCHDOG + // Pinned debug buffer written by the debug kernel, read by the host + // watchdog. Layout: debug_buf[rank * GGML_CUDA_AR_DEBUG_INTS + field]. + int * debug_buf; + int wdog_timeout_ms; // 0 = watchdog disabled (env: GGML_CUDA_AR_WATCHDOG) + int wdog_max_spin; // 0 = spin forever (env: GGML_CUDA_AR_MAX_SPIN) +#endif }; // Return a pointer to the arrival int for (slot, rank). @@ -141,6 +273,79 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, return reinterpret_cast(p->arrival + offset); } +// --------------------------------------------------------------------------- +// Watchdog poll — blocks the calling thread, polling ker events and reading +// debug state from pinned host memory. Only active when wdog_timeout_ms > 0. +// +// Called after all kernels for the current slot have been queued. The GPU +// streams proceed independently; this function only observes via +// cudaEventQuery and volatile reads of debug_buf. +// +// Output is deliberately sparse on the fast path: nothing is logged when +// both kernels complete before the first poll tick. If a kernel is still +// running on the first tick, all subsequent ticks are logged (including the +// final "done" tick) so the log captures the full timeline. +// --------------------------------------------------------------------------- +#ifdef GGML_CUDA_AR_WATCHDOG +static void ggml_cuda_ar_watchdog_poll( + const ggml_cuda_ar_pipeline * p, + int slot, int n, + const cudaEvent_t * ker_events) { + if (p->wdog_timeout_ms <= 0) { + return; + } + + int elapsed_ms = 0; + bool observed_busy = false; + + while (elapsed_ms <= p->wdog_timeout_ms) { + // Query completion state of every GPU's kernel event. + bool all_done = true; + cudaError_t qstat[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) { + ggml_cuda_set_device(p->devices[i]); + qstat[i] = cudaEventQuery(ker_events[i]); + if (qstat[i] != cudaSuccess) { + all_done = false; + } + } + + if (!all_done) { + observed_busy = true; + } + + // Log on every tick that is either slow or the first "done" after slow. + if (!all_done || observed_busy) { + char msg[512]; + int pos = 0; + pos += snprintf(msg + pos, sizeof(msg) - pos, + "ggml_cuda_ar watchdog +%dms slot=%d:", + elapsed_ms, slot); + for (int i = 0; i < n; ++i) { + const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; + int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, slot, i); + int spin = *(volatile int *)&dbg[0]; + int last = *(volatile int *)&dbg[1]; + int wb = *(volatile int *)&dbg[2]; + pos += snprintf(msg + pos, sizeof(msg) - pos, + " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d]", + p->devices[i], + qstat[i] == cudaSuccess ? "done" : "busy", + arr, spin, last, wb); + } + GGML_LOG_WARN("%s\n", msg); + } + + if (all_done) { + break; + } + + std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); + elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; + } +} +#endif // GGML_CUDA_AR_WATCHDOG + // --------------------------------------------------------------------------- // Init / free // --------------------------------------------------------------------------- @@ -160,6 +365,11 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( p->streams[i] = nullptr; p->ev_pool[i] = nullptr; } +#ifdef GGML_CUDA_AR_WATCHDOG + p->debug_buf = nullptr; + p->wdog_timeout_ms = 0; + p->wdog_max_spin = 0; +#endif // Per-device streams and event pools. for (int i = 0; i < n_devices; ++i) { @@ -211,11 +421,32 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( memset(p->host_buf[i], 0, max_bytes); } - // Warmup: run the kernel N times at the expected tensor size to pay the - // first-use driver / PCIe / page-mapping cost during model load rather - // than during the first inference step, and to encourage the GPU clock - // governor to boost before timing begins. - // Currently limited to the 2-GPU case. +#ifdef GGML_CUDA_AR_WATCHDOG + // Debug buffer: written by the instrumented kernel, polled by the host. + { + const size_t dbg_bytes = (size_t)n_devices * GGML_CUDA_AR_DEBUG_INTS * sizeof(int); + if (cudaHostAlloc(reinterpret_cast(&p->debug_buf), dbg_bytes, + cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for debug buffer failed (%zu bytes)\n", + __func__, dbg_bytes); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + memset(p->debug_buf, 0, dbg_bytes); + + const char * wdog_env = getenv("GGML_CUDA_AR_WATCHDOG"); + const char * spin_env = getenv("GGML_CUDA_AR_MAX_SPIN"); + p->wdog_timeout_ms = (wdog_env && wdog_env[0]) ? atoi(wdog_env) : 0; + p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; + GGML_LOG_INFO("%s: AR watchdog enabled — timeout=%dms max_spin=%d " + "(set GGML_CUDA_AR_WATCHDOG= / GGML_CUDA_AR_MAX_SPIN= to adjust)\n", + __func__, p->wdog_timeout_ms, p->wdog_max_spin); + } +#endif + + // Warmup: run the kernel N times to pay first-use driver / PCIe / + // page-mapping costs during model load and encourage the GPU clock + // governor to boost before inference begins. if (n_devices == 2) { constexpr int WARMUP_ITERS = 64; constexpr size_t WARMUP_COUNT = 8192; // 32 KB of fp32 @@ -234,7 +465,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( } if (warmup_ok) { - // Reuse slot 0 for every iteration, resetting arrival before each. + // Warmup always uses the production kernel (no debug overhead). for (int iter = 0; iter < WARMUP_ITERS; ++iter) { for (int r = 0; r < 2; ++r) { *ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r) = 0; @@ -296,6 +527,11 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { if (p->arrival) { cudaFreeHost(p->arrival); } +#ifdef GGML_CUDA_AR_WATCHDOG + if (p->debug_buf) { + cudaFreeHost(p->debug_buf); + } +#endif delete p; } @@ -316,8 +552,7 @@ bool ggml_cuda_ar_allreduce( return false; } - // Only FP32 tensors are handled by the kernel; other types need a - // separate implementation. + // Only FP32 tensors are handled by the kernel. if (tensors[0]->type != GGML_TYPE_F32) { return false; } @@ -330,15 +565,13 @@ bool ggml_cuda_ar_allreduce( } if (bytes > p->buf_bytes) { - // Staging buffers too small; the caller should fall back. - // TODO: reallocate or chunk for larger tensors. return false; } // Cycle through the event pool. On the second pass through the ring, // synchronise on the slot's ker event before touching arrival ints — - // the event and arrival pools wrap in lock-step so this guarantees that - // the kernels which last used this slot have finished. + // the event and arrival pools wrap in lock-step so this guarantees the + // kernels which last used this slot have finished. const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; p->call_count++; @@ -355,12 +588,20 @@ bool ggml_cuda_ar_allreduce( *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; } +#ifdef GGML_CUDA_AR_WATCHDOG + // Clear per-device debug state so the watchdog reads reflect only this call. + memset(p->debug_buf, 0, (size_t)n * GGML_CUDA_AR_DEBUG_INTS * sizeof(int)); + + // Collect ker events for the watchdog poll after the launch loop. + cudaEvent_t ker_events[GGML_CUDA_MAX_DEVICES]; +#endif + // Insert the kernel into each GPU's existing compute stream via events: - // record(app, compute_stream) — capture "upstream done" point - // wait(internal_stream, app) — internal stream defers until then + // record(app, compute_stream) — capture "upstream done" + // wait(internal_stream, app) — internal stream defers until then // launch kernel on internal_stream - // record(ker, internal_stream) — capture "kernel done" point - // wait(compute_stream, ker) — compute stream resumes after kernel + // record(ker, internal_stream) — capture "kernel done" + // wait(compute_stream, ker) — compute stream resumes after kernel for (int i = 0; i < n; ++i) { const int peer = 1 - i; // valid for n == 2 only ggml_cuda_set_device(p->devices[i]); @@ -370,6 +611,19 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); +#ifdef GGML_CUDA_AR_WATCHDOG + ggml_cuda_ar_f32_kernel_dbg<<streams[i]>>>( + static_cast(tensors[i]->data), + static_cast(tensors[i]->data), + p->host_buf[i], + p->host_buf[peer], + static_cast(ne), + ggml_cuda_ar_arrival_ptr(p, slot, i), + ggml_cuda_ar_arrival_ptr(p, slot, peer), + p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS, + p->wdog_max_spin, + i); +#else ggml_cuda_ar_f32_kernel<<streams[i]>>>( static_cast(tensors[i]->data), static_cast(tensors[i]->data), @@ -378,11 +632,22 @@ bool ggml_cuda_ar_allreduce( static_cast(ne), ggml_cuda_ar_arrival_ptr(p, slot, i), ggml_cuda_ar_arrival_ptr(p, slot, peer)); +#endif CUDA_CHECK(cudaGetLastError()); CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); + +#ifdef GGML_CUDA_AR_WATCHDOG + ker_events[i] = ev.ker; +#endif } +#ifdef GGML_CUDA_AR_WATCHDOG + // Block the calling thread and poll until both kernels complete or the + // timeout expires. The GPU streams continue independently. + ggml_cuda_ar_watchdog_poll(p, slot, n, ker_events); +#endif + return true; } From 172cba2e8ccf9e12491a8f51fbfbd302f6b809ac Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 21 Apr 2026 23:25:13 -0700 Subject: [PATCH 07/81] ggml-cuda: fix intermittent AllReduce hang on Blackwell PCIe Add __threadfence_system() before the arrival signal write in signal_set to ensure D2H data is globally visible before the peer observes the arrival flag. Without this fence, the peer could enter Phase 3 host reads before the data had fully landed, causing an intermittent deadlock on RTX 5090 (Blackwell, PCIe-only). Also redesign the watchdog from a blocking dispatch-thread poll to a non-blocking background thread, eliminating the ~20ms per-slot latency the old design added. Verified: 30/30 soak test runs clean at ~50 t/s (previously ~1-in-15 hang rate). Co-Authored-By: Claude Sonnet 4.6 - INTERNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/src/ggml-cuda/allreduce.cu | 179 ++++++++++++++++++++------------ 1 file changed, 112 insertions(+), 67 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index eb239f71d43..81263ba0ec4 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -6,6 +6,9 @@ #ifdef GGML_CUDA_AR_WATCHDOG #include +#include +#include +#include #include #endif @@ -23,8 +26,9 @@ // --------------------------------------------------------------------------- static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { + __threadfence_system(); // ensure all prior writes (D2H data) are globally visible *(volatile int *)p = 1; - __threadfence_system(); + __threadfence_system(); // ensure the signal itself is globally visible } static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { @@ -32,7 +36,7 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { } // --------------------------------------------------------------------------- -// Single-phase AllReduce kernel — float32, 2 GPUs (production) +// Single-kernel AllReduce — float32, 2 GPUs (production) // // Both GPUs run this kernel simultaneously in independent streams. Each GPU: // @@ -234,8 +238,17 @@ static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; // 3 reserved static constexpr int GGML_CUDA_AR_DEBUG_INTS = 4; -// Host poll interval for the blocking watchdog loop. -static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 20; +// Background-thread poll interval. This has no effect on dispatch latency +// because the poll runs in a dedicated thread. +static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 100; + +// One work item posted to the background watchdog thread per dispatch. +struct ggml_cuda_ar_wdog_item { + int slot; + int n; + int devices[GGML_CUDA_MAX_DEVICES]; + cudaEvent_t ker_events[GGML_CUDA_MAX_DEVICES]; +}; #endif struct ggml_cuda_ar_event_slot { @@ -259,11 +272,18 @@ struct ggml_cuda_ar_pipeline { char * arrival; #ifdef GGML_CUDA_AR_WATCHDOG - // Pinned debug buffer written by the debug kernel, read by the host - // watchdog. Layout: debug_buf[rank * GGML_CUDA_AR_DEBUG_INTS + field]. + // Pinned debug buffer written by the debug kernel, read by the background + // watchdog thread. Layout: debug_buf[rank * GGML_CUDA_AR_DEBUG_INTS + field]. int * debug_buf; - int wdog_timeout_ms; // 0 = watchdog disabled (env: GGML_CUDA_AR_WATCHDOG) - int wdog_max_spin; // 0 = spin forever (env: GGML_CUDA_AR_MAX_SPIN) + int wdog_timeout_ms; // 0 = disabled (env: GGML_CUDA_AR_WATCHDOG) + int wdog_max_spin; // 0 = no limit (env: GGML_CUDA_AR_MAX_SPIN) + + // Background watchdog thread: polls kernel events without blocking dispatch. + std::mutex wdog_mtx; + std::condition_variable wdog_cv; + std::deque wdog_queue; + bool wdog_stop = false; + std::thread wdog_thr; #endif }; @@ -274,74 +294,78 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, } // --------------------------------------------------------------------------- -// Watchdog poll — blocks the calling thread, polling ker events and reading -// debug state from pinned host memory. Only active when wdog_timeout_ms > 0. +// Background watchdog thread — polls kernel events and debug state without +// blocking the dispatch path. One work item is posted per dispatch; the +// thread logs any slot that doesn't complete within wdog_timeout_ms. // -// Called after all kernels for the current slot have been queued. The GPU -// streams proceed independently; this function only observes via -// cudaEventQuery and volatile reads of debug_buf. -// -// Output is deliberately sparse on the fast path: nothing is logged when -// both kernels complete before the first poll tick. If a kernel is still -// running on the first tick, all subsequent ticks are logged (including the -// final "done" tick) so the log captures the full timeline. +// Output is sparse on the fast path: nothing is logged when a slot completes +// before the first poll tick. On slow or hanging slots every tick is logged, +// including the final "done" tick, to capture the full timeline. // --------------------------------------------------------------------------- #ifdef GGML_CUDA_AR_WATCHDOG -static void ggml_cuda_ar_watchdog_poll( - const ggml_cuda_ar_pipeline * p, - int slot, int n, - const cudaEvent_t * ker_events) { - if (p->wdog_timeout_ms <= 0) { - return; - } - - int elapsed_ms = 0; - bool observed_busy = false; - - while (elapsed_ms <= p->wdog_timeout_ms) { - // Query completion state of every GPU's kernel event. - bool all_done = true; - cudaError_t qstat[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) { - ggml_cuda_set_device(p->devices[i]); - qstat[i] = cudaEventQuery(ker_events[i]); - if (qstat[i] != cudaSuccess) { - all_done = false; +static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { + while (true) { + ggml_cuda_ar_wdog_item item; + { + std::unique_lock lk(p->wdog_mtx); + p->wdog_cv.wait(lk, [p] { return !p->wdog_queue.empty() || p->wdog_stop; }); + if (p->wdog_stop && p->wdog_queue.empty()) { + break; } + item = p->wdog_queue.front(); + p->wdog_queue.pop_front(); } - if (!all_done) { - observed_busy = true; + if (p->wdog_timeout_ms <= 0) { + continue; // watchdog disabled — drain queue, do nothing } - // Log on every tick that is either slow or the first "done" after slow. - if (!all_done || observed_busy) { - char msg[512]; - int pos = 0; - pos += snprintf(msg + pos, sizeof(msg) - pos, - "ggml_cuda_ar watchdog +%dms slot=%d:", - elapsed_ms, slot); - for (int i = 0; i < n; ++i) { - const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; - int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, slot, i); - int spin = *(volatile int *)&dbg[0]; - int last = *(volatile int *)&dbg[1]; - int wb = *(volatile int *)&dbg[2]; + int elapsed_ms = 0; + bool observed_busy = false; + + while (elapsed_ms <= p->wdog_timeout_ms) { + bool all_done = true; + cudaError_t qstat[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < item.n; ++i) { + ggml_cuda_set_device(item.devices[i]); + qstat[i] = cudaEventQuery(item.ker_events[i]); + if (qstat[i] != cudaSuccess) { + all_done = false; + } + } + + if (!all_done) { + observed_busy = true; + } + + if (!all_done || observed_busy) { + char msg[512]; + int pos = 0; pos += snprintf(msg + pos, sizeof(msg) - pos, - " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d]", - p->devices[i], - qstat[i] == cudaSuccess ? "done" : "busy", - arr, spin, last, wb); + "ggml_cuda_ar watchdog +%dms slot=%d:", + elapsed_ms, item.slot); + for (int i = 0; i < item.n; ++i) { + const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; + int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, item.slot, i); + int spin = *(volatile int *)&dbg[0]; + int last = *(volatile int *)&dbg[1]; + int wb = *(volatile int *)&dbg[2]; + pos += snprintf(msg + pos, sizeof(msg) - pos, + " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d]", + item.devices[i], + qstat[i] == cudaSuccess ? "done" : "busy", + arr, spin, last, wb); + } + GGML_LOG_WARN("%s\n", msg); } - GGML_LOG_WARN("%s\n", msg); - } - if (all_done) { - break; - } + if (all_done) { + break; + } - std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); - elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; + std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); + elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; + } } } #endif // GGML_CUDA_AR_WATCHDOG @@ -441,6 +465,9 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( GGML_LOG_INFO("%s: AR watchdog enabled — timeout=%dms max_spin=%d " "(set GGML_CUDA_AR_WATCHDOG= / GGML_CUDA_AR_MAX_SPIN= to adjust)\n", __func__, p->wdog_timeout_ms, p->wdog_max_spin); + + // Start the background polling thread. + p->wdog_thr = std::thread(ggml_cuda_ar_wdog_thread, p); } #endif @@ -528,6 +555,14 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaFreeHost(p->arrival); } #ifdef GGML_CUDA_AR_WATCHDOG + if (p->wdog_thr.joinable()) { + { + std::lock_guard lk(p->wdog_mtx); + p->wdog_stop = true; + } + p->wdog_cv.notify_one(); + p->wdog_thr.join(); + } if (p->debug_buf) { cudaFreeHost(p->debug_buf); } @@ -644,9 +679,19 @@ bool ggml_cuda_ar_allreduce( } #ifdef GGML_CUDA_AR_WATCHDOG - // Block the calling thread and poll until both kernels complete or the - // timeout expires. The GPU streams continue independently. - ggml_cuda_ar_watchdog_poll(p, slot, n, ker_events); + // Post this slot to the background watchdog thread — non-blocking. + { + ggml_cuda_ar_wdog_item item; + item.slot = slot; + item.n = n; + for (int i = 0; i < n; ++i) { + item.devices[i] = p->devices[i]; + item.ker_events[i] = ker_events[i]; + } + std::lock_guard lk(p->wdog_mtx); + p->wdog_queue.push_back(item); + } + p->wdog_cv.notify_one(); #endif return true; From 3b584e14b0d52579977e88bfdb25465634fec818 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 19:32:17 -0700 Subject: [PATCH 08/81] ggml-cuda: fix watchdog shutdown ordering and pipeline_free drain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Stop watchdog thread BEFORE destroying GPU resources (events, streams) to prevent polling destroyed handles → spurious "busy" readings - Add cudaStreamSynchronize in pipeline_free to drain in-flight kernels before freeing pinned host buffers they may still be reading - Sleep-first watchdog polling: no +0ms noise, only logs when a kernel is genuinely stuck past the poll interval - Check wdog_stop in both outer and inner loops so join() returns promptly instead of draining the entire queue - Add Phase 3 breadcrumbs to debug[3] for hang localization Co-Authored-By: Claude Sonnet 4.6 RNAL provider initialises the pipeline at comm_init time - Dispatch routes to ggml_cuda_ar_allreduce(); falls back to meta-backend CPU reduce for unsupported sizes or GPU counts (> 2) Current scope: 2 GPUs, FP32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/src/ggml-cuda/allreduce.cu | 135 ++++++++++++++++++++++---------- 1 file changed, 94 insertions(+), 41 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 81263ba0ec4..9ba6972f978 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -30,10 +30,17 @@ static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { *(volatile int *)p = 1; __threadfence_system(); // ensure the signal itself is globally visible } - +#if 1 static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; } +#else +static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int* addr) { + int val; + asm("ld.global.cv.b32 %0, [%1];" : "=r"(val) : "l"(addr)); + return val; +} +#endif // --------------------------------------------------------------------------- // Single-kernel AllReduce — float32, 2 GPUs (production) @@ -82,7 +89,12 @@ static __global__ void ggml_cuda_ar_f32_kernel( // Phase 2: thread 0 signals arrival, then spins for the peer. if (tid == 0) { ggml_cuda_ar_signal_set(arrival_mine); + + // ensure all GPUs have access to the arrival signal + __threadfence_system(); + while (ggml_cuda_ar_signal_get(arrival_other) == 0) { + //__threadfence_system(); __nanosleep(100); } } @@ -169,9 +181,13 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( int writeback = ggml_cuda_ar_signal_get(arrival_mine); debug[2] = writeback; + // ensure all GPUs have access to the arrival signal + __threadfence_system(); + int spin = 0; int last = 0; while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { + //printf("ggml_cuda_ar: just testing\n"); ++spin; // Periodically expose progress so the host watchdog can read it. if ((spin & 0xFFF) == 0) { @@ -198,8 +214,13 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( } // Phase 3: reduce (proceeds even after bailout; output will be wrong). + // debug[3] breadcrumbs: 1 = entering syncthreads, 2 = past syncthreads, + // 3 = past threadfence, 4 = phase 3 complete. + if (tid == 0) { debug[3] = 1; } __syncthreads(); + if (tid == 0) { debug[3] = 2; } __threadfence_system(); + if (tid == 0) { debug[3] = 3; } { const float4 * s4 = reinterpret_cast(sendbuf); @@ -214,6 +235,7 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; } } + if (tid == 0) { debug[3] = 4; } } #endif // GGML_CUDA_AR_WATCHDOG @@ -309,8 +331,8 @@ static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { { std::unique_lock lk(p->wdog_mtx); p->wdog_cv.wait(lk, [p] { return !p->wdog_queue.empty() || p->wdog_stop; }); - if (p->wdog_stop && p->wdog_queue.empty()) { - break; + if (p->wdog_stop) { + break; // exit immediately — don't drain remaining items } item = p->wdog_queue.front(); p->wdog_queue.pop_front(); @@ -320,10 +342,20 @@ static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { continue; // watchdog disabled — drain queue, do nothing } - int elapsed_ms = 0; - bool observed_busy = false; + int elapsed_ms = 0; while (elapsed_ms <= p->wdog_timeout_ms) { + // Sleep first — give the kernel time to complete before checking. + // On the fast path the kernel finishes during this sleep and we + // never log anything. + std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); + elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; + + // Check for shutdown during the poll loop so join() returns promptly. + if (p->wdog_stop) { + break; + } + bool all_done = true; cudaError_t qstat[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < item.n; ++i) { @@ -334,37 +366,30 @@ static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { } } - if (!all_done) { - observed_busy = true; - } - - if (!all_done || observed_busy) { - char msg[512]; - int pos = 0; - pos += snprintf(msg + pos, sizeof(msg) - pos, - "ggml_cuda_ar watchdog +%dms slot=%d:", - elapsed_ms, item.slot); - for (int i = 0; i < item.n; ++i) { - const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; - int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, item.slot, i); - int spin = *(volatile int *)&dbg[0]; - int last = *(volatile int *)&dbg[1]; - int wb = *(volatile int *)&dbg[2]; - pos += snprintf(msg + pos, sizeof(msg) - pos, - " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d]", - item.devices[i], - qstat[i] == cudaSuccess ? "done" : "busy", - arr, spin, last, wb); - } - GGML_LOG_WARN("%s\n", msg); - } - if (all_done) { break; } - std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); - elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; + // Kernel still running after the grace period — log diagnostics. + char msg[512]; + int pos = 0; + pos += snprintf(msg + pos, sizeof(msg) - pos, + "ggml_cuda_ar watchdog +%dms slot=%d:", + elapsed_ms, item.slot); + for (int i = 0; i < item.n; ++i) { + const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; + int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, item.slot, i); + int spin = *(volatile int *)&dbg[0]; + int last = *(volatile int *)&dbg[1]; + int wb = *(volatile int *)&dbg[2]; + int phase = *(volatile int *)&dbg[3]; + pos += snprintf(msg + pos, sizeof(msg) - pos, + " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d ph=%d]", + item.devices[i], + qstat[i] == cudaSuccess ? "done" : "busy", + arr, spin, last, wb, phase); + } + GGML_LOG_WARN("%s\n", msg); } } } @@ -379,6 +404,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( GGML_ASSERT(n_devices >= 2 && n_devices <= GGML_CUDA_MAX_DEVICES); auto * p = new ggml_cuda_ar_pipeline{}; + printf("ggml_cuda_ar_pipeline_init: p=%p\n", p); + p->n_devices = n_devices; p->buf_bytes = 0; p->call_count = 0; @@ -471,10 +498,13 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( } #endif +#if 0 // Warmup: run the kernel N times to pay first-use driver / PCIe / // page-mapping costs during model load and encourage the GPU clock // governor to boost before inference begins. if (n_devices == 2) { + printf("ggml_cuda_ar_pipeline_init warmup\n"); + constexpr int WARMUP_ITERS = 64; constexpr size_t WARMUP_COUNT = 8192; // 32 KB of fp32 constexpr size_t WARMUP_BYTES = WARMUP_COUNT * sizeof(float); @@ -522,18 +552,48 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( cudaFree(dev_buf[i]); } } + + printf("ggml_cuda_ar_pipeline_init warmup finished\n"); } - +#endif GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", __func__, n_devices, max_bytes >> 10); + + printf("ggml_cuda_ar_pipeline_init finished\n"); + return p; } void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { + printf("ggml_cuda_ar_pipeline_free: p=%p\n", p); if (!p) { return; } + +#ifdef GGML_CUDA_AR_WATCHDOG + // Stop the watchdog thread FIRST, before destroying any GPU resources. + // Otherwise it polls cudaEventQuery on destroyed events → spurious "busy." + if (p->wdog_thr.joinable()) { + printf("ggml_cuda_ar_pipeline_free stopping watchdog\n"); + { + std::lock_guard lk(p->wdog_mtx); + p->wdog_stop = true; + } + p->wdog_cv.notify_one(); + p->wdog_thr.join(); + printf("ggml_cuda_ar_pipeline_free watchdog joined\n"); + } +#endif + + // Drain all in-flight kernels before tearing down resources. + for (int i = 0; i < p->n_devices; ++i) { + if (p->streams[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaStreamSynchronize(p->streams[i]); + } + } + for (int i = 0; i < p->n_devices; ++i) { if (p->host_buf[i]) { cudaFreeHost(p->host_buf[i]); @@ -555,14 +615,6 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaFreeHost(p->arrival); } #ifdef GGML_CUDA_AR_WATCHDOG - if (p->wdog_thr.joinable()) { - { - std::lock_guard lk(p->wdog_mtx); - p->wdog_stop = true; - } - p->wdog_cv.notify_one(); - p->wdog_thr.join(); - } if (p->debug_buf) { cudaFreeHost(p->debug_buf); } @@ -578,6 +630,7 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, ggml_tensor ** tensors) { + //printf("ggml_cuda_ar_allreduce\n"); GGML_ASSERT(p != nullptr); const int n = p->n_devices; From 860ee2a8a4db005f2f9eb83b91ea5bb7b6b5bb1a Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 19:54:03 -0700 Subject: [PATCH 09/81] ggml-cuda: replace event-based watchdog with per-GPU ring buffer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completely rework the GGML_CUDA_AR_WATCHDOG system: - Replace the shared debug_buf + event-polling + queue design with per-GPU ring buffers in pinned host memory - Kernel writes a debug record only on spin-limit bailout: claims a ring slot via atomicAdd (single-GPU host atomics work on RTX 5090), writes fields, fences, sets completion flag, then all threads exit - Watchdog thread simply polls ring head counters every 1ms and prints any new complete records — no CUDA event queries, no mutex, no queue - Zero overhead on the dispatch path (no queue posting, no memset) - Watchdog shutdown returns within ~1ms (atomic bool, no drain) - On bailout the kernel skips Phase 3 entirely and exits cleanly Verified: 20/20 prefill soak test clean at ~1112 t/s, no hangs. Co-Authored-By: Claude Sonnet 4.6 P32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/src/ggml-cuda/allreduce.cu | 349 ++++++++++++-------------------- 1 file changed, 128 insertions(+), 221 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 9ba6972f978..c65449c490f 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -5,10 +5,8 @@ #include #ifdef GGML_CUDA_AR_WATCHDOG +#include #include -#include -#include -#include #include #endif @@ -30,17 +28,9 @@ static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { *(volatile int *)p = 1; __threadfence_system(); // ensure the signal itself is globally visible } -#if 1 static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; } -#else -static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int* addr) { - int val; - asm("ld.global.cv.b32 %0, [%1];" : "=r"(val) : "l"(addr)); - return val; -} -#endif // --------------------------------------------------------------------------- // Single-kernel AllReduce — float32, 2 GPUs (production) @@ -122,23 +112,39 @@ static __global__ void ggml_cuda_ar_f32_kernel( // --------------------------------------------------------------------------- // Watchdog debug variant — compiled only when GGML_CUDA_AR_WATCHDOG is defined. // -// Adds three extra parameters to the kernel and instruments Phase 2: -// -// debug[0] spin iteration count, updated every ~4096 iterations -// (-1 written on max_spin bailout as a sentinel) -// debug[1] last value of arrival_other observed during the spin -// debug[2] readback of arrival_mine immediately after signal_set; -// should always be 1 — if 0, the write did not reach host memory -// debug[3] reserved (always 0) +// Identical to the production kernel except Phase 2 has a spin limit +// (max_spin). If the limit is reached the kernel writes a debug record to +// a per-GPU ring buffer in pinned host memory, then bails out — all threads +// exit the kernel immediately (Phase 3 is skipped). // -// max_spin if > 0, the spin bails out after this many iterations and the -// kernel logs via printf before proceeding to Phase 3 with stale -// data. Output will be numerically wrong, but the kernel exits -// rather than hanging, which is useful for post-mortem logging. -// -// rank this GPU's rank within the communicator (for printf output) +// The ring slot is claimed with atomicAdd on the ring head counter. Host +// memory atomics work for a single GPU on RTX 5090 (just not cross-GPU). +// After writing the record fields the kernel issues __threadfence_system() +// and then sets the completion flag so the host watchdog thread can safely +// read the record. // --------------------------------------------------------------------------- #ifdef GGML_CUDA_AR_WATCHDOG + +// One debug record written by the kernel on spin-limit bailout. +struct ggml_cuda_ar_debug_record { + int rank; // GPU rank (0 or 1) + int slot; // AllReduce pool slot + int spin_count; // spins before bailout + int arrival_mine; // readback of own arrival flag after signal_set + int arrival_other; // last value of peer's arrival flag + int count; // element count of the AllReduce call + int complete; // 1 = record fully written (set last, after fence) +}; + +static constexpr int GGML_CUDA_AR_RING_SIZE = 64; + +// Per-GPU ring buffer in pinned host memory. head is incremented by the +// GPU via atomicAdd; records[] is written by the GPU and read by the host. +struct ggml_cuda_ar_debug_ring { + int head; // next slot to write (GPU atomicAdd) + ggml_cuda_ar_debug_record records[GGML_CUDA_AR_RING_SIZE]; +}; + static __global__ void ggml_cuda_ar_f32_kernel_dbg( const float * __restrict__ sendbuf, float * __restrict__ recvbuf, @@ -147,15 +153,21 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( int count, int * arrival_mine, int * arrival_other, - int * debug, + ggml_cuda_ar_debug_ring * ring, int max_spin, - int rank) { + int rank, + int ar_slot) { + + __shared__ int bail; const int tid = threadIdx.x; const int nt = blockDim.x; const int count4 = count >> 2; const int tail = count4 << 2; + if (tid == 0) { bail = 0; } + __syncthreads(); + // Phase 1: D2H copy (identical to production kernel). { const float4 * s4 = reinterpret_cast(sendbuf); @@ -175,53 +187,44 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( if (tid == 0) { ggml_cuda_ar_signal_set(arrival_mine); - // Readback: can this GPU see its own signal? If writeback == 0 the - // write did not reach host-visible memory, which would explain why - // the peer never observes arrival. int writeback = ggml_cuda_ar_signal_get(arrival_mine); - debug[2] = writeback; - - // ensure all GPUs have access to the arrival signal - __threadfence_system(); int spin = 0; int last = 0; while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { - //printf("ggml_cuda_ar: just testing\n"); ++spin; - // Periodically expose progress so the host watchdog can read it. - if ((spin & 0xFFF) == 0) { - debug[0] = spin; - debug[1] = last; - } if (max_spin > 0 && spin >= max_spin) { - debug[0] = -1; // bailout sentinel - debug[1] = last; - printf("ggml_cuda_ar: rank=%d BAILOUT after %d spins; " - "writeback_mine=%d arrival_other=%d " - "(pm=%p po=%p)\n", - rank, spin, writeback, last, - (void *)arrival_mine, (void *)arrival_other); + // Acquire a ring slot via atomicAdd (single-GPU host atomics OK). + int ri = atomicAdd(&ring->head, 1) % GGML_CUDA_AR_RING_SIZE; + ggml_cuda_ar_debug_record * rec = &ring->records[ri]; + + rec->rank = rank; + rec->slot = ar_slot; + rec->spin_count = spin; + rec->arrival_mine = writeback; + rec->arrival_other = last; + rec->count = count; + + __threadfence_system(); // ensure fields visible before completion flag + rec->complete = 1; + __threadfence_system(); // ensure completion flag visible to host + + bail = 1; break; } __nanosleep(100); } - // Write final spin count, unless we already wrote the bailout sentinel. - if (debug[0] != -1) { - debug[0] = spin; - debug[1] = last; - } } - // Phase 3: reduce (proceeds even after bailout; output will be wrong). - // debug[3] breadcrumbs: 1 = entering syncthreads, 2 = past syncthreads, - // 3 = past threadfence, 4 = phase 3 complete. - if (tid == 0) { debug[3] = 1; } __syncthreads(); - if (tid == 0) { debug[3] = 2; } + if (bail) { + return; // all threads exit — skip Phase 3 + } + + // Broadcast "peer has arrived" and acquire peer's host_other writes. __threadfence_system(); - if (tid == 0) { debug[3] = 3; } + // Phase 3: reduce. { const float4 * s4 = reinterpret_cast(sendbuf); const float4 * o4 = reinterpret_cast(host_other); @@ -235,7 +238,6 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; } } - if (tid == 0) { debug[3] = 4; } } #endif // GGML_CUDA_AR_WATCHDOG @@ -253,24 +255,8 @@ static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; #ifdef GGML_CUDA_AR_WATCHDOG -// Ints per device in the debug buffer. Layout (index → meaning): -// 0 spin count, updated every ~4096 iterations (-1 = bailed out) -// 1 last arrival_other value observed during the spin -// 2 readback of arrival_mine after signal_set (should always be 1) -// 3 reserved -static constexpr int GGML_CUDA_AR_DEBUG_INTS = 4; - -// Background-thread poll interval. This has no effect on dispatch latency -// because the poll runs in a dedicated thread. -static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 100; - -// One work item posted to the background watchdog thread per dispatch. -struct ggml_cuda_ar_wdog_item { - int slot; - int n; - int devices[GGML_CUDA_MAX_DEVICES]; - cudaEvent_t ker_events[GGML_CUDA_MAX_DEVICES]; -}; +// Watchdog poll interval in milliseconds. +static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 1; #endif struct ggml_cuda_ar_event_slot { @@ -294,18 +280,12 @@ struct ggml_cuda_ar_pipeline { char * arrival; #ifdef GGML_CUDA_AR_WATCHDOG - // Pinned debug buffer written by the debug kernel, read by the background - // watchdog thread. Layout: debug_buf[rank * GGML_CUDA_AR_DEBUG_INTS + field]. - int * debug_buf; - int wdog_timeout_ms; // 0 = disabled (env: GGML_CUDA_AR_WATCHDOG) - int wdog_max_spin; // 0 = no limit (env: GGML_CUDA_AR_MAX_SPIN) - - // Background watchdog thread: polls kernel events without blocking dispatch. - std::mutex wdog_mtx; - std::condition_variable wdog_cv; - std::deque wdog_queue; - bool wdog_stop = false; - std::thread wdog_thr; + // Per-GPU debug ring buffers in pinned host memory. Written by the debug + // kernel on spin-limit bailout, read by the background watchdog thread. + ggml_cuda_ar_debug_ring * debug_ring[GGML_CUDA_MAX_DEVICES]; + int wdog_max_spin; // 0 = no limit (env: GGML_CUDA_AR_MAX_SPIN) + std::atomic wdog_stop{false}; + std::thread wdog_thr; #endif }; @@ -316,81 +296,39 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, } // --------------------------------------------------------------------------- -// Background watchdog thread — polls kernel events and debug state without -// blocking the dispatch path. One work item is posted per dispatch; the -// thread logs any slot that doesn't complete within wdog_timeout_ms. -// -// Output is sparse on the fast path: nothing is logged when a slot completes -// before the first poll tick. On slow or hanging slots every tick is logged, -// including the final "done" tick, to capture the full timeline. +// Background watchdog thread — monitors per-GPU debug ring buffers for new +// bailout records. The kernel writes a record when it hits the spin limit; +// this thread polls the ring head counters every 1ms and prints any new +// complete records. Zero overhead on the dispatch path (no queue, no events). // --------------------------------------------------------------------------- #ifdef GGML_CUDA_AR_WATCHDOG static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { - while (true) { - ggml_cuda_ar_wdog_item item; - { - std::unique_lock lk(p->wdog_mtx); - p->wdog_cv.wait(lk, [p] { return !p->wdog_queue.empty() || p->wdog_stop; }); - if (p->wdog_stop) { - break; // exit immediately — don't drain remaining items - } - item = p->wdog_queue.front(); - p->wdog_queue.pop_front(); - } - - if (p->wdog_timeout_ms <= 0) { - continue; // watchdog disabled — drain queue, do nothing - } - - int elapsed_ms = 0; - - while (elapsed_ms <= p->wdog_timeout_ms) { - // Sleep first — give the kernel time to complete before checking. - // On the fast path the kernel finishes during this sleep and we - // never log anything. - std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); - elapsed_ms += GGML_CUDA_AR_WDOG_POLL_MS; - - // Check for shutdown during the poll loop so join() returns promptly. - if (p->wdog_stop) { - break; - } - - bool all_done = true; - cudaError_t qstat[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < item.n; ++i) { - ggml_cuda_set_device(item.devices[i]); - qstat[i] = cudaEventQuery(item.ker_events[i]); - if (qstat[i] != cudaSuccess) { - all_done = false; + int last_seen[GGML_CUDA_MAX_DEVICES] = {}; + + while (!p->wdog_stop.load(std::memory_order_relaxed)) { + for (int i = 0; i < p->n_devices; ++i) { + ggml_cuda_ar_debug_ring * ring = p->debug_ring[i]; + if (!ring) { continue; } + + int head = *(volatile int *)&ring->head; + while (last_seen[i] < head) { + int ri = last_seen[i] % GGML_CUDA_AR_RING_SIZE; + const ggml_cuda_ar_debug_record * rec = &ring->records[ri]; + + // Wait for the completion flag (kernel writes it last after fence). + if (*(volatile int *)&rec->complete) { + GGML_LOG_WARN("ggml_cuda_ar BAILOUT: gpu%d rank=%d slot=%d " + "spins=%d arrival_mine=%d arrival_other=%d count=%d\n", + p->devices[i], rec->rank, rec->slot, + rec->spin_count, rec->arrival_mine, + rec->arrival_other, rec->count); + last_seen[i]++; + } else { + break; // record not yet complete — check again next poll } } - - if (all_done) { - break; - } - - // Kernel still running after the grace period — log diagnostics. - char msg[512]; - int pos = 0; - pos += snprintf(msg + pos, sizeof(msg) - pos, - "ggml_cuda_ar watchdog +%dms slot=%d:", - elapsed_ms, item.slot); - for (int i = 0; i < item.n; ++i) { - const int * dbg = p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS; - int arr = *(volatile int *)ggml_cuda_ar_arrival_ptr(p, item.slot, i); - int spin = *(volatile int *)&dbg[0]; - int last = *(volatile int *)&dbg[1]; - int wb = *(volatile int *)&dbg[2]; - int phase = *(volatile int *)&dbg[3]; - pos += snprintf(msg + pos, sizeof(msg) - pos, - " gpu%d[%s arr=%d spin=%d lastOther=%d wbMine=%d ph=%d]", - item.devices[i], - qstat[i] == cudaSuccess ? "done" : "busy", - arr, spin, last, wb, phase); - } - GGML_LOG_WARN("%s\n", msg); } + std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); } } #endif // GGML_CUDA_AR_WATCHDOG @@ -404,8 +342,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( GGML_ASSERT(n_devices >= 2 && n_devices <= GGML_CUDA_MAX_DEVICES); auto * p = new ggml_cuda_ar_pipeline{}; - printf("ggml_cuda_ar_pipeline_init: p=%p\n", p); - p->n_devices = n_devices; p->buf_bytes = 0; p->call_count = 0; @@ -417,9 +353,10 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( p->ev_pool[i] = nullptr; } #ifdef GGML_CUDA_AR_WATCHDOG - p->debug_buf = nullptr; - p->wdog_timeout_ms = 0; - p->wdog_max_spin = 0; + for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { + p->debug_ring[i] = nullptr; + } + p->wdog_max_spin = 0; #endif // Per-device streams and event pools. @@ -473,27 +410,29 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( } #ifdef GGML_CUDA_AR_WATCHDOG - // Debug buffer: written by the instrumented kernel, polled by the host. + // Per-GPU debug ring buffers: written by the kernel on spin-limit bailout, + // polled by the background watchdog thread. Each ring is pinned host + // memory accessed only by its owning GPU (single-GPU host atomics OK). { - const size_t dbg_bytes = (size_t)n_devices * GGML_CUDA_AR_DEBUG_INTS * sizeof(int); - if (cudaHostAlloc(reinterpret_cast(&p->debug_buf), dbg_bytes, - cudaHostAllocPortable) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for debug buffer failed (%zu bytes)\n", - __func__, dbg_bytes); - ggml_cuda_ar_pipeline_free(p); - return nullptr; + for (int i = 0; i < n_devices; ++i) { + if (cudaHostAlloc(reinterpret_cast(&p->debug_ring[i]), + sizeof(ggml_cuda_ar_debug_ring), + cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for debug ring failed on device %d\n", + __func__, p->devices[i]); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + memset(p->debug_ring[i], 0, sizeof(ggml_cuda_ar_debug_ring)); } - memset(p->debug_buf, 0, dbg_bytes); - const char * wdog_env = getenv("GGML_CUDA_AR_WATCHDOG"); const char * spin_env = getenv("GGML_CUDA_AR_MAX_SPIN"); - p->wdog_timeout_ms = (wdog_env && wdog_env[0]) ? atoi(wdog_env) : 0; - p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; - GGML_LOG_INFO("%s: AR watchdog enabled — timeout=%dms max_spin=%d " - "(set GGML_CUDA_AR_WATCHDOG= / GGML_CUDA_AR_MAX_SPIN= to adjust)\n", - __func__, p->wdog_timeout_ms, p->wdog_max_spin); + p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; + GGML_LOG_INFO("%s: AR watchdog enabled — max_spin=%d " + "(set GGML_CUDA_AR_MAX_SPIN= to adjust)\n", + __func__, p->wdog_max_spin); - // Start the background polling thread. + p->wdog_stop.store(false); p->wdog_thr = std::thread(ggml_cuda_ar_wdog_thread, p); } #endif @@ -560,29 +499,20 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( "%zu KB staging per GPU\n", __func__, n_devices, max_bytes >> 10); - printf("ggml_cuda_ar_pipeline_init finished\n"); - return p; } void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { - printf("ggml_cuda_ar_pipeline_free: p=%p\n", p); if (!p) { return; } #ifdef GGML_CUDA_AR_WATCHDOG - // Stop the watchdog thread FIRST, before destroying any GPU resources. - // Otherwise it polls cudaEventQuery on destroyed events → spurious "busy." + // Stop the watchdog thread first — it only reads pinned host memory, + // no GPU resources, so this is safe and returns within ~1ms. + p->wdog_stop.store(true); if (p->wdog_thr.joinable()) { - printf("ggml_cuda_ar_pipeline_free stopping watchdog\n"); - { - std::lock_guard lk(p->wdog_mtx); - p->wdog_stop = true; - } - p->wdog_cv.notify_one(); p->wdog_thr.join(); - printf("ggml_cuda_ar_pipeline_free watchdog joined\n"); } #endif @@ -615,8 +545,10 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaFreeHost(p->arrival); } #ifdef GGML_CUDA_AR_WATCHDOG - if (p->debug_buf) { - cudaFreeHost(p->debug_buf); + for (int i = 0; i < p->n_devices; ++i) { + if (p->debug_ring[i]) { + cudaFreeHost(p->debug_ring[i]); + } } #endif delete p; @@ -676,13 +608,6 @@ bool ggml_cuda_ar_allreduce( *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; } -#ifdef GGML_CUDA_AR_WATCHDOG - // Clear per-device debug state so the watchdog reads reflect only this call. - memset(p->debug_buf, 0, (size_t)n * GGML_CUDA_AR_DEBUG_INTS * sizeof(int)); - - // Collect ker events for the watchdog poll after the launch loop. - cudaEvent_t ker_events[GGML_CUDA_MAX_DEVICES]; -#endif // Insert the kernel into each GPU's existing compute stream via events: // record(app, compute_stream) — capture "upstream done" @@ -708,9 +633,10 @@ bool ggml_cuda_ar_allreduce( static_cast(ne), ggml_cuda_ar_arrival_ptr(p, slot, i), ggml_cuda_ar_arrival_ptr(p, slot, peer), - p->debug_buf + i * GGML_CUDA_AR_DEBUG_INTS, + p->debug_ring[i], p->wdog_max_spin, - i); + i, + slot); #else ggml_cuda_ar_f32_kernel<<streams[i]>>>( static_cast(tensors[i]->data), @@ -726,26 +652,7 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); -#ifdef GGML_CUDA_AR_WATCHDOG - ker_events[i] = ev.ker; -#endif } -#ifdef GGML_CUDA_AR_WATCHDOG - // Post this slot to the background watchdog thread — non-blocking. - { - ggml_cuda_ar_wdog_item item; - item.slot = slot; - item.n = n; - for (int i = 0; i < n; ++i) { - item.devices[i] = p->devices[i]; - item.ker_events[i] = ker_events[i]; - } - std::lock_guard lk(p->wdog_mtx); - p->wdog_queue.push_back(item); - } - p->wdog_cv.notify_one(); -#endif - return true; } From 433c35a2d43ead3a0453d60c7bd3b763ec09f54b Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 20:58:28 -0700 Subject: [PATCH 10/81] fix: normalize line endings to LF (undo Windows CRLF conversion) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five files were inadvertently converted to CRLF by the Windows development environment, causing every line to show as changed in diffs against master. Co-Authored-By: Claude Sonnet 4.6 imit bailout: claims a ring slot via atomicAdd (single-GPU host atomics work on RTX 5090), writes fields, fences, sets completion flag, then all threads exit - Watchdog thread simply polls ring head counters every 1ms and prints any new complete records — no CUDA event queries, no mutex, no queue - Zero overhead on the dispatch path (no queue posting, no memset) - Watchdog shutdown returns within ~1ms (atomic bool, no drain) - On bailout the kernel skips Phase 3 entirely and exits cleanly Verified: 20/20 prefill soak test clean at ~1112 t/s, no hangs. Co-Authored-By: Claude Sonnet 4.6 P32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/CMakeLists.txt | 1012 +-- ggml/cmake/FindNCCL.cmake | 152 +- ggml/src/ggml-cuda/CMakeLists.txt | 550 +- ggml/src/ggml-cuda/ggml-cuda.cu | 10968 ++++++++++++++-------------- tools/llama-bench/llama-bench.cpp | 4944 ++++++------- 5 files changed, 8813 insertions(+), 8813 deletions(-) diff --git a/ggml/CMakeLists.txt b/ggml/CMakeLists.txt index bbc7800ab89..820053a96c9 100644 --- a/ggml/CMakeLists.txt +++ b/ggml/CMakeLists.txt @@ -1,506 +1,506 @@ -cmake_minimum_required(VERSION 3.14...3.28) # for add_link_options and implicit target directories. - -project("ggml" C CXX ASM) - -### GGML Version -set(GGML_VERSION_MAJOR 0) -set(GGML_VERSION_MINOR 10) -set(GGML_VERSION_PATCH 0) -set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}") - -list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/") - -find_program(GIT_EXE NAMES git git.exe NO_CMAKE_FIND_ROOT_PATH) -if(GIT_EXE) - # Get current git commit hash - execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD - WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} - OUTPUT_VARIABLE GGML_BUILD_COMMIT - OUTPUT_STRIP_TRAILING_WHITESPACE - ERROR_QUIET - ) - - # Check if the working directory is dirty (i.e., has uncommitted changes) - execute_process(COMMAND ${GIT_EXE} diff-index --quiet HEAD -- . - WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} - RESULT_VARIABLE GGML_GIT_DIRTY - ERROR_QUIET - ) -endif() - -set(GGML_VERSION "${GGML_VERSION_BASE}") - -if(NOT GGML_BUILD_COMMIT) - set(GGML_BUILD_COMMIT "unknown") -endif() - -# Build the commit string with optional dirty flag -if(DEFINED GGML_GIT_DIRTY AND GGML_GIT_DIRTY EQUAL 1) - set(GGML_BUILD_COMMIT "${GGML_BUILD_COMMIT}-dirty") -endif() - -include(CheckIncludeFileCXX) - -set(CMAKE_EXPORT_COMPILE_COMMANDS ON) - -if (NOT XCODE AND NOT MSVC AND NOT CMAKE_BUILD_TYPE) - set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE) - set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo") -endif() - -if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) - set(GGML_STANDALONE ON) - - set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) - - # configure project version - # TODO -else() - set(GGML_STANDALONE OFF) - - if (NOT CMAKE_RUNTIME_OUTPUT_DIRECTORY) - set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) - endif() -endif() - -if (EMSCRIPTEN) - set(BUILD_SHARED_LIBS_DEFAULT OFF) - - option(GGML_WASM_SINGLE_FILE "ggml: embed WASM inside the generated ggml.js" ON) -else() - if (MINGW) - set(BUILD_SHARED_LIBS_DEFAULT OFF) - else() - set(BUILD_SHARED_LIBS_DEFAULT ON) - endif() -endif() - -# remove the lib prefix on win32 mingw -if (WIN32) - set(CMAKE_STATIC_LIBRARY_PREFIX "") - set(CMAKE_SHARED_LIBRARY_PREFIX "") - set(CMAKE_SHARED_MODULE_PREFIX "") -endif() - -option(BUILD_SHARED_LIBS "ggml: build shared libraries" ${BUILD_SHARED_LIBS_DEFAULT}) -option(GGML_BACKEND_DL "ggml: build backends as dynamic libraries (requires BUILD_SHARED_LIBS)" OFF) -set(GGML_BACKEND_DIR "" CACHE PATH "ggml: directory to load dynamic backends from (requires GGML_BACKEND_DL") - -# -# option list -# - -# TODO: mark all options as advanced when not GGML_STANDALONE - -if (APPLE) - set(GGML_METAL_DEFAULT ON) - set(GGML_BLAS_DEFAULT ON) - set(GGML_BLAS_VENDOR_DEFAULT "Apple") -else() - set(GGML_METAL_DEFAULT OFF) - set(GGML_BLAS_DEFAULT OFF) - set(GGML_BLAS_VENDOR_DEFAULT "Generic") -endif() - -if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) - message(STATUS "Setting GGML_NATIVE_DEFAULT to OFF") - set(GGML_NATIVE_DEFAULT OFF) -else() - set(GGML_NATIVE_DEFAULT ON) -endif() - -# defaults -if (NOT GGML_LLAMAFILE_DEFAULT) - set(GGML_LLAMAFILE_DEFAULT OFF) -endif() - -if (NOT GGML_CUDA_GRAPHS_DEFAULT) - set(GGML_CUDA_GRAPHS_DEFAULT OFF) -endif() - -# general -option(GGML_STATIC "ggml: static link libraries" OFF) -option(GGML_NATIVE "ggml: optimize the build for the current system" ${GGML_NATIVE_DEFAULT}) -option(GGML_LTO "ggml: enable link time optimization" OFF) -option(GGML_CCACHE "ggml: use ccache if available" ON) - -# debug -option(GGML_ALL_WARNINGS "ggml: enable all compiler warnings" ON) -option(GGML_ALL_WARNINGS_3RD_PARTY "ggml: enable all compiler warnings in 3rd party libs" OFF) -option(GGML_GPROF "ggml: enable gprof" OFF) - -# build -option(GGML_FATAL_WARNINGS "ggml: enable -Werror flag" OFF) - -# sanitizers -option(GGML_SANITIZE_THREAD "ggml: enable thread sanitizer" OFF) -option(GGML_SANITIZE_ADDRESS "ggml: enable address sanitizer" OFF) -option(GGML_SANITIZE_UNDEFINED "ggml: enable undefined sanitizer" OFF) - -# instruction set specific -if (GGML_NATIVE OR NOT GGML_NATIVE_DEFAULT) - set(INS_ENB OFF) -else() - set(INS_ENB ON) -endif() - -message(DEBUG "GGML_NATIVE : ${GGML_NATIVE}") -message(DEBUG "GGML_NATIVE_DEFAULT : ${GGML_NATIVE_DEFAULT}") -message(DEBUG "INS_ENB : ${INS_ENB}") - -option(GGML_CPU_HBM "ggml: use memkind for CPU HBM" OFF) -option(GGML_CPU_REPACK "ggml: use runtime weight conversion of Q4_0 to Q4_X_X" ON) -option(GGML_CPU_KLEIDIAI "ggml: use KleidiAI optimized kernels if applicable" OFF) -option(GGML_SSE42 "ggml: enable SSE 4.2" ${INS_ENB}) -option(GGML_AVX "ggml: enable AVX" ${INS_ENB}) -option(GGML_AVX_VNNI "ggml: enable AVX-VNNI" OFF) -option(GGML_AVX2 "ggml: enable AVX2" ${INS_ENB}) -option(GGML_BMI2 "ggml: enable BMI2" ${INS_ENB}) -option(GGML_AVX512 "ggml: enable AVX512F" OFF) -option(GGML_AVX512_VBMI "ggml: enable AVX512-VBMI" OFF) -option(GGML_AVX512_VNNI "ggml: enable AVX512-VNNI" OFF) -option(GGML_AVX512_BF16 "ggml: enable AVX512-BF16" OFF) -if (NOT MSVC) - # in MSVC F16C and FMA is implied with AVX2/AVX512 - option(GGML_FMA "ggml: enable FMA" ${INS_ENB}) - option(GGML_F16C "ggml: enable F16C" ${INS_ENB}) - # MSVC does not seem to support AMX - option(GGML_AMX_TILE "ggml: enable AMX-TILE" OFF) - option(GGML_AMX_INT8 "ggml: enable AMX-INT8" OFF) - option(GGML_AMX_BF16 "ggml: enable AMX-BF16" OFF) -endif() -option(GGML_LASX "ggml: enable lasx" ON) -option(GGML_LSX "ggml: enable lsx" ON) -option(GGML_RVV "ggml: enable rvv" ON) -option(GGML_RV_ZFH "ggml: enable riscv zfh" ON) -option(GGML_RV_ZVFH "ggml: enable riscv zvfh" ON) -option(GGML_RV_ZICBOP "ggml: enable riscv zicbop" ON) -option(GGML_RV_ZIHINTPAUSE "ggml: enable riscv zihintpause" ON) -option(GGML_RV_ZVFBFWMA "ggml: enable riscv zvfbfwma" OFF) -option(GGML_XTHEADVECTOR "ggml: enable xtheadvector" OFF) -option(GGML_VXE "ggml: enable vxe" ${GGML_NATIVE}) - -option(GGML_CPU_ALL_VARIANTS "ggml: build all variants of the CPU backend (requires GGML_BACKEND_DL)" OFF) -set(GGML_CPU_ARM_ARCH "" CACHE STRING "ggml: CPU architecture for ARM") -set(GGML_CPU_POWERPC_CPUTYPE "" CACHE STRING "ggml: CPU type for PowerPC") - -# ggml core -set(GGML_SCHED_MAX_COPIES "4" CACHE STRING "ggml: max input copies for pipeline parallelism") -option(GGML_CPU "ggml: enable CPU backend" ON) -option(GGML_SCHED_NO_REALLOC "ggml: disallow reallocations in ggml-alloc (for debugging)" OFF) - -# 3rd party libs / backends -option(GGML_ACCELERATE "ggml: enable Accelerate framework" ON) -option(GGML_BLAS "ggml: use BLAS" ${GGML_BLAS_DEFAULT}) -set(GGML_BLAS_VENDOR ${GGML_BLAS_VENDOR_DEFAULT} CACHE STRING - "ggml: BLAS library vendor") -option(GGML_LLAMAFILE "ggml: use LLAMAFILE" ${GGML_LLAMAFILE_DEFAULT}) - -option(GGML_CUDA "ggml: use CUDA" OFF) -option(GGML_MUSA "ggml: use MUSA" OFF) -option(GGML_CUDA_FORCE_MMQ "ggml: use mmq kernels instead of cuBLAS" OFF) -option(GGML_CUDA_FORCE_CUBLAS "ggml: always use cuBLAS instead of mmq kernels" OFF) -set (GGML_CUDA_PEER_MAX_BATCH_SIZE "128" CACHE STRING - "ggml: max. batch size for using peer access") -option(GGML_CUDA_NO_PEER_COPY "ggml: do not use peer to peer copies" OFF) -option(GGML_CUDA_NO_VMM "ggml: do not try to use CUDA VMM" OFF) -option(GGML_CUDA_FA "ggml: compile ggml FlashAttention CUDA kernels" ON) -option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashAttention" OFF) -option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) -option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) -option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) -option(GGML_CUDA_AR_WATCHDOG "ggml: enable internal AllReduce hang watchdog (debug)" OFF) -set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING - "ggml: cuda link binary compression mode; requires cuda 12.8+") -set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") - -option(GGML_HIP "ggml: use HIP" OFF) -option(GGML_HIP_GRAPHS "ggml: use HIP graph, experimental, slow" OFF) -option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF) -option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON) -option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF) -option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON) -option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF) -option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF) -option(GGML_MUSA_MUDNN_COPY "ggml: enable muDNN for accelerated copy" OFF) -option(GGML_VULKAN "ggml: use Vulkan" OFF) -option(GGML_VULKAN_CHECK_RESULTS "ggml: run Vulkan op checks" OFF) -option(GGML_VULKAN_DEBUG "ggml: enable Vulkan debug output" OFF) -option(GGML_VULKAN_MEMORY_DEBUG "ggml: enable Vulkan memory debug output" OFF) -option(GGML_VULKAN_SHADER_DEBUG_INFO "ggml: enable Vulkan shader debug info" OFF) -option(GGML_VULKAN_VALIDATE "ggml: enable Vulkan validation" OFF) -option(GGML_VULKAN_RUN_TESTS "ggml: run Vulkan tests" OFF) -option(GGML_WEBGPU "ggml: use WebGPU" OFF) -option(GGML_WEBGPU_DEBUG "ggml: enable WebGPU debug output" OFF) -option(GGML_WEBGPU_CPU_PROFILE "ggml: enable WebGPU profiling (CPU)" OFF) -option(GGML_WEBGPU_GPU_PROFILE "ggml: enable WebGPU profiling (GPU)" OFF) -option(GGML_WEBGPU_JSPI "ggml: use JSPI for WebGPU" ON) -option(GGML_ZDNN "ggml: use zDNN" OFF) -option(GGML_VIRTGPU "ggml: use the VirtGPU/Virglrenderer API Remoting frontend" OFF) -option(GGML_VIRTGPU_BACKEND "ggml: build the VirtGPU/Virglrenderer API Remoting backend" OFF) -option(GGML_METAL "ggml: use Metal" ${GGML_METAL_DEFAULT}) -option(GGML_METAL_NDEBUG "ggml: disable Metal debugging" OFF) -option(GGML_METAL_SHADER_DEBUG "ggml: compile Metal with -fno-fast-math" OFF) -option(GGML_METAL_EMBED_LIBRARY "ggml: embed Metal library" ${GGML_METAL}) -set (GGML_METAL_MACOSX_VERSION_MIN "" CACHE STRING - "ggml: metal minimum macOS version") -set (GGML_METAL_STD "" CACHE STRING "ggml: metal standard version (-std flag)") -option(GGML_OPENMP "ggml: use OpenMP" ON) -option(GGML_RPC "ggml: use RPC" OFF) -option(GGML_SYCL "ggml: use SYCL" OFF) -option(GGML_SYCL_F16 "ggml: use 16 bit floats for sycl calculations" OFF) -option(GGML_SYCL_GRAPH "ggml: enable graphs in the SYCL backend" ON) -option(GGML_SYCL_HOST_MEM_FALLBACK "ggml: allow host memory fallback in SYCL reorder (requires kernel 6.8+)" ON) -option(GGML_SYCL_DNN "ggml: enable oneDNN in the SYCL backend" ON) -set (GGML_SYCL_TARGET "INTEL" CACHE STRING - "ggml: sycl target device") -set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING - "ggml: sycl device architecture") - -option(GGML_OPENVINO "ggml: use OPENVINO" OFF) - -option(GGML_OPENCL "ggml: use OpenCL" OFF) -option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF) -option(GGML_OPENCL_EMBED_KERNELS "ggml: embed kernels" ON) -option(GGML_OPENCL_USE_ADRENO_KERNELS "ggml: use optimized kernels for Adreno" ON) -set (GGML_OPENCL_TARGET_VERSION "300" CACHE STRING - "ggml: OpenCL API version to target") - -option(GGML_HEXAGON "ggml: enable Hexagon backend" OFF) -set(GGML_HEXAGON_FP32_QUANTIZE_GROUP_SIZE 128 CACHE STRING "ggml: quantize group size (32, 64, or 128)") - -# toolchain for vulkan-shaders-gen -set (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN "" CACHE FILEPATH "ggml: toolchain file for vulkan-shaders-gen") - -option(GGML_ZENDNN "ggml: use ZenDNN" OFF) -option(ZENDNN_ROOT "ggml: path to ZenDNN installation" "") - -# extra artifacts -option(GGML_BUILD_TESTS "ggml: build tests" ${GGML_STANDALONE}) -option(GGML_BUILD_EXAMPLES "ggml: build examples" ${GGML_STANDALONE}) - -# -# dependencies -# - -set(CMAKE_C_STANDARD 11) -set(CMAKE_C_STANDARD_REQUIRED true) - -set(CMAKE_CXX_STANDARD 17) -set(CMAKE_CXX_STANDARD_REQUIRED true) - -set(THREADS_PREFER_PTHREAD_FLAG ON) - -find_package(Threads REQUIRED) - -include(GNUInstallDirs) - -# -# build the library -# - -add_subdirectory(src) - -# -# tests and examples -# - -if (GGML_BUILD_TESTS) - enable_testing() - add_subdirectory(tests) -endif () - -if (GGML_BUILD_EXAMPLES) - add_subdirectory(examples) -endif () - -# -# install -# - -include(CMakePackageConfigHelpers) - -# all public headers -set(GGML_PUBLIC_HEADERS - include/ggml.h - include/ggml-cpu.h - include/ggml-alloc.h - include/ggml-backend.h - include/ggml-blas.h - include/ggml-cann.h - include/ggml-cpp.h - include/ggml-cuda.h - include/ggml-opt.h - include/ggml-metal.h - include/ggml-rpc.h - include/ggml-virtgpu.h - include/ggml-sycl.h - include/ggml-vulkan.h - include/ggml-webgpu.h - include/ggml-zendnn.h - include/ggml-openvino.h - include/gguf.h) - -set_target_properties(ggml PROPERTIES PUBLIC_HEADER "${GGML_PUBLIC_HEADERS}") -#if (GGML_METAL) -# set_target_properties(ggml PROPERTIES RESOURCE "${CMAKE_CURRENT_SOURCE_DIR}/src/ggml-metal.metal") -#endif() -install(TARGETS ggml LIBRARY PUBLIC_HEADER) -install(TARGETS ggml-base LIBRARY) - -if (GGML_STANDALONE) - configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ggml.pc.in - ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc - @ONLY) - - install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc - DESTINATION share/pkgconfig) -endif() - -# -# Create CMake package -# - - - -# Capture variables prefixed with GGML_. - -set(variable_set_statements -" -####### Expanded from @GGML_VARIABLES_EXPANED@ by configure_package_config_file() ####### -####### Any changes to this file will be overwritten by the next CMake run ####### - -") - -set(GGML_SHARED_LIB ${BUILD_SHARED_LIBS}) - -get_cmake_property(all_variables VARIABLES) -foreach(variable_name IN LISTS all_variables) - if(variable_name MATCHES "^GGML_") - string(REPLACE ";" "\\;" - variable_value "${${variable_name}}") - - set(variable_set_statements - "${variable_set_statements}set(${variable_name} \"${variable_value}\")\n") - endif() -endforeach() - -set(GGML_VARIABLES_EXPANDED ${variable_set_statements}) - -# Create the CMake package and set install location. - -set(GGML_INSTALL_VERSION ${GGML_VERSION}) -set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header files") -set(GGML_LIB_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR} CACHE PATH "Location of library files") -set(GGML_BIN_INSTALL_DIR ${CMAKE_INSTALL_BINDIR} CACHE PATH "Location of binary files") - -configure_package_config_file( - ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in - ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake - INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml - PATH_VARS GGML_INCLUDE_INSTALL_DIR - GGML_LIB_INSTALL_DIR - GGML_BIN_INSTALL_DIR) - -write_basic_package_version_file( - ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake - VERSION ${GGML_INSTALL_VERSION} - COMPATIBILITY SameMajorVersion) - -target_compile_definitions(ggml-base PRIVATE - GGML_VERSION="${GGML_INSTALL_VERSION}" - GGML_COMMIT="${GGML_BUILD_COMMIT}" -) -message(STATUS "ggml version: ${GGML_INSTALL_VERSION}") -message(STATUS "ggml commit: ${GGML_BUILD_COMMIT}") - -install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake - ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake - DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml) - -if (MSVC) - set(MSVC_WARNING_FLAGS - /wd4005 # Macro redefinition - /wd4244 # Conversion from one type to another type, possible loss of data - /wd4267 # Conversion from 'size_t' to a smaller type, possible loss of data - /wd4305 # Conversion from 'type1' to 'type2', possible loss of data - /wd4566 # Conversion from 'char' to 'wchar_t', possible loss of data - /wd4996 # Disable POSIX deprecation warnings - /wd4702 # Unreachable code warnings - ) - set(MSVC_COMPILE_OPTIONS - "$<$:/utf-8>" - "$<$:/utf-8>" - ) - function(configure_msvc_target target_name) - if(TARGET ${target_name}) - target_compile_options(${target_name} PRIVATE ${MSVC_WARNING_FLAGS}) - target_compile_options(${target_name} PRIVATE ${MSVC_COMPILE_OPTIONS}) - endif() - endfunction() - - configure_msvc_target(ggml-base) - configure_msvc_target(ggml) - configure_msvc_target(ggml-cpu) - configure_msvc_target(ggml-cpu-x64) - configure_msvc_target(ggml-cpu-sse42) - configure_msvc_target(ggml-cpu-sandybridge) - # __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512 - # skipping ggml-cpu-ivybridge - # skipping ggml-cpu-piledriver - configure_msvc_target(ggml-cpu-haswell) - configure_msvc_target(ggml-cpu-skylakex) - configure_msvc_target(ggml-cpu-cannonlake) - configure_msvc_target(ggml-cpu-cascadelake) - configure_msvc_target(ggml-cpu-icelake) - # MSVC 2022 doesn't support BF16 intrinsics without `/arch:AVX10.1` ?! - # https://learn.microsoft.com/en-us/cpp/intrinsics/x64-amd64-intrinsics-list?view=msvc-170 - # https://learn.microsoft.com/en-us/cpp/build/reference/arch-x64?view=msvc-170 - # skipping ggml-cpu-cooperlake - # skipping ggml-cpu-zen4 - configure_msvc_target(ggml-cpu-alderlake) - # MSVC doesn't support AMX - # skipping ggml-cpu-sapphirerapids - - if (GGML_BUILD_EXAMPLES) - configure_msvc_target(common-ggml) - configure_msvc_target(common) - - configure_msvc_target(mnist-common) - configure_msvc_target(mnist-eval) - configure_msvc_target(mnist-train) - - configure_msvc_target(gpt-2-ctx) - configure_msvc_target(gpt-2-alloc) - configure_msvc_target(gpt-2-backend) - configure_msvc_target(gpt-2-sched) - configure_msvc_target(gpt-2-quantize) - configure_msvc_target(gpt-2-batched) - - configure_msvc_target(gpt-j) - configure_msvc_target(gpt-j-quantize) - - configure_msvc_target(magika) - configure_msvc_target(yolov3-tiny) - configure_msvc_target(sam) - - configure_msvc_target(simple-ctx) - configure_msvc_target(simple-backend) - endif() - - if (GGML_BUILD_TESTS) - configure_msvc_target(test-mul-mat) - configure_msvc_target(test-arange) - configure_msvc_target(test-backend-ops) - configure_msvc_target(test-cont) - configure_msvc_target(test-conv-transpose) - configure_msvc_target(test-conv-transpose-1d) - configure_msvc_target(test-conv1d) - configure_msvc_target(test-conv2d) - configure_msvc_target(test-conv2d-dw) - configure_msvc_target(test-customop) - configure_msvc_target(test-dup) - configure_msvc_target(test-opt) - configure_msvc_target(test-pool) - endif () -endif() +cmake_minimum_required(VERSION 3.14...3.28) # for add_link_options and implicit target directories. + +project("ggml" C CXX ASM) + +### GGML Version +set(GGML_VERSION_MAJOR 0) +set(GGML_VERSION_MINOR 10) +set(GGML_VERSION_PATCH 0) +set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}") + +list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/cmake/") + +find_program(GIT_EXE NAMES git git.exe NO_CMAKE_FIND_ROOT_PATH) +if(GIT_EXE) + # Get current git commit hash + execute_process(COMMAND ${GIT_EXE} rev-parse --short HEAD + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + OUTPUT_VARIABLE GGML_BUILD_COMMIT + OUTPUT_STRIP_TRAILING_WHITESPACE + ERROR_QUIET + ) + + # Check if the working directory is dirty (i.e., has uncommitted changes) + execute_process(COMMAND ${GIT_EXE} diff-index --quiet HEAD -- . + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR} + RESULT_VARIABLE GGML_GIT_DIRTY + ERROR_QUIET + ) +endif() + +set(GGML_VERSION "${GGML_VERSION_BASE}") + +if(NOT GGML_BUILD_COMMIT) + set(GGML_BUILD_COMMIT "unknown") +endif() + +# Build the commit string with optional dirty flag +if(DEFINED GGML_GIT_DIRTY AND GGML_GIT_DIRTY EQUAL 1) + set(GGML_BUILD_COMMIT "${GGML_BUILD_COMMIT}-dirty") +endif() + +include(CheckIncludeFileCXX) + +set(CMAKE_EXPORT_COMPILE_COMMANDS ON) + +if (NOT XCODE AND NOT MSVC AND NOT CMAKE_BUILD_TYPE) + set(CMAKE_BUILD_TYPE Release CACHE STRING "Build type" FORCE) + set_property(CACHE CMAKE_BUILD_TYPE PROPERTY STRINGS "Debug" "Release" "MinSizeRel" "RelWithDebInfo") +endif() + +if (CMAKE_SOURCE_DIR STREQUAL CMAKE_CURRENT_SOURCE_DIR) + set(GGML_STANDALONE ON) + + set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) + + # configure project version + # TODO +else() + set(GGML_STANDALONE OFF) + + if (NOT CMAKE_RUNTIME_OUTPUT_DIRECTORY) + set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin) + endif() +endif() + +if (EMSCRIPTEN) + set(BUILD_SHARED_LIBS_DEFAULT OFF) + + option(GGML_WASM_SINGLE_FILE "ggml: embed WASM inside the generated ggml.js" ON) +else() + if (MINGW) + set(BUILD_SHARED_LIBS_DEFAULT OFF) + else() + set(BUILD_SHARED_LIBS_DEFAULT ON) + endif() +endif() + +# remove the lib prefix on win32 mingw +if (WIN32) + set(CMAKE_STATIC_LIBRARY_PREFIX "") + set(CMAKE_SHARED_LIBRARY_PREFIX "") + set(CMAKE_SHARED_MODULE_PREFIX "") +endif() + +option(BUILD_SHARED_LIBS "ggml: build shared libraries" ${BUILD_SHARED_LIBS_DEFAULT}) +option(GGML_BACKEND_DL "ggml: build backends as dynamic libraries (requires BUILD_SHARED_LIBS)" OFF) +set(GGML_BACKEND_DIR "" CACHE PATH "ggml: directory to load dynamic backends from (requires GGML_BACKEND_DL") + +# +# option list +# + +# TODO: mark all options as advanced when not GGML_STANDALONE + +if (APPLE) + set(GGML_METAL_DEFAULT ON) + set(GGML_BLAS_DEFAULT ON) + set(GGML_BLAS_VENDOR_DEFAULT "Apple") +else() + set(GGML_METAL_DEFAULT OFF) + set(GGML_BLAS_DEFAULT OFF) + set(GGML_BLAS_VENDOR_DEFAULT "Generic") +endif() + +if (CMAKE_CROSSCOMPILING OR DEFINED ENV{SOURCE_DATE_EPOCH}) + message(STATUS "Setting GGML_NATIVE_DEFAULT to OFF") + set(GGML_NATIVE_DEFAULT OFF) +else() + set(GGML_NATIVE_DEFAULT ON) +endif() + +# defaults +if (NOT GGML_LLAMAFILE_DEFAULT) + set(GGML_LLAMAFILE_DEFAULT OFF) +endif() + +if (NOT GGML_CUDA_GRAPHS_DEFAULT) + set(GGML_CUDA_GRAPHS_DEFAULT OFF) +endif() + +# general +option(GGML_STATIC "ggml: static link libraries" OFF) +option(GGML_NATIVE "ggml: optimize the build for the current system" ${GGML_NATIVE_DEFAULT}) +option(GGML_LTO "ggml: enable link time optimization" OFF) +option(GGML_CCACHE "ggml: use ccache if available" ON) + +# debug +option(GGML_ALL_WARNINGS "ggml: enable all compiler warnings" ON) +option(GGML_ALL_WARNINGS_3RD_PARTY "ggml: enable all compiler warnings in 3rd party libs" OFF) +option(GGML_GPROF "ggml: enable gprof" OFF) + +# build +option(GGML_FATAL_WARNINGS "ggml: enable -Werror flag" OFF) + +# sanitizers +option(GGML_SANITIZE_THREAD "ggml: enable thread sanitizer" OFF) +option(GGML_SANITIZE_ADDRESS "ggml: enable address sanitizer" OFF) +option(GGML_SANITIZE_UNDEFINED "ggml: enable undefined sanitizer" OFF) + +# instruction set specific +if (GGML_NATIVE OR NOT GGML_NATIVE_DEFAULT) + set(INS_ENB OFF) +else() + set(INS_ENB ON) +endif() + +message(DEBUG "GGML_NATIVE : ${GGML_NATIVE}") +message(DEBUG "GGML_NATIVE_DEFAULT : ${GGML_NATIVE_DEFAULT}") +message(DEBUG "INS_ENB : ${INS_ENB}") + +option(GGML_CPU_HBM "ggml: use memkind for CPU HBM" OFF) +option(GGML_CPU_REPACK "ggml: use runtime weight conversion of Q4_0 to Q4_X_X" ON) +option(GGML_CPU_KLEIDIAI "ggml: use KleidiAI optimized kernels if applicable" OFF) +option(GGML_SSE42 "ggml: enable SSE 4.2" ${INS_ENB}) +option(GGML_AVX "ggml: enable AVX" ${INS_ENB}) +option(GGML_AVX_VNNI "ggml: enable AVX-VNNI" OFF) +option(GGML_AVX2 "ggml: enable AVX2" ${INS_ENB}) +option(GGML_BMI2 "ggml: enable BMI2" ${INS_ENB}) +option(GGML_AVX512 "ggml: enable AVX512F" OFF) +option(GGML_AVX512_VBMI "ggml: enable AVX512-VBMI" OFF) +option(GGML_AVX512_VNNI "ggml: enable AVX512-VNNI" OFF) +option(GGML_AVX512_BF16 "ggml: enable AVX512-BF16" OFF) +if (NOT MSVC) + # in MSVC F16C and FMA is implied with AVX2/AVX512 + option(GGML_FMA "ggml: enable FMA" ${INS_ENB}) + option(GGML_F16C "ggml: enable F16C" ${INS_ENB}) + # MSVC does not seem to support AMX + option(GGML_AMX_TILE "ggml: enable AMX-TILE" OFF) + option(GGML_AMX_INT8 "ggml: enable AMX-INT8" OFF) + option(GGML_AMX_BF16 "ggml: enable AMX-BF16" OFF) +endif() +option(GGML_LASX "ggml: enable lasx" ON) +option(GGML_LSX "ggml: enable lsx" ON) +option(GGML_RVV "ggml: enable rvv" ON) +option(GGML_RV_ZFH "ggml: enable riscv zfh" ON) +option(GGML_RV_ZVFH "ggml: enable riscv zvfh" ON) +option(GGML_RV_ZICBOP "ggml: enable riscv zicbop" ON) +option(GGML_RV_ZIHINTPAUSE "ggml: enable riscv zihintpause" ON) +option(GGML_RV_ZVFBFWMA "ggml: enable riscv zvfbfwma" OFF) +option(GGML_XTHEADVECTOR "ggml: enable xtheadvector" OFF) +option(GGML_VXE "ggml: enable vxe" ${GGML_NATIVE}) + +option(GGML_CPU_ALL_VARIANTS "ggml: build all variants of the CPU backend (requires GGML_BACKEND_DL)" OFF) +set(GGML_CPU_ARM_ARCH "" CACHE STRING "ggml: CPU architecture for ARM") +set(GGML_CPU_POWERPC_CPUTYPE "" CACHE STRING "ggml: CPU type for PowerPC") + +# ggml core +set(GGML_SCHED_MAX_COPIES "4" CACHE STRING "ggml: max input copies for pipeline parallelism") +option(GGML_CPU "ggml: enable CPU backend" ON) +option(GGML_SCHED_NO_REALLOC "ggml: disallow reallocations in ggml-alloc (for debugging)" OFF) + +# 3rd party libs / backends +option(GGML_ACCELERATE "ggml: enable Accelerate framework" ON) +option(GGML_BLAS "ggml: use BLAS" ${GGML_BLAS_DEFAULT}) +set(GGML_BLAS_VENDOR ${GGML_BLAS_VENDOR_DEFAULT} CACHE STRING + "ggml: BLAS library vendor") +option(GGML_LLAMAFILE "ggml: use LLAMAFILE" ${GGML_LLAMAFILE_DEFAULT}) + +option(GGML_CUDA "ggml: use CUDA" OFF) +option(GGML_MUSA "ggml: use MUSA" OFF) +option(GGML_CUDA_FORCE_MMQ "ggml: use mmq kernels instead of cuBLAS" OFF) +option(GGML_CUDA_FORCE_CUBLAS "ggml: always use cuBLAS instead of mmq kernels" OFF) +set (GGML_CUDA_PEER_MAX_BATCH_SIZE "128" CACHE STRING + "ggml: max. batch size for using peer access") +option(GGML_CUDA_NO_PEER_COPY "ggml: do not use peer to peer copies" OFF) +option(GGML_CUDA_NO_VMM "ggml: do not try to use CUDA VMM" OFF) +option(GGML_CUDA_FA "ggml: compile ggml FlashAttention CUDA kernels" ON) +option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashAttention" OFF) +option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) +option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) +option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) +option(GGML_CUDA_AR_WATCHDOG "ggml: enable internal AllReduce hang watchdog (debug)" OFF) +set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING + "ggml: cuda link binary compression mode; requires cuda 12.8+") +set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") + +option(GGML_HIP "ggml: use HIP" OFF) +option(GGML_HIP_GRAPHS "ggml: use HIP graph, experimental, slow" OFF) +option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF) +option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON) +option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF) +option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON) +option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF) +option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF) +option(GGML_MUSA_MUDNN_COPY "ggml: enable muDNN for accelerated copy" OFF) +option(GGML_VULKAN "ggml: use Vulkan" OFF) +option(GGML_VULKAN_CHECK_RESULTS "ggml: run Vulkan op checks" OFF) +option(GGML_VULKAN_DEBUG "ggml: enable Vulkan debug output" OFF) +option(GGML_VULKAN_MEMORY_DEBUG "ggml: enable Vulkan memory debug output" OFF) +option(GGML_VULKAN_SHADER_DEBUG_INFO "ggml: enable Vulkan shader debug info" OFF) +option(GGML_VULKAN_VALIDATE "ggml: enable Vulkan validation" OFF) +option(GGML_VULKAN_RUN_TESTS "ggml: run Vulkan tests" OFF) +option(GGML_WEBGPU "ggml: use WebGPU" OFF) +option(GGML_WEBGPU_DEBUG "ggml: enable WebGPU debug output" OFF) +option(GGML_WEBGPU_CPU_PROFILE "ggml: enable WebGPU profiling (CPU)" OFF) +option(GGML_WEBGPU_GPU_PROFILE "ggml: enable WebGPU profiling (GPU)" OFF) +option(GGML_WEBGPU_JSPI "ggml: use JSPI for WebGPU" ON) +option(GGML_ZDNN "ggml: use zDNN" OFF) +option(GGML_VIRTGPU "ggml: use the VirtGPU/Virglrenderer API Remoting frontend" OFF) +option(GGML_VIRTGPU_BACKEND "ggml: build the VirtGPU/Virglrenderer API Remoting backend" OFF) +option(GGML_METAL "ggml: use Metal" ${GGML_METAL_DEFAULT}) +option(GGML_METAL_NDEBUG "ggml: disable Metal debugging" OFF) +option(GGML_METAL_SHADER_DEBUG "ggml: compile Metal with -fno-fast-math" OFF) +option(GGML_METAL_EMBED_LIBRARY "ggml: embed Metal library" ${GGML_METAL}) +set (GGML_METAL_MACOSX_VERSION_MIN "" CACHE STRING + "ggml: metal minimum macOS version") +set (GGML_METAL_STD "" CACHE STRING "ggml: metal standard version (-std flag)") +option(GGML_OPENMP "ggml: use OpenMP" ON) +option(GGML_RPC "ggml: use RPC" OFF) +option(GGML_SYCL "ggml: use SYCL" OFF) +option(GGML_SYCL_F16 "ggml: use 16 bit floats for sycl calculations" OFF) +option(GGML_SYCL_GRAPH "ggml: enable graphs in the SYCL backend" ON) +option(GGML_SYCL_HOST_MEM_FALLBACK "ggml: allow host memory fallback in SYCL reorder (requires kernel 6.8+)" ON) +option(GGML_SYCL_DNN "ggml: enable oneDNN in the SYCL backend" ON) +set (GGML_SYCL_TARGET "INTEL" CACHE STRING + "ggml: sycl target device") +set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING + "ggml: sycl device architecture") + +option(GGML_OPENVINO "ggml: use OPENVINO" OFF) + +option(GGML_OPENCL "ggml: use OpenCL" OFF) +option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF) +option(GGML_OPENCL_EMBED_KERNELS "ggml: embed kernels" ON) +option(GGML_OPENCL_USE_ADRENO_KERNELS "ggml: use optimized kernels for Adreno" ON) +set (GGML_OPENCL_TARGET_VERSION "300" CACHE STRING + "ggml: OpenCL API version to target") + +option(GGML_HEXAGON "ggml: enable Hexagon backend" OFF) +set(GGML_HEXAGON_FP32_QUANTIZE_GROUP_SIZE 128 CACHE STRING "ggml: quantize group size (32, 64, or 128)") + +# toolchain for vulkan-shaders-gen +set (GGML_VULKAN_SHADERS_GEN_TOOLCHAIN "" CACHE FILEPATH "ggml: toolchain file for vulkan-shaders-gen") + +option(GGML_ZENDNN "ggml: use ZenDNN" OFF) +option(ZENDNN_ROOT "ggml: path to ZenDNN installation" "") + +# extra artifacts +option(GGML_BUILD_TESTS "ggml: build tests" ${GGML_STANDALONE}) +option(GGML_BUILD_EXAMPLES "ggml: build examples" ${GGML_STANDALONE}) + +# +# dependencies +# + +set(CMAKE_C_STANDARD 11) +set(CMAKE_C_STANDARD_REQUIRED true) + +set(CMAKE_CXX_STANDARD 17) +set(CMAKE_CXX_STANDARD_REQUIRED true) + +set(THREADS_PREFER_PTHREAD_FLAG ON) + +find_package(Threads REQUIRED) + +include(GNUInstallDirs) + +# +# build the library +# + +add_subdirectory(src) + +# +# tests and examples +# + +if (GGML_BUILD_TESTS) + enable_testing() + add_subdirectory(tests) +endif () + +if (GGML_BUILD_EXAMPLES) + add_subdirectory(examples) +endif () + +# +# install +# + +include(CMakePackageConfigHelpers) + +# all public headers +set(GGML_PUBLIC_HEADERS + include/ggml.h + include/ggml-cpu.h + include/ggml-alloc.h + include/ggml-backend.h + include/ggml-blas.h + include/ggml-cann.h + include/ggml-cpp.h + include/ggml-cuda.h + include/ggml-opt.h + include/ggml-metal.h + include/ggml-rpc.h + include/ggml-virtgpu.h + include/ggml-sycl.h + include/ggml-vulkan.h + include/ggml-webgpu.h + include/ggml-zendnn.h + include/ggml-openvino.h + include/gguf.h) + +set_target_properties(ggml PROPERTIES PUBLIC_HEADER "${GGML_PUBLIC_HEADERS}") +#if (GGML_METAL) +# set_target_properties(ggml PROPERTIES RESOURCE "${CMAKE_CURRENT_SOURCE_DIR}/src/ggml-metal.metal") +#endif() +install(TARGETS ggml LIBRARY PUBLIC_HEADER) +install(TARGETS ggml-base LIBRARY) + +if (GGML_STANDALONE) + configure_file(${CMAKE_CURRENT_SOURCE_DIR}/ggml.pc.in + ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc + @ONLY) + + install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml.pc + DESTINATION share/pkgconfig) +endif() + +# +# Create CMake package +# + + + +# Capture variables prefixed with GGML_. + +set(variable_set_statements +" +####### Expanded from @GGML_VARIABLES_EXPANED@ by configure_package_config_file() ####### +####### Any changes to this file will be overwritten by the next CMake run ####### + +") + +set(GGML_SHARED_LIB ${BUILD_SHARED_LIBS}) + +get_cmake_property(all_variables VARIABLES) +foreach(variable_name IN LISTS all_variables) + if(variable_name MATCHES "^GGML_") + string(REPLACE ";" "\\;" + variable_value "${${variable_name}}") + + set(variable_set_statements + "${variable_set_statements}set(${variable_name} \"${variable_value}\")\n") + endif() +endforeach() + +set(GGML_VARIABLES_EXPANDED ${variable_set_statements}) + +# Create the CMake package and set install location. + +set(GGML_INSTALL_VERSION ${GGML_VERSION}) +set(GGML_INCLUDE_INSTALL_DIR ${CMAKE_INSTALL_INCLUDEDIR} CACHE PATH "Location of header files") +set(GGML_LIB_INSTALL_DIR ${CMAKE_INSTALL_LIBDIR} CACHE PATH "Location of library files") +set(GGML_BIN_INSTALL_DIR ${CMAKE_INSTALL_BINDIR} CACHE PATH "Location of binary files") + +configure_package_config_file( + ${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-config.cmake.in + ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake + INSTALL_DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml + PATH_VARS GGML_INCLUDE_INSTALL_DIR + GGML_LIB_INSTALL_DIR + GGML_BIN_INSTALL_DIR) + +write_basic_package_version_file( + ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake + VERSION ${GGML_INSTALL_VERSION} + COMPATIBILITY SameMajorVersion) + +target_compile_definitions(ggml-base PRIVATE + GGML_VERSION="${GGML_INSTALL_VERSION}" + GGML_COMMIT="${GGML_BUILD_COMMIT}" +) +message(STATUS "ggml version: ${GGML_INSTALL_VERSION}") +message(STATUS "ggml commit: ${GGML_BUILD_COMMIT}") + +install(FILES ${CMAKE_CURRENT_BINARY_DIR}/ggml-config.cmake + ${CMAKE_CURRENT_BINARY_DIR}/ggml-version.cmake + DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/ggml) + +if (MSVC) + set(MSVC_WARNING_FLAGS + /wd4005 # Macro redefinition + /wd4244 # Conversion from one type to another type, possible loss of data + /wd4267 # Conversion from 'size_t' to a smaller type, possible loss of data + /wd4305 # Conversion from 'type1' to 'type2', possible loss of data + /wd4566 # Conversion from 'char' to 'wchar_t', possible loss of data + /wd4996 # Disable POSIX deprecation warnings + /wd4702 # Unreachable code warnings + ) + set(MSVC_COMPILE_OPTIONS + "$<$:/utf-8>" + "$<$:/utf-8>" + ) + function(configure_msvc_target target_name) + if(TARGET ${target_name}) + target_compile_options(${target_name} PRIVATE ${MSVC_WARNING_FLAGS}) + target_compile_options(${target_name} PRIVATE ${MSVC_COMPILE_OPTIONS}) + endif() + endfunction() + + configure_msvc_target(ggml-base) + configure_msvc_target(ggml) + configure_msvc_target(ggml-cpu) + configure_msvc_target(ggml-cpu-x64) + configure_msvc_target(ggml-cpu-sse42) + configure_msvc_target(ggml-cpu-sandybridge) + # __FMA__ and __F16C__ are not defined in MSVC, however they are implied with AVX2/AVX512 + # skipping ggml-cpu-ivybridge + # skipping ggml-cpu-piledriver + configure_msvc_target(ggml-cpu-haswell) + configure_msvc_target(ggml-cpu-skylakex) + configure_msvc_target(ggml-cpu-cannonlake) + configure_msvc_target(ggml-cpu-cascadelake) + configure_msvc_target(ggml-cpu-icelake) + # MSVC 2022 doesn't support BF16 intrinsics without `/arch:AVX10.1` ?! + # https://learn.microsoft.com/en-us/cpp/intrinsics/x64-amd64-intrinsics-list?view=msvc-170 + # https://learn.microsoft.com/en-us/cpp/build/reference/arch-x64?view=msvc-170 + # skipping ggml-cpu-cooperlake + # skipping ggml-cpu-zen4 + configure_msvc_target(ggml-cpu-alderlake) + # MSVC doesn't support AMX + # skipping ggml-cpu-sapphirerapids + + if (GGML_BUILD_EXAMPLES) + configure_msvc_target(common-ggml) + configure_msvc_target(common) + + configure_msvc_target(mnist-common) + configure_msvc_target(mnist-eval) + configure_msvc_target(mnist-train) + + configure_msvc_target(gpt-2-ctx) + configure_msvc_target(gpt-2-alloc) + configure_msvc_target(gpt-2-backend) + configure_msvc_target(gpt-2-sched) + configure_msvc_target(gpt-2-quantize) + configure_msvc_target(gpt-2-batched) + + configure_msvc_target(gpt-j) + configure_msvc_target(gpt-j-quantize) + + configure_msvc_target(magika) + configure_msvc_target(yolov3-tiny) + configure_msvc_target(sam) + + configure_msvc_target(simple-ctx) + configure_msvc_target(simple-backend) + endif() + + if (GGML_BUILD_TESTS) + configure_msvc_target(test-mul-mat) + configure_msvc_target(test-arange) + configure_msvc_target(test-backend-ops) + configure_msvc_target(test-cont) + configure_msvc_target(test-conv-transpose) + configure_msvc_target(test-conv-transpose-1d) + configure_msvc_target(test-conv1d) + configure_msvc_target(test-conv2d) + configure_msvc_target(test-conv2d-dw) + configure_msvc_target(test-customop) + configure_msvc_target(test-dup) + configure_msvc_target(test-opt) + configure_msvc_target(test-pool) + endif () +endif() diff --git a/ggml/cmake/FindNCCL.cmake b/ggml/cmake/FindNCCL.cmake index 9cba6e882c0..674ecddd25f 100644 --- a/ggml/cmake/FindNCCL.cmake +++ b/ggml/cmake/FindNCCL.cmake @@ -1,76 +1,76 @@ -# cmake/FindNCCL.cmake - -# NVIDIA does not distribute CMake files with NCCL, therefore use this file to find it instead. -# -# Inputs: -# NCCL_ROOT — root of an NCCL installation or source build tree -# NCCL_STATIC — if ON, prefer the static library and search cmake/lib/Release (or Debug); -# if OFF (default), prefer the shared/import library and search cmake/src/Release - -if(NCCL_STATIC) - # cmake source-build layout: cmake/lib//nccl_static.lib (or nccl.lib) - set(_nccl_lib_names nccl_static nccl) - set(_nccl_extra_lib_hints - "${NCCL_ROOT}/cmake/lib/Release" - "${NCCL_ROOT}/cmake/lib/Debug" - "${NCCL_ROOT}/cmake/lib" - ) -else() - # cmake source-build layout: cmake/src//nccl.lib (import lib for nccl.dll) - set(_nccl_lib_names nccl) - set(_nccl_extra_lib_hints - "${NCCL_ROOT}/cmake/src/Release" - "${NCCL_ROOT}/cmake/src/Debug" - "${NCCL_ROOT}/cmake/src" - ) -endif() - -find_path(NCCL_INCLUDE_DIR - NAMES nccl.h - HINTS - ${NCCL_ROOT} - "${NCCL_ROOT}/cmake/src/Release" - "${NCCL_ROOT}/cmake/src/Debug" - "${NCCL_ROOT}/cmake/src" - "${NCCL_ROOT}/cmake" - $ENV{NCCL_ROOT} - $ENV{CUDA_HOME} - /usr/local/cuda - PATH_SUFFIXES include src/include -) - -find_library(NCCL_LIBRARY - NAMES ${_nccl_lib_names} - HINTS - ${_nccl_extra_lib_hints} - ${NCCL_ROOT} - $ENV{NCCL_ROOT} - $ENV{CUDA_HOME} - /usr/local/cuda - PATH_SUFFIXES lib lib64 -) - -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(NCCL - DEFAULT_MSG - NCCL_LIBRARY NCCL_INCLUDE_DIR -) - -if(NCCL_FOUND) - set(NCCL_LIBRARIES ${NCCL_LIBRARY}) - set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR}) - - if(NOT TARGET NCCL::NCCL) - if(NCCL_STATIC) - add_library(NCCL::NCCL STATIC IMPORTED) - else() - add_library(NCCL::NCCL UNKNOWN IMPORTED) - endif() - set_target_properties(NCCL::NCCL PROPERTIES - IMPORTED_LOCATION "${NCCL_LIBRARY}" - INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIR}" - ) - endif() -endif() - -mark_as_advanced(NCCL_INCLUDE_DIR NCCL_LIBRARY) +# cmake/FindNCCL.cmake + +# NVIDIA does not distribute CMake files with NCCL, therefore use this file to find it instead. +# +# Inputs: +# NCCL_ROOT — root of an NCCL installation or source build tree +# NCCL_STATIC — if ON, prefer the static library and search cmake/lib/Release (or Debug); +# if OFF (default), prefer the shared/import library and search cmake/src/Release + +if(NCCL_STATIC) + # cmake source-build layout: cmake/lib//nccl_static.lib (or nccl.lib) + set(_nccl_lib_names nccl_static nccl) + set(_nccl_extra_lib_hints + "${NCCL_ROOT}/cmake/lib/Release" + "${NCCL_ROOT}/cmake/lib/Debug" + "${NCCL_ROOT}/cmake/lib" + ) +else() + # cmake source-build layout: cmake/src//nccl.lib (import lib for nccl.dll) + set(_nccl_lib_names nccl) + set(_nccl_extra_lib_hints + "${NCCL_ROOT}/cmake/src/Release" + "${NCCL_ROOT}/cmake/src/Debug" + "${NCCL_ROOT}/cmake/src" + ) +endif() + +find_path(NCCL_INCLUDE_DIR + NAMES nccl.h + HINTS + ${NCCL_ROOT} + "${NCCL_ROOT}/cmake/src/Release" + "${NCCL_ROOT}/cmake/src/Debug" + "${NCCL_ROOT}/cmake/src" + "${NCCL_ROOT}/cmake" + $ENV{NCCL_ROOT} + $ENV{CUDA_HOME} + /usr/local/cuda + PATH_SUFFIXES include src/include +) + +find_library(NCCL_LIBRARY + NAMES ${_nccl_lib_names} + HINTS + ${_nccl_extra_lib_hints} + ${NCCL_ROOT} + $ENV{NCCL_ROOT} + $ENV{CUDA_HOME} + /usr/local/cuda + PATH_SUFFIXES lib lib64 +) + +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(NCCL + DEFAULT_MSG + NCCL_LIBRARY NCCL_INCLUDE_DIR +) + +if(NCCL_FOUND) + set(NCCL_LIBRARIES ${NCCL_LIBRARY}) + set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR}) + + if(NOT TARGET NCCL::NCCL) + if(NCCL_STATIC) + add_library(NCCL::NCCL STATIC IMPORTED) + else() + add_library(NCCL::NCCL UNKNOWN IMPORTED) + endif() + set_target_properties(NCCL::NCCL PROPERTIES + IMPORTED_LOCATION "${NCCL_LIBRARY}" + INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIR}" + ) + endif() +endif() + +mark_as_advanced(NCCL_INCLUDE_DIR NCCL_LIBRARY) diff --git a/ggml/src/ggml-cuda/CMakeLists.txt b/ggml/src/ggml-cuda/CMakeLists.txt index 3286a6840aa..8f6c54d6cc0 100644 --- a/ggml/src/ggml-cuda/CMakeLists.txt +++ b/ggml/src/ggml-cuda/CMakeLists.txt @@ -1,275 +1,275 @@ -cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES - -find_package(CUDAToolkit) - -if (CUDAToolkit_FOUND) - message(STATUS "CUDA Toolkit found") - - if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES) - # native == GPUs available at build time - # 50 == Maxwell, lowest CUDA 12 standard - # 60 == P100, FP16 CUDA intrinsics - # 61 == Pascal, __dp4a instruction (per-byte integer dot product) - # 70 == V100, FP16 tensor cores - # 75 == Turing, int8 tensor cores - # 80 == Ampere, asynchronous data loading, faster tensor core instructions - # 86 == RTX 3000, needs CUDA v11.1 - # 89 == RTX 4000, needs CUDA v11.8 - # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores - # - # XX-virtual == compile CUDA code as PTX, do JIT compilation to binary code on first run - # XX-real == compile CUDA code as device code for this specific architecture - # no suffix == compile as both PTX and device code - # - # The default behavior for a non-native is to build virtual architectures as needed to cover all features needed - # for best performance and to also build real architectures for the most commonly used GPUs. - if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24") - set(CMAKE_CUDA_ARCHITECTURES "native") - else() - if (CUDAToolkit_VERSION VERSION_LESS "13") - list(APPEND CMAKE_CUDA_ARCHITECTURES 50-virtual 61-virtual 70-virtual) - endif () - - list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real) - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8") - list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real) - endif() - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") - # The CUDA architecture 120f-virtual would in principle work for Blackwell support - # but the newly added "f" suffix conflicted with a preexising regex for validating CUDA architectures in CMake. - # So either a recent CMake version or one with the backported fix is needed. - # The following versions should work: - # - CMake >= v3.31.8 && CMake < v4.0.0 - # - CMake >= v4.0.2 - # This is NOT documented in the CMake release notes, - # check Modules/Internal/CMakeCUDAArchitecturesValidate.cmake in the CMake git repository instead. - # However, the architectures 120a-real and 121a-real should work with basically any CMake version and - # until the release of e.g. Rubin there is no benefit to shipping virtual architectures for Blackwell. - list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real) - endif() - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9") - list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real) - endif() - endif() - endif() - - enable_language(CUDA) - - # TODO: Remove once CCCL 3.2 has been released and bundled with CUDA Toolkit - if (GGML_CUDA_CUB_3DOT2) - include(FetchContent) - - FetchContent_Declare( - CCCL - GIT_REPOSITORY https://github.com/nvidia/cccl.git - GIT_TAG v3.2.0 - GIT_SHALLOW TRUE - ) - - FetchContent_MakeAvailable(CCCL) - endif() - - # Replace any plain 12X CUDA architectures with their "architecture-specific" equivalents 12Xa. - # 12X is forwards-compatible, 12Xa is not. - # Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa. - # But while 12X vs. 12Xa can be checked in device code there is (to my knowledge) no easy way to do the same check in host code. - # So for now just replace all instances of 12X with 12Xa, this should be fine until Rubin is released. - foreach(ARCHS IN ITEMS CMAKE_CUDA_ARCHITECTURES CMAKE_CUDA_ARCHITECTURES_NATIVE) - set(FIXED_ARCHS "") - foreach(ARCH IN LISTS ${ARCHS}) - if (ARCH MATCHES "^12[0-9](-real|-virtual)?$") - string(REGEX REPLACE "^(12[0-9])((-real|-virtual)?)$" "\\1a\\2" FIXED_ARCH ${ARCH}) - message(STATUS "Replacing ${ARCH} in ${ARCHS} with ${FIXED_ARCH}") - list(APPEND FIXED_ARCHS "${FIXED_ARCH}") - else() - list(APPEND FIXED_ARCHS "${ARCH}") - endif() - endforeach() - set(${ARCHS} ${FIXED_ARCHS}) - endforeach() - - # If we try to compile a "native" build it will use the 12X architectures and fail. - # So we should instead use the native architectures as determined by CMake after replacing 12X with 12Xa. - # But if at the time of the build no GPUs are connected at all CMAKE_CUDA_ARCHITECTURES will contain garbage that we should not use. - if (CMAKE_CUDA_ARCHITECTURES STREQUAL "native" AND CMAKE_CUDA_ARCHITECTURES_NATIVE MATCHES "^[0-9]+(a|f)?(-real|-virtual)?(;[0-9]+(a|f)?(-real|-virtual)?|;)*$") - set(CMAKE_CUDA_ARCHITECTURES ${CMAKE_CUDA_ARCHITECTURES_NATIVE}) - endif() - message(STATUS "Using CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES} CMAKE_CUDA_ARCHITECTURES_NATIVE=${CMAKE_CUDA_ARCHITECTURES_NATIVE}") - - file(GLOB GGML_HEADERS_CUDA "*.cuh") - list(APPEND GGML_HEADERS_CUDA "../../include/ggml-cuda.h") - - file(GLOB GGML_SOURCES_CUDA "*.cu") - file(GLOB SRCS "template-instances/fattn-tile*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/fattn-mma*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/mmq*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - file(GLOB SRCS "template-instances/mmf*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - - if (GGML_CUDA_FA_ALL_QUANTS) - file(GLOB SRCS "template-instances/fattn-vec*.cu") - list(APPEND GGML_SOURCES_CUDA ${SRCS}) - add_compile_definitions(GGML_CUDA_FA_ALL_QUANTS) - else() - list(APPEND GGML_SOURCES_CUDA - template-instances/fattn-vec-instance-f16-f16.cu - template-instances/fattn-vec-instance-q4_0-q4_0.cu - template-instances/fattn-vec-instance-q8_0-q8_0.cu - template-instances/fattn-vec-instance-bf16-bf16.cu) - endif() - - ggml_add_backend_library(ggml-cuda - ${GGML_HEADERS_CUDA} - ${GGML_SOURCES_CUDA} - ) - - add_compile_definitions(GGML_CUDA_PEER_MAX_BATCH_SIZE=${GGML_CUDA_PEER_MAX_BATCH_SIZE}) - - if (GGML_CUDA_GRAPHS) - add_compile_definitions(GGML_CUDA_USE_GRAPHS) - endif() - - if (GGML_CUDA_FORCE_MMQ) - add_compile_definitions(GGML_CUDA_FORCE_MMQ) - endif() - - if (GGML_CUDA_FORCE_CUBLAS) - add_compile_definitions(GGML_CUDA_FORCE_CUBLAS) - endif() - - if (GGML_CUDA_NO_VMM) - add_compile_definitions(GGML_CUDA_NO_VMM) - endif() - - if (NOT GGML_CUDA_FA) - add_compile_definitions(GGML_CUDA_NO_FA) - endif() - - if (GGML_CUDA_NO_PEER_COPY) - add_compile_definitions(GGML_CUDA_NO_PEER_COPY) - endif() - - if (GGML_STATIC) - if (WIN32) - # As of 12.3.1 CUDA Toolkit for Windows does not offer a static cublas library - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas) - else () - if (GGML_CUDA_CUB_3DOT2) - target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) - endif() - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "10.1") - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static CUDA::cublasLt_static) - else() - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static) - endif() - endif() - else() - if (GGML_CUDA_CUB_3DOT2) - target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) - endif() - target_link_libraries(ggml-cuda PRIVATE CUDA::cudart CUDA::cublas) - endif() - - if (GGML_CUDA_NO_VMM) - # No VMM requested, no need to link directly with the cuda driver lib (libcuda.so) - else() - target_link_libraries(ggml-cuda PRIVATE CUDA::cuda_driver) - endif() - - if (GGML_CUDA_NCCL) - if (GGML_CUDA_NCCL_STATIC) - set(NCCL_STATIC ON) - endif() - find_package(NCCL) - if (NCCL_FOUND) - add_compile_definitions(GGML_USE_NCCL) - target_link_libraries(ggml-cuda PRIVATE NCCL::NCCL) - else() - message(STATUS "Warning: NCCL not found, performance for multiple CUDA GPUs will be suboptimal") - endif() - endif() - - if (GGML_CUDA_AR_WATCHDOG) - add_compile_definitions(GGML_CUDA_AR_WATCHDOG) - endif() - - set(CUDA_CXX_FLAGS "") - - set(CUDA_FLAGS -use_fast_math -extended-lambda) - - if (GGML_CUDA_DEBUG) - list(APPEND CUDA_FLAGS -lineinfo) - add_compile_definitions(GGML_CUDA_DEBUG) - endif() - - if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") - # Options are: - # - none (not recommended) - # - speed (nvcc's default) - # - balance - # - size - list(APPEND CUDA_FLAGS -compress-mode=${GGML_CUDA_COMPRESSION_MODE}) - endif() - - if (GGML_FATAL_WARNINGS) - list(APPEND CUDA_FLAGS -Werror all-warnings) - endif() - - if (GGML_ALL_WARNINGS AND NOT MSVC) - set(NVCC_CMD ${CMAKE_CUDA_COMPILER} .c) - if (NOT CMAKE_CUDA_HOST_COMPILER STREQUAL "") - list(APPEND NVCC_CMD -ccbin ${CMAKE_CUDA_HOST_COMPILER}) - endif() - - execute_process( - COMMAND ${NVCC_CMD} -Xcompiler --version - OUTPUT_VARIABLE CUDA_CCFULLVER - ERROR_QUIET - ) - - if (NOT CUDA_CCFULLVER MATCHES clang) - set(CUDA_CCID "GNU") - execute_process( - COMMAND ${NVCC_CMD} -Xcompiler "-dumpfullversion -dumpversion" - OUTPUT_VARIABLE CUDA_CCVER - ERROR_QUIET - OUTPUT_STRIP_TRAILING_WHITESPACE - ) - else() - if (CUDA_CCFULLVER MATCHES Apple) - set(CUDA_CCID "AppleClang") - else() - set(CUDA_CCID "Clang") - endif() - string(REGEX REPLACE "^.* version ([0-9.]*).*$" "\\1" CUDA_CCVER ${CUDA_CCFULLVER}) - endif() - - message(STATUS "CUDA host compiler is ${CUDA_CCID} ${CUDA_CCVER}") - - ggml_get_flags(${CUDA_CCID} ${CUDA_CCVER}) - list(APPEND CUDA_CXX_FLAGS ${CXX_FLAGS} ${GF_CXX_FLAGS}) # This is passed to -Xcompiler later - endif() - - if (NOT MSVC) - list(APPEND CUDA_CXX_FLAGS -Wno-pedantic) - else() - # CCCL 3.2 onwards will require a cpp-standard-compliant preprocessor for MSVC - # https://github.com/NVIDIA/cccl/pull/6827 - list(APPEND CUDA_CXX_FLAGS /Zc:preprocessor) - endif() - - list(JOIN CUDA_CXX_FLAGS " " CUDA_CXX_FLAGS_JOINED) # pass host compiler flags as a single argument - - if (NOT CUDA_CXX_FLAGS_JOINED STREQUAL "") - list(APPEND CUDA_FLAGS -Xcompiler ${CUDA_CXX_FLAGS_JOINED}) - endif() - - target_compile_options(ggml-cuda PRIVATE "$<$:${CUDA_FLAGS}>") -else() - message(FATAL_ERROR "CUDA Toolkit not found") -endif() +cmake_minimum_required(VERSION 3.18) # for CMAKE_CUDA_ARCHITECTURES + +find_package(CUDAToolkit) + +if (CUDAToolkit_FOUND) + message(STATUS "CUDA Toolkit found") + + if (NOT DEFINED CMAKE_CUDA_ARCHITECTURES) + # native == GPUs available at build time + # 50 == Maxwell, lowest CUDA 12 standard + # 60 == P100, FP16 CUDA intrinsics + # 61 == Pascal, __dp4a instruction (per-byte integer dot product) + # 70 == V100, FP16 tensor cores + # 75 == Turing, int8 tensor cores + # 80 == Ampere, asynchronous data loading, faster tensor core instructions + # 86 == RTX 3000, needs CUDA v11.1 + # 89 == RTX 4000, needs CUDA v11.8 + # 120 == Blackwell, needs CUDA v12.8, FP4 tensor cores + # + # XX-virtual == compile CUDA code as PTX, do JIT compilation to binary code on first run + # XX-real == compile CUDA code as device code for this specific architecture + # no suffix == compile as both PTX and device code + # + # The default behavior for a non-native is to build virtual architectures as needed to cover all features needed + # for best performance and to also build real architectures for the most commonly used GPUs. + if (GGML_NATIVE AND CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.6" AND CMAKE_VERSION VERSION_GREATER_EQUAL "3.24") + set(CMAKE_CUDA_ARCHITECTURES "native") + else() + if (CUDAToolkit_VERSION VERSION_LESS "13") + list(APPEND CMAKE_CUDA_ARCHITECTURES 50-virtual 61-virtual 70-virtual) + endif () + + list(APPEND CMAKE_CUDA_ARCHITECTURES 75-virtual 80-virtual 86-real) + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "11.8") + list(APPEND CMAKE_CUDA_ARCHITECTURES 89-real) + endif() + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") + # The CUDA architecture 120f-virtual would in principle work for Blackwell support + # but the newly added "f" suffix conflicted with a preexising regex for validating CUDA architectures in CMake. + # So either a recent CMake version or one with the backported fix is needed. + # The following versions should work: + # - CMake >= v3.31.8 && CMake < v4.0.0 + # - CMake >= v4.0.2 + # This is NOT documented in the CMake release notes, + # check Modules/Internal/CMakeCUDAArchitecturesValidate.cmake in the CMake git repository instead. + # However, the architectures 120a-real and 121a-real should work with basically any CMake version and + # until the release of e.g. Rubin there is no benefit to shipping virtual architectures for Blackwell. + list(APPEND CMAKE_CUDA_ARCHITECTURES 120a-real) + endif() + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.9") + list(APPEND CMAKE_CUDA_ARCHITECTURES 121a-real) + endif() + endif() + endif() + + enable_language(CUDA) + + # TODO: Remove once CCCL 3.2 has been released and bundled with CUDA Toolkit + if (GGML_CUDA_CUB_3DOT2) + include(FetchContent) + + FetchContent_Declare( + CCCL + GIT_REPOSITORY https://github.com/nvidia/cccl.git + GIT_TAG v3.2.0 + GIT_SHALLOW TRUE + ) + + FetchContent_MakeAvailable(CCCL) + endif() + + # Replace any plain 12X CUDA architectures with their "architecture-specific" equivalents 12Xa. + # 12X is forwards-compatible, 12Xa is not. + # Notably the Blackwell FP4 tensor core instructions are not forwards compatible and therefore need 12Xa. + # But while 12X vs. 12Xa can be checked in device code there is (to my knowledge) no easy way to do the same check in host code. + # So for now just replace all instances of 12X with 12Xa, this should be fine until Rubin is released. + foreach(ARCHS IN ITEMS CMAKE_CUDA_ARCHITECTURES CMAKE_CUDA_ARCHITECTURES_NATIVE) + set(FIXED_ARCHS "") + foreach(ARCH IN LISTS ${ARCHS}) + if (ARCH MATCHES "^12[0-9](-real|-virtual)?$") + string(REGEX REPLACE "^(12[0-9])((-real|-virtual)?)$" "\\1a\\2" FIXED_ARCH ${ARCH}) + message(STATUS "Replacing ${ARCH} in ${ARCHS} with ${FIXED_ARCH}") + list(APPEND FIXED_ARCHS "${FIXED_ARCH}") + else() + list(APPEND FIXED_ARCHS "${ARCH}") + endif() + endforeach() + set(${ARCHS} ${FIXED_ARCHS}) + endforeach() + + # If we try to compile a "native" build it will use the 12X architectures and fail. + # So we should instead use the native architectures as determined by CMake after replacing 12X with 12Xa. + # But if at the time of the build no GPUs are connected at all CMAKE_CUDA_ARCHITECTURES will contain garbage that we should not use. + if (CMAKE_CUDA_ARCHITECTURES STREQUAL "native" AND CMAKE_CUDA_ARCHITECTURES_NATIVE MATCHES "^[0-9]+(a|f)?(-real|-virtual)?(;[0-9]+(a|f)?(-real|-virtual)?|;)*$") + set(CMAKE_CUDA_ARCHITECTURES ${CMAKE_CUDA_ARCHITECTURES_NATIVE}) + endif() + message(STATUS "Using CMAKE_CUDA_ARCHITECTURES=${CMAKE_CUDA_ARCHITECTURES} CMAKE_CUDA_ARCHITECTURES_NATIVE=${CMAKE_CUDA_ARCHITECTURES_NATIVE}") + + file(GLOB GGML_HEADERS_CUDA "*.cuh") + list(APPEND GGML_HEADERS_CUDA "../../include/ggml-cuda.h") + + file(GLOB GGML_SOURCES_CUDA "*.cu") + file(GLOB SRCS "template-instances/fattn-tile*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/fattn-mma*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/mmq*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + file(GLOB SRCS "template-instances/mmf*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + + if (GGML_CUDA_FA_ALL_QUANTS) + file(GLOB SRCS "template-instances/fattn-vec*.cu") + list(APPEND GGML_SOURCES_CUDA ${SRCS}) + add_compile_definitions(GGML_CUDA_FA_ALL_QUANTS) + else() + list(APPEND GGML_SOURCES_CUDA + template-instances/fattn-vec-instance-f16-f16.cu + template-instances/fattn-vec-instance-q4_0-q4_0.cu + template-instances/fattn-vec-instance-q8_0-q8_0.cu + template-instances/fattn-vec-instance-bf16-bf16.cu) + endif() + + ggml_add_backend_library(ggml-cuda + ${GGML_HEADERS_CUDA} + ${GGML_SOURCES_CUDA} + ) + + add_compile_definitions(GGML_CUDA_PEER_MAX_BATCH_SIZE=${GGML_CUDA_PEER_MAX_BATCH_SIZE}) + + if (GGML_CUDA_GRAPHS) + add_compile_definitions(GGML_CUDA_USE_GRAPHS) + endif() + + if (GGML_CUDA_FORCE_MMQ) + add_compile_definitions(GGML_CUDA_FORCE_MMQ) + endif() + + if (GGML_CUDA_FORCE_CUBLAS) + add_compile_definitions(GGML_CUDA_FORCE_CUBLAS) + endif() + + if (GGML_CUDA_NO_VMM) + add_compile_definitions(GGML_CUDA_NO_VMM) + endif() + + if (NOT GGML_CUDA_FA) + add_compile_definitions(GGML_CUDA_NO_FA) + endif() + + if (GGML_CUDA_NO_PEER_COPY) + add_compile_definitions(GGML_CUDA_NO_PEER_COPY) + endif() + + if (GGML_STATIC) + if (WIN32) + # As of 12.3.1 CUDA Toolkit for Windows does not offer a static cublas library + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas) + else () + if (GGML_CUDA_CUB_3DOT2) + target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) + endif() + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "10.1") + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static CUDA::cublasLt_static) + else() + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart_static CUDA::cublas_static) + endif() + endif() + else() + if (GGML_CUDA_CUB_3DOT2) + target_link_libraries(ggml-cuda PRIVATE CCCL::CCCL) + endif() + target_link_libraries(ggml-cuda PRIVATE CUDA::cudart CUDA::cublas) + endif() + + if (GGML_CUDA_NO_VMM) + # No VMM requested, no need to link directly with the cuda driver lib (libcuda.so) + else() + target_link_libraries(ggml-cuda PRIVATE CUDA::cuda_driver) + endif() + + if (GGML_CUDA_NCCL) + if (GGML_CUDA_NCCL_STATIC) + set(NCCL_STATIC ON) + endif() + find_package(NCCL) + if (NCCL_FOUND) + add_compile_definitions(GGML_USE_NCCL) + target_link_libraries(ggml-cuda PRIVATE NCCL::NCCL) + else() + message(STATUS "Warning: NCCL not found, performance for multiple CUDA GPUs will be suboptimal") + endif() + endif() + + if (GGML_CUDA_AR_WATCHDOG) + add_compile_definitions(GGML_CUDA_AR_WATCHDOG) + endif() + + set(CUDA_CXX_FLAGS "") + + set(CUDA_FLAGS -use_fast_math -extended-lambda) + + if (GGML_CUDA_DEBUG) + list(APPEND CUDA_FLAGS -lineinfo) + add_compile_definitions(GGML_CUDA_DEBUG) + endif() + + if (CUDAToolkit_VERSION VERSION_GREATER_EQUAL "12.8") + # Options are: + # - none (not recommended) + # - speed (nvcc's default) + # - balance + # - size + list(APPEND CUDA_FLAGS -compress-mode=${GGML_CUDA_COMPRESSION_MODE}) + endif() + + if (GGML_FATAL_WARNINGS) + list(APPEND CUDA_FLAGS -Werror all-warnings) + endif() + + if (GGML_ALL_WARNINGS AND NOT MSVC) + set(NVCC_CMD ${CMAKE_CUDA_COMPILER} .c) + if (NOT CMAKE_CUDA_HOST_COMPILER STREQUAL "") + list(APPEND NVCC_CMD -ccbin ${CMAKE_CUDA_HOST_COMPILER}) + endif() + + execute_process( + COMMAND ${NVCC_CMD} -Xcompiler --version + OUTPUT_VARIABLE CUDA_CCFULLVER + ERROR_QUIET + ) + + if (NOT CUDA_CCFULLVER MATCHES clang) + set(CUDA_CCID "GNU") + execute_process( + COMMAND ${NVCC_CMD} -Xcompiler "-dumpfullversion -dumpversion" + OUTPUT_VARIABLE CUDA_CCVER + ERROR_QUIET + OUTPUT_STRIP_TRAILING_WHITESPACE + ) + else() + if (CUDA_CCFULLVER MATCHES Apple) + set(CUDA_CCID "AppleClang") + else() + set(CUDA_CCID "Clang") + endif() + string(REGEX REPLACE "^.* version ([0-9.]*).*$" "\\1" CUDA_CCVER ${CUDA_CCFULLVER}) + endif() + + message(STATUS "CUDA host compiler is ${CUDA_CCID} ${CUDA_CCVER}") + + ggml_get_flags(${CUDA_CCID} ${CUDA_CCVER}) + list(APPEND CUDA_CXX_FLAGS ${CXX_FLAGS} ${GF_CXX_FLAGS}) # This is passed to -Xcompiler later + endif() + + if (NOT MSVC) + list(APPEND CUDA_CXX_FLAGS -Wno-pedantic) + else() + # CCCL 3.2 onwards will require a cpp-standard-compliant preprocessor for MSVC + # https://github.com/NVIDIA/cccl/pull/6827 + list(APPEND CUDA_CXX_FLAGS /Zc:preprocessor) + endif() + + list(JOIN CUDA_CXX_FLAGS " " CUDA_CXX_FLAGS_JOINED) # pass host compiler flags as a single argument + + if (NOT CUDA_CXX_FLAGS_JOINED STREQUAL "") + list(APPEND CUDA_FLAGS -Xcompiler ${CUDA_CXX_FLAGS_JOINED}) + endif() + + target_compile_options(ggml-cuda PRIVATE "$<$:${CUDA_FLAGS}>") +else() + message(FATAL_ERROR "CUDA Toolkit not found") +endif() diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 1e00cbeb6d7..8e489a092f5 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1,5484 +1,5484 @@ -#include "ggml-cuda.h" -#include "ggml-impl.h" -#include "ggml-backend-impl.h" - -#include "ggml-cuda/allreduce.cuh" -#include "ggml-cuda/comm.cuh" -#include "ggml-cuda/common.cuh" -#include "ggml-cuda/acc.cuh" -#include "ggml-cuda/add-id.cuh" -#include "ggml-cuda/arange.cuh" -#include "ggml-cuda/argmax.cuh" -#include "ggml-cuda/argsort.cuh" -#include "ggml-cuda/binbcast.cuh" -#include "ggml-cuda/clamp.cuh" -#include "ggml-cuda/concat.cuh" -#include "ggml-cuda/conv-transpose-1d.cuh" -#include "ggml-cuda/conv2d.cuh" -#include "ggml-cuda/conv2d-dw.cuh" -#include "ggml-cuda/conv2d-transpose.cuh" -#include "ggml-cuda/convert.cuh" -#include "ggml-cuda/count-equal.cuh" -#include "ggml-cuda/cpy.cuh" -#include "ggml-cuda/cross-entropy-loss.cuh" -#include "ggml-cuda/cumsum.cuh" -#include "ggml-cuda/diagmask.cuh" -#include "ggml-cuda/diag.cuh" -#include "ggml-cuda/fattn.cuh" -#include "ggml-cuda/getrows.cuh" -#include "ggml-cuda/im2col.cuh" -#include "ggml-cuda/mmf.cuh" -#include "ggml-cuda/mmq.cuh" -#include "ggml-cuda/mmvf.cuh" -#include "ggml-cuda/mmvq.cuh" -#include "ggml-cuda/norm.cuh" -#include "ggml-cuda/opt-step-adamw.cuh" -#include "ggml-cuda/opt-step-sgd.cuh" -#include "ggml-cuda/out-prod.cuh" -#include "ggml-cuda/pad.cuh" -#include "ggml-cuda/pool2d.cuh" -#include "ggml-cuda/quantize.cuh" -#include "ggml-cuda/rope.cuh" -#include "ggml-cuda/roll.cuh" -#include "ggml-cuda/scale.cuh" -#include "ggml-cuda/softcap.cuh" -#include "ggml-cuda/softmax.cuh" -#include "ggml-cuda/ssm-conv.cuh" -#include "ggml-cuda/ssm-scan.cuh" -#include "ggml-cuda/sum.cuh" -#include "ggml-cuda/sumrows.cuh" -#include "ggml-cuda/top-k.cuh" -#include "ggml-cuda/mean.cuh" -#include "ggml-cuda/tsembd.cuh" -#include "ggml-cuda/topk-moe.cuh" -#include "ggml-cuda/unary.cuh" -#include "ggml-cuda/upscale.cuh" -#include "ggml-cuda/wkv.cuh" -#include "ggml-cuda/gla.cuh" -#include "ggml-cuda/gated_delta_net.cuh" -#include "ggml-cuda/set.cuh" -#include "ggml-cuda/set-rows.cuh" -#include "ggml-cuda/pad_reflect_1d.cuh" -#include "ggml-cuda/solve_tri.cuh" -#include "ggml-cuda/tri.cuh" -#include "ggml-cuda/cumsum.cuh" -#include "ggml-cuda/fill.cuh" -#include "ggml.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size"); - -[[noreturn]] -void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) { - int id = -1; // in case cudaGetDevice fails - (void)cudaGetDevice(&id); - - GGML_LOG_ERROR(GGML_CUDA_NAME " error: %s\n", msg); - GGML_LOG_ERROR(" current device: %d, in function %s at %s:%d\n", id, func, file, line); - GGML_LOG_ERROR(" %s\n", stmt); - // abort with GGML_ABORT to get a stack trace - GGML_ABORT(GGML_CUDA_NAME " error"); -} - -// this is faster on Windows -// probably because the Windows CUDA libraries forget to make this check before invoking the drivers -void ggml_cuda_set_device(int device) { - int current_device; - CUDA_CHECK(cudaGetDevice(¤t_device)); - - if (device == current_device) { - return; - } - - CUDA_CHECK(cudaSetDevice(device)); -} - -int ggml_cuda_get_device() { - int id; - CUDA_CHECK(cudaGetDevice(&id)); - return id; -} - -static cudaError_t ggml_cuda_device_malloc(void ** ptr, size_t size, int device) { - ggml_cuda_set_device(device); - cudaError_t err; - if (getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr) { - err = cudaMallocManaged(ptr, size); -#if defined(GGML_USE_HIP) - if (err == hipSuccess) { - // hipMemAdviseSetCoarseGrain is an optional performance hint; - // ignore errors (e.g. hipErrorInvalidValue on some APU/iGPU configs). - (void)cudaMemAdvise(*ptr, size, hipMemAdviseSetCoarseGrain, device); - (void)hipGetLastError(); // clear any error - } - - // fall back to cudaMalloc if not supported (e.g. on Windows) - if (err == hipErrorNotSupported) { - static bool warned_unsupported = false; - if (!warned_unsupported) { - GGML_LOG_WARN("hipMallocManaged unsupported, falling back to hipMalloc.\n"); - warned_unsupported = true; - } - - err = cudaMalloc(ptr, size); - } -#endif // defined(GGML_USE_HIP) - } else { - err = cudaMalloc(ptr, size); - } - return err; -} - -#if defined(GGML_USE_HIP) -static int ggml_cuda_parse_id(char devName[]) { - // A list of possible Target IDs can be found under the rocclr/clr repo in device.cpp - // these values are not stable so this is susceptible to breakage - // https://github.com/ROCm/clr/blob/amd-staging/rocclr/device/device.cpp - int archMajor = 0x0; - int archMinor = 0x0; - int archNum = GGML_CUDA_CC_OFFSET_AMD; - int archLen = strlen(devName); - char archName[archLen + 1]; - - // strip leading 'gfx' while copying into our buffer - if (archLen > 3) { - strcpy(archName, &devName[3]); - archLen -= 3; - } - - // trim trailing :xnack- or :sramecc- statuses - archLen = strcspn(archName, ":"); - archName[archLen] = '\0'; - - // tease out the version information - if (archLen > 8) { - // versions labeled generic use '-' as delimiter - // strip the trailing "-generic" then iterate through what remains - if ((strstr(archName, "-generic"))) { - archName[archLen - 8] = '\0'; - char * pch; - if ((pch = strtok(archName, "-"))) { - archMajor = (int)strtoul(pch, 0, 16); - if ((pch = strtok(NULL, "-"))) { - archMinor = 0x10 * (int)strtoul(pch, 0, 16); - } - } - } - } else if (archLen >= 3) { - // last two digits should be the minor * 0x10 + stepping - archMinor = (int)strtoul(&archName[archLen - 2], 0, 16); - archName[archLen - 2] = '\0'; - - // only the major version remains - archMajor = (int)strtoul(archName, 0, 16); - } - archNum += archMajor * 0x100; - archNum += archMinor; - return archNum; -} -#endif // defined(GGML_USE_HIP) - -static ggml_cuda_device_info ggml_cuda_init() { - ggml_cuda_device_info info = {}; - - cudaError_t err = cudaGetDeviceCount(&info.device_count); - if (err != cudaSuccess) { - GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err)); - return info; - } - - GGML_ASSERT(info.device_count <= GGML_CUDA_MAX_DEVICES); - - int64_t total_vram = 0; - for (int id = 0; id < info.device_count; ++id) { - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); - total_vram += prop.totalGlobalMem; - } - GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n", - __func__, info.device_count, (size_t)(total_vram / (1024 * 1024))); - total_vram = 0; - - std::vector> turing_devices_without_mma; - for (int id = 0; id < info.device_count; ++id) { - int device_vmm = 0; - -#if defined(GGML_USE_VMM) - CUdevice device; - CU_CHECK(cuDeviceGet(&device, id)); - CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device)); - - if (device_vmm) { - CUmemAllocationProp alloc_prop = {}; - alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; - alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - alloc_prop.location.id = id; - CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED)); - } -#endif // defined(GGML_USE_VMM) - info.devices[id].vmm = !!device_vmm; - - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); - - info.default_tensor_split[id] = total_vram; - total_vram += prop.totalGlobalMem; - info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034) - info.devices[id].nsm = prop.multiProcessorCount; - info.devices[id].smpb = prop.sharedMemPerBlock; - info.devices[id].warp_size = prop.warpSize; - -#ifndef GGML_USE_MUSA - int supports_coop_launch = 0; - CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, id)); - info.devices[id].supports_cooperative_launch = !!supports_coop_launch; -#else - info.devices[id].supports_cooperative_launch = false; -#endif // !(GGML_USE_MUSA) - -#if defined(GGML_USE_HIP) - info.devices[id].smpbo = prop.sharedMemPerBlock; - - info.devices[id].cc = ggml_cuda_parse_id(prop.gcnArchName); - if ((info.devices[id].cc & 0xff00) == 0x0) { - GGML_LOG_WARN("invalid architecture ID received for device %d %s: %s cc %d.%d\n", - id, prop.name, prop.gcnArchName, prop.major, prop.minor); - - // Fallback to prop.major and prop.minor - if (prop.major > 0) { - info.devices[id].cc = GGML_CUDA_CC_OFFSET_AMD + prop.major * 0x100; - info.devices[id].cc += prop.minor * 0x10; - } - } - GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n", - id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff, - device_vmm ? "yes" : "no", prop.warpSize, - (size_t)(prop.totalGlobalMem / (1024 * 1024))); -#elif defined(GGML_USE_MUSA) - // FIXME: Ensure compatibility with varying warp sizes across different MUSA archs. - info.devices[id].warp_size = 32; - info.devices[id].smpbo = prop.sharedMemPerBlockOptin; - info.devices[id].cc = GGML_CUDA_CC_OFFSET_MTHREADS + prop.major * 0x100; - info.devices[id].cc += prop.minor * 0x10; - GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", - id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", - (size_t)(prop.totalGlobalMem / (1024 * 1024))); -#else - info.devices[id].smpbo = prop.sharedMemPerBlockOptin; - info.devices[id].cc = 100*prop.major + 10*prop.minor; - GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", - id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", - (size_t)(prop.totalGlobalMem / (1024 * 1024))); - std::string device_name(prop.name); - if (device_name == "NVIDIA GeForce MX450") { - turing_devices_without_mma.push_back({ id, device_name }); - } else if (device_name == "NVIDIA GeForce MX550") { - turing_devices_without_mma.push_back({ id, device_name }); - } else if (device_name.substr(0, 21) == "NVIDIA GeForce GTX 16") { - turing_devices_without_mma.push_back({ id, device_name }); - } - - // Temporary performance fix: - // Setting device scheduling strategy for iGPUs with cc121 to "spinning" to avoid delays in cuda synchronize calls. - // TODO: Check for future drivers the default scheduling strategy and - // remove this call again when cudaDeviceScheduleSpin is default. - if (prop.major == 12 && prop.minor == 1) { - CUDA_CHECK(cudaSetDevice(id)); - CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin)); - } - -#endif // defined(GGML_USE_HIP) - } - - if (ggml_cuda_highest_compiled_arch(GGML_CUDA_CC_TURING) >= GGML_CUDA_CC_TURING && !turing_devices_without_mma.empty()) { - GGML_LOG_INFO("The following devices will have suboptimal performance due to a lack of tensor cores:\n"); - for (size_t device_pos = 0; device_pos < turing_devices_without_mma.size(); device_pos++) { - GGML_LOG_INFO( - " Device %d: %s\n", turing_devices_without_mma[device_pos].first, turing_devices_without_mma[device_pos].second.c_str()); - } - GGML_LOG_INFO( - "Consider compiling with CMAKE_CUDA_ARCHITECTURES=61-virtual;80-virtual and DGGML_CUDA_FORCE_MMQ to force the use of the Pascal code for Turing.\n"); - } - - for (int id = 0; id < info.device_count; ++id) { - info.default_tensor_split[id] /= total_vram; - } - - // configure logging to stdout - // CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr)); - - if (getenv("GGML_CUDA_P2P") != nullptr) { - for (int id = 0; id < info.device_count; ++id) { - ggml_cuda_set_device(id); - for (int id_other = 0; id_other < info.device_count; ++id_other) { - if (id == id_other) { - continue; - } - int can_access_peer; - CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id, id_other)); - if (can_access_peer) { - CUDA_CHECK(cudaDeviceEnablePeerAccess(id_other, 0)); - } - } - } - } - - return info; -} - -const ggml_cuda_device_info & ggml_cuda_info() { - static ggml_cuda_device_info info = ggml_cuda_init(); - return info; -} - -// #define DEBUG_CUDA_MALLOC - -// buffer pool for cuda (legacy) -struct ggml_cuda_pool_leg : public ggml_cuda_pool { - static const int MAX_BUFFERS = 256; - - int device; - struct ggml_cuda_buffer { - void * ptr = nullptr; - size_t size = 0; - }; - - ggml_cuda_buffer buffer_pool[MAX_BUFFERS] = {}; - size_t pool_size = 0; - - explicit ggml_cuda_pool_leg(int device) : - device(device) { - } - - ~ggml_cuda_pool_leg() { - clear_pool(); - GGML_ASSERT(pool_size == 0); - } - - void clear_pool() { - ggml_cuda_set_device(device); - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer & b = buffer_pool[i]; - if (b.ptr != nullptr) { - CUDA_CHECK(cudaFree(b.ptr)); - pool_size -= b.size; - b.ptr = nullptr; - b.size = 0; - } - } - } - - void * alloc(size_t size, size_t * actual_size) override { -#ifdef DEBUG_CUDA_MALLOC - int nnz = 0; - size_t max_size = 0; -#endif - size_t best_diff = 1ull << 36; - int ibest = -1; - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer& b = buffer_pool[i]; - if (b.ptr != nullptr) { -#ifdef DEBUG_CUDA_MALLOC - ++nnz; - if (b.size > max_size) max_size = b.size; -#endif - if (b.size >= size) { - size_t diff = b.size - size; - if (diff < best_diff) { - best_diff = diff; - ibest = i; - if (!best_diff) { - void * ptr = b.ptr; - *actual_size = b.size; - b.ptr = nullptr; - b.size = 0; - return ptr; - } - } - } - } - } - if (ibest >= 0) { - ggml_cuda_buffer& b = buffer_pool[ibest]; - void * ptr = b.ptr; - *actual_size = b.size; - b.ptr = nullptr; - b.size = 0; - return ptr; - } - void * ptr; - size_t look_ahead_size = (size_t) (1.05 * size); - look_ahead_size = 256 * ((look_ahead_size + 255)/256); - ggml_cuda_set_device(device); - cudaError_t err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); - if (err == cudaErrorMemoryAllocation) { - (void)cudaGetLastError(); - const size_t cached_bytes = pool_size; - GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: alloc of %.2f MiB failed, flushing %.2f MiB of cached buffers and retrying\n", - device, look_ahead_size/1024.0/1024.0, cached_bytes/1024.0/1024.0); - CUDA_CHECK(cudaDeviceSynchronize()); - clear_pool(); - err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); - if (err == cudaSuccess) { - GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: retry succeeded\n", device); - } - } - CUDA_CHECK(err); - *actual_size = look_ahead_size; - pool_size += look_ahead_size; -#ifdef DEBUG_CUDA_MALLOC - GGML_LOG_INFO("%s[%d]: %d buffers, max_size = %u MB, pool_size = %u MB, requested %u MB\n", __func__, device, nnz, - (uint32_t)(max_size / 1024 / 1024), (uint32_t)(pool_size / 1024 / 1024), (uint32_t)(size / 1024 / 1024)); -#endif - return ptr; - } - - void free(void * ptr, size_t size) override { - for (int i = 0; i < MAX_BUFFERS; ++i) { - ggml_cuda_buffer& b = buffer_pool[i]; - if (b.ptr == nullptr) { - b.ptr = ptr; - b.size = size; - return; - } - } - GGML_LOG_DEBUG(GGML_CUDA_NAME " buffer pool full, increase MAX_CUDA_BUFFERS\n"); - ggml_cuda_set_device(device); - CUDA_CHECK(cudaFree(ptr)); - pool_size -= size; - } -}; - -// pool with virtual memory -#if defined(GGML_USE_VMM) -struct ggml_cuda_pool_vmm : public ggml_cuda_pool { - static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB - - int device; - CUdeviceptr pool_addr = 0; - size_t pool_used = 0; - size_t pool_size = 0; - size_t granularity; -#if defined(GGML_USE_HIP) - std::vector> mappings; -#endif - - explicit ggml_cuda_pool_vmm(int device) : - device(device), - granularity(ggml_cuda_info().devices[device].vmm_granularity) { - } - - ~ggml_cuda_pool_vmm() { - if (pool_addr != 0) { -#if defined(GGML_USE_HIP) - // Workaround for https://github.com/ROCm/ROCR-Runtime/issues/285 - for (std::pair & mapping : mappings) { - CU_CHECK(cuMemUnmap(mapping.first, mapping.second)); - } -#else - CU_CHECK(cuMemUnmap(pool_addr, pool_size)); -#endif - CU_CHECK(cuMemAddressFree(pool_addr, CUDA_POOL_VMM_MAX_SIZE)); - } - } - - void * alloc(size_t size, size_t * actual_size) override { - // round up the allocation size to the alignment to ensure that all allocations are aligned for all data types - const size_t alignment = 128; - size = alignment * ((size + alignment - 1) / alignment); - - size_t avail = pool_size - pool_used; - - if (size > avail) { - // round up to the next multiple of the granularity - size_t reserve_size = size - avail; - reserve_size = granularity * ((reserve_size + granularity - 1) / granularity); - - GGML_ASSERT(pool_size + reserve_size <= CUDA_POOL_VMM_MAX_SIZE); - - // allocate more physical memory - CUmemAllocationProp prop = {}; - prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; - prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - prop.location.id = device; - CUmemGenericAllocationHandle handle; - CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0)); - - // reserve virtual address space (if not already reserved) - if (pool_addr == 0) { - CU_CHECK(cuMemAddressReserve(&pool_addr, CUDA_POOL_VMM_MAX_SIZE, 0, 0, 0)); - } - - // map at the end of the pool - CUdeviceptr start_ptr = (CUdeviceptr)((char *)(pool_addr) + pool_size); - CU_CHECK(cuMemMap(start_ptr, reserve_size, 0, handle, 0)); -#if defined(GGML_USE_HIP) - mappings.push_back({start_ptr, reserve_size}); -#endif - - // the memory allocation handle is no longer needed after mapping - CU_CHECK(cuMemRelease(handle)); - - // set access - CUmemAccessDesc access = {}; - access.location.type = CU_MEM_LOCATION_TYPE_DEVICE; - access.location.id = device; - access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; - CU_CHECK(cuMemSetAccess((CUdeviceptr)((char *)(pool_addr) + pool_size), reserve_size, &access, 1)); - - // add to the pool - pool_size += reserve_size; - - //printf("cuda pool[%d]: size increased to %llu MB (reserved %llu MB)\n", - // device, (unsigned long long) (pool_size/1024/1024), - // (unsigned long long) (reserve_size/1024/1024)); - } - - GGML_ASSERT(pool_addr != 0); - - void * ptr = (void *) ((CUdeviceptr)((char *)(pool_addr) + pool_used)); - *actual_size = size; - pool_used += size; - -#ifdef DEBUG_CUDA_MALLOC - printf("cuda pool[%d]: allocated %llu bytes at %llx\n", device, (unsigned long long) size, ptr); -#endif - - return ptr; - } - - void free(void * ptr, size_t size) override { -#ifdef DEBUG_CUDA_MALLOC - printf("cuda pool[%d]: freed %llu bytes at %llx\n", device, (unsigned long long) size, ptr); -#endif - - pool_used -= size; - - // all deallocations must be in reverse order of the allocations - GGML_ASSERT(ptr == (void *) ((char *)(pool_addr) + pool_used)); - } -}; -#endif // defined(GGML_USE_VMM) - -std::unique_ptr ggml_backend_cuda_context::new_pool_for_device(int device, - [[maybe_unused]] int stream_no) { -#if defined(GGML_USE_VMM) - if (ggml_cuda_info().devices[device].vmm) { - return std::unique_ptr(new ggml_cuda_pool_vmm(device)); - } -#endif // defined(GGML_USE_VMM) - return std::unique_ptr(new ggml_cuda_pool_leg(device)); -} - -// destroying a cuBLAS handle while a graph is being captured in a different thread can result in a CUDA error -// this lock is used to ensure that no cuBLAS handle is destroyed while a graph is being captured - -static std::mutex ggml_cuda_lock; -static std::condition_variable ggml_cuda_lock_cv; -static std::atomic ggml_cuda_lock_counter; - -ggml_backend_cuda_context::~ggml_backend_cuda_context() { - std::unique_lock lock(ggml_cuda_lock); - ggml_cuda_lock_cv.wait(lock, []{ return ggml_cuda_lock_counter.load(std::memory_order_relaxed) == 0; }); - - if (copy_event != nullptr) { - CUDA_CHECK(cudaEventDestroy(copy_event)); - } - for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { - for (int j = 0; j < GGML_CUDA_MAX_STREAMS; ++j) { - if (streams[i][j] != nullptr) { - CUDA_CHECK(cudaStreamDestroy(streams[i][j])); - } - } - if (cublas_handles[i] != nullptr) { - CUBLAS_CHECK(cublasDestroy(cublas_handles[i])); - } - } -} - - -// cuda buffer - -struct ggml_backend_cuda_buffer_context { - int device; - void * dev_ptr = nullptr; - std::string name; - - ggml_backend_cuda_buffer_context(int device, void * dev_ptr) : - device(device), dev_ptr(dev_ptr), - name(GGML_CUDA_NAME + std::to_string(device)) { - } - - ~ggml_backend_cuda_buffer_context() { - CUDA_CHECK(cudaFree(dev_ptr)); - } -}; - -static void ggml_backend_cuda_buffer_free_buffer(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - delete ctx; -} - -static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) { - return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer; -} - -static void * ggml_backend_cuda_buffer_get_base(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - return ctx->dev_ptr; -} - -static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - if (tensor->view_src != NULL) { - assert(tensor->view_src->buffer->buft == buffer->buft); - return GGML_STATUS_SUCCESS; - } - - if (ggml_is_quantized(tensor->type) && tensor->view_src == nullptr && ggml_backend_buffer_get_usage(buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE) { - // initialize padding to 0 to avoid possible NaN values - const size_t original_size = ggml_nbytes(tensor); - const size_t padded_size = ggml_backend_buft_get_alloc_size(buffer->buft, tensor); - - if (padded_size > original_size) { - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size)); - } - } - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemsetAsync((char *) tensor->data + offset, value, size, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpy2DAsync( - (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemcpy2DAsync( - data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) { - if (ggml_backend_buffer_is_cuda(src->buffer)) { - ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context; - ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context; - if (src_ctx->device == dst_ctx->device) { - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread)); - } else { -#ifdef GGML_CUDA_NO_PEER_COPY - return false; -#else - CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_ctx->device, src->data, src_ctx->device, ggml_nbytes(src), cudaStreamPerThread)); -#endif - } - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - return true; - } - return false; - - GGML_UNUSED(buffer); -} - -static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { - ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; - - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemsetAsync(ctx->dev_ptr, value, buffer->size, cudaStreamPerThread)); - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); -} - -static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = { - /* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer, - /* .get_base = */ ggml_backend_cuda_buffer_get_base, - /* .init_tensor = */ ggml_backend_cuda_buffer_init_tensor, - /* .memset_tensor = */ ggml_backend_cuda_buffer_memset_tensor, - /* .set_tensor = */ ggml_backend_cuda_buffer_set_tensor, - /* .get_tensor = */ ggml_backend_cuda_buffer_get_tensor, - /* .set_tensor_2d = */ ggml_backend_cuda_buffer_set_tensor_2d, - /* .get_tensor_2d = */ ggml_backend_cuda_buffer_get_tensor_2d, - /* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor, - /* .clear = */ ggml_backend_cuda_buffer_clear, - /* .reset = */ NULL, -}; - -// cuda buffer type -struct ggml_backend_cuda_buffer_type_context { - int device; - std::string name; -}; - -static const char * ggml_backend_cuda_buffer_type_get_name(ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; - - return ctx->name.c_str(); -} - -static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name; -} - -static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; - - ggml_cuda_set_device(buft_ctx->device); - - void * dev_ptr; - cudaError_t err = ggml_cuda_device_malloc(&dev_ptr, size, buft_ctx->device); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - GGML_LOG_ERROR("%s: allocating %.2f MiB on device %d: cudaMalloc failed: %s\n", __func__, size / 1024.0 / 1024.0, buft_ctx->device, cudaGetErrorString(err)); - return nullptr; - } - - ggml_backend_cuda_buffer_context * ctx = new ggml_backend_cuda_buffer_context(buft_ctx->device, dev_ptr); - - return ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size); -} - -static size_t ggml_backend_cuda_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { - return 128; - - GGML_UNUSED(buft); -} - -static size_t ggml_backend_cuda_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { - size_t size = ggml_nbytes(tensor); - int64_t ne0 = tensor->ne[0]; - - if (ggml_is_quantized(tensor->type)) { - if (ne0 % MATRIX_ROW_PADDING != 0) { - GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor)); - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - } - - return size; - - GGML_UNUSED(buft); -} - -static const ggml_backend_buffer_type_i ggml_backend_cuda_buffer_type_interface = { - /* .get_name = */ ggml_backend_cuda_buffer_type_get_name, - /* .alloc_buffer = */ ggml_backend_cuda_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cuda_buffer_type_get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cuda_buffer_type_get_alloc_size, - /* .is_host = */ NULL, -}; - -ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int device) { - static std::mutex mutex; - std::lock_guard lock(mutex); - - if (device >= ggml_backend_cuda_get_device_count()) { - return nullptr; - } - - static ggml_backend_buffer_type ggml_backend_cuda_buffer_types[GGML_CUDA_MAX_DEVICES]; - - static bool ggml_backend_cuda_buffer_type_initialized = false; - - if (!ggml_backend_cuda_buffer_type_initialized) { - for (int i = 0; i < ggml_backend_cuda_get_device_count(); i++) { - ggml_backend_cuda_buffer_types[i] = { - /* .iface = */ ggml_backend_cuda_buffer_type_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), i), - /* .context = */ new ggml_backend_cuda_buffer_type_context{i, GGML_CUDA_NAME + std::to_string(i)}, - }; - } - ggml_backend_cuda_buffer_type_initialized = true; - } - - return &ggml_backend_cuda_buffer_types[device]; -} - -// cuda split buffer - -static int64_t get_row_rounding(const std::array & tensor_split) { - int64_t row_rounding = 0; - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { - continue; - } - - const int cc = ggml_cuda_info().devices[id].cc; - row_rounding = std::max(row_rounding, (int64_t)get_mmq_y_host(cc)); - } - return row_rounding; -} - -static void get_row_split(int64_t * row_low, int64_t * row_high, const ggml_tensor * tensor, const std::array & tensor_split, int id) { - const int64_t nrows = ggml_nrows(tensor); - const int64_t rounding = get_row_rounding(tensor_split); - - *row_low = id == 0 ? 0 : nrows*tensor_split[id]; - *row_low -= *row_low % rounding; - - if (id == ggml_backend_cuda_get_device_count() - 1) { - *row_high = nrows; - } else { - *row_high = nrows*tensor_split[id + 1]; - *row_high -= *row_high % rounding; - } -} - -static size_t ggml_nbytes_split(const struct ggml_tensor * tensor, int nrows_split) { - static_assert(GGML_MAX_DIMS == 4, "GGML_MAX_DIMS is not 4 - update this function"); - - return nrows_split*ggml_row_size(tensor->type, tensor->ne[0]); -} - -struct ggml_backend_cuda_split_buffer_type_context { - int main_device; - std::array tensor_split; - std::string name; -}; - -struct ggml_backend_cuda_split_buffer_context { - ~ggml_backend_cuda_split_buffer_context() { - for (ggml_tensor_extra_gpu * extra : tensor_extras) { - for (int id = 0; id < GGML_CUDA_MAX_DEVICES; ++id) { - for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { - if (extra->events[id][is] != nullptr) { - CUDA_CHECK(cudaEventDestroy(extra->events[id][is])); - } - } - if (extra->data_device[id] != nullptr) { - CUDA_CHECK(cudaFree(extra->data_device[id])); - } - } - delete extra; - } - } - - std::vector tensor_extras; -}; - - -static void ggml_backend_cuda_split_buffer_free_buffer(ggml_backend_buffer_t buffer) { - ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; - delete ctx; -} - -static void * ggml_backend_cuda_split_buffer_get_base(ggml_backend_buffer_t buffer) { - // the pointers are stored in the tensor extras, this is just a dummy address and never dereferenced - return (void *)0x1000; - - GGML_UNUSED(buffer); -} - -static enum ggml_status ggml_backend_cuda_split_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { - GGML_ASSERT(tensor->view_src == nullptr); // views of split tensors are not supported - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - - ggml_tensor_extra_gpu * extra = new ggml_tensor_extra_gpu{}; - ctx->tensor_extras.push_back(extra); - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - // FIXME: do not crash if cudaMalloc fails - // currently, init_tensor cannot fail, it needs to be fixed in ggml-backend first - ggml_cuda_set_device(id); - char * buf; - CUDA_CHECK(ggml_cuda_device_malloc((void**)&buf, size, id)); - - // set padding to 0 to avoid possible NaN values - if (size > original_size) { - CUDA_CHECK(cudaMemset(buf + original_size, 0, size - original_size)); - } - - extra->data_device[id] = buf; - - for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { - CUDA_CHECK(cudaEventCreateWithFlags(&extra->events[id][is], cudaEventDisableTiming)); - } - } - tensor->extra = extra; - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_split_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - // split tensors must always be set in their entirety at once - GGML_ASSERT(offset == 0); - GGML_ASSERT(size == ggml_nbytes(tensor)); - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - const size_t nb1 = tensor->nb[1]; - ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - const size_t offset_split = row_low*nb1; - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - const char * buf_host = (const char *)data + offset_split; - CUDA_CHECK(cudaMemcpyAsync(extra->data_device[id], buf_host, original_size, cudaMemcpyHostToDevice, cudaStreamPerThread)); - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - } -} - -static void ggml_backend_cuda_split_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - // split tensors must always be set in their entirety at once - GGML_ASSERT(offset == 0); - GGML_ASSERT(size == ggml_nbytes(tensor)); - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; - - const int64_t ne0 = tensor->ne[0]; - const size_t nb1 = tensor->nb[1]; - ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - const size_t offset_split = row_low*nb1; - size_t size = ggml_nbytes_split(tensor, nrows_split); - const size_t original_size = size; - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - - char * buf_host = (char *)data + offset_split; - CUDA_CHECK(cudaMemcpyAsync(buf_host, extra->data_device[id], original_size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); - } -} - -static void ggml_backend_cuda_split_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { - GGML_UNUSED(buffer); - GGML_UNUSED(value); -} - -static const ggml_backend_buffer_i ggml_backend_cuda_split_buffer_interface = { - /* .free_buffer = */ ggml_backend_cuda_split_buffer_free_buffer, - /* .get_base = */ ggml_backend_cuda_split_buffer_get_base, - /* .init_tensor = */ ggml_backend_cuda_split_buffer_init_tensor, - /* .memset_tensor = */ NULL, - /* .set_tensor = */ ggml_backend_cuda_split_buffer_set_tensor, - /* .get_tensor = */ ggml_backend_cuda_split_buffer_get_tensor, - /* .set_tensor_2d = */ NULL, - /* .get_tensor_2d = */ NULL, - /* .cpy_tensor = */ NULL, - /* .clear = */ ggml_backend_cuda_split_buffer_clear, - /* .reset = */ NULL, -}; - -// cuda split buffer type - -static const char * ggml_backend_cuda_split_buffer_type_get_name(ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; - - return ctx->name.c_str(); -} - -static bool ggml_backend_buft_is_cuda_split(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_split_buffer_type_get_name; -} - -static ggml_backend_buffer_t ggml_backend_cuda_split_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - // since we don't know the exact split after rounding, we cannot allocate the device buffers at this point - // instead, we allocate them for each tensor separately in init_tensor - // however, the size still represents the maximum cumulative size of all the device buffers after the tensors are allocated, - // as returned by get_alloc_size. this limit is enforced during tensor allocation by ggml-alloc, so it must be correct. - ggml_backend_cuda_split_buffer_context * ctx = new ggml_backend_cuda_split_buffer_context(); - - return ggml_backend_buffer_init(buft, ggml_backend_cuda_split_buffer_interface, ctx, size); -} - -static size_t ggml_backend_cuda_split_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { - return 128; - - GGML_UNUSED(buft); -} - -static size_t ggml_backend_cuda_split_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { - ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; - GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); - - size_t total_size = 0; - - const int64_t ne0 = tensor->ne[0]; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - int64_t row_low, row_high; - get_row_split(&row_low, &row_high, tensor, ctx->tensor_split, id); - - int64_t nrows_split = row_high - row_low; - if (nrows_split == 0) { - continue; - } - - total_size += ggml_nbytes_split(tensor, nrows_split); - - // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses - if (ne0 % MATRIX_ROW_PADDING != 0) { - total_size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); - } - } - - return total_size; -} - -static bool ggml_backend_cuda_split_buffer_type_is_host(ggml_backend_buffer_type_t buft) { - return false; - - GGML_UNUSED(buft); -} - -static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_interface = { - /* .get_name = */ ggml_backend_cuda_split_buffer_type_get_name, - /* .alloc_buffer = */ ggml_backend_cuda_split_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cuda_split_buffer_type_get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cuda_split_buffer_type_get_alloc_size, - /* .is_host = */ ggml_backend_cuda_split_buffer_type_is_host, -}; - -// Communication context for multi-GPU AllReduce during tensor parallelism. -// Created once per meta backend instance; provider is fixed at init time. -struct ggml_backend_cuda_comm_context { - ggml_cuda_allreduce_provider provider; - std::vector backends; - -#ifdef GGML_USE_NCCL - std::vector comms; // valid when provider == GGML_CUDA_ALLREDUCE_NCCL -#endif - - ggml_cuda_ar_pipeline * ar_pipeline = nullptr; // valid when provider == GGML_CUDA_ALLREDUCE_INTERNAL - - ~ggml_backend_cuda_comm_context() { -#ifdef GGML_USE_NCCL - if (provider == GGML_CUDA_ALLREDUCE_NCCL) { - for (ncclComm_t comm : comms) { - NCCL_CHECK(ncclCommDestroy(comm)); - } - } -#endif - ggml_cuda_ar_pipeline_free(ar_pipeline); - } -}; - -// Select an AllReduce provider for the given set of CUDA device IDs. -// -// Priority: -// 1. GGML_CUDA_ALLREDUCE env var ("nccl" or "internal") — explicit override. -// 2. NCCL when compiled in (GGML_USE_NCCL defined). -// 3. Internal otherwise. -// -// Future: inspect NVLink topology via cudaDeviceGetP2PAttribute() with -// cudaDevP2PAttrNativeAtomicSupported to prefer INTERNAL on PCIe-only systems -// where host-staged reduction can beat NCCL for small tensors. -static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( - const std::vector & device_ids) { - const char * env = getenv("GGML_CUDA_ALLREDUCE"); - if (env != nullptr && env[0] != '\0') { - if (strcmp(env, "internal") == 0) { - return GGML_CUDA_ALLREDUCE_INTERNAL; - } - if (strcmp(env, "nccl") == 0) { -#ifdef GGML_USE_NCCL - return GGML_CUDA_ALLREDUCE_NCCL; -#else - GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in, using internal provider\n", __func__); - return GGML_CUDA_ALLREDUCE_INTERNAL; -#endif - } - GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); - } - -#ifdef GGML_USE_NCCL - GGML_UNUSED(device_ids); - return GGML_CUDA_ALLREDUCE_NCCL; -#else - GGML_UNUSED(device_ids); - return GGML_CUDA_ALLREDUCE_INTERNAL; -#endif -} - -static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { - if (comm_ctx_v == nullptr) { - return; - } - delete static_cast(comm_ctx_v); -} - -static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { - for (size_t i = 0; i < n_backends; i++) { - if (!ggml_backend_is_cuda(backends[i])) { - return nullptr; - } - } - - std::vector dev_ids; - dev_ids.reserve(n_backends); - for (size_t i = 0; i < n_backends; i++) { - dev_ids.push_back(static_cast(backends[i]->context)->device); - } - - const ggml_cuda_allreduce_provider provider = ggml_cuda_select_allreduce_provider(dev_ids); - - auto * ret = new ggml_backend_cuda_comm_context; - ret->provider = provider; - ret->backends.assign(backends, backends + n_backends); - - switch (provider) { - case GGML_CUDA_ALLREDUCE_NCCL: { -#ifdef GGML_USE_NCCL - ret->comms.resize(n_backends); - NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); -#else - // Unreachable: ggml_cuda_select_allreduce_provider() only returns - // GGML_CUDA_ALLREDUCE_NCCL when GGML_USE_NCCL is defined. - GGML_ABORT("NCCL provider selected but NCCL not compiled in"); -#endif - } break; - - case GGML_CUDA_ALLREDUCE_INTERNAL: { - ret->ar_pipeline = ggml_cuda_ar_pipeline_init( - dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); - if (ret->ar_pipeline == nullptr) { - GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); - delete ret; - return nullptr; - } - } break; - } - - return ret; -} - -#ifdef GGML_USE_NCCL -// AllReduce via NCCL. Reduces as FP32 for small tensors and BF16 for large -// tensors (bandwidth-bound), then converts back to FP32. -static bool ggml_backend_cuda_comm_allreduce_nccl( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - const int64_t ne = ggml_nelements(tensors[0]); - // FIXME the input of llm_graph_context::build_in_out_ids can produce a tensor with 0 elements if n_outputs == 0 - // This then causes a crash in this function - if (ne == 0) { - return true; - } - - const size_t n_backends = comm_ctx->backends.size(); - - for (size_t i = 0; i < n_backends; ++i) { - GGML_ASSERT(tensors[i] != nullptr); - GGML_ASSERT(ggml_nelements(tensors[i]) == ne); - GGML_ASSERT(ggml_is_contiguously_allocated(tensors[i])); - } - - // For small tensors, simply reduce them as FP32. - // The following heuristic for how "small" a tensor should be is based on RTX 4090s connected via 16x PCIe 4.0. - if ((n_backends <= 2 && ne < 32768) || (n_backends == 3 && ne < 131072) || (n_backends >= 4 && ne < 262144)) { - for (size_t i = 0; i < n_backends; ++i) { - if ((tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - ggml_cuda_set_device(cuda_ctx->device); - CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, ggml_nbytes(tensors[i]), cuda_ctx->stream())); - } - } - NCCL_CHECK(ncclGroupStart()); - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - NCCL_CHECK(ncclAllReduce(tensors[i]->data, tensors[i]->data, ne, ncclFloat, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); - } - NCCL_CHECK(ncclGroupEnd()); - return true; - } - - // For large tensors it's faster to compress them to BF16 for the reduction: - to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); - to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); - - ggml_cuda_pool_alloc tmp[GGML_CUDA_MAX_DEVICES]; - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - tmp[i].pool = &cuda_ctx->pool(); - tmp[i].alloc(ne); - - ggml_cuda_set_device(cuda_ctx->device); - if (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) { - to_bf16(tensors[i]->data, tmp[i].get(), ne, cuda_ctx->stream()); - } else { - CUDA_CHECK(cudaMemsetAsync(tmp[i].get(), 0, ne * sizeof(nv_bfloat16), cuda_ctx->stream())); - } - CUDA_CHECK(cudaGetLastError()); - } - - NCCL_CHECK(ncclGroupStart()); - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - NCCL_CHECK(ncclAllReduce(tmp[i].get(), tmp[i].get(), ne, ncclBfloat16, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); - } - NCCL_CHECK(ncclGroupEnd()); - - for (size_t i = 0; i < n_backends; ++i) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; - - ggml_cuda_set_device(cuda_ctx->device); - to_fp32(tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); - CUDA_CHECK(cudaGetLastError()); - } - - return true; -} -#endif // GGML_USE_NCCL - -static bool ggml_backend_cuda_comm_allreduce_internal( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); -} - -static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { - if (comm_ctx_v == nullptr) { - return false; - } - auto * comm_ctx = static_cast(comm_ctx_v); - switch (comm_ctx->provider) { -#ifdef GGML_USE_NCCL - case GGML_CUDA_ALLREDUCE_NCCL: - return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); -#endif - case GGML_CUDA_ALLREDUCE_INTERNAL: - return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors); - default: - return false; - } -} - -ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { - static std::mutex mutex; - std::lock_guard lock(mutex); - - static std::map>, struct ggml_backend_buffer_type> buft_map; - - std::array tensor_split_arr = {}; - - bool all_zero = tensor_split == nullptr || std::all_of(tensor_split, tensor_split + GGML_CUDA_MAX_DEVICES, [](float x) { return x == 0.0f; }); - if (all_zero) { - tensor_split_arr = ggml_cuda_info().default_tensor_split; - } else { - float split_sum = 0.0f; - for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { - tensor_split_arr[i] = split_sum; - split_sum += tensor_split[i]; - } - for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { - tensor_split_arr[i] /= split_sum; - } - } - - auto it = buft_map.find({main_device, tensor_split_arr}); - if (it != buft_map.end()) { - return &it->second; - } - auto * ctx = new ggml_backend_cuda_split_buffer_type_context{ - main_device, - tensor_split_arr, - GGML_CUDA_NAME + std::to_string(main_device) + "_Split", - }; - - struct ggml_backend_buffer_type buft { - /* .iface = */ ggml_backend_cuda_split_buffer_type_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), main_device), - /* .context = */ ctx, - }; - - auto result = buft_map.emplace(std::make_pair(main_device, tensor_split_arr), buft); - return &result.first->second; -} - -// host buffer type - -static const char * ggml_backend_cuda_host_buffer_type_name(ggml_backend_buffer_type_t buft) { - return GGML_CUDA_NAME "_Host"; - - GGML_UNUSED(buft); -} - -static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) { - return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; -} - -static void ggml_backend_cuda_host_buffer_free_buffer(ggml_backend_buffer_t buffer) { - CUDA_CHECK(cudaFreeHost(buffer->context)); -} - -static void * ggml_cuda_host_malloc(size_t size) { - if (getenv("GGML_CUDA_NO_PINNED") != nullptr) { - return nullptr; - } - - void * ptr = nullptr; - cudaError_t err = cudaMallocHost((void **) &ptr, size); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - GGML_LOG_DEBUG("%s: failed to allocate %.2f MiB of pinned memory: %s\n", __func__, - size / 1024.0 / 1024.0, cudaGetErrorString(err)); - return nullptr; - } - - return ptr; -} - -static ggml_backend_buffer_t ggml_backend_cuda_host_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { - void * ptr = ggml_cuda_host_malloc(size); - - if (ptr == nullptr) { - // fallback to cpu buffer - return ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size); - } - - ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size); - buffer->buft = buft; - buffer->iface.free_buffer = ggml_backend_cuda_host_buffer_free_buffer; - - return buffer; -} - -ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type() { - static struct ggml_backend_buffer_type ggml_backend_cuda_buffer_type_host = { - /* .iface = */ { - /* .get_name = */ ggml_backend_cuda_host_buffer_type_name, - /* .alloc_buffer = */ ggml_backend_cuda_host_buffer_type_alloc_buffer, - /* .get_alignment = */ ggml_backend_cpu_buffer_type()->iface.get_alignment, - /* .get_max_size = */ NULL, // defaults to SIZE_MAX - /* .get_alloc_size = */ ggml_backend_cpu_buffer_type()->iface.get_alloc_size, - /* .is_host = */ ggml_backend_cpu_buffer_type()->iface.is_host, - }, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), 0), - /* .context = */ nullptr, - }; - - return &ggml_backend_cuda_buffer_type_host; -} - -//static bool ggml_backend_buffer_is_cuda_host(ggml_backend_buffer_t buffer) { -// return buffer->buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; -//} - -/// kernels - -typedef void (*ggml_cuda_op_mul_mat_t)( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, - const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, - const int64_t src1_padded_row_size, cudaStream_t stream); - -#ifndef GGML_CUDA_PEER_MAX_BATCH_SIZE -#define GGML_CUDA_PEER_MAX_BATCH_SIZE 128 -#endif // GGML_CUDA_PEER_MAX_BATCH_SIZE - -#define MUL_MAT_SRC1_COL_STRIDE 128 - -static cudaError_t ggml_cuda_cpy_tensor_2d( - void * dst, const struct ggml_tensor * src, int64_t i3, int64_t i2, int64_t i1_low, int64_t i1_high, cudaStream_t stream) { - - const char * src_ptr = (const char *) src->data; - char * dst_ptr = (char *) dst; - - const int64_t ne0 = src->ne[0]; - const int64_t nb0 = src->nb[0]; - const int64_t nb1 = src->nb[1]; - const int64_t nb2 = src->nb[2]; - const int64_t nb3 = src->nb[3]; - const enum ggml_type type = src->type; - const int64_t ts = ggml_type_size(type); - const int64_t bs = ggml_blck_size(type); - const int64_t i1_diff = i1_high - i1_low; - - const char * x = src_ptr + i1_low*nb1 + i2*nb2 + i3*nb3; - if (nb0 == ts && nb1 == ts*ne0/bs) { - return cudaMemcpyAsync(dst_ptr, x, i1_diff*nb1, cudaMemcpyDeviceToDevice, stream); - } else if (nb0 == ts) { - return cudaMemcpy2DAsync(dst_ptr, ts*ne0/bs, x, nb1, ts*ne0/bs, i1_diff, cudaMemcpyDeviceToDevice, stream); - } else { - for (int64_t i1 = 0; i1 < i1_diff; i1++) { - const void * rx = (const void *) ((const char *) x + i1*nb1); - void * rd = (void *) (dst_ptr + i1*ts*ne0/bs); - // pretend the row is a matrix with cols=1 - cudaError_t r = cudaMemcpy2DAsync(rd, ts/bs, rx, nb0, ts/bs, ne0, cudaMemcpyDeviceToDevice, stream); - if (r != cudaSuccess) { - return r; - } - } - return cudaSuccess; - } -} - -struct cublas_force_compute_type { - bool fp32 = false; - bool fp16 = false; -}; - -static const cublas_force_compute_type & ggml_cuda_cublas_get_force_compute_type() { - static const cublas_force_compute_type compute_type = [] { - cublas_force_compute_type result; - - const bool ggml_cuda_force_cublas_compute_32f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F") != nullptr; - const bool ggml_cuda_force_cublas_compute_16f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F") != nullptr; - - GGML_ASSERT(ggml_cuda_force_cublas_compute_16f_env == false || ggml_cuda_force_cublas_compute_32f_env == false); - - if (ggml_cuda_force_cublas_compute_32f_env) { - GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F\n"); - result.fp32 = true; - } else if (ggml_cuda_force_cublas_compute_16f_env) { - GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F\n"); - result.fp16 = true; - } - - return result; - }(); - - return compute_type; -} - -static void ggml_cuda_op_mul_mat_cublas( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, - const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, - const int64_t src1_padded_row_size, cudaStream_t stream) { - - GGML_ASSERT(src0_dd_i != nullptr); - GGML_ASSERT(src1_ddf_i != nullptr); - GGML_ASSERT(dst_dd_i != nullptr); - - const int64_t ne00 = src0->ne[0]; - const int64_t ne10 = src1->ne[0]; - - const int64_t ne0 = dst->ne[0]; - - const int64_t row_diff = row_high - row_low; - - int id = ggml_cuda_get_device(); - - // the main device has a larger memory buffer to hold the results from all GPUs - // ldc == nrows of the matrix that cuBLAS writes into - int64_t ldc = id == ctx.device ? ne0 : row_diff; - - const int cc = ggml_cuda_info().devices[id].cc; - - const bool supports_bf16 = GGML_CUDA_CC_IS_NVIDIA(cc) || GGML_CUDA_CC_IS_AMD(cc) || - (GGML_CUDA_CC_IS_MTHREADS(cc) && cc >= GGML_CUDA_CC_QY2); - - const bool use_fp16 = - src0->type != GGML_TYPE_NVFP4 && - (src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && - ggml_is_contiguous(src0) && - row_diff == src0->ne[1] && - dst->op_params[0] == GGML_PREC_DEFAULT; - - if (supports_bf16 && src0->type == GGML_TYPE_BF16 && ggml_is_contiguous(src0) && row_diff == src0->ne[1]) { - ggml_cuda_pool_alloc src1_as_bf16(ctx.pool(id)); - if (src1->type != GGML_TYPE_BF16) { - const to_bf16_cuda_t to_bf16_cuda = ggml_get_to_bf16_cuda(src1->type); - GGML_ASSERT(to_bf16_cuda != nullptr); - size_t ne = src1_ncols*ne10; - src1_as_bf16.alloc(ne); - to_bf16_cuda(src1_ddf_i, src1_as_bf16.get(), ne, stream); - } - const nv_bfloat16 * src1_ptr = src1->type == GGML_TYPE_BF16 ? (const nv_bfloat16 *) src1_ddf_i : src1_as_bf16.get(); - const nv_bfloat16 * src0_ptr = (const nv_bfloat16 *)src0_dd_i; - ggml_cuda_pool_alloc dst_bf16(ctx.pool(id), row_diff*src1_ncols); - - const float alpha_f32 = 1.0f; - const float beta_f32 = 0.0f; - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha_f32, src0_ptr, CUDA_R_16BF, ne00, - src1_ptr, CUDA_R_16BF, ne10, - &beta_f32, dst_bf16.get(), CUDA_R_16BF, ldc, - CUBLAS_COMPUTE_32F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); - to_fp32_cuda(dst_bf16.get(), dst_dd_i, row_diff*src1_ncols, stream); - } else if (fast_fp16_hardware_available(cc) && use_fp16) { - // convert src0 and src1 to fp16, multiply as fp16, convert dst to fp32 - ggml_cuda_pool_alloc src0_as_f16(ctx.pool(id)); - if (src0->type != GGML_TYPE_F16) { - const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src0->type); - GGML_ASSERT(to_fp16_cuda != nullptr); - size_t ne = row_diff*ne00; - src0_as_f16.alloc(ne); - to_fp16_cuda(src0_dd_i, src0_as_f16.get(), ne, stream); - } - const half * src0_ptr = src0->type == GGML_TYPE_F16 ? (const half *) src0_dd_i : src0_as_f16.get(); - - ggml_cuda_pool_alloc src1_as_f16(ctx.pool(id)); - if (src1->type != GGML_TYPE_F16) { - const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type); - GGML_ASSERT(to_fp16_cuda != nullptr); - size_t ne = src1_ncols*ne10; - src1_as_f16.alloc(ne); - to_fp16_cuda(src1_ddf_i, src1_as_f16.get(), ne, stream); - } - const half * src1_ptr = src1->type == GGML_TYPE_F16 ? (const half *) src1_ddf_i : src1_as_f16.get(); - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - - const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); - - if (!force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) - || GGML_CUDA_CC_IS_RDNA4(cc) - || cc == GGML_CUDA_CC_VOLTA - || force_compute_type.fp32)) - { - const float alpha = 1.0f; - const float beta = 0.0f; - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha, src0_ptr, CUDA_R_16F, ne00, - src1_ptr, CUDA_R_16F, ne10, - &beta, dst_dd_i, CUDA_R_32F, ldc, - CUBLAS_COMPUTE_32F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } else { - ggml_cuda_pool_alloc dst_f16(ctx.pool(id), row_diff*src1_ncols); - - const half alpha_f16 = 1.0f; - const half beta_f16 = 0.0f; - - CUBLAS_CHECK( - cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha_f16, src0_ptr, CUDA_R_16F, ne00, - src1_ptr, CUDA_R_16F, ne10, - &beta_f16, dst_f16.get(), CUDA_R_16F, ldc, - CUBLAS_COMPUTE_16F, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_F16); - to_fp32_cuda(dst_f16.get(), dst_dd_i, row_diff*src1_ncols, stream); - } - } else { - ggml_cuda_pool_alloc src0_ddq_as_f32(ctx.pool(id)); - ggml_cuda_pool_alloc src1_ddq_as_f32(ctx.pool(id)); - - if (src0->type != GGML_TYPE_F32) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src0->type); - GGML_ASSERT(to_fp32_cuda != nullptr); - src0_ddq_as_f32.alloc(row_diff*ne00); - to_fp32_cuda(src0_dd_i, src0_ddq_as_f32.get(), row_diff*ne00, stream); - } - if (src1->type != GGML_TYPE_F32) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src1->type); - GGML_ASSERT(to_fp32_cuda != nullptr); - src1_ddq_as_f32.alloc(src1_ncols*ne10); - to_fp32_cuda(src1_ddf_i, src1_ddq_as_f32.get(), src1_ncols*ne10, stream); - } - - const float * src0_ddf_i = src0->type == GGML_TYPE_F32 ? (const float *) src0_dd_i : src0_ddq_as_f32.get(); - const float * src1_ddf1_i = src1->type == GGML_TYPE_F32 ? (const float *) src1_ddf_i : src1_ddq_as_f32.get(); - - const float alpha = 1.0f; - const float beta = 0.0f; - - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); - CUBLAS_CHECK( - cublasSgemm(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, - row_diff, src1_ncols, ne10, - &alpha, src0_ddf_i, ne00, - src1_ddf1_i, ne10, - &beta, dst_dd_i, ldc)); - } - - GGML_UNUSED_VARS(dst, src1_ddq_i, src1_padded_row_size); -} - -static cudaError_t ggml_cuda_Memcpy2DPeerAsync( - void * dst, int dstDevice, size_t dpitch, void * src, int srcDevice, size_t spitch, size_t width, size_t height, cudaStream_t stream) { - -#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) - // cudaMemcpy2DAsync may fail with copies between vmm pools of different devices - cudaMemcpy3DPeerParms p = {}; - p.dstDevice = dstDevice; - p.dstPtr = make_cudaPitchedPtr(dst, dpitch, dpitch, height); - p.srcDevice = srcDevice; - p.srcPtr = make_cudaPitchedPtr(src, spitch, spitch, height); - p.extent = make_cudaExtent(width, height, 1); - return cudaMemcpy3DPeerAsync(&p, stream); -#else - // HIP does not support cudaMemcpy3DPeerAsync or vmm pools - GGML_UNUSED(dstDevice); - GGML_UNUSED(srcDevice); - return cudaMemcpy2DAsync(dst, dpitch, src, spitch, width, height, cudaMemcpyDeviceToDevice, stream); -#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) -} - -static void ggml_cuda_op_mul_mat( - ggml_backend_cuda_context & ctx, - const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, ggml_cuda_op_mul_mat_t op, - quantize_cuda_t quantize_src1) { - - const int64_t ne00 = src0->ne[0]; - const int64_t ne01 = src0->ne[1]; - const int64_t ne02 = src0->ne[2]; - const int64_t ne03 = src0->ne[3]; - - const int64_t ne10 = src1->ne[0]; - const int64_t ne11 = src1->ne[1]; - const int64_t ne12 = src1->ne[2]; - const int64_t ne13 = src1->ne[3]; - const int64_t nrows1 = ggml_nrows(src1); - - const int64_t ne0 = dst->ne[0]; - const int64_t ne1 = dst->ne[1]; - - // const int64_t nb10 = src1->nb[0]; - const int64_t nb11 = src1->nb[1]; - const int64_t nb12 = src1->nb[2]; - const int64_t nb13 = src1->nb[3]; - - const int64_t nb2 = dst->nb[2]; - const int64_t nb3 = dst->nb[3]; - - ggml_backend_cuda_buffer_context * src1_ctx = (ggml_backend_cuda_buffer_context *) src1->buffer->context; - ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *) dst->buffer->context; - - GGML_ASSERT(src1->type == GGML_TYPE_F32 || (src1->ne[2] == 1 && src1->ne[3] == 1)); - - GGML_ASSERT(ne12 % ne02 == 0); - GGML_ASSERT(ne13 % ne03 == 0); - - const int64_t i02_divisor = ne12 / ne02; - const int64_t i03_divisor = ne13 / ne03; - - const size_t src0_ts = ggml_type_size(src0->type); - const size_t src0_bs = ggml_blck_size(src0->type); - const size_t q8_1_ts = sizeof(block_q8_1); - const size_t q8_1_bs = QK8_1; - - const bool src0_is_contiguous = ggml_is_contiguous(src0); - const bool src1_is_contiguous = ggml_is_contiguous(src1); - - const int64_t src1_padded_col_size = GGML_PAD(ne10, MATRIX_ROW_PADDING); - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); - GGML_ASSERT(!(split && ne02 > 1)); - GGML_ASSERT(!(split && ne03 > 1)); - GGML_ASSERT(!(split && ne02 < ne12)); - GGML_ASSERT(!(split && ne03 < ne13)); - - ggml_tensor_extra_gpu * src0_extra = split ? (ggml_tensor_extra_gpu *) src0->extra : nullptr; - - - std::array tensor_split; - if (split) { - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; - tensor_split = buft_ctx->tensor_split; - } - - struct dev_data { - int cc; - - ggml_cuda_pool_alloc src0_dd_alloc; - ggml_cuda_pool_alloc src1_ddf_alloc; - ggml_cuda_pool_alloc src1_ddq_alloc; - ggml_cuda_pool_alloc dst_dd_alloc; - - char * src0_dd = nullptr; - float * src1_ddf = nullptr; // float - char * src1_ddq = nullptr; // q8_1 - float * dst_dd = nullptr; - - int64_t row_low; - int64_t row_high; - }; - - dev_data dev[GGML_CUDA_MAX_DEVICES]; - - int used_devices = 0; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - dev[id].cc = ggml_cuda_info().devices[id].cc; - - // by default, use all rows - dev[id].row_low = 0; - dev[id].row_high = ne01; - - // for multi GPU, get the row boundaries from tensor split - // and round to mul_mat_q tile sizes - if (split) { - const int64_t rounding = get_row_rounding(tensor_split); - - if (id != 0) { - dev[id].row_low = ne01*tensor_split[id]; - if (dev[id].row_low < ne01) { - dev[id].row_low -= dev[id].row_low % rounding; - } - } - - if (id != ggml_backend_cuda_get_device_count() - 1) { - dev[id].row_high = ne01*tensor_split[id + 1]; - if (dev[id].row_high < ne01) { - dev[id].row_high -= dev[id].row_high % rounding; - } - } - } - } - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { - continue; - } - - used_devices++; - - const bool src1_on_device = id == src1_ctx->device; - const bool dst_on_device = id == dst_ctx->device; - - ggml_cuda_set_device(id); - cudaStream_t stream = ctx.stream(id, 0); - - if (src0_is_contiguous) { - dev[id].src0_dd = split ? (char *) src0_extra->data_device[id] : (char *) src0->data; - } else { - // If src0 is not contiguous it will be copied to a temporary buffer. - // This buffer needs to be cleared entirely because multiple regions will function as padding. - const size_t nbytes_data = ggml_nbytes(src0); - const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); - dev[id].src0_dd = dev[id].src0_dd_alloc.alloc(ctx.pool(id), nbytes_data + nbytes_padding); - CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd, 0, nbytes_data + nbytes_padding, stream)); - } - - // If src0 is on a temporary compute buffer (partial offloading) there may be some padding that needs to be cleared: - if (ne00 % MATRIX_ROW_PADDING != 0 && ggml_is_quantized(src0->type) && ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && src0->view_src == nullptr) { - GGML_ASSERT(ggml_is_contiguously_allocated(src0)); - GGML_ASSERT(!src0->view_src); - const size_t nbytes_data = ggml_row_size(src0->type, (dev[id].row_high - dev[id].row_low)*ne00); - const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); - CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd + nbytes_data, 0, nbytes_padding, stream)); - } - - if (src1_on_device && src1_is_contiguous) { - dev[id].src1_ddf = (float *) src1->data; - } else { - dev[id].src1_ddf = dev[id].src1_ddf_alloc.alloc(ctx.pool(id), ggml_nelements(src1)); - } - - if (quantize_src1) { - size_t src_1_ddq_size = nrows1*src1_padded_col_size*q8_1_ts/q8_1_bs; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - src_1_ddq_size += get_mmq_x_max_host(dev[id].cc)*sizeof(block_q8_1_mmq); - } - dev[id].src1_ddq = dev[id].src1_ddq_alloc.alloc(ctx.pool(id), src_1_ddq_size); - - if (src1_on_device && src1_is_contiguous) { - quantize_src1( - dev[id].src1_ddf, nullptr, dev[id].src1_ddq, src0->type, ne10, - nb11/sizeof(float), nb12/sizeof(float), nb13/sizeof(float), - src1_padded_col_size, ne11, ne12, ne13, stream); - CUDA_CHECK(cudaGetLastError()); - } - } - - if (dst_on_device) { - dev[id].dst_dd = (float *) dst->data; - } else { - const size_t size_dst_ddf = split ? (dev[id].row_high - dev[id].row_low)*ne1 : ggml_nelements(dst); - dev[id].dst_dd = dev[id].dst_dd_alloc.alloc(ctx.pool(id), size_dst_ddf); - } - } - - // if multiple devices are used they need to wait for the main device - // here an event is recorded that signals that the main device has finished calculating the input data - if (split && used_devices > 1) { - ggml_cuda_set_device(ctx.device); - CUDA_CHECK(cudaEventRecord(src0_extra->events[ctx.device][0], ctx.stream())); - } - - const int64_t src1_col_stride = split && used_devices > 1 ? MUL_MAT_SRC1_COL_STRIDE : ne11; - for (int64_t src1_col_0 = 0; src1_col_0 < ne11; src1_col_0 += src1_col_stride) { - const int64_t is = split ? (src1_col_0/src1_col_stride) % GGML_CUDA_MAX_STREAMS : 0; - const int64_t src1_ncols = src1_col_0 + src1_col_stride > ne11 ? ne11 - src1_col_0 : src1_col_stride; - - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { - continue; - } - - const bool src1_on_device = id == src1_ctx->device; - const bool dst_on_device = id == dst_ctx->device; - const int64_t row_diff = dev[id].row_high - dev[id].row_low; - - ggml_cuda_set_device(id); - cudaStream_t stream = ctx.stream(id, is); - - // wait for main GPU data if necessary - if (split && (id != ctx.device || is != 0)) { - CUDA_CHECK(cudaStreamWaitEvent(stream, src0_extra->events[ctx.device][0], 0)); - } - - for (int64_t i0 = 0; i0 < ne13*ne12; ++i0) { - const int64_t i03 = i0 / ne12; - const int64_t i02 = i0 % ne12; - - size_t src1_ddq_i_offset = i0*ne11 * src1_padded_col_size*q8_1_ts/q8_1_bs; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - src1_ddq_i_offset += src1_col_0 * sizeof(block_q8_1_mmq); - } else { - src1_ddq_i_offset += src1_col_0 * src1_padded_col_size*q8_1_ts/q8_1_bs; - } - - // for split tensors the data begins at i0 == i0_offset_low - const size_t nbytes_src0_matrix = ne01*ne00*src0_ts / src0_bs; - char * src0_dd_i = dev[id].src0_dd + ((i03/i03_divisor)*ne02 + (i02/i02_divisor)) * nbytes_src0_matrix; - float * src1_ddf_i = dev[id].src1_ddf + (i0*ne11 + src1_col_0) * ne10; - char * src1_ddq_i = dev[id].src1_ddq + src1_ddq_i_offset; - float * dst_dd_i = dev[id].dst_dd + (i0*ne1 + src1_col_0) * (dst_on_device ? ne0 : row_diff); - - // the main device memory buffer can be on VRAM scratch, with space for all partial results - // in that case an offset on dst_ddf_i is needed - if (id == ctx.device) { - dst_dd_i += dev[id].row_low; // offset is 0 if no tensor split - } - - // copy src0, src1 to device if necessary - if (src1_is_contiguous) { - if (id != ctx.device) { - if (quantize_src1) { - char * src1_ddq_i_source = dev[ctx.device].src1_ddq + src1_ddq_i_offset; - if (quantize_src1 == quantize_mmq_q8_1_cuda) { - const size_t pitch = ne11*sizeof(block_q8_1_mmq); - const size_t width = src1_ncols*sizeof(block_q8_1_mmq); - const size_t height = src1_padded_col_size/(4*QK8_1); - CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync(src1_ddq_i, id, pitch, src1_ddq_i_source, ctx.device, pitch, width, height, stream)); - } else { - CUDA_CHECK(cudaMemcpyPeerAsync( - src1_ddq_i, id, src1_ddq_i_source, ctx.device, src1_ncols*src1_padded_col_size*q8_1_ts/q8_1_bs, stream)); - } - } else { - float * src1_ddf_i_source = (float *) src1->data; - src1_ddf_i_source += (i0*ne11 + src1_col_0) * ne10; - CUDA_CHECK(cudaMemcpyPeerAsync(src1_ddf_i, id, src1_ddf_i_source, ctx.device, - src1_ncols*ne10*sizeof(float), stream)); - } - } - } else if (src1_on_device && !src1_is_contiguous) { - CUDA_CHECK(ggml_cuda_cpy_tensor_2d( - src1_ddf_i, src1, i03, i02, src1_col_0, src1_col_0+src1_ncols, stream)); - } else { - GGML_ABORT("fatal error"); - } - - if (quantize_src1 && !src1_is_contiguous) { - quantize_src1( - src1_ddf_i, nullptr, src1_ddq_i, src0->type, ne10, ne10, ne11*ne10, ne12*ne11*ne10, - src1_padded_col_size, src1_ncols, 1, 1, stream); - CUDA_CHECK(cudaGetLastError()); - } - - if (src1_col_0 == 0 && !src0_is_contiguous && i03 % i03_divisor == 0 && i02 % i02_divisor == 0) { - CUDA_CHECK(ggml_cuda_cpy_tensor_2d( - src0_dd_i, src0, i03/i03_divisor, i02/i02_divisor, dev[id].row_low, dev[id].row_high, stream)); - } - - // do the computation - op(ctx, src0, src1, dst, src0_dd_i, src1_ddf_i, src1_ddq_i, dst_dd_i, - dev[id].row_low, dev[id].row_high, src1_ncols, src1_padded_col_size, stream); - CUDA_CHECK(cudaGetLastError()); - - // copy dst to host or other device if necessary - if (!dst_on_device) { - void * dst_off_device = dst->data; - if (split) { - // src0 = weight matrix is saved as a transposed matrix for better memory layout. - // dst is NOT transposed. - // The outputs of matrix matrix multiplications can therefore NOT simply be concatenated for >1 GPU. - // Instead they need to be copied to the correct slice in ne0 = dst row index. - // If dst is a vector with ne0 == 1 then you don't have to do this but it still produces correct results. - float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); - GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); - dhf_dst_i += src1_col_0*ne0 + dev[id].row_low; - CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync( - dhf_dst_i, ctx.device, ne0*sizeof(float), dst_dd_i, id, row_diff*sizeof(float), row_diff*sizeof(float), src1_ncols, stream)); - } else { - float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); - GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); - dhf_dst_i += src1_col_0*ne0; - CUDA_CHECK(cudaMemcpyAsync(dhf_dst_i, dst_dd_i, src1_ncols*ne0*sizeof(float), cudaMemcpyDeviceToDevice, stream)); - } - } - - // add event for the main device to wait on until other device is done - if (split && (id != ctx.device || is != 0)) { - CUDA_CHECK(cudaEventRecord(src0_extra->events[id][is], stream)); - } - } - } - } - - // main device waits for all other devices to be finished - if (split && ggml_backend_cuda_get_device_count() > 1) { - int64_t is_max = (ne11 + MUL_MAT_SRC1_COL_STRIDE - 1) / MUL_MAT_SRC1_COL_STRIDE; - is_max = is_max <= GGML_CUDA_MAX_STREAMS ? is_max : GGML_CUDA_MAX_STREAMS; - - ggml_cuda_set_device(ctx.device); - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - if (dev[id].row_low == dev[id].row_high) { - continue; - } - for (int64_t is = 0; is < is_max; ++is) { - CUDA_CHECK(cudaStreamWaitEvent(ctx.stream(), src0_extra->events[id][is], 0)); - } - } - } -} - -static __global__ void k_compute_batched_ptrs( - const void * src0_as_f16, const void * src1_as_f16, char * dst, - const void ** ptrs_src, void ** ptrs_dst, - int64_t ne12, int64_t ne13, - int64_t ne23, - size_t nb02, size_t nb03, - size_t nb12, size_t nb13, - size_t nbd2, size_t nbd3, - int64_t r2, int64_t r3) { - const int64_t i13 = blockIdx.x * blockDim.x + threadIdx.x; - const int64_t i12 = blockIdx.y * blockDim.y + threadIdx.y; - - if (i13 >= ne13 || i12 >= ne12) { - return; - } - - const int64_t i03 = i13 / r3; - const int64_t i02 = i12 / r2; - - ptrs_src[0*ne23 + i12 + i13*ne12] = (const char *) src0_as_f16 + i02*nb02 + i03*nb03; - ptrs_src[1*ne23 + i12 + i13*ne12] = (const char *) src1_as_f16 + i12*nb12 + i13*nb13; - ptrs_dst[0*ne23 + i12 + i13*ne12] = ( char *) dst + i12*nbd2 + i13*nbd3; -} - -// Type traits for mapping ggml types to CUDA/cuBLAS types -template -struct batched_mul_mat_traits; - -template<> -struct batched_mul_mat_traits { - using cuda_type = float; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; - static inline const cudaDataType_t data_type = CUDA_R_32F; - static inline const ggml_type ggml_type_val = GGML_TYPE_F32; - static inline const float alpha = 1.0f; - static inline const float beta = 0.0f; - static inline const void* get_alpha() { static const float val = alpha; return &val; } - static inline const void* get_beta() { static const float val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp32_nc_cuda(src_type); } -}; - -template<> -struct batched_mul_mat_traits { - using cuda_type = nv_bfloat16; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; - static inline const cudaDataType_t data_type = CUDA_R_16BF; - static inline const ggml_type ggml_type_val = GGML_TYPE_BF16; - static inline const float alpha = 1.0f; - static inline const float beta = 0.0f; - static inline const void* get_alpha() { static const float val = alpha; return &val; } - static inline const void* get_beta() { static const float val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_bf16_nc_cuda(src_type); } -}; - -template<> -struct batched_mul_mat_traits { - using cuda_type = half; - static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_16F; - static inline const cudaDataType_t data_type = CUDA_R_16F; - static inline const ggml_type ggml_type_val = GGML_TYPE_F16; - static inline const half alpha = 1.0; - static inline const half beta = 0.0; - static inline const void* get_alpha() { static const half val = alpha; return &val; } - static inline const void* get_beta() { static const half val = beta; return &val; } - static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp16_nc_cuda(src_type); } -}; - -template -static void ggml_cuda_mul_mat_batched_cublas_impl(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - using traits = batched_mul_mat_traits; - using cuda_t = typename traits::cuda_type; - - GGML_ASSERT(!ggml_is_transposed(src0)); - GGML_ASSERT(!ggml_is_transposed(src1)); - GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft)); - GGML_ASSERT(src0->type == src0_type); - GGML_ASSERT(ggml_is_contiguous(dst)); - - // Byte offsets and tensor dimensions are currently used in an inconsistent way for dst. - // As long as dst is contiguous this does not matter though. - - GGML_TENSOR_BINARY_OP_LOCALS - - const int64_t ne_dst = ggml_nelements(dst); - cudaStream_t main_stream = ctx.stream(); - CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(), main_stream)); - - float * dst_ddf = (float *) dst->data; - const size_t ts_src1 = ggml_type_size(src1->type); - GGML_ASSERT(nb10 == ts_src1); - int64_t s11 = nb11 / ts_src1; - int64_t s12 = nb12 / ts_src1; - int64_t s13 = nb13 / ts_src1; - - const cuda_t * src0_ptr = nullptr; - const cuda_t * src1_ptr = nullptr; - - ggml_cuda_pool_alloc src0_alloc(ctx.pool()); - ggml_cuda_pool_alloc src1_alloc(ctx.pool()); - - bool is_src0_cont_2 = ggml_is_contiguous_2(src0); - bool is_src1_cont_2 = ggml_is_contiguous_2(src1); - - // Handle src0 - src0_ptr = (const cuda_t *) src0->data; - - // Handle src1 - convert if necessary - if (src1->type == src0_type) { - src1_ptr = (const cuda_t *) src1->data; - } else { - // Convert src1 to target type using traits conversion functions - const int64_t ne_src1 = ggml_nelements(src1); - src1_alloc.alloc(ne_src1); - - const auto convert_func = traits::get_nc_converter(src1->type); - GGML_ASSERT(convert_func != nullptr); - convert_func(src1->data, src1_alloc.get(), ne10, ne11, ne12, ne13, s11, s12, s13, main_stream); - src1_ptr = src1_alloc.get(); - s11 = ne10; - s12 = ne11*s11; - s13 = ne12*s12; - - is_src1_cont_2 = true; - } - - // Setup destination buffer - ggml_cuda_pool_alloc dst_temp(ctx.pool()); - char * dst_t; - size_t nbd2 = dst->nb[2]; - size_t nbd3 = dst->nb[3]; - - cublasComputeType_t cu_compute_type = traits::compute_type; - cudaDataType_t cu_data_type = traits::data_type; - cudaDataType_t cu_data_type_a = traits::data_type; - cudaDataType_t cu_data_type_b = traits::data_type; - const void * alpha = traits::get_alpha(); - const void * beta = traits::get_beta(); - - const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); - - int id = ggml_cuda_get_device(); - const int cc = ggml_cuda_info().devices[id].cc; - static constexpr bool is_src0_type_f16 = src0_type == GGML_TYPE_F16; - - // bf16 and fp32 are already being computed in fp32 (ensure it using static_assert), - // so checking necessity of forced fp32 only for fp16 src0_type - static_assert(is_src0_type_f16 || traits::compute_type == CUBLAS_COMPUTE_32F); - - const bool need_compute_32f = is_src0_type_f16 && !force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) - || GGML_CUDA_CC_IS_RDNA4(cc) - || cc == GGML_CUDA_CC_VOLTA - || force_compute_type.fp32); - - if (dst->op_params[0] == GGML_PREC_DEFAULT && !need_compute_32f) { - if constexpr (src0_type == GGML_TYPE_F32) { - dst_t = (char *) dst_ddf; // Direct F32 output - } else { - dst_t = (char *) dst_temp.alloc(ne_dst); - nbd2 /= sizeof(float) / sizeof(cuda_t); - nbd3 /= sizeof(float) / sizeof(cuda_t); - } - } else { - dst_t = (char *) dst_ddf; - cu_compute_type = batched_mul_mat_traits::compute_type; - cu_data_type = batched_mul_mat_traits::data_type; - alpha = batched_mul_mat_traits::get_alpha(); - beta = batched_mul_mat_traits::get_beta(); - } - - GGML_ASSERT(ne12 % ne02 == 0); - GGML_ASSERT(ne13 % ne03 == 0); - - // broadcast factors - const int64_t r2 = ne12/ne02; - const int64_t r3 = ne13/ne03; - - if (r2 == 1 && r3 == 1 && is_src0_cont_2 && is_src1_cont_2) { - // with a [0, 2, 1, 3] perm. and ne02==1 the matrix strides need to be determined from dim 3: - const int64_t sma = ne02 == 1 ? nb03/nb00 : nb02/nb00; - const int64_t smb = ne12 == 1 ? s13 : s12; - - // there is no broadcast and src0, src1 are contiguous across dims 2, 3 - // use cublasGemmStridedBatchedEx - CUBLAS_CHECK( - cublasGemmStridedBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, - ne01, ne11, ne10, - alpha, src0_ptr, cu_data_type_a, nb01/nb00, sma, // strideA - src1_ptr, cu_data_type_b, s11, smb, // strideB - beta, dst_t, cu_data_type, ne0, ne1*ne0, // strideC - ne12*ne13, - cu_compute_type, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } else { - // use cublasGemmBatchedEx - const int64_t ne23 = ne12*ne13; - - ggml_cuda_pool_alloc ptrs_src(ctx.pool(), 2*ne23); - ggml_cuda_pool_alloc< void *> ptrs_dst(ctx.pool(), 1*ne23); - - size_t src1_stride_size = sizeof(cuda_t); - - const int threads_x = 16; - const int threads_y = 16; - dim3 block_dims(threads_x, threads_y); - - dim3 grid_dims( - (ne13 + threads_x - 1) / threads_x, - (ne12 + threads_y - 1) / threads_y - ); - k_compute_batched_ptrs<<>>( - src0_ptr, src1_ptr, dst_t, - ptrs_src.get(), ptrs_dst.get(), - ne12, ne13, - ne23, - nb02, nb03, - (src1->type == src0_type) ? nb12 : s12*src1_stride_size, - (src1->type == src0_type) ? nb13 : s13*src1_stride_size, - nbd2, nbd3, - r2, r3); - - CUDA_CHECK(cudaGetLastError()); - - CUBLAS_CHECK( - cublasGemmBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, - ne01, ne11, ne10, - alpha, (const void **) (ptrs_src.get() + 0*ne23), cu_data_type_a, nb01/nb00, - (const void **) (ptrs_src.get() + 1*ne23), cu_data_type_b, s11, - beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne0, - ne23, - cu_compute_type, - CUBLAS_GEMM_DEFAULT_TENSOR_OP)); - } - - // Convert output back to F32 if needed - if (dst->op_params[0] == GGML_PREC_DEFAULT && cu_data_type != CUDA_R_32F) { - const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(traits::ggml_type_val); - to_fp32_cuda(dst_temp.get(), dst_ddf, ne_dst, main_stream); - } -} - -static void ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - GGML_ASSERT(src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16 || src0->type == GGML_TYPE_F32); - - switch (src0->type) { - case GGML_TYPE_F32: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - case GGML_TYPE_BF16: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - case GGML_TYPE_F16: - ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); - break; - default: - GGML_ABORT("Unsupported type"); - } -} - -static bool ggml_cuda_should_fuse_mul_mat(const ggml_tensor * ffn_up, - const ggml_tensor * ffn_gate, - const ggml_tensor * glu, - const ggml_tensor * ffn_up_bias = nullptr, - const ggml_tensor * ffn_gate_bias = nullptr) { - const bool has_bias = ffn_up_bias != nullptr || ffn_gate_bias != nullptr; - - if (has_bias && (!ffn_up_bias || !ffn_gate_bias)) { - return false; - } - - const bool is_mul_mat = ffn_up->op == GGML_OP_MUL_MAT && ffn_gate->op == GGML_OP_MUL_MAT && glu->op == GGML_OP_GLU; - const bool is_mul_mat_id = ffn_up->op == GGML_OP_MUL_MAT_ID && ffn_gate->op == GGML_OP_MUL_MAT_ID && glu->op == GGML_OP_GLU; - - GGML_ASSERT(ffn_up && ffn_gate && glu); - - if (!is_mul_mat && !is_mul_mat_id) { - return false; - } - - const ggml_op expected_bias_op = is_mul_mat ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (has_bias) { - if (ffn_up_bias->op != expected_bias_op || ffn_gate_bias->op != expected_bias_op) { - return false; - } - - if (glu->src[0] != ffn_gate_bias || glu->src[1] != ffn_up_bias) { - return false; - } - - if (expected_bias_op == GGML_OP_ADD) { - const bool up_has_mul = ffn_up_bias->src[0] == ffn_up || ffn_up_bias->src[1] == ffn_up; - const bool gate_has_mul = ffn_gate_bias->src[0] == ffn_gate || ffn_gate_bias->src[1] == ffn_gate; - if (!up_has_mul || !gate_has_mul) { - return false; - } - } else { // GGML_OP_ADD_ID - if (ffn_up_bias->src[0] != ffn_up || ffn_gate_bias->src[0] != ffn_gate) { - return false; - } - if (ffn_up_bias->src[2] != ffn_up->src[2] || ffn_gate_bias->src[2] != ffn_gate->src[2]) { - return false; - } - } - } else { - if (glu->src[0] != ffn_gate && glu->src[1] != ffn_up) { - return false; - } - } - - if (ffn_up->src[0]->type != ffn_gate->src[0]->type || !ggml_are_same_shape(ffn_up->src[0], ffn_gate->src[0]) || - !ggml_are_same_stride(ffn_up->src[0], ffn_gate->src[0])) { - return false; - } - - if (ffn_up->src[1] != ffn_gate->src[1]) { - return false; - } - - if (ffn_up->src[2] && (ffn_up->src[2] != ffn_gate->src[2])) { - return false; - } - - static constexpr std::array valid_glu_ops = { GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU, GGML_GLU_OP_SWIGLU_OAI }; - - if (std::find(valid_glu_ops.begin(), valid_glu_ops.end(), ggml_get_glu_op(glu)) == valid_glu_ops.end()) { - return false; - } - - if (const bool swapped = ggml_get_op_params_i32(glu, 1); swapped) { - return false; - } - - const bool split = ggml_backend_buft_is_cuda_split(ffn_up->src[0]->buffer->buft) || - ggml_backend_buft_is_cuda_split(ffn_gate->src[0]->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - return true; -} - -static bool ggml_cuda_should_fuse_mul_mat_vec_f(const ggml_tensor * tensor) { - ggml_tensor * src0 = tensor->src[0]; - ggml_tensor * src1 = tensor->src[1]; - const ggml_tensor * dst = tensor; - - const bool is_mul_mat_id = tensor->op == GGML_OP_MUL_MAT_ID; - - bool use_mul_mat_vec_f = - (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) && - src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, is_mul_mat_id ? src1->ne[2] : src1->ne[1]); - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || - ggml_backend_buft_is_cuda_split(src1->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - //we only support fusion for ncols_dst = 1 - if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { - return false; - } - - if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { - return false; - } - - - return use_mul_mat_vec_f; -} - -static bool ggml_cuda_should_fuse_mul_mat_vec_q(const ggml_tensor * tensor) { - ggml_tensor * src0 = tensor->src[0]; - ggml_tensor * src1 = tensor->src[1]; - const ggml_tensor * dst = tensor; - - const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && - ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && - src0->view_src; - - bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear && src1->type == GGML_TYPE_F32 && - dst->type == GGML_TYPE_F32 && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; - - // fusion is not universally faster on Pascal - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - if (cc <= GGML_CUDA_CC_PASCAL) { - return false; - } - //we only support fusion for ncols_dst = 1 - if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { - return false; - } - - if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { - return false; - } - - - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || - ggml_backend_buft_is_cuda_split(src1->buffer->buft); - - //TODO: add support for fusion for split buffers - if (split) { - return false; - } - - return use_mul_mat_vec_q; -} - -static void ggml_cuda_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { - const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); - - // If src0 is a temporary compute buffer it may have some padding that needs to be cleared for mul_mat_vec_q or mul_mat_q. - // But if src0 is also a view of another tensor then this cannot be done safely because it may overwrite valid tensor data. - // Therefore, in such cases use cuBLAS. - const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE - && ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && src0->view_src; - - bool use_mul_mat_vec_f = (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - bool use_mul_mat_f = !ggml_is_quantized(src0->type) - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32 - && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; - bool use_mul_mat_q = ggml_is_quantized(src0->type) && !bad_padding_clear - && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; - - bool any_gpus_with_slow_fp16 = false; - - if (split) { - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; - auto & tensor_split = buft_ctx->tensor_split; - for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { - // skip devices that are not going to do any work: - if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { - continue; - } - - const int cc = ggml_cuda_info().devices[id].cc; - const int warp_size = ggml_cuda_info().devices[id].warp_size; - use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); - use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); - any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); - } - } else { - const int cc = ggml_cuda_info().devices[ctx.device].cc; - const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size; - use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); - use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); - use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); - any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); - } - - // debug helpers - //printf("src0: %8d %8d %8d %8d\n", src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3]); - //printf(" %8d %8d %8d %8d\n", src0->nb[0], src0->nb[1], src0->nb[2], src0->nb[3]); - //printf("src1: %8d %8d %8d %8d\n", src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3]); - //printf(" %8d %8d %8d %8d\n", src1->nb[0], src1->nb[1], src1->nb[2], src1->nb[3]); - //printf("src0 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src0), ggml_is_transposed(src0), ggml_type_name(src0->type), src0->name); - //printf("src1 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src1), ggml_is_transposed(src1), ggml_type_name(src1->type), src1->name); - - //TODO update for generic tensor parallelism - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - bool use_batched_cublas_f16 = src0->type == GGML_TYPE_F16 && (src1->type == GGML_TYPE_F16 || !any_gpus_with_slow_fp16); - bool use_batched_cublas_bf16 = src0->type == GGML_TYPE_BF16 && bf16_mma_hardware_available(cc); - bool use_batched_cublas_f32 = src0->type == GGML_TYPE_F32; - - if (!split && use_mul_mat_vec_f) { - // the custom F16 vector kernel can be used over batched cuBLAS GEMM - // but this is only faster for GPUs without tensor cores or with a thin src0 matrix (particularly KQV in attention) - ggml_cuda_mul_mat_vec_f(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_f) { - ggml_cuda_mul_mat_f(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_vec_q) { - ggml_cuda_mul_mat_vec_q(ctx, src0, src1, nullptr, dst); - } else if (!split && use_mul_mat_q) { - ggml_cuda_mul_mat_q(ctx, src0, src1, nullptr, dst); - } else if (!split && (use_batched_cublas_f16 || use_batched_cublas_bf16 || use_batched_cublas_f32) - && !ggml_is_transposed(src0) && !ggml_is_transposed(src1) && src1->ne[2]*src1->ne[3] > 1) { - // general KQ + KQV multi-batch without FlashAttention - ggml_cuda_mul_mat_batched_cublas(ctx, src0, src1, dst); - } else if (use_mul_mat_vec_f) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_f, nullptr); - } else if (use_mul_mat_vec_q) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_q, quantize_row_q8_1_cuda); - } else if (use_mul_mat_q) { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_q, quantize_mmq_q8_1_cuda); - } else { - ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_cublas, nullptr); - } -} - -static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor * dst) { - const ggml_tensor * src0 = dst->src[0]; - const ggml_tensor * src1 = dst->src[1]; - const ggml_tensor * ids = dst->src[2]; - - GGML_ASSERT(src1->type == GGML_TYPE_F32); - GGML_ASSERT(dst->type == GGML_TYPE_F32); - GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft) && "mul_mat_id does not support split buffers"); - - GGML_TENSOR_BINARY_OP_LOCALS - - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - - // [TAG_MUL_MAT_ID_CUDA_GRAPHS] - if (src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { - static_assert(MMVQ_MAX_BATCH_SIZE == MMVF_MAX_BATCH_SIZE); - if (ne2 <= MMVQ_MAX_BATCH_SIZE) { - if (ggml_is_quantized(src0->type)) { - const int mmvq_mmid_max = get_mmvq_mmid_max_batch(src0->type, cc); - if (ne2 <= mmvq_mmid_max) { - ggml_cuda_mul_mat_vec_q(ctx, src0, src1, ids, dst); - return; - } - } else { - if (GGML_CUDA_CC_IS_AMD(cc)) { - ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst); - return; - } - } - } - - if (ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) { - ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst); - return; - } - - if (ggml_cuda_should_use_mmf(src0->type, cc, WARP_SIZE, src0->ne, src0->nb, src1->ne[2], /*mul_mat_id=*/true)) { - ggml_cuda_mul_mat_f(ctx, src0, src1, ids, dst); - return; - } - } - - // note: this path should not be reached when recording CUDA graphs, because it requires stream synchronization - // TODO: add asserts to verify this. should work with CUDA, HIP, etc. - cudaStream_t stream = ctx.stream(); - - GGML_ASSERT(nb12 % nb11 == 0); - GGML_ASSERT(nb2 % nb1 == 0); - - const ggml_type type_src1_sorted = (src0->type == GGML_TYPE_F16 && !fast_fp16_hardware_available(cc)) - || ggml_is_quantized(src0->type) ? GGML_TYPE_F32 : src0->type; - const ggml_type type_dst_sorted = GGML_TYPE_F32; - const size_t ts_src1_sorted = ggml_type_size(type_src1_sorted); - const size_t ts_dst_sorted = ggml_type_size(type_dst_sorted); - - const int64_t n_expert_used = ids->ne[0]; - const int64_t ne_get_rows = ne12 * n_expert_used; - - std::vector ids_to_sorted_host; - ids_to_sorted_host.reserve(2*ne_get_rows); - std::vector ids_from_sorted_host(ne_get_rows); - - ggml_cuda_pool_alloc ids_buf_dev(ctx.pool(), 2*ne_get_rows); - - std::vector tokens_per_expert(ne02); - - ggml_cuda_pool_alloc src1_sorted(ctx.pool(), ne12*n_expert_used*ne10*ts_src1_sorted); - ggml_cuda_pool_alloc dst_sorted(ctx.pool(), ne2 *n_expert_used* ne0*ts_dst_sorted); - - std::vector ids_host(ggml_nbytes(ids)); - CUDA_CHECK(cudaMemcpyAsync(ids_host.data(), ids->data, ggml_nbytes(ids), cudaMemcpyDeviceToHost, stream)); - CUDA_CHECK(cudaStreamSynchronize(stream)); - - for (int64_t i02 = 0; i02 < ne02; ++i02) { // expert matrices - for (int64_t i12 = 0; i12 < ne12; ++i12) { // tokens - for (int64_t iex = 0; iex < n_expert_used; ++iex) { - const int32_t expert_to_use = *(const int32_t *)(ids_host.data() + i12*ids->nb[1] + iex*ids->nb[0]); - assert(expert_to_use >= 0 && expert_to_use < ne02); - if (expert_to_use == i02) { - ids_from_sorted_host[i12*n_expert_used + iex] = ids_to_sorted_host.size(); - ids_to_sorted_host.push_back(i12*ne11 + iex % ne11); - tokens_per_expert[i02]++; - break; - } - } - } - } - GGML_ASSERT(ids_to_sorted_host.size() == size_t(ne_get_rows)); - - ids_to_sorted_host.insert(ids_to_sorted_host.end(), ids_from_sorted_host.begin(), ids_from_sorted_host.end()); - - CUDA_CHECK(cudaMemcpyAsync(ids_buf_dev.ptr, ids_to_sorted_host.data(), 2*ne_get_rows*sizeof(int32_t), cudaMemcpyHostToDevice, stream)); - CUDA_CHECK(cudaStreamSynchronize(stream)); - - const int32_t * ids_to_sorted = ids_buf_dev.ptr + 0*ne_get_rows; - const int32_t * ids_from_sorted = ids_buf_dev.ptr + 1*ne_get_rows; - - get_rows_cuda(src1->data, src1->type, ids_to_sorted, src1_sorted.ptr, type_src1_sorted, - ne10, nb11, nb12, nb13, - ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), - ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, stream); - CUDA_CHECK(cudaGetLastError()); - - char * src1_data_cur = (char *) src1_sorted.ptr; - char * dst_data_cur = (char *) dst_sorted.ptr; - for (int64_t i02 = 0; i02 < ne02; ++i02) { - if (tokens_per_expert[i02] == 0) { - continue; - } - - ggml_tensor src0_slice = *src0; - src0_slice.ne[2] = 1; - src0_slice.nb[3] = src0_slice.nb[2]; - src0_slice.op = GGML_OP_VIEW; - src0_slice.view_src = dst->src[0]; // non-const pointer to src0 - src0_slice.data = (char *) src0->data + i02*nb02; - - ggml_tensor src1_slice; - memset(&src1_slice, 0, sizeof(src1_slice)); - src1_slice.buffer = src1->buffer; - src1_slice.type = type_src1_sorted; - src1_slice.ne[0] = ne10; - src1_slice.ne[1] = tokens_per_expert[i02]; - src1_slice.ne[2] = 1; - src1_slice.ne[3] = 1; - src1_slice.nb[0] = ts_src1_sorted; - src1_slice.nb[1] = src1_slice.ne[0] * src1_slice.nb[0]; - src1_slice.nb[2] = src1_slice.ne[1] * src1_slice.nb[1]; - src1_slice.nb[3] = src1_slice.ne[2] * src1_slice.nb[2]; - src1_slice.data = src1_data_cur; - - ggml_tensor dst_slice; - memset(&dst_slice, 0, sizeof(dst_slice)); - dst_slice.buffer = dst->buffer; - dst_slice.type = type_dst_sorted; - dst_slice.ne[0] = ne0; - dst_slice.ne[1] = tokens_per_expert[i02]; - dst_slice.ne[2] = 1; - dst_slice.ne[3] = 1; - dst_slice.nb[0] = ts_dst_sorted; - dst_slice.nb[1] = dst_slice.ne[0] * dst_slice.nb[0]; - dst_slice.nb[2] = dst_slice.ne[1] * dst_slice.nb[1]; - dst_slice.nb[3] = dst_slice.ne[2] * dst_slice.nb[2]; - dst_slice.data = dst_data_cur; - - ggml_cuda_mul_mat(ctx, &src0_slice, &src1_slice, &dst_slice); - CUDA_CHECK(cudaGetLastError()); - - src1_data_cur += src1_slice.nb[2]; - dst_data_cur += dst_slice.nb[2]; - } - - get_rows_cuda(dst_sorted.ptr, type_dst_sorted, ids_from_sorted, dst->data, dst->type, - ne0, ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, - ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), - nb1, nb2, nb3, stream); -} - -static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct ggml_tensor * dst) { - switch (dst->op) { - case GGML_OP_ARGMAX: - ggml_cuda_argmax(ctx, dst); - break; - case GGML_OP_COUNT_EQUAL: - ggml_cuda_count_equal(ctx, dst); - break; - case GGML_OP_REPEAT: - ggml_cuda_op_repeat(ctx, dst); - break; - case GGML_OP_REPEAT_BACK: - ggml_cuda_op_repeat_back(ctx, dst); - break; - case GGML_OP_GET_ROWS: - ggml_cuda_op_get_rows(ctx, dst); - break; - case GGML_OP_GET_ROWS_BACK: - ggml_cuda_op_get_rows_back(ctx, dst); - break; - case GGML_OP_SET_ROWS: - ggml_cuda_op_set_rows(ctx, dst); - break; - case GGML_OP_SET: - ggml_cuda_op_set(ctx, dst); - break; - case GGML_OP_DUP: - ggml_cuda_dup(ctx, dst); - break; - case GGML_OP_CPY: - ggml_cuda_cpy(ctx, dst->src[0], dst->src[1]); - break; - case GGML_OP_CONT: - ggml_cuda_dup(ctx, dst); - break; - case GGML_OP_ADD: - case GGML_OP_ADD1: // TODO: more efficient implementation - ggml_cuda_op_add(ctx, dst); - break; - case GGML_OP_ADD_ID: - ggml_cuda_op_add_id(ctx, dst); - break; - case GGML_OP_SUB: - ggml_cuda_op_sub(ctx, dst); - break; - case GGML_OP_ACC: - ggml_cuda_op_acc(ctx, dst); - break; - case GGML_OP_MUL: - ggml_cuda_op_mul(ctx, dst); - break; - case GGML_OP_DIV: - ggml_cuda_op_div(ctx, dst); - break; - case GGML_OP_UNARY: - switch (ggml_get_unary_op(dst)) { - case GGML_UNARY_OP_ABS: - ggml_cuda_op_abs(ctx, dst); - break; - case GGML_UNARY_OP_SGN: - ggml_cuda_op_sgn(ctx, dst); - break; - case GGML_UNARY_OP_NEG: - ggml_cuda_op_neg(ctx, dst); - break; - case GGML_UNARY_OP_STEP: - ggml_cuda_op_step(ctx, dst); - break; - case GGML_UNARY_OP_GELU: - ggml_cuda_op_gelu(ctx, dst); - break; - case GGML_UNARY_OP_SILU: - ggml_cuda_op_silu(ctx, dst); - break; - case GGML_UNARY_OP_GELU_ERF: - ggml_cuda_op_gelu_erf(ctx, dst); - break; - case GGML_UNARY_OP_GELU_QUICK: - ggml_cuda_op_gelu_quick(ctx, dst); - break; - case GGML_UNARY_OP_TANH: - ggml_cuda_op_tanh(ctx, dst); - break; - case GGML_UNARY_OP_RELU: - ggml_cuda_op_relu(ctx, dst); - break; - case GGML_UNARY_OP_SIGMOID: - ggml_cuda_op_sigmoid(ctx, dst); - break; - case GGML_UNARY_OP_HARDSIGMOID: - ggml_cuda_op_hardsigmoid(ctx, dst); - break; - case GGML_UNARY_OP_HARDSWISH: - ggml_cuda_op_hardswish(ctx, dst); - break; - case GGML_UNARY_OP_EXP: - ggml_cuda_op_exp(ctx, dst); - break; - case GGML_UNARY_OP_ELU: - ggml_cuda_op_elu(ctx, dst); - break; - case GGML_UNARY_OP_XIELU: - ggml_cuda_op_xielu(ctx, dst); - break; - case GGML_UNARY_OP_FLOOR: - ggml_cuda_op_floor(ctx, dst); - break; - case GGML_UNARY_OP_CEIL: - ggml_cuda_op_ceil(ctx, dst); - break; - case GGML_UNARY_OP_ROUND: - ggml_cuda_op_round(ctx, dst); - break; - case GGML_UNARY_OP_TRUNC: - ggml_cuda_op_trunc(ctx, dst); - break; - case GGML_UNARY_OP_EXPM1: - ggml_cuda_op_expm1(ctx, dst); - break; - case GGML_UNARY_OP_SOFTPLUS: - ggml_cuda_op_softplus(ctx, dst); - break; - default: - return false; - } - break; - case GGML_OP_GLU: - switch (ggml_get_glu_op(dst)) { - case GGML_GLU_OP_REGLU: - ggml_cuda_op_reglu(ctx, dst); - break; - case GGML_GLU_OP_GEGLU: - ggml_cuda_op_geglu(ctx, dst); - break; - case GGML_GLU_OP_SWIGLU: - ggml_cuda_op_swiglu(ctx, dst); - break; - case GGML_GLU_OP_SWIGLU_OAI: - ggml_cuda_op_swiglu_oai(ctx, dst); - break; - case GGML_GLU_OP_GEGLU_ERF: - ggml_cuda_op_geglu_erf(ctx, dst); - break; - case GGML_GLU_OP_GEGLU_QUICK: - ggml_cuda_op_geglu_quick(ctx, dst); - break; - default: - return false; - } - break; - case GGML_OP_NORM: - ggml_cuda_op_norm(ctx, dst); - break; - case GGML_OP_GROUP_NORM: - ggml_cuda_op_group_norm(ctx, dst); - break; - case GGML_OP_L2_NORM: - ggml_cuda_op_l2_norm(ctx, dst); - break; - case GGML_OP_CONCAT: - ggml_cuda_op_concat(ctx, dst); - break; - case GGML_OP_UPSCALE: - ggml_cuda_op_upscale(ctx, dst); - break; - case GGML_OP_PAD: - ggml_cuda_op_pad(ctx, dst); - break; - case GGML_OP_PAD_REFLECT_1D: - ggml_cuda_op_pad_reflect_1d(ctx, dst); - break; - case GGML_OP_ARANGE: - ggml_cuda_op_arange(ctx, dst); - break; - case GGML_OP_TIMESTEP_EMBEDDING: - ggml_cuda_op_timestep_embedding(ctx, dst); - break; - case GGML_OP_LEAKY_RELU: - ggml_cuda_op_leaky_relu(ctx, dst); - break; - case GGML_OP_SILU_BACK: - ggml_cuda_op_silu_back(ctx, dst); - break; - case GGML_OP_RMS_NORM: - ggml_cuda_op_rms_norm(ctx, dst); - break; - case GGML_OP_RMS_NORM_BACK: - ggml_cuda_op_rms_norm_back(ctx, dst); - break; - case GGML_OP_MUL_MAT: - ggml_cuda_mul_mat(ctx, dst->src[0], dst->src[1], dst); - break; - case GGML_OP_MUL_MAT_ID: - ggml_cuda_mul_mat_id(ctx, dst); - break; - case GGML_OP_OUT_PROD: - ggml_cuda_out_prod(ctx, dst); - break; - case GGML_OP_SCALE: - ggml_cuda_op_scale(ctx, dst); - break; - case GGML_OP_SQR: - ggml_cuda_op_sqr(ctx, dst); - break; - case GGML_OP_SQRT: - ggml_cuda_op_sqrt(ctx, dst); - break; - case GGML_OP_SIN: - ggml_cuda_op_sin(ctx, dst); - break; - case GGML_OP_COS: - ggml_cuda_op_cos(ctx, dst); - break; - case GGML_OP_CLAMP: - ggml_cuda_op_clamp(ctx, dst); - break; - case GGML_OP_LOG: - ggml_cuda_op_log(ctx, dst); - break; - case GGML_OP_NONE: - case GGML_OP_RESHAPE: - case GGML_OP_VIEW: - case GGML_OP_PERMUTE: - case GGML_OP_TRANSPOSE: - break; - case GGML_OP_DIAG: - ggml_cuda_op_diag(ctx, dst); - break; - case GGML_OP_DIAG_MASK_INF: - ggml_cuda_op_diag_mask_inf(ctx, dst); - break; - case GGML_OP_SOFT_MAX: - ggml_cuda_op_soft_max(ctx, dst); - break; - case GGML_OP_SOFT_MAX_BACK: - ggml_cuda_op_soft_max_back(ctx, dst); - break; - case GGML_OP_ROPE: - ggml_cuda_op_rope(ctx, dst); - break; - case GGML_OP_ROPE_BACK: - ggml_cuda_op_rope_back(ctx, dst); - break; - case GGML_OP_ROLL: - ggml_cuda_op_roll(ctx, dst); - break; - case GGML_OP_IM2COL: - ggml_cuda_op_im2col(ctx, dst); - break; - case GGML_OP_IM2COL_3D: - ggml_cuda_op_im2col_3d(ctx, dst); - break; - case GGML_OP_CONV_2D: - ggml_cuda_op_conv2d(ctx, dst); - break; - case GGML_OP_CONV_2D_DW: - ggml_cuda_op_conv2d_dw(ctx, dst); - break; - case GGML_OP_CONV_TRANSPOSE_2D: - ggml_cuda_conv_2d_transpose_p0(ctx, dst); - break; - case GGML_OP_CONV_TRANSPOSE_1D: - ggml_cuda_op_conv_transpose_1d(ctx,dst); - break; - case GGML_OP_POOL_2D: - ggml_cuda_op_pool2d(ctx, dst); - break; - case GGML_OP_SUM: - ggml_cuda_op_sum(ctx, dst); - break; - case GGML_OP_CUMSUM: - ggml_cuda_op_cumsum(ctx, dst); - break; - case GGML_OP_SUM_ROWS: - ggml_cuda_op_sum_rows(ctx, dst); - break; - case GGML_OP_MEAN: - ggml_cuda_op_mean(ctx, dst); - break; - case GGML_OP_SSM_CONV: - ggml_cuda_op_ssm_conv(ctx, dst); - break; - case GGML_OP_SSM_SCAN: - ggml_cuda_op_ssm_scan(ctx, dst); - break; - case GGML_OP_TOP_K: - ggml_cuda_op_top_k(ctx, dst); - break; - case GGML_OP_ARGSORT: - ggml_cuda_op_argsort(ctx, dst); - break; - case GGML_OP_FLASH_ATTN_EXT: - ggml_cuda_flash_attn_ext(ctx, dst); - break; - case GGML_OP_CROSS_ENTROPY_LOSS: - ggml_cuda_cross_entropy_loss(ctx, dst); - break; - case GGML_OP_TRI: - ggml_cuda_op_tri(ctx, dst); - break; - case GGML_OP_RWKV_WKV6: - ggml_cuda_op_rwkv_wkv6(ctx, dst); - break; - case GGML_OP_GATED_LINEAR_ATTN: - ggml_cuda_op_gated_linear_attn(ctx, dst); - break; - case GGML_OP_GATED_DELTA_NET: - ggml_cuda_op_gated_delta_net(ctx, dst); - break; - case GGML_OP_RWKV_WKV7: - ggml_cuda_op_rwkv_wkv7(ctx, dst); - break; - case GGML_OP_CROSS_ENTROPY_LOSS_BACK: - ggml_cuda_cross_entropy_loss_back(ctx, dst); - break; - case GGML_OP_OPT_STEP_ADAMW: - ggml_cuda_opt_step_adamw(ctx, dst); - break; - case GGML_OP_OPT_STEP_SGD: - ggml_cuda_opt_step_sgd(ctx, dst); - break; - case GGML_OP_SOLVE_TRI: - ggml_cuda_op_solve_tri(ctx, dst); - break; - case GGML_OP_FILL: - ggml_cuda_op_fill(ctx, dst); - break; - default: - return false; - } - - cudaError_t err = cudaGetLastError(); - if (err != cudaSuccess) { - GGML_LOG_ERROR("%s: %s failed\n", __func__, ggml_op_desc(dst)); - CUDA_CHECK(err); - } - - return true; -} - -//////////////////////////////////////////////////////////////////////////////// - -// backend - -static const char * ggml_backend_cuda_get_name(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - return cuda_ctx->name.c_str(); -} - -static void ggml_backend_cuda_free(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - delete cuda_ctx; - delete backend; -} - -static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpy2DAsync( - (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, - size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; - - GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); - - CUDA_CHECK(cudaMemcpy2DAsync( - data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream())); -} - -static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) { - ggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer; - ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer; - - if (!ggml_backend_is_cuda(backend_src) || !ggml_backend_is_cuda(backend_dst)) { - return false; - } - - if (!ggml_backend_buffer_is_cuda(buf_src) || !ggml_backend_buffer_is_cuda(buf_dst)) { - return false; - } - - // device -> device copy - ggml_backend_cuda_context * cuda_ctx_src = (ggml_backend_cuda_context *) backend_src->context; - ggml_backend_cuda_context * cuda_ctx_dst = (ggml_backend_cuda_context *) backend_dst->context; - - ggml_backend_cuda_buffer_context * buf_ctx_src = (ggml_backend_cuda_buffer_context *) buf_src->context; - ggml_backend_cuda_buffer_context * buf_ctx_dst = (ggml_backend_cuda_buffer_context *) buf_dst->context; - - if (cuda_ctx_src->device != buf_ctx_src->device || cuda_ctx_dst->device != buf_ctx_dst->device) { -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: backend and buffer devices do not match\n", __func__); -#endif // NDEBUG - return false; - } - - if (backend_src != backend_dst) { - // copy on src stream - if (cuda_ctx_src->device == cuda_ctx_dst->device) { - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); - } else { -#ifdef GGML_CUDA_NO_PEER_COPY - return false; -#else - CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream())); -#endif // GGML_CUDA_NO_PEER_COPY - } - - // record event on src stream after the copy - if (!cuda_ctx_src->copy_event) { - ggml_cuda_set_device(cuda_ctx_src->device); - CUDA_CHECK(cudaEventCreateWithFlags(&cuda_ctx_src->copy_event, cudaEventDisableTiming)); - } - - CUDA_CHECK(cudaEventRecord(cuda_ctx_src->copy_event, cuda_ctx_src->stream())); - - // wait on dst stream for the copy to complete - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx_dst->stream(), cuda_ctx_src->copy_event, 0)); - } else { - // src and dst are on the same backend - CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); - } - return true; -} - -static void ggml_backend_cuda_synchronize(ggml_backend_t backend) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream())); - - GGML_UNUSED(backend); -} - -#ifdef USE_CUDA_GRAPH -static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) { - - bool use_cuda_graph = true; - // Loop over nodes in GGML graph to obtain info needed for CUDA graph - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_tensor * node = cgraph->nodes[i]; - - if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { - continue; - } - - if (node->src[0] && node->src[0]->buffer && ggml_backend_buft_is_cuda_split(node->src[0]->buffer->buft)) { - use_cuda_graph = false; // Split buffers are not supported by CUDA graph capture -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to split buffer\n", __func__); -#endif - } - - // [TAG_MUL_MAT_ID_CUDA_GRAPHS] - if (node->op == GGML_OP_MUL_MAT_ID) { - const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; - const int mmvq_mmid_max = get_mmvq_mmid_max_batch(node->src[0]->type, cc); - if (!ggml_is_quantized(node->src[0]->type) || node->ne[2] > mmvq_mmid_max) { - // under these conditions, the mul_mat_id operation will need to synchronize the stream, so we cannot use CUDA graphs - // TODO: figure out a way to enable for larger batch sizes, without hurting performance - // ref: https://github.com/ggml-org/llama.cpp/pull/18958 - use_cuda_graph = false; -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to unsupported node type\n", __func__); -#endif - } - } - - if (!use_cuda_graph) { - break; - } - } - - return use_cuda_graph; -} - -static const void * ggml_cuda_graph_get_key(ggml_cgraph * cgraph) { - return cgraph->nodes[0]; -} - -static bool ggml_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph) { - bool res = false; - - const void * graph_key = ggml_cuda_graph_get_key(cgraph); - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - - if (cgraph->uid != 0 && - cgraph->uid == graph->uid) { - GGML_LOG_DEBUG("CUDA Graph id %zu reused\n", cgraph->uid); - GGML_ASSERT((int)graph->node_props.size() == cgraph->n_nodes); - return false; - } - - graph->uid = cgraph->uid; - - // Check if the graph size has changed - if ((int)graph->node_props.size() != cgraph->n_nodes) { - res = true; - graph->node_props.resize(cgraph->n_nodes); - } - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_cuda_graph::node_properties prop = {}; - memcpy(&prop.node, cgraph->nodes[i], sizeof(ggml_tensor)); - - for (int j = 0; j < GGML_MAX_SRC; ++j) { - if (cgraph->nodes[i]->src[j]) { - prop.node_src_data_ptrs[j] = cgraph->nodes[i]->src[j]->data; - memcpy(prop.node_src_ne[j], cgraph->nodes[i]->src[j]->ne, sizeof(prop.node_src_ne[j])); - memcpy(prop.node_src_nb[j], cgraph->nodes[i]->src[j]->nb, sizeof(prop.node_src_nb[j])); - } - } - - if (res || memcmp(&graph->node_props[i], &prop, sizeof(prop)) != 0) { - graph->node_props[i] = prop; - res = true; - } - } - - return res; -} - -static void ggml_cuda_graph_update_executable(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - -#if CUDART_VERSION >= 12000 - cudaGraphExecUpdateResultInfo result_info; - cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &result_info); -#else - cudaGraphNode_t errorNode; - cudaGraphExecUpdateResult result_info; - cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &errorNode, &result_info); -#endif // CUDART_VERSION >= 12000 - - if (stat == cudaErrorGraphExecUpdateFailure) { -#ifndef NDEBUG - GGML_LOG_DEBUG("%s: CUDA graph update failed\n", __func__); -#endif - - // The pre-existing graph exec cannot be updated due to violated constraints - // so instead clear error and re-instantiate - (void)cudaGetLastError(); - CUDA_CHECK(cudaGraphExecDestroy(graph->instance)); - graph->instance = nullptr; - CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); - } else { - GGML_ASSERT(stat == cudaSuccess); - } -} -#endif // USE_CUDA_GRAPH - -static bool ggml_cuda_should_fuse_rope_set_rows(const ggml_tensor * rope, - const ggml_tensor * view, - const ggml_tensor * set_rows) { - - if (rope->op != GGML_OP_ROPE || view->op != GGML_OP_VIEW || set_rows->op != GGML_OP_SET_ROWS) { - return false; - } - // ne3 not tested - if (rope->src[0]->ne[3] != 1) { - return false; - } - - if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16) { - return false; - } - - if (set_rows->src[1]->type != GGML_TYPE_I64) { - return false; - } - - // The view should flatten two dims of rope into one dim - if (!ggml_is_contiguous(view) || view->ne[0] != rope->ne[0] * rope->ne[1]) { - return false; - } - - // Only norm/neox shaders have the fusion code - const int mode = ((const int32_t *) rope->op_params)[2]; - if (mode != GGML_ROPE_TYPE_NORMAL && mode != GGML_ROPE_TYPE_NEOX) { - return false; - } - - return true; -} - -static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) { - args.sigmoid = false; - args.softmax = false; - args.delayed_softmax = false; - args.prob_bias = false; - args.norm = false; - - const int n_nodes = cgraph->n_nodes; - ggml_tensor ** nodes = cgraph->nodes; - - if (nodes[node_idx]->op == GGML_OP_SOFT_MAX) { - args.softmax = true; - } - - if (nodes[node_idx]->op == GGML_OP_UNARY) { - if (ggml_get_unary_op(nodes[node_idx]) != GGML_UNARY_OP_SIGMOID) { - return false; - } - args.sigmoid = true; - } - - if (nodes[node_idx]->op == GGML_OP_ARGSORT) { - args.delayed_softmax = true; - } - - node_idx++; - - if (args.sigmoid || args.softmax) { - // SOFTMAX -> RESHAPE - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - ggml_tensor * probs_reshaped = nodes[node_idx]; - node_idx++; - - if (node_idx >= n_nodes) { - return false; - } - - // src of bias add is the unreshaped probs (-2 instead of -1) - if (nodes[node_idx]->op == GGML_OP_ADD && nodes[node_idx]->src[0] == nodes[node_idx - 2]) { - args.prob_bias = true; - node_idx++; - } - // RESHAPE/ADD -> ARGSORT - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_ARGSORT) { - return false; - } - - if (args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } else if (!args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 2]) { - return false; - } - - node_idx++; - - // ARGSORT-> VIEW - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_GET_ROWS) { - return false; - } - - // GET_ROWS - if (nodes[node_idx]->src[0] != probs_reshaped || nodes[node_idx]->src[1] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - } else if (args.delayed_softmax) { - if (node_idx - 2 < 0) { - return false; - } - ggml_tensor * probs_reshaped = nodes[node_idx - 2]; - - // VIEW->ARGSORT - if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || - nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - - // GET_ROWS - if (node_idx >= n_nodes || nodes[node_idx]->src[1] != nodes[node_idx - 1] || - nodes[node_idx]->src[0] != probs_reshaped) { - return false; - } - node_idx++; - - static const std::vector remaining_ops = { GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }; - - for (const ggml_op op : remaining_ops) { - if (node_idx >= n_nodes || nodes[node_idx]->op != op || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - return false; - } - node_idx++; - } - } - - // At this point we can check for norm + scale. Everything is now at least valid till the norm - if (node_idx >= n_nodes) { - return true; - } - - if (nodes[node_idx]->op == GGML_OP_RESHAPE) { - //check RESHAPE->SUM_ROWS->CLAMP->DIV->RESHAPE - static const std::vector norm_ops = { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP }; - - args.norm = true; - for (const ggml_op op : norm_ops) { - if (nodes[node_idx]->op == op && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { - node_idx++; - } else { - args.norm = false; - return true; - } - } - - // DIV <- CLAMP, RESHAPE - if (nodes[node_idx]->op != GGML_OP_DIV || nodes[node_idx]->src[1] != nodes[node_idx - 1] || - nodes[node_idx]->src[0] != nodes[node_idx - 3]) { - args.norm = false; - return true; - } - node_idx++; - - if (nodes[node_idx]->op != GGML_OP_RESHAPE || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { - args.norm = false; - return true; - } - - node_idx++; - } - - if (nodes[node_idx]->op == GGML_OP_SCALE && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { - args.scale = true; - } - - return true; -} - -// returns whether the write (out) nodes overwrite the read nodes in operation -static bool ggml_cuda_check_fusion_memory_ranges(const ggml_cgraph * cgraph, - const int node_idx, - const int node_count, - const int * out_nodes, - const int out_count, - const bool is_topk_moe = false) { - auto nodes_overlap = [&](const ggml_tensor * a, const ggml_tensor * b) { - const int64_t a_start = (int64_t) a->data; - const int64_t a_end = a_start + ggml_backend_buft_get_alloc_size(a->buffer->buft, a); - - const int64_t b_start = (int64_t) b->data; - const int64_t b_end = b_start + ggml_backend_buft_get_alloc_size(b->buffer->buft, b); - - if ((b_start <= a_start && a_start < b_end) || (a_start <= b_start && b_start < a_end)) { - return true; - } - - return false; - }; - - bool is_ok = true; - // exception for topk-moe, as each row is read entirely before writing - if (ggml_nrows(cgraph->nodes[node_idx]) == 1 && is_topk_moe) { - return true; - } - - for (int i = 0; i < out_count; ++i) { - const ggml_tensor * dst = cgraph->nodes[out_nodes[i]]; - - for (int j = node_idx; j < node_idx + node_count; ++j) { - // Loop over all srcs of all nodes in the fusion. If the src overlaps - // the destination and the src is not an intermediate node that's being - // elided, then disable fusion. - - for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { - const ggml_tensor * src = cgraph->nodes[j]->src[src_idx]; - - if (!src || src->op == GGML_OP_NONE) { - continue; - } - - if (nodes_overlap(dst, src)) { - bool found = false; - - for (int k = node_idx; k < j; ++k) { - if (cgraph->nodes[k] == src) { - found = true; - break; - } - } - - if (!found) { - is_ok = false; - break; - } - } - } - } - } - - return is_ok; -} - - -static bool ggml_cuda_can_fuse(const struct ggml_cgraph * cgraph, - int node_idx, - std::initializer_list ops, - std::initializer_list unary_ops) { -#ifndef NDEBUG - const size_t num_unary = std::count(ops.begin(), ops.end(), GGML_OP_UNARY); - GGML_ASSERT(unary_ops.size() == num_unary); -#endif - - const auto is_equal = [](const std::initializer_list & list1, - const std::initializer_list & list2) { - return std::equal(list1.begin(), list1.end(), list2.begin(), list2.end()); - }; - - std::initializer_list mul_mat_bias_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_GLU }; - std::initializer_list mul_mat_id_bias_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_GLU }; - - std::initializer_list mul_mat_id_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_MUL_MAT_ID, GGML_OP_GLU }; - std::initializer_list mul_mat_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT, GGML_OP_GLU }; - - if ((is_equal(mul_mat_bias_glu_ops, ops) || is_equal(mul_mat_id_bias_glu_ops, ops)) && - ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 4 })) { - const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; - const ggml_tensor * ffn_gate_bias = cgraph->nodes[node_idx + 1]; - const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 2]; - const ggml_tensor * ffn_up_bias = cgraph->nodes[node_idx + 3]; - const ggml_tensor * glu = cgraph->nodes[node_idx + 4]; - - if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu, ffn_up_bias, ffn_gate_bias)) { - int out_nodes[] = { node_idx + 4 }; - return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); - } - } - - if ((is_equal(mul_mat_id_glu_ops, ops) || is_equal(mul_mat_glu_ops, ops)) && - ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { - const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; - const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 1]; - const ggml_tensor * glu = cgraph->nodes[node_idx + 2]; - - if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu)) { - int out_nodes[] = { node_idx + 2 }; - return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); - } - } - - std::initializer_list rope_set_rows_ops = { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }; - - if (is_equal(rope_set_rows_ops, ops) && ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { - const ggml_tensor * rope = cgraph->nodes[node_idx]; - const ggml_tensor * view = cgraph->nodes[node_idx + 1]; - const ggml_tensor * set_rows = cgraph->nodes[node_idx + 2]; - - if (ggml_cuda_should_fuse_rope_set_rows(rope, view, set_rows)) { - return true; - } - } - - if (!ggml_can_fuse(cgraph, node_idx, ops)) { - return false; - } - - if ((ops.size() == 2 || ops.size() == 3) && ops.begin()[0] == GGML_OP_RMS_NORM && ops.begin()[1] == GGML_OP_MUL) { - const ggml_tensor *rms_norm = cgraph->nodes[node_idx]; - const ggml_tensor *mul = cgraph->nodes[node_idx+1]; - const ggml_tensor *add = nullptr; - - if (ops.size() == 3 && ops.begin()[2] == GGML_OP_ADD) { - add = cgraph->nodes[node_idx+2]; - } - - GGML_ASSERT(rms_norm->src[0]->type == GGML_TYPE_F32); - GGML_ASSERT(rms_norm->type == GGML_TYPE_F32); - - //rms norm only supports F32 - if (mul->src[0]->type != GGML_TYPE_F32 || - mul->src[1]->type != GGML_TYPE_F32 || - mul->type != GGML_TYPE_F32) { - return false; - } - - if (add && (add->src[0]->type != GGML_TYPE_F32 || - add->src[1]->type != GGML_TYPE_F32 || - add->type != GGML_TYPE_F32) ) { - return false; - } - - //if rms norm is the B operand, then we don't handle broadcast - if (rms_norm == mul->src[1] && !ggml_are_same_shape(mul->src[0], rms_norm)) { - return false; - } - - //rms_norm kernel assumes contiguous rows - if (!ggml_is_contiguous_rows(mul->src[0]) || !ggml_is_contiguous_rows(mul->src[1])) { - return false; - } - - if (add && (!ggml_is_contiguous(add->src[0]) || !ggml_is_contiguous_rows(add->src[1]))) { - return false; - } - - return true; - } - - if (ops.size() == 2 && ops.begin()[0] == GGML_OP_SSM_CONV && ops.begin()[1] == GGML_OP_UNARY - && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_SILU) { - const ggml_tensor * ssm_conv = cgraph->nodes[node_idx]; - const ggml_tensor * silu = cgraph->nodes[node_idx+1]; - - if (ssm_conv->type != GGML_TYPE_F32 || silu->type != GGML_TYPE_F32) { - return false; - } - - return true; - } - - if (ops.size() == 2 && ops.begin()[0] == GGML_OP_UNARY && ops.begin()[1] == GGML_OP_MUL - && unary_ops.size() == 1 && (unary_ops.begin()[0] == GGML_UNARY_OP_SILU || unary_ops.begin()[0] == GGML_UNARY_OP_SIGMOID || unary_ops.begin()[0] == GGML_UNARY_OP_SOFTPLUS)) { - const ggml_tensor * unary = cgraph->nodes[node_idx]; - const ggml_tensor * mul = cgraph->nodes[node_idx+1]; - - if (ggml_get_unary_op(unary) != unary_ops.begin()[0]) { - return false; - } - - if (unary->type != GGML_TYPE_F32 && unary->type != GGML_TYPE_F16) { - return false; - } - - if (unary->type != mul->type) { - return false; - } - - const ggml_tensor * other = (mul->src[0] == unary) ? mul->src[1] : mul->src[0]; - if (other->type != unary->type) { - return false; - } - if (!ggml_is_contiguous_1(other) || !ggml_is_contiguous_1(unary->src[0]) || !ggml_are_same_shape(other, unary)) { - return false; - } - - return true; - } - - if (ops.size() == 3 && ops.begin()[0] == GGML_OP_SCALE && ops.begin()[1] == GGML_OP_UNARY && ops.begin()[2] == GGML_OP_SCALE - && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_TANH) { - const ggml_tensor *scale = cgraph->nodes[node_idx]; - const ggml_tensor *tanh = cgraph->nodes[node_idx+1]; - const ggml_tensor *scale2 = cgraph->nodes[node_idx+2]; - - GGML_ASSERT(scale->src[0]->type == GGML_TYPE_F32); - GGML_ASSERT(scale->type == GGML_TYPE_F32); - - if (ggml_get_unary_op(tanh) != GGML_UNARY_OP_TANH) { - return false; - } - - // Check for bias - if (ggml_get_op_params_f32(scale, 1) != 0.0f || ggml_get_op_params_f32(scale2, 1) != 0.0f) { - return false; - } - - return true; - } - - return false; -} - -static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph, const bool use_cuda_graph, const bool cuda_graph_update_required, const void * graph_key) { - bool graph_evaluated_or_captured = false; - - // flag used to determine whether it is an integrated_gpu - const bool integrated = ggml_cuda_info().devices[cuda_ctx->device].integrated; - - ggml_cuda_stream_context & stream_ctx = cuda_ctx->stream_context(); - bool is_concurrent_event_active = false; - ggml_cuda_concurrent_event * concurrent_event = nullptr; - bool should_launch_concurrent_events = false; - - const auto try_launch_concurrent_event = [&](const ggml_tensor * node) { - if (stream_ctx.concurrent_events.find(node) != stream_ctx.concurrent_events.end()) { - concurrent_event = &stream_ctx.concurrent_events[node]; - - is_concurrent_event_active = true; - - GGML_LOG_DEBUG("Launching %d streams at %s\n", concurrent_event->n_streams, node->name); - - cudaStream_t main_stream = cuda_ctx->stream(); // this should be stream 0 - GGML_ASSERT(cuda_ctx->curr_stream_no == 0); - CUDA_CHECK(cudaEventRecord(concurrent_event->fork_event, main_stream)); - - for (int i = 1; i <= concurrent_event->n_streams; ++i) { - cudaStream_t stream = cuda_ctx->stream(cuda_ctx->device, i); - CUDA_CHECK(cudaStreamWaitEvent(stream, concurrent_event->fork_event)); - } - } - }; - - while (!graph_evaluated_or_captured) { - // Only perform the graph execution if CUDA graphs are not enabled, or we are capturing the graph. - // With the use of CUDA graphs, the execution will be performed by the graph launch. - if (!use_cuda_graph || cuda_graph_update_required) { - [[maybe_unused]] int prev_i = 0; - - if (stream_ctx.concurrent_events.size() > 0) { - should_launch_concurrent_events = true; - for (const auto & [tensor, event] : stream_ctx.concurrent_events) { - should_launch_concurrent_events = should_launch_concurrent_events && event.is_valid(); - } - } - - if (should_launch_concurrent_events) { - // Restore original node order within each concurrent region to enable fusion within streams - - std::unordered_map node_to_idx; - node_to_idx.reserve(cgraph->n_nodes); - for (int i = 0; i < cgraph->n_nodes; ++i) { - node_to_idx[cgraph->nodes[i]] = i; - } - - for (auto & [fork_node, event] : stream_ctx.concurrent_events) { - // Find positions of all nodes from this event in the current graph - std::vector positions; - positions.reserve(event.original_order.size()); - - bool all_found = true; - for (const ggml_tensor * orig_node : event.original_order) { - auto it = node_to_idx.find(orig_node); - if (it != node_to_idx.end()) { - positions.push_back(it->second); - } else { - all_found = false; - break; - } - } - - if (!all_found || positions.size() != event.original_order.size()) { - continue; - } - - // Sort positions to get contiguous range - std::vector sorted_positions = positions; - std::sort(sorted_positions.begin(), sorted_positions.end()); - - bool is_contiguous = true; - for (size_t i = 1; i < sorted_positions.size(); ++i) { - if (sorted_positions[i] != sorted_positions[i-1] + 1) { - is_contiguous = false; - break; - } - } - - if (!is_contiguous) { - continue; - } - - // Restore original order at the sorted positions - int start_pos = sorted_positions[0]; - for (size_t i = 0; i < event.original_order.size(); ++i) { - cgraph->nodes[start_pos + i] = const_cast(event.original_order[i]); - } - } - } else { - stream_ctx.concurrent_events.clear(); - } - - for (int i = 0; i < cgraph->n_nodes; i++) { - ggml_tensor * node = cgraph->nodes[i]; - if (is_concurrent_event_active) { - GGML_ASSERT(concurrent_event); - - if (node == concurrent_event->join_node) { - cuda_ctx->curr_stream_no = 0; - for (int i = 1; i <= concurrent_event->n_streams; ++i) { - // Wait on join events of forked streams in the main stream - CUDA_CHECK(cudaEventRecord(concurrent_event->join_events[i - 1], - cuda_ctx->stream(cuda_ctx->device, i))); - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), concurrent_event->join_events[i - 1])); - } - - is_concurrent_event_active = false; - concurrent_event = nullptr; - } else { - GGML_ASSERT (concurrent_event->stream_mapping.find(node) != concurrent_event->stream_mapping.end()); - cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; - GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); - } - } else if (i - prev_i > 1) { - //the previous node was fused - const ggml_tensor * prev_node = cgraph->nodes[i - 1]; - try_launch_concurrent_event(prev_node); - - if (is_concurrent_event_active) { - cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; - GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); - } - } - -#ifdef GGML_CUDA_DEBUG - const int nodes_fused = i - prev_i - 1; - if (nodes_fused > 0) { - GGML_LOG_INFO("nodes_fused: %d\n", nodes_fused); - } -#endif - prev_i = i; - - if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { - continue; - } - - if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { - continue; - } - - // start of fusion operations - static bool disable_fusion = (getenv("GGML_CUDA_DISABLE_FUSION") != nullptr); - if (!disable_fusion) { - ggml_cuda_topk_moe_args args; - - if (cgraph->nodes[i]->op == GGML_OP_UNARY || cgraph->nodes[i]->op == GGML_OP_SOFT_MAX || - cgraph->nodes[i]->op == GGML_OP_ARGSORT) { - const bool can_fuse = ggml_cuda_topk_moe_fusion(cgraph, i, args); - - std::vector ops; - - if (can_fuse) { - const ggml_tensor * logits = node->src[0]; - ggml_tensor * weights = nullptr; - ggml_tensor * ids = nullptr; - const ggml_tensor * bias = nullptr; - const ggml_tensor * clamp = nullptr; - const ggml_tensor * scale = nullptr; - - if (!args.delayed_softmax) { - ggml_op gating_op = args.sigmoid ? GGML_OP_UNARY : GGML_OP_SOFT_MAX; - int out_nodes[2]; // nodes which can't be elided - - if (args.prob_bias) { - bias = cgraph->nodes[i + 2]->src[1]; - ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, - GGML_OP_VIEW, GGML_OP_GET_ROWS }); - out_nodes[0] = i + 4; - ids = cgraph->nodes[i + 4]; - } else { - ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, - GGML_OP_GET_ROWS }); - out_nodes[0] = i + 3; - ids = cgraph->nodes[i + 3]; - } - - if (args.norm) { - ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP, - GGML_OP_DIV, GGML_OP_RESHAPE }); - clamp = cgraph->nodes[i + ops.size() - 3]; - } - if (args.scale) { - ops.insert(ops.end(), { GGML_OP_SCALE }); - scale = cgraph->nodes[i + ops.size() - 1]; - } - - weights = cgraph->nodes[i + ops.size() - 1]; - out_nodes[1] = i + ops.size() - 1; - - if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && - ggml_cuda_should_use_topk_moe(node, logits, weights, ids) && - ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { - ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); - i += ops.size() - 1; - continue; - } - } else if (!args.norm && !args.prob_bias) { - //special case gpt-oss, no norm, no bias. - ops.insert(ops.end(), { GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS, - GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }); - weights = cgraph->nodes[i + 5]; - ids = cgraph->nodes[i + 1]; - const ggml_tensor * softmax = cgraph->nodes[i + 4]; - - int out_nodes[2] = { i + 1, i + 5 }; - if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && - ggml_cuda_should_use_topk_moe(softmax, logits, weights, ids) && - ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { - ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); - i += ops.size() - 1; - continue; - } - } - } - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }, {})) { - ggml_tensor * rope = cgraph->nodes[i]; - ggml_tensor * set_rows = cgraph->nodes[i + 2]; - - ggml_cuda_op_rope_fused(*cuda_ctx, rope, set_rows); - i += 2; - continue; - } - - if (node->op == GGML_OP_ADD || node->op == GGML_OP_MUL) { - int n_fuse = 0; - ggml_op ops[8]; - std::fill(ops, ops + 8, node->op); - - for (; n_fuse <= 6; ++n_fuse){ - if (!ggml_can_fuse(cgraph, i + n_fuse, ops + n_fuse, 2)) { - break; - } - if (cgraph->nodes[i + n_fuse] != cgraph->nodes[i + n_fuse + 1]->src[0]) { - break; - } - if (!ggml_are_same_layout(cgraph->nodes[i + n_fuse]->src[1], cgraph->nodes[i + n_fuse + 1]->src[1])) { - break; - } - } - - n_fuse++; - - if (n_fuse > 1) { - ggml_tensor fused_node; - memcpy(&fused_node, node, sizeof(ggml_tensor)); - for (int j = 0; j < n_fuse - 1; ++j) { - fused_node.src[j + 2] = cgraph->nodes[i + j + 1]->src[1]; - } - fused_node.data = cgraph->nodes[i + n_fuse - 1]->data; - if (node->op == GGML_OP_ADD) { - ggml_cuda_op_fused_add(*cuda_ctx, &fused_node, n_fuse); - } else { - ggml_cuda_op_fused_mul(*cuda_ctx, &fused_node, n_fuse); - } - i += n_fuse - 1; - - continue; - } - } - - bool fused_mul_mat_vec = false; - int fused_node_count = 0; - - for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { - const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (ggml_cuda_can_fuse(cgraph, i, { op, bias_op, op, bias_op, GGML_OP_GLU }, {})) { - ggml_tensor * glu = cgraph->nodes[i + 4]; - ggml_tensor * gate_bias_n = glu->src[0]; - ggml_tensor * up_bias_n = glu->src[1]; - - //we don't assume the order for {gate, up}. Instead infer it from the bias tensor - ggml_tensor * gate_n = nullptr; - ggml_tensor * up_n = nullptr; - - if (gate_bias_n->src[0] == cgraph->nodes[i] || gate_bias_n->src[1] == cgraph->nodes[i]) { - gate_n = cgraph->nodes[i]; - up_n = cgraph->nodes[i + 2]; - } else if (gate_bias_n->src[0] == cgraph->nodes[i + 2] || gate_bias_n->src[1] == cgraph->nodes[i + 2]) { - gate_n = cgraph->nodes[i + 2]; - up_n = cgraph->nodes[i]; - } else { - continue; - } - - auto get_bias_tensor = [](const ggml_tensor * bias_node, const ggml_tensor * mul_node, ggml_op op_bias) { - if (op_bias == GGML_OP_ADD) { - if (bias_node->src[0] == mul_node) { - return bias_node->src[1]; - } - if (bias_node->src[1] == mul_node) { - return bias_node->src[0]; - } - return (ggml_tensor *) nullptr; - } - GGML_ASSERT(op_bias == GGML_OP_ADD_ID); - GGML_ASSERT(bias_node->src[0] == mul_node); - return bias_node->src[1]; - }; - - ggml_tensor * up_bias_tensor = get_bias_tensor(up_bias_n, up_n, bias_op); - ggml_tensor * gate_bias_tensor = get_bias_tensor(gate_bias_n, gate_n, bias_op); - - if (!up_bias_tensor || !gate_bias_tensor) { - continue; - } - - // we don't support repeating adds - if (bias_op == GGML_OP_ADD && - (!ggml_are_same_shape(gate_bias_n->src[0], gate_bias_n->src[1]) || - !ggml_are_same_shape(up_bias_n->src[0], up_bias_n->src[1]))) { - continue; - } - - const ggml_tensor * src0 = up_n->src[0]; - const ggml_tensor * src1 = up_n->src[1]; - const ggml_tensor * ids = up_n->src[2]; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(up_n)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate_n->src[0]; - fusion_data.x_bias = up_bias_tensor; - fusion_data.gate_bias = gate_bias_tensor; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 5; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(up_n)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate_n->src[0]; - fusion_data.x_bias = up_bias_tensor; - fusion_data.gate_bias = gate_bias_tensor; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 5; - break; - } - } else if (ggml_cuda_can_fuse(cgraph, i, { op, op, GGML_OP_GLU }, {})) { - ggml_tensor * glu = cgraph->nodes[i + 2]; - ggml_tensor * gate = glu->src[0]; - ggml_tensor * up = glu->src[1]; - - bool ok = (gate == cgraph->nodes[i] && up == cgraph->nodes[i + 1]) - || (gate == cgraph->nodes[i + 1] && up == cgraph->nodes[i]); - - if (!ok) continue; - - const ggml_tensor * src0 = up->src[0]; - const ggml_tensor * src1 = up->src[1]; - const ggml_tensor * ids = up->src[2]; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(up)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate->src[0]; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 3; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(up)) { - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.gate = gate->src[0]; - fusion_data.glu_op = ggml_get_glu_op(glu); - - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 3; - break; - } - } - } - - if (fused_mul_mat_vec) { - i += fused_node_count - 1; - continue; - } - - fused_mul_mat_vec = false; - fused_node_count = 0; - - for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { - const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; - - if (!ggml_can_fuse(cgraph, i, { op, bias_op })) { - continue; - } - - ggml_tensor * mm_node = cgraph->nodes[i]; - ggml_tensor * bias_node = cgraph->nodes[i + 1]; - - ggml_tensor * bias_tensor = nullptr; - if (bias_op == GGML_OP_ADD) { - if (bias_node->src[0] == mm_node) { - bias_tensor = bias_node->src[1]; - } else if (bias_node->src[1] == mm_node) { - bias_tensor = bias_node->src[0]; - } else { - continue; - } - } else { - if (bias_node->src[0] != mm_node) { - continue; - } - bias_tensor = bias_node->src[1]; - } - - const ggml_tensor * src0 = mm_node->src[0]; - const ggml_tensor * src1 = mm_node->src[1]; - const ggml_tensor * ids = mm_node->src[2]; - - if (bias_op == GGML_OP_ADD_ID && bias_node->src[2] != ids) { - continue; - } - - if (bias_op == GGML_OP_ADD && !ggml_are_same_shape(bias_node->src[0], bias_node->src[1])) { - continue; - } - - ggml_cuda_mm_fusion_args_host fusion_data{}; - fusion_data.x_bias = bias_tensor; - - if (ggml_cuda_should_fuse_mul_mat_vec_f(mm_node)) { - ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 2; - break; - } - - if (ggml_cuda_should_fuse_mul_mat_vec_q(mm_node)) { - ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); - fused_mul_mat_vec = true; - fused_node_count = 2; - break; - } - } - - if (fused_mul_mat_vec) { - i += fused_node_count - 1; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL, GGML_OP_ADD}, {})) { - ggml_cuda_op_rms_norm_fused_add(*cuda_ctx, node, cgraph->nodes[i+1], cgraph->nodes[i+2]); - i += 2; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL}, {})) { - ggml_cuda_op_rms_norm_fused(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SSM_CONV, GGML_OP_UNARY }, { GGML_UNARY_OP_SILU })) { - ggml_cuda_op_ssm_conv(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SILU }) || - ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SIGMOID }) || - ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SOFTPLUS })) { - ggml_cuda_op_unary_mul(*cuda_ctx, node, cgraph->nodes[i+1]); - i++; - continue; - } - - if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SCALE, GGML_OP_UNARY, GGML_OP_SCALE }, { GGML_UNARY_OP_TANH })) { - i += 2; - ggml_cuda_op_softcap(*cuda_ctx, cgraph->nodes[i], node); - continue; - } - } -#ifndef NDEBUG - assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device)); - for (int j = 0; j < GGML_MAX_SRC; j++) { - if (node->src[j] != nullptr) { - assert(node->src[j]->buffer); - assert(node->src[j]->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) || - ggml_backend_buft_is_cuda_split(node->src[j]->buffer->buft) || (integrated && ggml_backend_buft_is_cuda_host(node->src[j]->buffer->buft))); - } - } -#else - GGML_UNUSED(integrated); -#endif // NDEBUG - - bool ok = ggml_cuda_compute_forward(*cuda_ctx, node); - if (!ok) { - GGML_LOG_ERROR("%s: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op)); - } - GGML_ASSERT(ok); - - if (!is_concurrent_event_active) { - try_launch_concurrent_event(node); - } - } - } - -#ifdef USE_CUDA_GRAPH - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (use_cuda_graph && cuda_graph_update_required) { // End CUDA graph capture - if (graph->graph != nullptr) { - CUDA_CHECK(cudaGraphDestroy(graph->graph)); - graph->graph = nullptr; - } - - CUDA_CHECK(cudaStreamEndCapture(cuda_ctx->stream(), &graph->graph)); - graph_evaluated_or_captured = true; // CUDA graph has been captured - - std::lock_guard lock(ggml_cuda_lock); - if (ggml_cuda_lock_counter.fetch_sub(1, std::memory_order_relaxed) == 1) { - ggml_cuda_lock_cv.notify_all(); - } - } else { - graph_evaluated_or_captured = true; // ggml graph has been directly evaluated - } - } - - if (use_cuda_graph) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (graph->instance == nullptr) { // Create executable graph from captured graph. - CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); - } - if (cuda_graph_update_required) { // Update graph executable - ggml_cuda_graph_update_executable(cuda_ctx, graph_key); - } - // Launch graph - CUDA_CHECK(cudaGraphLaunch(graph->instance, cuda_ctx->stream())); -#else - GGML_UNUSED(graph_key); - graph_evaluated_or_captured = true; -#endif // USE_CUDA_GRAPH - } -} - -#ifdef USE_CUDA_GRAPH -static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - - if (graph->graph == nullptr) { - if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_AMPERE) { - if (!graph->disable_due_to_gpu_arch) { - GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__); - } - graph->disable_due_to_gpu_arch = true; - } - } - - return graph->is_enabled(); -} -#endif // USE_CUDA_GRAPH - -static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend, ggml_cgraph * cgraph) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - - ggml_cuda_set_device(cuda_ctx->device); - - bool use_cuda_graph = false; - bool cuda_graph_update_required = false; - const void * graph_key = nullptr; - -#ifdef USE_CUDA_GRAPH - graph_key = ggml_cuda_graph_get_key(cgraph); - - ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); - - ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); - if (graph->is_enabled()) { - const bool graph_compatible = ggml_cuda_graph_check_compability(cgraph); - if (graph_compatible) { - const bool properties_changed = ggml_cuda_graph_update_required(cuda_ctx, cgraph); - - if (!graph->warmup_complete) { - // Warmup: need at least 2 calls with no property change on the 2nd call - if (!properties_changed) { - graph->warmup_complete = true; - GGML_LOG_DEBUG("%s: CUDA graph warmup complete\n", __func__); - use_cuda_graph = true; - cuda_graph_update_required = true; - } - // else: properties changed or first call - execute directly (use_cuda_graph stays false) - } else { - // Post-warmup: normal CUDA graph operation - if (properties_changed) { - // Properties changed - reset warmup, execute directly until stable again - graph->warmup_complete = false; - GGML_LOG_DEBUG("%s: CUDA graph warmup reset\n", __func__); - } else { - use_cuda_graph = true; - cuda_graph_update_required = graph->instance == nullptr; - } - } - } - } -#endif // USE_CUDA_GRAPH - - if (use_cuda_graph && cuda_graph_update_required) { - // Start CUDA graph capture - { - std::lock_guard lock(ggml_cuda_lock); - ggml_cuda_lock_counter.fetch_add(1, std::memory_order_relaxed); - } - - CUDA_CHECK(cudaStreamBeginCapture(cuda_ctx->stream(), cudaStreamCaptureModeRelaxed)); - } - - ggml_cuda_graph_evaluate_and_capture(cuda_ctx, cgraph, use_cuda_graph, cuda_graph_update_required, graph_key); - - return GGML_STATUS_SUCCESS; -} - -static void ggml_backend_cuda_event_record(ggml_backend_t backend, ggml_backend_event_t event) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - CUDA_CHECK(cudaEventRecord((cudaEvent_t)event->context, cuda_ctx->stream())); -} - -static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_event_t event) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; - - if (ggml_backend_is_cuda(backend)) { - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), (cudaEvent_t)event->context, 0)); - } else { -#if 0 - // untested - auto wait_fn = [](void * user_data) { - ggml_backend_event_t event = (ggml_backend_event_t)user_data; - ggml_backend_event_synchronize(event); - }; - - CUDA_CHECK(cudaLaunchHostFunc(cuda_ctx->stream(), wait_fn, event)); -#endif - GGML_ABORT("fatal error"); - } -} - -static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) { - ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; - -#ifdef USE_CUDA_GRAPH - const void * graph_key = ggml_cuda_graph_get_key(cgraph); - const bool use_cuda_graph = ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); -#else - const bool use_cuda_graph = false; - GGML_UNUSED(cuda_ctx); - GGML_UNUSED(cgraph); -#endif - - static bool enable_graph_optimization = [] { - const char * env = getenv("GGML_CUDA_GRAPH_OPT"); - return env != nullptr && atoi(env) == 1; - }(); - - if (!enable_graph_optimization) { - return; - } - - ggml_cuda_stream_context & stream_context = cuda_ctx->stream_context(); - stream_context.reset(); - - if (!use_cuda_graph || ggml_backend_cuda_get_device_count() != 1) { - return; - } - - // number of out-degrees for a particular node - std::unordered_map fan_out; - // reverse mapping of node to index in the cgraph - std::unordered_map node_indices; - - const auto & is_noop = [](const ggml_tensor * node) -> bool { - return ggml_is_empty(node) || node->op == GGML_OP_NONE || node->op == GGML_OP_RESHAPE || - node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE; - }; - - const auto & depends_on = [](const ggml_tensor * dst, const ggml_tensor * src) -> bool { - for (uint32_t s = 0; s < GGML_MAX_SRC; ++s) { - if (dst->src[s] == src) { - return true; - } - } - // implicit dependency if they view the same tensor - const ggml_tensor * dst2 = dst->view_src ? dst->view_src : dst; - const ggml_tensor * src2 = src->view_src ? src->view_src : src; - if (dst2 == src2) { - return true; - } - return false; - }; - - for (int node_idx = 0; node_idx < cgraph->n_nodes; node_idx++) { - const ggml_tensor * node = cgraph->nodes[node_idx]; - node_indices[node] = node_idx; - - if (is_noop(node)) { - continue; - } - for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { - const ggml_tensor * src = cgraph->nodes[node_idx]->src[src_idx]; - //TODO: check why nrows > 1 fails - if (node && !is_noop(node) && ggml_nrows(node) <= 1) { - fan_out[src] += 1; - } - } - } - - // Target Q, K, V for concurrency - // this is a more general way to find nodes which can be candidates for concurrency (although it has not been tested for anything else): - // 1. find fan-out (fork) nodes where the same input is used at least N times (in QKV, it would be "attn-norm") - // 2. find the join node, where 2 or more of the outputs are required (in QKV, this would "KQ" or "flash-attn") - // 3. account for all branches from the fork to the join - // 4. To extend lifetimes of the tensors, we interleave the branches (see below for more details) - // 5. save the original cgraph and restore it in graph_compute, to enable fusion within streams - // See discussion: https://github.com/ggml-org/llama.cpp/pull/16991#issuecomment-3522620030 - - const int min_fan_out = 3; - const int max_fan_out = 3; - - // store {fork_idx, join_idx} - std::vector> concurrent_node_ranges; - - for (const auto & [root_node, count] : fan_out) { - if (count >= min_fan_out && count <= max_fan_out) { - const int root_node_idx = node_indices[root_node]; - - // only optimize for attn_norm - // TODO: make this more generic - if (!strstr(root_node->name, "attn_norm")) { - continue; - } - - bool is_part_of_event = false; - for (const auto & [start, end] : concurrent_node_ranges) { - if (root_node_idx >= start && root_node_idx <= end) { - is_part_of_event = true; - } - } - - if (is_part_of_event) { - continue; - } - - std::vector> nodes_per_branch; - for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { - const ggml_tensor * node = cgraph->nodes[i]; - if (!is_noop(node) && depends_on(node, root_node)) { - nodes_per_branch.push_back({ node }); - } - } - - GGML_ASSERT(nodes_per_branch.size() == (size_t) count); - - //find the join point - const ggml_tensor * join_node = nullptr; - - const auto & belongs_to_branch = [&](const ggml_tensor * node, - const std::vector & branch) -> bool { - for (const ggml_tensor * n : branch) { - if (depends_on(node, n)) { - return true; - } - } - return false; - }; - - for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { - const ggml_tensor * curr_node = cgraph->nodes[i]; - - int num_joins = 0; - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - if (belongs_to_branch(curr_node, nodes_per_branch[branch_idx])) { - num_joins++; - } - } - - if (num_joins >= 2) { - join_node = curr_node; - break; - } - - bool found_branch = false; - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - std::vector & branch_vec = nodes_per_branch[branch_idx]; - if (belongs_to_branch(curr_node, branch_vec)) { - //continue accumulating - if (std::find(branch_vec.begin(), branch_vec.end(), curr_node) == branch_vec.end()) { - branch_vec.push_back(curr_node); - } - found_branch = true; - } - } - - if (!found_branch && is_noop(curr_node)) { - // we can put it in any branch because it will be ignored - nodes_per_branch[0].push_back({ curr_node }); - } - } - - if (join_node) { - //Create ggml_cuda_concurrent_event - ggml_cuda_concurrent_event concurrent_event(nodes_per_branch.size()); - concurrent_event.join_node = join_node; - - for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { - for (const ggml_tensor * n : nodes_per_branch[branch_idx]) { - concurrent_event.stream_mapping[n] = branch_idx + 1; - } - } - - int fork_node_idx = node_indices[root_node]; - int join_node_idx = node_indices[join_node]; - - int current_branch_idx = 0; - int current_node_idx = fork_node_idx + 1; - const int n_branches = nodes_per_branch.size(); - - int total_branch_nodes = 0; - for (std::vector branch_nodes : nodes_per_branch) { - total_branch_nodes += branch_nodes.size(); - } - - // there are other nodes in the middle which are unaccounted for - // usually (cpy) nodes, then ignore this fork - if (join_node_idx - fork_node_idx - 1 != total_branch_nodes) { - GGML_LOG_DEBUG( - "Skipping %s because the number of nodes in the middle is not equal to the total number of " - "branch nodes %d != %d\n", - root_node->name, join_node_idx - fork_node_idx - 1, total_branch_nodes); - continue; - } - - // Save the original order of nodes in this region before interleaving - // This is used later to restore grouping for fusion within streams - concurrent_event.original_order.reserve(total_branch_nodes); - for (int i = fork_node_idx + 1; i < join_node_idx; ++i) { - concurrent_event.original_order.push_back(cgraph->nodes[i]); - } - - std::unordered_map & concurrent_events = cuda_ctx->stream_context().concurrent_events; - GGML_ASSERT(concurrent_events.find(root_node) == concurrent_events.end()); - concurrent_events.emplace(root_node, std::move(concurrent_event)); - GGML_LOG_DEBUG("Adding stream at node %s %p\n", root_node->name, root_node); - concurrent_node_ranges.emplace_back(fork_node_idx, join_node_idx); - - // interleave tensors to extend lifetimes so that ggml graph doesn't recycle them - // example transformation: - // [attn-norm, QMul, QNorm, QRope, KMul, KNorm, KRope, VMul, attn] -> - // [attn-norm, QMul, KMul, VMul, QNorm, VNorm, QRope, KRope, attn] - while (current_node_idx < join_node_idx) { - std::vector & branch_nodes = nodes_per_branch[current_branch_idx]; - - bool has_node = false; - for (std::vector branch_node : nodes_per_branch) { - has_node |= branch_node.size() > 0; - } - - GGML_ASSERT(has_node); - - if (branch_nodes.empty()) { - current_branch_idx = (current_branch_idx + 1) % n_branches; - continue; - } - - cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); - current_node_idx++; - branch_nodes.erase(branch_nodes.begin()); - - // append all empty nodes - while (!branch_nodes.empty() && is_noop(branch_nodes.front())) { - cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); - current_node_idx++; - branch_nodes.erase(branch_nodes.begin()); - } - - current_branch_idx = (current_branch_idx + 1) % n_branches; - } - } - } - } -} - -static const ggml_backend_i ggml_backend_cuda_interface = { - /* .get_name = */ ggml_backend_cuda_get_name, - /* .free = */ ggml_backend_cuda_free, - /* .set_tensor_async = */ ggml_backend_cuda_set_tensor_async, - /* .get_tensor_async = */ ggml_backend_cuda_get_tensor_async, - /* .get_tensor_2d_async = */ ggml_backend_cuda_set_tensor_2d_async, - /* .set_tensor_2d_async = */ ggml_backend_cuda_get_tensor_2d_async, - /* .cpy_tensor_async = */ ggml_backend_cuda_cpy_tensor_async, - /* .synchronize = */ ggml_backend_cuda_synchronize, - /* .graph_plan_create = */ NULL, - /* .graph_plan_free = */ NULL, - /* .graph_plan_update = */ NULL, - /* .graph_plan_compute = */ NULL, - /* .graph_compute = */ ggml_backend_cuda_graph_compute, - /* .event_record = */ ggml_backend_cuda_event_record, - /* .event_wait = */ ggml_backend_cuda_event_wait, - /* .graph_optimize = */ ggml_backend_cuda_graph_optimize, -}; - -static ggml_guid_t ggml_backend_cuda_guid() { - static ggml_guid guid = { 0x2c, 0xdd, 0xe8, 0x1c, 0x65, 0xb3, 0x65, 0x73, 0x6a, 0x12, 0x88, 0x61, 0x1c, 0xc9, 0xdc, 0x25 }; - return &guid; -} - -bool ggml_backend_is_cuda(ggml_backend_t backend) { - return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_cuda_guid()); -} - -int ggml_backend_cuda_get_device_count() { - return ggml_cuda_info().device_count; -} - -void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) { - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, device)); - snprintf(description, description_size, "%s", prop.name); -} - -void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) { - ggml_cuda_set_device(device); - - CUDA_CHECK(cudaMemGetInfo(free, total)); -} - -bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) { - if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { - return false; - } - -#if CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) || defined(GGML_USE_HIP) - cudaError_t err = cudaHostRegister(buffer, size, cudaHostRegisterPortable | cudaHostRegisterReadOnly); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - - GGML_LOG_DEBUG("%s: failed to register %.2f MiB of pinned memory: %s\n", __func__, - size / 1024.0 / 1024.0, cudaGetErrorString(err)); - return false; - } - return true; -#else - GGML_UNUSED(buffer); - GGML_UNUSED(size); - return false; -#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) -} - -void ggml_backend_cuda_unregister_host_buffer(void * buffer) { - if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { - return; - } - - cudaError_t err = cudaHostUnregister(buffer); - if (err != cudaSuccess) { - // clear the error - (void)cudaGetLastError(); - } -} - - -// backend device - -struct ggml_backend_cuda_device_context { - int device; - std::string name; - std::string description; - std::string pci_bus_id; - int op_offload_min_batch_size; -}; - -static const char * ggml_backend_cuda_device_get_name(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ctx->name.c_str(); -} - -static const char * ggml_backend_cuda_device_get_description(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ctx->description.c_str(); -} - -#if defined(__linux__) -// Helper function to get available memory from /proc/meminfo for UMA systems -static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_kb, long * free_swap_kb) { - FILE * meminfo_file = nullptr; - // 2KB buffer for reading /proc/meminfo since it does not report size info, should be enough - const size_t BUFFER_SIZE = 2048; - auto file_buffer = std::make_unique(BUFFER_SIZE); - size_t bytes_read = 0; - long huge_tlb_total_pages = -1; - long huge_tlb_free_pages = -1; - long huge_tlb_page_size = -1; - - if (available_memory_kb == nullptr || free_swap_kb == nullptr) { - return false; - } - - meminfo_file = fopen("/proc/meminfo", "r"); - if (meminfo_file == nullptr) { - GGML_LOG_ERROR("%s: failed to open /proc/meminfo\n", __func__); - return false; - } - - // Read file into buffer - bytes_read = fread(file_buffer.get(), 1, BUFFER_SIZE - 1, meminfo_file); - fclose(meminfo_file); - - if (bytes_read == 0) { - GGML_LOG_ERROR("%s: failed to read from /proc/meminfo\n", __func__); - return false; - } - file_buffer[bytes_read] = '\0'; - - *available_memory_kb = -1; - *free_swap_kb = -1; - - // Parse the file buffer line by line - char * line = file_buffer.get(); - char * line_next; - while (line < file_buffer.get() + bytes_read) { - // Find the end of the current line - line_next = strchr(line, '\n'); - if (line_next != nullptr) { - *line_next = '\0'; - line_next++; - } else { - line_next = file_buffer.get() + bytes_read; - } - - long value; - if (sscanf(line, "MemAvailable: %ld kB", &value) == 1) { - *available_memory_kb = value; - } else if (sscanf(line, "SwapFree: %ld kB", &value) == 1) { - *free_swap_kb = value; - } else if (sscanf(line, "HugePages_Total: %ld", &value) == 1) { - huge_tlb_total_pages = value; - } else if (sscanf(line, "HugePages_Free: %ld", &value) == 1) { - huge_tlb_free_pages = value; - } else if (sscanf(line, "Hugepagesize: %ld kB", &value) == 1) { - huge_tlb_page_size = value; - } - - line = line_next; - } - - if (huge_tlb_total_pages != 0 && huge_tlb_total_pages != -1) { - *available_memory_kb = huge_tlb_free_pages * huge_tlb_page_size; - - // Hugetlbfs pages are not swappable. - *free_swap_kb = 0; - } - - GGML_LOG_DEBUG("%s: final available_memory_kb: %ld\n", __func__, *available_memory_kb); - return true; -} -#endif // defined(__linux__) - -static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - ggml_cuda_set_device(ctx->device); - CUDA_CHECK(cudaMemGetInfo(free, total)); - -// ref: https://github.com/ggml-org/llama.cpp/pull/17368 -#if defined(__linux__) - // Check if this is a UMA (Unified Memory Architecture) system - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device)); - - // Check if UMA is explicitly enabled via environment variable - bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr; - bool is_uma = prop.integrated > 0 || uma_env; - - if (is_uma) { - // For UMA systems (like DGX Spark), use system memory info - long available_memory_kb = 0; - long free_swap_kb = 0; - - if (ggml_backend_cuda_get_available_uma_memory(&available_memory_kb, &free_swap_kb) && available_memory_kb > 0) { - *free = (size_t)available_memory_kb * 1024; - } else { - GGML_LOG_ERROR("%s: /proc/meminfo reading failed, using cudaMemGetInfo\n", __func__); - } - } -#endif // defined(__linux__) - -} - -static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) { - GGML_UNUSED(dev); - return GGML_BACKEND_DEVICE_TYPE_GPU; -} - -static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_backend_dev_props * props) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - - props->name = ggml_backend_cuda_device_get_name(dev); - props->description = ggml_backend_cuda_device_get_description(dev); - props->type = ggml_backend_cuda_device_get_type(dev); - props->device_id = ctx->pci_bus_id.empty() ? nullptr : ctx->pci_bus_id.c_str(); - ggml_backend_cuda_device_get_memory(dev, &props->memory_free, &props->memory_total); - - bool host_buffer = getenv("GGML_CUDA_NO_PINNED") == nullptr; -#ifdef GGML_CUDA_NO_PEER_COPY - bool events = false; -#else - bool events = true; -#endif - - props->caps = { - /* .async = */ true, - /* .host_buffer = */ host_buffer, - /* .buffer_from_host_ptr = */ false, - /* .events = */ events, - }; -} - -static ggml_backend_t ggml_backend_cuda_device_init_backend(ggml_backend_dev_t dev, const char * params) { - GGML_UNUSED(params); - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ggml_backend_cuda_init(ctx->device); -} - -static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_buffer_type(ggml_backend_dev_t dev) { - ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; - return ggml_backend_cuda_buffer_type(ctx->device); -} - -static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_host_buffer_type(ggml_backend_dev_t dev) { - GGML_UNUSED(dev); - return ggml_backend_cuda_host_buffer_type(); -} - -// TODO: move these functions here -static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - - // split buffers can only be used with GGML_OP_MUL_MAT - if (op->op != GGML_OP_MUL_MAT) { - for (int i = 0; i < GGML_MAX_SRC; i++) { - if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda_split(op->src[i]->buffer->buft)) { - return false; - } - } - } - - // check if all the sources are allocated on this device - for (int i = 0; i < GGML_MAX_SRC; i++) { - if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda(op->src[i]->buffer->buft)) { - ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)op->src[i]->buffer->buft->context; - if (buft_ctx->device != dev_ctx->device) { - return false; - } - } - } - - switch (op->op) { - case GGML_OP_UNARY: - switch (ggml_get_unary_op(op)) { - case GGML_UNARY_OP_ABS: - case GGML_UNARY_OP_SGN: - case GGML_UNARY_OP_NEG: - case GGML_UNARY_OP_STEP: - case GGML_UNARY_OP_GELU: - case GGML_UNARY_OP_SILU: - case GGML_UNARY_OP_RELU: - case GGML_UNARY_OP_SIGMOID: - case GGML_UNARY_OP_HARDSIGMOID: - case GGML_UNARY_OP_HARDSWISH: - case GGML_UNARY_OP_GELU_ERF: - case GGML_UNARY_OP_GELU_QUICK: - case GGML_UNARY_OP_TANH: - case GGML_UNARY_OP_EXP: - case GGML_UNARY_OP_EXPM1: - case GGML_UNARY_OP_SOFTPLUS: - case GGML_UNARY_OP_ELU: - case GGML_UNARY_OP_XIELU: - case GGML_UNARY_OP_FLOOR: - case GGML_UNARY_OP_CEIL: - case GGML_UNARY_OP_ROUND: - case GGML_UNARY_OP_TRUNC: - // TODO: should become: - //return ggml_is_contiguous_rows(op->src[0]); - return ggml_is_contiguous(op->src[0]); - default: - return false; - } - break; - case GGML_OP_GLU: - switch (ggml_get_glu_op(op)) { - case GGML_GLU_OP_REGLU: - case GGML_GLU_OP_GEGLU: - case GGML_GLU_OP_SWIGLU: - case GGML_GLU_OP_SWIGLU_OAI: - case GGML_GLU_OP_GEGLU_ERF: - case GGML_GLU_OP_GEGLU_QUICK: - return ggml_is_contiguous_1(op->src[0]); - default: - return false; - } - break; - case GGML_OP_MUL_MAT: - case GGML_OP_MUL_MAT_ID: - { - struct ggml_tensor * a = op->src[0]; - struct ggml_tensor * b = op->src[1]; - if (a->buffer && ggml_backend_buft_is_cuda_split(a->buffer->buft)) { - if (a->ne[2] > 1 || a->ne[3] > 1) { - return false; - } - // for small weight matrices the active device can end up without any rows, don't use row split in those cases - // this avoids some edge cases (and the performance would not be good anyways) - ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) a->buffer->buft->context; - int64_t row_low; - int64_t row_high; - get_row_split(&row_low, &row_high, a, buft_ctx->tensor_split, dev_ctx->device); - if (row_low == row_high) { - return false; - } - } - if (b->type == GGML_TYPE_F16 && a->type != GGML_TYPE_F16) { - return false; - } -#ifdef GGML_USE_MUSA - const int cc = ggml_cuda_info().devices[dev_ctx->device].cc; - if (b->ne[2]*b->ne[3] > 1 && !ggml_is_transposed(a) && !ggml_is_transposed(b)) { - if (GGML_CUDA_CC_IS_QY1(cc) && op->op == GGML_OP_MUL_MAT && - a->type == GGML_TYPE_F16 && b->type == GGML_TYPE_F16) { - return false; - } - if (GGML_CUDA_CC_IS_QY2(cc) && op->op == GGML_OP_MUL_MAT_ID && - a->type == GGML_TYPE_Q2_K && b->type == GGML_TYPE_F32) { - return false; - } - } -#endif // GGML_USE_MUSA - switch (a->type) { - case GGML_TYPE_F32: - case GGML_TYPE_F16: - case GGML_TYPE_Q1_0: - case GGML_TYPE_Q4_0: - case GGML_TYPE_Q4_1: - case GGML_TYPE_Q5_0: - case GGML_TYPE_Q5_1: - case GGML_TYPE_Q8_0: - case GGML_TYPE_MXFP4: - case GGML_TYPE_NVFP4: - case GGML_TYPE_Q2_K: - case GGML_TYPE_Q3_K: - case GGML_TYPE_Q4_K: - case GGML_TYPE_Q5_K: - case GGML_TYPE_Q6_K: - case GGML_TYPE_Q8_K: - case GGML_TYPE_IQ1_M: - case GGML_TYPE_IQ1_S: - case GGML_TYPE_IQ2_S: - case GGML_TYPE_IQ2_XS: - case GGML_TYPE_IQ2_XXS: - case GGML_TYPE_IQ3_S: - case GGML_TYPE_IQ3_XXS: - case GGML_TYPE_IQ4_NL: - case GGML_TYPE_IQ4_XS: - case GGML_TYPE_BF16: - return true; - default: - return false; - } - } break; - case GGML_OP_OUT_PROD: - return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32; - case GGML_OP_GET_ROWS: - { - switch (op->src[0]->type) { - case GGML_TYPE_F16: - case GGML_TYPE_F32: - case GGML_TYPE_BF16: - case GGML_TYPE_I32: - case GGML_TYPE_Q1_0: - case GGML_TYPE_Q4_0: - case GGML_TYPE_Q4_1: - case GGML_TYPE_Q5_0: - case GGML_TYPE_Q5_1: - case GGML_TYPE_Q8_0: - return true; - default: - return false; - } - } break; - case GGML_OP_GET_ROWS_BACK: - { - return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->ne[2] == 1 && op->ne[3] == 1; - } break; - case GGML_OP_SET_ROWS: - { - return (op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_BF16 || - op->type == GGML_TYPE_Q4_0 || op->type == GGML_TYPE_Q4_1 || op->type == GGML_TYPE_Q5_0 || - op->type == GGML_TYPE_Q5_1 || op->type == GGML_TYPE_Q8_0 || op->type == GGML_TYPE_IQ4_NL) && - op->src[0]->type == GGML_TYPE_F32 && - (op->src[1]->type == GGML_TYPE_I64 || op->src[1]->type == GGML_TYPE_I32); - } break; - case GGML_OP_SET: - { - const ggml_type t = op->type; - return (t == GGML_TYPE_F32 || t == GGML_TYPE_I32) && - t == op->src[0]->type && - t == op->src[1]->type; - } break; - case GGML_OP_CPY: - { - ggml_type src0_type = op->src[0]->type; - ggml_type src1_type = op->src[1]->type; - if ((src0_type == GGML_TYPE_F32 || src0_type == GGML_TYPE_BF16 || src0_type == GGML_TYPE_F16) && - (src1_type == GGML_TYPE_F32 || src1_type == GGML_TYPE_BF16 || src1_type == GGML_TYPE_F16) - ) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q8_0) { - return true; - } - if (src0_type == GGML_TYPE_Q8_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_0) { - return true; - } - if (src0_type == GGML_TYPE_Q4_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_1) { - return true; - } - if (src0_type == GGML_TYPE_Q4_1 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_0) { - return true; - } - if (src0_type == GGML_TYPE_Q5_0 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_1) { - return true; - } - if (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_IQ4_NL) { - return true; - } - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) { - return true; - } - if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_F32) { - return true; - } - if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_I32) { - return true; - } - if (src0_type == src1_type && ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1])) { - return true; - } - return false; - } break; - case GGML_OP_DUP: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_ARGMAX: - case GGML_OP_COUNT_EQUAL: - { - return true; - } break; - case GGML_OP_REPEAT: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_REPEAT_BACK: - return op->type == GGML_TYPE_F32 && (op->src[0]->ne[2]*op->src[0]->ne[3]) <= (1 << 15); - case GGML_OP_CONCAT: - { - ggml_type src0_type = op->src[0]->type; - return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; - } break; - case GGML_OP_CONV_TRANSPOSE_1D: - { - ggml_type src0_type = op->src[0]->type; - ggml_type src1_type = op->src[1]->type; - if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_F32) { - return true; - } - return false; - } break; - case GGML_OP_SILU_BACK: - return ggml_is_contiguous(op->src[0]) && op->src[0]->type == GGML_TYPE_F32; - break; - case GGML_OP_NORM: - case GGML_OP_RMS_NORM: - case GGML_OP_L2_NORM: - return true; - case GGML_OP_RMS_NORM_BACK: - return ggml_is_contiguous(op->src[0]); - break; - case GGML_OP_NONE: - case GGML_OP_RESHAPE: - case GGML_OP_VIEW: - case GGML_OP_PERMUTE: - case GGML_OP_TRANSPOSE: - case GGML_OP_ADD: - case GGML_OP_ADD_ID: - case GGML_OP_ADD1: - case GGML_OP_SUB: - case GGML_OP_MUL: - case GGML_OP_DIV: - case GGML_OP_SCALE: - case GGML_OP_SQR: - case GGML_OP_SQRT: - case GGML_OP_SIN: - case GGML_OP_COS: - case GGML_OP_CLAMP: - case GGML_OP_LOG: - return true; - case GGML_OP_SSM_SCAN: { - if (op->src[3]->ne[0] == 1) { - // Mamba2 - // (kernel only supports (d_state == 128 || d_state == 256) && d_head % 16 == 0) - return (op->src[0]->ne[0] == 128 || op->src[0]->ne[0] == 256) && op->src[0]->ne[1] % 16 == 0; - } else { - // Mamba - // (kernel only supports d_state == 16, d_head == 1, n_head % 128 == 0, n_group == 1) - return op->src[0]->ne[0] == 16 && op->src[0]->ne[1] == 1 && op->src[0]->ne[2] % 128 == 0 && op->src[4]->ne[1] == 1; - } - } - case GGML_OP_SSM_CONV: { - // assumes d_inner % threads == 0 - return op->src[0]->ne[1] % 128 == 0; - } - case GGML_OP_CONT: - return true; - case GGML_OP_DIAG_MASK_INF: - return true; - case GGML_OP_SOFT_MAX: - return true; - case GGML_OP_SOFT_MAX_BACK: { - float max_bias = 0.0f; - memcpy(&max_bias, (const float *) op->op_params + 1, sizeof(float)); - return max_bias == 0.0f; - } - case GGML_OP_ROLL: - if(op->src[0]->type == GGML_TYPE_F32) { - return true; - } - return false; - case GGML_OP_ROPE: - case GGML_OP_ROPE_BACK: { - return op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]); - } - case GGML_OP_IM2COL: - case GGML_OP_IM2COL_3D: - case GGML_OP_CONV_2D: - case GGML_OP_CONV_2D_DW: - case GGML_OP_CONV_TRANSPOSE_2D: - case GGML_OP_POOL_2D: - return true; - case GGML_OP_ACC: - // TODO: extend support like so: - //return ggml_is_contiguous_rows(op->src[0]) && ggml_is_contiguous_rows(op->src[1]); - return ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1]); - case GGML_OP_SUM: - return ggml_is_contiguous_rows(op->src[0]); - case GGML_OP_TOP_K: - case GGML_OP_ARGSORT: -#ifndef GGML_CUDA_USE_CUB - return op->src[0]->ne[0] <= 1024; -#else - return true; -#endif - case GGML_OP_SUM_ROWS: - case GGML_OP_MEAN: - case GGML_OP_GROUP_NORM: - return ggml_is_contiguous(op->src[0]); - case GGML_OP_PAD: - return true; - case GGML_OP_UPSCALE: - case GGML_OP_PAD_REFLECT_1D: - case GGML_OP_ARANGE: - case GGML_OP_TIMESTEP_EMBEDDING: - case GGML_OP_LEAKY_RELU: - case GGML_OP_RWKV_WKV6: - case GGML_OP_GATED_LINEAR_ATTN: - case GGML_OP_RWKV_WKV7: - return true; - case GGML_OP_GATED_DELTA_NET: - //TODO: enable once MUSA compiler is solved https://github.com/ggml-org/llama.cpp/pull/19504#issuecomment-4018634327 -#ifdef GGML_USE_MUSA - return false; -#else - return true; -#endif // GGML_USE_MUSA - case GGML_OP_FLASH_ATTN_EXT: - return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op); - case GGML_OP_CROSS_ENTROPY_LOSS: - case GGML_OP_CROSS_ENTROPY_LOSS_BACK: - case GGML_OP_OPT_STEP_ADAMW: - case GGML_OP_OPT_STEP_SGD: - case GGML_OP_FILL: - case GGML_OP_CUMSUM: - case GGML_OP_TRI: - case GGML_OP_DIAG: - case GGML_OP_SOLVE_TRI: - return true; - - default: - return false; - } -} - -static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated; - return (((ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft))); -} - -static int64_t get_op_batch_size(const ggml_tensor * op) { - switch (op->op) { - case GGML_OP_GET_ROWS: - return 0; - case GGML_OP_MUL_MAT: - return op->ne[1]; - case GGML_OP_MUL_MAT_ID: - case GGML_OP_ROPE: - case GGML_OP_ROPE_BACK: - return op->ne[2]; - default: - return ggml_nrows(op); - } -} - -static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) { - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; - - return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size; -} - -static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_t dev) { -#ifdef GGML_CUDA_NO_PEER_COPY - return nullptr; -#else - ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *)dev->context; - - ggml_cuda_set_device(dev_ctx->device); - - cudaEvent_t event; - CUDA_CHECK(cudaEventCreateWithFlags(&event, cudaEventDisableTiming)); - - return new ggml_backend_event { - /* .device = */ dev, - /* .context = */ event, - }; -#endif -} - -static void ggml_backend_cuda_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) { - GGML_UNUSED(dev); - - CUDA_CHECK(cudaEventDestroy((cudaEvent_t)event->context)); - delete event; -} - -static void ggml_backend_cuda_device_event_synchronize(ggml_backend_dev_t dev, ggml_backend_event_t event) { - GGML_UNUSED(dev); - CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context)); -} - -static const ggml_backend_device_i ggml_backend_cuda_device_interface = { - /* .get_name = */ ggml_backend_cuda_device_get_name, - /* .get_description = */ ggml_backend_cuda_device_get_description, - /* .get_memory = */ ggml_backend_cuda_device_get_memory, - /* .get_type = */ ggml_backend_cuda_device_get_type, - /* .get_props = */ ggml_backend_cuda_device_get_props, - /* .init_backend = */ ggml_backend_cuda_device_init_backend, - /* .get_buffer_type = */ ggml_backend_cuda_device_get_buffer_type, - /* .get_host_buffer_type = */ ggml_backend_cuda_device_get_host_buffer_type, - /* .buffer_from_host_ptr = */ NULL, - /* .supports_op = */ ggml_backend_cuda_device_supports_op, - /* .supports_buft = */ ggml_backend_cuda_device_supports_buft, - /* .offload_op = */ ggml_backend_cuda_device_offload_op, - /* .event_new = */ ggml_backend_cuda_device_event_new, - /* .event_free = */ ggml_backend_cuda_device_event_free, - /* .event_synchronize = */ ggml_backend_cuda_device_event_synchronize, -}; - -// backend reg - -struct ggml_backend_cuda_reg_context { - std::vector devices; -}; - -static const char * ggml_backend_cuda_reg_get_name(ggml_backend_reg_t reg) { - GGML_UNUSED(reg); - return GGML_CUDA_NAME; -} - -static size_t ggml_backend_cuda_reg_get_device_count(ggml_backend_reg_t reg) { - ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; - return ctx->devices.size(); -} - -static ggml_backend_dev_t ggml_backend_cuda_reg_get_device(ggml_backend_reg_t reg, size_t index) { - ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; - GGML_ASSERT(index < ctx->devices.size()); - return ctx->devices[index]; -} - -static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t reg) { - static std::vector features = []() { - std::vector features; - #define _STRINGIFY(...) #__VA_ARGS__ - #define STRINGIFY(...) _STRINGIFY(__VA_ARGS__) - - #ifdef __CUDA_ARCH_LIST__ - features.push_back({ "ARCHS", STRINGIFY(__CUDA_ARCH_LIST__) }); - #endif - - #ifdef GGML_CUDA_FORCE_MMQ - features.push_back({ "FORCE_MMQ", "1" }); - #endif - - #ifdef GGML_CUDA_FORCE_CUBLAS - features.push_back({ "FORCE_CUBLAS", "1" }); - #endif - - #ifndef GGML_USE_VMM - features.push_back({ "NO_VMM", "1" }); - #endif - - #ifdef GGML_CUDA_NO_PEER_COPY - features.push_back({ "NO_PEER_COPY", "1" }); - #endif - - #ifdef GGML_CUDA_USE_GRAPHS - features.push_back({ "USE_GRAPHS", "1" }); - #endif - - #ifdef GGML_CUDA_PEER_MAX_BATCH_SIZE - features.push_back({ "PEER_MAX_BATCH_SIZE", STRINGIFY(GGML_CUDA_PEER_MAX_BATCH_SIZE) }); - #endif - - #ifdef GGML_CUDA_FA_ALL_QUANTS - features.push_back({ "FA_ALL_QUANTS", "1" }); - #endif - - { - const auto & info = ggml_cuda_info(); - for (int id = 0; id < info.device_count; ++id) { - if (blackwell_mma_available(info.devices[id].cc)) { - features.push_back({ "BLACKWELL_NATIVE_FP4", "1"}); - break; - } - } - } - - #undef _STRINGIFY - #undef STRINGIFY - - features.push_back({ nullptr, nullptr }); - - return features; - }(); - - return features.data(); - - GGML_UNUSED(reg); -} - -static void * ggml_backend_cuda_reg_get_proc_address(ggml_backend_reg_t reg, const char * name) { - GGML_UNUSED(reg); - if (strcmp(name, "ggml_backend_comm_init") == 0) { - return (void *)ggml_backend_cuda_comm_init; - } - if (strcmp(name, "ggml_backend_comm_free") == 0) { - return (void *)ggml_backend_cuda_comm_free; - } - if (strcmp(name, "ggml_backend_comm_allreduce_tensor") == 0) { - return (void *)ggml_backend_cuda_comm_allreduce_tensor; - } - if (strcmp(name, "ggml_backend_split_buffer_type") == 0) { - return (void *)ggml_backend_cuda_split_buffer_type; - } - if (strcmp(name, "ggml_backend_register_host_buffer") == 0) { - return (void *)ggml_backend_cuda_register_host_buffer; - } - if (strcmp(name, "ggml_backend_unregister_host_buffer") == 0) { - return (void *)ggml_backend_cuda_unregister_host_buffer; - } - if (strcmp(name, "ggml_backend_get_features") == 0) { - return (void *)ggml_backend_cuda_get_features; - } - return nullptr; -} - -static const ggml_backend_reg_i ggml_backend_cuda_reg_interface = { - /* .get_name = */ ggml_backend_cuda_reg_get_name, - /* .get_device_count = */ ggml_backend_cuda_reg_get_device_count, - /* .get_device = */ ggml_backend_cuda_reg_get_device, - /* .get_proc_address = */ ggml_backend_cuda_reg_get_proc_address, -}; - -// backend registry -ggml_backend_reg_t ggml_backend_cuda_reg() { - static ggml_backend_reg reg; - static bool initialized = false; - - { - static std::mutex mutex; - std::lock_guard lock(mutex); - if (!initialized) { - ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context; - const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32; - - for (int i = 0; i < ggml_cuda_info().device_count; i++) { - ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context; - dev_ctx->device = i; - dev_ctx->name = GGML_CUDA_NAME + std::to_string(i); - - cudaDeviceProp prop; - CUDA_CHECK(cudaGetDeviceProperties(&prop, i)); - dev_ctx->description = prop.name; - - char pci_bus_id[16] = {}; - snprintf(pci_bus_id, sizeof(pci_bus_id), "%04x:%02x:%02x.0", prop.pciDomainID, prop.pciBusID, prop.pciDeviceID); - dev_ctx->pci_bus_id = pci_bus_id; - dev_ctx->op_offload_min_batch_size = min_batch_size; - - ggml_backend_dev_t dev = new ggml_backend_device { - /* .iface = */ ggml_backend_cuda_device_interface, - /* .reg = */ ®, - /* .context = */ dev_ctx - }; - ctx->devices.push_back(dev); - } - - reg = ggml_backend_reg { - /* .api_version = */ GGML_BACKEND_API_VERSION, - /* .iface = */ ggml_backend_cuda_reg_interface, - /* .context = */ ctx - }; - } - - initialized = true; - } - - return ® -} - -ggml_backend_t ggml_backend_cuda_init(int device) { - if (device < 0 || device >= ggml_backend_cuda_get_device_count()) { - GGML_LOG_ERROR("%s: invalid device %d\n", __func__, device); - return nullptr; - } - - ggml_backend_cuda_context * ctx = new ggml_backend_cuda_context(device); - if (ctx == nullptr) { - GGML_LOG_ERROR("%s: failed to allocate context\n", __func__); - return nullptr; - } - - ggml_backend_t cuda_backend = new ggml_backend { - /* .guid = */ ggml_backend_cuda_guid(), - /* .iface = */ ggml_backend_cuda_interface, - /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device), - /* .context = */ ctx, - }; - - return cuda_backend; -} - -GGML_BACKEND_DL_IMPL(ggml_backend_cuda_reg) +#include "ggml-cuda.h" +#include "ggml-impl.h" +#include "ggml-backend-impl.h" + +#include "ggml-cuda/allreduce.cuh" +#include "ggml-cuda/comm.cuh" +#include "ggml-cuda/common.cuh" +#include "ggml-cuda/acc.cuh" +#include "ggml-cuda/add-id.cuh" +#include "ggml-cuda/arange.cuh" +#include "ggml-cuda/argmax.cuh" +#include "ggml-cuda/argsort.cuh" +#include "ggml-cuda/binbcast.cuh" +#include "ggml-cuda/clamp.cuh" +#include "ggml-cuda/concat.cuh" +#include "ggml-cuda/conv-transpose-1d.cuh" +#include "ggml-cuda/conv2d.cuh" +#include "ggml-cuda/conv2d-dw.cuh" +#include "ggml-cuda/conv2d-transpose.cuh" +#include "ggml-cuda/convert.cuh" +#include "ggml-cuda/count-equal.cuh" +#include "ggml-cuda/cpy.cuh" +#include "ggml-cuda/cross-entropy-loss.cuh" +#include "ggml-cuda/cumsum.cuh" +#include "ggml-cuda/diagmask.cuh" +#include "ggml-cuda/diag.cuh" +#include "ggml-cuda/fattn.cuh" +#include "ggml-cuda/getrows.cuh" +#include "ggml-cuda/im2col.cuh" +#include "ggml-cuda/mmf.cuh" +#include "ggml-cuda/mmq.cuh" +#include "ggml-cuda/mmvf.cuh" +#include "ggml-cuda/mmvq.cuh" +#include "ggml-cuda/norm.cuh" +#include "ggml-cuda/opt-step-adamw.cuh" +#include "ggml-cuda/opt-step-sgd.cuh" +#include "ggml-cuda/out-prod.cuh" +#include "ggml-cuda/pad.cuh" +#include "ggml-cuda/pool2d.cuh" +#include "ggml-cuda/quantize.cuh" +#include "ggml-cuda/rope.cuh" +#include "ggml-cuda/roll.cuh" +#include "ggml-cuda/scale.cuh" +#include "ggml-cuda/softcap.cuh" +#include "ggml-cuda/softmax.cuh" +#include "ggml-cuda/ssm-conv.cuh" +#include "ggml-cuda/ssm-scan.cuh" +#include "ggml-cuda/sum.cuh" +#include "ggml-cuda/sumrows.cuh" +#include "ggml-cuda/top-k.cuh" +#include "ggml-cuda/mean.cuh" +#include "ggml-cuda/tsembd.cuh" +#include "ggml-cuda/topk-moe.cuh" +#include "ggml-cuda/unary.cuh" +#include "ggml-cuda/upscale.cuh" +#include "ggml-cuda/wkv.cuh" +#include "ggml-cuda/gla.cuh" +#include "ggml-cuda/gated_delta_net.cuh" +#include "ggml-cuda/set.cuh" +#include "ggml-cuda/set-rows.cuh" +#include "ggml-cuda/pad_reflect_1d.cuh" +#include "ggml-cuda/solve_tri.cuh" +#include "ggml-cuda/tri.cuh" +#include "ggml-cuda/cumsum.cuh" +#include "ggml-cuda/fill.cuh" +#include "ggml.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size"); + +[[noreturn]] +void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) { + int id = -1; // in case cudaGetDevice fails + (void)cudaGetDevice(&id); + + GGML_LOG_ERROR(GGML_CUDA_NAME " error: %s\n", msg); + GGML_LOG_ERROR(" current device: %d, in function %s at %s:%d\n", id, func, file, line); + GGML_LOG_ERROR(" %s\n", stmt); + // abort with GGML_ABORT to get a stack trace + GGML_ABORT(GGML_CUDA_NAME " error"); +} + +// this is faster on Windows +// probably because the Windows CUDA libraries forget to make this check before invoking the drivers +void ggml_cuda_set_device(int device) { + int current_device; + CUDA_CHECK(cudaGetDevice(¤t_device)); + + if (device == current_device) { + return; + } + + CUDA_CHECK(cudaSetDevice(device)); +} + +int ggml_cuda_get_device() { + int id; + CUDA_CHECK(cudaGetDevice(&id)); + return id; +} + +static cudaError_t ggml_cuda_device_malloc(void ** ptr, size_t size, int device) { + ggml_cuda_set_device(device); + cudaError_t err; + if (getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr) { + err = cudaMallocManaged(ptr, size); +#if defined(GGML_USE_HIP) + if (err == hipSuccess) { + // hipMemAdviseSetCoarseGrain is an optional performance hint; + // ignore errors (e.g. hipErrorInvalidValue on some APU/iGPU configs). + (void)cudaMemAdvise(*ptr, size, hipMemAdviseSetCoarseGrain, device); + (void)hipGetLastError(); // clear any error + } + + // fall back to cudaMalloc if not supported (e.g. on Windows) + if (err == hipErrorNotSupported) { + static bool warned_unsupported = false; + if (!warned_unsupported) { + GGML_LOG_WARN("hipMallocManaged unsupported, falling back to hipMalloc.\n"); + warned_unsupported = true; + } + + err = cudaMalloc(ptr, size); + } +#endif // defined(GGML_USE_HIP) + } else { + err = cudaMalloc(ptr, size); + } + return err; +} + +#if defined(GGML_USE_HIP) +static int ggml_cuda_parse_id(char devName[]) { + // A list of possible Target IDs can be found under the rocclr/clr repo in device.cpp + // these values are not stable so this is susceptible to breakage + // https://github.com/ROCm/clr/blob/amd-staging/rocclr/device/device.cpp + int archMajor = 0x0; + int archMinor = 0x0; + int archNum = GGML_CUDA_CC_OFFSET_AMD; + int archLen = strlen(devName); + char archName[archLen + 1]; + + // strip leading 'gfx' while copying into our buffer + if (archLen > 3) { + strcpy(archName, &devName[3]); + archLen -= 3; + } + + // trim trailing :xnack- or :sramecc- statuses + archLen = strcspn(archName, ":"); + archName[archLen] = '\0'; + + // tease out the version information + if (archLen > 8) { + // versions labeled generic use '-' as delimiter + // strip the trailing "-generic" then iterate through what remains + if ((strstr(archName, "-generic"))) { + archName[archLen - 8] = '\0'; + char * pch; + if ((pch = strtok(archName, "-"))) { + archMajor = (int)strtoul(pch, 0, 16); + if ((pch = strtok(NULL, "-"))) { + archMinor = 0x10 * (int)strtoul(pch, 0, 16); + } + } + } + } else if (archLen >= 3) { + // last two digits should be the minor * 0x10 + stepping + archMinor = (int)strtoul(&archName[archLen - 2], 0, 16); + archName[archLen - 2] = '\0'; + + // only the major version remains + archMajor = (int)strtoul(archName, 0, 16); + } + archNum += archMajor * 0x100; + archNum += archMinor; + return archNum; +} +#endif // defined(GGML_USE_HIP) + +static ggml_cuda_device_info ggml_cuda_init() { + ggml_cuda_device_info info = {}; + + cudaError_t err = cudaGetDeviceCount(&info.device_count); + if (err != cudaSuccess) { + GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err)); + return info; + } + + GGML_ASSERT(info.device_count <= GGML_CUDA_MAX_DEVICES); + + int64_t total_vram = 0; + for (int id = 0; id < info.device_count; ++id) { + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); + total_vram += prop.totalGlobalMem; + } + GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n", + __func__, info.device_count, (size_t)(total_vram / (1024 * 1024))); + total_vram = 0; + + std::vector> turing_devices_without_mma; + for (int id = 0; id < info.device_count; ++id) { + int device_vmm = 0; + +#if defined(GGML_USE_VMM) + CUdevice device; + CU_CHECK(cuDeviceGet(&device, id)); + CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device)); + + if (device_vmm) { + CUmemAllocationProp alloc_prop = {}; + alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; + alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + alloc_prop.location.id = id; + CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED)); + } +#endif // defined(GGML_USE_VMM) + info.devices[id].vmm = !!device_vmm; + + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, id)); + + info.default_tensor_split[id] = total_vram; + total_vram += prop.totalGlobalMem; + info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034) + info.devices[id].nsm = prop.multiProcessorCount; + info.devices[id].smpb = prop.sharedMemPerBlock; + info.devices[id].warp_size = prop.warpSize; + +#ifndef GGML_USE_MUSA + int supports_coop_launch = 0; + CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, id)); + info.devices[id].supports_cooperative_launch = !!supports_coop_launch; +#else + info.devices[id].supports_cooperative_launch = false; +#endif // !(GGML_USE_MUSA) + +#if defined(GGML_USE_HIP) + info.devices[id].smpbo = prop.sharedMemPerBlock; + + info.devices[id].cc = ggml_cuda_parse_id(prop.gcnArchName); + if ((info.devices[id].cc & 0xff00) == 0x0) { + GGML_LOG_WARN("invalid architecture ID received for device %d %s: %s cc %d.%d\n", + id, prop.name, prop.gcnArchName, prop.major, prop.minor); + + // Fallback to prop.major and prop.minor + if (prop.major > 0) { + info.devices[id].cc = GGML_CUDA_CC_OFFSET_AMD + prop.major * 0x100; + info.devices[id].cc += prop.minor * 0x10; + } + } + GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n", + id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff, + device_vmm ? "yes" : "no", prop.warpSize, + (size_t)(prop.totalGlobalMem / (1024 * 1024))); +#elif defined(GGML_USE_MUSA) + // FIXME: Ensure compatibility with varying warp sizes across different MUSA archs. + info.devices[id].warp_size = 32; + info.devices[id].smpbo = prop.sharedMemPerBlockOptin; + info.devices[id].cc = GGML_CUDA_CC_OFFSET_MTHREADS + prop.major * 0x100; + info.devices[id].cc += prop.minor * 0x10; + GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", + id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", + (size_t)(prop.totalGlobalMem / (1024 * 1024))); +#else + info.devices[id].smpbo = prop.sharedMemPerBlockOptin; + info.devices[id].cc = 100*prop.major + 10*prop.minor; + GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n", + id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no", + (size_t)(prop.totalGlobalMem / (1024 * 1024))); + std::string device_name(prop.name); + if (device_name == "NVIDIA GeForce MX450") { + turing_devices_without_mma.push_back({ id, device_name }); + } else if (device_name == "NVIDIA GeForce MX550") { + turing_devices_without_mma.push_back({ id, device_name }); + } else if (device_name.substr(0, 21) == "NVIDIA GeForce GTX 16") { + turing_devices_without_mma.push_back({ id, device_name }); + } + + // Temporary performance fix: + // Setting device scheduling strategy for iGPUs with cc121 to "spinning" to avoid delays in cuda synchronize calls. + // TODO: Check for future drivers the default scheduling strategy and + // remove this call again when cudaDeviceScheduleSpin is default. + if (prop.major == 12 && prop.minor == 1) { + CUDA_CHECK(cudaSetDevice(id)); + CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin)); + } + +#endif // defined(GGML_USE_HIP) + } + + if (ggml_cuda_highest_compiled_arch(GGML_CUDA_CC_TURING) >= GGML_CUDA_CC_TURING && !turing_devices_without_mma.empty()) { + GGML_LOG_INFO("The following devices will have suboptimal performance due to a lack of tensor cores:\n"); + for (size_t device_pos = 0; device_pos < turing_devices_without_mma.size(); device_pos++) { + GGML_LOG_INFO( + " Device %d: %s\n", turing_devices_without_mma[device_pos].first, turing_devices_without_mma[device_pos].second.c_str()); + } + GGML_LOG_INFO( + "Consider compiling with CMAKE_CUDA_ARCHITECTURES=61-virtual;80-virtual and DGGML_CUDA_FORCE_MMQ to force the use of the Pascal code for Turing.\n"); + } + + for (int id = 0; id < info.device_count; ++id) { + info.default_tensor_split[id] /= total_vram; + } + + // configure logging to stdout + // CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr)); + + if (getenv("GGML_CUDA_P2P") != nullptr) { + for (int id = 0; id < info.device_count; ++id) { + ggml_cuda_set_device(id); + for (int id_other = 0; id_other < info.device_count; ++id_other) { + if (id == id_other) { + continue; + } + int can_access_peer; + CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id, id_other)); + if (can_access_peer) { + CUDA_CHECK(cudaDeviceEnablePeerAccess(id_other, 0)); + } + } + } + } + + return info; +} + +const ggml_cuda_device_info & ggml_cuda_info() { + static ggml_cuda_device_info info = ggml_cuda_init(); + return info; +} + +// #define DEBUG_CUDA_MALLOC + +// buffer pool for cuda (legacy) +struct ggml_cuda_pool_leg : public ggml_cuda_pool { + static const int MAX_BUFFERS = 256; + + int device; + struct ggml_cuda_buffer { + void * ptr = nullptr; + size_t size = 0; + }; + + ggml_cuda_buffer buffer_pool[MAX_BUFFERS] = {}; + size_t pool_size = 0; + + explicit ggml_cuda_pool_leg(int device) : + device(device) { + } + + ~ggml_cuda_pool_leg() { + clear_pool(); + GGML_ASSERT(pool_size == 0); + } + + void clear_pool() { + ggml_cuda_set_device(device); + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer & b = buffer_pool[i]; + if (b.ptr != nullptr) { + CUDA_CHECK(cudaFree(b.ptr)); + pool_size -= b.size; + b.ptr = nullptr; + b.size = 0; + } + } + } + + void * alloc(size_t size, size_t * actual_size) override { +#ifdef DEBUG_CUDA_MALLOC + int nnz = 0; + size_t max_size = 0; +#endif + size_t best_diff = 1ull << 36; + int ibest = -1; + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer& b = buffer_pool[i]; + if (b.ptr != nullptr) { +#ifdef DEBUG_CUDA_MALLOC + ++nnz; + if (b.size > max_size) max_size = b.size; +#endif + if (b.size >= size) { + size_t diff = b.size - size; + if (diff < best_diff) { + best_diff = diff; + ibest = i; + if (!best_diff) { + void * ptr = b.ptr; + *actual_size = b.size; + b.ptr = nullptr; + b.size = 0; + return ptr; + } + } + } + } + } + if (ibest >= 0) { + ggml_cuda_buffer& b = buffer_pool[ibest]; + void * ptr = b.ptr; + *actual_size = b.size; + b.ptr = nullptr; + b.size = 0; + return ptr; + } + void * ptr; + size_t look_ahead_size = (size_t) (1.05 * size); + look_ahead_size = 256 * ((look_ahead_size + 255)/256); + ggml_cuda_set_device(device); + cudaError_t err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); + if (err == cudaErrorMemoryAllocation) { + (void)cudaGetLastError(); + const size_t cached_bytes = pool_size; + GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: alloc of %.2f MiB failed, flushing %.2f MiB of cached buffers and retrying\n", + device, look_ahead_size/1024.0/1024.0, cached_bytes/1024.0/1024.0); + CUDA_CHECK(cudaDeviceSynchronize()); + clear_pool(); + err = ggml_cuda_device_malloc(&ptr, look_ahead_size, device); + if (err == cudaSuccess) { + GGML_LOG_DEBUG(GGML_CUDA_NAME " pool[%d]: retry succeeded\n", device); + } + } + CUDA_CHECK(err); + *actual_size = look_ahead_size; + pool_size += look_ahead_size; +#ifdef DEBUG_CUDA_MALLOC + GGML_LOG_INFO("%s[%d]: %d buffers, max_size = %u MB, pool_size = %u MB, requested %u MB\n", __func__, device, nnz, + (uint32_t)(max_size / 1024 / 1024), (uint32_t)(pool_size / 1024 / 1024), (uint32_t)(size / 1024 / 1024)); +#endif + return ptr; + } + + void free(void * ptr, size_t size) override { + for (int i = 0; i < MAX_BUFFERS; ++i) { + ggml_cuda_buffer& b = buffer_pool[i]; + if (b.ptr == nullptr) { + b.ptr = ptr; + b.size = size; + return; + } + } + GGML_LOG_DEBUG(GGML_CUDA_NAME " buffer pool full, increase MAX_CUDA_BUFFERS\n"); + ggml_cuda_set_device(device); + CUDA_CHECK(cudaFree(ptr)); + pool_size -= size; + } +}; + +// pool with virtual memory +#if defined(GGML_USE_VMM) +struct ggml_cuda_pool_vmm : public ggml_cuda_pool { + static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB + + int device; + CUdeviceptr pool_addr = 0; + size_t pool_used = 0; + size_t pool_size = 0; + size_t granularity; +#if defined(GGML_USE_HIP) + std::vector> mappings; +#endif + + explicit ggml_cuda_pool_vmm(int device) : + device(device), + granularity(ggml_cuda_info().devices[device].vmm_granularity) { + } + + ~ggml_cuda_pool_vmm() { + if (pool_addr != 0) { +#if defined(GGML_USE_HIP) + // Workaround for https://github.com/ROCm/ROCR-Runtime/issues/285 + for (std::pair & mapping : mappings) { + CU_CHECK(cuMemUnmap(mapping.first, mapping.second)); + } +#else + CU_CHECK(cuMemUnmap(pool_addr, pool_size)); +#endif + CU_CHECK(cuMemAddressFree(pool_addr, CUDA_POOL_VMM_MAX_SIZE)); + } + } + + void * alloc(size_t size, size_t * actual_size) override { + // round up the allocation size to the alignment to ensure that all allocations are aligned for all data types + const size_t alignment = 128; + size = alignment * ((size + alignment - 1) / alignment); + + size_t avail = pool_size - pool_used; + + if (size > avail) { + // round up to the next multiple of the granularity + size_t reserve_size = size - avail; + reserve_size = granularity * ((reserve_size + granularity - 1) / granularity); + + GGML_ASSERT(pool_size + reserve_size <= CUDA_POOL_VMM_MAX_SIZE); + + // allocate more physical memory + CUmemAllocationProp prop = {}; + prop.type = CU_MEM_ALLOCATION_TYPE_PINNED; + prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + prop.location.id = device; + CUmemGenericAllocationHandle handle; + CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0)); + + // reserve virtual address space (if not already reserved) + if (pool_addr == 0) { + CU_CHECK(cuMemAddressReserve(&pool_addr, CUDA_POOL_VMM_MAX_SIZE, 0, 0, 0)); + } + + // map at the end of the pool + CUdeviceptr start_ptr = (CUdeviceptr)((char *)(pool_addr) + pool_size); + CU_CHECK(cuMemMap(start_ptr, reserve_size, 0, handle, 0)); +#if defined(GGML_USE_HIP) + mappings.push_back({start_ptr, reserve_size}); +#endif + + // the memory allocation handle is no longer needed after mapping + CU_CHECK(cuMemRelease(handle)); + + // set access + CUmemAccessDesc access = {}; + access.location.type = CU_MEM_LOCATION_TYPE_DEVICE; + access.location.id = device; + access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE; + CU_CHECK(cuMemSetAccess((CUdeviceptr)((char *)(pool_addr) + pool_size), reserve_size, &access, 1)); + + // add to the pool + pool_size += reserve_size; + + //printf("cuda pool[%d]: size increased to %llu MB (reserved %llu MB)\n", + // device, (unsigned long long) (pool_size/1024/1024), + // (unsigned long long) (reserve_size/1024/1024)); + } + + GGML_ASSERT(pool_addr != 0); + + void * ptr = (void *) ((CUdeviceptr)((char *)(pool_addr) + pool_used)); + *actual_size = size; + pool_used += size; + +#ifdef DEBUG_CUDA_MALLOC + printf("cuda pool[%d]: allocated %llu bytes at %llx\n", device, (unsigned long long) size, ptr); +#endif + + return ptr; + } + + void free(void * ptr, size_t size) override { +#ifdef DEBUG_CUDA_MALLOC + printf("cuda pool[%d]: freed %llu bytes at %llx\n", device, (unsigned long long) size, ptr); +#endif + + pool_used -= size; + + // all deallocations must be in reverse order of the allocations + GGML_ASSERT(ptr == (void *) ((char *)(pool_addr) + pool_used)); + } +}; +#endif // defined(GGML_USE_VMM) + +std::unique_ptr ggml_backend_cuda_context::new_pool_for_device(int device, + [[maybe_unused]] int stream_no) { +#if defined(GGML_USE_VMM) + if (ggml_cuda_info().devices[device].vmm) { + return std::unique_ptr(new ggml_cuda_pool_vmm(device)); + } +#endif // defined(GGML_USE_VMM) + return std::unique_ptr(new ggml_cuda_pool_leg(device)); +} + +// destroying a cuBLAS handle while a graph is being captured in a different thread can result in a CUDA error +// this lock is used to ensure that no cuBLAS handle is destroyed while a graph is being captured + +static std::mutex ggml_cuda_lock; +static std::condition_variable ggml_cuda_lock_cv; +static std::atomic ggml_cuda_lock_counter; + +ggml_backend_cuda_context::~ggml_backend_cuda_context() { + std::unique_lock lock(ggml_cuda_lock); + ggml_cuda_lock_cv.wait(lock, []{ return ggml_cuda_lock_counter.load(std::memory_order_relaxed) == 0; }); + + if (copy_event != nullptr) { + CUDA_CHECK(cudaEventDestroy(copy_event)); + } + for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { + for (int j = 0; j < GGML_CUDA_MAX_STREAMS; ++j) { + if (streams[i][j] != nullptr) { + CUDA_CHECK(cudaStreamDestroy(streams[i][j])); + } + } + if (cublas_handles[i] != nullptr) { + CUBLAS_CHECK(cublasDestroy(cublas_handles[i])); + } + } +} + + +// cuda buffer + +struct ggml_backend_cuda_buffer_context { + int device; + void * dev_ptr = nullptr; + std::string name; + + ggml_backend_cuda_buffer_context(int device, void * dev_ptr) : + device(device), dev_ptr(dev_ptr), + name(GGML_CUDA_NAME + std::to_string(device)) { + } + + ~ggml_backend_cuda_buffer_context() { + CUDA_CHECK(cudaFree(dev_ptr)); + } +}; + +static void ggml_backend_cuda_buffer_free_buffer(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + delete ctx; +} + +static bool ggml_backend_buffer_is_cuda(ggml_backend_buffer_t buffer) { + return buffer->iface.free_buffer == ggml_backend_cuda_buffer_free_buffer; +} + +static void * ggml_backend_cuda_buffer_get_base(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + return ctx->dev_ptr; +} + +static enum ggml_status ggml_backend_cuda_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + if (tensor->view_src != NULL) { + assert(tensor->view_src->buffer->buft == buffer->buft); + return GGML_STATUS_SUCCESS; + } + + if (ggml_is_quantized(tensor->type) && tensor->view_src == nullptr && ggml_backend_buffer_get_usage(buffer) != GGML_BACKEND_BUFFER_USAGE_COMPUTE) { + // initialize padding to 0 to avoid possible NaN values + const size_t original_size = ggml_nbytes(tensor); + const size_t padded_size = ggml_backend_buft_get_alloc_size(buffer->buft, tensor); + + if (padded_size > original_size) { + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size)); + } + } + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_buffer_memset_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, uint8_t value, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemsetAsync((char *) tensor->data + offset, value, size, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_set_tensor_2d(ggml_backend_buffer_t buffer, struct ggml_tensor * tensor, const void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *) buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpy2DAsync( + (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static void ggml_backend_cuda_buffer_get_tensor_2d(ggml_backend_buffer_t buffer, const struct ggml_tensor * tensor, void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemcpy2DAsync( + data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * src, ggml_tensor * dst) { + if (ggml_backend_buffer_is_cuda(src->buffer)) { + ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context; + ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context; + if (src_ctx->device == dst_ctx->device) { + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread)); + } else { +#ifdef GGML_CUDA_NO_PEER_COPY + return false; +#else + CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_ctx->device, src->data, src_ctx->device, ggml_nbytes(src), cudaStreamPerThread)); +#endif + } + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + return true; + } + return false; + + GGML_UNUSED(buffer); +} + +static void ggml_backend_cuda_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { + ggml_backend_cuda_buffer_context * ctx = (ggml_backend_cuda_buffer_context *)buffer->context; + + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemsetAsync(ctx->dev_ptr, value, buffer->size, cudaStreamPerThread)); + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); +} + +static const ggml_backend_buffer_i ggml_backend_cuda_buffer_interface = { + /* .free_buffer = */ ggml_backend_cuda_buffer_free_buffer, + /* .get_base = */ ggml_backend_cuda_buffer_get_base, + /* .init_tensor = */ ggml_backend_cuda_buffer_init_tensor, + /* .memset_tensor = */ ggml_backend_cuda_buffer_memset_tensor, + /* .set_tensor = */ ggml_backend_cuda_buffer_set_tensor, + /* .get_tensor = */ ggml_backend_cuda_buffer_get_tensor, + /* .set_tensor_2d = */ ggml_backend_cuda_buffer_set_tensor_2d, + /* .get_tensor_2d = */ ggml_backend_cuda_buffer_get_tensor_2d, + /* .cpy_tensor = */ ggml_backend_cuda_buffer_cpy_tensor, + /* .clear = */ ggml_backend_cuda_buffer_clear, + /* .reset = */ NULL, +}; + +// cuda buffer type +struct ggml_backend_cuda_buffer_type_context { + int device; + std::string name; +}; + +static const char * ggml_backend_cuda_buffer_type_get_name(ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_buffer_type_context * ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; + + return ctx->name.c_str(); +} + +static bool ggml_backend_buft_is_cuda(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_buffer_type_get_name; +} + +static ggml_backend_buffer_t ggml_backend_cuda_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)buft->context; + + ggml_cuda_set_device(buft_ctx->device); + + void * dev_ptr; + cudaError_t err = ggml_cuda_device_malloc(&dev_ptr, size, buft_ctx->device); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + GGML_LOG_ERROR("%s: allocating %.2f MiB on device %d: cudaMalloc failed: %s\n", __func__, size / 1024.0 / 1024.0, buft_ctx->device, cudaGetErrorString(err)); + return nullptr; + } + + ggml_backend_cuda_buffer_context * ctx = new ggml_backend_cuda_buffer_context(buft_ctx->device, dev_ptr); + + return ggml_backend_buffer_init(buft, ggml_backend_cuda_buffer_interface, ctx, size); +} + +static size_t ggml_backend_cuda_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { + return 128; + + GGML_UNUSED(buft); +} + +static size_t ggml_backend_cuda_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { + size_t size = ggml_nbytes(tensor); + int64_t ne0 = tensor->ne[0]; + + if (ggml_is_quantized(tensor->type)) { + if (ne0 % MATRIX_ROW_PADDING != 0) { + GGML_ASSERT(tensor->nb[0] == ggml_element_size(tensor)); + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + } + + return size; + + GGML_UNUSED(buft); +} + +static const ggml_backend_buffer_type_i ggml_backend_cuda_buffer_type_interface = { + /* .get_name = */ ggml_backend_cuda_buffer_type_get_name, + /* .alloc_buffer = */ ggml_backend_cuda_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cuda_buffer_type_get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cuda_buffer_type_get_alloc_size, + /* .is_host = */ NULL, +}; + +ggml_backend_buffer_type_t ggml_backend_cuda_buffer_type(int device) { + static std::mutex mutex; + std::lock_guard lock(mutex); + + if (device >= ggml_backend_cuda_get_device_count()) { + return nullptr; + } + + static ggml_backend_buffer_type ggml_backend_cuda_buffer_types[GGML_CUDA_MAX_DEVICES]; + + static bool ggml_backend_cuda_buffer_type_initialized = false; + + if (!ggml_backend_cuda_buffer_type_initialized) { + for (int i = 0; i < ggml_backend_cuda_get_device_count(); i++) { + ggml_backend_cuda_buffer_types[i] = { + /* .iface = */ ggml_backend_cuda_buffer_type_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), i), + /* .context = */ new ggml_backend_cuda_buffer_type_context{i, GGML_CUDA_NAME + std::to_string(i)}, + }; + } + ggml_backend_cuda_buffer_type_initialized = true; + } + + return &ggml_backend_cuda_buffer_types[device]; +} + +// cuda split buffer + +static int64_t get_row_rounding(const std::array & tensor_split) { + int64_t row_rounding = 0; + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { + continue; + } + + const int cc = ggml_cuda_info().devices[id].cc; + row_rounding = std::max(row_rounding, (int64_t)get_mmq_y_host(cc)); + } + return row_rounding; +} + +static void get_row_split(int64_t * row_low, int64_t * row_high, const ggml_tensor * tensor, const std::array & tensor_split, int id) { + const int64_t nrows = ggml_nrows(tensor); + const int64_t rounding = get_row_rounding(tensor_split); + + *row_low = id == 0 ? 0 : nrows*tensor_split[id]; + *row_low -= *row_low % rounding; + + if (id == ggml_backend_cuda_get_device_count() - 1) { + *row_high = nrows; + } else { + *row_high = nrows*tensor_split[id + 1]; + *row_high -= *row_high % rounding; + } +} + +static size_t ggml_nbytes_split(const struct ggml_tensor * tensor, int nrows_split) { + static_assert(GGML_MAX_DIMS == 4, "GGML_MAX_DIMS is not 4 - update this function"); + + return nrows_split*ggml_row_size(tensor->type, tensor->ne[0]); +} + +struct ggml_backend_cuda_split_buffer_type_context { + int main_device; + std::array tensor_split; + std::string name; +}; + +struct ggml_backend_cuda_split_buffer_context { + ~ggml_backend_cuda_split_buffer_context() { + for (ggml_tensor_extra_gpu * extra : tensor_extras) { + for (int id = 0; id < GGML_CUDA_MAX_DEVICES; ++id) { + for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { + if (extra->events[id][is] != nullptr) { + CUDA_CHECK(cudaEventDestroy(extra->events[id][is])); + } + } + if (extra->data_device[id] != nullptr) { + CUDA_CHECK(cudaFree(extra->data_device[id])); + } + } + delete extra; + } + } + + std::vector tensor_extras; +}; + + +static void ggml_backend_cuda_split_buffer_free_buffer(ggml_backend_buffer_t buffer) { + ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; + delete ctx; +} + +static void * ggml_backend_cuda_split_buffer_get_base(ggml_backend_buffer_t buffer) { + // the pointers are stored in the tensor extras, this is just a dummy address and never dereferenced + return (void *)0x1000; + + GGML_UNUSED(buffer); +} + +static enum ggml_status ggml_backend_cuda_split_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) { + GGML_ASSERT(tensor->view_src == nullptr); // views of split tensors are not supported + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_context * ctx = (ggml_backend_cuda_split_buffer_context *)buffer->context; + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + + ggml_tensor_extra_gpu * extra = new ggml_tensor_extra_gpu{}; + ctx->tensor_extras.push_back(extra); + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + // FIXME: do not crash if cudaMalloc fails + // currently, init_tensor cannot fail, it needs to be fixed in ggml-backend first + ggml_cuda_set_device(id); + char * buf; + CUDA_CHECK(ggml_cuda_device_malloc((void**)&buf, size, id)); + + // set padding to 0 to avoid possible NaN values + if (size > original_size) { + CUDA_CHECK(cudaMemset(buf + original_size, 0, size - original_size)); + } + + extra->data_device[id] = buf; + + for (int64_t is = 0; is < GGML_CUDA_MAX_STREAMS; ++is) { + CUDA_CHECK(cudaEventCreateWithFlags(&extra->events[id][is], cudaEventDisableTiming)); + } + } + tensor->extra = extra; + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_split_buffer_set_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + // split tensors must always be set in their entirety at once + GGML_ASSERT(offset == 0); + GGML_ASSERT(size == ggml_nbytes(tensor)); + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + const size_t nb1 = tensor->nb[1]; + ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + const size_t offset_split = row_low*nb1; + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + const char * buf_host = (const char *)data + offset_split; + CUDA_CHECK(cudaMemcpyAsync(extra->data_device[id], buf_host, original_size, cudaMemcpyHostToDevice, cudaStreamPerThread)); + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + } +} + +static void ggml_backend_cuda_split_buffer_get_tensor(ggml_backend_buffer_t buffer, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + // split tensors must always be set in their entirety at once + GGML_ASSERT(offset == 0); + GGML_ASSERT(size == ggml_nbytes(tensor)); + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *)buffer->buft->context; + + const int64_t ne0 = tensor->ne[0]; + const size_t nb1 = tensor->nb[1]; + ggml_tensor_extra_gpu * extra = (ggml_tensor_extra_gpu *)tensor->extra; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, buft_ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + const size_t offset_split = row_low*nb1; + size_t size = ggml_nbytes_split(tensor, nrows_split); + const size_t original_size = size; + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + + char * buf_host = (char *)data + offset_split; + CUDA_CHECK(cudaMemcpyAsync(buf_host, extra->data_device[id], original_size, cudaMemcpyDeviceToHost, cudaStreamPerThread)); + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread)); + } +} + +static void ggml_backend_cuda_split_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) { + GGML_UNUSED(buffer); + GGML_UNUSED(value); +} + +static const ggml_backend_buffer_i ggml_backend_cuda_split_buffer_interface = { + /* .free_buffer = */ ggml_backend_cuda_split_buffer_free_buffer, + /* .get_base = */ ggml_backend_cuda_split_buffer_get_base, + /* .init_tensor = */ ggml_backend_cuda_split_buffer_init_tensor, + /* .memset_tensor = */ NULL, + /* .set_tensor = */ ggml_backend_cuda_split_buffer_set_tensor, + /* .get_tensor = */ ggml_backend_cuda_split_buffer_get_tensor, + /* .set_tensor_2d = */ NULL, + /* .get_tensor_2d = */ NULL, + /* .cpy_tensor = */ NULL, + /* .clear = */ ggml_backend_cuda_split_buffer_clear, + /* .reset = */ NULL, +}; + +// cuda split buffer type + +static const char * ggml_backend_cuda_split_buffer_type_get_name(ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; + + return ctx->name.c_str(); +} + +static bool ggml_backend_buft_is_cuda_split(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_split_buffer_type_get_name; +} + +static ggml_backend_buffer_t ggml_backend_cuda_split_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + // since we don't know the exact split after rounding, we cannot allocate the device buffers at this point + // instead, we allocate them for each tensor separately in init_tensor + // however, the size still represents the maximum cumulative size of all the device buffers after the tensors are allocated, + // as returned by get_alloc_size. this limit is enforced during tensor allocation by ggml-alloc, so it must be correct. + ggml_backend_cuda_split_buffer_context * ctx = new ggml_backend_cuda_split_buffer_context(); + + return ggml_backend_buffer_init(buft, ggml_backend_cuda_split_buffer_interface, ctx, size); +} + +static size_t ggml_backend_cuda_split_buffer_type_get_alignment(ggml_backend_buffer_type_t buft) { + return 128; + + GGML_UNUSED(buft); +} + +static size_t ggml_backend_cuda_split_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const ggml_tensor * tensor) { + ggml_backend_cuda_split_buffer_type_context * ctx = (ggml_backend_cuda_split_buffer_type_context *)buft->context; + GGML_ASSERT(ggml_is_contiguous(tensor) && "split buffers only supported for contiguous tensors"); + + size_t total_size = 0; + + const int64_t ne0 = tensor->ne[0]; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + int64_t row_low, row_high; + get_row_split(&row_low, &row_high, tensor, ctx->tensor_split, id); + + int64_t nrows_split = row_high - row_low; + if (nrows_split == 0) { + continue; + } + + total_size += ggml_nbytes_split(tensor, nrows_split); + + // pad last row to a multiple of 512 elements to avoid out-of-bounds memory accesses + if (ne0 % MATRIX_ROW_PADDING != 0) { + total_size += ggml_row_size(tensor->type, MATRIX_ROW_PADDING - ne0 % MATRIX_ROW_PADDING); + } + } + + return total_size; +} + +static bool ggml_backend_cuda_split_buffer_type_is_host(ggml_backend_buffer_type_t buft) { + return false; + + GGML_UNUSED(buft); +} + +static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_interface = { + /* .get_name = */ ggml_backend_cuda_split_buffer_type_get_name, + /* .alloc_buffer = */ ggml_backend_cuda_split_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cuda_split_buffer_type_get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cuda_split_buffer_type_get_alloc_size, + /* .is_host = */ ggml_backend_cuda_split_buffer_type_is_host, +}; + +// Communication context for multi-GPU AllReduce during tensor parallelism. +// Created once per meta backend instance; provider is fixed at init time. +struct ggml_backend_cuda_comm_context { + ggml_cuda_allreduce_provider provider; + std::vector backends; + +#ifdef GGML_USE_NCCL + std::vector comms; // valid when provider == GGML_CUDA_ALLREDUCE_NCCL +#endif + + ggml_cuda_ar_pipeline * ar_pipeline = nullptr; // valid when provider == GGML_CUDA_ALLREDUCE_INTERNAL + + ~ggml_backend_cuda_comm_context() { +#ifdef GGML_USE_NCCL + if (provider == GGML_CUDA_ALLREDUCE_NCCL) { + for (ncclComm_t comm : comms) { + NCCL_CHECK(ncclCommDestroy(comm)); + } + } +#endif + ggml_cuda_ar_pipeline_free(ar_pipeline); + } +}; + +// Select an AllReduce provider for the given set of CUDA device IDs. +// +// Priority: +// 1. GGML_CUDA_ALLREDUCE env var ("nccl" or "internal") — explicit override. +// 2. NCCL when compiled in (GGML_USE_NCCL defined). +// 3. Internal otherwise. +// +// Future: inspect NVLink topology via cudaDeviceGetP2PAttribute() with +// cudaDevP2PAttrNativeAtomicSupported to prefer INTERNAL on PCIe-only systems +// where host-staged reduction can beat NCCL for small tensors. +static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( + const std::vector & device_ids) { + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + if (env != nullptr && env[0] != '\0') { + if (strcmp(env, "internal") == 0) { + return GGML_CUDA_ALLREDUCE_INTERNAL; + } + if (strcmp(env, "nccl") == 0) { +#ifdef GGML_USE_NCCL + return GGML_CUDA_ALLREDUCE_NCCL; +#else + GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in, using internal provider\n", __func__); + return GGML_CUDA_ALLREDUCE_INTERNAL; +#endif + } + GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); + } + +#ifdef GGML_USE_NCCL + GGML_UNUSED(device_ids); + return GGML_CUDA_ALLREDUCE_NCCL; +#else + GGML_UNUSED(device_ids); + return GGML_CUDA_ALLREDUCE_INTERNAL; +#endif +} + +static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { + if (comm_ctx_v == nullptr) { + return; + } + delete static_cast(comm_ctx_v); +} + +static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { + for (size_t i = 0; i < n_backends; i++) { + if (!ggml_backend_is_cuda(backends[i])) { + return nullptr; + } + } + + std::vector dev_ids; + dev_ids.reserve(n_backends); + for (size_t i = 0; i < n_backends; i++) { + dev_ids.push_back(static_cast(backends[i]->context)->device); + } + + const ggml_cuda_allreduce_provider provider = ggml_cuda_select_allreduce_provider(dev_ids); + + auto * ret = new ggml_backend_cuda_comm_context; + ret->provider = provider; + ret->backends.assign(backends, backends + n_backends); + + switch (provider) { + case GGML_CUDA_ALLREDUCE_NCCL: { +#ifdef GGML_USE_NCCL + ret->comms.resize(n_backends); + NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); +#else + // Unreachable: ggml_cuda_select_allreduce_provider() only returns + // GGML_CUDA_ALLREDUCE_NCCL when GGML_USE_NCCL is defined. + GGML_ABORT("NCCL provider selected but NCCL not compiled in"); +#endif + } break; + + case GGML_CUDA_ALLREDUCE_INTERNAL: { + ret->ar_pipeline = ggml_cuda_ar_pipeline_init( + dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); + if (ret->ar_pipeline == nullptr) { + GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); + delete ret; + return nullptr; + } + } break; + } + + return ret; +} + +#ifdef GGML_USE_NCCL +// AllReduce via NCCL. Reduces as FP32 for small tensors and BF16 for large +// tensors (bandwidth-bound), then converts back to FP32. +static bool ggml_backend_cuda_comm_allreduce_nccl( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + const int64_t ne = ggml_nelements(tensors[0]); + // FIXME the input of llm_graph_context::build_in_out_ids can produce a tensor with 0 elements if n_outputs == 0 + // This then causes a crash in this function + if (ne == 0) { + return true; + } + + const size_t n_backends = comm_ctx->backends.size(); + + for (size_t i = 0; i < n_backends; ++i) { + GGML_ASSERT(tensors[i] != nullptr); + GGML_ASSERT(ggml_nelements(tensors[i]) == ne); + GGML_ASSERT(ggml_is_contiguously_allocated(tensors[i])); + } + + // For small tensors, simply reduce them as FP32. + // The following heuristic for how "small" a tensor should be is based on RTX 4090s connected via 16x PCIe 4.0. + if ((n_backends <= 2 && ne < 32768) || (n_backends == 3 && ne < 131072) || (n_backends >= 4 && ne < 262144)) { + for (size_t i = 0; i < n_backends; ++i) { + if ((tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + ggml_cuda_set_device(cuda_ctx->device); + CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, ggml_nbytes(tensors[i]), cuda_ctx->stream())); + } + } + NCCL_CHECK(ncclGroupStart()); + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + NCCL_CHECK(ncclAllReduce(tensors[i]->data, tensors[i]->data, ne, ncclFloat, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); + } + NCCL_CHECK(ncclGroupEnd()); + return true; + } + + // For large tensors it's faster to compress them to BF16 for the reduction: + to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); + to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); + + ggml_cuda_pool_alloc tmp[GGML_CUDA_MAX_DEVICES]; + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + tmp[i].pool = &cuda_ctx->pool(); + tmp[i].alloc(ne); + + ggml_cuda_set_device(cuda_ctx->device); + if (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) { + to_bf16(tensors[i]->data, tmp[i].get(), ne, cuda_ctx->stream()); + } else { + CUDA_CHECK(cudaMemsetAsync(tmp[i].get(), 0, ne * sizeof(nv_bfloat16), cuda_ctx->stream())); + } + CUDA_CHECK(cudaGetLastError()); + } + + NCCL_CHECK(ncclGroupStart()); + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + NCCL_CHECK(ncclAllReduce(tmp[i].get(), tmp[i].get(), ne, ncclBfloat16, ncclSum, comm_ctx->comms[i], cuda_ctx->stream())); + } + NCCL_CHECK(ncclGroupEnd()); + + for (size_t i = 0; i < n_backends; ++i) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) comm_ctx->backends[i]->context; + + ggml_cuda_set_device(cuda_ctx->device); + to_fp32(tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); + CUDA_CHECK(cudaGetLastError()); + } + + return true; +} +#endif // GGML_USE_NCCL + +static bool ggml_backend_cuda_comm_allreduce_internal( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); +} + +static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { + if (comm_ctx_v == nullptr) { + return false; + } + auto * comm_ctx = static_cast(comm_ctx_v); + switch (comm_ctx->provider) { +#ifdef GGML_USE_NCCL + case GGML_CUDA_ALLREDUCE_NCCL: + return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); +#endif + case GGML_CUDA_ALLREDUCE_INTERNAL: + return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors); + default: + return false; + } +} + +ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { + static std::mutex mutex; + std::lock_guard lock(mutex); + + static std::map>, struct ggml_backend_buffer_type> buft_map; + + std::array tensor_split_arr = {}; + + bool all_zero = tensor_split == nullptr || std::all_of(tensor_split, tensor_split + GGML_CUDA_MAX_DEVICES, [](float x) { return x == 0.0f; }); + if (all_zero) { + tensor_split_arr = ggml_cuda_info().default_tensor_split; + } else { + float split_sum = 0.0f; + for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { + tensor_split_arr[i] = split_sum; + split_sum += tensor_split[i]; + } + for (int i = 0; i < ggml_backend_cuda_get_device_count(); ++i) { + tensor_split_arr[i] /= split_sum; + } + } + + auto it = buft_map.find({main_device, tensor_split_arr}); + if (it != buft_map.end()) { + return &it->second; + } + auto * ctx = new ggml_backend_cuda_split_buffer_type_context{ + main_device, + tensor_split_arr, + GGML_CUDA_NAME + std::to_string(main_device) + "_Split", + }; + + struct ggml_backend_buffer_type buft { + /* .iface = */ ggml_backend_cuda_split_buffer_type_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), main_device), + /* .context = */ ctx, + }; + + auto result = buft_map.emplace(std::make_pair(main_device, tensor_split_arr), buft); + return &result.first->second; +} + +// host buffer type + +static const char * ggml_backend_cuda_host_buffer_type_name(ggml_backend_buffer_type_t buft) { + return GGML_CUDA_NAME "_Host"; + + GGML_UNUSED(buft); +} + +static bool ggml_backend_buft_is_cuda_host(ggml_backend_buffer_type_t buft) { + return buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; +} + +static void ggml_backend_cuda_host_buffer_free_buffer(ggml_backend_buffer_t buffer) { + CUDA_CHECK(cudaFreeHost(buffer->context)); +} + +static void * ggml_cuda_host_malloc(size_t size) { + if (getenv("GGML_CUDA_NO_PINNED") != nullptr) { + return nullptr; + } + + void * ptr = nullptr; + cudaError_t err = cudaMallocHost((void **) &ptr, size); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + GGML_LOG_DEBUG("%s: failed to allocate %.2f MiB of pinned memory: %s\n", __func__, + size / 1024.0 / 1024.0, cudaGetErrorString(err)); + return nullptr; + } + + return ptr; +} + +static ggml_backend_buffer_t ggml_backend_cuda_host_buffer_type_alloc_buffer(ggml_backend_buffer_type_t buft, size_t size) { + void * ptr = ggml_cuda_host_malloc(size); + + if (ptr == nullptr) { + // fallback to cpu buffer + return ggml_backend_buft_alloc_buffer(ggml_backend_cpu_buffer_type(), size); + } + + ggml_backend_buffer_t buffer = ggml_backend_cpu_buffer_from_ptr(ptr, size); + buffer->buft = buft; + buffer->iface.free_buffer = ggml_backend_cuda_host_buffer_free_buffer; + + return buffer; +} + +ggml_backend_buffer_type_t ggml_backend_cuda_host_buffer_type() { + static struct ggml_backend_buffer_type ggml_backend_cuda_buffer_type_host = { + /* .iface = */ { + /* .get_name = */ ggml_backend_cuda_host_buffer_type_name, + /* .alloc_buffer = */ ggml_backend_cuda_host_buffer_type_alloc_buffer, + /* .get_alignment = */ ggml_backend_cpu_buffer_type()->iface.get_alignment, + /* .get_max_size = */ NULL, // defaults to SIZE_MAX + /* .get_alloc_size = */ ggml_backend_cpu_buffer_type()->iface.get_alloc_size, + /* .is_host = */ ggml_backend_cpu_buffer_type()->iface.is_host, + }, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), 0), + /* .context = */ nullptr, + }; + + return &ggml_backend_cuda_buffer_type_host; +} + +//static bool ggml_backend_buffer_is_cuda_host(ggml_backend_buffer_t buffer) { +// return buffer->buft->iface.get_name == ggml_backend_cuda_host_buffer_type_name; +//} + +/// kernels + +typedef void (*ggml_cuda_op_mul_mat_t)( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, + const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, + const int64_t src1_padded_row_size, cudaStream_t stream); + +#ifndef GGML_CUDA_PEER_MAX_BATCH_SIZE +#define GGML_CUDA_PEER_MAX_BATCH_SIZE 128 +#endif // GGML_CUDA_PEER_MAX_BATCH_SIZE + +#define MUL_MAT_SRC1_COL_STRIDE 128 + +static cudaError_t ggml_cuda_cpy_tensor_2d( + void * dst, const struct ggml_tensor * src, int64_t i3, int64_t i2, int64_t i1_low, int64_t i1_high, cudaStream_t stream) { + + const char * src_ptr = (const char *) src->data; + char * dst_ptr = (char *) dst; + + const int64_t ne0 = src->ne[0]; + const int64_t nb0 = src->nb[0]; + const int64_t nb1 = src->nb[1]; + const int64_t nb2 = src->nb[2]; + const int64_t nb3 = src->nb[3]; + const enum ggml_type type = src->type; + const int64_t ts = ggml_type_size(type); + const int64_t bs = ggml_blck_size(type); + const int64_t i1_diff = i1_high - i1_low; + + const char * x = src_ptr + i1_low*nb1 + i2*nb2 + i3*nb3; + if (nb0 == ts && nb1 == ts*ne0/bs) { + return cudaMemcpyAsync(dst_ptr, x, i1_diff*nb1, cudaMemcpyDeviceToDevice, stream); + } else if (nb0 == ts) { + return cudaMemcpy2DAsync(dst_ptr, ts*ne0/bs, x, nb1, ts*ne0/bs, i1_diff, cudaMemcpyDeviceToDevice, stream); + } else { + for (int64_t i1 = 0; i1 < i1_diff; i1++) { + const void * rx = (const void *) ((const char *) x + i1*nb1); + void * rd = (void *) (dst_ptr + i1*ts*ne0/bs); + // pretend the row is a matrix with cols=1 + cudaError_t r = cudaMemcpy2DAsync(rd, ts/bs, rx, nb0, ts/bs, ne0, cudaMemcpyDeviceToDevice, stream); + if (r != cudaSuccess) { + return r; + } + } + return cudaSuccess; + } +} + +struct cublas_force_compute_type { + bool fp32 = false; + bool fp16 = false; +}; + +static const cublas_force_compute_type & ggml_cuda_cublas_get_force_compute_type() { + static const cublas_force_compute_type compute_type = [] { + cublas_force_compute_type result; + + const bool ggml_cuda_force_cublas_compute_32f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F") != nullptr; + const bool ggml_cuda_force_cublas_compute_16f_env = getenv("GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F") != nullptr; + + GGML_ASSERT(ggml_cuda_force_cublas_compute_16f_env == false || ggml_cuda_force_cublas_compute_32f_env == false); + + if (ggml_cuda_force_cublas_compute_32f_env) { + GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F\n"); + result.fp32 = true; + } else if (ggml_cuda_force_cublas_compute_16f_env) { + GGML_LOG_INFO("Detected GGML_CUDA_FORCE_CUBLAS_COMPUTE_16F\n"); + result.fp16 = true; + } + + return result; + }(); + + return compute_type; +} + +static void ggml_cuda_op_mul_mat_cublas( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i, + const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols, + const int64_t src1_padded_row_size, cudaStream_t stream) { + + GGML_ASSERT(src0_dd_i != nullptr); + GGML_ASSERT(src1_ddf_i != nullptr); + GGML_ASSERT(dst_dd_i != nullptr); + + const int64_t ne00 = src0->ne[0]; + const int64_t ne10 = src1->ne[0]; + + const int64_t ne0 = dst->ne[0]; + + const int64_t row_diff = row_high - row_low; + + int id = ggml_cuda_get_device(); + + // the main device has a larger memory buffer to hold the results from all GPUs + // ldc == nrows of the matrix that cuBLAS writes into + int64_t ldc = id == ctx.device ? ne0 : row_diff; + + const int cc = ggml_cuda_info().devices[id].cc; + + const bool supports_bf16 = GGML_CUDA_CC_IS_NVIDIA(cc) || GGML_CUDA_CC_IS_AMD(cc) || + (GGML_CUDA_CC_IS_MTHREADS(cc) && cc >= GGML_CUDA_CC_QY2); + + const bool use_fp16 = + src0->type != GGML_TYPE_NVFP4 && + (src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && + ggml_is_contiguous(src0) && + row_diff == src0->ne[1] && + dst->op_params[0] == GGML_PREC_DEFAULT; + + if (supports_bf16 && src0->type == GGML_TYPE_BF16 && ggml_is_contiguous(src0) && row_diff == src0->ne[1]) { + ggml_cuda_pool_alloc src1_as_bf16(ctx.pool(id)); + if (src1->type != GGML_TYPE_BF16) { + const to_bf16_cuda_t to_bf16_cuda = ggml_get_to_bf16_cuda(src1->type); + GGML_ASSERT(to_bf16_cuda != nullptr); + size_t ne = src1_ncols*ne10; + src1_as_bf16.alloc(ne); + to_bf16_cuda(src1_ddf_i, src1_as_bf16.get(), ne, stream); + } + const nv_bfloat16 * src1_ptr = src1->type == GGML_TYPE_BF16 ? (const nv_bfloat16 *) src1_ddf_i : src1_as_bf16.get(); + const nv_bfloat16 * src0_ptr = (const nv_bfloat16 *)src0_dd_i; + ggml_cuda_pool_alloc dst_bf16(ctx.pool(id), row_diff*src1_ncols); + + const float alpha_f32 = 1.0f; + const float beta_f32 = 0.0f; + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha_f32, src0_ptr, CUDA_R_16BF, ne00, + src1_ptr, CUDA_R_16BF, ne10, + &beta_f32, dst_bf16.get(), CUDA_R_16BF, ldc, + CUBLAS_COMPUTE_32F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); + to_fp32_cuda(dst_bf16.get(), dst_dd_i, row_diff*src1_ncols, stream); + } else if (fast_fp16_hardware_available(cc) && use_fp16) { + // convert src0 and src1 to fp16, multiply as fp16, convert dst to fp32 + ggml_cuda_pool_alloc src0_as_f16(ctx.pool(id)); + if (src0->type != GGML_TYPE_F16) { + const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src0->type); + GGML_ASSERT(to_fp16_cuda != nullptr); + size_t ne = row_diff*ne00; + src0_as_f16.alloc(ne); + to_fp16_cuda(src0_dd_i, src0_as_f16.get(), ne, stream); + } + const half * src0_ptr = src0->type == GGML_TYPE_F16 ? (const half *) src0_dd_i : src0_as_f16.get(); + + ggml_cuda_pool_alloc src1_as_f16(ctx.pool(id)); + if (src1->type != GGML_TYPE_F16) { + const to_fp16_cuda_t to_fp16_cuda = ggml_get_to_fp16_cuda(src1->type); + GGML_ASSERT(to_fp16_cuda != nullptr); + size_t ne = src1_ncols*ne10; + src1_as_f16.alloc(ne); + to_fp16_cuda(src1_ddf_i, src1_as_f16.get(), ne, stream); + } + const half * src1_ptr = src1->type == GGML_TYPE_F16 ? (const half *) src1_ddf_i : src1_as_f16.get(); + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + + const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); + + if (!force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) + || GGML_CUDA_CC_IS_RDNA4(cc) + || cc == GGML_CUDA_CC_VOLTA + || force_compute_type.fp32)) + { + const float alpha = 1.0f; + const float beta = 0.0f; + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha, src0_ptr, CUDA_R_16F, ne00, + src1_ptr, CUDA_R_16F, ne10, + &beta, dst_dd_i, CUDA_R_32F, ldc, + CUBLAS_COMPUTE_32F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } else { + ggml_cuda_pool_alloc dst_f16(ctx.pool(id), row_diff*src1_ncols); + + const half alpha_f16 = 1.0f; + const half beta_f16 = 0.0f; + + CUBLAS_CHECK( + cublasGemmEx(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha_f16, src0_ptr, CUDA_R_16F, ne00, + src1_ptr, CUDA_R_16F, ne10, + &beta_f16, dst_f16.get(), CUDA_R_16F, ldc, + CUBLAS_COMPUTE_16F, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(GGML_TYPE_F16); + to_fp32_cuda(dst_f16.get(), dst_dd_i, row_diff*src1_ncols, stream); + } + } else { + ggml_cuda_pool_alloc src0_ddq_as_f32(ctx.pool(id)); + ggml_cuda_pool_alloc src1_ddq_as_f32(ctx.pool(id)); + + if (src0->type != GGML_TYPE_F32) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src0->type); + GGML_ASSERT(to_fp32_cuda != nullptr); + src0_ddq_as_f32.alloc(row_diff*ne00); + to_fp32_cuda(src0_dd_i, src0_ddq_as_f32.get(), row_diff*ne00, stream); + } + if (src1->type != GGML_TYPE_F32) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(src1->type); + GGML_ASSERT(to_fp32_cuda != nullptr); + src1_ddq_as_f32.alloc(src1_ncols*ne10); + to_fp32_cuda(src1_ddf_i, src1_ddq_as_f32.get(), src1_ncols*ne10, stream); + } + + const float * src0_ddf_i = src0->type == GGML_TYPE_F32 ? (const float *) src0_dd_i : src0_ddq_as_f32.get(); + const float * src1_ddf1_i = src1->type == GGML_TYPE_F32 ? (const float *) src1_ddf_i : src1_ddq_as_f32.get(); + + const float alpha = 1.0f; + const float beta = 0.0f; + + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(id), stream)); + CUBLAS_CHECK( + cublasSgemm(ctx.cublas_handle(id), CUBLAS_OP_T, CUBLAS_OP_N, + row_diff, src1_ncols, ne10, + &alpha, src0_ddf_i, ne00, + src1_ddf1_i, ne10, + &beta, dst_dd_i, ldc)); + } + + GGML_UNUSED_VARS(dst, src1_ddq_i, src1_padded_row_size); +} + +static cudaError_t ggml_cuda_Memcpy2DPeerAsync( + void * dst, int dstDevice, size_t dpitch, void * src, int srcDevice, size_t spitch, size_t width, size_t height, cudaStream_t stream) { + +#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) + // cudaMemcpy2DAsync may fail with copies between vmm pools of different devices + cudaMemcpy3DPeerParms p = {}; + p.dstDevice = dstDevice; + p.dstPtr = make_cudaPitchedPtr(dst, dpitch, dpitch, height); + p.srcDevice = srcDevice; + p.srcPtr = make_cudaPitchedPtr(src, spitch, spitch, height); + p.extent = make_cudaExtent(width, height, 1); + return cudaMemcpy3DPeerAsync(&p, stream); +#else + // HIP does not support cudaMemcpy3DPeerAsync or vmm pools + GGML_UNUSED(dstDevice); + GGML_UNUSED(srcDevice); + return cudaMemcpy2DAsync(dst, dpitch, src, spitch, width, height, cudaMemcpyDeviceToDevice, stream); +#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) +} + +static void ggml_cuda_op_mul_mat( + ggml_backend_cuda_context & ctx, + const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, ggml_cuda_op_mul_mat_t op, + quantize_cuda_t quantize_src1) { + + const int64_t ne00 = src0->ne[0]; + const int64_t ne01 = src0->ne[1]; + const int64_t ne02 = src0->ne[2]; + const int64_t ne03 = src0->ne[3]; + + const int64_t ne10 = src1->ne[0]; + const int64_t ne11 = src1->ne[1]; + const int64_t ne12 = src1->ne[2]; + const int64_t ne13 = src1->ne[3]; + const int64_t nrows1 = ggml_nrows(src1); + + const int64_t ne0 = dst->ne[0]; + const int64_t ne1 = dst->ne[1]; + + // const int64_t nb10 = src1->nb[0]; + const int64_t nb11 = src1->nb[1]; + const int64_t nb12 = src1->nb[2]; + const int64_t nb13 = src1->nb[3]; + + const int64_t nb2 = dst->nb[2]; + const int64_t nb3 = dst->nb[3]; + + ggml_backend_cuda_buffer_context * src1_ctx = (ggml_backend_cuda_buffer_context *) src1->buffer->context; + ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *) dst->buffer->context; + + GGML_ASSERT(src1->type == GGML_TYPE_F32 || (src1->ne[2] == 1 && src1->ne[3] == 1)); + + GGML_ASSERT(ne12 % ne02 == 0); + GGML_ASSERT(ne13 % ne03 == 0); + + const int64_t i02_divisor = ne12 / ne02; + const int64_t i03_divisor = ne13 / ne03; + + const size_t src0_ts = ggml_type_size(src0->type); + const size_t src0_bs = ggml_blck_size(src0->type); + const size_t q8_1_ts = sizeof(block_q8_1); + const size_t q8_1_bs = QK8_1; + + const bool src0_is_contiguous = ggml_is_contiguous(src0); + const bool src1_is_contiguous = ggml_is_contiguous(src1); + + const int64_t src1_padded_col_size = GGML_PAD(ne10, MATRIX_ROW_PADDING); + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); + GGML_ASSERT(!(split && ne02 > 1)); + GGML_ASSERT(!(split && ne03 > 1)); + GGML_ASSERT(!(split && ne02 < ne12)); + GGML_ASSERT(!(split && ne03 < ne13)); + + ggml_tensor_extra_gpu * src0_extra = split ? (ggml_tensor_extra_gpu *) src0->extra : nullptr; + + + std::array tensor_split; + if (split) { + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; + tensor_split = buft_ctx->tensor_split; + } + + struct dev_data { + int cc; + + ggml_cuda_pool_alloc src0_dd_alloc; + ggml_cuda_pool_alloc src1_ddf_alloc; + ggml_cuda_pool_alloc src1_ddq_alloc; + ggml_cuda_pool_alloc dst_dd_alloc; + + char * src0_dd = nullptr; + float * src1_ddf = nullptr; // float + char * src1_ddq = nullptr; // q8_1 + float * dst_dd = nullptr; + + int64_t row_low; + int64_t row_high; + }; + + dev_data dev[GGML_CUDA_MAX_DEVICES]; + + int used_devices = 0; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + dev[id].cc = ggml_cuda_info().devices[id].cc; + + // by default, use all rows + dev[id].row_low = 0; + dev[id].row_high = ne01; + + // for multi GPU, get the row boundaries from tensor split + // and round to mul_mat_q tile sizes + if (split) { + const int64_t rounding = get_row_rounding(tensor_split); + + if (id != 0) { + dev[id].row_low = ne01*tensor_split[id]; + if (dev[id].row_low < ne01) { + dev[id].row_low -= dev[id].row_low % rounding; + } + } + + if (id != ggml_backend_cuda_get_device_count() - 1) { + dev[id].row_high = ne01*tensor_split[id + 1]; + if (dev[id].row_high < ne01) { + dev[id].row_high -= dev[id].row_high % rounding; + } + } + } + } + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { + continue; + } + + used_devices++; + + const bool src1_on_device = id == src1_ctx->device; + const bool dst_on_device = id == dst_ctx->device; + + ggml_cuda_set_device(id); + cudaStream_t stream = ctx.stream(id, 0); + + if (src0_is_contiguous) { + dev[id].src0_dd = split ? (char *) src0_extra->data_device[id] : (char *) src0->data; + } else { + // If src0 is not contiguous it will be copied to a temporary buffer. + // This buffer needs to be cleared entirely because multiple regions will function as padding. + const size_t nbytes_data = ggml_nbytes(src0); + const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); + dev[id].src0_dd = dev[id].src0_dd_alloc.alloc(ctx.pool(id), nbytes_data + nbytes_padding); + CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd, 0, nbytes_data + nbytes_padding, stream)); + } + + // If src0 is on a temporary compute buffer (partial offloading) there may be some padding that needs to be cleared: + if (ne00 % MATRIX_ROW_PADDING != 0 && ggml_is_quantized(src0->type) && ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && src0->view_src == nullptr) { + GGML_ASSERT(ggml_is_contiguously_allocated(src0)); + GGML_ASSERT(!src0->view_src); + const size_t nbytes_data = ggml_row_size(src0->type, (dev[id].row_high - dev[id].row_low)*ne00); + const size_t nbytes_padding = ggml_row_size(src0->type, MATRIX_ROW_PADDING - ne00 % MATRIX_ROW_PADDING); + CUDA_CHECK(cudaMemsetAsync(dev[id].src0_dd + nbytes_data, 0, nbytes_padding, stream)); + } + + if (src1_on_device && src1_is_contiguous) { + dev[id].src1_ddf = (float *) src1->data; + } else { + dev[id].src1_ddf = dev[id].src1_ddf_alloc.alloc(ctx.pool(id), ggml_nelements(src1)); + } + + if (quantize_src1) { + size_t src_1_ddq_size = nrows1*src1_padded_col_size*q8_1_ts/q8_1_bs; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + src_1_ddq_size += get_mmq_x_max_host(dev[id].cc)*sizeof(block_q8_1_mmq); + } + dev[id].src1_ddq = dev[id].src1_ddq_alloc.alloc(ctx.pool(id), src_1_ddq_size); + + if (src1_on_device && src1_is_contiguous) { + quantize_src1( + dev[id].src1_ddf, nullptr, dev[id].src1_ddq, src0->type, ne10, + nb11/sizeof(float), nb12/sizeof(float), nb13/sizeof(float), + src1_padded_col_size, ne11, ne12, ne13, stream); + CUDA_CHECK(cudaGetLastError()); + } + } + + if (dst_on_device) { + dev[id].dst_dd = (float *) dst->data; + } else { + const size_t size_dst_ddf = split ? (dev[id].row_high - dev[id].row_low)*ne1 : ggml_nelements(dst); + dev[id].dst_dd = dev[id].dst_dd_alloc.alloc(ctx.pool(id), size_dst_ddf); + } + } + + // if multiple devices are used they need to wait for the main device + // here an event is recorded that signals that the main device has finished calculating the input data + if (split && used_devices > 1) { + ggml_cuda_set_device(ctx.device); + CUDA_CHECK(cudaEventRecord(src0_extra->events[ctx.device][0], ctx.stream())); + } + + const int64_t src1_col_stride = split && used_devices > 1 ? MUL_MAT_SRC1_COL_STRIDE : ne11; + for (int64_t src1_col_0 = 0; src1_col_0 < ne11; src1_col_0 += src1_col_stride) { + const int64_t is = split ? (src1_col_0/src1_col_stride) % GGML_CUDA_MAX_STREAMS : 0; + const int64_t src1_ncols = src1_col_0 + src1_col_stride > ne11 ? ne11 - src1_col_0 : src1_col_stride; + + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if ((!split && id != ctx.device) || dev[id].row_low == dev[id].row_high) { + continue; + } + + const bool src1_on_device = id == src1_ctx->device; + const bool dst_on_device = id == dst_ctx->device; + const int64_t row_diff = dev[id].row_high - dev[id].row_low; + + ggml_cuda_set_device(id); + cudaStream_t stream = ctx.stream(id, is); + + // wait for main GPU data if necessary + if (split && (id != ctx.device || is != 0)) { + CUDA_CHECK(cudaStreamWaitEvent(stream, src0_extra->events[ctx.device][0], 0)); + } + + for (int64_t i0 = 0; i0 < ne13*ne12; ++i0) { + const int64_t i03 = i0 / ne12; + const int64_t i02 = i0 % ne12; + + size_t src1_ddq_i_offset = i0*ne11 * src1_padded_col_size*q8_1_ts/q8_1_bs; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + src1_ddq_i_offset += src1_col_0 * sizeof(block_q8_1_mmq); + } else { + src1_ddq_i_offset += src1_col_0 * src1_padded_col_size*q8_1_ts/q8_1_bs; + } + + // for split tensors the data begins at i0 == i0_offset_low + const size_t nbytes_src0_matrix = ne01*ne00*src0_ts / src0_bs; + char * src0_dd_i = dev[id].src0_dd + ((i03/i03_divisor)*ne02 + (i02/i02_divisor)) * nbytes_src0_matrix; + float * src1_ddf_i = dev[id].src1_ddf + (i0*ne11 + src1_col_0) * ne10; + char * src1_ddq_i = dev[id].src1_ddq + src1_ddq_i_offset; + float * dst_dd_i = dev[id].dst_dd + (i0*ne1 + src1_col_0) * (dst_on_device ? ne0 : row_diff); + + // the main device memory buffer can be on VRAM scratch, with space for all partial results + // in that case an offset on dst_ddf_i is needed + if (id == ctx.device) { + dst_dd_i += dev[id].row_low; // offset is 0 if no tensor split + } + + // copy src0, src1 to device if necessary + if (src1_is_contiguous) { + if (id != ctx.device) { + if (quantize_src1) { + char * src1_ddq_i_source = dev[ctx.device].src1_ddq + src1_ddq_i_offset; + if (quantize_src1 == quantize_mmq_q8_1_cuda) { + const size_t pitch = ne11*sizeof(block_q8_1_mmq); + const size_t width = src1_ncols*sizeof(block_q8_1_mmq); + const size_t height = src1_padded_col_size/(4*QK8_1); + CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync(src1_ddq_i, id, pitch, src1_ddq_i_source, ctx.device, pitch, width, height, stream)); + } else { + CUDA_CHECK(cudaMemcpyPeerAsync( + src1_ddq_i, id, src1_ddq_i_source, ctx.device, src1_ncols*src1_padded_col_size*q8_1_ts/q8_1_bs, stream)); + } + } else { + float * src1_ddf_i_source = (float *) src1->data; + src1_ddf_i_source += (i0*ne11 + src1_col_0) * ne10; + CUDA_CHECK(cudaMemcpyPeerAsync(src1_ddf_i, id, src1_ddf_i_source, ctx.device, + src1_ncols*ne10*sizeof(float), stream)); + } + } + } else if (src1_on_device && !src1_is_contiguous) { + CUDA_CHECK(ggml_cuda_cpy_tensor_2d( + src1_ddf_i, src1, i03, i02, src1_col_0, src1_col_0+src1_ncols, stream)); + } else { + GGML_ABORT("fatal error"); + } + + if (quantize_src1 && !src1_is_contiguous) { + quantize_src1( + src1_ddf_i, nullptr, src1_ddq_i, src0->type, ne10, ne10, ne11*ne10, ne12*ne11*ne10, + src1_padded_col_size, src1_ncols, 1, 1, stream); + CUDA_CHECK(cudaGetLastError()); + } + + if (src1_col_0 == 0 && !src0_is_contiguous && i03 % i03_divisor == 0 && i02 % i02_divisor == 0) { + CUDA_CHECK(ggml_cuda_cpy_tensor_2d( + src0_dd_i, src0, i03/i03_divisor, i02/i02_divisor, dev[id].row_low, dev[id].row_high, stream)); + } + + // do the computation + op(ctx, src0, src1, dst, src0_dd_i, src1_ddf_i, src1_ddq_i, dst_dd_i, + dev[id].row_low, dev[id].row_high, src1_ncols, src1_padded_col_size, stream); + CUDA_CHECK(cudaGetLastError()); + + // copy dst to host or other device if necessary + if (!dst_on_device) { + void * dst_off_device = dst->data; + if (split) { + // src0 = weight matrix is saved as a transposed matrix for better memory layout. + // dst is NOT transposed. + // The outputs of matrix matrix multiplications can therefore NOT simply be concatenated for >1 GPU. + // Instead they need to be copied to the correct slice in ne0 = dst row index. + // If dst is a vector with ne0 == 1 then you don't have to do this but it still produces correct results. + float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); + GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); + dhf_dst_i += src1_col_0*ne0 + dev[id].row_low; + CUDA_CHECK(ggml_cuda_Memcpy2DPeerAsync( + dhf_dst_i, ctx.device, ne0*sizeof(float), dst_dd_i, id, row_diff*sizeof(float), row_diff*sizeof(float), src1_ncols, stream)); + } else { + float * dhf_dst_i = (float *) ((char *) dst_off_device + i02*nb2 + i03*nb3); + GGML_ASSERT(dst->nb[1] == ne0*sizeof(float)); + dhf_dst_i += src1_col_0*ne0; + CUDA_CHECK(cudaMemcpyAsync(dhf_dst_i, dst_dd_i, src1_ncols*ne0*sizeof(float), cudaMemcpyDeviceToDevice, stream)); + } + } + + // add event for the main device to wait on until other device is done + if (split && (id != ctx.device || is != 0)) { + CUDA_CHECK(cudaEventRecord(src0_extra->events[id][is], stream)); + } + } + } + } + + // main device waits for all other devices to be finished + if (split && ggml_backend_cuda_get_device_count() > 1) { + int64_t is_max = (ne11 + MUL_MAT_SRC1_COL_STRIDE - 1) / MUL_MAT_SRC1_COL_STRIDE; + is_max = is_max <= GGML_CUDA_MAX_STREAMS ? is_max : GGML_CUDA_MAX_STREAMS; + + ggml_cuda_set_device(ctx.device); + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + if (dev[id].row_low == dev[id].row_high) { + continue; + } + for (int64_t is = 0; is < is_max; ++is) { + CUDA_CHECK(cudaStreamWaitEvent(ctx.stream(), src0_extra->events[id][is], 0)); + } + } + } +} + +static __global__ void k_compute_batched_ptrs( + const void * src0_as_f16, const void * src1_as_f16, char * dst, + const void ** ptrs_src, void ** ptrs_dst, + int64_t ne12, int64_t ne13, + int64_t ne23, + size_t nb02, size_t nb03, + size_t nb12, size_t nb13, + size_t nbd2, size_t nbd3, + int64_t r2, int64_t r3) { + const int64_t i13 = blockIdx.x * blockDim.x + threadIdx.x; + const int64_t i12 = blockIdx.y * blockDim.y + threadIdx.y; + + if (i13 >= ne13 || i12 >= ne12) { + return; + } + + const int64_t i03 = i13 / r3; + const int64_t i02 = i12 / r2; + + ptrs_src[0*ne23 + i12 + i13*ne12] = (const char *) src0_as_f16 + i02*nb02 + i03*nb03; + ptrs_src[1*ne23 + i12 + i13*ne12] = (const char *) src1_as_f16 + i12*nb12 + i13*nb13; + ptrs_dst[0*ne23 + i12 + i13*ne12] = ( char *) dst + i12*nbd2 + i13*nbd3; +} + +// Type traits for mapping ggml types to CUDA/cuBLAS types +template +struct batched_mul_mat_traits; + +template<> +struct batched_mul_mat_traits { + using cuda_type = float; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; + static inline const cudaDataType_t data_type = CUDA_R_32F; + static inline const ggml_type ggml_type_val = GGML_TYPE_F32; + static inline const float alpha = 1.0f; + static inline const float beta = 0.0f; + static inline const void* get_alpha() { static const float val = alpha; return &val; } + static inline const void* get_beta() { static const float val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp32_nc_cuda(src_type); } +}; + +template<> +struct batched_mul_mat_traits { + using cuda_type = nv_bfloat16; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_32F; + static inline const cudaDataType_t data_type = CUDA_R_16BF; + static inline const ggml_type ggml_type_val = GGML_TYPE_BF16; + static inline const float alpha = 1.0f; + static inline const float beta = 0.0f; + static inline const void* get_alpha() { static const float val = alpha; return &val; } + static inline const void* get_beta() { static const float val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_bf16_nc_cuda(src_type); } +}; + +template<> +struct batched_mul_mat_traits { + using cuda_type = half; + static inline const cublasComputeType_t compute_type = CUBLAS_COMPUTE_16F; + static inline const cudaDataType_t data_type = CUDA_R_16F; + static inline const ggml_type ggml_type_val = GGML_TYPE_F16; + static inline const half alpha = 1.0; + static inline const half beta = 0.0; + static inline const void* get_alpha() { static const half val = alpha; return &val; } + static inline const void* get_beta() { static const half val = beta; return &val; } + static inline auto get_nc_converter(ggml_type src_type) { return ggml_get_to_fp16_nc_cuda(src_type); } +}; + +template +static void ggml_cuda_mul_mat_batched_cublas_impl(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + using traits = batched_mul_mat_traits; + using cuda_t = typename traits::cuda_type; + + GGML_ASSERT(!ggml_is_transposed(src0)); + GGML_ASSERT(!ggml_is_transposed(src1)); + GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft)); + GGML_ASSERT(src0->type == src0_type); + GGML_ASSERT(ggml_is_contiguous(dst)); + + // Byte offsets and tensor dimensions are currently used in an inconsistent way for dst. + // As long as dst is contiguous this does not matter though. + + GGML_TENSOR_BINARY_OP_LOCALS + + const int64_t ne_dst = ggml_nelements(dst); + cudaStream_t main_stream = ctx.stream(); + CUBLAS_CHECK(cublasSetStream(ctx.cublas_handle(), main_stream)); + + float * dst_ddf = (float *) dst->data; + const size_t ts_src1 = ggml_type_size(src1->type); + GGML_ASSERT(nb10 == ts_src1); + int64_t s11 = nb11 / ts_src1; + int64_t s12 = nb12 / ts_src1; + int64_t s13 = nb13 / ts_src1; + + const cuda_t * src0_ptr = nullptr; + const cuda_t * src1_ptr = nullptr; + + ggml_cuda_pool_alloc src0_alloc(ctx.pool()); + ggml_cuda_pool_alloc src1_alloc(ctx.pool()); + + bool is_src0_cont_2 = ggml_is_contiguous_2(src0); + bool is_src1_cont_2 = ggml_is_contiguous_2(src1); + + // Handle src0 + src0_ptr = (const cuda_t *) src0->data; + + // Handle src1 - convert if necessary + if (src1->type == src0_type) { + src1_ptr = (const cuda_t *) src1->data; + } else { + // Convert src1 to target type using traits conversion functions + const int64_t ne_src1 = ggml_nelements(src1); + src1_alloc.alloc(ne_src1); + + const auto convert_func = traits::get_nc_converter(src1->type); + GGML_ASSERT(convert_func != nullptr); + convert_func(src1->data, src1_alloc.get(), ne10, ne11, ne12, ne13, s11, s12, s13, main_stream); + src1_ptr = src1_alloc.get(); + s11 = ne10; + s12 = ne11*s11; + s13 = ne12*s12; + + is_src1_cont_2 = true; + } + + // Setup destination buffer + ggml_cuda_pool_alloc dst_temp(ctx.pool()); + char * dst_t; + size_t nbd2 = dst->nb[2]; + size_t nbd3 = dst->nb[3]; + + cublasComputeType_t cu_compute_type = traits::compute_type; + cudaDataType_t cu_data_type = traits::data_type; + cudaDataType_t cu_data_type_a = traits::data_type; + cudaDataType_t cu_data_type_b = traits::data_type; + const void * alpha = traits::get_alpha(); + const void * beta = traits::get_beta(); + + const auto & force_compute_type = ggml_cuda_cublas_get_force_compute_type(); + + int id = ggml_cuda_get_device(); + const int cc = ggml_cuda_info().devices[id].cc; + static constexpr bool is_src0_type_f16 = src0_type == GGML_TYPE_F16; + + // bf16 and fp32 are already being computed in fp32 (ensure it using static_assert), + // so checking necessity of forced fp32 only for fp16 src0_type + static_assert(is_src0_type_f16 || traits::compute_type == CUBLAS_COMPUTE_32F); + + const bool need_compute_32f = is_src0_type_f16 && !force_compute_type.fp16 && (GGML_CUDA_CC_IS_CDNA(cc) + || GGML_CUDA_CC_IS_RDNA4(cc) + || cc == GGML_CUDA_CC_VOLTA + || force_compute_type.fp32); + + if (dst->op_params[0] == GGML_PREC_DEFAULT && !need_compute_32f) { + if constexpr (src0_type == GGML_TYPE_F32) { + dst_t = (char *) dst_ddf; // Direct F32 output + } else { + dst_t = (char *) dst_temp.alloc(ne_dst); + nbd2 /= sizeof(float) / sizeof(cuda_t); + nbd3 /= sizeof(float) / sizeof(cuda_t); + } + } else { + dst_t = (char *) dst_ddf; + cu_compute_type = batched_mul_mat_traits::compute_type; + cu_data_type = batched_mul_mat_traits::data_type; + alpha = batched_mul_mat_traits::get_alpha(); + beta = batched_mul_mat_traits::get_beta(); + } + + GGML_ASSERT(ne12 % ne02 == 0); + GGML_ASSERT(ne13 % ne03 == 0); + + // broadcast factors + const int64_t r2 = ne12/ne02; + const int64_t r3 = ne13/ne03; + + if (r2 == 1 && r3 == 1 && is_src0_cont_2 && is_src1_cont_2) { + // with a [0, 2, 1, 3] perm. and ne02==1 the matrix strides need to be determined from dim 3: + const int64_t sma = ne02 == 1 ? nb03/nb00 : nb02/nb00; + const int64_t smb = ne12 == 1 ? s13 : s12; + + // there is no broadcast and src0, src1 are contiguous across dims 2, 3 + // use cublasGemmStridedBatchedEx + CUBLAS_CHECK( + cublasGemmStridedBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, + ne01, ne11, ne10, + alpha, src0_ptr, cu_data_type_a, nb01/nb00, sma, // strideA + src1_ptr, cu_data_type_b, s11, smb, // strideB + beta, dst_t, cu_data_type, ne0, ne1*ne0, // strideC + ne12*ne13, + cu_compute_type, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } else { + // use cublasGemmBatchedEx + const int64_t ne23 = ne12*ne13; + + ggml_cuda_pool_alloc ptrs_src(ctx.pool(), 2*ne23); + ggml_cuda_pool_alloc< void *> ptrs_dst(ctx.pool(), 1*ne23); + + size_t src1_stride_size = sizeof(cuda_t); + + const int threads_x = 16; + const int threads_y = 16; + dim3 block_dims(threads_x, threads_y); + + dim3 grid_dims( + (ne13 + threads_x - 1) / threads_x, + (ne12 + threads_y - 1) / threads_y + ); + k_compute_batched_ptrs<<>>( + src0_ptr, src1_ptr, dst_t, + ptrs_src.get(), ptrs_dst.get(), + ne12, ne13, + ne23, + nb02, nb03, + (src1->type == src0_type) ? nb12 : s12*src1_stride_size, + (src1->type == src0_type) ? nb13 : s13*src1_stride_size, + nbd2, nbd3, + r2, r3); + + CUDA_CHECK(cudaGetLastError()); + + CUBLAS_CHECK( + cublasGemmBatchedEx(ctx.cublas_handle(), CUBLAS_OP_T, CUBLAS_OP_N, + ne01, ne11, ne10, + alpha, (const void **) (ptrs_src.get() + 0*ne23), cu_data_type_a, nb01/nb00, + (const void **) (ptrs_src.get() + 1*ne23), cu_data_type_b, s11, + beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne0, + ne23, + cu_compute_type, + CUBLAS_GEMM_DEFAULT_TENSOR_OP)); + } + + // Convert output back to F32 if needed + if (dst->op_params[0] == GGML_PREC_DEFAULT && cu_data_type != CUDA_R_32F) { + const to_fp32_cuda_t to_fp32_cuda = ggml_get_to_fp32_cuda(traits::ggml_type_val); + to_fp32_cuda(dst_temp.get(), dst_ddf, ne_dst, main_stream); + } +} + +static void ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + GGML_ASSERT(src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16 || src0->type == GGML_TYPE_F32); + + switch (src0->type) { + case GGML_TYPE_F32: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + case GGML_TYPE_BF16: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + case GGML_TYPE_F16: + ggml_cuda_mul_mat_batched_cublas_impl(ctx, src0, src1, dst); + break; + default: + GGML_ABORT("Unsupported type"); + } +} + +static bool ggml_cuda_should_fuse_mul_mat(const ggml_tensor * ffn_up, + const ggml_tensor * ffn_gate, + const ggml_tensor * glu, + const ggml_tensor * ffn_up_bias = nullptr, + const ggml_tensor * ffn_gate_bias = nullptr) { + const bool has_bias = ffn_up_bias != nullptr || ffn_gate_bias != nullptr; + + if (has_bias && (!ffn_up_bias || !ffn_gate_bias)) { + return false; + } + + const bool is_mul_mat = ffn_up->op == GGML_OP_MUL_MAT && ffn_gate->op == GGML_OP_MUL_MAT && glu->op == GGML_OP_GLU; + const bool is_mul_mat_id = ffn_up->op == GGML_OP_MUL_MAT_ID && ffn_gate->op == GGML_OP_MUL_MAT_ID && glu->op == GGML_OP_GLU; + + GGML_ASSERT(ffn_up && ffn_gate && glu); + + if (!is_mul_mat && !is_mul_mat_id) { + return false; + } + + const ggml_op expected_bias_op = is_mul_mat ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (has_bias) { + if (ffn_up_bias->op != expected_bias_op || ffn_gate_bias->op != expected_bias_op) { + return false; + } + + if (glu->src[0] != ffn_gate_bias || glu->src[1] != ffn_up_bias) { + return false; + } + + if (expected_bias_op == GGML_OP_ADD) { + const bool up_has_mul = ffn_up_bias->src[0] == ffn_up || ffn_up_bias->src[1] == ffn_up; + const bool gate_has_mul = ffn_gate_bias->src[0] == ffn_gate || ffn_gate_bias->src[1] == ffn_gate; + if (!up_has_mul || !gate_has_mul) { + return false; + } + } else { // GGML_OP_ADD_ID + if (ffn_up_bias->src[0] != ffn_up || ffn_gate_bias->src[0] != ffn_gate) { + return false; + } + if (ffn_up_bias->src[2] != ffn_up->src[2] || ffn_gate_bias->src[2] != ffn_gate->src[2]) { + return false; + } + } + } else { + if (glu->src[0] != ffn_gate && glu->src[1] != ffn_up) { + return false; + } + } + + if (ffn_up->src[0]->type != ffn_gate->src[0]->type || !ggml_are_same_shape(ffn_up->src[0], ffn_gate->src[0]) || + !ggml_are_same_stride(ffn_up->src[0], ffn_gate->src[0])) { + return false; + } + + if (ffn_up->src[1] != ffn_gate->src[1]) { + return false; + } + + if (ffn_up->src[2] && (ffn_up->src[2] != ffn_gate->src[2])) { + return false; + } + + static constexpr std::array valid_glu_ops = { GGML_GLU_OP_SWIGLU, GGML_GLU_OP_GEGLU, GGML_GLU_OP_SWIGLU_OAI }; + + if (std::find(valid_glu_ops.begin(), valid_glu_ops.end(), ggml_get_glu_op(glu)) == valid_glu_ops.end()) { + return false; + } + + if (const bool swapped = ggml_get_op_params_i32(glu, 1); swapped) { + return false; + } + + const bool split = ggml_backend_buft_is_cuda_split(ffn_up->src[0]->buffer->buft) || + ggml_backend_buft_is_cuda_split(ffn_gate->src[0]->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + return true; +} + +static bool ggml_cuda_should_fuse_mul_mat_vec_f(const ggml_tensor * tensor) { + ggml_tensor * src0 = tensor->src[0]; + ggml_tensor * src1 = tensor->src[1]; + const ggml_tensor * dst = tensor; + + const bool is_mul_mat_id = tensor->op == GGML_OP_MUL_MAT_ID; + + bool use_mul_mat_vec_f = + (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) && + src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, is_mul_mat_id ? src1->ne[2] : src1->ne[1]); + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || + ggml_backend_buft_is_cuda_split(src1->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + //we only support fusion for ncols_dst = 1 + if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { + return false; + } + + if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { + return false; + } + + + return use_mul_mat_vec_f; +} + +static bool ggml_cuda_should_fuse_mul_mat_vec_q(const ggml_tensor * tensor) { + ggml_tensor * src0 = tensor->src[0]; + ggml_tensor * src1 = tensor->src[1]; + const ggml_tensor * dst = tensor; + + const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE && + ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && + src0->view_src; + + bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear && src1->type == GGML_TYPE_F32 && + dst->type == GGML_TYPE_F32 && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; + + // fusion is not universally faster on Pascal + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + if (cc <= GGML_CUDA_CC_PASCAL) { + return false; + } + //we only support fusion for ncols_dst = 1 + if (tensor->op == GGML_OP_MUL_MAT && dst->ne[1] != 1) { + return false; + } + + if (tensor->op == GGML_OP_MUL_MAT_ID && dst->ne[2] != 1) { + return false; + } + + + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft) || + ggml_backend_buft_is_cuda_split(src1->buffer->buft); + + //TODO: add support for fusion for split buffers + if (split) { + return false; + } + + return use_mul_mat_vec_q; +} + +static void ggml_cuda_mul_mat(ggml_backend_cuda_context & ctx, const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst) { + const bool split = ggml_backend_buft_is_cuda_split(src0->buffer->buft); + + // If src0 is a temporary compute buffer it may have some padding that needs to be cleared for mul_mat_vec_q or mul_mat_q. + // But if src0 is also a view of another tensor then this cannot be done safely because it may overwrite valid tensor data. + // Therefore, in such cases use cuBLAS. + const bool bad_padding_clear = ggml_backend_buffer_get_usage(src0->buffer) == GGML_BACKEND_BUFFER_USAGE_COMPUTE + && ggml_nbytes(src0) != ggml_backend_buffer_get_alloc_size(src0->buffer, src0) && src0->view_src; + + bool use_mul_mat_vec_f = (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_BF16) + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + bool use_mul_mat_f = !ggml_is_quantized(src0->type) + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + bool use_mul_mat_vec_q = ggml_is_quantized(src0->type) && !bad_padding_clear + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32 + && src1->ne[1] <= MMVQ_MAX_BATCH_SIZE; + bool use_mul_mat_q = ggml_is_quantized(src0->type) && !bad_padding_clear + && src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32; + + bool any_gpus_with_slow_fp16 = false; + + if (split) { + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) src0->buffer->buft->context; + auto & tensor_split = buft_ctx->tensor_split; + for (int id = 0; id < ggml_backend_cuda_get_device_count(); ++id) { + // skip devices that are not going to do any work: + if (tensor_split[id] >= (id + 1 < ggml_backend_cuda_get_device_count() ? tensor_split[id + 1] : 1.0f)) { + continue; + } + + const int cc = ggml_cuda_info().devices[id].cc; + const int warp_size = ggml_cuda_info().devices[id].warp_size; + use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); + use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); + any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); + } + } else { + const int cc = ggml_cuda_info().devices[ctx.device].cc; + const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size; + use_mul_mat_q = use_mul_mat_q && ggml_cuda_should_use_mmq(src0->type, cc, src1->ne[1], /*n_experts=*/0); + use_mul_mat_f = use_mul_mat_f && ggml_cuda_should_use_mmf(src0->type, cc, warp_size, src0->ne, src0->nb, src1->ne[1], /*mul_mat_id=*/false); + use_mul_mat_vec_f = use_mul_mat_vec_f && ggml_cuda_should_use_mmvf(src0->type, cc, src0->ne, src0->nb, src1->ne[1]); + any_gpus_with_slow_fp16 = any_gpus_with_slow_fp16 || !fast_fp16_hardware_available(cc); + } + + // debug helpers + //printf("src0: %8d %8d %8d %8d\n", src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3]); + //printf(" %8d %8d %8d %8d\n", src0->nb[0], src0->nb[1], src0->nb[2], src0->nb[3]); + //printf("src1: %8d %8d %8d %8d\n", src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3]); + //printf(" %8d %8d %8d %8d\n", src1->nb[0], src1->nb[1], src1->nb[2], src1->nb[3]); + //printf("src0 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src0), ggml_is_transposed(src0), ggml_type_name(src0->type), src0->name); + //printf("src1 is contiguous %d, transposed %d, type = %s, name = %s\n", ggml_is_contiguous(src1), ggml_is_transposed(src1), ggml_type_name(src1->type), src1->name); + + //TODO update for generic tensor parallelism + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + bool use_batched_cublas_f16 = src0->type == GGML_TYPE_F16 && (src1->type == GGML_TYPE_F16 || !any_gpus_with_slow_fp16); + bool use_batched_cublas_bf16 = src0->type == GGML_TYPE_BF16 && bf16_mma_hardware_available(cc); + bool use_batched_cublas_f32 = src0->type == GGML_TYPE_F32; + + if (!split && use_mul_mat_vec_f) { + // the custom F16 vector kernel can be used over batched cuBLAS GEMM + // but this is only faster for GPUs without tensor cores or with a thin src0 matrix (particularly KQV in attention) + ggml_cuda_mul_mat_vec_f(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_f) { + ggml_cuda_mul_mat_f(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_vec_q) { + ggml_cuda_mul_mat_vec_q(ctx, src0, src1, nullptr, dst); + } else if (!split && use_mul_mat_q) { + ggml_cuda_mul_mat_q(ctx, src0, src1, nullptr, dst); + } else if (!split && (use_batched_cublas_f16 || use_batched_cublas_bf16 || use_batched_cublas_f32) + && !ggml_is_transposed(src0) && !ggml_is_transposed(src1) && src1->ne[2]*src1->ne[3] > 1) { + // general KQ + KQV multi-batch without FlashAttention + ggml_cuda_mul_mat_batched_cublas(ctx, src0, src1, dst); + } else if (use_mul_mat_vec_f) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_f, nullptr); + } else if (use_mul_mat_vec_q) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_vec_q, quantize_row_q8_1_cuda); + } else if (use_mul_mat_q) { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_q, quantize_mmq_q8_1_cuda); + } else { + ggml_cuda_op_mul_mat(ctx, src0, src1, dst, ggml_cuda_op_mul_mat_cublas, nullptr); + } +} + +static void ggml_cuda_mul_mat_id(ggml_backend_cuda_context & ctx, ggml_tensor * dst) { + const ggml_tensor * src0 = dst->src[0]; + const ggml_tensor * src1 = dst->src[1]; + const ggml_tensor * ids = dst->src[2]; + + GGML_ASSERT(src1->type == GGML_TYPE_F32); + GGML_ASSERT(dst->type == GGML_TYPE_F32); + GGML_ASSERT(!ggml_backend_buft_is_cuda_split(src0->buffer->buft) && "mul_mat_id does not support split buffers"); + + GGML_TENSOR_BINARY_OP_LOCALS + + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + + // [TAG_MUL_MAT_ID_CUDA_GRAPHS] + if (src1->type == GGML_TYPE_F32 && dst->type == GGML_TYPE_F32) { + static_assert(MMVQ_MAX_BATCH_SIZE == MMVF_MAX_BATCH_SIZE); + if (ne2 <= MMVQ_MAX_BATCH_SIZE) { + if (ggml_is_quantized(src0->type)) { + const int mmvq_mmid_max = get_mmvq_mmid_max_batch(src0->type, cc); + if (ne2 <= mmvq_mmid_max) { + ggml_cuda_mul_mat_vec_q(ctx, src0, src1, ids, dst); + return; + } + } else { + if (GGML_CUDA_CC_IS_AMD(cc)) { + ggml_cuda_mul_mat_vec_f(ctx, src0, src1, ids, dst); + return; + } + } + } + + if (ggml_cuda_should_use_mmq(src0->type, cc, ne12, /*n_experts=*/ne02)) { + ggml_cuda_mul_mat_q(ctx, src0, src1, ids, dst); + return; + } + + if (ggml_cuda_should_use_mmf(src0->type, cc, WARP_SIZE, src0->ne, src0->nb, src1->ne[2], /*mul_mat_id=*/true)) { + ggml_cuda_mul_mat_f(ctx, src0, src1, ids, dst); + return; + } + } + + // note: this path should not be reached when recording CUDA graphs, because it requires stream synchronization + // TODO: add asserts to verify this. should work with CUDA, HIP, etc. + cudaStream_t stream = ctx.stream(); + + GGML_ASSERT(nb12 % nb11 == 0); + GGML_ASSERT(nb2 % nb1 == 0); + + const ggml_type type_src1_sorted = (src0->type == GGML_TYPE_F16 && !fast_fp16_hardware_available(cc)) + || ggml_is_quantized(src0->type) ? GGML_TYPE_F32 : src0->type; + const ggml_type type_dst_sorted = GGML_TYPE_F32; + const size_t ts_src1_sorted = ggml_type_size(type_src1_sorted); + const size_t ts_dst_sorted = ggml_type_size(type_dst_sorted); + + const int64_t n_expert_used = ids->ne[0]; + const int64_t ne_get_rows = ne12 * n_expert_used; + + std::vector ids_to_sorted_host; + ids_to_sorted_host.reserve(2*ne_get_rows); + std::vector ids_from_sorted_host(ne_get_rows); + + ggml_cuda_pool_alloc ids_buf_dev(ctx.pool(), 2*ne_get_rows); + + std::vector tokens_per_expert(ne02); + + ggml_cuda_pool_alloc src1_sorted(ctx.pool(), ne12*n_expert_used*ne10*ts_src1_sorted); + ggml_cuda_pool_alloc dst_sorted(ctx.pool(), ne2 *n_expert_used* ne0*ts_dst_sorted); + + std::vector ids_host(ggml_nbytes(ids)); + CUDA_CHECK(cudaMemcpyAsync(ids_host.data(), ids->data, ggml_nbytes(ids), cudaMemcpyDeviceToHost, stream)); + CUDA_CHECK(cudaStreamSynchronize(stream)); + + for (int64_t i02 = 0; i02 < ne02; ++i02) { // expert matrices + for (int64_t i12 = 0; i12 < ne12; ++i12) { // tokens + for (int64_t iex = 0; iex < n_expert_used; ++iex) { + const int32_t expert_to_use = *(const int32_t *)(ids_host.data() + i12*ids->nb[1] + iex*ids->nb[0]); + assert(expert_to_use >= 0 && expert_to_use < ne02); + if (expert_to_use == i02) { + ids_from_sorted_host[i12*n_expert_used + iex] = ids_to_sorted_host.size(); + ids_to_sorted_host.push_back(i12*ne11 + iex % ne11); + tokens_per_expert[i02]++; + break; + } + } + } + } + GGML_ASSERT(ids_to_sorted_host.size() == size_t(ne_get_rows)); + + ids_to_sorted_host.insert(ids_to_sorted_host.end(), ids_from_sorted_host.begin(), ids_from_sorted_host.end()); + + CUDA_CHECK(cudaMemcpyAsync(ids_buf_dev.ptr, ids_to_sorted_host.data(), 2*ne_get_rows*sizeof(int32_t), cudaMemcpyHostToDevice, stream)); + CUDA_CHECK(cudaStreamSynchronize(stream)); + + const int32_t * ids_to_sorted = ids_buf_dev.ptr + 0*ne_get_rows; + const int32_t * ids_from_sorted = ids_buf_dev.ptr + 1*ne_get_rows; + + get_rows_cuda(src1->data, src1->type, ids_to_sorted, src1_sorted.ptr, type_src1_sorted, + ne10, nb11, nb12, nb13, + ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), + ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, ne_get_rows*ne10*ts_src1_sorted, stream); + CUDA_CHECK(cudaGetLastError()); + + char * src1_data_cur = (char *) src1_sorted.ptr; + char * dst_data_cur = (char *) dst_sorted.ptr; + for (int64_t i02 = 0; i02 < ne02; ++i02) { + if (tokens_per_expert[i02] == 0) { + continue; + } + + ggml_tensor src0_slice = *src0; + src0_slice.ne[2] = 1; + src0_slice.nb[3] = src0_slice.nb[2]; + src0_slice.op = GGML_OP_VIEW; + src0_slice.view_src = dst->src[0]; // non-const pointer to src0 + src0_slice.data = (char *) src0->data + i02*nb02; + + ggml_tensor src1_slice; + memset(&src1_slice, 0, sizeof(src1_slice)); + src1_slice.buffer = src1->buffer; + src1_slice.type = type_src1_sorted; + src1_slice.ne[0] = ne10; + src1_slice.ne[1] = tokens_per_expert[i02]; + src1_slice.ne[2] = 1; + src1_slice.ne[3] = 1; + src1_slice.nb[0] = ts_src1_sorted; + src1_slice.nb[1] = src1_slice.ne[0] * src1_slice.nb[0]; + src1_slice.nb[2] = src1_slice.ne[1] * src1_slice.nb[1]; + src1_slice.nb[3] = src1_slice.ne[2] * src1_slice.nb[2]; + src1_slice.data = src1_data_cur; + + ggml_tensor dst_slice; + memset(&dst_slice, 0, sizeof(dst_slice)); + dst_slice.buffer = dst->buffer; + dst_slice.type = type_dst_sorted; + dst_slice.ne[0] = ne0; + dst_slice.ne[1] = tokens_per_expert[i02]; + dst_slice.ne[2] = 1; + dst_slice.ne[3] = 1; + dst_slice.nb[0] = ts_dst_sorted; + dst_slice.nb[1] = dst_slice.ne[0] * dst_slice.nb[0]; + dst_slice.nb[2] = dst_slice.ne[1] * dst_slice.nb[1]; + dst_slice.nb[3] = dst_slice.ne[2] * dst_slice.nb[2]; + dst_slice.data = dst_data_cur; + + ggml_cuda_mul_mat(ctx, &src0_slice, &src1_slice, &dst_slice); + CUDA_CHECK(cudaGetLastError()); + + src1_data_cur += src1_slice.nb[2]; + dst_data_cur += dst_slice.nb[2]; + } + + get_rows_cuda(dst_sorted.ptr, type_dst_sorted, ids_from_sorted, dst->data, dst->type, + ne0, ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, ne_get_rows*ne0*ts_dst_sorted, + ne_get_rows, 1, 1, sizeof(int32_t), ne_get_rows*sizeof(int32_t), ne_get_rows*sizeof(int32_t), + nb1, nb2, nb3, stream); +} + +static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct ggml_tensor * dst) { + switch (dst->op) { + case GGML_OP_ARGMAX: + ggml_cuda_argmax(ctx, dst); + break; + case GGML_OP_COUNT_EQUAL: + ggml_cuda_count_equal(ctx, dst); + break; + case GGML_OP_REPEAT: + ggml_cuda_op_repeat(ctx, dst); + break; + case GGML_OP_REPEAT_BACK: + ggml_cuda_op_repeat_back(ctx, dst); + break; + case GGML_OP_GET_ROWS: + ggml_cuda_op_get_rows(ctx, dst); + break; + case GGML_OP_GET_ROWS_BACK: + ggml_cuda_op_get_rows_back(ctx, dst); + break; + case GGML_OP_SET_ROWS: + ggml_cuda_op_set_rows(ctx, dst); + break; + case GGML_OP_SET: + ggml_cuda_op_set(ctx, dst); + break; + case GGML_OP_DUP: + ggml_cuda_dup(ctx, dst); + break; + case GGML_OP_CPY: + ggml_cuda_cpy(ctx, dst->src[0], dst->src[1]); + break; + case GGML_OP_CONT: + ggml_cuda_dup(ctx, dst); + break; + case GGML_OP_ADD: + case GGML_OP_ADD1: // TODO: more efficient implementation + ggml_cuda_op_add(ctx, dst); + break; + case GGML_OP_ADD_ID: + ggml_cuda_op_add_id(ctx, dst); + break; + case GGML_OP_SUB: + ggml_cuda_op_sub(ctx, dst); + break; + case GGML_OP_ACC: + ggml_cuda_op_acc(ctx, dst); + break; + case GGML_OP_MUL: + ggml_cuda_op_mul(ctx, dst); + break; + case GGML_OP_DIV: + ggml_cuda_op_div(ctx, dst); + break; + case GGML_OP_UNARY: + switch (ggml_get_unary_op(dst)) { + case GGML_UNARY_OP_ABS: + ggml_cuda_op_abs(ctx, dst); + break; + case GGML_UNARY_OP_SGN: + ggml_cuda_op_sgn(ctx, dst); + break; + case GGML_UNARY_OP_NEG: + ggml_cuda_op_neg(ctx, dst); + break; + case GGML_UNARY_OP_STEP: + ggml_cuda_op_step(ctx, dst); + break; + case GGML_UNARY_OP_GELU: + ggml_cuda_op_gelu(ctx, dst); + break; + case GGML_UNARY_OP_SILU: + ggml_cuda_op_silu(ctx, dst); + break; + case GGML_UNARY_OP_GELU_ERF: + ggml_cuda_op_gelu_erf(ctx, dst); + break; + case GGML_UNARY_OP_GELU_QUICK: + ggml_cuda_op_gelu_quick(ctx, dst); + break; + case GGML_UNARY_OP_TANH: + ggml_cuda_op_tanh(ctx, dst); + break; + case GGML_UNARY_OP_RELU: + ggml_cuda_op_relu(ctx, dst); + break; + case GGML_UNARY_OP_SIGMOID: + ggml_cuda_op_sigmoid(ctx, dst); + break; + case GGML_UNARY_OP_HARDSIGMOID: + ggml_cuda_op_hardsigmoid(ctx, dst); + break; + case GGML_UNARY_OP_HARDSWISH: + ggml_cuda_op_hardswish(ctx, dst); + break; + case GGML_UNARY_OP_EXP: + ggml_cuda_op_exp(ctx, dst); + break; + case GGML_UNARY_OP_ELU: + ggml_cuda_op_elu(ctx, dst); + break; + case GGML_UNARY_OP_XIELU: + ggml_cuda_op_xielu(ctx, dst); + break; + case GGML_UNARY_OP_FLOOR: + ggml_cuda_op_floor(ctx, dst); + break; + case GGML_UNARY_OP_CEIL: + ggml_cuda_op_ceil(ctx, dst); + break; + case GGML_UNARY_OP_ROUND: + ggml_cuda_op_round(ctx, dst); + break; + case GGML_UNARY_OP_TRUNC: + ggml_cuda_op_trunc(ctx, dst); + break; + case GGML_UNARY_OP_EXPM1: + ggml_cuda_op_expm1(ctx, dst); + break; + case GGML_UNARY_OP_SOFTPLUS: + ggml_cuda_op_softplus(ctx, dst); + break; + default: + return false; + } + break; + case GGML_OP_GLU: + switch (ggml_get_glu_op(dst)) { + case GGML_GLU_OP_REGLU: + ggml_cuda_op_reglu(ctx, dst); + break; + case GGML_GLU_OP_GEGLU: + ggml_cuda_op_geglu(ctx, dst); + break; + case GGML_GLU_OP_SWIGLU: + ggml_cuda_op_swiglu(ctx, dst); + break; + case GGML_GLU_OP_SWIGLU_OAI: + ggml_cuda_op_swiglu_oai(ctx, dst); + break; + case GGML_GLU_OP_GEGLU_ERF: + ggml_cuda_op_geglu_erf(ctx, dst); + break; + case GGML_GLU_OP_GEGLU_QUICK: + ggml_cuda_op_geglu_quick(ctx, dst); + break; + default: + return false; + } + break; + case GGML_OP_NORM: + ggml_cuda_op_norm(ctx, dst); + break; + case GGML_OP_GROUP_NORM: + ggml_cuda_op_group_norm(ctx, dst); + break; + case GGML_OP_L2_NORM: + ggml_cuda_op_l2_norm(ctx, dst); + break; + case GGML_OP_CONCAT: + ggml_cuda_op_concat(ctx, dst); + break; + case GGML_OP_UPSCALE: + ggml_cuda_op_upscale(ctx, dst); + break; + case GGML_OP_PAD: + ggml_cuda_op_pad(ctx, dst); + break; + case GGML_OP_PAD_REFLECT_1D: + ggml_cuda_op_pad_reflect_1d(ctx, dst); + break; + case GGML_OP_ARANGE: + ggml_cuda_op_arange(ctx, dst); + break; + case GGML_OP_TIMESTEP_EMBEDDING: + ggml_cuda_op_timestep_embedding(ctx, dst); + break; + case GGML_OP_LEAKY_RELU: + ggml_cuda_op_leaky_relu(ctx, dst); + break; + case GGML_OP_SILU_BACK: + ggml_cuda_op_silu_back(ctx, dst); + break; + case GGML_OP_RMS_NORM: + ggml_cuda_op_rms_norm(ctx, dst); + break; + case GGML_OP_RMS_NORM_BACK: + ggml_cuda_op_rms_norm_back(ctx, dst); + break; + case GGML_OP_MUL_MAT: + ggml_cuda_mul_mat(ctx, dst->src[0], dst->src[1], dst); + break; + case GGML_OP_MUL_MAT_ID: + ggml_cuda_mul_mat_id(ctx, dst); + break; + case GGML_OP_OUT_PROD: + ggml_cuda_out_prod(ctx, dst); + break; + case GGML_OP_SCALE: + ggml_cuda_op_scale(ctx, dst); + break; + case GGML_OP_SQR: + ggml_cuda_op_sqr(ctx, dst); + break; + case GGML_OP_SQRT: + ggml_cuda_op_sqrt(ctx, dst); + break; + case GGML_OP_SIN: + ggml_cuda_op_sin(ctx, dst); + break; + case GGML_OP_COS: + ggml_cuda_op_cos(ctx, dst); + break; + case GGML_OP_CLAMP: + ggml_cuda_op_clamp(ctx, dst); + break; + case GGML_OP_LOG: + ggml_cuda_op_log(ctx, dst); + break; + case GGML_OP_NONE: + case GGML_OP_RESHAPE: + case GGML_OP_VIEW: + case GGML_OP_PERMUTE: + case GGML_OP_TRANSPOSE: + break; + case GGML_OP_DIAG: + ggml_cuda_op_diag(ctx, dst); + break; + case GGML_OP_DIAG_MASK_INF: + ggml_cuda_op_diag_mask_inf(ctx, dst); + break; + case GGML_OP_SOFT_MAX: + ggml_cuda_op_soft_max(ctx, dst); + break; + case GGML_OP_SOFT_MAX_BACK: + ggml_cuda_op_soft_max_back(ctx, dst); + break; + case GGML_OP_ROPE: + ggml_cuda_op_rope(ctx, dst); + break; + case GGML_OP_ROPE_BACK: + ggml_cuda_op_rope_back(ctx, dst); + break; + case GGML_OP_ROLL: + ggml_cuda_op_roll(ctx, dst); + break; + case GGML_OP_IM2COL: + ggml_cuda_op_im2col(ctx, dst); + break; + case GGML_OP_IM2COL_3D: + ggml_cuda_op_im2col_3d(ctx, dst); + break; + case GGML_OP_CONV_2D: + ggml_cuda_op_conv2d(ctx, dst); + break; + case GGML_OP_CONV_2D_DW: + ggml_cuda_op_conv2d_dw(ctx, dst); + break; + case GGML_OP_CONV_TRANSPOSE_2D: + ggml_cuda_conv_2d_transpose_p0(ctx, dst); + break; + case GGML_OP_CONV_TRANSPOSE_1D: + ggml_cuda_op_conv_transpose_1d(ctx,dst); + break; + case GGML_OP_POOL_2D: + ggml_cuda_op_pool2d(ctx, dst); + break; + case GGML_OP_SUM: + ggml_cuda_op_sum(ctx, dst); + break; + case GGML_OP_CUMSUM: + ggml_cuda_op_cumsum(ctx, dst); + break; + case GGML_OP_SUM_ROWS: + ggml_cuda_op_sum_rows(ctx, dst); + break; + case GGML_OP_MEAN: + ggml_cuda_op_mean(ctx, dst); + break; + case GGML_OP_SSM_CONV: + ggml_cuda_op_ssm_conv(ctx, dst); + break; + case GGML_OP_SSM_SCAN: + ggml_cuda_op_ssm_scan(ctx, dst); + break; + case GGML_OP_TOP_K: + ggml_cuda_op_top_k(ctx, dst); + break; + case GGML_OP_ARGSORT: + ggml_cuda_op_argsort(ctx, dst); + break; + case GGML_OP_FLASH_ATTN_EXT: + ggml_cuda_flash_attn_ext(ctx, dst); + break; + case GGML_OP_CROSS_ENTROPY_LOSS: + ggml_cuda_cross_entropy_loss(ctx, dst); + break; + case GGML_OP_TRI: + ggml_cuda_op_tri(ctx, dst); + break; + case GGML_OP_RWKV_WKV6: + ggml_cuda_op_rwkv_wkv6(ctx, dst); + break; + case GGML_OP_GATED_LINEAR_ATTN: + ggml_cuda_op_gated_linear_attn(ctx, dst); + break; + case GGML_OP_GATED_DELTA_NET: + ggml_cuda_op_gated_delta_net(ctx, dst); + break; + case GGML_OP_RWKV_WKV7: + ggml_cuda_op_rwkv_wkv7(ctx, dst); + break; + case GGML_OP_CROSS_ENTROPY_LOSS_BACK: + ggml_cuda_cross_entropy_loss_back(ctx, dst); + break; + case GGML_OP_OPT_STEP_ADAMW: + ggml_cuda_opt_step_adamw(ctx, dst); + break; + case GGML_OP_OPT_STEP_SGD: + ggml_cuda_opt_step_sgd(ctx, dst); + break; + case GGML_OP_SOLVE_TRI: + ggml_cuda_op_solve_tri(ctx, dst); + break; + case GGML_OP_FILL: + ggml_cuda_op_fill(ctx, dst); + break; + default: + return false; + } + + cudaError_t err = cudaGetLastError(); + if (err != cudaSuccess) { + GGML_LOG_ERROR("%s: %s failed\n", __func__, ggml_op_desc(dst)); + CUDA_CHECK(err); + } + + return true; +} + +//////////////////////////////////////////////////////////////////////////////// + +// backend + +static const char * ggml_backend_cuda_get_name(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + return cuda_ctx->name.c_str(); +} + +static void ggml_backend_cuda_free(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + delete cuda_ctx; + delete backend; +} + +static void ggml_backend_cuda_set_tensor_async(ggml_backend_t backend, ggml_tensor * tensor, const void * data, size_t offset, size_t size) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpyAsync((char *) tensor->data + offset, data, size, cudaMemcpyHostToDevice, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_get_tensor_async(ggml_backend_t backend, const ggml_tensor * tensor, void * data, size_t offset, size_t size) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpyAsync(data, (const char *) tensor->data + offset, size, cudaMemcpyDeviceToHost, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_set_tensor_2d_async(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpy2DAsync( + (char *) tensor->data + offset, stride_tensor, data, stride_data, size, n_copies, cudaMemcpyHostToDevice, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_get_tensor_2d_async(ggml_backend_t backend, const struct ggml_tensor * tensor, void * data, + size_t offset, size_t size, size_t n_copies, size_t stride_tensor, size_t stride_data) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + ggml_backend_buffer_t buf = tensor->view_src ? tensor->view_src->buffer : tensor->buffer; + + GGML_ASSERT(buf->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) && "unsupported buffer type"); + + CUDA_CHECK(cudaMemcpy2DAsync( + data, stride_data, (const char *) tensor->data + offset, stride_tensor, size, n_copies, cudaMemcpyDeviceToHost, cuda_ctx->stream())); +} + +static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) { + ggml_backend_buffer_t buf_src = src->view_src ? src->view_src->buffer : src->buffer; + ggml_backend_buffer_t buf_dst = dst->view_src ? dst->view_src->buffer : dst->buffer; + + if (!ggml_backend_is_cuda(backend_src) || !ggml_backend_is_cuda(backend_dst)) { + return false; + } + + if (!ggml_backend_buffer_is_cuda(buf_src) || !ggml_backend_buffer_is_cuda(buf_dst)) { + return false; + } + + // device -> device copy + ggml_backend_cuda_context * cuda_ctx_src = (ggml_backend_cuda_context *) backend_src->context; + ggml_backend_cuda_context * cuda_ctx_dst = (ggml_backend_cuda_context *) backend_dst->context; + + ggml_backend_cuda_buffer_context * buf_ctx_src = (ggml_backend_cuda_buffer_context *) buf_src->context; + ggml_backend_cuda_buffer_context * buf_ctx_dst = (ggml_backend_cuda_buffer_context *) buf_dst->context; + + if (cuda_ctx_src->device != buf_ctx_src->device || cuda_ctx_dst->device != buf_ctx_dst->device) { +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: backend and buffer devices do not match\n", __func__); +#endif // NDEBUG + return false; + } + + if (backend_src != backend_dst) { + // copy on src stream + if (cuda_ctx_src->device == cuda_ctx_dst->device) { + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); + } else { +#ifdef GGML_CUDA_NO_PEER_COPY + return false; +#else + CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream())); +#endif // GGML_CUDA_NO_PEER_COPY + } + + // record event on src stream after the copy + if (!cuda_ctx_src->copy_event) { + ggml_cuda_set_device(cuda_ctx_src->device); + CUDA_CHECK(cudaEventCreateWithFlags(&cuda_ctx_src->copy_event, cudaEventDisableTiming)); + } + + CUDA_CHECK(cudaEventRecord(cuda_ctx_src->copy_event, cuda_ctx_src->stream())); + + // wait on dst stream for the copy to complete + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx_dst->stream(), cuda_ctx_src->copy_event, 0)); + } else { + // src and dst are on the same backend + CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream())); + } + return true; +} + +static void ggml_backend_cuda_synchronize(ggml_backend_t backend) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + CUDA_CHECK(cudaStreamSynchronize(cuda_ctx->stream())); + + GGML_UNUSED(backend); +} + +#ifdef USE_CUDA_GRAPH +static bool ggml_cuda_graph_check_compability(ggml_cgraph * cgraph) { + + bool use_cuda_graph = true; + // Loop over nodes in GGML graph to obtain info needed for CUDA graph + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_tensor * node = cgraph->nodes[i]; + + if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { + continue; + } + + if (node->src[0] && node->src[0]->buffer && ggml_backend_buft_is_cuda_split(node->src[0]->buffer->buft)) { + use_cuda_graph = false; // Split buffers are not supported by CUDA graph capture +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to split buffer\n", __func__); +#endif + } + + // [TAG_MUL_MAT_ID_CUDA_GRAPHS] + if (node->op == GGML_OP_MUL_MAT_ID) { + const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc; + const int mmvq_mmid_max = get_mmvq_mmid_max_batch(node->src[0]->type, cc); + if (!ggml_is_quantized(node->src[0]->type) || node->ne[2] > mmvq_mmid_max) { + // under these conditions, the mul_mat_id operation will need to synchronize the stream, so we cannot use CUDA graphs + // TODO: figure out a way to enable for larger batch sizes, without hurting performance + // ref: https://github.com/ggml-org/llama.cpp/pull/18958 + use_cuda_graph = false; +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to unsupported node type\n", __func__); +#endif + } + } + + if (!use_cuda_graph) { + break; + } + } + + return use_cuda_graph; +} + +static const void * ggml_cuda_graph_get_key(ggml_cgraph * cgraph) { + return cgraph->nodes[0]; +} + +static bool ggml_cuda_graph_update_required(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph) { + bool res = false; + + const void * graph_key = ggml_cuda_graph_get_key(cgraph); + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + + if (cgraph->uid != 0 && + cgraph->uid == graph->uid) { + GGML_LOG_DEBUG("CUDA Graph id %zu reused\n", cgraph->uid); + GGML_ASSERT((int)graph->node_props.size() == cgraph->n_nodes); + return false; + } + + graph->uid = cgraph->uid; + + // Check if the graph size has changed + if ((int)graph->node_props.size() != cgraph->n_nodes) { + res = true; + graph->node_props.resize(cgraph->n_nodes); + } + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_cuda_graph::node_properties prop = {}; + memcpy(&prop.node, cgraph->nodes[i], sizeof(ggml_tensor)); + + for (int j = 0; j < GGML_MAX_SRC; ++j) { + if (cgraph->nodes[i]->src[j]) { + prop.node_src_data_ptrs[j] = cgraph->nodes[i]->src[j]->data; + memcpy(prop.node_src_ne[j], cgraph->nodes[i]->src[j]->ne, sizeof(prop.node_src_ne[j])); + memcpy(prop.node_src_nb[j], cgraph->nodes[i]->src[j]->nb, sizeof(prop.node_src_nb[j])); + } + } + + if (res || memcmp(&graph->node_props[i], &prop, sizeof(prop)) != 0) { + graph->node_props[i] = prop; + res = true; + } + } + + return res; +} + +static void ggml_cuda_graph_update_executable(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + +#if CUDART_VERSION >= 12000 + cudaGraphExecUpdateResultInfo result_info; + cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &result_info); +#else + cudaGraphNode_t errorNode; + cudaGraphExecUpdateResult result_info; + cudaError_t stat = cudaGraphExecUpdate(graph->instance, graph->graph, &errorNode, &result_info); +#endif // CUDART_VERSION >= 12000 + + if (stat == cudaErrorGraphExecUpdateFailure) { +#ifndef NDEBUG + GGML_LOG_DEBUG("%s: CUDA graph update failed\n", __func__); +#endif + + // The pre-existing graph exec cannot be updated due to violated constraints + // so instead clear error and re-instantiate + (void)cudaGetLastError(); + CUDA_CHECK(cudaGraphExecDestroy(graph->instance)); + graph->instance = nullptr; + CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); + } else { + GGML_ASSERT(stat == cudaSuccess); + } +} +#endif // USE_CUDA_GRAPH + +static bool ggml_cuda_should_fuse_rope_set_rows(const ggml_tensor * rope, + const ggml_tensor * view, + const ggml_tensor * set_rows) { + + if (rope->op != GGML_OP_ROPE || view->op != GGML_OP_VIEW || set_rows->op != GGML_OP_SET_ROWS) { + return false; + } + // ne3 not tested + if (rope->src[0]->ne[3] != 1) { + return false; + } + + if (set_rows->type != GGML_TYPE_F32 && set_rows->type != GGML_TYPE_F16) { + return false; + } + + if (set_rows->src[1]->type != GGML_TYPE_I64) { + return false; + } + + // The view should flatten two dims of rope into one dim + if (!ggml_is_contiguous(view) || view->ne[0] != rope->ne[0] * rope->ne[1]) { + return false; + } + + // Only norm/neox shaders have the fusion code + const int mode = ((const int32_t *) rope->op_params)[2]; + if (mode != GGML_ROPE_TYPE_NORMAL && mode != GGML_ROPE_TYPE_NEOX) { + return false; + } + + return true; +} + +static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) { + args.sigmoid = false; + args.softmax = false; + args.delayed_softmax = false; + args.prob_bias = false; + args.norm = false; + + const int n_nodes = cgraph->n_nodes; + ggml_tensor ** nodes = cgraph->nodes; + + if (nodes[node_idx]->op == GGML_OP_SOFT_MAX) { + args.softmax = true; + } + + if (nodes[node_idx]->op == GGML_OP_UNARY) { + if (ggml_get_unary_op(nodes[node_idx]) != GGML_UNARY_OP_SIGMOID) { + return false; + } + args.sigmoid = true; + } + + if (nodes[node_idx]->op == GGML_OP_ARGSORT) { + args.delayed_softmax = true; + } + + node_idx++; + + if (args.sigmoid || args.softmax) { + // SOFTMAX -> RESHAPE + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + ggml_tensor * probs_reshaped = nodes[node_idx]; + node_idx++; + + if (node_idx >= n_nodes) { + return false; + } + + // src of bias add is the unreshaped probs (-2 instead of -1) + if (nodes[node_idx]->op == GGML_OP_ADD && nodes[node_idx]->src[0] == nodes[node_idx - 2]) { + args.prob_bias = true; + node_idx++; + } + // RESHAPE/ADD -> ARGSORT + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_ARGSORT) { + return false; + } + + if (args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } else if (!args.prob_bias && nodes[node_idx]->src[0] != nodes[node_idx - 2]) { + return false; + } + + node_idx++; + + // ARGSORT-> VIEW + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_GET_ROWS) { + return false; + } + + // GET_ROWS + if (nodes[node_idx]->src[0] != probs_reshaped || nodes[node_idx]->src[1] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + } else if (args.delayed_softmax) { + if (node_idx - 2 < 0) { + return false; + } + ggml_tensor * probs_reshaped = nodes[node_idx - 2]; + + // VIEW->ARGSORT + if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_VIEW || + nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + + // GET_ROWS + if (node_idx >= n_nodes || nodes[node_idx]->src[1] != nodes[node_idx - 1] || + nodes[node_idx]->src[0] != probs_reshaped) { + return false; + } + node_idx++; + + static const std::vector remaining_ops = { GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }; + + for (const ggml_op op : remaining_ops) { + if (node_idx >= n_nodes || nodes[node_idx]->op != op || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + return false; + } + node_idx++; + } + } + + // At this point we can check for norm + scale. Everything is now at least valid till the norm + if (node_idx >= n_nodes) { + return true; + } + + if (nodes[node_idx]->op == GGML_OP_RESHAPE) { + //check RESHAPE->SUM_ROWS->CLAMP->DIV->RESHAPE + static const std::vector norm_ops = { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP }; + + args.norm = true; + for (const ggml_op op : norm_ops) { + if (nodes[node_idx]->op == op && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { + node_idx++; + } else { + args.norm = false; + return true; + } + } + + // DIV <- CLAMP, RESHAPE + if (nodes[node_idx]->op != GGML_OP_DIV || nodes[node_idx]->src[1] != nodes[node_idx - 1] || + nodes[node_idx]->src[0] != nodes[node_idx - 3]) { + args.norm = false; + return true; + } + node_idx++; + + if (nodes[node_idx]->op != GGML_OP_RESHAPE || nodes[node_idx]->src[0] != nodes[node_idx - 1]) { + args.norm = false; + return true; + } + + node_idx++; + } + + if (nodes[node_idx]->op == GGML_OP_SCALE && nodes[node_idx]->src[0] == nodes[node_idx - 1]) { + args.scale = true; + } + + return true; +} + +// returns whether the write (out) nodes overwrite the read nodes in operation +static bool ggml_cuda_check_fusion_memory_ranges(const ggml_cgraph * cgraph, + const int node_idx, + const int node_count, + const int * out_nodes, + const int out_count, + const bool is_topk_moe = false) { + auto nodes_overlap = [&](const ggml_tensor * a, const ggml_tensor * b) { + const int64_t a_start = (int64_t) a->data; + const int64_t a_end = a_start + ggml_backend_buft_get_alloc_size(a->buffer->buft, a); + + const int64_t b_start = (int64_t) b->data; + const int64_t b_end = b_start + ggml_backend_buft_get_alloc_size(b->buffer->buft, b); + + if ((b_start <= a_start && a_start < b_end) || (a_start <= b_start && b_start < a_end)) { + return true; + } + + return false; + }; + + bool is_ok = true; + // exception for topk-moe, as each row is read entirely before writing + if (ggml_nrows(cgraph->nodes[node_idx]) == 1 && is_topk_moe) { + return true; + } + + for (int i = 0; i < out_count; ++i) { + const ggml_tensor * dst = cgraph->nodes[out_nodes[i]]; + + for (int j = node_idx; j < node_idx + node_count; ++j) { + // Loop over all srcs of all nodes in the fusion. If the src overlaps + // the destination and the src is not an intermediate node that's being + // elided, then disable fusion. + + for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { + const ggml_tensor * src = cgraph->nodes[j]->src[src_idx]; + + if (!src || src->op == GGML_OP_NONE) { + continue; + } + + if (nodes_overlap(dst, src)) { + bool found = false; + + for (int k = node_idx; k < j; ++k) { + if (cgraph->nodes[k] == src) { + found = true; + break; + } + } + + if (!found) { + is_ok = false; + break; + } + } + } + } + } + + return is_ok; +} + + +static bool ggml_cuda_can_fuse(const struct ggml_cgraph * cgraph, + int node_idx, + std::initializer_list ops, + std::initializer_list unary_ops) { +#ifndef NDEBUG + const size_t num_unary = std::count(ops.begin(), ops.end(), GGML_OP_UNARY); + GGML_ASSERT(unary_ops.size() == num_unary); +#endif + + const auto is_equal = [](const std::initializer_list & list1, + const std::initializer_list & list2) { + return std::equal(list1.begin(), list1.end(), list2.begin(), list2.end()); + }; + + std::initializer_list mul_mat_bias_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_GLU }; + std::initializer_list mul_mat_id_bias_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_MUL_MAT_ID, GGML_OP_ADD_ID, GGML_OP_GLU }; + + std::initializer_list mul_mat_id_glu_ops = { GGML_OP_MUL_MAT_ID, GGML_OP_MUL_MAT_ID, GGML_OP_GLU }; + std::initializer_list mul_mat_glu_ops = { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT, GGML_OP_GLU }; + + if ((is_equal(mul_mat_bias_glu_ops, ops) || is_equal(mul_mat_id_bias_glu_ops, ops)) && + ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 4 })) { + const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; + const ggml_tensor * ffn_gate_bias = cgraph->nodes[node_idx + 1]; + const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 2]; + const ggml_tensor * ffn_up_bias = cgraph->nodes[node_idx + 3]; + const ggml_tensor * glu = cgraph->nodes[node_idx + 4]; + + if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu, ffn_up_bias, ffn_gate_bias)) { + int out_nodes[] = { node_idx + 4 }; + return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); + } + } + + if ((is_equal(mul_mat_id_glu_ops, ops) || is_equal(mul_mat_glu_ops, ops)) && + ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { + const ggml_tensor * ffn_gate = cgraph->nodes[node_idx]; + const ggml_tensor * ffn_up = cgraph->nodes[node_idx + 1]; + const ggml_tensor * glu = cgraph->nodes[node_idx + 2]; + + if (ggml_cuda_should_fuse_mul_mat(ffn_up, ffn_gate, glu)) { + int out_nodes[] = { node_idx + 2 }; + return ggml_cuda_check_fusion_memory_ranges(cgraph, node_idx, (int)ops.size(), out_nodes, 1); + } + } + + std::initializer_list rope_set_rows_ops = { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }; + + if (is_equal(rope_set_rows_ops, ops) && ggml_can_fuse_subgraph(cgraph, node_idx, ops, { node_idx + 2 })) { + const ggml_tensor * rope = cgraph->nodes[node_idx]; + const ggml_tensor * view = cgraph->nodes[node_idx + 1]; + const ggml_tensor * set_rows = cgraph->nodes[node_idx + 2]; + + if (ggml_cuda_should_fuse_rope_set_rows(rope, view, set_rows)) { + return true; + } + } + + if (!ggml_can_fuse(cgraph, node_idx, ops)) { + return false; + } + + if ((ops.size() == 2 || ops.size() == 3) && ops.begin()[0] == GGML_OP_RMS_NORM && ops.begin()[1] == GGML_OP_MUL) { + const ggml_tensor *rms_norm = cgraph->nodes[node_idx]; + const ggml_tensor *mul = cgraph->nodes[node_idx+1]; + const ggml_tensor *add = nullptr; + + if (ops.size() == 3 && ops.begin()[2] == GGML_OP_ADD) { + add = cgraph->nodes[node_idx+2]; + } + + GGML_ASSERT(rms_norm->src[0]->type == GGML_TYPE_F32); + GGML_ASSERT(rms_norm->type == GGML_TYPE_F32); + + //rms norm only supports F32 + if (mul->src[0]->type != GGML_TYPE_F32 || + mul->src[1]->type != GGML_TYPE_F32 || + mul->type != GGML_TYPE_F32) { + return false; + } + + if (add && (add->src[0]->type != GGML_TYPE_F32 || + add->src[1]->type != GGML_TYPE_F32 || + add->type != GGML_TYPE_F32) ) { + return false; + } + + //if rms norm is the B operand, then we don't handle broadcast + if (rms_norm == mul->src[1] && !ggml_are_same_shape(mul->src[0], rms_norm)) { + return false; + } + + //rms_norm kernel assumes contiguous rows + if (!ggml_is_contiguous_rows(mul->src[0]) || !ggml_is_contiguous_rows(mul->src[1])) { + return false; + } + + if (add && (!ggml_is_contiguous(add->src[0]) || !ggml_is_contiguous_rows(add->src[1]))) { + return false; + } + + return true; + } + + if (ops.size() == 2 && ops.begin()[0] == GGML_OP_SSM_CONV && ops.begin()[1] == GGML_OP_UNARY + && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_SILU) { + const ggml_tensor * ssm_conv = cgraph->nodes[node_idx]; + const ggml_tensor * silu = cgraph->nodes[node_idx+1]; + + if (ssm_conv->type != GGML_TYPE_F32 || silu->type != GGML_TYPE_F32) { + return false; + } + + return true; + } + + if (ops.size() == 2 && ops.begin()[0] == GGML_OP_UNARY && ops.begin()[1] == GGML_OP_MUL + && unary_ops.size() == 1 && (unary_ops.begin()[0] == GGML_UNARY_OP_SILU || unary_ops.begin()[0] == GGML_UNARY_OP_SIGMOID || unary_ops.begin()[0] == GGML_UNARY_OP_SOFTPLUS)) { + const ggml_tensor * unary = cgraph->nodes[node_idx]; + const ggml_tensor * mul = cgraph->nodes[node_idx+1]; + + if (ggml_get_unary_op(unary) != unary_ops.begin()[0]) { + return false; + } + + if (unary->type != GGML_TYPE_F32 && unary->type != GGML_TYPE_F16) { + return false; + } + + if (unary->type != mul->type) { + return false; + } + + const ggml_tensor * other = (mul->src[0] == unary) ? mul->src[1] : mul->src[0]; + if (other->type != unary->type) { + return false; + } + if (!ggml_is_contiguous_1(other) || !ggml_is_contiguous_1(unary->src[0]) || !ggml_are_same_shape(other, unary)) { + return false; + } + + return true; + } + + if (ops.size() == 3 && ops.begin()[0] == GGML_OP_SCALE && ops.begin()[1] == GGML_OP_UNARY && ops.begin()[2] == GGML_OP_SCALE + && unary_ops.size() == 1 && unary_ops.begin()[0] == GGML_UNARY_OP_TANH) { + const ggml_tensor *scale = cgraph->nodes[node_idx]; + const ggml_tensor *tanh = cgraph->nodes[node_idx+1]; + const ggml_tensor *scale2 = cgraph->nodes[node_idx+2]; + + GGML_ASSERT(scale->src[0]->type == GGML_TYPE_F32); + GGML_ASSERT(scale->type == GGML_TYPE_F32); + + if (ggml_get_unary_op(tanh) != GGML_UNARY_OP_TANH) { + return false; + } + + // Check for bias + if (ggml_get_op_params_f32(scale, 1) != 0.0f || ggml_get_op_params_f32(scale2, 1) != 0.0f) { + return false; + } + + return true; + } + + return false; +} + +static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph * cgraph, const bool use_cuda_graph, const bool cuda_graph_update_required, const void * graph_key) { + bool graph_evaluated_or_captured = false; + + // flag used to determine whether it is an integrated_gpu + const bool integrated = ggml_cuda_info().devices[cuda_ctx->device].integrated; + + ggml_cuda_stream_context & stream_ctx = cuda_ctx->stream_context(); + bool is_concurrent_event_active = false; + ggml_cuda_concurrent_event * concurrent_event = nullptr; + bool should_launch_concurrent_events = false; + + const auto try_launch_concurrent_event = [&](const ggml_tensor * node) { + if (stream_ctx.concurrent_events.find(node) != stream_ctx.concurrent_events.end()) { + concurrent_event = &stream_ctx.concurrent_events[node]; + + is_concurrent_event_active = true; + + GGML_LOG_DEBUG("Launching %d streams at %s\n", concurrent_event->n_streams, node->name); + + cudaStream_t main_stream = cuda_ctx->stream(); // this should be stream 0 + GGML_ASSERT(cuda_ctx->curr_stream_no == 0); + CUDA_CHECK(cudaEventRecord(concurrent_event->fork_event, main_stream)); + + for (int i = 1; i <= concurrent_event->n_streams; ++i) { + cudaStream_t stream = cuda_ctx->stream(cuda_ctx->device, i); + CUDA_CHECK(cudaStreamWaitEvent(stream, concurrent_event->fork_event)); + } + } + }; + + while (!graph_evaluated_or_captured) { + // Only perform the graph execution if CUDA graphs are not enabled, or we are capturing the graph. + // With the use of CUDA graphs, the execution will be performed by the graph launch. + if (!use_cuda_graph || cuda_graph_update_required) { + [[maybe_unused]] int prev_i = 0; + + if (stream_ctx.concurrent_events.size() > 0) { + should_launch_concurrent_events = true; + for (const auto & [tensor, event] : stream_ctx.concurrent_events) { + should_launch_concurrent_events = should_launch_concurrent_events && event.is_valid(); + } + } + + if (should_launch_concurrent_events) { + // Restore original node order within each concurrent region to enable fusion within streams + + std::unordered_map node_to_idx; + node_to_idx.reserve(cgraph->n_nodes); + for (int i = 0; i < cgraph->n_nodes; ++i) { + node_to_idx[cgraph->nodes[i]] = i; + } + + for (auto & [fork_node, event] : stream_ctx.concurrent_events) { + // Find positions of all nodes from this event in the current graph + std::vector positions; + positions.reserve(event.original_order.size()); + + bool all_found = true; + for (const ggml_tensor * orig_node : event.original_order) { + auto it = node_to_idx.find(orig_node); + if (it != node_to_idx.end()) { + positions.push_back(it->second); + } else { + all_found = false; + break; + } + } + + if (!all_found || positions.size() != event.original_order.size()) { + continue; + } + + // Sort positions to get contiguous range + std::vector sorted_positions = positions; + std::sort(sorted_positions.begin(), sorted_positions.end()); + + bool is_contiguous = true; + for (size_t i = 1; i < sorted_positions.size(); ++i) { + if (sorted_positions[i] != sorted_positions[i-1] + 1) { + is_contiguous = false; + break; + } + } + + if (!is_contiguous) { + continue; + } + + // Restore original order at the sorted positions + int start_pos = sorted_positions[0]; + for (size_t i = 0; i < event.original_order.size(); ++i) { + cgraph->nodes[start_pos + i] = const_cast(event.original_order[i]); + } + } + } else { + stream_ctx.concurrent_events.clear(); + } + + for (int i = 0; i < cgraph->n_nodes; i++) { + ggml_tensor * node = cgraph->nodes[i]; + if (is_concurrent_event_active) { + GGML_ASSERT(concurrent_event); + + if (node == concurrent_event->join_node) { + cuda_ctx->curr_stream_no = 0; + for (int i = 1; i <= concurrent_event->n_streams; ++i) { + // Wait on join events of forked streams in the main stream + CUDA_CHECK(cudaEventRecord(concurrent_event->join_events[i - 1], + cuda_ctx->stream(cuda_ctx->device, i))); + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), concurrent_event->join_events[i - 1])); + } + + is_concurrent_event_active = false; + concurrent_event = nullptr; + } else { + GGML_ASSERT (concurrent_event->stream_mapping.find(node) != concurrent_event->stream_mapping.end()); + cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; + GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); + } + } else if (i - prev_i > 1) { + //the previous node was fused + const ggml_tensor * prev_node = cgraph->nodes[i - 1]; + try_launch_concurrent_event(prev_node); + + if (is_concurrent_event_active) { + cuda_ctx->curr_stream_no = concurrent_event->stream_mapping[node]; + GGML_LOG_DEBUG("Setting stream no to %d for node %s\n", cuda_ctx->curr_stream_no, node->name); + } + } + +#ifdef GGML_CUDA_DEBUG + const int nodes_fused = i - prev_i - 1; + if (nodes_fused > 0) { + GGML_LOG_INFO("nodes_fused: %d\n", nodes_fused); + } +#endif + prev_i = i; + + if (ggml_is_empty(node) || node->op == GGML_OP_RESHAPE || node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE || node->op == GGML_OP_NONE) { + continue; + } + + if ((node->flags & GGML_TENSOR_FLAG_COMPUTE) == 0) { + continue; + } + + // start of fusion operations + static bool disable_fusion = (getenv("GGML_CUDA_DISABLE_FUSION") != nullptr); + if (!disable_fusion) { + ggml_cuda_topk_moe_args args; + + if (cgraph->nodes[i]->op == GGML_OP_UNARY || cgraph->nodes[i]->op == GGML_OP_SOFT_MAX || + cgraph->nodes[i]->op == GGML_OP_ARGSORT) { + const bool can_fuse = ggml_cuda_topk_moe_fusion(cgraph, i, args); + + std::vector ops; + + if (can_fuse) { + const ggml_tensor * logits = node->src[0]; + ggml_tensor * weights = nullptr; + ggml_tensor * ids = nullptr; + const ggml_tensor * bias = nullptr; + const ggml_tensor * clamp = nullptr; + const ggml_tensor * scale = nullptr; + + if (!args.delayed_softmax) { + ggml_op gating_op = args.sigmoid ? GGML_OP_UNARY : GGML_OP_SOFT_MAX; + int out_nodes[2]; // nodes which can't be elided + + if (args.prob_bias) { + bias = cgraph->nodes[i + 2]->src[1]; + ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, + GGML_OP_VIEW, GGML_OP_GET_ROWS }); + out_nodes[0] = i + 4; + ids = cgraph->nodes[i + 4]; + } else { + ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, + GGML_OP_GET_ROWS }); + out_nodes[0] = i + 3; + ids = cgraph->nodes[i + 3]; + } + + if (args.norm) { + ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_SUM_ROWS, GGML_OP_CLAMP, + GGML_OP_DIV, GGML_OP_RESHAPE }); + clamp = cgraph->nodes[i + ops.size() - 3]; + } + if (args.scale) { + ops.insert(ops.end(), { GGML_OP_SCALE }); + scale = cgraph->nodes[i + ops.size() - 1]; + } + + weights = cgraph->nodes[i + ops.size() - 1]; + out_nodes[1] = i + ops.size() - 1; + + if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && + ggml_cuda_should_use_topk_moe(node, logits, weights, ids) && + ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { + ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); + i += ops.size() - 1; + continue; + } + } else if (!args.norm && !args.prob_bias) { + //special case gpt-oss, no norm, no bias. + ops.insert(ops.end(), { GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS, + GGML_OP_RESHAPE, GGML_OP_SOFT_MAX, GGML_OP_RESHAPE }); + weights = cgraph->nodes[i + 5]; + ids = cgraph->nodes[i + 1]; + const ggml_tensor * softmax = cgraph->nodes[i + 4]; + + int out_nodes[2] = { i + 1, i + 5 }; + if (ggml_can_fuse_subgraph(cgraph, i, ops.size(), ops.data(), out_nodes, 2) && + ggml_cuda_should_use_topk_moe(softmax, logits, weights, ids) && + ggml_cuda_check_fusion_memory_ranges(cgraph, i, ops.size(), out_nodes, 2, /*is_topk_moe=*/ true)) { + ggml_cuda_op_topk_moe(*cuda_ctx, logits, weights, ids, clamp, scale, bias, args); + i += ops.size() - 1; + continue; + } + } + } + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_ROPE, GGML_OP_VIEW, GGML_OP_SET_ROWS }, {})) { + ggml_tensor * rope = cgraph->nodes[i]; + ggml_tensor * set_rows = cgraph->nodes[i + 2]; + + ggml_cuda_op_rope_fused(*cuda_ctx, rope, set_rows); + i += 2; + continue; + } + + if (node->op == GGML_OP_ADD || node->op == GGML_OP_MUL) { + int n_fuse = 0; + ggml_op ops[8]; + std::fill(ops, ops + 8, node->op); + + for (; n_fuse <= 6; ++n_fuse){ + if (!ggml_can_fuse(cgraph, i + n_fuse, ops + n_fuse, 2)) { + break; + } + if (cgraph->nodes[i + n_fuse] != cgraph->nodes[i + n_fuse + 1]->src[0]) { + break; + } + if (!ggml_are_same_layout(cgraph->nodes[i + n_fuse]->src[1], cgraph->nodes[i + n_fuse + 1]->src[1])) { + break; + } + } + + n_fuse++; + + if (n_fuse > 1) { + ggml_tensor fused_node; + memcpy(&fused_node, node, sizeof(ggml_tensor)); + for (int j = 0; j < n_fuse - 1; ++j) { + fused_node.src[j + 2] = cgraph->nodes[i + j + 1]->src[1]; + } + fused_node.data = cgraph->nodes[i + n_fuse - 1]->data; + if (node->op == GGML_OP_ADD) { + ggml_cuda_op_fused_add(*cuda_ctx, &fused_node, n_fuse); + } else { + ggml_cuda_op_fused_mul(*cuda_ctx, &fused_node, n_fuse); + } + i += n_fuse - 1; + + continue; + } + } + + bool fused_mul_mat_vec = false; + int fused_node_count = 0; + + for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { + const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (ggml_cuda_can_fuse(cgraph, i, { op, bias_op, op, bias_op, GGML_OP_GLU }, {})) { + ggml_tensor * glu = cgraph->nodes[i + 4]; + ggml_tensor * gate_bias_n = glu->src[0]; + ggml_tensor * up_bias_n = glu->src[1]; + + //we don't assume the order for {gate, up}. Instead infer it from the bias tensor + ggml_tensor * gate_n = nullptr; + ggml_tensor * up_n = nullptr; + + if (gate_bias_n->src[0] == cgraph->nodes[i] || gate_bias_n->src[1] == cgraph->nodes[i]) { + gate_n = cgraph->nodes[i]; + up_n = cgraph->nodes[i + 2]; + } else if (gate_bias_n->src[0] == cgraph->nodes[i + 2] || gate_bias_n->src[1] == cgraph->nodes[i + 2]) { + gate_n = cgraph->nodes[i + 2]; + up_n = cgraph->nodes[i]; + } else { + continue; + } + + auto get_bias_tensor = [](const ggml_tensor * bias_node, const ggml_tensor * mul_node, ggml_op op_bias) { + if (op_bias == GGML_OP_ADD) { + if (bias_node->src[0] == mul_node) { + return bias_node->src[1]; + } + if (bias_node->src[1] == mul_node) { + return bias_node->src[0]; + } + return (ggml_tensor *) nullptr; + } + GGML_ASSERT(op_bias == GGML_OP_ADD_ID); + GGML_ASSERT(bias_node->src[0] == mul_node); + return bias_node->src[1]; + }; + + ggml_tensor * up_bias_tensor = get_bias_tensor(up_bias_n, up_n, bias_op); + ggml_tensor * gate_bias_tensor = get_bias_tensor(gate_bias_n, gate_n, bias_op); + + if (!up_bias_tensor || !gate_bias_tensor) { + continue; + } + + // we don't support repeating adds + if (bias_op == GGML_OP_ADD && + (!ggml_are_same_shape(gate_bias_n->src[0], gate_bias_n->src[1]) || + !ggml_are_same_shape(up_bias_n->src[0], up_bias_n->src[1]))) { + continue; + } + + const ggml_tensor * src0 = up_n->src[0]; + const ggml_tensor * src1 = up_n->src[1]; + const ggml_tensor * ids = up_n->src[2]; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(up_n)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate_n->src[0]; + fusion_data.x_bias = up_bias_tensor; + fusion_data.gate_bias = gate_bias_tensor; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 5; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(up_n)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate_n->src[0]; + fusion_data.x_bias = up_bias_tensor; + fusion_data.gate_bias = gate_bias_tensor; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 5; + break; + } + } else if (ggml_cuda_can_fuse(cgraph, i, { op, op, GGML_OP_GLU }, {})) { + ggml_tensor * glu = cgraph->nodes[i + 2]; + ggml_tensor * gate = glu->src[0]; + ggml_tensor * up = glu->src[1]; + + bool ok = (gate == cgraph->nodes[i] && up == cgraph->nodes[i + 1]) + || (gate == cgraph->nodes[i + 1] && up == cgraph->nodes[i]); + + if (!ok) continue; + + const ggml_tensor * src0 = up->src[0]; + const ggml_tensor * src1 = up->src[1]; + const ggml_tensor * ids = up->src[2]; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(up)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate->src[0]; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 3; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(up)) { + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.gate = gate->src[0]; + fusion_data.glu_op = ggml_get_glu_op(glu); + + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, glu, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 3; + break; + } + } + } + + if (fused_mul_mat_vec) { + i += fused_node_count - 1; + continue; + } + + fused_mul_mat_vec = false; + fused_node_count = 0; + + for (ggml_op op : { GGML_OP_MUL_MAT, GGML_OP_MUL_MAT_ID }) { + const ggml_op bias_op = op == GGML_OP_MUL_MAT ? GGML_OP_ADD : GGML_OP_ADD_ID; + + if (!ggml_can_fuse(cgraph, i, { op, bias_op })) { + continue; + } + + ggml_tensor * mm_node = cgraph->nodes[i]; + ggml_tensor * bias_node = cgraph->nodes[i + 1]; + + ggml_tensor * bias_tensor = nullptr; + if (bias_op == GGML_OP_ADD) { + if (bias_node->src[0] == mm_node) { + bias_tensor = bias_node->src[1]; + } else if (bias_node->src[1] == mm_node) { + bias_tensor = bias_node->src[0]; + } else { + continue; + } + } else { + if (bias_node->src[0] != mm_node) { + continue; + } + bias_tensor = bias_node->src[1]; + } + + const ggml_tensor * src0 = mm_node->src[0]; + const ggml_tensor * src1 = mm_node->src[1]; + const ggml_tensor * ids = mm_node->src[2]; + + if (bias_op == GGML_OP_ADD_ID && bias_node->src[2] != ids) { + continue; + } + + if (bias_op == GGML_OP_ADD && !ggml_are_same_shape(bias_node->src[0], bias_node->src[1])) { + continue; + } + + ggml_cuda_mm_fusion_args_host fusion_data{}; + fusion_data.x_bias = bias_tensor; + + if (ggml_cuda_should_fuse_mul_mat_vec_f(mm_node)) { + ggml_cuda_mul_mat_vec_f(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 2; + break; + } + + if (ggml_cuda_should_fuse_mul_mat_vec_q(mm_node)) { + ggml_cuda_mul_mat_vec_q(*cuda_ctx, src0, src1, ids, bias_node, &fusion_data); + fused_mul_mat_vec = true; + fused_node_count = 2; + break; + } + } + + if (fused_mul_mat_vec) { + i += fused_node_count - 1; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL, GGML_OP_ADD}, {})) { + ggml_cuda_op_rms_norm_fused_add(*cuda_ctx, node, cgraph->nodes[i+1], cgraph->nodes[i+2]); + i += 2; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL}, {})) { + ggml_cuda_op_rms_norm_fused(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SSM_CONV, GGML_OP_UNARY }, { GGML_UNARY_OP_SILU })) { + ggml_cuda_op_ssm_conv(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SILU }) || + ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SIGMOID }) || + ggml_cuda_can_fuse(cgraph, i, { GGML_OP_UNARY, GGML_OP_MUL }, { GGML_UNARY_OP_SOFTPLUS })) { + ggml_cuda_op_unary_mul(*cuda_ctx, node, cgraph->nodes[i+1]); + i++; + continue; + } + + if (ggml_cuda_can_fuse(cgraph, i, { GGML_OP_SCALE, GGML_OP_UNARY, GGML_OP_SCALE }, { GGML_UNARY_OP_TANH })) { + i += 2; + ggml_cuda_op_softcap(*cuda_ctx, cgraph->nodes[i], node); + continue; + } + } +#ifndef NDEBUG + assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device)); + for (int j = 0; j < GGML_MAX_SRC; j++) { + if (node->src[j] != nullptr) { + assert(node->src[j]->buffer); + assert(node->src[j]->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) || + ggml_backend_buft_is_cuda_split(node->src[j]->buffer->buft) || (integrated && ggml_backend_buft_is_cuda_host(node->src[j]->buffer->buft))); + } + } +#else + GGML_UNUSED(integrated); +#endif // NDEBUG + + bool ok = ggml_cuda_compute_forward(*cuda_ctx, node); + if (!ok) { + GGML_LOG_ERROR("%s: op not supported %s (%s)\n", __func__, node->name, ggml_op_name(node->op)); + } + GGML_ASSERT(ok); + + if (!is_concurrent_event_active) { + try_launch_concurrent_event(node); + } + } + } + +#ifdef USE_CUDA_GRAPH + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (use_cuda_graph && cuda_graph_update_required) { // End CUDA graph capture + if (graph->graph != nullptr) { + CUDA_CHECK(cudaGraphDestroy(graph->graph)); + graph->graph = nullptr; + } + + CUDA_CHECK(cudaStreamEndCapture(cuda_ctx->stream(), &graph->graph)); + graph_evaluated_or_captured = true; // CUDA graph has been captured + + std::lock_guard lock(ggml_cuda_lock); + if (ggml_cuda_lock_counter.fetch_sub(1, std::memory_order_relaxed) == 1) { + ggml_cuda_lock_cv.notify_all(); + } + } else { + graph_evaluated_or_captured = true; // ggml graph has been directly evaluated + } + } + + if (use_cuda_graph) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (graph->instance == nullptr) { // Create executable graph from captured graph. + CUDA_CHECK(cudaGraphInstantiate(&graph->instance, graph->graph, NULL, NULL, 0)); + } + if (cuda_graph_update_required) { // Update graph executable + ggml_cuda_graph_update_executable(cuda_ctx, graph_key); + } + // Launch graph + CUDA_CHECK(cudaGraphLaunch(graph->instance, cuda_ctx->stream())); +#else + GGML_UNUSED(graph_key); + graph_evaluated_or_captured = true; +#endif // USE_CUDA_GRAPH + } +} + +#ifdef USE_CUDA_GRAPH +static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, const void * graph_key) { + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + + if (graph->graph == nullptr) { + if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_AMPERE) { + if (!graph->disable_due_to_gpu_arch) { + GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__); + } + graph->disable_due_to_gpu_arch = true; + } + } + + return graph->is_enabled(); +} +#endif // USE_CUDA_GRAPH + +static enum ggml_status ggml_backend_cuda_graph_compute(ggml_backend_t backend, ggml_cgraph * cgraph) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + + ggml_cuda_set_device(cuda_ctx->device); + + bool use_cuda_graph = false; + bool cuda_graph_update_required = false; + const void * graph_key = nullptr; + +#ifdef USE_CUDA_GRAPH + graph_key = ggml_cuda_graph_get_key(cgraph); + + ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); + + ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key); + if (graph->is_enabled()) { + const bool graph_compatible = ggml_cuda_graph_check_compability(cgraph); + if (graph_compatible) { + const bool properties_changed = ggml_cuda_graph_update_required(cuda_ctx, cgraph); + + if (!graph->warmup_complete) { + // Warmup: need at least 2 calls with no property change on the 2nd call + if (!properties_changed) { + graph->warmup_complete = true; + GGML_LOG_DEBUG("%s: CUDA graph warmup complete\n", __func__); + use_cuda_graph = true; + cuda_graph_update_required = true; + } + // else: properties changed or first call - execute directly (use_cuda_graph stays false) + } else { + // Post-warmup: normal CUDA graph operation + if (properties_changed) { + // Properties changed - reset warmup, execute directly until stable again + graph->warmup_complete = false; + GGML_LOG_DEBUG("%s: CUDA graph warmup reset\n", __func__); + } else { + use_cuda_graph = true; + cuda_graph_update_required = graph->instance == nullptr; + } + } + } + } +#endif // USE_CUDA_GRAPH + + if (use_cuda_graph && cuda_graph_update_required) { + // Start CUDA graph capture + { + std::lock_guard lock(ggml_cuda_lock); + ggml_cuda_lock_counter.fetch_add(1, std::memory_order_relaxed); + } + + CUDA_CHECK(cudaStreamBeginCapture(cuda_ctx->stream(), cudaStreamCaptureModeRelaxed)); + } + + ggml_cuda_graph_evaluate_and_capture(cuda_ctx, cgraph, use_cuda_graph, cuda_graph_update_required, graph_key); + + return GGML_STATUS_SUCCESS; +} + +static void ggml_backend_cuda_event_record(ggml_backend_t backend, ggml_backend_event_t event) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + CUDA_CHECK(cudaEventRecord((cudaEvent_t)event->context, cuda_ctx->stream())); +} + +static void ggml_backend_cuda_event_wait(ggml_backend_t backend, ggml_backend_event_t event) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *)backend->context; + + if (ggml_backend_is_cuda(backend)) { + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), (cudaEvent_t)event->context, 0)); + } else { +#if 0 + // untested + auto wait_fn = [](void * user_data) { + ggml_backend_event_t event = (ggml_backend_event_t)user_data; + ggml_backend_event_synchronize(event); + }; + + CUDA_CHECK(cudaLaunchHostFunc(cuda_ctx->stream(), wait_fn, event)); +#endif + GGML_ABORT("fatal error"); + } +} + +static void ggml_backend_cuda_graph_optimize(ggml_backend_t backend, ggml_cgraph * cgraph) { + ggml_backend_cuda_context * cuda_ctx = (ggml_backend_cuda_context *) backend->context; + +#ifdef USE_CUDA_GRAPH + const void * graph_key = ggml_cuda_graph_get_key(cgraph); + const bool use_cuda_graph = ggml_cuda_graph_set_enabled(cuda_ctx, graph_key); +#else + const bool use_cuda_graph = false; + GGML_UNUSED(cuda_ctx); + GGML_UNUSED(cgraph); +#endif + + static bool enable_graph_optimization = [] { + const char * env = getenv("GGML_CUDA_GRAPH_OPT"); + return env != nullptr && atoi(env) == 1; + }(); + + if (!enable_graph_optimization) { + return; + } + + ggml_cuda_stream_context & stream_context = cuda_ctx->stream_context(); + stream_context.reset(); + + if (!use_cuda_graph || ggml_backend_cuda_get_device_count() != 1) { + return; + } + + // number of out-degrees for a particular node + std::unordered_map fan_out; + // reverse mapping of node to index in the cgraph + std::unordered_map node_indices; + + const auto & is_noop = [](const ggml_tensor * node) -> bool { + return ggml_is_empty(node) || node->op == GGML_OP_NONE || node->op == GGML_OP_RESHAPE || + node->op == GGML_OP_TRANSPOSE || node->op == GGML_OP_VIEW || node->op == GGML_OP_PERMUTE; + }; + + const auto & depends_on = [](const ggml_tensor * dst, const ggml_tensor * src) -> bool { + for (uint32_t s = 0; s < GGML_MAX_SRC; ++s) { + if (dst->src[s] == src) { + return true; + } + } + // implicit dependency if they view the same tensor + const ggml_tensor * dst2 = dst->view_src ? dst->view_src : dst; + const ggml_tensor * src2 = src->view_src ? src->view_src : src; + if (dst2 == src2) { + return true; + } + return false; + }; + + for (int node_idx = 0; node_idx < cgraph->n_nodes; node_idx++) { + const ggml_tensor * node = cgraph->nodes[node_idx]; + node_indices[node] = node_idx; + + if (is_noop(node)) { + continue; + } + for (int src_idx = 0; src_idx < GGML_MAX_SRC; ++src_idx) { + const ggml_tensor * src = cgraph->nodes[node_idx]->src[src_idx]; + //TODO: check why nrows > 1 fails + if (node && !is_noop(node) && ggml_nrows(node) <= 1) { + fan_out[src] += 1; + } + } + } + + // Target Q, K, V for concurrency + // this is a more general way to find nodes which can be candidates for concurrency (although it has not been tested for anything else): + // 1. find fan-out (fork) nodes where the same input is used at least N times (in QKV, it would be "attn-norm") + // 2. find the join node, where 2 or more of the outputs are required (in QKV, this would "KQ" or "flash-attn") + // 3. account for all branches from the fork to the join + // 4. To extend lifetimes of the tensors, we interleave the branches (see below for more details) + // 5. save the original cgraph and restore it in graph_compute, to enable fusion within streams + // See discussion: https://github.com/ggml-org/llama.cpp/pull/16991#issuecomment-3522620030 + + const int min_fan_out = 3; + const int max_fan_out = 3; + + // store {fork_idx, join_idx} + std::vector> concurrent_node_ranges; + + for (const auto & [root_node, count] : fan_out) { + if (count >= min_fan_out && count <= max_fan_out) { + const int root_node_idx = node_indices[root_node]; + + // only optimize for attn_norm + // TODO: make this more generic + if (!strstr(root_node->name, "attn_norm")) { + continue; + } + + bool is_part_of_event = false; + for (const auto & [start, end] : concurrent_node_ranges) { + if (root_node_idx >= start && root_node_idx <= end) { + is_part_of_event = true; + } + } + + if (is_part_of_event) { + continue; + } + + std::vector> nodes_per_branch; + for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { + const ggml_tensor * node = cgraph->nodes[i]; + if (!is_noop(node) && depends_on(node, root_node)) { + nodes_per_branch.push_back({ node }); + } + } + + GGML_ASSERT(nodes_per_branch.size() == (size_t) count); + + //find the join point + const ggml_tensor * join_node = nullptr; + + const auto & belongs_to_branch = [&](const ggml_tensor * node, + const std::vector & branch) -> bool { + for (const ggml_tensor * n : branch) { + if (depends_on(node, n)) { + return true; + } + } + return false; + }; + + for (int i = root_node_idx + 1; i < cgraph->n_nodes; ++i) { + const ggml_tensor * curr_node = cgraph->nodes[i]; + + int num_joins = 0; + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + if (belongs_to_branch(curr_node, nodes_per_branch[branch_idx])) { + num_joins++; + } + } + + if (num_joins >= 2) { + join_node = curr_node; + break; + } + + bool found_branch = false; + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + std::vector & branch_vec = nodes_per_branch[branch_idx]; + if (belongs_to_branch(curr_node, branch_vec)) { + //continue accumulating + if (std::find(branch_vec.begin(), branch_vec.end(), curr_node) == branch_vec.end()) { + branch_vec.push_back(curr_node); + } + found_branch = true; + } + } + + if (!found_branch && is_noop(curr_node)) { + // we can put it in any branch because it will be ignored + nodes_per_branch[0].push_back({ curr_node }); + } + } + + if (join_node) { + //Create ggml_cuda_concurrent_event + ggml_cuda_concurrent_event concurrent_event(nodes_per_branch.size()); + concurrent_event.join_node = join_node; + + for (size_t branch_idx = 0; branch_idx < nodes_per_branch.size(); branch_idx++) { + for (const ggml_tensor * n : nodes_per_branch[branch_idx]) { + concurrent_event.stream_mapping[n] = branch_idx + 1; + } + } + + int fork_node_idx = node_indices[root_node]; + int join_node_idx = node_indices[join_node]; + + int current_branch_idx = 0; + int current_node_idx = fork_node_idx + 1; + const int n_branches = nodes_per_branch.size(); + + int total_branch_nodes = 0; + for (std::vector branch_nodes : nodes_per_branch) { + total_branch_nodes += branch_nodes.size(); + } + + // there are other nodes in the middle which are unaccounted for + // usually (cpy) nodes, then ignore this fork + if (join_node_idx - fork_node_idx - 1 != total_branch_nodes) { + GGML_LOG_DEBUG( + "Skipping %s because the number of nodes in the middle is not equal to the total number of " + "branch nodes %d != %d\n", + root_node->name, join_node_idx - fork_node_idx - 1, total_branch_nodes); + continue; + } + + // Save the original order of nodes in this region before interleaving + // This is used later to restore grouping for fusion within streams + concurrent_event.original_order.reserve(total_branch_nodes); + for (int i = fork_node_idx + 1; i < join_node_idx; ++i) { + concurrent_event.original_order.push_back(cgraph->nodes[i]); + } + + std::unordered_map & concurrent_events = cuda_ctx->stream_context().concurrent_events; + GGML_ASSERT(concurrent_events.find(root_node) == concurrent_events.end()); + concurrent_events.emplace(root_node, std::move(concurrent_event)); + GGML_LOG_DEBUG("Adding stream at node %s %p\n", root_node->name, root_node); + concurrent_node_ranges.emplace_back(fork_node_idx, join_node_idx); + + // interleave tensors to extend lifetimes so that ggml graph doesn't recycle them + // example transformation: + // [attn-norm, QMul, QNorm, QRope, KMul, KNorm, KRope, VMul, attn] -> + // [attn-norm, QMul, KMul, VMul, QNorm, VNorm, QRope, KRope, attn] + while (current_node_idx < join_node_idx) { + std::vector & branch_nodes = nodes_per_branch[current_branch_idx]; + + bool has_node = false; + for (std::vector branch_node : nodes_per_branch) { + has_node |= branch_node.size() > 0; + } + + GGML_ASSERT(has_node); + + if (branch_nodes.empty()) { + current_branch_idx = (current_branch_idx + 1) % n_branches; + continue; + } + + cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); + current_node_idx++; + branch_nodes.erase(branch_nodes.begin()); + + // append all empty nodes + while (!branch_nodes.empty() && is_noop(branch_nodes.front())) { + cgraph->nodes[current_node_idx] = const_cast(branch_nodes.front()); + current_node_idx++; + branch_nodes.erase(branch_nodes.begin()); + } + + current_branch_idx = (current_branch_idx + 1) % n_branches; + } + } + } + } +} + +static const ggml_backend_i ggml_backend_cuda_interface = { + /* .get_name = */ ggml_backend_cuda_get_name, + /* .free = */ ggml_backend_cuda_free, + /* .set_tensor_async = */ ggml_backend_cuda_set_tensor_async, + /* .get_tensor_async = */ ggml_backend_cuda_get_tensor_async, + /* .get_tensor_2d_async = */ ggml_backend_cuda_set_tensor_2d_async, + /* .set_tensor_2d_async = */ ggml_backend_cuda_get_tensor_2d_async, + /* .cpy_tensor_async = */ ggml_backend_cuda_cpy_tensor_async, + /* .synchronize = */ ggml_backend_cuda_synchronize, + /* .graph_plan_create = */ NULL, + /* .graph_plan_free = */ NULL, + /* .graph_plan_update = */ NULL, + /* .graph_plan_compute = */ NULL, + /* .graph_compute = */ ggml_backend_cuda_graph_compute, + /* .event_record = */ ggml_backend_cuda_event_record, + /* .event_wait = */ ggml_backend_cuda_event_wait, + /* .graph_optimize = */ ggml_backend_cuda_graph_optimize, +}; + +static ggml_guid_t ggml_backend_cuda_guid() { + static ggml_guid guid = { 0x2c, 0xdd, 0xe8, 0x1c, 0x65, 0xb3, 0x65, 0x73, 0x6a, 0x12, 0x88, 0x61, 0x1c, 0xc9, 0xdc, 0x25 }; + return &guid; +} + +bool ggml_backend_is_cuda(ggml_backend_t backend) { + return backend != NULL && ggml_guid_matches(backend->guid, ggml_backend_cuda_guid()); +} + +int ggml_backend_cuda_get_device_count() { + return ggml_cuda_info().device_count; +} + +void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) { + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, device)); + snprintf(description, description_size, "%s", prop.name); +} + +void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) { + ggml_cuda_set_device(device); + + CUDA_CHECK(cudaMemGetInfo(free, total)); +} + +bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) { + if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { + return false; + } + +#if CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) || defined(GGML_USE_HIP) + cudaError_t err = cudaHostRegister(buffer, size, cudaHostRegisterPortable | cudaHostRegisterReadOnly); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + + GGML_LOG_DEBUG("%s: failed to register %.2f MiB of pinned memory: %s\n", __func__, + size / 1024.0 / 1024.0, cudaGetErrorString(err)); + return false; + } + return true; +#else + GGML_UNUSED(buffer); + GGML_UNUSED(size); + return false; +#endif // CUDART_VERSION >= 11010 || defined(GGML_USE_MUSA) +} + +void ggml_backend_cuda_unregister_host_buffer(void * buffer) { + if (getenv("GGML_CUDA_REGISTER_HOST") == nullptr) { + return; + } + + cudaError_t err = cudaHostUnregister(buffer); + if (err != cudaSuccess) { + // clear the error + (void)cudaGetLastError(); + } +} + + +// backend device + +struct ggml_backend_cuda_device_context { + int device; + std::string name; + std::string description; + std::string pci_bus_id; + int op_offload_min_batch_size; +}; + +static const char * ggml_backend_cuda_device_get_name(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ctx->name.c_str(); +} + +static const char * ggml_backend_cuda_device_get_description(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ctx->description.c_str(); +} + +#if defined(__linux__) +// Helper function to get available memory from /proc/meminfo for UMA systems +static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_kb, long * free_swap_kb) { + FILE * meminfo_file = nullptr; + // 2KB buffer for reading /proc/meminfo since it does not report size info, should be enough + const size_t BUFFER_SIZE = 2048; + auto file_buffer = std::make_unique(BUFFER_SIZE); + size_t bytes_read = 0; + long huge_tlb_total_pages = -1; + long huge_tlb_free_pages = -1; + long huge_tlb_page_size = -1; + + if (available_memory_kb == nullptr || free_swap_kb == nullptr) { + return false; + } + + meminfo_file = fopen("/proc/meminfo", "r"); + if (meminfo_file == nullptr) { + GGML_LOG_ERROR("%s: failed to open /proc/meminfo\n", __func__); + return false; + } + + // Read file into buffer + bytes_read = fread(file_buffer.get(), 1, BUFFER_SIZE - 1, meminfo_file); + fclose(meminfo_file); + + if (bytes_read == 0) { + GGML_LOG_ERROR("%s: failed to read from /proc/meminfo\n", __func__); + return false; + } + file_buffer[bytes_read] = '\0'; + + *available_memory_kb = -1; + *free_swap_kb = -1; + + // Parse the file buffer line by line + char * line = file_buffer.get(); + char * line_next; + while (line < file_buffer.get() + bytes_read) { + // Find the end of the current line + line_next = strchr(line, '\n'); + if (line_next != nullptr) { + *line_next = '\0'; + line_next++; + } else { + line_next = file_buffer.get() + bytes_read; + } + + long value; + if (sscanf(line, "MemAvailable: %ld kB", &value) == 1) { + *available_memory_kb = value; + } else if (sscanf(line, "SwapFree: %ld kB", &value) == 1) { + *free_swap_kb = value; + } else if (sscanf(line, "HugePages_Total: %ld", &value) == 1) { + huge_tlb_total_pages = value; + } else if (sscanf(line, "HugePages_Free: %ld", &value) == 1) { + huge_tlb_free_pages = value; + } else if (sscanf(line, "Hugepagesize: %ld kB", &value) == 1) { + huge_tlb_page_size = value; + } + + line = line_next; + } + + if (huge_tlb_total_pages != 0 && huge_tlb_total_pages != -1) { + *available_memory_kb = huge_tlb_free_pages * huge_tlb_page_size; + + // Hugetlbfs pages are not swappable. + *free_swap_kb = 0; + } + + GGML_LOG_DEBUG("%s: final available_memory_kb: %ld\n", __func__, *available_memory_kb); + return true; +} +#endif // defined(__linux__) + +static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + ggml_cuda_set_device(ctx->device); + CUDA_CHECK(cudaMemGetInfo(free, total)); + +// ref: https://github.com/ggml-org/llama.cpp/pull/17368 +#if defined(__linux__) + // Check if this is a UMA (Unified Memory Architecture) system + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device)); + + // Check if UMA is explicitly enabled via environment variable + bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr; + bool is_uma = prop.integrated > 0 || uma_env; + + if (is_uma) { + // For UMA systems (like DGX Spark), use system memory info + long available_memory_kb = 0; + long free_swap_kb = 0; + + if (ggml_backend_cuda_get_available_uma_memory(&available_memory_kb, &free_swap_kb) && available_memory_kb > 0) { + *free = (size_t)available_memory_kb * 1024; + } else { + GGML_LOG_ERROR("%s: /proc/meminfo reading failed, using cudaMemGetInfo\n", __func__); + } + } +#endif // defined(__linux__) + +} + +static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) { + GGML_UNUSED(dev); + return GGML_BACKEND_DEVICE_TYPE_GPU; +} + +static void ggml_backend_cuda_device_get_props(ggml_backend_dev_t dev, ggml_backend_dev_props * props) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + + props->name = ggml_backend_cuda_device_get_name(dev); + props->description = ggml_backend_cuda_device_get_description(dev); + props->type = ggml_backend_cuda_device_get_type(dev); + props->device_id = ctx->pci_bus_id.empty() ? nullptr : ctx->pci_bus_id.c_str(); + ggml_backend_cuda_device_get_memory(dev, &props->memory_free, &props->memory_total); + + bool host_buffer = getenv("GGML_CUDA_NO_PINNED") == nullptr; +#ifdef GGML_CUDA_NO_PEER_COPY + bool events = false; +#else + bool events = true; +#endif + + props->caps = { + /* .async = */ true, + /* .host_buffer = */ host_buffer, + /* .buffer_from_host_ptr = */ false, + /* .events = */ events, + }; +} + +static ggml_backend_t ggml_backend_cuda_device_init_backend(ggml_backend_dev_t dev, const char * params) { + GGML_UNUSED(params); + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ggml_backend_cuda_init(ctx->device); +} + +static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_buffer_type(ggml_backend_dev_t dev) { + ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context; + return ggml_backend_cuda_buffer_type(ctx->device); +} + +static ggml_backend_buffer_type_t ggml_backend_cuda_device_get_host_buffer_type(ggml_backend_dev_t dev) { + GGML_UNUSED(dev); + return ggml_backend_cuda_host_buffer_type(); +} + +// TODO: move these functions here +static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const ggml_tensor * op) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + + // split buffers can only be used with GGML_OP_MUL_MAT + if (op->op != GGML_OP_MUL_MAT) { + for (int i = 0; i < GGML_MAX_SRC; i++) { + if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda_split(op->src[i]->buffer->buft)) { + return false; + } + } + } + + // check if all the sources are allocated on this device + for (int i = 0; i < GGML_MAX_SRC; i++) { + if (op->src[i] && op->src[i]->buffer && ggml_backend_buft_is_cuda(op->src[i]->buffer->buft)) { + ggml_backend_cuda_buffer_type_context * buft_ctx = (ggml_backend_cuda_buffer_type_context *)op->src[i]->buffer->buft->context; + if (buft_ctx->device != dev_ctx->device) { + return false; + } + } + } + + switch (op->op) { + case GGML_OP_UNARY: + switch (ggml_get_unary_op(op)) { + case GGML_UNARY_OP_ABS: + case GGML_UNARY_OP_SGN: + case GGML_UNARY_OP_NEG: + case GGML_UNARY_OP_STEP: + case GGML_UNARY_OP_GELU: + case GGML_UNARY_OP_SILU: + case GGML_UNARY_OP_RELU: + case GGML_UNARY_OP_SIGMOID: + case GGML_UNARY_OP_HARDSIGMOID: + case GGML_UNARY_OP_HARDSWISH: + case GGML_UNARY_OP_GELU_ERF: + case GGML_UNARY_OP_GELU_QUICK: + case GGML_UNARY_OP_TANH: + case GGML_UNARY_OP_EXP: + case GGML_UNARY_OP_EXPM1: + case GGML_UNARY_OP_SOFTPLUS: + case GGML_UNARY_OP_ELU: + case GGML_UNARY_OP_XIELU: + case GGML_UNARY_OP_FLOOR: + case GGML_UNARY_OP_CEIL: + case GGML_UNARY_OP_ROUND: + case GGML_UNARY_OP_TRUNC: + // TODO: should become: + //return ggml_is_contiguous_rows(op->src[0]); + return ggml_is_contiguous(op->src[0]); + default: + return false; + } + break; + case GGML_OP_GLU: + switch (ggml_get_glu_op(op)) { + case GGML_GLU_OP_REGLU: + case GGML_GLU_OP_GEGLU: + case GGML_GLU_OP_SWIGLU: + case GGML_GLU_OP_SWIGLU_OAI: + case GGML_GLU_OP_GEGLU_ERF: + case GGML_GLU_OP_GEGLU_QUICK: + return ggml_is_contiguous_1(op->src[0]); + default: + return false; + } + break; + case GGML_OP_MUL_MAT: + case GGML_OP_MUL_MAT_ID: + { + struct ggml_tensor * a = op->src[0]; + struct ggml_tensor * b = op->src[1]; + if (a->buffer && ggml_backend_buft_is_cuda_split(a->buffer->buft)) { + if (a->ne[2] > 1 || a->ne[3] > 1) { + return false; + } + // for small weight matrices the active device can end up without any rows, don't use row split in those cases + // this avoids some edge cases (and the performance would not be good anyways) + ggml_backend_cuda_split_buffer_type_context * buft_ctx = (ggml_backend_cuda_split_buffer_type_context *) a->buffer->buft->context; + int64_t row_low; + int64_t row_high; + get_row_split(&row_low, &row_high, a, buft_ctx->tensor_split, dev_ctx->device); + if (row_low == row_high) { + return false; + } + } + if (b->type == GGML_TYPE_F16 && a->type != GGML_TYPE_F16) { + return false; + } +#ifdef GGML_USE_MUSA + const int cc = ggml_cuda_info().devices[dev_ctx->device].cc; + if (b->ne[2]*b->ne[3] > 1 && !ggml_is_transposed(a) && !ggml_is_transposed(b)) { + if (GGML_CUDA_CC_IS_QY1(cc) && op->op == GGML_OP_MUL_MAT && + a->type == GGML_TYPE_F16 && b->type == GGML_TYPE_F16) { + return false; + } + if (GGML_CUDA_CC_IS_QY2(cc) && op->op == GGML_OP_MUL_MAT_ID && + a->type == GGML_TYPE_Q2_K && b->type == GGML_TYPE_F32) { + return false; + } + } +#endif // GGML_USE_MUSA + switch (a->type) { + case GGML_TYPE_F32: + case GGML_TYPE_F16: + case GGML_TYPE_Q1_0: + case GGML_TYPE_Q4_0: + case GGML_TYPE_Q4_1: + case GGML_TYPE_Q5_0: + case GGML_TYPE_Q5_1: + case GGML_TYPE_Q8_0: + case GGML_TYPE_MXFP4: + case GGML_TYPE_NVFP4: + case GGML_TYPE_Q2_K: + case GGML_TYPE_Q3_K: + case GGML_TYPE_Q4_K: + case GGML_TYPE_Q5_K: + case GGML_TYPE_Q6_K: + case GGML_TYPE_Q8_K: + case GGML_TYPE_IQ1_M: + case GGML_TYPE_IQ1_S: + case GGML_TYPE_IQ2_S: + case GGML_TYPE_IQ2_XS: + case GGML_TYPE_IQ2_XXS: + case GGML_TYPE_IQ3_S: + case GGML_TYPE_IQ3_XXS: + case GGML_TYPE_IQ4_NL: + case GGML_TYPE_IQ4_XS: + case GGML_TYPE_BF16: + return true; + default: + return false; + } + } break; + case GGML_OP_OUT_PROD: + return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32; + case GGML_OP_GET_ROWS: + { + switch (op->src[0]->type) { + case GGML_TYPE_F16: + case GGML_TYPE_F32: + case GGML_TYPE_BF16: + case GGML_TYPE_I32: + case GGML_TYPE_Q1_0: + case GGML_TYPE_Q4_0: + case GGML_TYPE_Q4_1: + case GGML_TYPE_Q5_0: + case GGML_TYPE_Q5_1: + case GGML_TYPE_Q8_0: + return true; + default: + return false; + } + } break; + case GGML_OP_GET_ROWS_BACK: + { + return op->type == GGML_TYPE_F32 && op->src[0]->type == GGML_TYPE_F32 && op->ne[2] == 1 && op->ne[3] == 1; + } break; + case GGML_OP_SET_ROWS: + { + return (op->type == GGML_TYPE_F32 || op->type == GGML_TYPE_F16 || op->type == GGML_TYPE_BF16 || + op->type == GGML_TYPE_Q4_0 || op->type == GGML_TYPE_Q4_1 || op->type == GGML_TYPE_Q5_0 || + op->type == GGML_TYPE_Q5_1 || op->type == GGML_TYPE_Q8_0 || op->type == GGML_TYPE_IQ4_NL) && + op->src[0]->type == GGML_TYPE_F32 && + (op->src[1]->type == GGML_TYPE_I64 || op->src[1]->type == GGML_TYPE_I32); + } break; + case GGML_OP_SET: + { + const ggml_type t = op->type; + return (t == GGML_TYPE_F32 || t == GGML_TYPE_I32) && + t == op->src[0]->type && + t == op->src[1]->type; + } break; + case GGML_OP_CPY: + { + ggml_type src0_type = op->src[0]->type; + ggml_type src1_type = op->src[1]->type; + if ((src0_type == GGML_TYPE_F32 || src0_type == GGML_TYPE_BF16 || src0_type == GGML_TYPE_F16) && + (src1_type == GGML_TYPE_F32 || src1_type == GGML_TYPE_BF16 || src1_type == GGML_TYPE_F16) + ) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q8_0) { + return true; + } + if (src0_type == GGML_TYPE_Q8_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_0) { + return true; + } + if (src0_type == GGML_TYPE_Q4_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q4_1) { + return true; + } + if (src0_type == GGML_TYPE_Q4_1 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_0) { + return true; + } + if (src0_type == GGML_TYPE_Q5_0 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_Q5_1) { + return true; + } + if (src0_type == GGML_TYPE_Q5_1 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_IQ4_NL) { + return true; + } + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_I32) { + return true; + } + if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_F32) { + return true; + } + if (src0_type == GGML_TYPE_I32 && src1_type == GGML_TYPE_I32) { + return true; + } + if (src0_type == src1_type && ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1])) { + return true; + } + return false; + } break; + case GGML_OP_DUP: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_ARGMAX: + case GGML_OP_COUNT_EQUAL: + { + return true; + } break; + case GGML_OP_REPEAT: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_REPEAT_BACK: + return op->type == GGML_TYPE_F32 && (op->src[0]->ne[2]*op->src[0]->ne[3]) <= (1 << 15); + case GGML_OP_CONCAT: + { + ggml_type src0_type = op->src[0]->type; + return src0_type != GGML_TYPE_I32 && src0_type != GGML_TYPE_I16; + } break; + case GGML_OP_CONV_TRANSPOSE_1D: + { + ggml_type src0_type = op->src[0]->type; + ggml_type src1_type = op->src[1]->type; + if (src0_type == GGML_TYPE_F32 && src1_type == GGML_TYPE_F32) { + return true; + } + return false; + } break; + case GGML_OP_SILU_BACK: + return ggml_is_contiguous(op->src[0]) && op->src[0]->type == GGML_TYPE_F32; + break; + case GGML_OP_NORM: + case GGML_OP_RMS_NORM: + case GGML_OP_L2_NORM: + return true; + case GGML_OP_RMS_NORM_BACK: + return ggml_is_contiguous(op->src[0]); + break; + case GGML_OP_NONE: + case GGML_OP_RESHAPE: + case GGML_OP_VIEW: + case GGML_OP_PERMUTE: + case GGML_OP_TRANSPOSE: + case GGML_OP_ADD: + case GGML_OP_ADD_ID: + case GGML_OP_ADD1: + case GGML_OP_SUB: + case GGML_OP_MUL: + case GGML_OP_DIV: + case GGML_OP_SCALE: + case GGML_OP_SQR: + case GGML_OP_SQRT: + case GGML_OP_SIN: + case GGML_OP_COS: + case GGML_OP_CLAMP: + case GGML_OP_LOG: + return true; + case GGML_OP_SSM_SCAN: { + if (op->src[3]->ne[0] == 1) { + // Mamba2 + // (kernel only supports (d_state == 128 || d_state == 256) && d_head % 16 == 0) + return (op->src[0]->ne[0] == 128 || op->src[0]->ne[0] == 256) && op->src[0]->ne[1] % 16 == 0; + } else { + // Mamba + // (kernel only supports d_state == 16, d_head == 1, n_head % 128 == 0, n_group == 1) + return op->src[0]->ne[0] == 16 && op->src[0]->ne[1] == 1 && op->src[0]->ne[2] % 128 == 0 && op->src[4]->ne[1] == 1; + } + } + case GGML_OP_SSM_CONV: { + // assumes d_inner % threads == 0 + return op->src[0]->ne[1] % 128 == 0; + } + case GGML_OP_CONT: + return true; + case GGML_OP_DIAG_MASK_INF: + return true; + case GGML_OP_SOFT_MAX: + return true; + case GGML_OP_SOFT_MAX_BACK: { + float max_bias = 0.0f; + memcpy(&max_bias, (const float *) op->op_params + 1, sizeof(float)); + return max_bias == 0.0f; + } + case GGML_OP_ROLL: + if(op->src[0]->type == GGML_TYPE_F32) { + return true; + } + return false; + case GGML_OP_ROPE: + case GGML_OP_ROPE_BACK: { + return op->src[0]->nb[0] == ggml_type_size(op->src[0]->type) && ggml_is_contiguous_2(op->src[0]); + } + case GGML_OP_IM2COL: + case GGML_OP_IM2COL_3D: + case GGML_OP_CONV_2D: + case GGML_OP_CONV_2D_DW: + case GGML_OP_CONV_TRANSPOSE_2D: + case GGML_OP_POOL_2D: + return true; + case GGML_OP_ACC: + // TODO: extend support like so: + //return ggml_is_contiguous_rows(op->src[0]) && ggml_is_contiguous_rows(op->src[1]); + return ggml_is_contiguous(op->src[0]) && ggml_is_contiguous(op->src[1]); + case GGML_OP_SUM: + return ggml_is_contiguous_rows(op->src[0]); + case GGML_OP_TOP_K: + case GGML_OP_ARGSORT: +#ifndef GGML_CUDA_USE_CUB + return op->src[0]->ne[0] <= 1024; +#else + return true; +#endif + case GGML_OP_SUM_ROWS: + case GGML_OP_MEAN: + case GGML_OP_GROUP_NORM: + return ggml_is_contiguous(op->src[0]); + case GGML_OP_PAD: + return true; + case GGML_OP_UPSCALE: + case GGML_OP_PAD_REFLECT_1D: + case GGML_OP_ARANGE: + case GGML_OP_TIMESTEP_EMBEDDING: + case GGML_OP_LEAKY_RELU: + case GGML_OP_RWKV_WKV6: + case GGML_OP_GATED_LINEAR_ATTN: + case GGML_OP_RWKV_WKV7: + return true; + case GGML_OP_GATED_DELTA_NET: + //TODO: enable once MUSA compiler is solved https://github.com/ggml-org/llama.cpp/pull/19504#issuecomment-4018634327 +#ifdef GGML_USE_MUSA + return false; +#else + return true; +#endif // GGML_USE_MUSA + case GGML_OP_FLASH_ATTN_EXT: + return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op); + case GGML_OP_CROSS_ENTROPY_LOSS: + case GGML_OP_CROSS_ENTROPY_LOSS_BACK: + case GGML_OP_OPT_STEP_ADAMW: + case GGML_OP_OPT_STEP_SGD: + case GGML_OP_FILL: + case GGML_OP_CUMSUM: + case GGML_OP_TRI: + case GGML_OP_DIAG: + case GGML_OP_SOLVE_TRI: + return true; + + default: + return false; + } +} + +static bool ggml_backend_cuda_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + const bool integrated = ggml_cuda_info().devices[dev_ctx->device].integrated; + return (((ggml_backend_buft_is_cuda(buft) || ggml_backend_buft_is_cuda_split(buft)) && buft->device == dev) || (integrated && ggml_backend_buft_is_cuda_host(buft))); +} + +static int64_t get_op_batch_size(const ggml_tensor * op) { + switch (op->op) { + case GGML_OP_GET_ROWS: + return 0; + case GGML_OP_MUL_MAT: + return op->ne[1]; + case GGML_OP_MUL_MAT_ID: + case GGML_OP_ROPE: + case GGML_OP_ROPE_BACK: + return op->ne[2]; + default: + return ggml_nrows(op); + } +} + +static bool ggml_backend_cuda_device_offload_op(ggml_backend_dev_t dev, const ggml_tensor * op) { + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *) dev->context; + + return get_op_batch_size(op) >= dev_ctx->op_offload_min_batch_size; +} + +static ggml_backend_event_t ggml_backend_cuda_device_event_new(ggml_backend_dev_t dev) { +#ifdef GGML_CUDA_NO_PEER_COPY + return nullptr; +#else + ggml_backend_cuda_device_context * dev_ctx = (ggml_backend_cuda_device_context *)dev->context; + + ggml_cuda_set_device(dev_ctx->device); + + cudaEvent_t event; + CUDA_CHECK(cudaEventCreateWithFlags(&event, cudaEventDisableTiming)); + + return new ggml_backend_event { + /* .device = */ dev, + /* .context = */ event, + }; +#endif +} + +static void ggml_backend_cuda_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) { + GGML_UNUSED(dev); + + CUDA_CHECK(cudaEventDestroy((cudaEvent_t)event->context)); + delete event; +} + +static void ggml_backend_cuda_device_event_synchronize(ggml_backend_dev_t dev, ggml_backend_event_t event) { + GGML_UNUSED(dev); + CUDA_CHECK(cudaEventSynchronize((cudaEvent_t)event->context)); +} + +static const ggml_backend_device_i ggml_backend_cuda_device_interface = { + /* .get_name = */ ggml_backend_cuda_device_get_name, + /* .get_description = */ ggml_backend_cuda_device_get_description, + /* .get_memory = */ ggml_backend_cuda_device_get_memory, + /* .get_type = */ ggml_backend_cuda_device_get_type, + /* .get_props = */ ggml_backend_cuda_device_get_props, + /* .init_backend = */ ggml_backend_cuda_device_init_backend, + /* .get_buffer_type = */ ggml_backend_cuda_device_get_buffer_type, + /* .get_host_buffer_type = */ ggml_backend_cuda_device_get_host_buffer_type, + /* .buffer_from_host_ptr = */ NULL, + /* .supports_op = */ ggml_backend_cuda_device_supports_op, + /* .supports_buft = */ ggml_backend_cuda_device_supports_buft, + /* .offload_op = */ ggml_backend_cuda_device_offload_op, + /* .event_new = */ ggml_backend_cuda_device_event_new, + /* .event_free = */ ggml_backend_cuda_device_event_free, + /* .event_synchronize = */ ggml_backend_cuda_device_event_synchronize, +}; + +// backend reg + +struct ggml_backend_cuda_reg_context { + std::vector devices; +}; + +static const char * ggml_backend_cuda_reg_get_name(ggml_backend_reg_t reg) { + GGML_UNUSED(reg); + return GGML_CUDA_NAME; +} + +static size_t ggml_backend_cuda_reg_get_device_count(ggml_backend_reg_t reg) { + ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; + return ctx->devices.size(); +} + +static ggml_backend_dev_t ggml_backend_cuda_reg_get_device(ggml_backend_reg_t reg, size_t index) { + ggml_backend_cuda_reg_context * ctx = (ggml_backend_cuda_reg_context *)reg->context; + GGML_ASSERT(index < ctx->devices.size()); + return ctx->devices[index]; +} + +static ggml_backend_feature * ggml_backend_cuda_get_features(ggml_backend_reg_t reg) { + static std::vector features = []() { + std::vector features; + #define _STRINGIFY(...) #__VA_ARGS__ + #define STRINGIFY(...) _STRINGIFY(__VA_ARGS__) + + #ifdef __CUDA_ARCH_LIST__ + features.push_back({ "ARCHS", STRINGIFY(__CUDA_ARCH_LIST__) }); + #endif + + #ifdef GGML_CUDA_FORCE_MMQ + features.push_back({ "FORCE_MMQ", "1" }); + #endif + + #ifdef GGML_CUDA_FORCE_CUBLAS + features.push_back({ "FORCE_CUBLAS", "1" }); + #endif + + #ifndef GGML_USE_VMM + features.push_back({ "NO_VMM", "1" }); + #endif + + #ifdef GGML_CUDA_NO_PEER_COPY + features.push_back({ "NO_PEER_COPY", "1" }); + #endif + + #ifdef GGML_CUDA_USE_GRAPHS + features.push_back({ "USE_GRAPHS", "1" }); + #endif + + #ifdef GGML_CUDA_PEER_MAX_BATCH_SIZE + features.push_back({ "PEER_MAX_BATCH_SIZE", STRINGIFY(GGML_CUDA_PEER_MAX_BATCH_SIZE) }); + #endif + + #ifdef GGML_CUDA_FA_ALL_QUANTS + features.push_back({ "FA_ALL_QUANTS", "1" }); + #endif + + { + const auto & info = ggml_cuda_info(); + for (int id = 0; id < info.device_count; ++id) { + if (blackwell_mma_available(info.devices[id].cc)) { + features.push_back({ "BLACKWELL_NATIVE_FP4", "1"}); + break; + } + } + } + + #undef _STRINGIFY + #undef STRINGIFY + + features.push_back({ nullptr, nullptr }); + + return features; + }(); + + return features.data(); + + GGML_UNUSED(reg); +} + +static void * ggml_backend_cuda_reg_get_proc_address(ggml_backend_reg_t reg, const char * name) { + GGML_UNUSED(reg); + if (strcmp(name, "ggml_backend_comm_init") == 0) { + return (void *)ggml_backend_cuda_comm_init; + } + if (strcmp(name, "ggml_backend_comm_free") == 0) { + return (void *)ggml_backend_cuda_comm_free; + } + if (strcmp(name, "ggml_backend_comm_allreduce_tensor") == 0) { + return (void *)ggml_backend_cuda_comm_allreduce_tensor; + } + if (strcmp(name, "ggml_backend_split_buffer_type") == 0) { + return (void *)ggml_backend_cuda_split_buffer_type; + } + if (strcmp(name, "ggml_backend_register_host_buffer") == 0) { + return (void *)ggml_backend_cuda_register_host_buffer; + } + if (strcmp(name, "ggml_backend_unregister_host_buffer") == 0) { + return (void *)ggml_backend_cuda_unregister_host_buffer; + } + if (strcmp(name, "ggml_backend_get_features") == 0) { + return (void *)ggml_backend_cuda_get_features; + } + return nullptr; +} + +static const ggml_backend_reg_i ggml_backend_cuda_reg_interface = { + /* .get_name = */ ggml_backend_cuda_reg_get_name, + /* .get_device_count = */ ggml_backend_cuda_reg_get_device_count, + /* .get_device = */ ggml_backend_cuda_reg_get_device, + /* .get_proc_address = */ ggml_backend_cuda_reg_get_proc_address, +}; + +// backend registry +ggml_backend_reg_t ggml_backend_cuda_reg() { + static ggml_backend_reg reg; + static bool initialized = false; + + { + static std::mutex mutex; + std::lock_guard lock(mutex); + if (!initialized) { + ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context; + const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32; + + for (int i = 0; i < ggml_cuda_info().device_count; i++) { + ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context; + dev_ctx->device = i; + dev_ctx->name = GGML_CUDA_NAME + std::to_string(i); + + cudaDeviceProp prop; + CUDA_CHECK(cudaGetDeviceProperties(&prop, i)); + dev_ctx->description = prop.name; + + char pci_bus_id[16] = {}; + snprintf(pci_bus_id, sizeof(pci_bus_id), "%04x:%02x:%02x.0", prop.pciDomainID, prop.pciBusID, prop.pciDeviceID); + dev_ctx->pci_bus_id = pci_bus_id; + dev_ctx->op_offload_min_batch_size = min_batch_size; + + ggml_backend_dev_t dev = new ggml_backend_device { + /* .iface = */ ggml_backend_cuda_device_interface, + /* .reg = */ ®, + /* .context = */ dev_ctx + }; + ctx->devices.push_back(dev); + } + + reg = ggml_backend_reg { + /* .api_version = */ GGML_BACKEND_API_VERSION, + /* .iface = */ ggml_backend_cuda_reg_interface, + /* .context = */ ctx + }; + } + + initialized = true; + } + + return ® +} + +ggml_backend_t ggml_backend_cuda_init(int device) { + if (device < 0 || device >= ggml_backend_cuda_get_device_count()) { + GGML_LOG_ERROR("%s: invalid device %d\n", __func__, device); + return nullptr; + } + + ggml_backend_cuda_context * ctx = new ggml_backend_cuda_context(device); + if (ctx == nullptr) { + GGML_LOG_ERROR("%s: failed to allocate context\n", __func__); + return nullptr; + } + + ggml_backend_t cuda_backend = new ggml_backend { + /* .guid = */ ggml_backend_cuda_guid(), + /* .iface = */ ggml_backend_cuda_interface, + /* .device = */ ggml_backend_reg_dev_get(ggml_backend_cuda_reg(), device), + /* .context = */ ctx, + }; + + return cuda_backend; +} + +GGML_BACKEND_DL_IMPL(ggml_backend_cuda_reg) diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index d57c564958c..366bf1cb57b 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -1,2472 +1,2472 @@ -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "build-info.h" -#include "common.h" -#include "download.h" -#include "fit.h" -#include "ggml.h" -#include "llama.h" - -#ifdef _WIN32 -# define WIN32_LEAN_AND_MEAN -# ifndef NOMINMAX -# define NOMINMAX -# endif -# include -#endif - -// utils -static uint64_t get_time_ns() { - using clock = std::chrono::high_resolution_clock; - return std::chrono::nanoseconds(clock::now().time_since_epoch()).count(); -} - -static bool tensor_buft_override_equal(const llama_model_tensor_buft_override& a, const llama_model_tensor_buft_override& b) { - if (a.pattern != b.pattern) { - // cString comparison that may be null - if (a.pattern == nullptr || b.pattern == nullptr) { - return false; - } - if (strcmp(a.pattern, b.pattern) != 0) { - return false; - } - } - if (a.buft != b.buft) { - return false; - } - return true; -} - -static bool vec_tensor_buft_override_equal(const std::vector& a, const std::vector& b) { - if (a.size() != b.size()) { - return false; - } - for (size_t i = 0; i < a.size(); i++) { - if (!tensor_buft_override_equal(a[i], b[i])) { - return false; - } - } - return true; -} - -static bool vec_vec_tensor_buft_override_equal(const std::vector>& a, const std::vector>& b) { - if (a.size() != b.size()) { - return false; - } - for (size_t i = 0; i < a.size(); i++) { - if (!vec_tensor_buft_override_equal(a[i], b[i])) { - return false; - } - } - return true; -} - -template static std::string join(const std::vector & values, const std::string & delim) { - std::ostringstream str; - for (size_t i = 0; i < values.size(); i++) { - str << values[i]; - if (i < values.size() - 1) { - str << delim; - } - } - return str.str(); -} - -template static std::vector transform_to_str(const std::vector & values, F f) { - std::vector str_values; - std::transform(values.begin(), values.end(), std::back_inserter(str_values), f); - return str_values; -} - -template static T avg(const std::vector & v) { - if (v.empty()) { - return 0; - } - T sum = std::accumulate(v.begin(), v.end(), T(0)); - return sum / (T) v.size(); -} - -template static T stdev(const std::vector & v) { - if (v.size() <= 1) { - return 0; - } - T mean = avg(v); - T sq_sum = std::inner_product(v.begin(), v.end(), v.begin(), T(0)); - T stdev = std::sqrt(sq_sum / (T) (v.size() - 1) - mean * mean * (T) v.size() / (T) (v.size() - 1)); - return stdev; -} - -static std::string get_cpu_info() { - std::vector cpu_list; - for (size_t i = 0; i < ggml_backend_dev_count(); i++) { - auto * dev = ggml_backend_dev_get(i); - auto dev_type = ggml_backend_dev_type(dev); - if (dev_type == GGML_BACKEND_DEVICE_TYPE_CPU || dev_type == GGML_BACKEND_DEVICE_TYPE_ACCEL) { - cpu_list.push_back(ggml_backend_dev_description(dev)); - } - } - return join(cpu_list, ", "); -} - -static std::string get_gpu_info() { - std::vector gpu_list; - for (size_t i = 0; i < ggml_backend_dev_count(); i++) { - auto * dev = ggml_backend_dev_get(i); - auto dev_type = ggml_backend_dev_type(dev); - if (dev_type == GGML_BACKEND_DEVICE_TYPE_GPU || dev_type == GGML_BACKEND_DEVICE_TYPE_IGPU) { - gpu_list.push_back(ggml_backend_dev_description(dev)); - } - } - return join(gpu_list, ", "); -} - -static std::vector parse_devices_arg(const std::string & value) { - std::vector devices; - std::string trimmed = string_strip(value); - if (trimmed.empty()) { - throw std::invalid_argument("no devices specified"); - } - if (trimmed == "auto") { - return devices; - } - - auto dev_names = string_split(trimmed, '/'); - if (dev_names.size() == 1 && string_strip(dev_names[0]) == "none") { - devices.push_back(nullptr); - return devices; - } - - for (auto & name : dev_names) { - std::string dev_name = string_strip(name); - if (dev_name.empty()) { - throw std::invalid_argument("invalid device specification"); - } - auto * dev = ggml_backend_dev_by_name(dev_name.c_str()); - if (!dev || ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU) { - throw std::invalid_argument(string_format("invalid device: %s", dev_name.c_str())); - } - devices.push_back(dev); - } - - devices.push_back(nullptr); - return devices; -} - -static void register_rpc_server_list(const std::string & servers) { - auto rpc_servers = string_split(servers, ','); - if (rpc_servers.empty()) { - throw std::invalid_argument("no RPC servers specified"); - } - - auto * rpc_reg = ggml_backend_reg_by_name("RPC"); - if (!rpc_reg) { - throw std::invalid_argument("failed to find RPC backend"); - } - - using add_rpc_server_fn = ggml_backend_reg_t (*)(const char * endpoint); - auto * ggml_backend_rpc_add_server_fn = (add_rpc_server_fn) ggml_backend_reg_get_proc_address(rpc_reg, "ggml_backend_rpc_add_server"); - if (!ggml_backend_rpc_add_server_fn) { - throw std::invalid_argument("failed to find RPC add server function"); - } - for (const auto & server : rpc_servers) { - auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); - ggml_backend_register(reg); - } -} - -static std::string devices_to_string(const std::vector & devices) { - if (devices.empty()) { - return "auto"; - } - - if (devices.size() == 1 && devices[0] == nullptr) { - return "none"; - } - - std::vector names; - for (auto * dev : devices) { - if (dev == nullptr) { - break; - } - names.push_back(ggml_backend_dev_name(dev)); - } - - return join(names, "/"); -} - -// command line params -enum output_formats { NONE, CSV, JSON, JSONL, MARKDOWN, SQL }; - -static const char * output_format_str(output_formats format) { - switch (format) { - case NONE: - return "none"; - case CSV: - return "csv"; - case JSON: - return "json"; - case JSONL: - return "jsonl"; - case MARKDOWN: - return "md"; - case SQL: - return "sql"; - default: - GGML_ABORT("invalid output format"); - } -} - -static bool output_format_from_str(const std::string & s, output_formats & format) { - if (s == "none") { - format = NONE; - } else if (s == "csv") { - format = CSV; - } else if (s == "json") { - format = JSON; - } else if (s == "jsonl") { - format = JSONL; - } else if (s == "md") { - format = MARKDOWN; - } else if (s == "sql") { - format = SQL; - } else { - return false; - } - return true; -} - -static const char * split_mode_str(llama_split_mode mode) { - switch (mode) { - case LLAMA_SPLIT_MODE_NONE: - return "none"; - case LLAMA_SPLIT_MODE_LAYER: - return "layer"; - case LLAMA_SPLIT_MODE_ROW: - return "row"; - case LLAMA_SPLIT_MODE_TENSOR: - return "tensor"; - default: - GGML_ABORT("invalid split mode"); - } -} - -static std::string pair_str(const std::pair & p) { - static char buf[32]; - snprintf(buf, sizeof(buf), "%d,%d", p.first, p.second); - return buf; -} - -static std::vector parse_int_range(const std::string & s) { - // first[-last[(+|*)step]] - std::regex range_regex(R"(^(\d+)(?:-(\d+)(?:([\+|\*])(\d+))?)?(?:,|$))"); - - std::smatch match; - std::string::const_iterator search_start(s.cbegin()); - std::vector result; - while (std::regex_search(search_start, s.cend(), match, range_regex)) { - int first = std::stoi(match[1]); - int last = match[2].matched ? std::stoi(match[2]) : first; - char op = match[3].matched ? match[3].str()[0] : '+'; - int step = match[4].matched ? std::stoi(match[4]) : 1; - - for (int i = first; i <= last;) { - result.push_back(i); - - int prev_i = i; - - if (op == '+') { - i += step; - } else if (op == '*') { - i *= step; - } else { - throw std::invalid_argument("invalid range format"); - } - - if (i <= prev_i) { - throw std::invalid_argument("invalid range"); - } - } - search_start = match.suffix().first; - } - - if (search_start != s.cend()) { - throw std::invalid_argument("invalid range format"); - } - - return result; -} - -struct cmd_params { - std::vector model; - std::vector hf_repo; - std::vector hf_file; - std::string hf_token; - std::vector n_prompt; - std::vector n_gen; - std::vector> n_pg; - std::vector n_depth; - std::vector n_batch; - std::vector n_ubatch; - std::vector type_k; - std::vector type_v; - std::vector n_threads; - std::vector cpu_mask; - std::vector cpu_strict; - std::vector poll; - std::vector n_gpu_layers; - std::vector n_cpu_moe; - std::vector split_mode; - std::vector reduction_provider; - std::vector main_gpu; - std::vector no_kv_offload; - std::vector flash_attn; - std::vector> devices; - std::vector> tensor_split; - std::vector> tensor_buft_overrides; - std::vector use_mmap; - std::vector use_direct_io; - std::vector embeddings; - std::vector no_op_offload; - std::vector no_host; - std::vector fit_params_target; - std::vector fit_params_min_ctx; - ggml_numa_strategy numa; - int reps; - ggml_sched_priority prio; - int delay; - bool verbose; - bool progress; - bool no_warmup; - output_formats output_format; - output_formats output_format_stderr; -}; - -static const cmd_params cmd_params_defaults = { - /* model */ { "models/7B/ggml-model-q4_0.gguf" }, - /* hf_repo */ {}, - /* hf_file */ {}, - /* hf_token */ "", - /* n_prompt */ { 512 }, - /* n_gen */ { 128 }, - /* n_pg */ {}, - /* n_depth */ { 0 }, - /* n_batch */ { 2048 }, - /* n_ubatch */ { 512 }, - /* type_k */ { GGML_TYPE_F16 }, - /* type_v */ { GGML_TYPE_F16 }, - /* n_threads */ { cpu_get_num_math() }, - /* cpu_mask */ { "0x0" }, - /* cpu_strict */ { false }, - /* poll */ { 50 }, - /* n_gpu_layers */ { 99 }, - /* n_cpu_moe */ { 0 }, - /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, - /* reduction_provider */ { "auto" }, - /* main_gpu */ { 0 }, - /* no_kv_offload */ { false }, - /* flash_attn */ { false }, - /* devices */ { {} }, - /* tensor_split */ { std::vector(llama_max_devices(), 0.0f) }, - /* tensor_buft_overrides*/ { std::vector{ { nullptr, nullptr } } }, - /* use_mmap */ { true }, - /* use_direct_io */ { false }, - /* embeddings */ { false }, - /* no_op_offload */ { false }, - /* no_host */ { false }, - /* fit_params_target */ { 0 }, - /* fit_params_min_ctx */ { 0 }, - /* numa */ GGML_NUMA_STRATEGY_DISABLED, - /* reps */ 5, - /* prio */ GGML_SCHED_PRIO_NORMAL, - /* delay */ 0, - /* verbose */ false, - /* progress */ false, - /* no_warmup */ false, - /* output_format */ MARKDOWN, - /* output_format_stderr */ NONE, -}; - -static void print_usage(int /* argc */, char ** argv) { - printf("usage: %s [options]\n", argv[0]); - printf("\n"); - printf("options:\n"); - printf(" -h, --help\n"); - printf(" --numa numa mode (default: disabled)\n"); - printf(" -r, --repetitions number of times to repeat each test (default: %d)\n", cmd_params_defaults.reps); - printf(" --prio <-1|0|1|2|3> process/thread priority (default: %d)\n", cmd_params_defaults.prio); - printf(" --delay <0...N> (seconds) delay between each test (default: %d)\n", cmd_params_defaults.delay); - printf(" -o, --output output format printed to stdout (default: %s)\n", output_format_str(cmd_params_defaults.output_format)); - printf(" -oe, --output-err output format printed to stderr (default: %s)\n", output_format_str(cmd_params_defaults.output_format_stderr)); - printf(" --list-devices list available devices and exit\n"); - printf(" -v, --verbose verbose output\n"); - printf(" --progress print test progress indicators\n"); - printf(" --no-warmup skip warmup runs before benchmarking\n"); - printf(" -fitt, --fit-target fit model to device memory with this margin per device in MiB (default: off)\n"); - printf(" -fitc, --fit-ctx minimum ctx size for --fit-target (default: 4096)\n"); - if (llama_supports_rpc()) { - printf(" -rpc, --rpc register RPC devices (comma separated)\n"); - } - printf("\n"); - printf("test parameters:\n"); - printf(" -m, --model (default: %s)\n", join(cmd_params_defaults.model, ",").c_str()); - printf(" -hf, -hfr, --hf-repo /[:quant] Hugging Face model repository; quant is optional, case-insensitive\n"); - printf(" default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.\n"); - printf(" example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M\n"); - printf(" (default: unused)\n"); - printf(" -hff, --hf-file Hugging Face model file. If specified, it will override the quant in --hf-repo\n"); - printf(" (default: unused)\n"); - printf(" -hft, --hf-token Hugging Face access token\n"); - printf(" (default: value from HF_TOKEN environment variable)\n"); - printf(" -p, --n-prompt (default: %s)\n", join(cmd_params_defaults.n_prompt, ",").c_str()); - printf(" -n, --n-gen (default: %s)\n", join(cmd_params_defaults.n_gen, ",").c_str()); - printf(" -pg (default: %s)\n", join(transform_to_str(cmd_params_defaults.n_pg, pair_str), ",").c_str()); - printf(" -d, --n-depth (default: %s)\n", join(cmd_params_defaults.n_depth, ",").c_str()); - printf(" -b, --batch-size (default: %s)\n", join(cmd_params_defaults.n_batch, ",").c_str()); - printf(" -ub, --ubatch-size (default: %s)\n", join(cmd_params_defaults.n_ubatch, ",").c_str()); - printf(" -ctk, --cache-type-k (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_k, ggml_type_name), ",").c_str()); - printf(" -ctv, --cache-type-v (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_v, ggml_type_name), ",").c_str()); - printf(" -t, --threads (default: %s)\n", join(cmd_params_defaults.n_threads, ",").c_str()); - printf(" -C, --cpu-mask (default: %s)\n", join(cmd_params_defaults.cpu_mask, ",").c_str()); - printf(" --cpu-strict <0|1> (default: %s)\n", join(cmd_params_defaults.cpu_strict, ",").c_str()); - printf(" --poll <0...100> (default: %s)\n", join(cmd_params_defaults.poll, ",").c_str()); - printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); - printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); - printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); - printf(" -rp, --reduction-provider allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.reduction_provider, ",").c_str()); - printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); - printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); - printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); - printf(" -dev, --device (default: auto)\n"); - printf(" -mmp, --mmap <0|1> (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str()); - printf(" -dio, --direct-io <0|1> (default: %s)\n", join(cmd_params_defaults.use_direct_io, ",").c_str()); - printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str()); - printf(" -ts, --tensor-split (default: 0)\n"); - printf(" -ot --override-tensor =;...\n"); - printf(" (default: disabled)\n"); - printf(" -nopo, --no-op-offload <0|1> (default: 0)\n"); - printf(" --no-host <0|1> (default: %s)\n", join(cmd_params_defaults.no_host, ",").c_str()); - printf("\n"); - printf( - "Multiple values can be given for each parameter by separating them with ','\n" - "or by specifying the parameter multiple times. Ranges can be given as\n" - "'first-last' or 'first-last+step' or 'first-last*mult'.\n"); -} - -static ggml_type ggml_type_from_name(const std::string & s) { - if (s == "f16") { - return GGML_TYPE_F16; - } - if (s == "bf16") { - return GGML_TYPE_BF16; - } - if (s == "q8_0") { - return GGML_TYPE_Q8_0; - } - if (s == "q4_0") { - return GGML_TYPE_Q4_0; - } - if (s == "q4_1") { - return GGML_TYPE_Q4_1; - } - if (s == "q5_0") { - return GGML_TYPE_Q5_0; - } - if (s == "q5_1") { - return GGML_TYPE_Q5_1; - } - if (s == "iq4_nl") { - return GGML_TYPE_IQ4_NL; - } - - return GGML_TYPE_COUNT; -} - -static cmd_params parse_cmd_params(int argc, char ** argv) { - cmd_params params; - std::string arg; - bool invalid_param = false; - const std::string arg_prefix = "--"; - const char split_delim = ','; - - params.verbose = cmd_params_defaults.verbose; - params.output_format = cmd_params_defaults.output_format; - params.output_format_stderr = cmd_params_defaults.output_format_stderr; - params.reps = cmd_params_defaults.reps; - params.numa = cmd_params_defaults.numa; - params.prio = cmd_params_defaults.prio; - params.delay = cmd_params_defaults.delay; - params.progress = cmd_params_defaults.progress; - params.no_warmup = cmd_params_defaults.no_warmup; - - if (const char * env = getenv("HF_TOKEN")) { - params.hf_token = env; - } - - for (int i = 1; i < argc; i++) { - arg = argv[i]; - if (arg.compare(0, arg_prefix.size(), arg_prefix) == 0) { - std::replace(arg.begin(), arg.end(), '_', '-'); - } - - try { - if (arg == "-h" || arg == "--help") { - print_usage(argc, argv); - exit(0); - } else if (arg == "-m" || arg == "--model") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.model.insert(params.model.end(), p.begin(), p.end()); - } else if (arg == "-hf" || arg == "-hfr" || arg == "--hf-repo") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.hf_repo.insert(params.hf_repo.end(), p.begin(), p.end()); - } else if (arg == "-hff" || arg == "--hf-file") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.hf_file.insert(params.hf_file.end(), p.begin(), p.end()); - } else if (arg == "-hft" || arg == "--hf-token") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.hf_token = argv[i]; - } else if (arg == "-p" || arg == "--n-prompt") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_prompt.insert(params.n_prompt.end(), p.begin(), p.end()); - } else if (arg == "-n" || arg == "--n-gen") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_gen.insert(params.n_gen.end(), p.begin(), p.end()); - } else if (arg == "-pg") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], ','); - if (p.size() != 2) { - invalid_param = true; - break; - } - params.n_pg.push_back({ std::stoi(p[0]), std::stoi(p[1]) }); - } else if (arg == "-d" || arg == "--n-depth") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_depth.insert(params.n_depth.end(), p.begin(), p.end()); - } else if (arg == "-b" || arg == "--batch-size") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_batch.insert(params.n_batch.end(), p.begin(), p.end()); - } else if (arg == "-ub" || arg == "--ubatch-size") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_ubatch.insert(params.n_ubatch.end(), p.begin(), p.end()); - } else if (arg == "-ctk" || arg == "--cache-type-k") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector types; - for (const auto & t : p) { - ggml_type gt = ggml_type_from_name(t); - if (gt == GGML_TYPE_COUNT) { - invalid_param = true; - break; - } - types.push_back(gt); - } - if (invalid_param) { - break; - } - params.type_k.insert(params.type_k.end(), types.begin(), types.end()); - } else if (arg == "-ctv" || arg == "--cache-type-v") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector types; - for (const auto & t : p) { - ggml_type gt = ggml_type_from_name(t); - if (gt == GGML_TYPE_COUNT) { - invalid_param = true; - break; - } - types.push_back(gt); - } - if (invalid_param) { - break; - } - params.type_v.insert(params.type_v.end(), types.begin(), types.end()); - } else if (arg == "-dev" || arg == "--device") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto combos = string_split(argv[i], split_delim); - for (const auto & combo : combos) { - try { - params.devices.push_back(parse_devices_arg(combo)); - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } - if (invalid_param) { - break; - } - } else if (arg == "--list-devices") { - std::vector devices; - for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { - auto * dev = ggml_backend_dev_get(i); - if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) { - devices.push_back(dev); - } - } - printf("Available devices:\n"); - if (devices.empty()) { - printf(" (none)\n"); - } - for (auto * dev : devices) { - size_t free, total; - ggml_backend_dev_memory(dev, &free, &total); - printf(" %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / 1024 / 1024, free / 1024 / 1024); - } - exit(0); - } else if (arg == "-t" || arg == "--threads") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_threads.insert(params.n_threads.end(), p.begin(), p.end()); - } else if (arg == "-C" || arg == "--cpu-mask") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.cpu_mask.insert(params.cpu_mask.end(), p.begin(), p.end()); - } else if (arg == "--cpu-strict") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.cpu_strict.insert(params.cpu_strict.end(), p.begin(), p.end()); - } else if (arg == "--poll") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.poll.insert(params.poll.end(), p.begin(), p.end()); - } else if (arg == "-ngl" || arg == "--n-gpu-layers") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_gpu_layers.insert(params.n_gpu_layers.end(), p.begin(), p.end()); - } else if (arg == "-ncmoe" || arg == "--n-cpu-moe") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = parse_int_range(argv[i]); - params.n_cpu_moe.insert(params.n_cpu_moe.end(), p.begin(), p.end()); - } else if (llama_supports_rpc() && (arg == "-rpc" || arg == "--rpc")) { - if (++i >= argc) { - invalid_param = true; - break; - } - try { - register_rpc_server_list(argv[i]); - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } else if (arg == "-sm" || arg == "--split-mode") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - - std::vector modes; - for (const auto & m : p) { - llama_split_mode mode; - if (m == "none") { - mode = LLAMA_SPLIT_MODE_NONE; - } else if (m == "layer") { - mode = LLAMA_SPLIT_MODE_LAYER; - } else if (m == "row") { - mode = LLAMA_SPLIT_MODE_ROW; - } else if (m == "tensor") { - mode = LLAMA_SPLIT_MODE_TENSOR; - } else { - invalid_param = true; - break; - } - modes.push_back(mode); - } - if (invalid_param) { - break; - } - params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); - } else if (arg == "-rp" || arg == "--reduction-provider") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - if (v != "auto" && v != "nccl" && v != "internal") { - invalid_param = true; - break; - } - } - if (invalid_param) { - break; - } - params.reduction_provider.insert(params.reduction_provider.end(), p.begin(), p.end()); - } else if (arg == "-mg" || arg == "--main-gpu") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.main_gpu = parse_int_range(argv[i]); - } else if (arg == "-nkvo" || arg == "--no-kv-offload") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_kv_offload.insert(params.no_kv_offload.end(), p.begin(), p.end()); - } else if (arg == "--numa") { - if (++i >= argc) { - invalid_param = true; - break; - } - std::string value(argv[i]); - if (value == "distribute" || value == "") { - params.numa = GGML_NUMA_STRATEGY_DISTRIBUTE; - } else if (value == "isolate") { - params.numa = GGML_NUMA_STRATEGY_ISOLATE; - } else if (value == "numactl") { - params.numa = GGML_NUMA_STRATEGY_NUMACTL; - } else { - invalid_param = true; - break; - } - } else if (arg == "-fa" || arg == "--flash-attn") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.flash_attn.insert(params.flash_attn.end(), p.begin(), p.end()); - } else if (arg == "-mmp" || arg == "--mmap") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.use_mmap.insert(params.use_mmap.end(), p.begin(), p.end()); - } else if (arg == "-dio" || arg == "--direct-io") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.use_direct_io.insert(params.use_direct_io.end(), p.begin(), p.end()); - } else if (arg == "-embd" || arg == "--embeddings") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.embeddings.insert(params.embeddings.end(), p.begin(), p.end()); - } else if (arg == "-nopo" || arg == "--no-op-offload") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_op_offload.insert(params.no_op_offload.end(), p.begin(), p.end()); - } else if (arg == "--no-host") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - params.no_host.insert(params.no_host.end(), p.begin(), p.end()); - } else if (arg == "-ts" || arg == "--tensor-split") { - if (++i >= argc) { - invalid_param = true; - break; - } - for (auto ts : string_split(argv[i], split_delim)) { - // split string by ; and / - const std::regex regex{ R"([;/]+)" }; - std::sregex_token_iterator it{ ts.begin(), ts.end(), regex, -1 }; - std::vector split_arg{ it, {} }; - GGML_ASSERT(split_arg.size() <= llama_max_devices()); - - std::vector tensor_split(llama_max_devices()); - for (size_t i = 0; i < llama_max_devices(); ++i) { - if (i < split_arg.size()) { - tensor_split[i] = std::stof(split_arg[i]); - } else { - tensor_split[i] = 0.0f; - } - } - params.tensor_split.push_back(tensor_split); - } - } else if (arg == "-ot" || arg == "--override-tensor") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto * value = argv[i]; - /* static */ std::map buft_list; - if (buft_list.empty()) { - // enumerate all the devices and add their buffer types to the list - for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { - auto * dev = ggml_backend_dev_get(i); - auto * buft = ggml_backend_dev_buffer_type(dev); - if (buft) { - buft_list[ggml_backend_buft_name(buft)] = buft; - } - } - } - auto override_group_span_len = std::strcspn(value, ","); - bool last_group = false; - do { - if (override_group_span_len == 0) { - // Adds an empty override-tensors for an empty span - params.tensor_buft_overrides.push_back({{}}); - if (value[override_group_span_len] == '\0') { - value = &value[override_group_span_len]; - last_group = true; - } else { - value = &value[override_group_span_len + 1]; - override_group_span_len = std::strcspn(value, ","); - } - continue; - } - // Stamps null terminators into the argv - // value for this option to avoid the - // memory leak present in the implementation - // over in arg.cpp. Acceptable because we - // only parse these args once in this program. - auto * override_group = value; - if (value[override_group_span_len] == '\0') { - value = &value[override_group_span_len]; - last_group = true; - } else { - value[override_group_span_len] = '\0'; - value = &value[override_group_span_len + 1]; - } - std::vector group_tensor_buft_overrides{}; - auto override_span_len = std::strcspn(override_group, ";"); - while (override_span_len > 0) { - auto * override = override_group; - if (override_group[override_span_len] != '\0') { - override_group[override_span_len] = '\0'; - override_group = &override_group[override_span_len + 1]; - } else { - override_group = &override_group[override_span_len]; - } - auto tensor_name_span_len = std::strcspn(override, "="); - if (tensor_name_span_len >= override_span_len) { - invalid_param = true; - break; - } - override[tensor_name_span_len] = '\0'; - auto * tensor_name = override; - auto * buffer_type = &override[tensor_name_span_len + 1]; - if (buft_list.find(buffer_type) == buft_list.end()) { - printf("error: unrecognized buffer type '%s'\n", buffer_type); - printf("Available buffer types:\n"); - for (const auto & it : buft_list) { - printf(" %s\n", ggml_backend_buft_name(it.second)); - } - invalid_param = true; - break; - } - group_tensor_buft_overrides.push_back({tensor_name, buft_list.at(buffer_type)}); - override_span_len = std::strcspn(override_group, ";"); - } - if (invalid_param) { - break; - } - group_tensor_buft_overrides.push_back({nullptr,nullptr}); - params.tensor_buft_overrides.push_back(group_tensor_buft_overrides); - override_group_span_len = std::strcspn(value, ","); - } while (!last_group); - } else if (arg == "-r" || arg == "--repetitions") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.reps = std::stoi(argv[i]); - } else if (arg == "--prio") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.prio = (enum ggml_sched_priority) std::stoi(argv[i]); - } else if (arg == "--delay") { - if (++i >= argc) { - invalid_param = true; - break; - } - params.delay = std::stoi(argv[i]); - } else if (arg == "-o" || arg == "--output") { - if (++i >= argc) { - invalid_param = true; - break; - } - invalid_param = !output_format_from_str(argv[i], params.output_format); - } else if (arg == "-oe" || arg == "--output-err") { - if (++i >= argc) { - invalid_param = true; - break; - } - invalid_param = !output_format_from_str(argv[i], params.output_format_stderr); - } else if (arg == "-v" || arg == "--verbose") { - params.verbose = true; - } else if (arg == "--progress") { - params.progress = true; - } else if (arg == "--no-warmup") { - params.no_warmup = true; - } else if (arg == "-fitt" || arg == "--fit-target") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - params.fit_params_target.push_back(std::stoull(v)); - } - } else if (arg == "-fitc" || arg == "--fit-ctx") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - params.fit_params_min_ctx.push_back(std::stoul(v)); - } - } else { - invalid_param = true; - break; - } - } catch (const std::exception & e) { - fprintf(stderr, "error: %s\n", e.what()); - invalid_param = true; - break; - } - } - - if (invalid_param) { - fprintf(stderr, "error: invalid parameter for argument: %s\n", arg.c_str()); - print_usage(argc, argv); - exit(1); - } - - if (!params.hf_repo.empty()) { - for (size_t i = 0; i < params.hf_repo.size(); i++) { - common_params_model model; - - if (params.hf_file.empty() || params.hf_file[i].empty()) { - model.hf_repo = params.hf_repo[i]; - } else { - model.hf_repo = params.hf_repo[i]; - model.hf_file = params.hf_file[i]; - } - - common_download_opts opts; - opts.bearer_token = params.hf_token; - auto download_result = common_download_model(model, opts); - if (download_result.model_path.empty()) { - fprintf(stderr, "error: failed to download model from HuggingFace\n"); - exit(1); - } - - params.model.push_back(download_result.model_path); - } - } - - // set defaults - if (params.model.empty()) { - params.model = cmd_params_defaults.model; - } - if (params.n_prompt.empty()) { - params.n_prompt = cmd_params_defaults.n_prompt; - } - if (params.n_gen.empty()) { - params.n_gen = cmd_params_defaults.n_gen; - } - if (params.n_pg.empty()) { - params.n_pg = cmd_params_defaults.n_pg; - } - if (params.n_depth.empty()) { - params.n_depth = cmd_params_defaults.n_depth; - } - if (params.n_batch.empty()) { - params.n_batch = cmd_params_defaults.n_batch; - } - if (params.n_ubatch.empty()) { - params.n_ubatch = cmd_params_defaults.n_ubatch; - } - if (params.type_k.empty()) { - params.type_k = cmd_params_defaults.type_k; - } - if (params.type_v.empty()) { - params.type_v = cmd_params_defaults.type_v; - } - if (params.n_gpu_layers.empty()) { - params.n_gpu_layers = cmd_params_defaults.n_gpu_layers; - } - if (params.n_cpu_moe.empty()) { - params.n_cpu_moe = cmd_params_defaults.n_cpu_moe; - } - if (params.split_mode.empty()) { - params.split_mode = cmd_params_defaults.split_mode; - } - if (params.reduction_provider.empty()) { - params.reduction_provider = cmd_params_defaults.reduction_provider; - } - if (params.main_gpu.empty()) { - params.main_gpu = cmd_params_defaults.main_gpu; - } - if (params.no_kv_offload.empty()) { - params.no_kv_offload = cmd_params_defaults.no_kv_offload; - } - if (params.flash_attn.empty()) { - params.flash_attn = cmd_params_defaults.flash_attn; - } - if (params.devices.empty()) { - params.devices = cmd_params_defaults.devices; - } - if (params.tensor_split.empty()) { - params.tensor_split = cmd_params_defaults.tensor_split; - } - if (params.tensor_buft_overrides.empty()) { - params.tensor_buft_overrides = cmd_params_defaults.tensor_buft_overrides; - } - if (params.use_mmap.empty()) { - params.use_mmap = cmd_params_defaults.use_mmap; - } - if (params.use_direct_io.empty()) { - params.use_direct_io = cmd_params_defaults.use_direct_io; - } - if (params.embeddings.empty()) { - params.embeddings = cmd_params_defaults.embeddings; - } - if (params.no_op_offload.empty()) { - params.no_op_offload = cmd_params_defaults.no_op_offload; - } - if (params.no_host.empty()) { - params.no_host = cmd_params_defaults.no_host; - } - if (params.n_threads.empty()) { - params.n_threads = cmd_params_defaults.n_threads; - } - if (params.cpu_mask.empty()) { - params.cpu_mask = cmd_params_defaults.cpu_mask; - } - if (params.cpu_strict.empty()) { - params.cpu_strict = cmd_params_defaults.cpu_strict; - } - if (params.poll.empty()) { - params.poll = cmd_params_defaults.poll; - } - if (params.fit_params_target.empty()) { - params.fit_params_target = cmd_params_defaults.fit_params_target; - } - if (params.fit_params_min_ctx.empty()) { - params.fit_params_min_ctx = cmd_params_defaults.fit_params_min_ctx; - } - - return params; -} - -struct cmd_params_instance { - std::string model; - int n_prompt; - int n_gen; - int n_depth; - int n_batch; - int n_ubatch; - ggml_type type_k; - ggml_type type_v; - int n_threads; - std::string cpu_mask; - bool cpu_strict; - int poll; - int n_gpu_layers; - int n_cpu_moe; - llama_split_mode split_mode; - std::string reduction_provider; - int main_gpu; - bool no_kv_offload; - bool flash_attn; - std::vector devices; - std::vector tensor_split; - std::vector tensor_buft_overrides; - bool use_mmap; - bool use_direct_io; - bool embeddings; - bool no_op_offload; - bool no_host; - size_t fit_target; - uint32_t fit_min_ctx; - - llama_model_params to_llama_mparams() const { - llama_model_params mparams = llama_model_default_params(); - - mparams.n_gpu_layers = n_gpu_layers; - if (!devices.empty()) { - mparams.devices = const_cast(devices.data()); - } - mparams.split_mode = split_mode; - mparams.main_gpu = main_gpu; - mparams.tensor_split = tensor_split.data(); - mparams.use_mmap = use_mmap; - mparams.use_direct_io = use_direct_io; - mparams.no_host = no_host; - - if (n_cpu_moe <= 0) { - if (tensor_buft_overrides.empty()) { - mparams.tensor_buft_overrides = nullptr; - } else { - GGML_ASSERT(tensor_buft_overrides.back().pattern == nullptr && - "Tensor buffer overrides not terminated with empty pattern"); - mparams.tensor_buft_overrides = tensor_buft_overrides.data(); - } - } else { - static std::vector merged; - static std::vector patterns; - - merged.clear(); - patterns.clear(); - - auto first = tensor_buft_overrides.begin(); - auto last = tensor_buft_overrides.end(); - if (first != last && (last - 1)->pattern == nullptr) { - --last; - } - merged.insert(merged.end(), first, last); - - patterns.reserve((size_t) n_cpu_moe); - merged.reserve(merged.size() + (size_t) n_cpu_moe + 1); - - for (int i = 0; i < n_cpu_moe; ++i) { - patterns.push_back(llm_ffn_exps_block_regex(i)); - merged.push_back({ patterns.back().c_str(), - ggml_backend_cpu_buffer_type() }); - } - - merged.push_back({ nullptr, nullptr }); - - mparams.tensor_buft_overrides = merged.data(); - } - - return mparams; - } - - bool equal_mparams(const cmd_params_instance & other) const { - return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && - split_mode == other.split_mode && reduction_provider == other.reduction_provider && - main_gpu == other.main_gpu && tensor_split == other.tensor_split && - use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && - devices == other.devices && - no_host == other.no_host && - vec_tensor_buft_override_equal(tensor_buft_overrides, other.tensor_buft_overrides); - } - - llama_context_params to_llama_cparams() const { - llama_context_params cparams = llama_context_default_params(); - - cparams.n_ctx = n_prompt + n_gen + n_depth; - cparams.n_batch = n_batch; - cparams.n_ubatch = n_ubatch; - cparams.type_k = type_k; - cparams.type_v = type_v; - cparams.offload_kqv = !no_kv_offload; - cparams.flash_attn_type = flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED : LLAMA_FLASH_ATTN_TYPE_DISABLED; - cparams.embeddings = embeddings; - cparams.op_offload = !no_op_offload; - cparams.swa_full = false; - - return cparams; - } -}; - -static std::vector get_cmd_params_instances(const cmd_params & params) { - std::vector instances; - - // this ordering minimizes the number of times that each model needs to be reloaded - // clang-format off - for (const auto & m : params.model) - for (const auto & fpt : params.fit_params_target) - for (const auto & fpc : params.fit_params_min_ctx) - for (const auto & nl : params.n_gpu_layers) - for (const auto & ncmoe : params.n_cpu_moe) - for (const auto & sm : params.split_mode) - for (const auto & rp : params.reduction_provider) - for (const auto & mg : params.main_gpu) - for (const auto & devs : params.devices) - for (const auto & ts : params.tensor_split) - for (const auto & ot : params.tensor_buft_overrides) - for (const auto & mmp : params.use_mmap) - for (const auto & dio : params.use_direct_io) - for (const auto & noh : params.no_host) - for (const auto & embd : params.embeddings) - for (const auto & nopo : params.no_op_offload) - for (const auto & nb : params.n_batch) - for (const auto & nub : params.n_ubatch) - for (const auto & tk : params.type_k) - for (const auto & tv : params.type_v) - for (const auto & nkvo : params.no_kv_offload) - for (const auto & fa : params.flash_attn) - for (const auto & nt : params.n_threads) - for (const auto & cm : params.cpu_mask) - for (const auto & cs : params.cpu_strict) - for (const auto & nd : params.n_depth) - for (const auto & pl : params.poll) { - for (const auto & n_prompt : params.n_prompt) { - if (n_prompt == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ n_prompt, - /* .n_gen = */ 0, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .reduction_provider = */ rp, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - - for (const auto & n_gen : params.n_gen) { - if (n_gen == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ 0, - /* .n_gen = */ n_gen, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .reduction_provider = */ rp, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - - for (const auto & n_pg : params.n_pg) { - if (n_pg.first == 0 && n_pg.second == 0) { - continue; - } - cmd_params_instance instance = { - /* .model = */ m, - /* .n_prompt = */ n_pg.first, - /* .n_gen = */ n_pg.second, - /* .n_depth = */ nd, - /* .n_batch = */ nb, - /* .n_ubatch = */ nub, - /* .type_k = */ tk, - /* .type_v = */ tv, - /* .n_threads = */ nt, - /* .cpu_mask = */ cm, - /* .cpu_strict = */ cs, - /* .poll = */ pl, - /* .n_gpu_layers = */ nl, - /* .n_cpu_moe = */ ncmoe, - /* .split_mode = */ sm, - /* .reduction_provider = */ rp, - /* .main_gpu = */ mg, - /* .no_kv_offload= */ nkvo, - /* .flash_attn = */ fa, - /* .devices = */ devs, - /* .tensor_split = */ ts, - /* .tensor_buft_overrides = */ ot, - /* .use_mmap = */ mmp, - /* .use_direct_io= */ dio, - /* .embeddings = */ embd, - /* .no_op_offload= */ nopo, - /* .no_host = */ noh, - /* .fit_target = */ fpt, - /* .fit_min_ctx = */ fpc, - }; - instances.push_back(instance); - } - } - // clang-format on - - return instances; -} - -struct test { - static const std::string build_commit; - static const int build_number; - const std::string cpu_info; - const std::string gpu_info; - std::string model_filename; - std::string model_type; - uint64_t model_size; - uint64_t model_n_params; - int n_batch; - int n_ubatch; - int n_threads; - std::string cpu_mask; - bool cpu_strict; - int poll; - ggml_type type_k; - ggml_type type_v; - int n_gpu_layers; - int n_cpu_moe; - llama_split_mode split_mode; - std::string reduction_provider; - int main_gpu; - bool no_kv_offload; - bool flash_attn; - std::vector devices; - std::vector tensor_split; - std::vector tensor_buft_overrides; - bool use_mmap; - bool use_direct_io; - bool embeddings; - bool no_op_offload; - bool no_host; - size_t fit_target; - uint32_t fit_min_ctx; - int n_prompt; - int n_gen; - int n_depth; - std::string test_time; - std::vector samples_ns; - - test(const cmd_params_instance & inst, const llama_model * lmodel, const llama_context * ctx) : - cpu_info(get_cpu_info()), - gpu_info(get_gpu_info()) { - - model_filename = inst.model; - char buf[128]; - llama_model_desc(lmodel, buf, sizeof(buf)); - model_type = buf; - model_size = llama_model_size(lmodel); - model_n_params = llama_model_n_params(lmodel); - n_batch = inst.n_batch; - n_ubatch = inst.n_ubatch; - n_threads = inst.n_threads; - cpu_mask = inst.cpu_mask; - cpu_strict = inst.cpu_strict; - poll = inst.poll; - type_k = inst.type_k; - type_v = inst.type_v; - n_gpu_layers = inst.n_gpu_layers; - n_cpu_moe = inst.n_cpu_moe; - split_mode = inst.split_mode; - reduction_provider = inst.reduction_provider; - main_gpu = inst.main_gpu; - no_kv_offload = inst.no_kv_offload; - flash_attn = inst.flash_attn; - devices = inst.devices; - tensor_split = inst.tensor_split; - tensor_buft_overrides = inst.tensor_buft_overrides; - use_mmap = inst.use_mmap; - use_direct_io = inst.use_direct_io; - embeddings = inst.embeddings; - no_op_offload = inst.no_op_offload; - no_host = inst.no_host; - fit_target = inst.fit_target; - fit_min_ctx = inst.fit_min_ctx; - n_prompt = inst.n_prompt; - n_gen = inst.n_gen; - n_depth = inst.n_depth; - // RFC 3339 date-time format - time_t t = time(NULL); - std::strftime(buf, sizeof(buf), "%FT%TZ", gmtime(&t)); - test_time = buf; - - (void) ctx; - } - - uint64_t avg_ns() const { return ::avg(samples_ns); } - - uint64_t stdev_ns() const { return ::stdev(samples_ns); } - - std::vector get_ts() const { - int n_tokens = n_prompt + n_gen; - std::vector ts; - std::transform(samples_ns.begin(), samples_ns.end(), std::back_inserter(ts), - [n_tokens](uint64_t t) { return 1e9 * n_tokens / t; }); - return ts; - } - - double avg_ts() const { return ::avg(get_ts()); } - - double stdev_ts() const { return ::stdev(get_ts()); } - - static std::string get_backend() { - std::vector backends; - bool rpc_used = false; - for (size_t i = 0; i < ggml_backend_reg_count(); i++) { - auto * reg = ggml_backend_reg_get(i); - std::string name = ggml_backend_reg_name(reg); - if (string_starts_with(name, "RPC")) { - if (ggml_backend_reg_dev_count(reg) > 0) { - rpc_used = true; - } - } else { - if (name != "CPU") { - backends.push_back(ggml_backend_reg_name(reg)); - } - } - } - if (rpc_used) { - backends.push_back("RPC"); - } - return backends.empty() ? "CPU" : join(backends, ","); - } - - static const std::vector & get_fields() { - static const std::vector fields = { - "build_commit", "build_number", "cpu_info", "gpu_info", "backends", - "model_filename", "model_type", "model_size", "model_n_params", "n_batch", - "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", - "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", - "reduction_provider", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", - "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", - "no_op_offload", "no_host", "fit_target", "fit_min_ctx", - "n_prompt", "n_gen", "n_depth", - "test_time", "avg_ns", "stddev_ns", "avg_ts", "stddev_ts" - }; - return fields; - } - - enum field_type { STRING, BOOL, INT, FLOAT }; - - static field_type get_field_type(const std::string & field) { - if (field == "build_number" || field == "n_batch" || field == "n_ubatch" || field == "n_threads" || - field == "poll" || field == "model_size" || field == "model_n_params" || field == "n_gpu_layers" || - field == "main_gpu" || field == "n_prompt" || field == "n_gen" || field == "n_depth" || field == "avg_ns" || - field == "stddev_ns" || field == "no_op_offload" || field == "n_cpu_moe" || - field == "fit_target" || field == "fit_min_ctx") { - return INT; - } - if (field == "f16_kv" || field == "no_kv_offload" || field == "cpu_strict" || field == "flash_attn" || - field == "use_mmap" || field == "use_direct_io" || field == "embeddings" || field == "no_host") { - return BOOL; - } - if (field == "avg_ts" || field == "stddev_ts") { - return FLOAT; - } - return STRING; - } - - std::vector get_values() const { - std::string tensor_split_str; - std::string tensor_buft_overrides_str; - int max_nonzero = 0; - for (size_t i = 0; i < llama_max_devices(); i++) { - if (tensor_split[i] > 0) { - max_nonzero = i; - } - } - for (int i = 0; i <= max_nonzero; i++) { - char buf[32]; - snprintf(buf, sizeof(buf), "%.2f", tensor_split[i]); - tensor_split_str += buf; - if (i < max_nonzero) { - tensor_split_str += "/"; - } - } - if (tensor_buft_overrides.size() == 1) { - // Last element of tensor_buft_overrides is always a null pattern - // so if it is only one element long, it must be a null pattern. - GGML_ASSERT(tensor_buft_overrides[0].pattern == nullptr); - tensor_buft_overrides_str += "none"; - } else { - for (size_t i = 0; i < tensor_buft_overrides.size()-1; i++) { - // Last element of tensor_buft_overrides is always a null pattern - if (tensor_buft_overrides[i].pattern == nullptr) { - tensor_buft_overrides_str += "none"; - } else { - tensor_buft_overrides_str += tensor_buft_overrides[i].pattern; - tensor_buft_overrides_str += "="; - tensor_buft_overrides_str += ggml_backend_buft_name(tensor_buft_overrides[i].buft); - } - if (i + 2 < tensor_buft_overrides.size()) { - tensor_buft_overrides_str += ";"; - } - } - } - std::vector values = { build_commit, - std::to_string(build_number), - cpu_info, - gpu_info, - get_backend(), - model_filename, - model_type, - std::to_string(model_size), - std::to_string(model_n_params), - std::to_string(n_batch), - std::to_string(n_ubatch), - std::to_string(n_threads), - cpu_mask, - std::to_string(cpu_strict), - std::to_string(poll), - ggml_type_name(type_k), - ggml_type_name(type_v), - std::to_string(n_gpu_layers), - std::to_string(n_cpu_moe), - split_mode_str(split_mode), - reduction_provider, - std::to_string(main_gpu), - std::to_string(no_kv_offload), - std::to_string(flash_attn), - devices_to_string(devices), - tensor_split_str, - tensor_buft_overrides_str, - std::to_string(use_mmap), - std::to_string(use_direct_io), - std::to_string(embeddings), - std::to_string(no_op_offload), - std::to_string(no_host), - std::to_string(fit_target), - std::to_string(fit_min_ctx), - std::to_string(n_prompt), - std::to_string(n_gen), - std::to_string(n_depth), - test_time, - std::to_string(avg_ns()), - std::to_string(stdev_ns()), - std::to_string(avg_ts()), - std::to_string(stdev_ts()) }; - return values; - } - - std::map get_map() const { - std::map map; - auto fields = get_fields(); - auto values = get_values(); - std::transform(fields.begin(), fields.end(), values.begin(), std::inserter(map, map.end()), - std::make_pair); - return map; - } -}; - -const std::string test::build_commit = llama_commit(); -const int test::build_number = llama_build_number(); - -struct printer { - virtual ~printer() {} - - FILE * fout; - - virtual void print_header(const cmd_params & params) { (void) params; } - - virtual void print_test(const test & t) = 0; - - virtual void print_footer() {} -}; - -struct csv_printer : public printer { - static std::string escape_csv(const std::string & field) { - std::string escaped = "\""; - for (auto c : field) { - if (c == '"') { - escaped += "\""; - } - escaped += c; - } - escaped += "\""; - return escaped; - } - - void print_header(const cmd_params & params) override { - std::vector fields = test::get_fields(); - fprintf(fout, "%s\n", join(fields, ",").c_str()); - (void) params; - } - - void print_test(const test & t) override { - std::vector values = t.get_values(); - std::transform(values.begin(), values.end(), values.begin(), escape_csv); - fprintf(fout, "%s\n", join(values, ",").c_str()); - } -}; - -static std::string escape_json(const std::string & value) { - std::string escaped; - for (auto c : value) { - if (c == '"') { - escaped += "\\\""; - } else if (c == '\\') { - escaped += "\\\\"; - } else if (c <= 0x1f) { - char buf[8]; - snprintf(buf, sizeof(buf), "\\u%04x", c); - escaped += buf; - } else { - escaped += c; - } - } - return escaped; -} - -static std::string format_json_value(const std::string & field, const std::string & value) { - switch (test::get_field_type(field)) { - case test::STRING: - return "\"" + escape_json(value) + "\""; - case test::BOOL: - return value == "0" ? "false" : "true"; - default: - return value; - } -} - -struct json_printer : public printer { - bool first = true; - - void print_header(const cmd_params & params) override { - fprintf(fout, "[\n"); - (void) params; - } - - void print_fields(const std::vector & fields, const std::vector & values) { - assert(fields.size() == values.size()); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, " \"%s\": %s,\n", fields.at(i).c_str(), - format_json_value(fields.at(i), values.at(i)).c_str()); - } - } - - void print_test(const test & t) override { - if (first) { - first = false; - } else { - fprintf(fout, ",\n"); - } - fprintf(fout, " {\n"); - print_fields(test::get_fields(), t.get_values()); - fprintf(fout, " \"samples_ns\": [ %s ],\n", join(t.samples_ns, ", ").c_str()); - fprintf(fout, " \"samples_ts\": [ %s ]\n", join(t.get_ts(), ", ").c_str()); - fprintf(fout, " }"); - fflush(fout); - } - - void print_footer() override { fprintf(fout, "\n]\n"); } -}; - -struct jsonl_printer : public printer { - void print_fields(const std::vector & fields, const std::vector & values) { - assert(fields.size() == values.size()); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, "\"%s\": %s, ", fields.at(i).c_str(), format_json_value(fields.at(i), values.at(i)).c_str()); - } - } - - void print_test(const test & t) override { - fprintf(fout, "{"); - print_fields(test::get_fields(), t.get_values()); - fprintf(fout, "\"samples_ns\": [ %s ],", join(t.samples_ns, ", ").c_str()); - fprintf(fout, "\"samples_ts\": [ %s ]", join(t.get_ts(), ", ").c_str()); - fprintf(fout, "}\n"); - fflush(fout); - } -}; - -struct markdown_printer : public printer { - std::vector fields; - - static int get_field_width(const std::string & field) { - if (field == "model") { - return -30; - } - if (field == "t/s") { - return 20; - } - if (field == "size" || field == "params") { - return 10; - } - if (field == "n_gpu_layers") { - return 3; - } - if (field == "n_threads") { - return 7; - } - if (field == "n_batch") { - return 7; - } - if (field == "n_ubatch") { - return 8; - } - if (field == "type_k" || field == "type_v") { - return 6; - } - if (field == "split_mode") { - return 6; - } - if (field == "flash_attn") { - return 2; - } - if (field == "devices") { - return -12; - } - if (field == "use_mmap") { - return 4; - } - if (field == "use_direct_io") { - return 3; - } - if (field == "test") { - return 15; - } - if (field == "no_op_offload") { - return 4; - } - if (field == "no_host") { - return 4; - } - - int width = std::max((int) field.length(), 10); - - if (test::get_field_type(field) == test::STRING) { - return -width; - } - return width; - } - - static std::string get_field_display_name(const std::string & field) { - if (field == "n_gpu_layers") { - return "ngl"; - } - if (field == "split_mode") { - return "sm"; - } - if (field == "n_threads") { - return "threads"; - } - if (field == "no_kv_offload") { - return "nkvo"; - } - if (field == "flash_attn") { - return "fa"; - } - if (field == "use_mmap") { - return "mmap"; - } - if (field == "use_direct_io") { - return "dio"; - } - if (field == "embeddings") { - return "embd"; - } - if (field == "no_op_offload") { - return "nopo"; - } - if (field == "no_host") { - return "noh"; - } - if (field == "devices") { - return "dev"; - } - if (field == "tensor_split") { - return "ts"; - } - if (field == "tensor_buft_overrides") { - return "ot"; - } - if (field == "fit_target") { - return "fitt"; - } - if (field == "fit_min_ctx") { - return "fitc"; - } - return field; - } - - void print_header(const cmd_params & params) override { - // select fields to print - fields.emplace_back("model"); - fields.emplace_back("size"); - fields.emplace_back("params"); - fields.emplace_back("backend"); - bool is_cpu_backend = test::get_backend().find("CPU") != std::string::npos || - test::get_backend().find("BLAS") != std::string::npos || - test::get_backend().find("ZenDNN") != std::string::npos; - if (!is_cpu_backend) { - fields.emplace_back("n_gpu_layers"); - } - if (params.n_cpu_moe.size() > 1 || params.n_cpu_moe != cmd_params_defaults.n_cpu_moe) { - fields.emplace_back("n_cpu_moe"); - } - if (params.n_threads.size() > 1 || params.n_threads != cmd_params_defaults.n_threads || is_cpu_backend) { - fields.emplace_back("n_threads"); - } - if (params.cpu_mask.size() > 1 || params.cpu_mask != cmd_params_defaults.cpu_mask) { - fields.emplace_back("cpu_mask"); - } - if (params.cpu_strict.size() > 1 || params.cpu_strict != cmd_params_defaults.cpu_strict) { - fields.emplace_back("cpu_strict"); - } - if (params.poll.size() > 1 || params.poll != cmd_params_defaults.poll) { - fields.emplace_back("poll"); - } - if (params.n_batch.size() > 1 || params.n_batch != cmd_params_defaults.n_batch) { - fields.emplace_back("n_batch"); - } - if (params.n_ubatch.size() > 1 || params.n_ubatch != cmd_params_defaults.n_ubatch) { - fields.emplace_back("n_ubatch"); - } - if (params.type_k.size() > 1 || params.type_k != cmd_params_defaults.type_k) { - fields.emplace_back("type_k"); - } - if (params.type_v.size() > 1 || params.type_v != cmd_params_defaults.type_v) { - fields.emplace_back("type_v"); - } - if (params.main_gpu.size() > 1 || params.main_gpu != cmd_params_defaults.main_gpu) { - fields.emplace_back("main_gpu"); - } - if (params.split_mode.size() > 1 || params.split_mode != cmd_params_defaults.split_mode) { - fields.emplace_back("split_mode"); - } - if (params.no_kv_offload.size() > 1 || params.no_kv_offload != cmd_params_defaults.no_kv_offload) { - fields.emplace_back("no_kv_offload"); - } - if (params.flash_attn.size() > 1 || params.flash_attn != cmd_params_defaults.flash_attn) { - fields.emplace_back("flash_attn"); - } - if (params.devices.size() > 1 || params.devices != cmd_params_defaults.devices) { - fields.emplace_back("devices"); - } - if (params.tensor_split.size() > 1 || params.tensor_split != cmd_params_defaults.tensor_split) { - fields.emplace_back("tensor_split"); - } - if (params.tensor_buft_overrides.size() > 1 || !vec_vec_tensor_buft_override_equal(params.tensor_buft_overrides, cmd_params_defaults.tensor_buft_overrides)) { - fields.emplace_back("tensor_buft_overrides"); - } - if (params.use_mmap.size() > 1 || params.use_mmap != cmd_params_defaults.use_mmap) { - fields.emplace_back("use_mmap"); - } - if (params.use_direct_io.size() > 1 || params.use_direct_io != cmd_params_defaults.use_direct_io) { - fields.emplace_back("use_direct_io"); - } - if (params.embeddings.size() > 1 || params.embeddings != cmd_params_defaults.embeddings) { - fields.emplace_back("embeddings"); - } - if (params.no_op_offload.size() > 1 || params.no_op_offload != cmd_params_defaults.no_op_offload) { - fields.emplace_back("no_op_offload"); - } - if (params.no_host.size() > 1 || params.no_host != cmd_params_defaults.no_host) { - fields.emplace_back("no_host"); - } - if (params.fit_params_target.size() > 1 || params.fit_params_target != cmd_params_defaults.fit_params_target) { - fields.emplace_back("fit_target"); - } - if (params.fit_params_min_ctx.size() > 1 || params.fit_params_min_ctx != cmd_params_defaults.fit_params_min_ctx) { - fields.emplace_back("fit_min_ctx"); - } - fields.emplace_back("test"); - fields.emplace_back("t/s"); - - fprintf(fout, "|"); - for (const auto & field : fields) { - fprintf(fout, " %*s |", get_field_width(field), get_field_display_name(field).c_str()); - } - fprintf(fout, "\n"); - fprintf(fout, "|"); - for (const auto & field : fields) { - int width = get_field_width(field); - fprintf(fout, " %s%s |", std::string(std::abs(width) - 1, '-').c_str(), width > 0 ? ":" : "-"); - } - fprintf(fout, "\n"); - } - - void print_test(const test & t) override { - std::map vmap = t.get_map(); - - fprintf(fout, "|"); - for (const auto & field : fields) { - std::string value; - char buf[128]; - if (field == "model") { - value = t.model_type; - } else if (field == "size") { - if (t.model_size < 1024 * 1024 * 1024) { - snprintf(buf, sizeof(buf), "%.2f MiB", t.model_size / 1024.0 / 1024.0); - } else { - snprintf(buf, sizeof(buf), "%.2f GiB", t.model_size / 1024.0 / 1024.0 / 1024.0); - } - value = buf; - } else if (field == "params") { - if (t.model_n_params < 1000 * 1000 * 1000) { - snprintf(buf, sizeof(buf), "%.2f M", t.model_n_params / 1e6); - } else { - snprintf(buf, sizeof(buf), "%.2f B", t.model_n_params / 1e9); - } - value = buf; - } else if (field == "backend") { - value = test::get_backend(); - } else if (field == "test") { - if (t.n_prompt > 0 && t.n_gen == 0) { - snprintf(buf, sizeof(buf), "pp%d", t.n_prompt); - } else if (t.n_gen > 0 && t.n_prompt == 0) { - snprintf(buf, sizeof(buf), "tg%d", t.n_gen); - } else { - snprintf(buf, sizeof(buf), "pp%d+tg%d", t.n_prompt, t.n_gen); - } - if (t.n_depth > 0) { - int len = strlen(buf); - snprintf(buf + len, sizeof(buf) - len, " @ d%d", t.n_depth); - } - value = buf; - } else if (field == "t/s") { - snprintf(buf, sizeof(buf), "%.2f ± %.2f", t.avg_ts(), t.stdev_ts()); - value = buf; - } else if (vmap.find(field) != vmap.end()) { - value = vmap.at(field); - } else { - assert(false); - exit(1); - } - - int width = get_field_width(field); - if (field == "t/s") { - // HACK: the utf-8 character is 2 bytes - width += 1; - } - fprintf(fout, " %*s |", width, value.c_str()); - } - fprintf(fout, "\n"); - } - - void print_footer() override { - fprintf(fout, "\nbuild: %s (%d)\n", test::build_commit.c_str(), test::build_number); - } -}; - -struct sql_printer : public printer { - static std::string get_sql_field_type(const std::string & field) { - switch (test::get_field_type(field)) { - case test::STRING: - return "TEXT"; - case test::BOOL: - case test::INT: - return "INTEGER"; - case test::FLOAT: - return "REAL"; - default: - assert(false); - exit(1); - } - } - - void print_header(const cmd_params & params) override { - std::vector fields = test::get_fields(); - fprintf(fout, "CREATE TABLE IF NOT EXISTS llama_bench (\n"); - for (size_t i = 0; i < fields.size(); i++) { - fprintf(fout, " %s %s%s\n", fields.at(i).c_str(), get_sql_field_type(fields.at(i)).c_str(), - i < fields.size() - 1 ? "," : ""); - } - fprintf(fout, ");\n"); - fprintf(fout, "\n"); - (void) params; - } - - void print_test(const test & t) override { - fprintf(fout, "INSERT INTO llama_bench (%s) ", join(test::get_fields(), ", ").c_str()); - fprintf(fout, "VALUES ("); - std::vector values = t.get_values(); - for (size_t i = 0; i < values.size(); i++) { - fprintf(fout, "'%s'%s", values.at(i).c_str(), i < values.size() - 1 ? ", " : ""); - } - fprintf(fout, ");\n"); - } -}; - -struct ctx_state { - int depth = 0; // in tokens - - std::vector buf; // the llama_context state buffer -}; - -static bool test_prompt(llama_context * ctx, int n_prompt, int n_batch, int n_threads) { - llama_set_n_threads(ctx, n_threads, n_threads); - - const llama_model * model = llama_get_model(ctx); - const llama_vocab * vocab = llama_model_get_vocab(model); - const int32_t n_vocab = llama_vocab_n_tokens(vocab); - - std::vector tokens(n_batch); - - int n_processed = 0; - - while (n_processed < n_prompt) { - int n_tokens = std::min(n_prompt - n_processed, n_batch); - tokens[0] = n_processed == 0 && llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; - for (int i = 1; i < n_tokens; i++) { - tokens[i] = std::rand() % n_vocab; - } - int res = llama_decode(ctx, llama_batch_get_one(tokens.data(), n_tokens)); - if (res != 0) { - fprintf(stderr, "%s: failed to decode prompt batch, res = %d\n", __func__, res); - return false; - } - n_processed += n_tokens; - } - - llama_synchronize(ctx); - return true; -} - -static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { - llama_set_n_threads(ctx, n_threads, n_threads); - - const llama_model * model = llama_get_model(ctx); - const llama_vocab * vocab = llama_model_get_vocab(model); - const int32_t n_vocab = llama_vocab_n_tokens(vocab); - - llama_token token = llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; - - for (int i = 0; i < n_gen; i++) { - int res = llama_decode(ctx, llama_batch_get_one(&token, 1)); - if (res != 0) { - fprintf(stderr, "%s: failed to decode generation batch, res = %d\n", __func__, res); - return false; - } - llama_synchronize(ctx); - token = std::rand() % n_vocab; - } - return true; -} - -static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { - (void) user_data; - if (level >= GGML_LOG_LEVEL_WARN) { - fputs(text, stderr); - } -} - -static std::unique_ptr create_printer(output_formats format) { - switch (format) { - case NONE: - return nullptr; - case CSV: - return std::unique_ptr(new csv_printer()); - case JSON: - return std::unique_ptr(new json_printer()); - case JSONL: - return std::unique_ptr(new jsonl_printer()); - case MARKDOWN: - return std::unique_ptr(new markdown_printer()); - case SQL: - return std::unique_ptr(new sql_printer()); - } - GGML_ABORT("fatal error"); -} - -int main(int argc, char ** argv) { - std::setlocale(LC_NUMERIC, "C"); - // try to set locale for unicode characters in markdown - std::setlocale(LC_CTYPE, ".UTF-8"); - -#if !defined(NDEBUG) - fprintf(stderr, "warning: asserts enabled, performance may be affected\n"); -#endif - -#if (defined(_MSC_VER) && defined(_DEBUG)) || (!defined(_MSC_VER) && !defined(__OPTIMIZE__)) - fprintf(stderr, "warning: debug build, performance may be affected\n"); -#endif - -#if defined(__SANITIZE_ADDRESS__) || defined(__SANITIZE_THREAD__) - fprintf(stderr, "warning: sanitizer enabled, performance may be affected\n"); -#endif - - // initialize backends - ggml_backend_load_all(); - - cmd_params params = parse_cmd_params(argc, argv); - - auto * cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU); - if (!cpu_dev) { - fprintf(stderr, "%s: error: CPU backend is not loaded\n", __func__); - return 1; - } - auto * cpu_reg = ggml_backend_dev_backend_reg(cpu_dev); - auto * ggml_threadpool_new_fn = (decltype(ggml_threadpool_new) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_new"); - auto * ggml_threadpool_free_fn = (decltype(ggml_threadpool_free) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_free"); - - // initialize llama.cpp - if (!params.verbose) { - llama_log_set(llama_null_log_callback, NULL); - } - llama_backend_init(); - llama_numa_init(params.numa); - - if (!set_process_priority(params.prio)) { - fprintf(stderr, "%s: error: failed to set process priority\n", __func__); - return 1; - } - - // initialize printer - std::unique_ptr p = create_printer(params.output_format); - std::unique_ptr p_err = create_printer(params.output_format_stderr); - - if (p) { - p->fout = stdout; - p->print_header(params); - } - - if (p_err) { - p_err->fout = stderr; - p_err->print_header(params); - } - - std::vector params_instances = get_cmd_params_instances(params); - - llama_model * lmodel = nullptr; - const cmd_params_instance * prev_inst = nullptr; - - // store the llama_context state at the previous depth that we performed a test - // ref: https://github.com/ggml-org/llama.cpp/pull/16944#issuecomment-3478151721 - ctx_state cstate; - - int params_idx = 0; - auto params_count = params_instances.size(); - for (const auto & inst : params_instances) { - params_idx++; - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: starting\n", params_idx, params_count); - } - auto mparams = inst.to_llama_mparams(); - auto cparams = inst.to_llama_cparams(); - - bool do_fit = inst.fit_target != cmd_params_defaults.fit_params_target[0] || - inst.fit_min_ctx != cmd_params_defaults.fit_params_min_ctx[0]; - - std::vector fit_tensor_split(llama_max_devices(), 0.0f); - std::vector fit_overrides(llama_max_tensor_buft_overrides(), {nullptr, nullptr}); - - if (do_fit) { - // free the previous model so fit sees full free VRAM - if (lmodel) { - llama_model_free(lmodel); - lmodel = nullptr; - prev_inst = nullptr; - } - - // use default n_gpu_layers and n_ctx so common_fit_params can adjust them - mparams.n_gpu_layers = llama_model_default_params().n_gpu_layers; - mparams.tensor_split = fit_tensor_split.data(); - mparams.tensor_buft_overrides = fit_overrides.data(); - cparams.n_ctx = 0; - - std::vector margins(llama_max_devices(), inst.fit_target * 1024 * 1024); - - uint32_t n_ctx_needed = inst.n_prompt + inst.n_gen + inst.n_depth; - cparams.n_ctx = std::max(cparams.n_ctx, n_ctx_needed); - - common_fit_params(inst.model.c_str(), &mparams, &cparams, - fit_tensor_split.data(), - fit_overrides.data(), - margins.data(), - inst.fit_min_ctx, - params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); - } - - // set reduction provider env var before model load (comm_init reads it) - { - const char * ar_val = (inst.reduction_provider == "auto") ? "" : inst.reduction_provider.c_str(); -#ifdef _WIN32 - _putenv_s("GGML_CUDA_ALLREDUCE", ar_val); -#else - setenv("GGML_CUDA_ALLREDUCE", ar_val, 1); -#endif - } - - // keep the same model between tests when possible - if (!lmodel || !prev_inst || !inst.equal_mparams(*prev_inst)) { - if (lmodel) { - llama_model_free(lmodel); - } - - lmodel = llama_model_load_from_file(inst.model.c_str(), mparams); - if (lmodel == NULL) { - fprintf(stderr, "%s: error: failed to load model '%s'\n", __func__, inst.model.c_str()); - return 1; - } - prev_inst = &inst; - } - - llama_context * ctx = llama_init_from_model(lmodel, cparams); - if (ctx == NULL) { - fprintf(stderr, "%s: error: failed to create context with model '%s'\n", __func__, inst.model.c_str()); - llama_model_free(lmodel); - return 1; - } - - test t(inst, lmodel, ctx); - - llama_memory_clear(llama_get_memory(ctx), false); - - // cool off before the test - if (params.delay) { - std::this_thread::sleep_for(std::chrono::seconds(params.delay)); - } - - struct ggml_threadpool_params tpp = ggml_threadpool_params_default(t.n_threads); - if (!parse_cpu_mask(t.cpu_mask, tpp.cpumask)) { - fprintf(stderr, "%s: failed to parse cpu-mask: %s\n", __func__, t.cpu_mask.c_str()); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - tpp.strict_cpu = t.cpu_strict; - tpp.poll = t.poll; - tpp.prio = params.prio; - - struct ggml_threadpool * threadpool = ggml_threadpool_new_fn(&tpp); - if (!threadpool) { - fprintf(stderr, "%s: threadpool create failed : n_threads %d\n", __func__, tpp.n_threads); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - - llama_attach_threadpool(ctx, threadpool, NULL); - - // warmup run - if (!params.no_warmup) { - if (t.n_prompt > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup prompt run\n", params_idx, params_count); - } - //test_prompt(ctx, std::min(t.n_batch, std::min(t.n_prompt, 32)), 0, t.n_batch, t.n_threads); - bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run prompt warmup\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - if (t.n_gen > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup generation run\n", params_idx, params_count); - } - bool res = test_gen(ctx, 1, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run gen warmup\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - } - - for (int i = 0; i < params.reps; i++) { - llama_memory_clear(llama_get_memory(ctx), false); - - if (t.n_depth > 0) { - bool is_cached = t.n_depth == cstate.depth; - - if (is_cached) { - // if previously we have computed at this depth, just restore the state - const size_t ret = llama_state_seq_set_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); - if (ret == 0) { - // if the old state is incompatible with the current context - reprocess from scratch - is_cached = false; - } - } - - if (!is_cached) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_prompt(ctx, t.n_depth, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run depth\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - - // store the context state for reuse in later runs - cstate.depth = t.n_depth; - cstate.buf.resize(llama_state_seq_get_size(ctx, 0)); - llama_state_seq_get_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); - } else { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d (cached)\n", params_idx, params_count, - i + 1, params.reps); - } - } - } - - uint64_t t_start = get_time_ns(); - - if (t.n_prompt > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: prompt run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run prompt\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - if (t.n_gen > 0) { - if (params.progress) { - fprintf(stderr, "llama-bench: benchmark %d/%zu: generation run %d/%d\n", params_idx, params_count, - i + 1, params.reps); - } - bool res = test_gen(ctx, t.n_gen, t.n_threads); - if (!res) { - fprintf(stderr, "%s: error: failed to run gen\n", __func__); - llama_free(ctx); - llama_model_free(lmodel); - exit(1); - } - } - - uint64_t t_ns = get_time_ns() - t_start; - t.samples_ns.push_back(t_ns); - } - - if (p) { - p->print_test(t); - fflush(p->fout); - } - - if (p_err) { - p_err->print_test(t); - fflush(p_err->fout); - } - - llama_perf_context_print(ctx); - - llama_free(ctx); - - ggml_threadpool_free_fn(threadpool); - } - - llama_model_free(lmodel); - - if (p) { - p->print_footer(); - } - - if (p_err) { - p_err->print_footer(); - } - - llama_backend_free(); - - return 0; -} +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "build-info.h" +#include "common.h" +#include "download.h" +#include "fit.h" +#include "ggml.h" +#include "llama.h" + +#ifdef _WIN32 +# define WIN32_LEAN_AND_MEAN +# ifndef NOMINMAX +# define NOMINMAX +# endif +# include +#endif + +// utils +static uint64_t get_time_ns() { + using clock = std::chrono::high_resolution_clock; + return std::chrono::nanoseconds(clock::now().time_since_epoch()).count(); +} + +static bool tensor_buft_override_equal(const llama_model_tensor_buft_override& a, const llama_model_tensor_buft_override& b) { + if (a.pattern != b.pattern) { + // cString comparison that may be null + if (a.pattern == nullptr || b.pattern == nullptr) { + return false; + } + if (strcmp(a.pattern, b.pattern) != 0) { + return false; + } + } + if (a.buft != b.buft) { + return false; + } + return true; +} + +static bool vec_tensor_buft_override_equal(const std::vector& a, const std::vector& b) { + if (a.size() != b.size()) { + return false; + } + for (size_t i = 0; i < a.size(); i++) { + if (!tensor_buft_override_equal(a[i], b[i])) { + return false; + } + } + return true; +} + +static bool vec_vec_tensor_buft_override_equal(const std::vector>& a, const std::vector>& b) { + if (a.size() != b.size()) { + return false; + } + for (size_t i = 0; i < a.size(); i++) { + if (!vec_tensor_buft_override_equal(a[i], b[i])) { + return false; + } + } + return true; +} + +template static std::string join(const std::vector & values, const std::string & delim) { + std::ostringstream str; + for (size_t i = 0; i < values.size(); i++) { + str << values[i]; + if (i < values.size() - 1) { + str << delim; + } + } + return str.str(); +} + +template static std::vector transform_to_str(const std::vector & values, F f) { + std::vector str_values; + std::transform(values.begin(), values.end(), std::back_inserter(str_values), f); + return str_values; +} + +template static T avg(const std::vector & v) { + if (v.empty()) { + return 0; + } + T sum = std::accumulate(v.begin(), v.end(), T(0)); + return sum / (T) v.size(); +} + +template static T stdev(const std::vector & v) { + if (v.size() <= 1) { + return 0; + } + T mean = avg(v); + T sq_sum = std::inner_product(v.begin(), v.end(), v.begin(), T(0)); + T stdev = std::sqrt(sq_sum / (T) (v.size() - 1) - mean * mean * (T) v.size() / (T) (v.size() - 1)); + return stdev; +} + +static std::string get_cpu_info() { + std::vector cpu_list; + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + auto * dev = ggml_backend_dev_get(i); + auto dev_type = ggml_backend_dev_type(dev); + if (dev_type == GGML_BACKEND_DEVICE_TYPE_CPU || dev_type == GGML_BACKEND_DEVICE_TYPE_ACCEL) { + cpu_list.push_back(ggml_backend_dev_description(dev)); + } + } + return join(cpu_list, ", "); +} + +static std::string get_gpu_info() { + std::vector gpu_list; + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + auto * dev = ggml_backend_dev_get(i); + auto dev_type = ggml_backend_dev_type(dev); + if (dev_type == GGML_BACKEND_DEVICE_TYPE_GPU || dev_type == GGML_BACKEND_DEVICE_TYPE_IGPU) { + gpu_list.push_back(ggml_backend_dev_description(dev)); + } + } + return join(gpu_list, ", "); +} + +static std::vector parse_devices_arg(const std::string & value) { + std::vector devices; + std::string trimmed = string_strip(value); + if (trimmed.empty()) { + throw std::invalid_argument("no devices specified"); + } + if (trimmed == "auto") { + return devices; + } + + auto dev_names = string_split(trimmed, '/'); + if (dev_names.size() == 1 && string_strip(dev_names[0]) == "none") { + devices.push_back(nullptr); + return devices; + } + + for (auto & name : dev_names) { + std::string dev_name = string_strip(name); + if (dev_name.empty()) { + throw std::invalid_argument("invalid device specification"); + } + auto * dev = ggml_backend_dev_by_name(dev_name.c_str()); + if (!dev || ggml_backend_dev_type(dev) == GGML_BACKEND_DEVICE_TYPE_CPU) { + throw std::invalid_argument(string_format("invalid device: %s", dev_name.c_str())); + } + devices.push_back(dev); + } + + devices.push_back(nullptr); + return devices; +} + +static void register_rpc_server_list(const std::string & servers) { + auto rpc_servers = string_split(servers, ','); + if (rpc_servers.empty()) { + throw std::invalid_argument("no RPC servers specified"); + } + + auto * rpc_reg = ggml_backend_reg_by_name("RPC"); + if (!rpc_reg) { + throw std::invalid_argument("failed to find RPC backend"); + } + + using add_rpc_server_fn = ggml_backend_reg_t (*)(const char * endpoint); + auto * ggml_backend_rpc_add_server_fn = (add_rpc_server_fn) ggml_backend_reg_get_proc_address(rpc_reg, "ggml_backend_rpc_add_server"); + if (!ggml_backend_rpc_add_server_fn) { + throw std::invalid_argument("failed to find RPC add server function"); + } + for (const auto & server : rpc_servers) { + auto reg = ggml_backend_rpc_add_server_fn(server.c_str()); + ggml_backend_register(reg); + } +} + +static std::string devices_to_string(const std::vector & devices) { + if (devices.empty()) { + return "auto"; + } + + if (devices.size() == 1 && devices[0] == nullptr) { + return "none"; + } + + std::vector names; + for (auto * dev : devices) { + if (dev == nullptr) { + break; + } + names.push_back(ggml_backend_dev_name(dev)); + } + + return join(names, "/"); +} + +// command line params +enum output_formats { NONE, CSV, JSON, JSONL, MARKDOWN, SQL }; + +static const char * output_format_str(output_formats format) { + switch (format) { + case NONE: + return "none"; + case CSV: + return "csv"; + case JSON: + return "json"; + case JSONL: + return "jsonl"; + case MARKDOWN: + return "md"; + case SQL: + return "sql"; + default: + GGML_ABORT("invalid output format"); + } +} + +static bool output_format_from_str(const std::string & s, output_formats & format) { + if (s == "none") { + format = NONE; + } else if (s == "csv") { + format = CSV; + } else if (s == "json") { + format = JSON; + } else if (s == "jsonl") { + format = JSONL; + } else if (s == "md") { + format = MARKDOWN; + } else if (s == "sql") { + format = SQL; + } else { + return false; + } + return true; +} + +static const char * split_mode_str(llama_split_mode mode) { + switch (mode) { + case LLAMA_SPLIT_MODE_NONE: + return "none"; + case LLAMA_SPLIT_MODE_LAYER: + return "layer"; + case LLAMA_SPLIT_MODE_ROW: + return "row"; + case LLAMA_SPLIT_MODE_TENSOR: + return "tensor"; + default: + GGML_ABORT("invalid split mode"); + } +} + +static std::string pair_str(const std::pair & p) { + static char buf[32]; + snprintf(buf, sizeof(buf), "%d,%d", p.first, p.second); + return buf; +} + +static std::vector parse_int_range(const std::string & s) { + // first[-last[(+|*)step]] + std::regex range_regex(R"(^(\d+)(?:-(\d+)(?:([\+|\*])(\d+))?)?(?:,|$))"); + + std::smatch match; + std::string::const_iterator search_start(s.cbegin()); + std::vector result; + while (std::regex_search(search_start, s.cend(), match, range_regex)) { + int first = std::stoi(match[1]); + int last = match[2].matched ? std::stoi(match[2]) : first; + char op = match[3].matched ? match[3].str()[0] : '+'; + int step = match[4].matched ? std::stoi(match[4]) : 1; + + for (int i = first; i <= last;) { + result.push_back(i); + + int prev_i = i; + + if (op == '+') { + i += step; + } else if (op == '*') { + i *= step; + } else { + throw std::invalid_argument("invalid range format"); + } + + if (i <= prev_i) { + throw std::invalid_argument("invalid range"); + } + } + search_start = match.suffix().first; + } + + if (search_start != s.cend()) { + throw std::invalid_argument("invalid range format"); + } + + return result; +} + +struct cmd_params { + std::vector model; + std::vector hf_repo; + std::vector hf_file; + std::string hf_token; + std::vector n_prompt; + std::vector n_gen; + std::vector> n_pg; + std::vector n_depth; + std::vector n_batch; + std::vector n_ubatch; + std::vector type_k; + std::vector type_v; + std::vector n_threads; + std::vector cpu_mask; + std::vector cpu_strict; + std::vector poll; + std::vector n_gpu_layers; + std::vector n_cpu_moe; + std::vector split_mode; + std::vector reduction_provider; + std::vector main_gpu; + std::vector no_kv_offload; + std::vector flash_attn; + std::vector> devices; + std::vector> tensor_split; + std::vector> tensor_buft_overrides; + std::vector use_mmap; + std::vector use_direct_io; + std::vector embeddings; + std::vector no_op_offload; + std::vector no_host; + std::vector fit_params_target; + std::vector fit_params_min_ctx; + ggml_numa_strategy numa; + int reps; + ggml_sched_priority prio; + int delay; + bool verbose; + bool progress; + bool no_warmup; + output_formats output_format; + output_formats output_format_stderr; +}; + +static const cmd_params cmd_params_defaults = { + /* model */ { "models/7B/ggml-model-q4_0.gguf" }, + /* hf_repo */ {}, + /* hf_file */ {}, + /* hf_token */ "", + /* n_prompt */ { 512 }, + /* n_gen */ { 128 }, + /* n_pg */ {}, + /* n_depth */ { 0 }, + /* n_batch */ { 2048 }, + /* n_ubatch */ { 512 }, + /* type_k */ { GGML_TYPE_F16 }, + /* type_v */ { GGML_TYPE_F16 }, + /* n_threads */ { cpu_get_num_math() }, + /* cpu_mask */ { "0x0" }, + /* cpu_strict */ { false }, + /* poll */ { 50 }, + /* n_gpu_layers */ { 99 }, + /* n_cpu_moe */ { 0 }, + /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, + /* reduction_provider */ { "auto" }, + /* main_gpu */ { 0 }, + /* no_kv_offload */ { false }, + /* flash_attn */ { false }, + /* devices */ { {} }, + /* tensor_split */ { std::vector(llama_max_devices(), 0.0f) }, + /* tensor_buft_overrides*/ { std::vector{ { nullptr, nullptr } } }, + /* use_mmap */ { true }, + /* use_direct_io */ { false }, + /* embeddings */ { false }, + /* no_op_offload */ { false }, + /* no_host */ { false }, + /* fit_params_target */ { 0 }, + /* fit_params_min_ctx */ { 0 }, + /* numa */ GGML_NUMA_STRATEGY_DISABLED, + /* reps */ 5, + /* prio */ GGML_SCHED_PRIO_NORMAL, + /* delay */ 0, + /* verbose */ false, + /* progress */ false, + /* no_warmup */ false, + /* output_format */ MARKDOWN, + /* output_format_stderr */ NONE, +}; + +static void print_usage(int /* argc */, char ** argv) { + printf("usage: %s [options]\n", argv[0]); + printf("\n"); + printf("options:\n"); + printf(" -h, --help\n"); + printf(" --numa numa mode (default: disabled)\n"); + printf(" -r, --repetitions number of times to repeat each test (default: %d)\n", cmd_params_defaults.reps); + printf(" --prio <-1|0|1|2|3> process/thread priority (default: %d)\n", cmd_params_defaults.prio); + printf(" --delay <0...N> (seconds) delay between each test (default: %d)\n", cmd_params_defaults.delay); + printf(" -o, --output output format printed to stdout (default: %s)\n", output_format_str(cmd_params_defaults.output_format)); + printf(" -oe, --output-err output format printed to stderr (default: %s)\n", output_format_str(cmd_params_defaults.output_format_stderr)); + printf(" --list-devices list available devices and exit\n"); + printf(" -v, --verbose verbose output\n"); + printf(" --progress print test progress indicators\n"); + printf(" --no-warmup skip warmup runs before benchmarking\n"); + printf(" -fitt, --fit-target fit model to device memory with this margin per device in MiB (default: off)\n"); + printf(" -fitc, --fit-ctx minimum ctx size for --fit-target (default: 4096)\n"); + if (llama_supports_rpc()) { + printf(" -rpc, --rpc register RPC devices (comma separated)\n"); + } + printf("\n"); + printf("test parameters:\n"); + printf(" -m, --model (default: %s)\n", join(cmd_params_defaults.model, ",").c_str()); + printf(" -hf, -hfr, --hf-repo /[:quant] Hugging Face model repository; quant is optional, case-insensitive\n"); + printf(" default to Q4_K_M, or falls back to the first file in the repo if Q4_K_M doesn't exist.\n"); + printf(" example: ggml-org/GLM-4.7-Flash-GGUF:Q4_K_M\n"); + printf(" (default: unused)\n"); + printf(" -hff, --hf-file Hugging Face model file. If specified, it will override the quant in --hf-repo\n"); + printf(" (default: unused)\n"); + printf(" -hft, --hf-token Hugging Face access token\n"); + printf(" (default: value from HF_TOKEN environment variable)\n"); + printf(" -p, --n-prompt (default: %s)\n", join(cmd_params_defaults.n_prompt, ",").c_str()); + printf(" -n, --n-gen (default: %s)\n", join(cmd_params_defaults.n_gen, ",").c_str()); + printf(" -pg (default: %s)\n", join(transform_to_str(cmd_params_defaults.n_pg, pair_str), ",").c_str()); + printf(" -d, --n-depth (default: %s)\n", join(cmd_params_defaults.n_depth, ",").c_str()); + printf(" -b, --batch-size (default: %s)\n", join(cmd_params_defaults.n_batch, ",").c_str()); + printf(" -ub, --ubatch-size (default: %s)\n", join(cmd_params_defaults.n_ubatch, ",").c_str()); + printf(" -ctk, --cache-type-k (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_k, ggml_type_name), ",").c_str()); + printf(" -ctv, --cache-type-v (default: %s)\n", join(transform_to_str(cmd_params_defaults.type_v, ggml_type_name), ",").c_str()); + printf(" -t, --threads (default: %s)\n", join(cmd_params_defaults.n_threads, ",").c_str()); + printf(" -C, --cpu-mask (default: %s)\n", join(cmd_params_defaults.cpu_mask, ",").c_str()); + printf(" --cpu-strict <0|1> (default: %s)\n", join(cmd_params_defaults.cpu_strict, ",").c_str()); + printf(" --poll <0...100> (default: %s)\n", join(cmd_params_defaults.poll, ",").c_str()); + printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); + printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); + printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); + printf(" -rp, --reduction-provider allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.reduction_provider, ",").c_str()); + printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); + printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); + printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); + printf(" -dev, --device (default: auto)\n"); + printf(" -mmp, --mmap <0|1> (default: %s)\n", join(cmd_params_defaults.use_mmap, ",").c_str()); + printf(" -dio, --direct-io <0|1> (default: %s)\n", join(cmd_params_defaults.use_direct_io, ",").c_str()); + printf(" -embd, --embeddings <0|1> (default: %s)\n", join(cmd_params_defaults.embeddings, ",").c_str()); + printf(" -ts, --tensor-split (default: 0)\n"); + printf(" -ot --override-tensor =;...\n"); + printf(" (default: disabled)\n"); + printf(" -nopo, --no-op-offload <0|1> (default: 0)\n"); + printf(" --no-host <0|1> (default: %s)\n", join(cmd_params_defaults.no_host, ",").c_str()); + printf("\n"); + printf( + "Multiple values can be given for each parameter by separating them with ','\n" + "or by specifying the parameter multiple times. Ranges can be given as\n" + "'first-last' or 'first-last+step' or 'first-last*mult'.\n"); +} + +static ggml_type ggml_type_from_name(const std::string & s) { + if (s == "f16") { + return GGML_TYPE_F16; + } + if (s == "bf16") { + return GGML_TYPE_BF16; + } + if (s == "q8_0") { + return GGML_TYPE_Q8_0; + } + if (s == "q4_0") { + return GGML_TYPE_Q4_0; + } + if (s == "q4_1") { + return GGML_TYPE_Q4_1; + } + if (s == "q5_0") { + return GGML_TYPE_Q5_0; + } + if (s == "q5_1") { + return GGML_TYPE_Q5_1; + } + if (s == "iq4_nl") { + return GGML_TYPE_IQ4_NL; + } + + return GGML_TYPE_COUNT; +} + +static cmd_params parse_cmd_params(int argc, char ** argv) { + cmd_params params; + std::string arg; + bool invalid_param = false; + const std::string arg_prefix = "--"; + const char split_delim = ','; + + params.verbose = cmd_params_defaults.verbose; + params.output_format = cmd_params_defaults.output_format; + params.output_format_stderr = cmd_params_defaults.output_format_stderr; + params.reps = cmd_params_defaults.reps; + params.numa = cmd_params_defaults.numa; + params.prio = cmd_params_defaults.prio; + params.delay = cmd_params_defaults.delay; + params.progress = cmd_params_defaults.progress; + params.no_warmup = cmd_params_defaults.no_warmup; + + if (const char * env = getenv("HF_TOKEN")) { + params.hf_token = env; + } + + for (int i = 1; i < argc; i++) { + arg = argv[i]; + if (arg.compare(0, arg_prefix.size(), arg_prefix) == 0) { + std::replace(arg.begin(), arg.end(), '_', '-'); + } + + try { + if (arg == "-h" || arg == "--help") { + print_usage(argc, argv); + exit(0); + } else if (arg == "-m" || arg == "--model") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.model.insert(params.model.end(), p.begin(), p.end()); + } else if (arg == "-hf" || arg == "-hfr" || arg == "--hf-repo") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.hf_repo.insert(params.hf_repo.end(), p.begin(), p.end()); + } else if (arg == "-hff" || arg == "--hf-file") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.hf_file.insert(params.hf_file.end(), p.begin(), p.end()); + } else if (arg == "-hft" || arg == "--hf-token") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.hf_token = argv[i]; + } else if (arg == "-p" || arg == "--n-prompt") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_prompt.insert(params.n_prompt.end(), p.begin(), p.end()); + } else if (arg == "-n" || arg == "--n-gen") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_gen.insert(params.n_gen.end(), p.begin(), p.end()); + } else if (arg == "-pg") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], ','); + if (p.size() != 2) { + invalid_param = true; + break; + } + params.n_pg.push_back({ std::stoi(p[0]), std::stoi(p[1]) }); + } else if (arg == "-d" || arg == "--n-depth") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_depth.insert(params.n_depth.end(), p.begin(), p.end()); + } else if (arg == "-b" || arg == "--batch-size") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_batch.insert(params.n_batch.end(), p.begin(), p.end()); + } else if (arg == "-ub" || arg == "--ubatch-size") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_ubatch.insert(params.n_ubatch.end(), p.begin(), p.end()); + } else if (arg == "-ctk" || arg == "--cache-type-k") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector types; + for (const auto & t : p) { + ggml_type gt = ggml_type_from_name(t); + if (gt == GGML_TYPE_COUNT) { + invalid_param = true; + break; + } + types.push_back(gt); + } + if (invalid_param) { + break; + } + params.type_k.insert(params.type_k.end(), types.begin(), types.end()); + } else if (arg == "-ctv" || arg == "--cache-type-v") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector types; + for (const auto & t : p) { + ggml_type gt = ggml_type_from_name(t); + if (gt == GGML_TYPE_COUNT) { + invalid_param = true; + break; + } + types.push_back(gt); + } + if (invalid_param) { + break; + } + params.type_v.insert(params.type_v.end(), types.begin(), types.end()); + } else if (arg == "-dev" || arg == "--device") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto combos = string_split(argv[i], split_delim); + for (const auto & combo : combos) { + try { + params.devices.push_back(parse_devices_arg(combo)); + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } + if (invalid_param) { + break; + } + } else if (arg == "--list-devices") { + std::vector devices; + for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { + auto * dev = ggml_backend_dev_get(i); + if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) { + devices.push_back(dev); + } + } + printf("Available devices:\n"); + if (devices.empty()) { + printf(" (none)\n"); + } + for (auto * dev : devices) { + size_t free, total; + ggml_backend_dev_memory(dev, &free, &total); + printf(" %s: %s (%zu MiB, %zu MiB free)\n", ggml_backend_dev_name(dev), ggml_backend_dev_description(dev), total / 1024 / 1024, free / 1024 / 1024); + } + exit(0); + } else if (arg == "-t" || arg == "--threads") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_threads.insert(params.n_threads.end(), p.begin(), p.end()); + } else if (arg == "-C" || arg == "--cpu-mask") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.cpu_mask.insert(params.cpu_mask.end(), p.begin(), p.end()); + } else if (arg == "--cpu-strict") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.cpu_strict.insert(params.cpu_strict.end(), p.begin(), p.end()); + } else if (arg == "--poll") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.poll.insert(params.poll.end(), p.begin(), p.end()); + } else if (arg == "-ngl" || arg == "--n-gpu-layers") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_gpu_layers.insert(params.n_gpu_layers.end(), p.begin(), p.end()); + } else if (arg == "-ncmoe" || arg == "--n-cpu-moe") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = parse_int_range(argv[i]); + params.n_cpu_moe.insert(params.n_cpu_moe.end(), p.begin(), p.end()); + } else if (llama_supports_rpc() && (arg == "-rpc" || arg == "--rpc")) { + if (++i >= argc) { + invalid_param = true; + break; + } + try { + register_rpc_server_list(argv[i]); + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } else if (arg == "-sm" || arg == "--split-mode") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + + std::vector modes; + for (const auto & m : p) { + llama_split_mode mode; + if (m == "none") { + mode = LLAMA_SPLIT_MODE_NONE; + } else if (m == "layer") { + mode = LLAMA_SPLIT_MODE_LAYER; + } else if (m == "row") { + mode = LLAMA_SPLIT_MODE_ROW; + } else if (m == "tensor") { + mode = LLAMA_SPLIT_MODE_TENSOR; + } else { + invalid_param = true; + break; + } + modes.push_back(mode); + } + if (invalid_param) { + break; + } + params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); + } else if (arg == "-rp" || arg == "--reduction-provider") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + if (v != "auto" && v != "nccl" && v != "internal") { + invalid_param = true; + break; + } + } + if (invalid_param) { + break; + } + params.reduction_provider.insert(params.reduction_provider.end(), p.begin(), p.end()); + } else if (arg == "-mg" || arg == "--main-gpu") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.main_gpu = parse_int_range(argv[i]); + } else if (arg == "-nkvo" || arg == "--no-kv-offload") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_kv_offload.insert(params.no_kv_offload.end(), p.begin(), p.end()); + } else if (arg == "--numa") { + if (++i >= argc) { + invalid_param = true; + break; + } + std::string value(argv[i]); + if (value == "distribute" || value == "") { + params.numa = GGML_NUMA_STRATEGY_DISTRIBUTE; + } else if (value == "isolate") { + params.numa = GGML_NUMA_STRATEGY_ISOLATE; + } else if (value == "numactl") { + params.numa = GGML_NUMA_STRATEGY_NUMACTL; + } else { + invalid_param = true; + break; + } + } else if (arg == "-fa" || arg == "--flash-attn") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.flash_attn.insert(params.flash_attn.end(), p.begin(), p.end()); + } else if (arg == "-mmp" || arg == "--mmap") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.use_mmap.insert(params.use_mmap.end(), p.begin(), p.end()); + } else if (arg == "-dio" || arg == "--direct-io") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.use_direct_io.insert(params.use_direct_io.end(), p.begin(), p.end()); + } else if (arg == "-embd" || arg == "--embeddings") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.embeddings.insert(params.embeddings.end(), p.begin(), p.end()); + } else if (arg == "-nopo" || arg == "--no-op-offload") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_op_offload.insert(params.no_op_offload.end(), p.begin(), p.end()); + } else if (arg == "--no-host") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + params.no_host.insert(params.no_host.end(), p.begin(), p.end()); + } else if (arg == "-ts" || arg == "--tensor-split") { + if (++i >= argc) { + invalid_param = true; + break; + } + for (auto ts : string_split(argv[i], split_delim)) { + // split string by ; and / + const std::regex regex{ R"([;/]+)" }; + std::sregex_token_iterator it{ ts.begin(), ts.end(), regex, -1 }; + std::vector split_arg{ it, {} }; + GGML_ASSERT(split_arg.size() <= llama_max_devices()); + + std::vector tensor_split(llama_max_devices()); + for (size_t i = 0; i < llama_max_devices(); ++i) { + if (i < split_arg.size()) { + tensor_split[i] = std::stof(split_arg[i]); + } else { + tensor_split[i] = 0.0f; + } + } + params.tensor_split.push_back(tensor_split); + } + } else if (arg == "-ot" || arg == "--override-tensor") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto * value = argv[i]; + /* static */ std::map buft_list; + if (buft_list.empty()) { + // enumerate all the devices and add their buffer types to the list + for (size_t i = 0; i < ggml_backend_dev_count(); ++i) { + auto * dev = ggml_backend_dev_get(i); + auto * buft = ggml_backend_dev_buffer_type(dev); + if (buft) { + buft_list[ggml_backend_buft_name(buft)] = buft; + } + } + } + auto override_group_span_len = std::strcspn(value, ","); + bool last_group = false; + do { + if (override_group_span_len == 0) { + // Adds an empty override-tensors for an empty span + params.tensor_buft_overrides.push_back({{}}); + if (value[override_group_span_len] == '\0') { + value = &value[override_group_span_len]; + last_group = true; + } else { + value = &value[override_group_span_len + 1]; + override_group_span_len = std::strcspn(value, ","); + } + continue; + } + // Stamps null terminators into the argv + // value for this option to avoid the + // memory leak present in the implementation + // over in arg.cpp. Acceptable because we + // only parse these args once in this program. + auto * override_group = value; + if (value[override_group_span_len] == '\0') { + value = &value[override_group_span_len]; + last_group = true; + } else { + value[override_group_span_len] = '\0'; + value = &value[override_group_span_len + 1]; + } + std::vector group_tensor_buft_overrides{}; + auto override_span_len = std::strcspn(override_group, ";"); + while (override_span_len > 0) { + auto * override = override_group; + if (override_group[override_span_len] != '\0') { + override_group[override_span_len] = '\0'; + override_group = &override_group[override_span_len + 1]; + } else { + override_group = &override_group[override_span_len]; + } + auto tensor_name_span_len = std::strcspn(override, "="); + if (tensor_name_span_len >= override_span_len) { + invalid_param = true; + break; + } + override[tensor_name_span_len] = '\0'; + auto * tensor_name = override; + auto * buffer_type = &override[tensor_name_span_len + 1]; + if (buft_list.find(buffer_type) == buft_list.end()) { + printf("error: unrecognized buffer type '%s'\n", buffer_type); + printf("Available buffer types:\n"); + for (const auto & it : buft_list) { + printf(" %s\n", ggml_backend_buft_name(it.second)); + } + invalid_param = true; + break; + } + group_tensor_buft_overrides.push_back({tensor_name, buft_list.at(buffer_type)}); + override_span_len = std::strcspn(override_group, ";"); + } + if (invalid_param) { + break; + } + group_tensor_buft_overrides.push_back({nullptr,nullptr}); + params.tensor_buft_overrides.push_back(group_tensor_buft_overrides); + override_group_span_len = std::strcspn(value, ","); + } while (!last_group); + } else if (arg == "-r" || arg == "--repetitions") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.reps = std::stoi(argv[i]); + } else if (arg == "--prio") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.prio = (enum ggml_sched_priority) std::stoi(argv[i]); + } else if (arg == "--delay") { + if (++i >= argc) { + invalid_param = true; + break; + } + params.delay = std::stoi(argv[i]); + } else if (arg == "-o" || arg == "--output") { + if (++i >= argc) { + invalid_param = true; + break; + } + invalid_param = !output_format_from_str(argv[i], params.output_format); + } else if (arg == "-oe" || arg == "--output-err") { + if (++i >= argc) { + invalid_param = true; + break; + } + invalid_param = !output_format_from_str(argv[i], params.output_format_stderr); + } else if (arg == "-v" || arg == "--verbose") { + params.verbose = true; + } else if (arg == "--progress") { + params.progress = true; + } else if (arg == "--no-warmup") { + params.no_warmup = true; + } else if (arg == "-fitt" || arg == "--fit-target") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + params.fit_params_target.push_back(std::stoull(v)); + } + } else if (arg == "-fitc" || arg == "--fit-ctx") { + if (++i >= argc) { + invalid_param = true; + break; + } + auto p = string_split(argv[i], split_delim); + for (const auto & v : p) { + params.fit_params_min_ctx.push_back(std::stoul(v)); + } + } else { + invalid_param = true; + break; + } + } catch (const std::exception & e) { + fprintf(stderr, "error: %s\n", e.what()); + invalid_param = true; + break; + } + } + + if (invalid_param) { + fprintf(stderr, "error: invalid parameter for argument: %s\n", arg.c_str()); + print_usage(argc, argv); + exit(1); + } + + if (!params.hf_repo.empty()) { + for (size_t i = 0; i < params.hf_repo.size(); i++) { + common_params_model model; + + if (params.hf_file.empty() || params.hf_file[i].empty()) { + model.hf_repo = params.hf_repo[i]; + } else { + model.hf_repo = params.hf_repo[i]; + model.hf_file = params.hf_file[i]; + } + + common_download_opts opts; + opts.bearer_token = params.hf_token; + auto download_result = common_download_model(model, opts); + if (download_result.model_path.empty()) { + fprintf(stderr, "error: failed to download model from HuggingFace\n"); + exit(1); + } + + params.model.push_back(download_result.model_path); + } + } + + // set defaults + if (params.model.empty()) { + params.model = cmd_params_defaults.model; + } + if (params.n_prompt.empty()) { + params.n_prompt = cmd_params_defaults.n_prompt; + } + if (params.n_gen.empty()) { + params.n_gen = cmd_params_defaults.n_gen; + } + if (params.n_pg.empty()) { + params.n_pg = cmd_params_defaults.n_pg; + } + if (params.n_depth.empty()) { + params.n_depth = cmd_params_defaults.n_depth; + } + if (params.n_batch.empty()) { + params.n_batch = cmd_params_defaults.n_batch; + } + if (params.n_ubatch.empty()) { + params.n_ubatch = cmd_params_defaults.n_ubatch; + } + if (params.type_k.empty()) { + params.type_k = cmd_params_defaults.type_k; + } + if (params.type_v.empty()) { + params.type_v = cmd_params_defaults.type_v; + } + if (params.n_gpu_layers.empty()) { + params.n_gpu_layers = cmd_params_defaults.n_gpu_layers; + } + if (params.n_cpu_moe.empty()) { + params.n_cpu_moe = cmd_params_defaults.n_cpu_moe; + } + if (params.split_mode.empty()) { + params.split_mode = cmd_params_defaults.split_mode; + } + if (params.reduction_provider.empty()) { + params.reduction_provider = cmd_params_defaults.reduction_provider; + } + if (params.main_gpu.empty()) { + params.main_gpu = cmd_params_defaults.main_gpu; + } + if (params.no_kv_offload.empty()) { + params.no_kv_offload = cmd_params_defaults.no_kv_offload; + } + if (params.flash_attn.empty()) { + params.flash_attn = cmd_params_defaults.flash_attn; + } + if (params.devices.empty()) { + params.devices = cmd_params_defaults.devices; + } + if (params.tensor_split.empty()) { + params.tensor_split = cmd_params_defaults.tensor_split; + } + if (params.tensor_buft_overrides.empty()) { + params.tensor_buft_overrides = cmd_params_defaults.tensor_buft_overrides; + } + if (params.use_mmap.empty()) { + params.use_mmap = cmd_params_defaults.use_mmap; + } + if (params.use_direct_io.empty()) { + params.use_direct_io = cmd_params_defaults.use_direct_io; + } + if (params.embeddings.empty()) { + params.embeddings = cmd_params_defaults.embeddings; + } + if (params.no_op_offload.empty()) { + params.no_op_offload = cmd_params_defaults.no_op_offload; + } + if (params.no_host.empty()) { + params.no_host = cmd_params_defaults.no_host; + } + if (params.n_threads.empty()) { + params.n_threads = cmd_params_defaults.n_threads; + } + if (params.cpu_mask.empty()) { + params.cpu_mask = cmd_params_defaults.cpu_mask; + } + if (params.cpu_strict.empty()) { + params.cpu_strict = cmd_params_defaults.cpu_strict; + } + if (params.poll.empty()) { + params.poll = cmd_params_defaults.poll; + } + if (params.fit_params_target.empty()) { + params.fit_params_target = cmd_params_defaults.fit_params_target; + } + if (params.fit_params_min_ctx.empty()) { + params.fit_params_min_ctx = cmd_params_defaults.fit_params_min_ctx; + } + + return params; +} + +struct cmd_params_instance { + std::string model; + int n_prompt; + int n_gen; + int n_depth; + int n_batch; + int n_ubatch; + ggml_type type_k; + ggml_type type_v; + int n_threads; + std::string cpu_mask; + bool cpu_strict; + int poll; + int n_gpu_layers; + int n_cpu_moe; + llama_split_mode split_mode; + std::string reduction_provider; + int main_gpu; + bool no_kv_offload; + bool flash_attn; + std::vector devices; + std::vector tensor_split; + std::vector tensor_buft_overrides; + bool use_mmap; + bool use_direct_io; + bool embeddings; + bool no_op_offload; + bool no_host; + size_t fit_target; + uint32_t fit_min_ctx; + + llama_model_params to_llama_mparams() const { + llama_model_params mparams = llama_model_default_params(); + + mparams.n_gpu_layers = n_gpu_layers; + if (!devices.empty()) { + mparams.devices = const_cast(devices.data()); + } + mparams.split_mode = split_mode; + mparams.main_gpu = main_gpu; + mparams.tensor_split = tensor_split.data(); + mparams.use_mmap = use_mmap; + mparams.use_direct_io = use_direct_io; + mparams.no_host = no_host; + + if (n_cpu_moe <= 0) { + if (tensor_buft_overrides.empty()) { + mparams.tensor_buft_overrides = nullptr; + } else { + GGML_ASSERT(tensor_buft_overrides.back().pattern == nullptr && + "Tensor buffer overrides not terminated with empty pattern"); + mparams.tensor_buft_overrides = tensor_buft_overrides.data(); + } + } else { + static std::vector merged; + static std::vector patterns; + + merged.clear(); + patterns.clear(); + + auto first = tensor_buft_overrides.begin(); + auto last = tensor_buft_overrides.end(); + if (first != last && (last - 1)->pattern == nullptr) { + --last; + } + merged.insert(merged.end(), first, last); + + patterns.reserve((size_t) n_cpu_moe); + merged.reserve(merged.size() + (size_t) n_cpu_moe + 1); + + for (int i = 0; i < n_cpu_moe; ++i) { + patterns.push_back(llm_ffn_exps_block_regex(i)); + merged.push_back({ patterns.back().c_str(), + ggml_backend_cpu_buffer_type() }); + } + + merged.push_back({ nullptr, nullptr }); + + mparams.tensor_buft_overrides = merged.data(); + } + + return mparams; + } + + bool equal_mparams(const cmd_params_instance & other) const { + return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && + split_mode == other.split_mode && reduction_provider == other.reduction_provider && + main_gpu == other.main_gpu && tensor_split == other.tensor_split && + use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && + devices == other.devices && + no_host == other.no_host && + vec_tensor_buft_override_equal(tensor_buft_overrides, other.tensor_buft_overrides); + } + + llama_context_params to_llama_cparams() const { + llama_context_params cparams = llama_context_default_params(); + + cparams.n_ctx = n_prompt + n_gen + n_depth; + cparams.n_batch = n_batch; + cparams.n_ubatch = n_ubatch; + cparams.type_k = type_k; + cparams.type_v = type_v; + cparams.offload_kqv = !no_kv_offload; + cparams.flash_attn_type = flash_attn ? LLAMA_FLASH_ATTN_TYPE_ENABLED : LLAMA_FLASH_ATTN_TYPE_DISABLED; + cparams.embeddings = embeddings; + cparams.op_offload = !no_op_offload; + cparams.swa_full = false; + + return cparams; + } +}; + +static std::vector get_cmd_params_instances(const cmd_params & params) { + std::vector instances; + + // this ordering minimizes the number of times that each model needs to be reloaded + // clang-format off + for (const auto & m : params.model) + for (const auto & fpt : params.fit_params_target) + for (const auto & fpc : params.fit_params_min_ctx) + for (const auto & nl : params.n_gpu_layers) + for (const auto & ncmoe : params.n_cpu_moe) + for (const auto & sm : params.split_mode) + for (const auto & rp : params.reduction_provider) + for (const auto & mg : params.main_gpu) + for (const auto & devs : params.devices) + for (const auto & ts : params.tensor_split) + for (const auto & ot : params.tensor_buft_overrides) + for (const auto & mmp : params.use_mmap) + for (const auto & dio : params.use_direct_io) + for (const auto & noh : params.no_host) + for (const auto & embd : params.embeddings) + for (const auto & nopo : params.no_op_offload) + for (const auto & nb : params.n_batch) + for (const auto & nub : params.n_ubatch) + for (const auto & tk : params.type_k) + for (const auto & tv : params.type_v) + for (const auto & nkvo : params.no_kv_offload) + for (const auto & fa : params.flash_attn) + for (const auto & nt : params.n_threads) + for (const auto & cm : params.cpu_mask) + for (const auto & cs : params.cpu_strict) + for (const auto & nd : params.n_depth) + for (const auto & pl : params.poll) { + for (const auto & n_prompt : params.n_prompt) { + if (n_prompt == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ n_prompt, + /* .n_gen = */ 0, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .reduction_provider = */ rp, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + + for (const auto & n_gen : params.n_gen) { + if (n_gen == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ 0, + /* .n_gen = */ n_gen, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .reduction_provider = */ rp, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + + for (const auto & n_pg : params.n_pg) { + if (n_pg.first == 0 && n_pg.second == 0) { + continue; + } + cmd_params_instance instance = { + /* .model = */ m, + /* .n_prompt = */ n_pg.first, + /* .n_gen = */ n_pg.second, + /* .n_depth = */ nd, + /* .n_batch = */ nb, + /* .n_ubatch = */ nub, + /* .type_k = */ tk, + /* .type_v = */ tv, + /* .n_threads = */ nt, + /* .cpu_mask = */ cm, + /* .cpu_strict = */ cs, + /* .poll = */ pl, + /* .n_gpu_layers = */ nl, + /* .n_cpu_moe = */ ncmoe, + /* .split_mode = */ sm, + /* .reduction_provider = */ rp, + /* .main_gpu = */ mg, + /* .no_kv_offload= */ nkvo, + /* .flash_attn = */ fa, + /* .devices = */ devs, + /* .tensor_split = */ ts, + /* .tensor_buft_overrides = */ ot, + /* .use_mmap = */ mmp, + /* .use_direct_io= */ dio, + /* .embeddings = */ embd, + /* .no_op_offload= */ nopo, + /* .no_host = */ noh, + /* .fit_target = */ fpt, + /* .fit_min_ctx = */ fpc, + }; + instances.push_back(instance); + } + } + // clang-format on + + return instances; +} + +struct test { + static const std::string build_commit; + static const int build_number; + const std::string cpu_info; + const std::string gpu_info; + std::string model_filename; + std::string model_type; + uint64_t model_size; + uint64_t model_n_params; + int n_batch; + int n_ubatch; + int n_threads; + std::string cpu_mask; + bool cpu_strict; + int poll; + ggml_type type_k; + ggml_type type_v; + int n_gpu_layers; + int n_cpu_moe; + llama_split_mode split_mode; + std::string reduction_provider; + int main_gpu; + bool no_kv_offload; + bool flash_attn; + std::vector devices; + std::vector tensor_split; + std::vector tensor_buft_overrides; + bool use_mmap; + bool use_direct_io; + bool embeddings; + bool no_op_offload; + bool no_host; + size_t fit_target; + uint32_t fit_min_ctx; + int n_prompt; + int n_gen; + int n_depth; + std::string test_time; + std::vector samples_ns; + + test(const cmd_params_instance & inst, const llama_model * lmodel, const llama_context * ctx) : + cpu_info(get_cpu_info()), + gpu_info(get_gpu_info()) { + + model_filename = inst.model; + char buf[128]; + llama_model_desc(lmodel, buf, sizeof(buf)); + model_type = buf; + model_size = llama_model_size(lmodel); + model_n_params = llama_model_n_params(lmodel); + n_batch = inst.n_batch; + n_ubatch = inst.n_ubatch; + n_threads = inst.n_threads; + cpu_mask = inst.cpu_mask; + cpu_strict = inst.cpu_strict; + poll = inst.poll; + type_k = inst.type_k; + type_v = inst.type_v; + n_gpu_layers = inst.n_gpu_layers; + n_cpu_moe = inst.n_cpu_moe; + split_mode = inst.split_mode; + reduction_provider = inst.reduction_provider; + main_gpu = inst.main_gpu; + no_kv_offload = inst.no_kv_offload; + flash_attn = inst.flash_attn; + devices = inst.devices; + tensor_split = inst.tensor_split; + tensor_buft_overrides = inst.tensor_buft_overrides; + use_mmap = inst.use_mmap; + use_direct_io = inst.use_direct_io; + embeddings = inst.embeddings; + no_op_offload = inst.no_op_offload; + no_host = inst.no_host; + fit_target = inst.fit_target; + fit_min_ctx = inst.fit_min_ctx; + n_prompt = inst.n_prompt; + n_gen = inst.n_gen; + n_depth = inst.n_depth; + // RFC 3339 date-time format + time_t t = time(NULL); + std::strftime(buf, sizeof(buf), "%FT%TZ", gmtime(&t)); + test_time = buf; + + (void) ctx; + } + + uint64_t avg_ns() const { return ::avg(samples_ns); } + + uint64_t stdev_ns() const { return ::stdev(samples_ns); } + + std::vector get_ts() const { + int n_tokens = n_prompt + n_gen; + std::vector ts; + std::transform(samples_ns.begin(), samples_ns.end(), std::back_inserter(ts), + [n_tokens](uint64_t t) { return 1e9 * n_tokens / t; }); + return ts; + } + + double avg_ts() const { return ::avg(get_ts()); } + + double stdev_ts() const { return ::stdev(get_ts()); } + + static std::string get_backend() { + std::vector backends; + bool rpc_used = false; + for (size_t i = 0; i < ggml_backend_reg_count(); i++) { + auto * reg = ggml_backend_reg_get(i); + std::string name = ggml_backend_reg_name(reg); + if (string_starts_with(name, "RPC")) { + if (ggml_backend_reg_dev_count(reg) > 0) { + rpc_used = true; + } + } else { + if (name != "CPU") { + backends.push_back(ggml_backend_reg_name(reg)); + } + } + } + if (rpc_used) { + backends.push_back("RPC"); + } + return backends.empty() ? "CPU" : join(backends, ","); + } + + static const std::vector & get_fields() { + static const std::vector fields = { + "build_commit", "build_number", "cpu_info", "gpu_info", "backends", + "model_filename", "model_type", "model_size", "model_n_params", "n_batch", + "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", + "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", + "reduction_provider", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", + "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", + "no_op_offload", "no_host", "fit_target", "fit_min_ctx", + "n_prompt", "n_gen", "n_depth", + "test_time", "avg_ns", "stddev_ns", "avg_ts", "stddev_ts" + }; + return fields; + } + + enum field_type { STRING, BOOL, INT, FLOAT }; + + static field_type get_field_type(const std::string & field) { + if (field == "build_number" || field == "n_batch" || field == "n_ubatch" || field == "n_threads" || + field == "poll" || field == "model_size" || field == "model_n_params" || field == "n_gpu_layers" || + field == "main_gpu" || field == "n_prompt" || field == "n_gen" || field == "n_depth" || field == "avg_ns" || + field == "stddev_ns" || field == "no_op_offload" || field == "n_cpu_moe" || + field == "fit_target" || field == "fit_min_ctx") { + return INT; + } + if (field == "f16_kv" || field == "no_kv_offload" || field == "cpu_strict" || field == "flash_attn" || + field == "use_mmap" || field == "use_direct_io" || field == "embeddings" || field == "no_host") { + return BOOL; + } + if (field == "avg_ts" || field == "stddev_ts") { + return FLOAT; + } + return STRING; + } + + std::vector get_values() const { + std::string tensor_split_str; + std::string tensor_buft_overrides_str; + int max_nonzero = 0; + for (size_t i = 0; i < llama_max_devices(); i++) { + if (tensor_split[i] > 0) { + max_nonzero = i; + } + } + for (int i = 0; i <= max_nonzero; i++) { + char buf[32]; + snprintf(buf, sizeof(buf), "%.2f", tensor_split[i]); + tensor_split_str += buf; + if (i < max_nonzero) { + tensor_split_str += "/"; + } + } + if (tensor_buft_overrides.size() == 1) { + // Last element of tensor_buft_overrides is always a null pattern + // so if it is only one element long, it must be a null pattern. + GGML_ASSERT(tensor_buft_overrides[0].pattern == nullptr); + tensor_buft_overrides_str += "none"; + } else { + for (size_t i = 0; i < tensor_buft_overrides.size()-1; i++) { + // Last element of tensor_buft_overrides is always a null pattern + if (tensor_buft_overrides[i].pattern == nullptr) { + tensor_buft_overrides_str += "none"; + } else { + tensor_buft_overrides_str += tensor_buft_overrides[i].pattern; + tensor_buft_overrides_str += "="; + tensor_buft_overrides_str += ggml_backend_buft_name(tensor_buft_overrides[i].buft); + } + if (i + 2 < tensor_buft_overrides.size()) { + tensor_buft_overrides_str += ";"; + } + } + } + std::vector values = { build_commit, + std::to_string(build_number), + cpu_info, + gpu_info, + get_backend(), + model_filename, + model_type, + std::to_string(model_size), + std::to_string(model_n_params), + std::to_string(n_batch), + std::to_string(n_ubatch), + std::to_string(n_threads), + cpu_mask, + std::to_string(cpu_strict), + std::to_string(poll), + ggml_type_name(type_k), + ggml_type_name(type_v), + std::to_string(n_gpu_layers), + std::to_string(n_cpu_moe), + split_mode_str(split_mode), + reduction_provider, + std::to_string(main_gpu), + std::to_string(no_kv_offload), + std::to_string(flash_attn), + devices_to_string(devices), + tensor_split_str, + tensor_buft_overrides_str, + std::to_string(use_mmap), + std::to_string(use_direct_io), + std::to_string(embeddings), + std::to_string(no_op_offload), + std::to_string(no_host), + std::to_string(fit_target), + std::to_string(fit_min_ctx), + std::to_string(n_prompt), + std::to_string(n_gen), + std::to_string(n_depth), + test_time, + std::to_string(avg_ns()), + std::to_string(stdev_ns()), + std::to_string(avg_ts()), + std::to_string(stdev_ts()) }; + return values; + } + + std::map get_map() const { + std::map map; + auto fields = get_fields(); + auto values = get_values(); + std::transform(fields.begin(), fields.end(), values.begin(), std::inserter(map, map.end()), + std::make_pair); + return map; + } +}; + +const std::string test::build_commit = llama_commit(); +const int test::build_number = llama_build_number(); + +struct printer { + virtual ~printer() {} + + FILE * fout; + + virtual void print_header(const cmd_params & params) { (void) params; } + + virtual void print_test(const test & t) = 0; + + virtual void print_footer() {} +}; + +struct csv_printer : public printer { + static std::string escape_csv(const std::string & field) { + std::string escaped = "\""; + for (auto c : field) { + if (c == '"') { + escaped += "\""; + } + escaped += c; + } + escaped += "\""; + return escaped; + } + + void print_header(const cmd_params & params) override { + std::vector fields = test::get_fields(); + fprintf(fout, "%s\n", join(fields, ",").c_str()); + (void) params; + } + + void print_test(const test & t) override { + std::vector values = t.get_values(); + std::transform(values.begin(), values.end(), values.begin(), escape_csv); + fprintf(fout, "%s\n", join(values, ",").c_str()); + } +}; + +static std::string escape_json(const std::string & value) { + std::string escaped; + for (auto c : value) { + if (c == '"') { + escaped += "\\\""; + } else if (c == '\\') { + escaped += "\\\\"; + } else if (c <= 0x1f) { + char buf[8]; + snprintf(buf, sizeof(buf), "\\u%04x", c); + escaped += buf; + } else { + escaped += c; + } + } + return escaped; +} + +static std::string format_json_value(const std::string & field, const std::string & value) { + switch (test::get_field_type(field)) { + case test::STRING: + return "\"" + escape_json(value) + "\""; + case test::BOOL: + return value == "0" ? "false" : "true"; + default: + return value; + } +} + +struct json_printer : public printer { + bool first = true; + + void print_header(const cmd_params & params) override { + fprintf(fout, "[\n"); + (void) params; + } + + void print_fields(const std::vector & fields, const std::vector & values) { + assert(fields.size() == values.size()); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, " \"%s\": %s,\n", fields.at(i).c_str(), + format_json_value(fields.at(i), values.at(i)).c_str()); + } + } + + void print_test(const test & t) override { + if (first) { + first = false; + } else { + fprintf(fout, ",\n"); + } + fprintf(fout, " {\n"); + print_fields(test::get_fields(), t.get_values()); + fprintf(fout, " \"samples_ns\": [ %s ],\n", join(t.samples_ns, ", ").c_str()); + fprintf(fout, " \"samples_ts\": [ %s ]\n", join(t.get_ts(), ", ").c_str()); + fprintf(fout, " }"); + fflush(fout); + } + + void print_footer() override { fprintf(fout, "\n]\n"); } +}; + +struct jsonl_printer : public printer { + void print_fields(const std::vector & fields, const std::vector & values) { + assert(fields.size() == values.size()); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, "\"%s\": %s, ", fields.at(i).c_str(), format_json_value(fields.at(i), values.at(i)).c_str()); + } + } + + void print_test(const test & t) override { + fprintf(fout, "{"); + print_fields(test::get_fields(), t.get_values()); + fprintf(fout, "\"samples_ns\": [ %s ],", join(t.samples_ns, ", ").c_str()); + fprintf(fout, "\"samples_ts\": [ %s ]", join(t.get_ts(), ", ").c_str()); + fprintf(fout, "}\n"); + fflush(fout); + } +}; + +struct markdown_printer : public printer { + std::vector fields; + + static int get_field_width(const std::string & field) { + if (field == "model") { + return -30; + } + if (field == "t/s") { + return 20; + } + if (field == "size" || field == "params") { + return 10; + } + if (field == "n_gpu_layers") { + return 3; + } + if (field == "n_threads") { + return 7; + } + if (field == "n_batch") { + return 7; + } + if (field == "n_ubatch") { + return 8; + } + if (field == "type_k" || field == "type_v") { + return 6; + } + if (field == "split_mode") { + return 6; + } + if (field == "flash_attn") { + return 2; + } + if (field == "devices") { + return -12; + } + if (field == "use_mmap") { + return 4; + } + if (field == "use_direct_io") { + return 3; + } + if (field == "test") { + return 15; + } + if (field == "no_op_offload") { + return 4; + } + if (field == "no_host") { + return 4; + } + + int width = std::max((int) field.length(), 10); + + if (test::get_field_type(field) == test::STRING) { + return -width; + } + return width; + } + + static std::string get_field_display_name(const std::string & field) { + if (field == "n_gpu_layers") { + return "ngl"; + } + if (field == "split_mode") { + return "sm"; + } + if (field == "n_threads") { + return "threads"; + } + if (field == "no_kv_offload") { + return "nkvo"; + } + if (field == "flash_attn") { + return "fa"; + } + if (field == "use_mmap") { + return "mmap"; + } + if (field == "use_direct_io") { + return "dio"; + } + if (field == "embeddings") { + return "embd"; + } + if (field == "no_op_offload") { + return "nopo"; + } + if (field == "no_host") { + return "noh"; + } + if (field == "devices") { + return "dev"; + } + if (field == "tensor_split") { + return "ts"; + } + if (field == "tensor_buft_overrides") { + return "ot"; + } + if (field == "fit_target") { + return "fitt"; + } + if (field == "fit_min_ctx") { + return "fitc"; + } + return field; + } + + void print_header(const cmd_params & params) override { + // select fields to print + fields.emplace_back("model"); + fields.emplace_back("size"); + fields.emplace_back("params"); + fields.emplace_back("backend"); + bool is_cpu_backend = test::get_backend().find("CPU") != std::string::npos || + test::get_backend().find("BLAS") != std::string::npos || + test::get_backend().find("ZenDNN") != std::string::npos; + if (!is_cpu_backend) { + fields.emplace_back("n_gpu_layers"); + } + if (params.n_cpu_moe.size() > 1 || params.n_cpu_moe != cmd_params_defaults.n_cpu_moe) { + fields.emplace_back("n_cpu_moe"); + } + if (params.n_threads.size() > 1 || params.n_threads != cmd_params_defaults.n_threads || is_cpu_backend) { + fields.emplace_back("n_threads"); + } + if (params.cpu_mask.size() > 1 || params.cpu_mask != cmd_params_defaults.cpu_mask) { + fields.emplace_back("cpu_mask"); + } + if (params.cpu_strict.size() > 1 || params.cpu_strict != cmd_params_defaults.cpu_strict) { + fields.emplace_back("cpu_strict"); + } + if (params.poll.size() > 1 || params.poll != cmd_params_defaults.poll) { + fields.emplace_back("poll"); + } + if (params.n_batch.size() > 1 || params.n_batch != cmd_params_defaults.n_batch) { + fields.emplace_back("n_batch"); + } + if (params.n_ubatch.size() > 1 || params.n_ubatch != cmd_params_defaults.n_ubatch) { + fields.emplace_back("n_ubatch"); + } + if (params.type_k.size() > 1 || params.type_k != cmd_params_defaults.type_k) { + fields.emplace_back("type_k"); + } + if (params.type_v.size() > 1 || params.type_v != cmd_params_defaults.type_v) { + fields.emplace_back("type_v"); + } + if (params.main_gpu.size() > 1 || params.main_gpu != cmd_params_defaults.main_gpu) { + fields.emplace_back("main_gpu"); + } + if (params.split_mode.size() > 1 || params.split_mode != cmd_params_defaults.split_mode) { + fields.emplace_back("split_mode"); + } + if (params.no_kv_offload.size() > 1 || params.no_kv_offload != cmd_params_defaults.no_kv_offload) { + fields.emplace_back("no_kv_offload"); + } + if (params.flash_attn.size() > 1 || params.flash_attn != cmd_params_defaults.flash_attn) { + fields.emplace_back("flash_attn"); + } + if (params.devices.size() > 1 || params.devices != cmd_params_defaults.devices) { + fields.emplace_back("devices"); + } + if (params.tensor_split.size() > 1 || params.tensor_split != cmd_params_defaults.tensor_split) { + fields.emplace_back("tensor_split"); + } + if (params.tensor_buft_overrides.size() > 1 || !vec_vec_tensor_buft_override_equal(params.tensor_buft_overrides, cmd_params_defaults.tensor_buft_overrides)) { + fields.emplace_back("tensor_buft_overrides"); + } + if (params.use_mmap.size() > 1 || params.use_mmap != cmd_params_defaults.use_mmap) { + fields.emplace_back("use_mmap"); + } + if (params.use_direct_io.size() > 1 || params.use_direct_io != cmd_params_defaults.use_direct_io) { + fields.emplace_back("use_direct_io"); + } + if (params.embeddings.size() > 1 || params.embeddings != cmd_params_defaults.embeddings) { + fields.emplace_back("embeddings"); + } + if (params.no_op_offload.size() > 1 || params.no_op_offload != cmd_params_defaults.no_op_offload) { + fields.emplace_back("no_op_offload"); + } + if (params.no_host.size() > 1 || params.no_host != cmd_params_defaults.no_host) { + fields.emplace_back("no_host"); + } + if (params.fit_params_target.size() > 1 || params.fit_params_target != cmd_params_defaults.fit_params_target) { + fields.emplace_back("fit_target"); + } + if (params.fit_params_min_ctx.size() > 1 || params.fit_params_min_ctx != cmd_params_defaults.fit_params_min_ctx) { + fields.emplace_back("fit_min_ctx"); + } + fields.emplace_back("test"); + fields.emplace_back("t/s"); + + fprintf(fout, "|"); + for (const auto & field : fields) { + fprintf(fout, " %*s |", get_field_width(field), get_field_display_name(field).c_str()); + } + fprintf(fout, "\n"); + fprintf(fout, "|"); + for (const auto & field : fields) { + int width = get_field_width(field); + fprintf(fout, " %s%s |", std::string(std::abs(width) - 1, '-').c_str(), width > 0 ? ":" : "-"); + } + fprintf(fout, "\n"); + } + + void print_test(const test & t) override { + std::map vmap = t.get_map(); + + fprintf(fout, "|"); + for (const auto & field : fields) { + std::string value; + char buf[128]; + if (field == "model") { + value = t.model_type; + } else if (field == "size") { + if (t.model_size < 1024 * 1024 * 1024) { + snprintf(buf, sizeof(buf), "%.2f MiB", t.model_size / 1024.0 / 1024.0); + } else { + snprintf(buf, sizeof(buf), "%.2f GiB", t.model_size / 1024.0 / 1024.0 / 1024.0); + } + value = buf; + } else if (field == "params") { + if (t.model_n_params < 1000 * 1000 * 1000) { + snprintf(buf, sizeof(buf), "%.2f M", t.model_n_params / 1e6); + } else { + snprintf(buf, sizeof(buf), "%.2f B", t.model_n_params / 1e9); + } + value = buf; + } else if (field == "backend") { + value = test::get_backend(); + } else if (field == "test") { + if (t.n_prompt > 0 && t.n_gen == 0) { + snprintf(buf, sizeof(buf), "pp%d", t.n_prompt); + } else if (t.n_gen > 0 && t.n_prompt == 0) { + snprintf(buf, sizeof(buf), "tg%d", t.n_gen); + } else { + snprintf(buf, sizeof(buf), "pp%d+tg%d", t.n_prompt, t.n_gen); + } + if (t.n_depth > 0) { + int len = strlen(buf); + snprintf(buf + len, sizeof(buf) - len, " @ d%d", t.n_depth); + } + value = buf; + } else if (field == "t/s") { + snprintf(buf, sizeof(buf), "%.2f ± %.2f", t.avg_ts(), t.stdev_ts()); + value = buf; + } else if (vmap.find(field) != vmap.end()) { + value = vmap.at(field); + } else { + assert(false); + exit(1); + } + + int width = get_field_width(field); + if (field == "t/s") { + // HACK: the utf-8 character is 2 bytes + width += 1; + } + fprintf(fout, " %*s |", width, value.c_str()); + } + fprintf(fout, "\n"); + } + + void print_footer() override { + fprintf(fout, "\nbuild: %s (%d)\n", test::build_commit.c_str(), test::build_number); + } +}; + +struct sql_printer : public printer { + static std::string get_sql_field_type(const std::string & field) { + switch (test::get_field_type(field)) { + case test::STRING: + return "TEXT"; + case test::BOOL: + case test::INT: + return "INTEGER"; + case test::FLOAT: + return "REAL"; + default: + assert(false); + exit(1); + } + } + + void print_header(const cmd_params & params) override { + std::vector fields = test::get_fields(); + fprintf(fout, "CREATE TABLE IF NOT EXISTS llama_bench (\n"); + for (size_t i = 0; i < fields.size(); i++) { + fprintf(fout, " %s %s%s\n", fields.at(i).c_str(), get_sql_field_type(fields.at(i)).c_str(), + i < fields.size() - 1 ? "," : ""); + } + fprintf(fout, ");\n"); + fprintf(fout, "\n"); + (void) params; + } + + void print_test(const test & t) override { + fprintf(fout, "INSERT INTO llama_bench (%s) ", join(test::get_fields(), ", ").c_str()); + fprintf(fout, "VALUES ("); + std::vector values = t.get_values(); + for (size_t i = 0; i < values.size(); i++) { + fprintf(fout, "'%s'%s", values.at(i).c_str(), i < values.size() - 1 ? ", " : ""); + } + fprintf(fout, ");\n"); + } +}; + +struct ctx_state { + int depth = 0; // in tokens + + std::vector buf; // the llama_context state buffer +}; + +static bool test_prompt(llama_context * ctx, int n_prompt, int n_batch, int n_threads) { + llama_set_n_threads(ctx, n_threads, n_threads); + + const llama_model * model = llama_get_model(ctx); + const llama_vocab * vocab = llama_model_get_vocab(model); + const int32_t n_vocab = llama_vocab_n_tokens(vocab); + + std::vector tokens(n_batch); + + int n_processed = 0; + + while (n_processed < n_prompt) { + int n_tokens = std::min(n_prompt - n_processed, n_batch); + tokens[0] = n_processed == 0 && llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; + for (int i = 1; i < n_tokens; i++) { + tokens[i] = std::rand() % n_vocab; + } + int res = llama_decode(ctx, llama_batch_get_one(tokens.data(), n_tokens)); + if (res != 0) { + fprintf(stderr, "%s: failed to decode prompt batch, res = %d\n", __func__, res); + return false; + } + n_processed += n_tokens; + } + + llama_synchronize(ctx); + return true; +} + +static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { + llama_set_n_threads(ctx, n_threads, n_threads); + + const llama_model * model = llama_get_model(ctx); + const llama_vocab * vocab = llama_model_get_vocab(model); + const int32_t n_vocab = llama_vocab_n_tokens(vocab); + + llama_token token = llama_vocab_get_add_bos(vocab) ? llama_vocab_bos(vocab) : std::rand() % n_vocab; + + for (int i = 0; i < n_gen; i++) { + int res = llama_decode(ctx, llama_batch_get_one(&token, 1)); + if (res != 0) { + fprintf(stderr, "%s: failed to decode generation batch, res = %d\n", __func__, res); + return false; + } + llama_synchronize(ctx); + token = std::rand() % n_vocab; + } + return true; +} + +static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { + (void) user_data; + if (level >= GGML_LOG_LEVEL_WARN) { + fputs(text, stderr); + } +} + +static std::unique_ptr create_printer(output_formats format) { + switch (format) { + case NONE: + return nullptr; + case CSV: + return std::unique_ptr(new csv_printer()); + case JSON: + return std::unique_ptr(new json_printer()); + case JSONL: + return std::unique_ptr(new jsonl_printer()); + case MARKDOWN: + return std::unique_ptr(new markdown_printer()); + case SQL: + return std::unique_ptr(new sql_printer()); + } + GGML_ABORT("fatal error"); +} + +int main(int argc, char ** argv) { + std::setlocale(LC_NUMERIC, "C"); + // try to set locale for unicode characters in markdown + std::setlocale(LC_CTYPE, ".UTF-8"); + +#if !defined(NDEBUG) + fprintf(stderr, "warning: asserts enabled, performance may be affected\n"); +#endif + +#if (defined(_MSC_VER) && defined(_DEBUG)) || (!defined(_MSC_VER) && !defined(__OPTIMIZE__)) + fprintf(stderr, "warning: debug build, performance may be affected\n"); +#endif + +#if defined(__SANITIZE_ADDRESS__) || defined(__SANITIZE_THREAD__) + fprintf(stderr, "warning: sanitizer enabled, performance may be affected\n"); +#endif + + // initialize backends + ggml_backend_load_all(); + + cmd_params params = parse_cmd_params(argc, argv); + + auto * cpu_dev = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU); + if (!cpu_dev) { + fprintf(stderr, "%s: error: CPU backend is not loaded\n", __func__); + return 1; + } + auto * cpu_reg = ggml_backend_dev_backend_reg(cpu_dev); + auto * ggml_threadpool_new_fn = (decltype(ggml_threadpool_new) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_new"); + auto * ggml_threadpool_free_fn = (decltype(ggml_threadpool_free) *) ggml_backend_reg_get_proc_address(cpu_reg, "ggml_threadpool_free"); + + // initialize llama.cpp + if (!params.verbose) { + llama_log_set(llama_null_log_callback, NULL); + } + llama_backend_init(); + llama_numa_init(params.numa); + + if (!set_process_priority(params.prio)) { + fprintf(stderr, "%s: error: failed to set process priority\n", __func__); + return 1; + } + + // initialize printer + std::unique_ptr p = create_printer(params.output_format); + std::unique_ptr p_err = create_printer(params.output_format_stderr); + + if (p) { + p->fout = stdout; + p->print_header(params); + } + + if (p_err) { + p_err->fout = stderr; + p_err->print_header(params); + } + + std::vector params_instances = get_cmd_params_instances(params); + + llama_model * lmodel = nullptr; + const cmd_params_instance * prev_inst = nullptr; + + // store the llama_context state at the previous depth that we performed a test + // ref: https://github.com/ggml-org/llama.cpp/pull/16944#issuecomment-3478151721 + ctx_state cstate; + + int params_idx = 0; + auto params_count = params_instances.size(); + for (const auto & inst : params_instances) { + params_idx++; + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: starting\n", params_idx, params_count); + } + auto mparams = inst.to_llama_mparams(); + auto cparams = inst.to_llama_cparams(); + + bool do_fit = inst.fit_target != cmd_params_defaults.fit_params_target[0] || + inst.fit_min_ctx != cmd_params_defaults.fit_params_min_ctx[0]; + + std::vector fit_tensor_split(llama_max_devices(), 0.0f); + std::vector fit_overrides(llama_max_tensor_buft_overrides(), {nullptr, nullptr}); + + if (do_fit) { + // free the previous model so fit sees full free VRAM + if (lmodel) { + llama_model_free(lmodel); + lmodel = nullptr; + prev_inst = nullptr; + } + + // use default n_gpu_layers and n_ctx so common_fit_params can adjust them + mparams.n_gpu_layers = llama_model_default_params().n_gpu_layers; + mparams.tensor_split = fit_tensor_split.data(); + mparams.tensor_buft_overrides = fit_overrides.data(); + cparams.n_ctx = 0; + + std::vector margins(llama_max_devices(), inst.fit_target * 1024 * 1024); + + uint32_t n_ctx_needed = inst.n_prompt + inst.n_gen + inst.n_depth; + cparams.n_ctx = std::max(cparams.n_ctx, n_ctx_needed); + + common_fit_params(inst.model.c_str(), &mparams, &cparams, + fit_tensor_split.data(), + fit_overrides.data(), + margins.data(), + inst.fit_min_ctx, + params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); + } + + // set reduction provider env var before model load (comm_init reads it) + { + const char * ar_val = (inst.reduction_provider == "auto") ? "" : inst.reduction_provider.c_str(); +#ifdef _WIN32 + _putenv_s("GGML_CUDA_ALLREDUCE", ar_val); +#else + setenv("GGML_CUDA_ALLREDUCE", ar_val, 1); +#endif + } + + // keep the same model between tests when possible + if (!lmodel || !prev_inst || !inst.equal_mparams(*prev_inst)) { + if (lmodel) { + llama_model_free(lmodel); + } + + lmodel = llama_model_load_from_file(inst.model.c_str(), mparams); + if (lmodel == NULL) { + fprintf(stderr, "%s: error: failed to load model '%s'\n", __func__, inst.model.c_str()); + return 1; + } + prev_inst = &inst; + } + + llama_context * ctx = llama_init_from_model(lmodel, cparams); + if (ctx == NULL) { + fprintf(stderr, "%s: error: failed to create context with model '%s'\n", __func__, inst.model.c_str()); + llama_model_free(lmodel); + return 1; + } + + test t(inst, lmodel, ctx); + + llama_memory_clear(llama_get_memory(ctx), false); + + // cool off before the test + if (params.delay) { + std::this_thread::sleep_for(std::chrono::seconds(params.delay)); + } + + struct ggml_threadpool_params tpp = ggml_threadpool_params_default(t.n_threads); + if (!parse_cpu_mask(t.cpu_mask, tpp.cpumask)) { + fprintf(stderr, "%s: failed to parse cpu-mask: %s\n", __func__, t.cpu_mask.c_str()); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + tpp.strict_cpu = t.cpu_strict; + tpp.poll = t.poll; + tpp.prio = params.prio; + + struct ggml_threadpool * threadpool = ggml_threadpool_new_fn(&tpp); + if (!threadpool) { + fprintf(stderr, "%s: threadpool create failed : n_threads %d\n", __func__, tpp.n_threads); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + + llama_attach_threadpool(ctx, threadpool, NULL); + + // warmup run + if (!params.no_warmup) { + if (t.n_prompt > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup prompt run\n", params_idx, params_count); + } + //test_prompt(ctx, std::min(t.n_batch, std::min(t.n_prompt, 32)), 0, t.n_batch, t.n_threads); + bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run prompt warmup\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + if (t.n_gen > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: warmup generation run\n", params_idx, params_count); + } + bool res = test_gen(ctx, 1, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run gen warmup\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + } + + for (int i = 0; i < params.reps; i++) { + llama_memory_clear(llama_get_memory(ctx), false); + + if (t.n_depth > 0) { + bool is_cached = t.n_depth == cstate.depth; + + if (is_cached) { + // if previously we have computed at this depth, just restore the state + const size_t ret = llama_state_seq_set_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); + if (ret == 0) { + // if the old state is incompatible with the current context - reprocess from scratch + is_cached = false; + } + } + + if (!is_cached) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_prompt(ctx, t.n_depth, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run depth\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + + // store the context state for reuse in later runs + cstate.depth = t.n_depth; + cstate.buf.resize(llama_state_seq_get_size(ctx, 0)); + llama_state_seq_get_data(ctx, cstate.buf.data(), cstate.buf.size(), 0); + } else { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: depth run %d/%d (cached)\n", params_idx, params_count, + i + 1, params.reps); + } + } + } + + uint64_t t_start = get_time_ns(); + + if (t.n_prompt > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: prompt run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_prompt(ctx, t.n_prompt, t.n_batch, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run prompt\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + if (t.n_gen > 0) { + if (params.progress) { + fprintf(stderr, "llama-bench: benchmark %d/%zu: generation run %d/%d\n", params_idx, params_count, + i + 1, params.reps); + } + bool res = test_gen(ctx, t.n_gen, t.n_threads); + if (!res) { + fprintf(stderr, "%s: error: failed to run gen\n", __func__); + llama_free(ctx); + llama_model_free(lmodel); + exit(1); + } + } + + uint64_t t_ns = get_time_ns() - t_start; + t.samples_ns.push_back(t_ns); + } + + if (p) { + p->print_test(t); + fflush(p->fout); + } + + if (p_err) { + p_err->print_test(t); + fflush(p_err->fout); + } + + llama_perf_context_print(ctx); + + llama_free(ctx); + + ggml_threadpool_free_fn(threadpool); + } + + llama_model_free(lmodel); + + if (p) { + p->print_footer(); + } + + if (p_err) { + p_err->print_footer(); + } + + llama_backend_free(); + + return 0; +} From bc8b080b4283272e33deee1382ff12810b2dc501 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 21:24:47 -0700 Subject: [PATCH 11/81] .gitattributes: force LF line endings to prevent Windows CRLF conversion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Sonnet 4.6 elopment environment, causing every line to show as changed in diffs against master. Co-Authored-By: Claude Sonnet 4.6 imit bailout: claims a ring slot via atomicAdd (single-GPU host atomics work on RTX 5090), writes fields, fences, sets completion flag, then all threads exit - Watchdog thread simply polls ring head counters every 1ms and prints any new complete records — no CUDA event queries, no mutex, no queue - Zero overhead on the dispatch path (no queue posting, no memset) - Watchdog shutdown returns within ~1ms (atomic bool, no drain) - On bailout the kernel skips Phase 3 entirely and exits cleanly Verified: 20/20 prefill soak test clean at ~1112 t/s, no hangs. Co-Authored-By: Claude Sonnet 4.6 P32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- .gitattributes | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.gitattributes b/.gitattributes index 06c85ad56e8..e9f9b494a72 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,3 +1,6 @@ +# Force LF line endings everywhere (prevent Windows CRLF conversion). +* text=auto eol=lf + # Treat the generated single-file WebUI build as binary for diff purposes. # Git's pack-file delta compression still works (byte-level), but this prevents # git diff from printing the entire minified file on every change. From 8da7e74e142be7761c1c58210f6282fda2541dca Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 21:57:59 -0700 Subject: [PATCH 12/81] ggml-cuda: move GGML_CUDA_AR_WATCHDOG from CMake option to local define MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The watchdog is development-only; a global CMake option is overkill. Move the toggle to a #define at the top of allreduce.cu (set to 0 by default) and remove the option from ggml/CMakeLists.txt and the CUDA CMakeLists.txt add_compile_definitions block. Co-Authored-By: Claude Sonnet 4.6 fences, sets completion flag, then all threads exit - Watchdog thread simply polls ring head counters every 1ms and prints any new complete records — no CUDA event queries, no mutex, no queue - Zero overhead on the dispatch path (no queue posting, no memset) - Watchdog shutdown returns within ~1ms (atomic bool, no drain) - On bailout the kernel skips Phase 3 entirely and exits cleanly Verified: 20/20 prefill soak test clean at ~1112 t/s, no hangs. Co-Authored-By: Claude Sonnet 4.6 P32, tensors <= 256 KB. Notes in NOTES-allreduce.md. Co-Authored-By: Claude Sonnet 4.6 --- ggml/CMakeLists.txt | 1 - ggml/src/ggml-cuda/CMakeLists.txt | 4 ---- ggml/src/ggml-cuda/allreduce.cu | 26 ++++++++++++++++---------- 3 files changed, 16 insertions(+), 15 deletions(-) diff --git a/ggml/CMakeLists.txt b/ggml/CMakeLists.txt index 820053a96c9..c721d76f188 100644 --- a/ggml/CMakeLists.txt +++ b/ggml/CMakeLists.txt @@ -209,7 +209,6 @@ option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashA option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) -option(GGML_CUDA_AR_WATCHDOG "ggml: enable internal AllReduce hang watchdog (debug)" OFF) set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING "ggml: cuda link binary compression mode; requires cuda 12.8+") set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") diff --git a/ggml/src/ggml-cuda/CMakeLists.txt b/ggml/src/ggml-cuda/CMakeLists.txt index 8f6c54d6cc0..1d94873c1a7 100644 --- a/ggml/src/ggml-cuda/CMakeLists.txt +++ b/ggml/src/ggml-cuda/CMakeLists.txt @@ -194,10 +194,6 @@ if (CUDAToolkit_FOUND) endif() endif() - if (GGML_CUDA_AR_WATCHDOG) - add_compile_definitions(GGML_CUDA_AR_WATCHDOG) - endif() - set(CUDA_CXX_FLAGS "") set(CUDA_FLAGS -use_fast_math -extended-lambda) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index c65449c490f..4225b037901 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -4,7 +4,13 @@ #include #include -#ifdef GGML_CUDA_AR_WATCHDOG +// Set to 1 to enable the AllReduce spin-limit watchdog (development only). +// When enabled, the debug kernel bails out after GGML_CUDA_AR_MAX_SPIN +// iterations and writes a record to a per-GPU ring buffer that the +// background watchdog thread prints. +#define GGML_CUDA_AR_WATCHDOG 0 + +#if GGML_CUDA_AR_WATCHDOG #include #include #include @@ -123,7 +129,7 @@ static __global__ void ggml_cuda_ar_f32_kernel( // and then sets the completion flag so the host watchdog thread can safely // read the record. // --------------------------------------------------------------------------- -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // One debug record written by the kernel on spin-limit bailout. struct ggml_cuda_ar_debug_record { @@ -254,7 +260,7 @@ static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; // preventing false-sharing stalls on the polling GPU. static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // Watchdog poll interval in milliseconds. static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 1; #endif @@ -279,7 +285,7 @@ struct ggml_cuda_ar_pipeline { // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // Per-GPU debug ring buffers in pinned host memory. Written by the debug // kernel on spin-limit bailout, read by the background watchdog thread. ggml_cuda_ar_debug_ring * debug_ring[GGML_CUDA_MAX_DEVICES]; @@ -301,7 +307,7 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, // this thread polls the ring head counters every 1ms and prints any new // complete records. Zero overhead on the dispatch path (no queue, no events). // --------------------------------------------------------------------------- -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { int last_seen[GGML_CUDA_MAX_DEVICES] = {}; @@ -352,7 +358,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( p->streams[i] = nullptr; p->ev_pool[i] = nullptr; } -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { p->debug_ring[i] = nullptr; } @@ -409,7 +415,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( memset(p->host_buf[i], 0, max_bytes); } -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // Per-GPU debug ring buffers: written by the kernel on spin-limit bailout, // polled by the background watchdog thread. Each ring is pinned host // memory accessed only by its owning GPU (single-GPU host atomics OK). @@ -507,7 +513,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { return; } -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // Stop the watchdog thread first — it only reads pinned host memory, // no GPU resources, so this is safe and returns within ~1ms. p->wdog_stop.store(true); @@ -544,7 +550,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { if (p->arrival) { cudaFreeHost(p->arrival); } -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG for (int i = 0; i < p->n_devices; ++i) { if (p->debug_ring[i]) { cudaFreeHost(p->debug_ring[i]); @@ -624,7 +630,7 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); -#ifdef GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG ggml_cuda_ar_f32_kernel_dbg<<streams[i]>>>( static_cast(tensors[i]->data), static_cast(tensors[i]->data), From 372d40830d9f7251a40698fbc0d6bfbda027d6bd Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 22:12:37 -0700 Subject: [PATCH 13/81] unify kernel debug paths --- ggml/src/ggml-cuda/allreduce.cu | 138 +++++++++----------------------- 1 file changed, 40 insertions(+), 98 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 4225b037901..5104343d558 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -39,7 +39,7 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { } // --------------------------------------------------------------------------- -// Single-kernel AllReduce — float32, 2 GPUs (production) +// Single-kernel AllReduce — float32, 2 GPUs // // Both GPUs run this kernel simultaneously in independent streams. Each GPU: // @@ -51,86 +51,14 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { // The single-block configuration means __syncthreads() is sufficient for // intra-block coordination and we can use the cheaper non-cooperative launch. // 256 threads gives good occupancy while keeping register pressure low. -// --------------------------------------------------------------------------- -static __global__ void ggml_cuda_ar_f32_kernel( - const float * __restrict__ sendbuf, - float * __restrict__ recvbuf, - float * __restrict__ host_mine, - const float * __restrict__ host_other, - int count, - int * arrival_mine, - int * arrival_other) { - - const int tid = threadIdx.x; - const int nt = blockDim.x; - const int count4 = count >> 2; - const int tail = count4 << 2; - - // Phase 1: vectorised D2H copy using float4 (16 bytes per load/store). - { - const float4 * s4 = reinterpret_cast(sendbuf); - float4 * d4 = reinterpret_cast(host_mine); - for (int i = tid; i < count4; i += nt) { - d4[i] = s4[i]; - } - if (tid < count - tail) { - host_mine[tail + tid] = sendbuf[tail + tid]; - } - } - - // Commit all host writes before signalling. - __threadfence_system(); - __syncthreads(); - - // Phase 2: thread 0 signals arrival, then spins for the peer. - if (tid == 0) { - ggml_cuda_ar_signal_set(arrival_mine); - - // ensure all GPUs have access to the arrival signal - __threadfence_system(); - - while (ggml_cuda_ar_signal_get(arrival_other) == 0) { - //__threadfence_system(); - __nanosleep(100); - } - } - - // Broadcast "peer has arrived" and acquire peer's host_other writes. - __syncthreads(); - __threadfence_system(); - - // Phase 3: reduce. - { - const float4 * s4 = reinterpret_cast(sendbuf); - const float4 * o4 = reinterpret_cast(host_other); - float4 * r4 = reinterpret_cast(recvbuf); - for (int i = tid; i < count4; i += nt) { - float4 a = s4[i]; - float4 b = o4[i]; - r4[i] = make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); - } - if (tid < count - tail) { - recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; - } - } -} - -// --------------------------------------------------------------------------- -// Watchdog debug variant — compiled only when GGML_CUDA_AR_WATCHDOG is defined. // -// Identical to the production kernel except Phase 2 has a spin limit +// When GGML_CUDA_AR_WATCHDOG is enabled, Phase 2 has a spin limit // (max_spin). If the limit is reached the kernel writes a debug record to // a per-GPU ring buffer in pinned host memory, then bails out — all threads // exit the kernel immediately (Phase 3 is skipped). -// -// The ring slot is claimed with atomicAdd on the ring head counter. Host -// memory atomics work for a single GPU on RTX 5090 (just not cross-GPU). -// After writing the record fields the kernel issues __threadfence_system() -// and then sets the completion flag so the host watchdog thread can safely -// read the record. // --------------------------------------------------------------------------- -#if GGML_CUDA_AR_WATCHDOG +#if GGML_CUDA_AR_WATCHDOG // One debug record written by the kernel on spin-limit bailout. struct ggml_cuda_ar_debug_record { int rank; // GPU rank (0 or 1) @@ -150,31 +78,39 @@ struct ggml_cuda_ar_debug_ring { int head; // next slot to write (GPU atomicAdd) ggml_cuda_ar_debug_record records[GGML_CUDA_AR_RING_SIZE]; }; +#endif // GGML_CUDA_AR_WATCHDOG -static __global__ void ggml_cuda_ar_f32_kernel_dbg( +static __global__ void ggml_cuda_ar_f32_kernel( const float * __restrict__ sendbuf, float * __restrict__ recvbuf, float * __restrict__ host_mine, const float * __restrict__ host_other, int count, int * arrival_mine, - int * arrival_other, - ggml_cuda_ar_debug_ring * ring, + int * arrival_other +#if GGML_CUDA_AR_WATCHDOG + ,ggml_cuda_ar_debug_ring * ring, int max_spin, int rank, - int ar_slot) { + int ar_slot +#endif + ) { +#if GGML_CUDA_AR_WATCHDOG __shared__ int bail; +#endif const int tid = threadIdx.x; const int nt = blockDim.x; const int count4 = count >> 2; const int tail = count4 << 2; +#if GGML_CUDA_AR_WATCHDOG if (tid == 0) { bail = 0; } __syncthreads(); +#endif - // Phase 1: D2H copy (identical to production kernel). + // Phase 1: vectorised D2H copy using float4 (16 bytes per load/store). { const float4 * s4 = reinterpret_cast(sendbuf); float4 * d4 = reinterpret_cast(host_mine); @@ -186,15 +122,16 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( } } + // Commit all host writes before signalling. __threadfence_system(); __syncthreads(); - // Phase 2: signal + instrumented spin. + // Phase 2: thread 0 signals arrival, then spins for the peer. if (tid == 0) { ggml_cuda_ar_signal_set(arrival_mine); +#if GGML_CUDA_AR_WATCHDOG int writeback = ggml_cuda_ar_signal_get(arrival_mine); - int spin = 0; int last = 0; while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { @@ -220,12 +157,19 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( } __nanosleep(100); } +#else + while (ggml_cuda_ar_signal_get(arrival_other) == 0) { + __nanosleep(100); + } +#endif } __syncthreads(); +#if GGML_CUDA_AR_WATCHDOG if (bail) { return; // all threads exit — skip Phase 3 } +#endif // Broadcast "peer has arrived" and acquire peer's host_other writes. __threadfence_system(); @@ -245,7 +189,6 @@ static __global__ void ggml_cuda_ar_f32_kernel_dbg( } } } -#endif // GGML_CUDA_AR_WATCHDOG // --------------------------------------------------------------------------- // Pipeline structure @@ -480,7 +423,14 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( p->host_buf[1 - r], static_cast(WARMUP_COUNT), ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r), - ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, 1 - r)); + ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, 1 - r) +#if GGML_CUDA_AR_WATCHDOG + ,p->debug_ring[r], + 0, // max_spin = 0 (no limit during warmup) + r, + 0 // slot = 0 +#endif + ); } } for (int i = 0; i < 2; ++i) { @@ -630,29 +580,21 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); -#if GGML_CUDA_AR_WATCHDOG - ggml_cuda_ar_f32_kernel_dbg<<streams[i]>>>( + ggml_cuda_ar_f32_kernel<<streams[i]>>>( static_cast(tensors[i]->data), static_cast(tensors[i]->data), p->host_buf[i], p->host_buf[peer], static_cast(ne), ggml_cuda_ar_arrival_ptr(p, slot, i), - ggml_cuda_ar_arrival_ptr(p, slot, peer), - p->debug_ring[i], + ggml_cuda_ar_arrival_ptr(p, slot, peer) +#if GGML_CUDA_AR_WATCHDOG + ,p->debug_ring[i], p->wdog_max_spin, i, - slot); -#else - ggml_cuda_ar_f32_kernel<<streams[i]>>>( - static_cast(tensors[i]->data), - static_cast(tensors[i]->data), - p->host_buf[i], - p->host_buf[peer], - static_cast(ne), - ggml_cuda_ar_arrival_ptr(p, slot, i), - ggml_cuda_ar_arrival_ptr(p, slot, peer)); + slot #endif + ); CUDA_CHECK(cudaGetLastError()); CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); From 5028250b52e0cd77cb2deb536fc1dd536e4dd368 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 22:22:35 -0700 Subject: [PATCH 14/81] use __threadfence_system explicitly (not in ggml_cuda_ar_signal_set) --- ggml/src/ggml-cuda/allreduce.cu | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 5104343d558..420eff8e365 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -30,9 +30,7 @@ // --------------------------------------------------------------------------- static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { - __threadfence_system(); // ensure all prior writes (D2H data) are globally visible *(volatile int *)p = 1; - __threadfence_system(); // ensure the signal itself is globally visible } static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; @@ -130,6 +128,8 @@ static __global__ void ggml_cuda_ar_f32_kernel( if (tid == 0) { ggml_cuda_ar_signal_set(arrival_mine); + __threadfence_system(); // ensure the signal itself is visible across all GPUs + #if GGML_CUDA_AR_WATCHDOG int writeback = ggml_cuda_ar_signal_get(arrival_mine); int spin = 0; From cfdb0a6bbe4e06712d648058f70a74371f138a25 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 22:49:27 -0700 Subject: [PATCH 15/81] preferentially use internal reduction for <=2 GPUs --- ggml/src/ggml-cuda/ggml-cuda.cu | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 8e489a092f5..a6920cbfd16 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1168,12 +1168,9 @@ struct ggml_backend_cuda_comm_context { // // Priority: // 1. GGML_CUDA_ALLREDUCE env var ("nccl" or "internal") — explicit override. -// 2. NCCL when compiled in (GGML_USE_NCCL defined). -// 3. Internal otherwise. -// -// Future: inspect NVLink topology via cudaDeviceGetP2PAttribute() with -// cudaDevP2PAttrNativeAtomicSupported to prefer INTERNAL on PCIe-only systems -// where host-staged reduction can beat NCCL for small tensors. +// 2. Internal for 2 GPUs (the optimised path). +// 3. NCCL as fallback for >2 GPUs when compiled in (GGML_USE_NCCL defined). +// 4. Internal otherwise. static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( const std::vector & device_ids) { const char * env = getenv("GGML_CUDA_ALLREDUCE"); @@ -1192,11 +1189,15 @@ static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); } + // Internal provider is the default for 2-GPU configurations. + if (device_ids.size() <= 2) { + return GGML_CUDA_ALLREDUCE_INTERNAL; + } + + // >2 GPUs: fall back to NCCL if available, otherwise internal. #ifdef GGML_USE_NCCL - GGML_UNUSED(device_ids); return GGML_CUDA_ALLREDUCE_NCCL; #else - GGML_UNUSED(device_ids); return GGML_CUDA_ALLREDUCE_INTERNAL; #endif } From 239f20e1e9b31aaba90a962330b396e19ac0b5cd Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 22 Apr 2026 23:08:23 -0700 Subject: [PATCH 16/81] templatize the main kernel to support fp16/bf16 --- ggml/src/ggml-cuda/allreduce.cu | 146 +++++++++++++++++++++----------- 1 file changed, 97 insertions(+), 49 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 420eff8e365..61612510a48 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -37,7 +37,7 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { } // --------------------------------------------------------------------------- -// Single-kernel AllReduce — float32, 2 GPUs +// Single-kernel AllReduce — 2 GPUs, supports float, half, and bfloat16. // // Both GPUs run this kernel simultaneously in independent streams. Each GPU: // @@ -78,30 +78,68 @@ struct ggml_cuda_ar_debug_ring { }; #endif // GGML_CUDA_AR_WATCHDOG -static __global__ void ggml_cuda_ar_f32_kernel( - const float * __restrict__ sendbuf, - float * __restrict__ recvbuf, - float * __restrict__ host_mine, - const float * __restrict__ host_other, - int count, - int * arrival_mine, - int * arrival_other +// --------------------------------------------------------------------------- +// Vectorised add helpers for Phase 3 reduction. All types use float4 +// (16 bytes) as the vector load unit for maximum PCIe throughput. +// --------------------------------------------------------------------------- +template +static __device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b); + +template <> +__device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b) { + return make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); +} + +template <> +__device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b) { + float4 r; + half2 * ha = reinterpret_cast(&a); + half2 * hb = reinterpret_cast(&b); + half2 * hr = reinterpret_cast(&r); + #pragma unroll + for (int k = 0; k < 4; ++k) { hr[k] = ha[k] + hb[k]; } + return r; +} + +template <> +__device__ __forceinline__ float4 ggml_cuda_ar_vec_add<__nv_bfloat16>(float4 a, float4 b) { + float4 r; + __nv_bfloat162 * ba = reinterpret_cast<__nv_bfloat162 *>(&a); + __nv_bfloat162 * bb = reinterpret_cast<__nv_bfloat162 *>(&b); + __nv_bfloat162 * br = reinterpret_cast<__nv_bfloat162 *>(&r); + #pragma unroll + for (int k = 0; k < 4; ++k) { br[k] = ba[k] + bb[k]; } + return r; +} + +template +static __global__ void ggml_cuda_ar_kernel( + const T * __restrict__ sendbuf, + T * __restrict__ recvbuf, + T * __restrict__ host_mine, + const T * __restrict__ host_other, + int count, + int * arrival_mine, + int * arrival_other #if GGML_CUDA_AR_WATCHDOG - ,ggml_cuda_ar_debug_ring * ring, - int max_spin, - int rank, - int ar_slot + ,ggml_cuda_ar_debug_ring * ring, + int max_spin, + int rank, + int ar_slot #endif ) { + // Number of elements of T per float4 vector (16 bytes). + constexpr int ELEMS_PER_VEC = 16 / sizeof(T); + #if GGML_CUDA_AR_WATCHDOG __shared__ int bail; #endif - const int tid = threadIdx.x; - const int nt = blockDim.x; - const int count4 = count >> 2; - const int tail = count4 << 2; + const int tid = threadIdx.x; + const int nt = blockDim.x; + const int count_vec = count / ELEMS_PER_VEC; + const int tail = count_vec * ELEMS_PER_VEC; #if GGML_CUDA_AR_WATCHDOG if (tid == 0) { bail = 0; } @@ -112,7 +150,7 @@ static __global__ void ggml_cuda_ar_f32_kernel( { const float4 * s4 = reinterpret_cast(sendbuf); float4 * d4 = reinterpret_cast(host_mine); - for (int i = tid; i < count4; i += nt) { + for (int i = tid; i < count_vec; i += nt) { d4[i] = s4[i]; } if (tid < count - tail) { @@ -137,7 +175,6 @@ static __global__ void ggml_cuda_ar_f32_kernel( while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { ++spin; if (max_spin > 0 && spin >= max_spin) { - // Acquire a ring slot via atomicAdd (single-GPU host atomics OK). int ri = atomicAdd(&ring->head, 1) % GGML_CUDA_AR_RING_SIZE; ggml_cuda_ar_debug_record * rec = &ring->records[ri]; @@ -148,9 +185,9 @@ static __global__ void ggml_cuda_ar_f32_kernel( rec->arrival_other = last; rec->count = count; - __threadfence_system(); // ensure fields visible before completion flag + __threadfence_system(); rec->complete = 1; - __threadfence_system(); // ensure completion flag visible to host + __threadfence_system(); bail = 1; break; @@ -167,7 +204,7 @@ static __global__ void ggml_cuda_ar_f32_kernel( __syncthreads(); #if GGML_CUDA_AR_WATCHDOG if (bail) { - return; // all threads exit — skip Phase 3 + return; } #endif @@ -179,10 +216,8 @@ static __global__ void ggml_cuda_ar_f32_kernel( const float4 * s4 = reinterpret_cast(sendbuf); const float4 * o4 = reinterpret_cast(host_other); float4 * r4 = reinterpret_cast(recvbuf); - for (int i = tid; i < count4; i += nt) { - float4 a = s4[i]; - float4 b = o4[i]; - r4[i] = make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); + for (int i = tid; i < count_vec; i += nt) { + r4[i] = ggml_cuda_ar_vec_add(s4[i], o4[i]); } if (tid < count - tail) { recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; @@ -220,7 +255,7 @@ struct ggml_cuda_ar_pipeline { uint64_t call_count; // Per-device resources. - float * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging + char * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking ggml_cuda_ar_event_slot *ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] @@ -348,8 +383,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( // Per-device pinned staging buffers. p->buf_bytes = max_bytes; for (int i = 0; i < n_devices; ++i) { - if (cudaHostAlloc(reinterpret_cast(&p->host_buf[i]), max_bytes, - cudaHostAllocPortable) != cudaSuccess) { + if (cudaHostAlloc(&p->host_buf[i], max_bytes, cudaHostAllocPortable) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for staging failed (%zu bytes)\n", __func__, max_bytes); ggml_cuda_ar_pipeline_free(p); @@ -410,17 +444,17 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( } if (warmup_ok) { - // Warmup always uses the production kernel (no debug overhead). + // Warmup uses float kernel. for (int iter = 0; iter < WARMUP_ITERS; ++iter) { for (int r = 0; r < 2; ++r) { *ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r) = 0; } for (int r = 0; r < 2; ++r) { ggml_cuda_set_device(p->devices[r]); - ggml_cuda_ar_f32_kernel<<streams[r]>>>( + ggml_cuda_ar_kernel<<streams[r]>>>( dev_buf[r], dev_buf[r], - p->host_buf[r], - p->host_buf[1 - r], + reinterpret_cast(p->host_buf[r]), + reinterpret_cast(p->host_buf[1 - r]), static_cast(WARMUP_COUNT), ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r), ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, 1 - r) @@ -528,13 +562,16 @@ bool ggml_cuda_ar_allreduce( return false; } - // Only FP32 tensors are handled by the kernel. - if (tensors[0]->type != GGML_TYPE_F32) { + const ggml_type type = tensors[0]->type; + const size_t type_size = ggml_type_size(type); + + // Only float, half, and bfloat16 tensors are handled by the kernel. + if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { return false; } const int64_t ne = ggml_nelements(tensors[0]); - const size_t bytes = (size_t)ne * sizeof(float); + const size_t bytes = (size_t)ne * type_size; if (ne == 0) { return true; @@ -580,21 +617,32 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); - ggml_cuda_ar_f32_kernel<<streams[i]>>>( - static_cast(tensors[i]->data), - static_cast(tensors[i]->data), - p->host_buf[i], - p->host_buf[peer], - static_cast(ne), - ggml_cuda_ar_arrival_ptr(p, slot, i), - ggml_cuda_ar_arrival_ptr(p, slot, peer) #if GGML_CUDA_AR_WATCHDOG - ,p->debug_ring[i], - p->wdog_max_spin, - i, - slot +#define GGML_CUDA_AR_WDOG_EXTRA_ARGS , p->debug_ring[i], p->wdog_max_spin, i, slot +#else +#define GGML_CUDA_AR_WDOG_EXTRA_ARGS #endif - ); + +#define LAUNCH_AR_KERNEL(T) \ + ggml_cuda_ar_kernel<<streams[i]>>>( \ + static_cast(tensors[i]->data), \ + static_cast(tensors[i]->data), \ + reinterpret_cast(p->host_buf[i]), \ + reinterpret_cast(p->host_buf[peer]), \ + static_cast(ne), \ + ggml_cuda_ar_arrival_ptr(p, slot, i), \ + ggml_cuda_ar_arrival_ptr(p, slot, peer) \ + GGML_CUDA_AR_WDOG_EXTRA_ARGS) + + switch (type) { + case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; + case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; + case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; + default: GGML_ASSERT(false); + } + +#undef LAUNCH_AR_KERNEL +#undef GGML_CUDA_AR_WDOG_EXTRA_ARGS CUDA_CHECK(cudaGetLastError()); CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); From cfcaee3509307b8424c7dcf273120869920e395b Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 13:17:38 -0700 Subject: [PATCH 17/81] restore llama-bench.cpp changes --- tools/llama-bench/llama-bench.cpp | 49 +++---------------------------- 1 file changed, 4 insertions(+), 45 deletions(-) diff --git a/tools/llama-bench/llama-bench.cpp b/tools/llama-bench/llama-bench.cpp index 366bf1cb57b..e21a80e697b 100644 --- a/tools/llama-bench/llama-bench.cpp +++ b/tools/llama-bench/llama-bench.cpp @@ -335,7 +335,6 @@ struct cmd_params { std::vector n_gpu_layers; std::vector n_cpu_moe; std::vector split_mode; - std::vector reduction_provider; std::vector main_gpu; std::vector no_kv_offload; std::vector flash_attn; @@ -380,7 +379,6 @@ static const cmd_params cmd_params_defaults = { /* n_gpu_layers */ { 99 }, /* n_cpu_moe */ { 0 }, /* split_mode */ { LLAMA_SPLIT_MODE_LAYER }, - /* reduction_provider */ { "auto" }, /* main_gpu */ { 0 }, /* no_kv_offload */ { false }, /* flash_attn */ { false }, @@ -451,7 +449,6 @@ static void print_usage(int /* argc */, char ** argv) { printf(" -ngl, --n-gpu-layers (default: %s)\n", join(cmd_params_defaults.n_gpu_layers, ",").c_str()); printf(" -ncmoe, --n-cpu-moe (default: %s)\n", join(cmd_params_defaults.n_cpu_moe, ",").c_str()); printf(" -sm, --split-mode (default: %s)\n", join(transform_to_str(cmd_params_defaults.split_mode, split_mode_str), ",").c_str()); - printf(" -rp, --reduction-provider allreduce provider for tensor split mode (default: %s)\n", join(cmd_params_defaults.reduction_provider, ",").c_str()); printf(" -mg, --main-gpu (default: %s)\n", join(cmd_params_defaults.main_gpu, ",").c_str()); printf(" -nkvo, --no-kv-offload <0|1> (default: %s)\n", join(cmd_params_defaults.no_kv_offload, ",").c_str()); printf(" -fa, --flash-attn <0|1> (default: %s)\n", join(cmd_params_defaults.flash_attn, ",").c_str()); @@ -762,22 +759,6 @@ static cmd_params parse_cmd_params(int argc, char ** argv) { break; } params.split_mode.insert(params.split_mode.end(), modes.begin(), modes.end()); - } else if (arg == "-rp" || arg == "--reduction-provider") { - if (++i >= argc) { - invalid_param = true; - break; - } - auto p = string_split(argv[i], split_delim); - for (const auto & v : p) { - if (v != "auto" && v != "nccl" && v != "internal") { - invalid_param = true; - break; - } - } - if (invalid_param) { - break; - } - params.reduction_provider.insert(params.reduction_provider.end(), p.begin(), p.end()); } else if (arg == "-mg" || arg == "--main-gpu") { if (++i >= argc) { invalid_param = true; @@ -1084,9 +1065,6 @@ static cmd_params parse_cmd_params(int argc, char ** argv) { if (params.split_mode.empty()) { params.split_mode = cmd_params_defaults.split_mode; } - if (params.reduction_provider.empty()) { - params.reduction_provider = cmd_params_defaults.reduction_provider; - } if (params.main_gpu.empty()) { params.main_gpu = cmd_params_defaults.main_gpu; } @@ -1158,7 +1136,6 @@ struct cmd_params_instance { int n_gpu_layers; int n_cpu_moe; llama_split_mode split_mode; - std::string reduction_provider; int main_gpu; bool no_kv_offload; bool flash_attn; @@ -1228,7 +1205,7 @@ struct cmd_params_instance { bool equal_mparams(const cmd_params_instance & other) const { return model == other.model && n_gpu_layers == other.n_gpu_layers && n_cpu_moe == other.n_cpu_moe && - split_mode == other.split_mode && reduction_provider == other.reduction_provider && + split_mode == other.split_mode && main_gpu == other.main_gpu && tensor_split == other.tensor_split && use_mmap == other.use_mmap && use_direct_io == other.use_direct_io && devices == other.devices && @@ -1265,7 +1242,6 @@ static std::vector get_cmd_params_instances(const cmd_param for (const auto & nl : params.n_gpu_layers) for (const auto & ncmoe : params.n_cpu_moe) for (const auto & sm : params.split_mode) - for (const auto & rp : params.reduction_provider) for (const auto & mg : params.main_gpu) for (const auto & devs : params.devices) for (const auto & ts : params.tensor_split) @@ -1306,7 +1282,6 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1344,7 +1319,6 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1382,7 +1356,6 @@ static std::vector get_cmd_params_instances(const cmd_param /* .n_gpu_layers = */ nl, /* .n_cpu_moe = */ ncmoe, /* .split_mode = */ sm, - /* .reduction_provider = */ rp, /* .main_gpu = */ mg, /* .no_kv_offload= */ nkvo, /* .flash_attn = */ fa, @@ -1425,7 +1398,6 @@ struct test { int n_gpu_layers; int n_cpu_moe; llama_split_mode split_mode; - std::string reduction_provider; int main_gpu; bool no_kv_offload; bool flash_attn; @@ -1466,7 +1438,6 @@ struct test { n_gpu_layers = inst.n_gpu_layers; n_cpu_moe = inst.n_cpu_moe; split_mode = inst.split_mode; - reduction_provider = inst.reduction_provider; main_gpu = inst.main_gpu; no_kv_offload = inst.no_kv_offload; flash_attn = inst.flash_attn; @@ -1535,7 +1506,7 @@ struct test { "model_filename", "model_type", "model_size", "model_n_params", "n_batch", "n_ubatch", "n_threads", "cpu_mask", "cpu_strict", "poll", "type_k", "type_v", "n_gpu_layers", "n_cpu_moe", "split_mode", - "reduction_provider", "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", + "main_gpu", "no_kv_offload", "flash_attn", "devices", "tensor_split", "tensor_buft_overrides", "use_mmap", "use_direct_io", "embeddings", "no_op_offload", "no_host", "fit_target", "fit_min_ctx", "n_prompt", "n_gen", "n_depth", @@ -1621,7 +1592,6 @@ struct test { std::to_string(n_gpu_layers), std::to_string(n_cpu_moe), split_mode_str(split_mode), - reduction_provider, std::to_string(main_gpu), std::to_string(no_kv_offload), std::to_string(flash_attn), @@ -2143,10 +2113,9 @@ static bool test_gen(llama_context * ctx, int n_gen, int n_threads) { } static void llama_null_log_callback(enum ggml_log_level level, const char * text, void * user_data) { + (void) level; + (void) text; (void) user_data; - if (level >= GGML_LOG_LEVEL_WARN) { - fputs(text, stderr); - } } static std::unique_ptr create_printer(output_formats format) { @@ -2276,16 +2245,6 @@ int main(int argc, char ** argv) { params.verbose ? GGML_LOG_LEVEL_DEBUG : GGML_LOG_LEVEL_ERROR); } - // set reduction provider env var before model load (comm_init reads it) - { - const char * ar_val = (inst.reduction_provider == "auto") ? "" : inst.reduction_provider.c_str(); -#ifdef _WIN32 - _putenv_s("GGML_CUDA_ALLREDUCE", ar_val); -#else - setenv("GGML_CUDA_ALLREDUCE", ar_val, 1); -#endif - } - // keep the same model between tests when possible if (!lmodel || !prev_inst || !inst.equal_mparams(*prev_inst)) { if (lmodel) { From bb01919be49a45814751a3bf0a769cbf145579bc Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 13:52:31 -0700 Subject: [PATCH 18/81] revert CMakeLists changes --- ggml/CMakeLists.txt | 1 - ggml/cmake/FindNCCL.cmake | 52 ++++--------------------------- ggml/src/ggml-cuda/CMakeLists.txt | 3 -- 3 files changed, 6 insertions(+), 50 deletions(-) diff --git a/ggml/CMakeLists.txt b/ggml/CMakeLists.txt index c721d76f188..2effd587b41 100644 --- a/ggml/CMakeLists.txt +++ b/ggml/CMakeLists.txt @@ -208,7 +208,6 @@ option(GGML_CUDA_FA "ggml: compile ggml FlashAttention C option(GGML_CUDA_FA_ALL_QUANTS "ggml: compile all quants for FlashAttention" OFF) option(GGML_CUDA_GRAPHS "ggml: use CUDA graphs (llama.cpp only)" ${GGML_CUDA_GRAPHS_DEFAULT}) option(GGML_CUDA_NCCL "ggml: use NVIDIA Collective Comm. Library" ON) -option(GGML_CUDA_NCCL_STATIC "ggml: link NCCL statically (ON) or dynamically (OFF)" OFF) set (GGML_CUDA_COMPRESSION_MODE "size" CACHE STRING "ggml: cuda link binary compression mode; requires cuda 12.8+") set_property(CACHE GGML_CUDA_COMPRESSION_MODE PROPERTY STRINGS "none;speed;balance;size") diff --git a/ggml/cmake/FindNCCL.cmake b/ggml/cmake/FindNCCL.cmake index 674ecddd25f..67511e2d56a 100644 --- a/ggml/cmake/FindNCCL.cmake +++ b/ggml/cmake/FindNCCL.cmake @@ -1,52 +1,16 @@ # cmake/FindNCCL.cmake -# NVIDIA does not distribute CMake files with NCCL, therefore use this file to find it instead. -# -# Inputs: -# NCCL_ROOT — root of an NCCL installation or source build tree -# NCCL_STATIC — if ON, prefer the static library and search cmake/lib/Release (or Debug); -# if OFF (default), prefer the shared/import library and search cmake/src/Release - -if(NCCL_STATIC) - # cmake source-build layout: cmake/lib//nccl_static.lib (or nccl.lib) - set(_nccl_lib_names nccl_static nccl) - set(_nccl_extra_lib_hints - "${NCCL_ROOT}/cmake/lib/Release" - "${NCCL_ROOT}/cmake/lib/Debug" - "${NCCL_ROOT}/cmake/lib" - ) -else() - # cmake source-build layout: cmake/src//nccl.lib (import lib for nccl.dll) - set(_nccl_lib_names nccl) - set(_nccl_extra_lib_hints - "${NCCL_ROOT}/cmake/src/Release" - "${NCCL_ROOT}/cmake/src/Debug" - "${NCCL_ROOT}/cmake/src" - ) -endif() +# NVIDIA does not distribute CMake files with NCCl, therefore use this file to find it instead. find_path(NCCL_INCLUDE_DIR NAMES nccl.h - HINTS - ${NCCL_ROOT} - "${NCCL_ROOT}/cmake/src/Release" - "${NCCL_ROOT}/cmake/src/Debug" - "${NCCL_ROOT}/cmake/src" - "${NCCL_ROOT}/cmake" - $ENV{NCCL_ROOT} - $ENV{CUDA_HOME} - /usr/local/cuda - PATH_SUFFIXES include src/include + HINTS ${NCCL_ROOT} $ENV{NCCL_ROOT} $ENV{CUDA_HOME} /usr/local/cuda + PATH_SUFFIXES include ) find_library(NCCL_LIBRARY - NAMES ${_nccl_lib_names} - HINTS - ${_nccl_extra_lib_hints} - ${NCCL_ROOT} - $ENV{NCCL_ROOT} - $ENV{CUDA_HOME} - /usr/local/cuda + NAMES nccl + HINTS ${NCCL_ROOT} $ENV{NCCL_ROOT} $ENV{CUDA_HOME} /usr/local/cuda PATH_SUFFIXES lib lib64 ) @@ -61,11 +25,7 @@ if(NCCL_FOUND) set(NCCL_INCLUDE_DIRS ${NCCL_INCLUDE_DIR}) if(NOT TARGET NCCL::NCCL) - if(NCCL_STATIC) - add_library(NCCL::NCCL STATIC IMPORTED) - else() - add_library(NCCL::NCCL UNKNOWN IMPORTED) - endif() + add_library(NCCL::NCCL UNKNOWN IMPORTED) set_target_properties(NCCL::NCCL PROPERTIES IMPORTED_LOCATION "${NCCL_LIBRARY}" INTERFACE_INCLUDE_DIRECTORIES "${NCCL_INCLUDE_DIR}" diff --git a/ggml/src/ggml-cuda/CMakeLists.txt b/ggml/src/ggml-cuda/CMakeLists.txt index 1d94873c1a7..b54d4a6b107 100644 --- a/ggml/src/ggml-cuda/CMakeLists.txt +++ b/ggml/src/ggml-cuda/CMakeLists.txt @@ -182,9 +182,6 @@ if (CUDAToolkit_FOUND) endif() if (GGML_CUDA_NCCL) - if (GGML_CUDA_NCCL_STATIC) - set(NCCL_STATIC ON) - endif() find_package(NCCL) if (NCCL_FOUND) add_compile_definitions(GGML_USE_NCCL) From ebc31bccfaa698ca78ae7ab05b4617f20b56e6db Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 14:15:25 -0700 Subject: [PATCH 19/81] remove notes from repo --- NOTES-allreduce.md | 311 --------------------------------------------- 1 file changed, 311 deletions(-) delete mode 100644 NOTES-allreduce.md diff --git a/NOTES-allreduce.md b/NOTES-allreduce.md deleted file mode 100644 index 49734567f76..00000000000 --- a/NOTES-allreduce.md +++ /dev/null @@ -1,311 +0,0 @@ -# AllReduce Provider Abstraction — Working Notes - -## Context - -Tensor-parallel mode (`LLAMA_SPLIT_MODE_TENSOR = 3`) splits attention and FFN weight -matrices across N GPUs. Each GPU computes a partial result; an AllReduce sums them -before the next layer begins. - -## Where the Reduction Happens - -``` -src/llama-context.cpp — validates SPLIT_MODE_TENSOR requirements (FlashAttn required, no KV quant) -src/llama-model.cpp — llama_meta_device_get_split_state(): assigns split axis per tensor - attn_q/k/v, ffn_up/gate → PARTIAL (needs AllReduce) - output → MIRRORED (no AllReduce) -ggml/src/ggml-backend-meta.cpp — ggml_backend_meta_graph_compute(): drives the subgraph loop - after each PARTIAL subgraph: calls comm_allreduce(), or - falls back to allreduce_fallback() (CPU-based) -ggml/src/ggml-cuda/ggml-cuda.cu — the CUDA-side implementations (NCCL + future internal) -``` - -### Subgraph Execution Loop (ggml-backend-meta.cpp ~line 2023) - -``` -for each subgraph i: - compute subgraph on each GPU in parallel - if i < last_subgraph: - if comm_ctx set: - try comm_allreduce(comm_ctx, last_nodes_per_gpu[]) - if allreduce failed (or no comm_ctx): - allreduce_fallback(i) ← copies to CPU, reduces, copies back -``` - -## Data Structures - -| Struct | File | Purpose | -|--------|------|---------| -| `ggml_backend_cuda_comm_context` | `ggml-cuda.cu` | holds provider enum + NCCL comms (or future internal state) | -| `ggml_backend_meta_context` | `ggml-backend-meta.cpp` | holds `comm_ctx` (opaque) + `comm_allreduce` fn ptr | -| `ggml_cuda_device_info` | `common.cuh` | per-device CC, VRAM, default split ratios | - -## Provider Abstraction Added - -### New file: `ggml/src/ggml-cuda/comm.cuh` - -Defines `enum ggml_cuda_allreduce_provider`: -- `GGML_CUDA_ALLREDUCE_NCCL` — NCCL/RCCL (default when compiled in) -- `GGML_CUDA_ALLREDUCE_INTERNAL` — internal host/CUDA staged reduction (stub for now) - -### Changes to `ggml/src/ggml-cuda/ggml-cuda.cu` - -- `ggml_backend_cuda_comm_context` now always exists; holds `provider` + conditionally `comms`. -- `ggml_cuda_select_allreduce_provider()` — new selection function (see below). -- `ggml_backend_cuda_comm_init()` — constructs context, selects provider, inits NCCL comms or internal state. -- `ggml_backend_cuda_comm_allreduce_tensor()` — dispatches to `_nccl` or `_internal` helper. -- `ggml_backend_cuda_comm_allreduce_nccl()` — extracted from old monolithic function; logic unchanged. -- `ggml_backend_cuda_comm_allreduce_internal()` — stub returning `false` (triggers meta fallback). - -The public interface (`ggml_backend_comm_init` / `_free` / `_allreduce_tensor` proc addresses) is unchanged. - -## Provider Selection Logic (`ggml_cuda_select_allreduce_provider`) - -Priority order: -1. `GGML_CUDA_ALLREDUCE=nccl` env var — force NCCL (warn if not compiled in). -2. `GGML_CUDA_ALLREDUCE=internal` env var — force internal. -3. NCCL when `GGML_USE_NCCL` is defined at compile time. -4. INTERNAL otherwise (with a warning on NVIDIA non-HIP/MUSA builds). - -Future: inspect hardware topology before choosing the default: - -```cpp -// Check if all device pairs have direct NVLink: -int native_atomic; -cudaDeviceGetP2PAttribute(&native_atomic, - cudaDevP2PAttrNativeAtomicSupported, dev_i, dev_j); -// If any pair lacks NVLink, internal may win for small tensors on PCIe. -``` - -## Files Changed - -``` -ggml/src/ggml-cuda/comm.cuh NEW — provider enum -ggml/src/ggml-cuda/ggml-cuda.cu MOD — provider selection, dispatch, NCCL helper extracted, internal stub -``` - -## Files NOT Changed (intentionally) - -- `ggml/src/ggml-backend-meta.cpp` — no changes needed; uses opaque `comm_ctx` + fn ptr already. -- `ggml/include/ggml-backend.h` — public `comm_*` typedef signatures unchanged. -- `include/llama.h` — `llama_split_mode` enum unchanged. - ---- - -## Prototype Analysis: `nccl_injector_prototype/` - -### What the prototype is - -A Windows DLL injected via Microsoft Detours that intercepts NCCL calls and reroutes -AllReduce to a faster internal kernel for the 2-GPU float32 case. We are NOT using the -injection/Detours machinery — we're implementing directly inside llama.cpp. - -### Single-Phase Kernel (what we're using) - -The prototype has two strategies. We only want the **single-phase merged kernel** -(`allreduce_f32_kernel` in `src/kernels.cu`). It merges D2H copy + cross-GPU -synchronization + reduction into one kernel launch per GPU. - -**Execution: 1 block × 256 threads per GPU.** - -``` -Phase A (all 256 threads): vectorized D2H copy, sendbuf → host_mine - - float4 loads (16 bytes/thread/iteration) for the bulk - - scalar tail for remainder if count % 4 != 0 - __threadfence_system() + __syncthreads() ← make D2H visible system-wide - -Phase B (thread 0 only): signal + spin - signal_publish(arrival_mine, 1) ← volatile write + __threadfence_system() - while signal_observe(arrival_other) == 0: ← volatile read, __nanosleep(100) between polls - (optional: log spin count to debug buf every 4096 iters) - __syncthreads() ← broadcast "both D2H done" to all threads - __threadfence_system() ← acquire peer's host_other writes - -Phase C (all 256 threads): reduce - recvbuf[i] = sendbuf[i] + host_other[i] ← float4 vectorized -``` - -**Why it's fast:** the D2H copy and the cross-GPU spin overlap naturally — GPU-0 starts -spinning while GPU-1's 256 threads are still copying their data. No extra kernel launches -or host round-trips. - -### Signal Mechanism - -Three options exist via `SIGNAL_MECHANISM` macro; default (and recommended) is 1: - -```cuda -// Publish: volatile write + system fence -*(volatile int*)p = value; -__threadfence_system(); - -// Observe: volatile read (no fence needed — __threadfence_system() after syncthreads covers it) -return *(const volatile int*)p; -``` - -One int per GPU. Values: 0 = not arrived, 1 = arrived. Reset to 0 before each call. -Single writer per slot (owning GPU), single reader (peer GPU) — no atomics needed. - -### Host-Side Setup (what to port, minus the NCCL hooks) - -**Per-pipeline state to allocate at `comm_init` time:** - -``` -host_buf[N] float* cudaMallocHost, one per GPU, >= max_tensor_bytes -arrival[POOL×N] int* cudaMallocHost, ring buffer, one int slot per GPU per in-flight call -stream[N] cudaStream_t cudaStreamCreateWithFlags(cudaStreamNonBlocking) -ev_pool[N][POOL] cudaEvent_t cudaEventCreateWithFlags(cudaEventDisableTiming) - × 2 events per slot (app = "wait for upstream work", ker = "kernel done") -debug[N×4] int* cudaMallocHost, optional, 4 ints per GPU for spin diagnostics -``` - -**Pool size:** 128 slots in the prototype. Events + arrival slots wrap together; must -sync on `ev_pool[r][slot].ker` before reusing arrival slot (slot ownership check). - -**Kernel dispatch sequence:** - -```cpp -// For each GPU r in parallel: -cudaEventRecord(ev[r].app, upstream_stream[r]); // capture upstream work -cudaStreamWaitEvent(internal_stream[r], ev[r].app); // internal stream waits for it -launch_allreduce_kernel(..., internal_stream[r]); // launch merged kernel -cudaEventRecord(ev[r].ker, internal_stream[r]); // record kernel completion -cudaStreamWaitEvent(upstream_stream[r], ev[r].ker); // upstream waits for kernel -``` - -This inserts the allreduce into the existing CUDA streams without blocking the host. - -**Warmup:** 64 iterations with 32 KB payloads at `comm_init` time. Amortizes -driver overhead and encourages GPU clock boost before real inference begins. - -**Watchdog (optional):** poll arrival + debug values from host every ~20 ms to detect -deadlocks without killing the process. - -### What to Discard (Detours/Injection Overhead) - -| File | Reason to skip | -|------|---------------| -| `src/dllmain.cpp` | DLL entry point, Detours attach/detach | -| `src/launcher.cpp` | Standalone DLL injector executable | -| `src/nccl_types.h` | NCCL function pointer typedefs (not needed when calling directly) | -| `src/hooks.cpp` (partially) | NCCL function wrapping, PendingOp queue, GroupStart/End logic | -| `src/hooks.h` | Hook declarations | - -**Keep from `hooks.cpp`:** -- `AllReducePipeline` struct (minus NCCL-specific fields) -- `init_ar_pipeline()` logic -- `execute_all_reduce_kernel()` dispatch logic (adapted for our stream model) - -**Keep from `kernels.cu`:** -- `allreduce_f32_kernel` exactly as-is (can rename) -- `signal_publish` / `signal_observe` device functions -- `launch_allreduce_f32` wrapper (adapt to our context) - -**Discard from `kernels.cu`:** -- `allreduce_d2h_f32_kernel` — phase 1 of two-phase approach -- `allreduce_reduce_f32_kernel` — phase 2 of two-phase approach - -### Current Limitations & Extension Plan - -**Current prototype only handles:** -- Exactly 2 GPUs -- `float32` data type -- Tensors ≤ 256 KB (64K floats, `AR_KERNEL_THRESHOLD`) -- Sum reduction only - ---- - -## Extension Plan for the Internal Implementation - -### Data Types Beyond float32 - -The prototype's kernel is float32 only. In llama.cpp the allreduce tensors are always -FP32 (the NCCL path already converts larger tensors to BF16 before sending and back -after). We should follow the same pattern: - -**Strategy A — FP32 kernel only (simplest, sufficient for most cases):** -- Tensors ≤ threshold: run internal kernel as FP32 directly (matches prototype) -- Tensors > threshold: convert F32→BF16 on GPU, run BF16 kernel, convert back - - Halves PCIe/pinned-host bandwidth for large tensors - - BF16 kernel is identical structure but with `__nv_bfloat16` / `__nv_bfloat162` - -**Strategy B — templated kernel:** - -```cuda -template -__global__ void allreduce_kernel( - const T* sendbuf, T* recvbuf, - AccT* host_mine, const AccT* host_other, - int count, int* arrival_mine, int* arrival_other, ...) -{ - // D2H: convert T → AccT on the fly (if T != AccT), store AccT to host_mine - // Reduce: read AccT from sendbuf (via on-the-fly upcast) + host_other, write T to recvbuf -} -``` - -Instantiate for: -- `` — FP32 direct (fast, bulk of tensors) -- `` — BF16 tensors, accumulate as FP32 in host_mine -- `` — FP16 tensors, accumulate as FP32 - -The host_mine staging buffer always stores the accumulation type (float), so size is -always `count * sizeof(float)` regardless of tensor type. Simpler than varying buffer types. - -### Tensor Size Beyond 256 KB - -The prototype bails to CPU sync for large tensors. Options: - -**Option 1 — Multi-block kernel (recommended):** -Launch `ceil(count / BLOCK_ELEMENTS)` blocks instead of 1. Each block handles its own -arrival signaling independently (need one arrival int pair per block, or use a shared -atomic). This allows pipelining — later blocks can start D2H while earlier blocks -have already signaled. - -**Option 2 — Chunked sequential:** -Call the single-block kernel in a loop, each call covering `CHUNK_SIZE` elements. -Simple but adds kernel launch overhead. - -**Option 3 — Keep threshold, fall back to NCCL/CPU for large:** -The NCCL path already handles large tensors well (BF16 compressed). Use internal -only for tensors under a tuned threshold where it beats NCCL. This is probably the -right first step — just match or beat NCCL in the size range where NCCL has latency -overhead. - -### More Than 2 GPUs - -The prototype is hardcoded 2-GPU. The single-phase approach generalizes to N GPUs: - -**For N=3 or N=4 (small N), tree or ring approach:** - -**Ring AllReduce (reduce-scatter + all-gather):** -1. Reduce-scatter: each GPU sends to next, keeps accumulated result for its chunk -2. All-gather: each GPU sends its final chunk to all others - -For N=2 the ring degenerates to the simple pairwise protocol already in the prototype. -The arrival mechanism needs one slot per `(gpu, neighbor)` pair. - -**Alternative for small N: star topology** (one GPU is root): -1. All non-root GPUs send to root's host_buf in parallel -2. Root reduces all contributions -3. Root broadcasts to all non-root - -Simpler to implement than ring but root becomes bottleneck for N > 2. - -For the initial implementation: focus on N=2 (covers the most common dual-GPU case), -then extend to N=4 for 4×GPU servers. - -### Size Threshold Tuning - -The prototype uses 256 KB for PCIe 4.0 x16. Our threshold should be determined by -benchmarking; likely different on PCIe 5.0 and definitely different on NVLink. -Expose as `GGML_CUDA_ALLREDUCE_INTERNAL_THRESHOLD` env var (elements, default 65536) -so users can tune without recompiling. - ---- - -## Open Questions - -1. What are the actual tensor shapes/sizes in the allreduce calls during inference? - Need a trace to know what the P50/P95 sizes are. -2. Target GPU topology? NVLink or PCIe? Determines whether internal can beat NCCL. -3. Is BF16 staging acceptable precision-wise, or is FP32 end-to-end required? -4. How many GPUs max? Design differs significantly between N=2 and N≥8. -5. Should we support the watchdog/spin limit for hang detection in production? From 014ad9f61d682c278ea417fdd90325cdd29eb755 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 14:47:49 -0700 Subject: [PATCH 20/81] remove dead warmup code --- ggml/src/ggml-cuda/allreduce.cu | 65 --------------------------------- 1 file changed, 65 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 61612510a48..a9dad4c13ab 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -420,71 +420,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( } #endif -#if 0 - // Warmup: run the kernel N times to pay first-use driver / PCIe / - // page-mapping costs during model load and encourage the GPU clock - // governor to boost before inference begins. - if (n_devices == 2) { - printf("ggml_cuda_ar_pipeline_init warmup\n"); - - constexpr int WARMUP_ITERS = 64; - constexpr size_t WARMUP_COUNT = 8192; // 32 KB of fp32 - constexpr size_t WARMUP_BYTES = WARMUP_COUNT * sizeof(float); - - float * dev_buf[2] = {}; - bool warmup_ok = true; - for (int i = 0; i < 2; ++i) { - ggml_cuda_set_device(p->devices[i]); - if (cudaMalloc(reinterpret_cast(&dev_buf[i]), WARMUP_BYTES) != cudaSuccess) { - GGML_LOG_WARN("%s: warmup alloc failed for device %d, skipping\n", - __func__, p->devices[i]); - warmup_ok = false; - break; - } - } - - if (warmup_ok) { - // Warmup uses float kernel. - for (int iter = 0; iter < WARMUP_ITERS; ++iter) { - for (int r = 0; r < 2; ++r) { - *ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r) = 0; - } - for (int r = 0; r < 2; ++r) { - ggml_cuda_set_device(p->devices[r]); - ggml_cuda_ar_kernel<<streams[r]>>>( - dev_buf[r], dev_buf[r], - reinterpret_cast(p->host_buf[r]), - reinterpret_cast(p->host_buf[1 - r]), - static_cast(WARMUP_COUNT), - ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, r), - ggml_cuda_ar_arrival_ptr(p, /*slot=*/0, 1 - r) -#if GGML_CUDA_AR_WATCHDOG - ,p->debug_ring[r], - 0, // max_spin = 0 (no limit during warmup) - r, - 0 // slot = 0 -#endif - ); - } - } - for (int i = 0; i < 2; ++i) { - ggml_cuda_set_device(p->devices[i]); - cudaStreamSynchronize(p->streams[i]); - } - GGML_LOG_DEBUG("%s: warmup complete (%d iters x %zu KB)\n", - __func__, WARMUP_ITERS, WARMUP_BYTES >> 10); - } - - for (int i = 0; i < 2; ++i) { - if (dev_buf[i]) { - ggml_cuda_set_device(p->devices[i]); - cudaFree(dev_buf[i]); - } - } - - printf("ggml_cuda_ar_pipeline_init warmup finished\n"); - } -#endif GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", __func__, n_devices, max_bytes >> 10); From a6981b576ad1852541e995b033801233ba2f05c9 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 14:55:44 -0700 Subject: [PATCH 21/81] fix comments --- ggml/src/ggml-cuda/allreduce.cuh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index f14ff4b6175..dbeedb63c0c 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -13,7 +13,7 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB // Opaque pipeline context — owns all pinned buffers, streams, and events. struct ggml_cuda_ar_pipeline; -// Allocate and warm up a pipeline for n_devices GPUs. +// Allocate a pipeline for n_devices GPUs. // devices[] holds the CUDA device IDs in rank order. // max_bytes is the staging buffer size per device; must be at least as large // as the largest tensor that will be reduced. @@ -26,7 +26,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); // Execute an in-place AllReduce (sum) across tensors[0..n_devices-1]. // tensors[i] must live on the device managed by backends[i] and be -// contiguous FP32. +// contiguous F32, F16, or BF16. // Returns true on success. Returns false when the tensor type or size is // outside the currently supported range; the caller should fall back to // another provider (NCCL or the meta-backend CPU reduce). @@ -34,3 +34,4 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * pipeline, ggml_backend_t * backends, ggml_tensor ** tensors); +** tensors); From 77c0e36ceb98a434b568d01a2a9feecf37c3def9 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 15:29:21 -0700 Subject: [PATCH 22/81] improve reduction provider fallback code --- ggml/src/ggml-cuda/allreduce.cuh | 3 +- ggml/src/ggml-cuda/comm.cuh | 4 +- ggml/src/ggml-cuda/ggml-cuda.cu | 127 ++++++++++++++++++++++++++++--- 3 files changed, 118 insertions(+), 16 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index dbeedb63c0c..77466011a04 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -27,11 +27,10 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); // Execute an in-place AllReduce (sum) across tensors[0..n_devices-1]. // tensors[i] must live on the device managed by backends[i] and be // contiguous F32, F16, or BF16. -// Returns true on success. Returns false when the tensor type or size is +// Returns true on success. Returns false when the tensor type or size is // outside the currently supported range; the caller should fall back to // another provider (NCCL or the meta-backend CPU reduce). bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * pipeline, ggml_backend_t * backends, ggml_tensor ** tensors); -** tensors); diff --git a/ggml/src/ggml-cuda/comm.cuh b/ggml/src/ggml-cuda/comm.cuh index ee738d4555c..c26d1e1dd76 100644 --- a/ggml/src/ggml-cuda/comm.cuh +++ b/ggml/src/ggml-cuda/comm.cuh @@ -6,9 +6,9 @@ // N GPUs and requires an AllReduce after each segment to sum the partial results. // This enum selects which implementation performs that reduction. // -// The active provider is chosen once at communicator init time by +// The preferred provider is chosen once at communicator init time by // ggml_cuda_select_allreduce_provider() and stored in -// ggml_backend_cuda_comm_context::provider. +// ggml_backend_cuda_comm_context::preferred_provider. enum ggml_cuda_allreduce_provider { // NVIDIA/AMD Collective Communications Library (NCCL/RCCL). // Optimal on NVLink/NVSwitch topologies; auto-selects the best transport. diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index a6920cbfd16..a016e881a48 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1141,20 +1141,22 @@ static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_inte }; // Communication context for multi-GPU AllReduce during tensor parallelism. -// Created once per meta backend instance; provider is fixed at init time. +// Created once per meta backend instance; the preferred provider is chosen at +// init time, but per-call dispatch may fall back to another CUDA provider when +// the preferred one does not support a tensor configuration. struct ggml_backend_cuda_comm_context { - ggml_cuda_allreduce_provider provider; + ggml_cuda_allreduce_provider preferred_provider; std::vector backends; #ifdef GGML_USE_NCCL - std::vector comms; // valid when provider == GGML_CUDA_ALLREDUCE_NCCL + std::vector comms; #endif - ggml_cuda_ar_pipeline * ar_pipeline = nullptr; // valid when provider == GGML_CUDA_ALLREDUCE_INTERNAL + ggml_cuda_ar_pipeline * ar_pipeline = nullptr; ~ggml_backend_cuda_comm_context() { #ifdef GGML_USE_NCCL - if (provider == GGML_CUDA_ALLREDUCE_NCCL) { + if (!comms.empty()) { for (ncclComm_t comm : comms) { NCCL_CHECK(ncclCommDestroy(comm)); } @@ -1225,7 +1227,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba const ggml_cuda_allreduce_provider provider = ggml_cuda_select_allreduce_provider(dev_ids); auto * ret = new ggml_backend_cuda_comm_context; - ret->provider = provider; + ret->preferred_provider = provider; ret->backends.assign(backends, backends + n_backends); switch (provider) { @@ -1248,6 +1250,10 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba delete ret; return nullptr; } +#ifdef GGML_USE_NCCL + ret->comms.resize(n_backends); + NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); +#endif } break; } @@ -1336,18 +1342,115 @@ static bool ggml_backend_cuda_comm_allreduce_internal( return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); } +enum ggml_cuda_comm_allreduce_result { + GGML_CUDA_COMM_ALLREDUCE_SUCCESS, + GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED, + GGML_CUDA_COMM_ALLREDUCE_FAILED, +}; + +static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_internal( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + if (comm_ctx->ar_pipeline == nullptr) { + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + } + + const size_t n_backends = comm_ctx->backends.size(); + GGML_ASSERT(n_backends >= 1); + GGML_ASSERT(tensors[0] != nullptr); + + const int64_t ne = ggml_nelements(tensors[0]); + const ggml_type type = tensors[0]->type; + + if (n_backends != 2) { + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + } + + if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + } + + if (ne == 0) { + return GGML_CUDA_COMM_ALLREDUCE_SUCCESS; + } + + const size_t bytes = (size_t) ne * ggml_type_size(type); + if (bytes > GGML_CUDA_AR_MAX_BYTES) { + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + } + + for (size_t i = 0; i < n_backends; ++i) { + if (tensors[i] == nullptr) { + return GGML_CUDA_COMM_ALLREDUCE_FAILED; + } + if (ggml_nelements(tensors[i]) != ne || tensors[i]->type != type) { + return GGML_CUDA_COMM_ALLREDUCE_FAILED; + } + } + + return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors) + ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS + : GGML_CUDA_COMM_ALLREDUCE_FAILED; +} + +#ifdef GGML_USE_NCCL +static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_nccl( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + if (comm_ctx->comms.empty()) { + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + } + return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors) + ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS + : GGML_CUDA_COMM_ALLREDUCE_FAILED; +} +#else +static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_nccl( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + GGML_UNUSED_VARS(comm_ctx, tensors); + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; +} +#endif + static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { if (comm_ctx_v == nullptr) { return false; } auto * comm_ctx = static_cast(comm_ctx_v); - switch (comm_ctx->provider) { -#ifdef GGML_USE_NCCL - case GGML_CUDA_ALLREDUCE_NCCL: - return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); -#endif + + auto try_in_order = [&](ggml_cuda_allreduce_provider first) -> bool { + const ggml_cuda_allreduce_provider second = + first == GGML_CUDA_ALLREDUCE_INTERNAL + ? GGML_CUDA_ALLREDUCE_NCCL + : GGML_CUDA_ALLREDUCE_INTERNAL; + const ggml_cuda_allreduce_provider order[2] = { first, second }; + + for (ggml_cuda_allreduce_provider provider : order) { + ggml_cuda_comm_allreduce_result result = GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + switch (provider) { + case GGML_CUDA_ALLREDUCE_INTERNAL: + result = ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); + break; + case GGML_CUDA_ALLREDUCE_NCCL: + result = ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); + break; + default: + GGML_ASSERT(false); + } + + if (result == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) { + return true; + } + if (result == GGML_CUDA_COMM_ALLREDUCE_FAILED) { + return false; + } + } + + return false; + }; + + switch (comm_ctx->preferred_provider) { case GGML_CUDA_ALLREDUCE_INTERNAL: - return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors); + case GGML_CUDA_ALLREDUCE_NCCL: + return try_in_order(comm_ctx->preferred_provider); default: return false; } From 43743e5ade26c239190b89ee2976331e1c5d49ec Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 16:32:14 -0700 Subject: [PATCH 23/81] add messages for allreduce fallback --- ggml/src/ggml-cuda/ggml-cuda.cu | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index a016e881a48..3894130a090 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1351,6 +1351,7 @@ enum ggml_cuda_comm_allreduce_result { static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { if (comm_ctx->ar_pipeline == nullptr) { + GGML_LOG_WARN("%s: internal unsupported: pipeline unavailable\n", __func__); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } @@ -1362,10 +1363,12 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte const ggml_type type = tensors[0]->type; if (n_backends != 2) { + GGML_LOG_WARN("%s: internal unsupported: n_backends=%zu\n", __func__, n_backends); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { + GGML_LOG_WARN("%s: internal unsupported: type=%d\n", __func__, (int) type); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } @@ -1375,16 +1378,30 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte const size_t bytes = (size_t) ne * ggml_type_size(type); if (bytes > GGML_CUDA_AR_MAX_BYTES) { + GGML_LOG_WARN("%s: internal unsupported: ne=%" PRId64 " type=%d bytes=%zu max=%zu\n", + __func__, ne, (int) type, bytes, GGML_CUDA_AR_MAX_BYTES); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } for (size_t i = 0; i < n_backends; ++i) { if (tensors[i] == nullptr) { + GGML_LOG_ERROR("%s: internal failed: tensor[%zu] is null\n", __func__, i); return GGML_CUDA_COMM_ALLREDUCE_FAILED; } if (ggml_nelements(tensors[i]) != ne || tensors[i]->type != type) { + GGML_LOG_ERROR("%s: internal failed: tensor[%zu] ne=%" PRId64 " type=%d expected ne=%" PRId64 " type=%d\n", + __func__, i, ggml_nelements(tensors[i]), (int) tensors[i]->type, ne, (int) type); return GGML_CUDA_COMM_ALLREDUCE_FAILED; } + if (!ggml_is_contiguously_allocated(tensors[i])) { + GGML_LOG_WARN("%s: internal tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", + __func__, i, ne, ggml_nbytes(tensors[i]), + (size_t) ne * ggml_type_size(type) / ggml_blck_size(type), (int) type); + } + if (((uintptr_t) tensors[i]->data & 0xF) != 0) { + GGML_LOG_WARN("%s: internal tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", + __func__, i, tensors[i]->data, (int) type, ne); + } } return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors) From 2573b7b3790890272cdc06401c6b139f6208b443 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 17:37:24 -0700 Subject: [PATCH 24/81] rework reduction provider init to not call ncclCommInitAll if using the internal provider --- ggml/src/ggml-cuda/ggml-cuda.cu | 43 +++++++++++++++++---------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 3894130a090..2dca6b3db40 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1230,34 +1230,35 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba ret->preferred_provider = provider; ret->backends.assign(backends, backends + n_backends); - switch (provider) { - case GGML_CUDA_ALLREDUCE_NCCL: { + GGML_ASSERT(provider == GGML_CUDA_ALLREDUCE_INTERNAL || provider == GGML_CUDA_ALLREDUCE_NCCL); + + if (provider == GGML_CUDA_ALLREDUCE_INTERNAL) { + ret->ar_pipeline = ggml_cuda_ar_pipeline_init( + dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); + if (ret->ar_pipeline != nullptr) { + return ret; + } + + GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); #ifdef GGML_USE_NCCL - ret->comms.resize(n_backends); - NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); + ret->preferred_provider = GGML_CUDA_ALLREDUCE_NCCL; #else - // Unreachable: ggml_cuda_select_allreduce_provider() only returns - // GGML_CUDA_ALLREDUCE_NCCL when GGML_USE_NCCL is defined. - GGML_ABORT("NCCL provider selected but NCCL not compiled in"); + delete ret; + return nullptr; #endif - } break; - - case GGML_CUDA_ALLREDUCE_INTERNAL: { - ret->ar_pipeline = ggml_cuda_ar_pipeline_init( - dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); - if (ret->ar_pipeline == nullptr) { - GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); - delete ret; - return nullptr; - } + } + + if (ret->preferred_provider == GGML_CUDA_ALLREDUCE_NCCL) { #ifdef GGML_USE_NCCL - ret->comms.resize(n_backends); - NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); + ret->comms.resize(n_backends); + NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); + return ret; +#else + GGML_ABORT("NCCL provider selected but NCCL not compiled in"); #endif - } break; } - return ret; + GGML_ABORT("unexpected AllReduce provider"); } #ifdef GGML_USE_NCCL From 4d7736c7619e7c98908400af98a9bb38d380f73d Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 19:15:08 -0700 Subject: [PATCH 25/81] fix case where a given tensor has not been computed --- ggml/src/ggml-cuda/allreduce.cu | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index a9dad4c13ab..2aba8d03b67 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -548,10 +548,17 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_set_device(p->devices[i]); auto * cuda_ctx = static_cast(backends[i]->context); ggml_cuda_ar_event_slot & ev = p->ev_pool[i][slot]; + const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); + // Match the NCCL and meta-backend semantics: inactive shards + // contribute zeros to the reduction. + if (!compute) { + CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, bytes, p->streams[i])); + } + #if GGML_CUDA_AR_WATCHDOG #define GGML_CUDA_AR_WDOG_EXTRA_ARGS , p->debug_ring[i], p->wdog_max_spin, i, slot #else From f046004c816792c925aaf4cde345044722941c5d Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 20:19:11 -0700 Subject: [PATCH 26/81] add chunked mode to the kernel for unlimited vector size --- ggml/src/ggml-cuda/allreduce.cu | 128 +++++++++++++++++-------------- ggml/src/ggml-cuda/allreduce.cuh | 5 +- ggml/src/ggml-cuda/ggml-cuda.cu | 7 -- 3 files changed, 72 insertions(+), 68 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 2aba8d03b67..c1df8d8d2ce 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -505,59 +505,69 @@ bool ggml_cuda_ar_allreduce( return false; } - const int64_t ne = ggml_nelements(tensors[0]); - const size_t bytes = (size_t)ne * type_size; + const int64_t ne = ggml_nelements(tensors[0]); if (ne == 0) { return true; } - if (bytes > p->buf_bytes) { + if (p->buf_bytes < type_size) { return false; } - // Cycle through the event pool. On the second pass through the ring, - // synchronise on the slot's ker event before touching arrival ints — - // the event and arrival pools wrap in lock-step so this guarantees the - // kernels which last used this slot have finished. - const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); - const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; - p->call_count++; + const size_t max_chunk_elems = p->buf_bytes / type_size; + GGML_ASSERT(max_chunk_elems > 0); - if (pool_lapped) { - for (int i = 0; i < n; ++i) { - ggml_cuda_set_device(p->devices[i]); - CUDA_CHECK(cudaEventSynchronize(p->ev_pool[i][slot].ker)); + // Insert chunked kernels into each GPU's existing compute stream via events: + // record(app, compute_stream) — capture "upstream done" + // wait(internal_stream, app) — internal stream defers until then + // launch one or more chunk kernels on internal_stream + // record(ker, internal_stream) — capture "final chunk done" + // wait(compute_stream, ker) — compute stream resumes after reduce + for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems) { + const size_t remaining_elems = (size_t) (ne - chunk_start); + const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; + const size_t chunk_bytes = chunk_elems * type_size; + + // Cycle through the event pool. On the second pass through the ring, + // synchronise on the slot's ker event before touching arrival ints — + // the event and arrival pools wrap in lock-step so this guarantees the + // kernels which last used this slot have finished. + const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); + const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; + p->call_count++; + + if (pool_lapped) { + for (int i = 0; i < n; ++i) { + ggml_cuda_set_device(p->devices[i]); + CUDA_CHECK(cudaEventSynchronize(p->ev_pool[i][slot].ker)); + } } - } - // Reset the arrival ints for this slot before any kernel can read them. - for (int i = 0; i < n; ++i) { - *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; - } + // Reset the arrival ints for this slot before any kernel can read them. + for (int i = 0; i < n; ++i) { + *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; + } + for (int i = 0; i < n; ++i) { + const int peer = 1 - i; // valid for n == 2 only + ggml_cuda_set_device(p->devices[i]); + auto * cuda_ctx = static_cast(backends[i]->context); + ggml_cuda_ar_event_slot & ev = p->ev_pool[i][slot]; + const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; - // Insert the kernel into each GPU's existing compute stream via events: - // record(app, compute_stream) — capture "upstream done" - // wait(internal_stream, app) — internal stream defers until then - // launch kernel on internal_stream - // record(ker, internal_stream) — capture "kernel done" - // wait(compute_stream, ker) — compute stream resumes after kernel - for (int i = 0; i < n; ++i) { - const int peer = 1 - i; // valid for n == 2 only - ggml_cuda_set_device(p->devices[i]); - auto * cuda_ctx = static_cast(backends[i]->context); - ggml_cuda_ar_event_slot & ev = p->ev_pool[i][slot]; - const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; + if (chunk_start == 0) { + CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); + } - CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); - CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); + char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) type_size; - // Match the NCCL and meta-backend semantics: inactive shards - // contribute zeros to the reduction. - if (!compute) { - CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, bytes, p->streams[i])); - } + // Match the NCCL and meta-backend semantics: inactive shards + // contribute zeros to the reduction. + if (!compute) { + CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_bytes, p->streams[i])); + } #if GGML_CUDA_AR_WATCHDOG #define GGML_CUDA_AR_WDOG_EXTRA_ARGS , p->debug_ring[i], p->wdog_max_spin, i, slot @@ -566,30 +576,32 @@ bool ggml_cuda_ar_allreduce( #endif #define LAUNCH_AR_KERNEL(T) \ - ggml_cuda_ar_kernel<<streams[i]>>>( \ - static_cast(tensors[i]->data), \ - static_cast(tensors[i]->data), \ - reinterpret_cast(p->host_buf[i]), \ - reinterpret_cast(p->host_buf[peer]), \ - static_cast(ne), \ - ggml_cuda_ar_arrival_ptr(p, slot, i), \ - ggml_cuda_ar_arrival_ptr(p, slot, peer) \ - GGML_CUDA_AR_WDOG_EXTRA_ARGS) - - switch (type) { - case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; - case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; - case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; - default: GGML_ASSERT(false); - } + ggml_cuda_ar_kernel<<streams[i]>>>( \ + reinterpret_cast(data), \ + reinterpret_cast(data), \ + reinterpret_cast(p->host_buf[i]), \ + reinterpret_cast(p->host_buf[peer]), \ + static_cast(chunk_elems), \ + ggml_cuda_ar_arrival_ptr(p, slot, i), \ + ggml_cuda_ar_arrival_ptr(p, slot, peer) \ + GGML_CUDA_AR_WDOG_EXTRA_ARGS) + + switch (type) { + case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; + case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; + case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; + default: GGML_ASSERT(false); + } #undef LAUNCH_AR_KERNEL #undef GGML_CUDA_AR_WDOG_EXTRA_ARGS - CUDA_CHECK(cudaGetLastError()); - - CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); + CUDA_CHECK(cudaGetLastError()); + CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); + if (chunk_start + (int64_t) chunk_elems == ne) { + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); + } + } } return true; diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index 77466011a04..4c2be439625 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -5,9 +5,8 @@ #include -// Maximum tensor size (bytes per GPU) handled by the internal kernel path. -// Tensors larger than this are not yet supported and ggml_cuda_ar_allreduce() -// returns false, allowing the caller to fall back to another provider. +// Maximum chunk size (bytes per GPU) handled by one internal kernel launch. +// Larger tensors are reduced by issuing multiple chunked launches. static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB // Opaque pipeline context — owns all pinned buffers, streams, and events. diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 2dca6b3db40..9a4dd062aa3 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1377,13 +1377,6 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte return GGML_CUDA_COMM_ALLREDUCE_SUCCESS; } - const size_t bytes = (size_t) ne * ggml_type_size(type); - if (bytes > GGML_CUDA_AR_MAX_BYTES) { - GGML_LOG_WARN("%s: internal unsupported: ne=%" PRId64 " type=%d bytes=%zu max=%zu\n", - __func__, ne, (int) type, bytes, GGML_CUDA_AR_MAX_BYTES); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; - } - for (size_t i = 0; i < n_backends; ++i) { if (tensors[i] == nullptr) { GGML_LOG_ERROR("%s: internal failed: tensor[%zu] is null\n", __func__, i); From fbcae511bddfc5973ea936c44dd4d86337232624 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 21:07:13 -0700 Subject: [PATCH 27/81] rework a few checks/fallbacks --- ggml/src/ggml-cuda/allreduce.cu | 7 +++++-- ggml/src/ggml-cuda/allreduce.cuh | 2 +- ggml/src/ggml-cuda/ggml-cuda.cu | 8 +++++--- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index c1df8d8d2ce..270a3d0e7f7 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -322,8 +322,11 @@ static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { // --------------------------------------------------------------------------- ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( - const int * devices, int n_devices, size_t max_bytes) { - GGML_ASSERT(n_devices >= 2 && n_devices <= GGML_CUDA_MAX_DEVICES); + const int * devices, size_t n_devices, size_t max_bytes) { + + if ((n_devices != 2) || (n_devices > GGML_CUDA_MAX_DEVICES)) { + return nullptr; + } auto * p = new ggml_cuda_ar_pipeline{}; p->n_devices = n_devices; diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index 4c2be439625..a30a4f8c287 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -18,7 +18,7 @@ struct ggml_cuda_ar_pipeline; // as the largest tensor that will be reduced. // Returns nullptr on allocation failure. ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( - const int * devices, int n_devices, size_t max_bytes); + const int * devices, size_t n_devices, size_t max_bytes); // Release all resources owned by the pipeline. void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 9a4dd062aa3..498679daa67 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1234,7 +1234,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba if (provider == GGML_CUDA_ALLREDUCE_INTERNAL) { ret->ar_pipeline = ggml_cuda_ar_pipeline_init( - dev_ids.data(), static_cast(n_backends), GGML_CUDA_AR_MAX_BYTES); + dev_ids.data(), n_backends, GGML_CUDA_AR_MAX_BYTES); if (ret->ar_pipeline != nullptr) { return ret; } @@ -1388,13 +1388,15 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte return GGML_CUDA_COMM_ALLREDUCE_FAILED; } if (!ggml_is_contiguously_allocated(tensors[i])) { - GGML_LOG_WARN("%s: internal tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", + GGML_LOG_WARN("%s: internal unsupported: tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", __func__, i, ne, ggml_nbytes(tensors[i]), (size_t) ne * ggml_type_size(type) / ggml_blck_size(type), (int) type); + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } if (((uintptr_t) tensors[i]->data & 0xF) != 0) { - GGML_LOG_WARN("%s: internal tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", + GGML_LOG_WARN("%s: internal unsupported: tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", __func__, i, tensors[i]->data, (int) type, ne); + return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } } From 7449c1d30a00d1ffdb00e2239a27c3d952a39c1e Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 23 Apr 2026 21:24:11 -0700 Subject: [PATCH 28/81] various small cleanups --- ggml/src/ggml-cuda/allreduce.cu | 5 +++-- ggml/src/ggml-cuda/ggml-cuda.cu | 3 +++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 270a3d0e7f7..4a72d91899e 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -350,12 +350,14 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( for (int i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); - if (cudaStreamCreateWithFlags(&p->streams[i], cudaStreamNonBlocking) != cudaSuccess) { + cudaStream_t stream = nullptr; + if (cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaStreamCreateWithFlags failed for device %d\n", __func__, p->devices[i]); ggml_cuda_ar_pipeline_free(p); return nullptr; } + p->streams[i] = stream; p->ev_pool[i] = new ggml_cuda_ar_event_slot[GGML_CUDA_AR_POOL_SIZE](); for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { @@ -490,7 +492,6 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, ggml_tensor ** tensors) { - //printf("ggml_cuda_ar_allreduce\n"); GGML_ASSERT(p != nullptr); const int n = p->n_devices; diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index d779c41adca..e545ca1ccae 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1241,6 +1241,9 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); #ifdef GGML_USE_NCCL + // Clear any sticky CUDA error left over from the failed pipeline init + // so NCCL's own error-check on entry doesn't observe it. + (void) cudaGetLastError(); ret->preferred_provider = GGML_CUDA_ALLREDUCE_NCCL; #else delete ret; From 2b91e21d1461879e6474e93d9668dfd549e8478f Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Fri, 24 Apr 2026 12:58:19 -0700 Subject: [PATCH 29/81] allow disabling CUDA reductions completely (falling back to the non-CUDA butterfly mode) --- ggml/src/ggml-cuda/ggml-cuda.cu | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index e545ca1ccae..f9d1d55ceab 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1218,6 +1218,16 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba } } + // GGML_CUDA_ALLREDUCE=none disables the CUDA-specific AllReduce entirely, + // so the meta-backend falls back to its generic butterfly reduction. + { + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + if (env != nullptr && strcmp(env, "none") == 0) { + GGML_LOG_INFO("%s: GGML_CUDA_ALLREDUCE=none; using meta-backend butterfly reduction\n", __func__); + return nullptr; + } + } + std::vector dev_ids; dev_ids.reserve(n_backends); for (size_t i = 0; i < n_backends; i++) { From 485ee232892cedf73f1f25bfebb4717c612b7878 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Fri, 24 Apr 2026 13:55:00 -0700 Subject: [PATCH 30/81] simplify reduction provider selection --- ggml/src/ggml-cuda/comm.cuh | 22 --- ggml/src/ggml-cuda/ggml-cuda.cu | 232 ++++++++++++++------------------ 2 files changed, 101 insertions(+), 153 deletions(-) delete mode 100644 ggml/src/ggml-cuda/comm.cuh diff --git a/ggml/src/ggml-cuda/comm.cuh b/ggml/src/ggml-cuda/comm.cuh deleted file mode 100644 index c26d1e1dd76..00000000000 --- a/ggml/src/ggml-cuda/comm.cuh +++ /dev/null @@ -1,22 +0,0 @@ -#pragma once - -// AllReduce provider for multi-GPU tensor parallelism. -// -// The meta backend splits each transformer layer's PARTIAL-axis subgraph across -// N GPUs and requires an AllReduce after each segment to sum the partial results. -// This enum selects which implementation performs that reduction. -// -// The preferred provider is chosen once at communicator init time by -// ggml_cuda_select_allreduce_provider() and stored in -// ggml_backend_cuda_comm_context::preferred_provider. -enum ggml_cuda_allreduce_provider { - // NVIDIA/AMD Collective Communications Library (NCCL/RCCL). - // Optimal on NVLink/NVSwitch topologies; auto-selects the best transport. - // Requires GGML_USE_NCCL at compile time. - GGML_CUDA_ALLREDUCE_NCCL = 0, - - // Internal host/CUDA staged reduction built into llama.cpp. - // Works on any interconnect (PCIe, NVLink) without an external library. - // Can outperform NCCL on PCIe-only systems for latency-sensitive tensor sizes. - GGML_CUDA_ALLREDUCE_INTERNAL = 1, -}; diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index f9d1d55ceab..6b6ca822a9a 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -3,7 +3,6 @@ #include "ggml-backend-impl.h" #include "ggml-cuda/allreduce.cuh" -#include "ggml-cuda/comm.cuh" #include "ggml-cuda/common.cuh" #include "ggml-cuda/acc.cuh" #include "ggml-cuda/add-id.cuh" @@ -1141,69 +1140,36 @@ static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_inte }; // Communication context for multi-GPU AllReduce during tensor parallelism. -// Created once per meta backend instance; the preferred provider is chosen at -// init time, but per-call dispatch may fall back to another CUDA provider when -// the preferred one does not support a tensor configuration. +// +// Created once per meta backend instance. The internal pipeline (if any) is +// allocated eagerly at init time; NCCL communicators are created lazily on +// first use so NCCL's init/runtime quirks don't interfere with configurations +// that never hit the fallback path. struct ggml_backend_cuda_comm_context { - ggml_cuda_allreduce_provider preferred_provider; - std::vector backends; + std::vector backends; + std::vector dev_ids; + + ggml_cuda_ar_pipeline * ar_pipeline = nullptr; + // NCCL is eligible when GGML_USE_NCCL is defined and the user did not + // force GGML_CUDA_ALLREDUCE=internal. Comms are initialised on first use. + bool nccl_eligible = false; #ifdef GGML_USE_NCCL - std::vector comms; + std::once_flag nccl_init_flag; + bool nccl_init_ok = false; + std::vector comms; #endif - ggml_cuda_ar_pipeline * ar_pipeline = nullptr; - ~ggml_backend_cuda_comm_context() { #ifdef GGML_USE_NCCL - if (!comms.empty()) { - for (ncclComm_t comm : comms) { - NCCL_CHECK(ncclCommDestroy(comm)); - } + for (ncclComm_t comm : comms) { + NCCL_CHECK(ncclCommDestroy(comm)); } #endif ggml_cuda_ar_pipeline_free(ar_pipeline); } }; -// Select an AllReduce provider for the given set of CUDA device IDs. -// -// Priority: -// 1. GGML_CUDA_ALLREDUCE env var ("nccl" or "internal") — explicit override. -// 2. Internal for 2 GPUs (the optimised path). -// 3. NCCL as fallback for >2 GPUs when compiled in (GGML_USE_NCCL defined). -// 4. Internal otherwise. -static ggml_cuda_allreduce_provider ggml_cuda_select_allreduce_provider( - const std::vector & device_ids) { - const char * env = getenv("GGML_CUDA_ALLREDUCE"); - if (env != nullptr && env[0] != '\0') { - if (strcmp(env, "internal") == 0) { - return GGML_CUDA_ALLREDUCE_INTERNAL; - } - if (strcmp(env, "nccl") == 0) { -#ifdef GGML_USE_NCCL - return GGML_CUDA_ALLREDUCE_NCCL; -#else - GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in, using internal provider\n", __func__); - return GGML_CUDA_ALLREDUCE_INTERNAL; -#endif - } - GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); - } - - // Internal provider is the default for 2-GPU configurations. - if (device_ids.size() <= 2) { - return GGML_CUDA_ALLREDUCE_INTERNAL; - } - - // >2 GPUs: fall back to NCCL if available, otherwise internal. -#ifdef GGML_USE_NCCL - return GGML_CUDA_ALLREDUCE_NCCL; -#else - return GGML_CUDA_ALLREDUCE_INTERNAL; -#endif -} - static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { if (comm_ctx_v == nullptr) { return; @@ -1211,6 +1177,16 @@ static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { delete static_cast(comm_ctx_v); } +// Create the comm context. +// +// GGML_CUDA_ALLREDUCE selects which provider(s) to enable: +// unset — try internal first, fall back to NCCL if compiled in, +// then to the meta-backend butterfly reduction. +// internal — internal only; on unsupported/failure, fall back to butterfly. +// nccl — NCCL only; on unsupported/failure, fall back to butterfly. +// none — skip the CUDA AllReduce entirely; always use butterfly. +// Returns nullptr when no CUDA provider is available, which causes the meta +// backend to run its generic butterfly reduction. static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { for (size_t i = 0; i < n_backends; i++) { if (!ggml_backend_is_cuda(backends[i])) { @@ -1218,60 +1194,60 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba } } - // GGML_CUDA_ALLREDUCE=none disables the CUDA-specific AllReduce entirely, - // so the meta-backend falls back to its generic butterfly reduction. - { - const char * env = getenv("GGML_CUDA_ALLREDUCE"); - if (env != nullptr && strcmp(env, "none") == 0) { - GGML_LOG_INFO("%s: GGML_CUDA_ALLREDUCE=none; using meta-backend butterfly reduction\n", __func__); - return nullptr; - } + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + const bool force_none = env && strcmp(env, "none") == 0; + const bool force_internal = env && strcmp(env, "internal") == 0; + const bool force_nccl = env && strcmp(env, "nccl") == 0; + if (env && *env && !force_none && !force_internal && !force_nccl) { + GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); } - std::vector dev_ids; - dev_ids.reserve(n_backends); - for (size_t i = 0; i < n_backends; i++) { - dev_ids.push_back(static_cast(backends[i]->context)->device); + if (force_none) { + GGML_LOG_INFO("%s: GGML_CUDA_ALLREDUCE=none; using meta-backend butterfly reduction\n", __func__); + return nullptr; } - const ggml_cuda_allreduce_provider provider = ggml_cuda_select_allreduce_provider(dev_ids); +#ifndef GGML_USE_NCCL + if (force_nccl) { + GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in; using meta-backend butterfly reduction\n", __func__); + return nullptr; + } +#endif auto * ret = new ggml_backend_cuda_comm_context; - ret->preferred_provider = provider; ret->backends.assign(backends, backends + n_backends); + ret->dev_ids.reserve(n_backends); + for (size_t i = 0; i < n_backends; i++) { + ret->dev_ids.push_back(static_cast(backends[i]->context)->device); + } - GGML_ASSERT(provider == GGML_CUDA_ALLREDUCE_INTERNAL || provider == GGML_CUDA_ALLREDUCE_NCCL); - - if (provider == GGML_CUDA_ALLREDUCE_INTERNAL) { + // Try to allocate the internal pipeline unless the user forced NCCL. + if (!force_nccl) { ret->ar_pipeline = ggml_cuda_ar_pipeline_init( - dev_ids.data(), n_backends, GGML_CUDA_AR_MAX_BYTES); - if (ret->ar_pipeline != nullptr) { - return ret; + ret->dev_ids.data(), n_backends, GGML_CUDA_AR_MAX_BYTES); + if (ret->ar_pipeline == nullptr) { + // Clear any sticky CUDA error from the failed init so it can't + // leak into a later NCCL call. + (void) cudaGetLastError(); + if (force_internal) { + GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed; falling back to butterfly\n", __func__); + } } - - GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed\n", __func__); -#ifdef GGML_USE_NCCL - // Clear any sticky CUDA error left over from the failed pipeline init - // so NCCL's own error-check on entry doesn't observe it. - (void) cudaGetLastError(); - ret->preferred_provider = GGML_CUDA_ALLREDUCE_NCCL; -#else - delete ret; - return nullptr; -#endif } - if (ret->preferred_provider == GGML_CUDA_ALLREDUCE_NCCL) { #ifdef GGML_USE_NCCL - ret->comms.resize(n_backends); - NCCL_CHECK(ncclCommInitAll(ret->comms.data(), (int) n_backends, dev_ids.data())); - return ret; + ret->nccl_eligible = !force_internal; #else - GGML_ABORT("NCCL provider selected but NCCL not compiled in"); + ret->nccl_eligible = false; #endif + + // If nothing is usable, return nullptr so the meta backend uses butterfly. + if (ret->ar_pipeline == nullptr && !ret->nccl_eligible) { + delete ret; + return nullptr; } - GGML_ABORT("unexpected AllReduce provider"); + return ret; } #ifdef GGML_USE_NCCL @@ -1419,67 +1395,61 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte } #ifdef GGML_USE_NCCL +// Lazily initialise NCCL communicators on first use. +// Returns true when comms are ready; false if init failed (dispatcher should skip NCCL). +static bool ggml_backend_cuda_comm_ensure_nccl(ggml_backend_cuda_comm_context * comm_ctx) { + std::call_once(comm_ctx->nccl_init_flag, [&] { + const size_t n = comm_ctx->dev_ids.size(); + comm_ctx->comms.resize(n); + ncclResult_t rc = ncclCommInitAll(comm_ctx->comms.data(), (int) n, comm_ctx->dev_ids.data()); + if (rc != ncclSuccess) { + GGML_LOG_ERROR("%s: ncclCommInitAll failed: %s\n", __func__, ncclGetErrorString(rc)); + comm_ctx->comms.clear(); + return; + } + comm_ctx->nccl_init_ok = true; + }); + return comm_ctx->nccl_init_ok; +} + static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_nccl( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (comm_ctx->comms.empty()) { + if (!ggml_backend_cuda_comm_ensure_nccl(comm_ctx)) { return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors) ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS : GGML_CUDA_COMM_ALLREDUCE_FAILED; } -#else -static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_nccl( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - GGML_UNUSED_VARS(comm_ctx, tensors); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; -} #endif +// Dispatch order is fixed: internal first (if allocated), then NCCL (if +// eligible, lazily initialised on first call). If neither handles the tensor, +// return false so the meta backend runs its butterfly reduction. static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { if (comm_ctx_v == nullptr) { return false; } auto * comm_ctx = static_cast(comm_ctx_v); - auto try_in_order = [&](ggml_cuda_allreduce_provider first) -> bool { - const ggml_cuda_allreduce_provider second = - first == GGML_CUDA_ALLREDUCE_INTERNAL - ? GGML_CUDA_ALLREDUCE_NCCL - : GGML_CUDA_ALLREDUCE_INTERNAL; - const ggml_cuda_allreduce_provider order[2] = { first, second }; - - for (ggml_cuda_allreduce_provider provider : order) { - ggml_cuda_comm_allreduce_result result = GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; - switch (provider) { - case GGML_CUDA_ALLREDUCE_INTERNAL: - result = ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); - break; - case GGML_CUDA_ALLREDUCE_NCCL: - result = ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); - break; - default: - GGML_ASSERT(false); - } - - if (result == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) { - return true; - } - if (result == GGML_CUDA_COMM_ALLREDUCE_FAILED) { - return false; - } - } - - return false; - }; + if (comm_ctx->ar_pipeline != nullptr) { + const ggml_cuda_comm_allreduce_result r = + ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); + if (r == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) return true; + if (r == GGML_CUDA_COMM_ALLREDUCE_FAILED) return false; + // UNSUPPORTED — fall through to NCCL (if eligible). + } - switch (comm_ctx->preferred_provider) { - case GGML_CUDA_ALLREDUCE_INTERNAL: - case GGML_CUDA_ALLREDUCE_NCCL: - return try_in_order(comm_ctx->preferred_provider); - default: - return false; +#ifdef GGML_USE_NCCL + if (comm_ctx->nccl_eligible) { + const ggml_cuda_comm_allreduce_result r = + ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); + if (r == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) return true; + if (r == GGML_CUDA_COMM_ALLREDUCE_FAILED) return false; } +#endif + + return false; } ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { From 892b2e388d61e48d9b35b56f312653151b9c0176 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Fri, 24 Apr 2026 14:38:06 -0700 Subject: [PATCH 31/81] minor simplifications --- ggml/src/ggml-cuda/allreduce.cu | 1 - ggml/src/ggml-cuda/allreduce.cuh | 5 +++-- ggml/src/ggml-cuda/ggml-cuda.cu | 7 +------ 3 files changed, 4 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 4a72d91899e..f5f3c53dd10 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -520,7 +520,6 @@ bool ggml_cuda_ar_allreduce( } const size_t max_chunk_elems = p->buf_bytes / type_size; - GGML_ASSERT(max_chunk_elems > 0); // Insert chunked kernels into each GPU's existing compute stream via events: // record(app, compute_stream) — capture "upstream done" diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index a30a4f8c287..fbf11521792 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -14,8 +14,9 @@ struct ggml_cuda_ar_pipeline; // Allocate a pipeline for n_devices GPUs. // devices[] holds the CUDA device IDs in rank order. -// max_bytes is the staging buffer size per device; must be at least as large -// as the largest tensor that will be reduced. +// max_bytes is the staging buffer size per device, which also bounds the +// per-launch chunk size; tensors larger than this are reduced via multiple +// chunked launches. // Returns nullptr on allocation failure. ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( const int * devices, size_t n_devices, size_t max_bytes); diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 6b6ca822a9a..9bec84551b2 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1327,11 +1327,6 @@ static bool ggml_backend_cuda_comm_allreduce_nccl( } #endif // GGML_USE_NCCL -static bool ggml_backend_cuda_comm_allreduce_internal( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); -} - enum ggml_cuda_comm_allreduce_result { GGML_CUDA_COMM_ALLREDUCE_SUCCESS, GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED, @@ -1389,7 +1384,7 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte } } - return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors) + return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors) ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS : GGML_CUDA_COMM_ALLREDUCE_FAILED; } From b34adf4805fcc9434bea02b35749de3675daedcd Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Fri, 24 Apr 2026 15:41:18 -0700 Subject: [PATCH 32/81] more cleanups/fixes --- .gitattributes | 3 - ggml/src/ggml-cuda/allreduce.cu | 183 +++++++++++++++++-------------- ggml/src/ggml-cuda/allreduce.cuh | 14 +-- ggml/src/ggml-cuda/ggml-cuda.cu | 19 ++-- 4 files changed, 110 insertions(+), 109 deletions(-) diff --git a/.gitattributes b/.gitattributes index e9f9b494a72..06c85ad56e8 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1,6 +1,3 @@ -# Force LF line endings everywhere (prevent Windows CRLF conversion). -* text=auto eol=lf - # Treat the generated single-file WebUI build as binary for diff purposes. # Git's pack-file delta compression still works (byte-level), but this prevents # git diff from printing the entire minified file on every change. diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index f5f3c53dd10..43999b5d569 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -233,6 +233,10 @@ static __global__ void ggml_cuda_ar_kernel( // in-flight depth (single digits in practice) while keeping init cost low. static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; +// Maximum chunk size (bytes per GPU) handled by one internal kernel launch. +// Larger tensors are reduced by issuing multiple chunked launches. +static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB + // Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) // ensures the arrival slots for the two GPUs never share a cache line, // preventing false-sharing stalls on the polling GPU. @@ -279,6 +283,41 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, return reinterpret_cast(p->arrival + offset); } +static int ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { + const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); + const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; + p->call_count++; + + if (pool_lapped) { + for (int i = 0; i < p->n_devices; ++i) { + ggml_cuda_set_device(p->devices[i]); + CUDA_CHECK(cudaEventSynchronize(p->ev_pool[i][slot].ker)); + } + } + + for (int i = 0; i < p->n_devices; ++i) { + *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; + } + + return slot; +} + +static void ggml_cuda_ar_wait_for_compute( + ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot) { + ggml_cuda_ar_event_slot & ev = p->ev_pool[rank][slot]; + CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); + CUDA_CHECK(cudaStreamWaitEvent(p->streams[rank], ev.app)); +} + +static void ggml_cuda_ar_record_chunk_done( + ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot, bool last_chunk) { + ggml_cuda_ar_event_slot & ev = p->ev_pool[rank][slot]; + CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[rank])); + if (last_chunk) { + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); + } +} + // --------------------------------------------------------------------------- // Background watchdog thread — monitors per-GPU debug ring buffers for new // bailout records. The kernel writes a record when it hits the spin limit; @@ -315,14 +354,51 @@ static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); } } + +static bool ggml_cuda_ar_wdog_init(ggml_cuda_ar_pipeline * p) { + for (int i = 0; i < p->n_devices; ++i) { + if (cudaHostAlloc(reinterpret_cast(&p->debug_ring[i]), + sizeof(ggml_cuda_ar_debug_ring), + cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for debug ring failed on device %d\n", + __func__, p->devices[i]); + return false; + } + memset(p->debug_ring[i], 0, sizeof(ggml_cuda_ar_debug_ring)); + } + + const char * spin_env = getenv("GGML_CUDA_AR_MAX_SPIN"); + p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; + GGML_LOG_INFO("%s: AR watchdog enabled — max_spin=%d " + "(set GGML_CUDA_AR_MAX_SPIN= to adjust)\n", + __func__, p->wdog_max_spin); + + p->wdog_stop.store(false); + p->wdog_thr = std::thread(ggml_cuda_ar_wdog_thread, p); + return true; +} + +static void ggml_cuda_ar_wdog_stop(ggml_cuda_ar_pipeline * p) { + p->wdog_stop.store(true); + if (p->wdog_thr.joinable()) { + p->wdog_thr.join(); + } +} + +static void ggml_cuda_ar_wdog_free(ggml_cuda_ar_pipeline * p) { + for (int i = 0; i < p->n_devices; ++i) { + if (p->debug_ring[i]) { + cudaFreeHost(p->debug_ring[i]); + } + } +} #endif // GGML_CUDA_AR_WATCHDOG // --------------------------------------------------------------------------- // Init / free // --------------------------------------------------------------------------- -ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( - const int * devices, size_t n_devices, size_t max_bytes) { +ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n_devices) { if ((n_devices != 2) || (n_devices > GGML_CUDA_MAX_DEVICES)) { return nullptr; @@ -386,49 +462,28 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( memset(p->arrival, 0, arrival_bytes); // Per-device pinned staging buffers. - p->buf_bytes = max_bytes; + p->buf_bytes = GGML_CUDA_AR_MAX_BYTES; for (int i = 0; i < n_devices; ++i) { - if (cudaHostAlloc(&p->host_buf[i], max_bytes, cudaHostAllocPortable) != cudaSuccess) { + if (cudaHostAlloc(&p->host_buf[i], p->buf_bytes, cudaHostAllocPortable) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for staging failed (%zu bytes)\n", - __func__, max_bytes); + __func__, p->buf_bytes); ggml_cuda_ar_pipeline_free(p); return nullptr; } - memset(p->host_buf[i], 0, max_bytes); + memset(p->host_buf[i], 0, p->buf_bytes); } #if GGML_CUDA_AR_WATCHDOG - // Per-GPU debug ring buffers: written by the kernel on spin-limit bailout, - // polled by the background watchdog thread. Each ring is pinned host - // memory accessed only by its owning GPU (single-GPU host atomics OK). - { - for (int i = 0; i < n_devices; ++i) { - if (cudaHostAlloc(reinterpret_cast(&p->debug_ring[i]), - sizeof(ggml_cuda_ar_debug_ring), - cudaHostAllocPortable) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for debug ring failed on device %d\n", - __func__, p->devices[i]); - ggml_cuda_ar_pipeline_free(p); - return nullptr; - } - memset(p->debug_ring[i], 0, sizeof(ggml_cuda_ar_debug_ring)); - } - - const char * spin_env = getenv("GGML_CUDA_AR_MAX_SPIN"); - p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; - GGML_LOG_INFO("%s: AR watchdog enabled — max_spin=%d " - "(set GGML_CUDA_AR_MAX_SPIN= to adjust)\n", - __func__, p->wdog_max_spin); - - p->wdog_stop.store(false); - p->wdog_thr = std::thread(ggml_cuda_ar_wdog_thread, p); + if (!ggml_cuda_ar_wdog_init(p)) { + ggml_cuda_ar_pipeline_free(p); + return nullptr; } #endif GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", - __func__, n_devices, max_bytes >> 10); - + __func__, n_devices, p->buf_bytes >> 10); + return p; } @@ -440,10 +495,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { #if GGML_CUDA_AR_WATCHDOG // Stop the watchdog thread first — it only reads pinned host memory, // no GPU resources, so this is safe and returns within ~1ms. - p->wdog_stop.store(true); - if (p->wdog_thr.joinable()) { - p->wdog_thr.join(); - } + ggml_cuda_ar_wdog_stop(p); #endif // Drain all in-flight kernels before tearing down resources. @@ -475,11 +527,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaFreeHost(p->arrival); } #if GGML_CUDA_AR_WATCHDOG - for (int i = 0; i < p->n_devices; ++i) { - if (p->debug_ring[i]) { - cudaFreeHost(p->debug_ring[i]); - } - } + ggml_cuda_ar_wdog_free(p); #endif delete p; } @@ -495,31 +543,18 @@ bool ggml_cuda_ar_allreduce( GGML_ASSERT(p != nullptr); const int n = p->n_devices; - - // Only the 2-GPU path is implemented; fall back for larger communicators. - if (n != 2) { - return false; - } + GGML_ASSERT(n == 2); const ggml_type type = tensors[0]->type; const size_t type_size = ggml_type_size(type); - - // Only float, half, and bfloat16 tensors are handled by the kernel. - if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { - return false; - } + GGML_ASSERT(type == GGML_TYPE_F32 || type == GGML_TYPE_F16 || type == GGML_TYPE_BF16); const int64_t ne = ggml_nelements(tensors[0]); - - if (ne == 0) { - return true; - } - - if (p->buf_bytes < type_size) { - return false; - } + GGML_ASSERT(ne > 0); + GGML_ASSERT(p->buf_bytes >= type_size); const size_t max_chunk_elems = p->buf_bytes / type_size; + GGML_ASSERT(max_chunk_elems > 0); // Insert chunked kernels into each GPU's existing compute stream via events: // record(app, compute_stream) — capture "upstream done" @@ -532,36 +567,17 @@ bool ggml_cuda_ar_allreduce( const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; const size_t chunk_bytes = chunk_elems * type_size; - // Cycle through the event pool. On the second pass through the ring, - // synchronise on the slot's ker event before touching arrival ints — - // the event and arrival pools wrap in lock-step so this guarantees the - // kernels which last used this slot have finished. - const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); - const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; - p->call_count++; - - if (pool_lapped) { - for (int i = 0; i < n; ++i) { - ggml_cuda_set_device(p->devices[i]); - CUDA_CHECK(cudaEventSynchronize(p->ev_pool[i][slot].ker)); - } - } - - // Reset the arrival ints for this slot before any kernel can read them. - for (int i = 0; i < n; ++i) { - *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; - } + const int slot = ggml_cuda_ar_acquire_slot(p); + const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; for (int i = 0; i < n; ++i) { const int peer = 1 - i; // valid for n == 2 only ggml_cuda_set_device(p->devices[i]); auto * cuda_ctx = static_cast(backends[i]->context); - ggml_cuda_ar_event_slot & ev = p->ev_pool[i][slot]; const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; if (chunk_start == 0) { - CUDA_CHECK(cudaEventRecord(ev.app, cuda_ctx->stream())); - CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], ev.app)); + ggml_cuda_ar_wait_for_compute(p, cuda_ctx, i, slot); } char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) type_size; @@ -600,10 +616,7 @@ bool ggml_cuda_ar_allreduce( #undef GGML_CUDA_AR_WDOG_EXTRA_ARGS CUDA_CHECK(cudaGetLastError()); - CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[i])); - if (chunk_start + (int64_t) chunk_elems == ne) { - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); - } + ggml_cuda_ar_record_chunk_done(p, cuda_ctx, i, slot, last_chunk); } } diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index fbf11521792..1f7f9f41ba8 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -5,21 +5,14 @@ #include -// Maximum chunk size (bytes per GPU) handled by one internal kernel launch. -// Larger tensors are reduced by issuing multiple chunked launches. -static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB - // Opaque pipeline context — owns all pinned buffers, streams, and events. struct ggml_cuda_ar_pipeline; // Allocate a pipeline for n_devices GPUs. // devices[] holds the CUDA device IDs in rank order. -// max_bytes is the staging buffer size per device, which also bounds the -// per-launch chunk size; tensors larger than this are reduced via multiple -// chunked launches. // Returns nullptr on allocation failure. ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init( - const int * devices, size_t n_devices, size_t max_bytes); + const int * devices, size_t n_devices); // Release all resources owned by the pipeline. void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); @@ -27,9 +20,8 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * pipeline); // Execute an in-place AllReduce (sum) across tensors[0..n_devices-1]. // tensors[i] must live on the device managed by backends[i] and be // contiguous F32, F16, or BF16. -// Returns true on success. Returns false when the tensor type or size is -// outside the currently supported range; the caller should fall back to -// another provider (NCCL or the meta-backend CPU reduce). +// Preconditions are checked by the CUDA comm dispatcher before calling this. +// Returns true once the reduction work has been enqueued successfully. bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * pipeline, ggml_backend_t * backends, diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 9bec84551b2..93308bcfbe0 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1223,8 +1223,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba // Try to allocate the internal pipeline unless the user forced NCCL. if (!force_nccl) { - ret->ar_pipeline = ggml_cuda_ar_pipeline_init( - ret->dev_ids.data(), n_backends, GGML_CUDA_AR_MAX_BYTES); + ret->ar_pipeline = ggml_cuda_ar_pipeline_init(ret->dev_ids.data(), n_backends); if (ret->ar_pipeline == nullptr) { // Clear any sticky CUDA error from the failed init so it can't // leak into a later NCCL call. @@ -1336,7 +1335,7 @@ enum ggml_cuda_comm_allreduce_result { static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { if (comm_ctx->ar_pipeline == nullptr) { - GGML_LOG_WARN("%s: internal unsupported: pipeline unavailable\n", __func__); + GGML_LOG_DEBUG("%s: internal unsupported: pipeline unavailable\n", __func__); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } @@ -1348,12 +1347,12 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte const ggml_type type = tensors[0]->type; if (n_backends != 2) { - GGML_LOG_WARN("%s: internal unsupported: n_backends=%zu\n", __func__, n_backends); + GGML_LOG_DEBUG("%s: internal unsupported: n_backends=%zu\n", __func__, n_backends); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { - GGML_LOG_WARN("%s: internal unsupported: type=%d\n", __func__, (int) type); + GGML_LOG_DEBUG("%s: internal unsupported: type=%d\n", __func__, (int) type); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } @@ -1372,14 +1371,14 @@ static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_inte return GGML_CUDA_COMM_ALLREDUCE_FAILED; } if (!ggml_is_contiguously_allocated(tensors[i])) { - GGML_LOG_WARN("%s: internal unsupported: tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", - __func__, i, ne, ggml_nbytes(tensors[i]), - (size_t) ne * ggml_type_size(type) / ggml_blck_size(type), (int) type); + GGML_LOG_DEBUG("%s: internal unsupported: tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", + __func__, i, ne, ggml_nbytes(tensors[i]), + (size_t) ne * ggml_type_size(type) / ggml_blck_size(type), (int) type); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } if (((uintptr_t) tensors[i]->data & 0xF) != 0) { - GGML_LOG_WARN("%s: internal unsupported: tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", - __func__, i, tensors[i]->data, (int) type, ne); + GGML_LOG_DEBUG("%s: internal unsupported: tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", + __func__, i, tensors[i]->data, (int) type, ne); return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; } } From b8caad377110ccf6f5debc14c0d91305b5542840 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Sat, 25 Apr 2026 01:04:16 -0700 Subject: [PATCH 33/81] prototype alternate path for large reductions --- ggml/src/ggml-cuda/allreduce.cu | 274 +++++++++++++++++++++++++++++++- 1 file changed, 268 insertions(+), 6 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 43999b5d569..8e21fdf1ff0 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -1,8 +1,11 @@ #include "allreduce.cuh" #include "ggml-impl.h" +#include +#include #include #include +#include // Set to 1 to enable the AllReduce spin-limit watchdog (development only). // When enabled, the debug kernel bails out after GGML_CUDA_AR_MAX_SPIN @@ -225,6 +228,17 @@ static __global__ void ggml_cuda_ar_kernel( } } +static __global__ void ggml_cuda_ar_add_f32_kernel( + float * __restrict__ dst, + const float * __restrict__ src, + int count) { + const int tid = blockIdx.x * blockDim.x + threadIdx.x; + const int nt = gridDim.x * blockDim.x; + for (int i = tid; i < count; i += nt) { + dst[i] += src[i]; + } +} + // --------------------------------------------------------------------------- // Pipeline structure // --------------------------------------------------------------------------- @@ -235,7 +249,11 @@ static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; // Maximum chunk size (bytes per GPU) handled by one internal kernel launch. // Larger tensors are reduced by issuing multiple chunked launches. -static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 256 * 1024; // 256 KB +static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB + +// Prototype copy-engine path for large F32 reductions. +static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB +static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB // Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) // ensures the arrival slots for the two GPUs never share a cache line, @@ -249,6 +267,7 @@ static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 1; struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete + cudaEvent_t cpy = nullptr; // copy-engine D2H complete cudaEvent_t ker = nullptr; // AllReduce kernel complete }; @@ -256,10 +275,15 @@ struct ggml_cuda_ar_pipeline { int n_devices; int devices[GGML_CUDA_MAX_DEVICES]; size_t buf_bytes; // bytes per device in host_buf[] + size_t copy_bytes; // bytes per device in host_large[] / dev_tmp[] + size_t copy_threshold; uint64_t call_count; + uint64_t reduce_count; // Per-device resources. char * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging + char * host_large[GGML_CUDA_MAX_DEVICES]; // pinned staging for copy-engine path + char * dev_tmp[GGML_CUDA_MAX_DEVICES]; // device scratch for copy-engine path cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking ggml_cuda_ar_event_slot *ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] @@ -267,6 +291,13 @@ struct ggml_cuda_ar_pipeline { // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; + // Temporary host-side tracing for prefill AllReduce analysis. + bool trace_enabled; + bool trace_chunks; + uint64_t trace_limit; + uint64_t trace_chunk_limit; + uint64_t trace_chunk_count; + #if GGML_CUDA_AR_WATCHDOG // Per-GPU debug ring buffers in pinned host memory. Written by the debug // kernel on spin-limit bailout, read by the background watchdog thread. @@ -283,6 +314,105 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, return reinterpret_cast(p->arrival + offset); } +static bool ggml_cuda_ar_env_enabled(const char * name) { + const char * value = getenv(name); + return value != nullptr && value[0] != '\0' && strcmp(value, "0") != 0 && + strcmp(value, "false") != 0 && strcmp(value, "FALSE") != 0; +} + +static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) { + const char * value = getenv(name); + if (value == nullptr || value[0] == '\0') { + return default_value; + } + + char * end = nullptr; + const unsigned long long parsed = strtoull(value, &end, 10); + return end != value ? (uint64_t) parsed : default_value; +} + +static void ggml_cuda_ar_trace_call( + const ggml_cuda_ar_pipeline * p, + uint64_t reduce_id, + const char * path, + ggml_tensor ** tensors, + ggml_type type, + int64_t ne, + size_t nbytes, + size_t max_chunk_elems, + size_t chunks) { + if (!p->trace_enabled || reduce_id >= p->trace_limit) { + return; + } + + fprintf(stdout, + "GGML_CUDA_AR_TRACE call=%" PRIu64 + " path=%s name=\"%s\" type=%s ne=%" PRId64 " nbytes=%zu chunks=%zu" + " max_chunk_elems=%zu max_chunk_bytes=%zu" + " flags=[0x%x,0x%x] compute=[%d,%d]" + " data=[%p,%p]" + " ne0=[%" PRId64 ",%" PRId64 "] ne1=[%" PRId64 ",%" PRId64 "]" + " ne2=[%" PRId64 ",%" PRId64 "] ne3=[%" PRId64 ",%" PRId64 "]" + " nb0=[%zu,%zu] nb1=[%zu,%zu] nb2=[%zu,%zu] nb3=[%zu,%zu]\n", + reduce_id, + path, + tensors[0]->name, + ggml_type_name(type), + ne, + nbytes, + chunks, + max_chunk_elems, + max_chunk_elems * ggml_type_size(type), + tensors[0]->flags, + tensors[1]->flags, + (tensors[0]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0, + (tensors[1]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0, + tensors[0]->data, + tensors[1]->data, + tensors[0]->ne[0], tensors[1]->ne[0], + tensors[0]->ne[1], tensors[1]->ne[1], + tensors[0]->ne[2], tensors[1]->ne[2], + tensors[0]->ne[3], tensors[1]->ne[3], + (size_t) tensors[0]->nb[0], (size_t) tensors[1]->nb[0], + (size_t) tensors[0]->nb[1], (size_t) tensors[1]->nb[1], + (size_t) tensors[0]->nb[2], (size_t) tensors[1]->nb[2], + (size_t) tensors[0]->nb[3], (size_t) tensors[1]->nb[3]); + fflush(stdout); +} + +static void ggml_cuda_ar_trace_chunk( + ggml_cuda_ar_pipeline * p, + uint64_t reduce_id, + const char * path, + size_t chunk_index, + int slot, + int64_t chunk_start, + size_t chunk_elems, + size_t chunk_bytes, + bool last_chunk) { + if (!p->trace_enabled || !p->trace_chunks || + reduce_id >= p->trace_limit || + p->trace_chunk_count >= p->trace_chunk_limit) { + return; + } + + p->trace_chunk_count++; + fprintf(stdout, + "GGML_CUDA_AR_TRACE_CHUNK call=%" PRIu64 + " path=%s chunk=%zu slot=%d start=%" PRId64 + " elems=%zu bytes=%zu launches=%d last=%d\n", + reduce_id, + path, + chunk_index, + slot, + chunk_start, + chunk_elems, + chunk_bytes, + p->n_devices, + last_chunk); + fflush(stdout); +} + static int ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; @@ -405,13 +535,24 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } auto * p = new ggml_cuda_ar_pipeline{}; - p->n_devices = n_devices; - p->buf_bytes = 0; - p->call_count = 0; - p->arrival = nullptr; + p->n_devices = n_devices; + p->buf_bytes = 0; + p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; + p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); + p->call_count = 0; + p->reduce_count = 0; + p->arrival = nullptr; + p->trace_enabled = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE", 0) != 0 && + !ggml_cuda_ar_env_enabled("GGML_CUDA_AR_TRACE_DISABLE"); + p->trace_chunks = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_CHUNKS", 1) != 0; + p->trace_limit = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_LIMIT", 2048); + p->trace_chunk_limit = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_CHUNK_LIMIT", 8192); + p->trace_chunk_count = 0; for (int i = 0; i < n_devices; ++i) { p->devices[i] = devices[i]; p->host_buf[i] = nullptr; + p->host_large[i] = nullptr; + p->dev_tmp[i] = nullptr; p->streams[i] = nullptr; p->ev_pool[i] = nullptr; } @@ -439,6 +580,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { const bool ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].app, cudaEventDisableTiming) == cudaSuccess && + cudaEventCreateWithFlags(&p->ev_pool[i][s].cpy, cudaEventDisableTiming) == cudaSuccess && cudaEventCreateWithFlags(&p->ev_pool[i][s].ker, cudaEventDisableTiming) == cudaSuccess; if (!ok) { GGML_LOG_ERROR("%s: cudaEventCreate failed for device %d slot %d\n", @@ -473,6 +615,24 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n memset(p->host_buf[i], 0, p->buf_bytes); } + // Prototype copy-engine path resources. Keep these deliberately large for + // now; memory footprint can be reduced after the bandwidth experiment. + for (int i = 0; i < n_devices; ++i) { + ggml_cuda_set_device(p->devices[i]); + if (cudaHostAlloc(&p->host_large[i], p->copy_bytes, cudaHostAllocPortable) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostAlloc for large staging failed (%zu bytes)\n", + __func__, p->copy_bytes); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + if (cudaMalloc(reinterpret_cast(&p->dev_tmp[i]), p->copy_bytes) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaMalloc for copy scratch failed (%zu bytes) on device %d\n", + __func__, p->copy_bytes, p->devices[i]); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + } + #if GGML_CUDA_AR_WATCHDOG if (!ggml_cuda_ar_wdog_init(p)) { ggml_cuda_ar_pipeline_free(p); @@ -483,6 +643,21 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", __func__, n_devices, p->buf_bytes >> 10); + if (p->trace_enabled) { + fprintf(stdout, + "GGML_CUDA_AR_TRACE_INIT devices=%d staging_bytes=%zu pool=%d chunks=%d" + " copy_bytes=%zu copy_threshold=%zu" + " trace_limit=%" PRIu64 " chunk_limit=%" PRIu64 "\n", + p->n_devices, + p->buf_bytes, + GGML_CUDA_AR_POOL_SIZE, + p->trace_chunks, + p->copy_bytes, + p->copy_threshold, + p->trace_limit, + p->trace_chunk_limit); + fflush(stdout); + } return p; } @@ -510,10 +685,18 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { if (p->host_buf[i]) { cudaFreeHost(p->host_buf[i]); } + if (p->host_large[i]) { + cudaFreeHost(p->host_large[i]); + } + if (p->dev_tmp[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaFree(p->dev_tmp[i]); + } if (p->ev_pool[i]) { ggml_cuda_set_device(p->devices[i]); for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { if (p->ev_pool[i][s].app) { cudaEventDestroy(p->ev_pool[i][s].app); } + if (p->ev_pool[i][s].cpy) { cudaEventDestroy(p->ev_pool[i][s].cpy); } if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } } delete[] p->ev_pool[i]; @@ -536,6 +719,66 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- +static bool ggml_cuda_ar_allreduce_copy_f32( + ggml_cuda_ar_pipeline * p, + ggml_backend_t * backends, + ggml_tensor ** tensors, + uint64_t reduce_id, + int64_t ne, + size_t nbytes) { + GGML_ASSERT(p->n_devices == 2); + GGML_ASSERT(nbytes <= p->copy_bytes); + GGML_ASSERT(ne <= std::numeric_limits::max()); + + const int slot = ggml_cuda_ar_acquire_slot(p); + ggml_cuda_ar_trace_chunk(p, reduce_id, "copy_engine", 0, slot, 0, (size_t) ne, nbytes, true); + + ggml_backend_cuda_context * cuda_ctx[2] = {}; + char * data[2] = {}; + + // Stage 1: both GPUs copy their local contribution to pinned host memory. + for (int i = 0; i < 2; ++i) { + ggml_cuda_set_device(p->devices[i]); + cuda_ctx[i] = static_cast(backends[i]->context); + data[i] = static_cast(tensors[i]->data); + + ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); + + const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; + if (!compute) { + CUDA_CHECK(cudaMemsetAsync(data[i], 0, nbytes, p->streams[i])); + } + + CUDA_CHECK(cudaMemcpyAsync(p->host_large[i], data[i], nbytes, cudaMemcpyDeviceToHost, p->streams[i])); + CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy, p->streams[i])); + } + + // Stage 2: each GPU waits for the peer D2H copy, pulls peer data back to + // local scratch, then performs a device-local add into the original tensor. + for (int i = 0; i < 2; ++i) { + const int peer = 1 - i; + ggml_cuda_set_device(p->devices[i]); + + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy)); + CUDA_CHECK(cudaMemcpyAsync(p->dev_tmp[i], p->host_large[peer], nbytes, cudaMemcpyHostToDevice, p->streams[i])); + + const int block_size = 256; + int n_blocks = (int) ((ne + block_size - 1) / block_size); + if (n_blocks > 1024) { + n_blocks = 1024; + } + ggml_cuda_ar_add_f32_kernel<<streams[i]>>>( + reinterpret_cast(data[i]), + reinterpret_cast(p->dev_tmp[i]), + (int) ne); + CUDA_CHECK(cudaGetLastError()); + + ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot, true); + } + + return true; +} + bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, @@ -556,19 +799,38 @@ bool ggml_cuda_ar_allreduce( const size_t max_chunk_elems = p->buf_bytes / type_size; GGML_ASSERT(max_chunk_elems > 0); + const uint64_t reduce_id = p->reduce_count++; + const size_t nbytes = ggml_nbytes(tensors[0]); + + const bool use_copy_engine = + type == GGML_TYPE_F32 && + p->copy_threshold > 0 && + nbytes >= p->copy_threshold && + nbytes <= p->copy_bytes; + if (use_copy_engine) { + const size_t copy_chunk_elems = p->copy_bytes / type_size; + ggml_cuda_ar_trace_call(p, reduce_id, "copy_engine", tensors, type, ne, nbytes, copy_chunk_elems, 1); + return ggml_cuda_ar_allreduce_copy_f32(p, backends, tensors, reduce_id, ne, nbytes); + } + + const size_t chunks = ((size_t) ne + max_chunk_elems - 1) / max_chunk_elems; + ggml_cuda_ar_trace_call(p, reduce_id, "kernel", tensors, type, ne, nbytes, max_chunk_elems, chunks); + // Insert chunked kernels into each GPU's existing compute stream via events: // record(app, compute_stream) — capture "upstream done" // wait(internal_stream, app) — internal stream defers until then // launch one or more chunk kernels on internal_stream // record(ker, internal_stream) — capture "final chunk done" // wait(compute_stream, ker) — compute stream resumes after reduce - for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems) { + size_t chunk_index = 0; + for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { const size_t remaining_elems = (size_t) (ne - chunk_start); const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; const size_t chunk_bytes = chunk_elems * type_size; const int slot = ggml_cuda_ar_acquire_slot(p); const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; + ggml_cuda_ar_trace_chunk(p, reduce_id, "kernel", chunk_index, slot, chunk_start, chunk_elems, chunk_bytes, last_chunk); for (int i = 0; i < n; ++i) { const int peer = 1 - i; // valid for n == 2 only From b0bdf83fd7999025cfbbd46c68c362bbb38a71ce Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Sat, 25 Apr 2026 01:54:35 -0700 Subject: [PATCH 34/81] chunked version of large reduction path --- ggml/src/ggml-cuda/allreduce.cu | 64 +++++++++++++++++++++++++-------- 1 file changed, 50 insertions(+), 14 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 8e21fdf1ff0..fd6b4f54ff1 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -254,6 +254,10 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB // Prototype copy-engine path for large F32 reductions. static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB +static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB +static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = + static_cast((GGML_CUDA_AR_COPY_MAX_BYTES + GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT - 1) / + GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); // Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) // ensures the arrival slots for the two GPUs never share a cache line, @@ -267,7 +271,7 @@ static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 1; struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete - cudaEvent_t cpy = nullptr; // copy-engine D2H complete + cudaEvent_t cpy[GGML_CUDA_AR_COPY_MAX_CHUNKS] = {}; // copy-engine D2H chunks complete cudaEvent_t ker = nullptr; // AllReduce kernel complete }; @@ -277,6 +281,7 @@ struct ggml_cuda_ar_pipeline { size_t buf_bytes; // bytes per device in host_buf[] size_t copy_bytes; // bytes per device in host_large[] / dev_tmp[] size_t copy_threshold; + size_t copy_chunk_bytes; uint64_t call_count; uint64_t reduce_count; @@ -539,6 +544,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->buf_bytes = 0; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); + p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT; p->call_count = 0; p->reduce_count = 0; p->arrival = nullptr; @@ -578,10 +584,12 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->ev_pool[i] = new ggml_cuda_ar_event_slot[GGML_CUDA_AR_POOL_SIZE](); for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { - const bool ok = + bool ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].app, cudaEventDisableTiming) == cudaSuccess && - cudaEventCreateWithFlags(&p->ev_pool[i][s].cpy, cudaEventDisableTiming) == cudaSuccess && cudaEventCreateWithFlags(&p->ev_pool[i][s].ker, cudaEventDisableTiming) == cudaSuccess; + for (int c = 0; ok && c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { + ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].cpy[c], cudaEventDisableTiming) == cudaSuccess; + } if (!ok) { GGML_LOG_ERROR("%s: cudaEventCreate failed for device %d slot %d\n", __func__, p->devices[i], s); @@ -646,7 +654,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n if (p->trace_enabled) { fprintf(stdout, "GGML_CUDA_AR_TRACE_INIT devices=%d staging_bytes=%zu pool=%d chunks=%d" - " copy_bytes=%zu copy_threshold=%zu" + " copy_bytes=%zu copy_threshold=%zu copy_chunk_bytes=%zu" " trace_limit=%" PRIu64 " chunk_limit=%" PRIu64 "\n", p->n_devices, p->buf_bytes, @@ -654,6 +662,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->trace_chunks, p->copy_bytes, p->copy_threshold, + p->copy_chunk_bytes, p->trace_limit, p->trace_chunk_limit); fflush(stdout); @@ -696,7 +705,9 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { ggml_cuda_set_device(p->devices[i]); for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { if (p->ev_pool[i][s].app) { cudaEventDestroy(p->ev_pool[i][s].app); } - if (p->ev_pool[i][s].cpy) { cudaEventDestroy(p->ev_pool[i][s].cpy); } + for (int c = 0; c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { + if (p->ev_pool[i][s].cpy[c]) { cudaEventDestroy(p->ev_pool[i][s].cpy[c]); } + } if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } } delete[] p->ev_pool[i]; @@ -729,9 +740,11 @@ static bool ggml_cuda_ar_allreduce_copy_f32( GGML_ASSERT(p->n_devices == 2); GGML_ASSERT(nbytes <= p->copy_bytes); GGML_ASSERT(ne <= std::numeric_limits::max()); + GGML_ASSERT(p->copy_chunk_bytes > 0); const int slot = ggml_cuda_ar_acquire_slot(p); - ggml_cuda_ar_trace_chunk(p, reduce_id, "copy_engine", 0, slot, 0, (size_t) ne, nbytes, true); + const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; + GGML_ASSERT(copy_chunks <= GGML_CUDA_AR_COPY_MAX_CHUNKS); ggml_backend_cuda_context * cuda_ctx[2] = {}; char * data[2] = {}; @@ -749,18 +762,40 @@ static bool ggml_cuda_ar_allreduce_copy_f32( CUDA_CHECK(cudaMemsetAsync(data[i], 0, nbytes, p->streams[i])); } - CUDA_CHECK(cudaMemcpyAsync(p->host_large[i], data[i], nbytes, cudaMemcpyDeviceToHost, p->streams[i])); - CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy, p->streams[i])); + for (size_t c = 0; c < copy_chunks; ++c) { + const size_t offset = c * p->copy_chunk_bytes; + const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? + (nbytes - offset) : p->copy_chunk_bytes; + + if (i == 0) { + ggml_cuda_ar_trace_chunk( + p, reduce_id, "copy_engine", c, slot, offset / sizeof(float), + chunk_bytes / sizeof(float), chunk_bytes, c + 1 == copy_chunks); + } + + CUDA_CHECK(cudaMemcpyAsync( + p->host_large[i] + offset, data[i] + offset, chunk_bytes, + cudaMemcpyDeviceToHost, p->streams[i])); + CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); + } } - // Stage 2: each GPU waits for the peer D2H copy, pulls peer data back to - // local scratch, then performs a device-local add into the original tensor. + // Stage 2: each GPU waits for each peer D2H chunk, pulls that chunk back to + // local scratch, then performs one device-local add over the assembled peer tensor. for (int i = 0; i < 2; ++i) { const int peer = 1 - i; ggml_cuda_set_device(p->devices[i]); - CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy)); - CUDA_CHECK(cudaMemcpyAsync(p->dev_tmp[i], p->host_large[peer], nbytes, cudaMemcpyHostToDevice, p->streams[i])); + for (size_t c = 0; c < copy_chunks; ++c) { + const size_t offset = c * p->copy_chunk_bytes; + const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? + (nbytes - offset) : p->copy_chunk_bytes; + + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy[c])); + CUDA_CHECK(cudaMemcpyAsync( + p->dev_tmp[i] + offset, p->host_large[peer] + offset, chunk_bytes, + cudaMemcpyHostToDevice, p->streams[i])); + } const int block_size = 256; int n_blocks = (int) ((ne + block_size - 1) / block_size); @@ -808,8 +843,9 @@ bool ggml_cuda_ar_allreduce( nbytes >= p->copy_threshold && nbytes <= p->copy_bytes; if (use_copy_engine) { - const size_t copy_chunk_elems = p->copy_bytes / type_size; - ggml_cuda_ar_trace_call(p, reduce_id, "copy_engine", tensors, type, ne, nbytes, copy_chunk_elems, 1); + const size_t copy_chunk_elems = p->copy_chunk_bytes / type_size; + const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; + ggml_cuda_ar_trace_call(p, reduce_id, "copy_engine", tensors, type, ne, nbytes, copy_chunk_elems, copy_chunks); return ggml_cuda_ar_allreduce_copy_f32(p, backends, tensors, reduce_id, ne, nbytes); } From 77b96e6f9aec005c29146b2e96b2bf300c16b837 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 16:52:43 -0700 Subject: [PATCH 35/81] use bf16 for large reductions --- ggml/src/ggml-cuda/allreduce.cu | 256 ++++++++++++++++++++++---------- 1 file changed, 174 insertions(+), 82 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index fd6b4f54ff1..7be4c1c5006 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -1,4 +1,5 @@ #include "allreduce.cuh" +#include "convert.cuh" #include "ggml-impl.h" #include @@ -228,14 +229,15 @@ static __global__ void ggml_cuda_ar_kernel( } } -static __global__ void ggml_cuda_ar_add_f32_kernel( - float * __restrict__ dst, - const float * __restrict__ src, +template +static __global__ void ggml_cuda_ar_add_kernel( + T * __restrict__ dst, + const T * __restrict__ src, int count) { const int tid = blockIdx.x * blockDim.x + threadIdx.x; const int nt = gridDim.x * blockDim.x; for (int i = tid; i < count; i += nt) { - dst[i] += src[i]; + dst[i] = dst[i] + src[i]; } } @@ -255,9 +257,12 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB +// Minimum chunk size the env-var override is allowed to set; this caps the +// per-slot copy-event array. 256 KB → up to 128 chunks per 32 MB tensor. +static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN = 256 * 1024; static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = - static_cast((GGML_CUDA_AR_COPY_MAX_BYTES + GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT - 1) / - GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); + static_cast((GGML_CUDA_AR_COPY_MAX_BYTES + GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN - 1) / + GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); // Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) // ensures the arrival slots for the two GPUs never share a cache line, @@ -282,6 +287,7 @@ struct ggml_cuda_ar_pipeline { size_t copy_bytes; // bytes per device in host_large[] / dev_tmp[] size_t copy_threshold; size_t copy_chunk_bytes; + size_t bf16_threshold; // tensors >= this size (bytes) are reduced via FP32->BF16 round-trip; 0 disables uint64_t call_count; uint64_t reduce_count; @@ -544,7 +550,13 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->buf_bytes = 0; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); - p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT; + p->copy_chunk_bytes = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_CHUNK_BYTES", GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); + if (p->copy_chunk_bytes < GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN) { + GGML_LOG_WARN("%s: GGML_CUDA_AR_COPY_CHUNK_BYTES=%zu below minimum %zu; clamping\n", + __func__, p->copy_chunk_bytes, GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); + p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; + } + p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default p->call_count = 0; p->reduce_count = 0; p->arrival = nullptr; @@ -730,13 +742,16 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- -static bool ggml_cuda_ar_allreduce_copy_f32( +template +static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, - ggml_tensor ** tensors, + T * const buf[GGML_CUDA_MAX_DEVICES], + const bool compute[GGML_CUDA_MAX_DEVICES], uint64_t reduce_id, int64_t ne, - size_t nbytes) { + size_t nbytes, + const char * trace_label) { GGML_ASSERT(p->n_devices == 2); GGML_ASSERT(nbytes <= p->copy_bytes); GGML_ASSERT(ne <= std::numeric_limits::max()); @@ -747,19 +762,16 @@ static bool ggml_cuda_ar_allreduce_copy_f32( GGML_ASSERT(copy_chunks <= GGML_CUDA_AR_COPY_MAX_CHUNKS); ggml_backend_cuda_context * cuda_ctx[2] = {}; - char * data[2] = {}; // Stage 1: both GPUs copy their local contribution to pinned host memory. for (int i = 0; i < 2; ++i) { ggml_cuda_set_device(p->devices[i]); cuda_ctx[i] = static_cast(backends[i]->context); - data[i] = static_cast(tensors[i]->data); ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); - const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; - if (!compute) { - CUDA_CHECK(cudaMemsetAsync(data[i], 0, nbytes, p->streams[i])); + if (!compute[i]) { + CUDA_CHECK(cudaMemsetAsync(buf[i], 0, nbytes, p->streams[i])); } for (size_t c = 0; c < copy_chunks; ++c) { @@ -769,12 +781,12 @@ static bool ggml_cuda_ar_allreduce_copy_f32( if (i == 0) { ggml_cuda_ar_trace_chunk( - p, reduce_id, "copy_engine", c, slot, offset / sizeof(float), - chunk_bytes / sizeof(float), chunk_bytes, c + 1 == copy_chunks); + p, reduce_id, trace_label, c, slot, offset / sizeof(T), + chunk_bytes / sizeof(T), chunk_bytes, c + 1 == copy_chunks); } CUDA_CHECK(cudaMemcpyAsync( - p->host_large[i] + offset, data[i] + offset, chunk_bytes, + p->host_large[i] + offset, reinterpret_cast(buf[i]) + offset, chunk_bytes, cudaMemcpyDeviceToHost, p->streams[i])); CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); } @@ -802,9 +814,9 @@ static bool ggml_cuda_ar_allreduce_copy_f32( if (n_blocks > 1024) { n_blocks = 1024; } - ggml_cuda_ar_add_f32_kernel<<streams[i]>>>( - reinterpret_cast(data[i]), - reinterpret_cast(p->dev_tmp[i]), + ggml_cuda_ar_add_kernel<<streams[i]>>>( + buf[i], + reinterpret_cast(p->dev_tmp[i]), (int) ne); CUDA_CHECK(cudaGetLastError()); @@ -823,68 +835,137 @@ bool ggml_cuda_ar_allreduce( const int n = p->n_devices; GGML_ASSERT(n == 2); - const ggml_type type = tensors[0]->type; - const size_t type_size = ggml_type_size(type); - GGML_ASSERT(type == GGML_TYPE_F32 || type == GGML_TYPE_F16 || type == GGML_TYPE_BF16); + const ggml_type input_type = tensors[0]->type; + GGML_ASSERT(input_type == GGML_TYPE_F32 || input_type == GGML_TYPE_F16 || input_type == GGML_TYPE_BF16); const int64_t ne = ggml_nelements(tensors[0]); GGML_ASSERT(ne > 0); + + const uint64_t reduce_id = p->reduce_count++; + const size_t input_nbytes = ggml_nbytes(tensors[0]); + + // BF16 round-trip: F32 inputs >= bf16_threshold are converted to BF16 for + // the reduction (chunked or copy-engine), halving on-wire bytes. Matches + // NCCL's behaviour. The pre-conversion zeroes inactive shards so the + // inner paths see them as already-prepared compute tensors. + const bool use_bf16 = + input_type == GGML_TYPE_F32 && + p->bf16_threshold > 0 && + input_nbytes >= p->bf16_threshold; + + const ggml_type kernel_type = use_bf16 ? GGML_TYPE_BF16 : input_type; + const size_t type_size = ggml_type_size(kernel_type); GGML_ASSERT(p->buf_bytes >= type_size); + const size_t nbytes = (size_t) ne * type_size; - const size_t max_chunk_elems = p->buf_bytes / type_size; - GGML_ASSERT(max_chunk_elems > 0); + bool compute_flag[GGML_CUDA_MAX_DEVICES] = {}; + for (int i = 0; i < n; ++i) { + compute_flag[i] = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; + } - const uint64_t reduce_id = p->reduce_count++; - const size_t nbytes = ggml_nbytes(tensors[0]); + ggml_cuda_pool_alloc bf16_tmp[GGML_CUDA_MAX_DEVICES]; + void * data_ptr[GGML_CUDA_MAX_DEVICES]; + + if (use_bf16) { + to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); + for (int i = 0; i < n; ++i) { + auto * cuda_ctx = static_cast(backends[i]->context); + bf16_tmp[i].pool = &cuda_ctx->pool(); + bf16_tmp[i].alloc(ne); + ggml_cuda_set_device(p->devices[i]); + if (compute_flag[i]) { + to_bf16(tensors[i]->data, bf16_tmp[i].get(), ne, cuda_ctx->stream()); + } else { + CUDA_CHECK(cudaMemsetAsync(bf16_tmp[i].get(), 0, nbytes, cuda_ctx->stream())); + } + CUDA_CHECK(cudaGetLastError()); + data_ptr[i] = bf16_tmp[i].get(); + } + } else { + for (int i = 0; i < n; ++i) { + data_ptr[i] = tensors[i]->data; + } + } + // Decide between copy-engine and chunked-kernel paths based on the working + // type's actual byte count. const bool use_copy_engine = - type == GGML_TYPE_F32 && p->copy_threshold > 0 && nbytes >= p->copy_threshold && nbytes <= p->copy_bytes; + + bool ok = true; if (use_copy_engine) { + const char * label = use_bf16 ? "copy_engine-bf16" : "copy_engine"; const size_t copy_chunk_elems = p->copy_chunk_bytes / type_size; const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; - ggml_cuda_ar_trace_call(p, reduce_id, "copy_engine", tensors, type, ne, nbytes, copy_chunk_elems, copy_chunks); - return ggml_cuda_ar_allreduce_copy_f32(p, backends, tensors, reduce_id, ne, nbytes); - } - - const size_t chunks = ((size_t) ne + max_chunk_elems - 1) / max_chunk_elems; - ggml_cuda_ar_trace_call(p, reduce_id, "kernel", tensors, type, ne, nbytes, max_chunk_elems, chunks); - - // Insert chunked kernels into each GPU's existing compute stream via events: - // record(app, compute_stream) — capture "upstream done" - // wait(internal_stream, app) — internal stream defers until then - // launch one or more chunk kernels on internal_stream - // record(ker, internal_stream) — capture "final chunk done" - // wait(compute_stream, ker) — compute stream resumes after reduce - size_t chunk_index = 0; - for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { - const size_t remaining_elems = (size_t) (ne - chunk_start); - const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; - const size_t chunk_bytes = chunk_elems * type_size; - - const int slot = ggml_cuda_ar_acquire_slot(p); - const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; - ggml_cuda_ar_trace_chunk(p, reduce_id, "kernel", chunk_index, slot, chunk_start, chunk_elems, chunk_bytes, last_chunk); + ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, copy_chunk_elems, copy_chunks); + // After up-front BF16 conversion, the tmp buffers already hold the + // (possibly zeroed-for-inactive) data, so the inner path can treat + // every shard as compute. + bool inner_compute[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) { - const int peer = 1 - i; // valid for n == 2 only - ggml_cuda_set_device(p->devices[i]); - auto * cuda_ctx = static_cast(backends[i]->context); - const bool compute = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; + inner_compute[i] = use_bf16 ? true : compute_flag[i]; + } - if (chunk_start == 0) { - ggml_cuda_ar_wait_for_compute(p, cuda_ctx, i, slot); + switch (kernel_type) { + case GGML_TYPE_F32: { + float * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); + ok = ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + break; + } + case GGML_TYPE_BF16: { + __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); + ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + break; } + default: + GGML_ASSERT(false); + } + } else { + const char * label = use_bf16 ? "kernel-bf16" : "kernel"; + const size_t max_chunk_elems = p->buf_bytes / type_size; + const size_t chunks = ((size_t) ne + max_chunk_elems - 1) / max_chunk_elems; + ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, max_chunk_elems, chunks); + + // Chunked-kernel path. Insert per-chunk kernels into each GPU's + // existing compute stream via events: + // record(app, compute_stream) — capture "upstream done" + // (incl. F32->BF16 conversion when applicable) + // wait(internal_stream, app) — internal stream defers until then + // launch one or more chunk kernels on internal_stream + // record(ker, internal_stream) — capture "final chunk done" + // wait(compute_stream, ker) — compute stream resumes + // (then runs BF16->F32 when applicable) + size_t chunk_index = 0; + for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { + const size_t remaining_elems = (size_t) (ne - chunk_start); + const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; + const size_t chunk_bytes = chunk_elems * type_size; + + const int slot = ggml_cuda_ar_acquire_slot(p); + const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; + ggml_cuda_ar_trace_chunk(p, reduce_id, label, chunk_index, slot, chunk_start, chunk_elems, chunk_bytes, last_chunk); + + for (int i = 0; i < n; ++i) { + const int peer = 1 - i; // valid for n == 2 only + ggml_cuda_set_device(p->devices[i]); + auto * cuda_ctx = static_cast(backends[i]->context); + + if (chunk_start == 0) { + ggml_cuda_ar_wait_for_compute(p, cuda_ctx, i, slot); + } - char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) type_size; + char * data = static_cast(data_ptr[i]) + chunk_start * (int64_t) type_size; - // Match the NCCL and meta-backend semantics: inactive shards - // contribute zeros to the reduction. - if (!compute) { - CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_bytes, p->streams[i])); - } + // Match NCCL/meta-backend semantics: inactive shards contribute + // zeros. On the BF16 path the tmp buffer was already zeroed. + if (!compute_flag[i] && !use_bf16) { + CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_bytes, p->streams[i])); + } #if GGML_CUDA_AR_WATCHDOG #define GGML_CUDA_AR_WDOG_EXTRA_ARGS , p->debug_ring[i], p->wdog_max_spin, i, slot @@ -893,30 +974,41 @@ bool ggml_cuda_ar_allreduce( #endif #define LAUNCH_AR_KERNEL(T) \ - ggml_cuda_ar_kernel<<streams[i]>>>( \ - reinterpret_cast(data), \ - reinterpret_cast(data), \ - reinterpret_cast(p->host_buf[i]), \ - reinterpret_cast(p->host_buf[peer]), \ - static_cast(chunk_elems), \ - ggml_cuda_ar_arrival_ptr(p, slot, i), \ - ggml_cuda_ar_arrival_ptr(p, slot, peer) \ - GGML_CUDA_AR_WDOG_EXTRA_ARGS) - - switch (type) { - case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; - case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; - case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; - default: GGML_ASSERT(false); - } + ggml_cuda_ar_kernel<<streams[i]>>>( \ + reinterpret_cast(data), \ + reinterpret_cast(data), \ + reinterpret_cast(p->host_buf[i]), \ + reinterpret_cast(p->host_buf[peer]), \ + static_cast(chunk_elems), \ + ggml_cuda_ar_arrival_ptr(p, slot, i), \ + ggml_cuda_ar_arrival_ptr(p, slot, peer) \ + GGML_CUDA_AR_WDOG_EXTRA_ARGS) + + switch (kernel_type) { + case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; + case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; + case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; + default: GGML_ASSERT(false); + } #undef LAUNCH_AR_KERNEL #undef GGML_CUDA_AR_WDOG_EXTRA_ARGS - CUDA_CHECK(cudaGetLastError()); + CUDA_CHECK(cudaGetLastError()); - ggml_cuda_ar_record_chunk_done(p, cuda_ctx, i, slot, last_chunk); + ggml_cuda_ar_record_chunk_done(p, cuda_ctx, i, slot, last_chunk); + } } } - return true; + if (use_bf16 && ok) { + to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); + for (int i = 0; i < n; ++i) { + auto * cuda_ctx = static_cast(backends[i]->context); + ggml_cuda_set_device(p->devices[i]); + to_fp32(bf16_tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); + CUDA_CHECK(cudaGetLastError()); + } + } + + return ok; } From 0805e6a82893481384abf4c621f2fbd0142769cf Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 17:37:20 -0700 Subject: [PATCH 36/81] experimental reduction using cudaMemcpyPeerAsync (slightly slower) --- ggml/src/ggml-cuda/allreduce.cu | 126 ++++++++++++++++++++++++++++++-- 1 file changed, 121 insertions(+), 5 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 7be4c1c5006..5a2667d3a44 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -288,6 +288,7 @@ struct ggml_cuda_ar_pipeline { size_t copy_threshold; size_t copy_chunk_bytes; size_t bf16_threshold; // tensors >= this size (bytes) are reduced via FP32->BF16 round-trip; 0 disables + bool use_peer; // for tensors in the copy-engine size range, use the cudaMemcpyPeerAsync path instead of D2H/H2D staging uint64_t call_count; uint64_t reduce_count; @@ -557,6 +558,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; } p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default + p->use_peer = ggml_cuda_ar_env_u64("GGML_CUDA_AR_PEER", 1) != 0; p->call_count = 0; p->reduce_count = 0; p->arrival = nullptr; @@ -742,6 +744,113 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- +// Asymmetric P2P AllReduce for two GPUs. GPU 0 issues both a D2H of its own +// data and a cudaMemcpyPeerAsync that pulls GPU 1's data directly to its +// device tmp; once both are done it records a single event and runs its +// reduction. GPU 1 waits on that event, does an H2D from the host buffer +// GPU 0 just filled, then runs its reduction. Five events total, no +// chunking — minimises the cross-device sync count that dominates the +// chunked copy path. +template +static bool ggml_cuda_ar_allreduce_peer_impl( + ggml_cuda_ar_pipeline * p, + ggml_backend_t * backends, + T * const buf[GGML_CUDA_MAX_DEVICES], + const bool compute[GGML_CUDA_MAX_DEVICES], + uint64_t reduce_id, + int64_t ne, + size_t nbytes, + const char * trace_label) { + GGML_ASSERT(p->n_devices == 2); + GGML_ASSERT(nbytes <= p->copy_bytes); + GGML_ASSERT(ne <= std::numeric_limits::max()); + + const int slot = ggml_cuda_ar_acquire_slot(p); + ggml_cuda_ar_trace_chunk(p, reduce_id, trace_label, 0, slot, 0, ne, nbytes, true); + + ggml_backend_cuda_context * cuda_ctx[2] = {}; + for (int i = 0; i < 2; ++i) { + cuda_ctx[i] = static_cast(backends[i]->context); + } + + // Inactive shards: zero on each device's compute stream so the wait below + // picks up the zero before any subsequent read of buf[i]. + for (int i = 0; i < 2; ++i) { + if (!compute[i]) { + ggml_cuda_set_device(p->devices[i]); + CUDA_CHECK(cudaMemsetAsync(buf[i], 0, nbytes, cuda_ctx[i]->stream())); + } + } + + // Each AR stream waits on its own compute stream (records ev.app per rank). + // Note: the cross-device wait on rank 1's compute is deferred to just + // before the PeerCopy below, so GPU 0's D2H can start as soon as its own + // compute is done — it doesn't depend on GPU 1. + for (int i = 0; i < 2; ++i) { + ggml_cuda_set_device(p->devices[i]); + ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); + } + + // GPU 0 stream: D2H(buf[0] -> host_large[0]) + // EventRecord(cpy[0]) — signals "host_large[0] is ready" + // WaitEvent(rank 1's compute) — only the PeerCopy needs this + // PeerCopy(buf[1] -> dev_tmp[0]) + // EventRecord(cpy[1]) — signals "buf[1] safe to overwrite" + // Reduction(buf[0] += dev_tmp[0]) + { + ggml_cuda_set_device(p->devices[0]); + + CUDA_CHECK(cudaMemcpyAsync(p->host_large[0], buf[0], nbytes, + cudaMemcpyDeviceToHost, p->streams[0])); + CUDA_CHECK(cudaEventRecord(p->ev_pool[0][slot].cpy[0], p->streams[0])); + + CUDA_CHECK(cudaStreamWaitEvent(p->streams[0], p->ev_pool[1][slot].app)); + CUDA_CHECK(cudaMemcpyPeerAsync(p->dev_tmp[0], p->devices[0], + buf[1], p->devices[1], + nbytes, p->streams[0])); + CUDA_CHECK(cudaEventRecord(p->ev_pool[0][slot].cpy[1], p->streams[0])); + + const int block_size = 256; + int n_blocks = (int) ((ne + block_size - 1) / block_size); + if (n_blocks > 1024) n_blocks = 1024; + ggml_cuda_ar_add_kernel<<streams[0]>>>( + buf[0], + reinterpret_cast(p->dev_tmp[0]), + (int) ne); + CUDA_CHECK(cudaGetLastError()); + } + + // GPU 1 stream: WaitEvent(cpy[0]) — D2H is done, host_large[0] ready + // H2D(host_large[0] -> dev_tmp[1]) — runs concurrently with GPU 0's PeerCopy + // WaitEvent(cpy[1]) — PeerCopy done, safe to write buf[1] + // Reduction(buf[1] += dev_tmp[1]) + { + ggml_cuda_set_device(p->devices[1]); + + CUDA_CHECK(cudaStreamWaitEvent(p->streams[1], p->ev_pool[0][slot].cpy[0])); + CUDA_CHECK(cudaMemcpyAsync(p->dev_tmp[1], p->host_large[0], nbytes, + cudaMemcpyHostToDevice, p->streams[1])); + CUDA_CHECK(cudaStreamWaitEvent(p->streams[1], p->ev_pool[0][slot].cpy[1])); + + const int block_size = 256; + int n_blocks = (int) ((ne + block_size - 1) / block_size); + if (n_blocks > 1024) n_blocks = 1024; + ggml_cuda_ar_add_kernel<<streams[1]>>>( + buf[1], + reinterpret_cast(p->dev_tmp[1]), + (int) ne); + CUDA_CHECK(cudaGetLastError()); + } + + // Per-device end events: AR stream → compute stream. + for (int i = 0; i < 2; ++i) { + ggml_cuda_set_device(p->devices[i]); + ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot, true); + } + + return true; +} + template static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_pipeline * p, @@ -896,9 +1005,12 @@ bool ggml_cuda_ar_allreduce( bool ok = true; if (use_copy_engine) { - const char * label = use_bf16 ? "copy_engine-bf16" : "copy_engine"; - const size_t copy_chunk_elems = p->copy_chunk_bytes / type_size; - const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; + const bool use_peer = p->use_peer; + const char * label = use_peer + ? (use_bf16 ? "peer-bf16" : "peer") + : (use_bf16 ? "copy_engine-bf16" : "copy_engine"); + const size_t copy_chunk_elems = use_peer ? (size_t) ne : (p->copy_chunk_bytes / type_size); + const size_t copy_chunks = use_peer ? 1 : ((nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes); ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, copy_chunk_elems, copy_chunks); // After up-front BF16 conversion, the tmp buffers already hold the @@ -913,13 +1025,17 @@ bool ggml_cuda_ar_allreduce( case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); - ok = ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + ok = use_peer + ? ggml_cuda_ar_allreduce_peer_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label) + : ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); break; } case GGML_TYPE_BF16: { __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); - ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + ok = use_peer + ? ggml_cuda_ar_allreduce_peer_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label) + : ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); break; } default: From 97eae7c5001ce5cf8a57d7bbbcdecb845780f1b0 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 17:49:24 -0700 Subject: [PATCH 37/81] revert experimental change --- ggml/src/ggml-cuda/allreduce.cu | 126 ++------------------------------ 1 file changed, 5 insertions(+), 121 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 5a2667d3a44..7be4c1c5006 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -288,7 +288,6 @@ struct ggml_cuda_ar_pipeline { size_t copy_threshold; size_t copy_chunk_bytes; size_t bf16_threshold; // tensors >= this size (bytes) are reduced via FP32->BF16 round-trip; 0 disables - bool use_peer; // for tensors in the copy-engine size range, use the cudaMemcpyPeerAsync path instead of D2H/H2D staging uint64_t call_count; uint64_t reduce_count; @@ -558,7 +557,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; } p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default - p->use_peer = ggml_cuda_ar_env_u64("GGML_CUDA_AR_PEER", 1) != 0; p->call_count = 0; p->reduce_count = 0; p->arrival = nullptr; @@ -744,113 +742,6 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- -// Asymmetric P2P AllReduce for two GPUs. GPU 0 issues both a D2H of its own -// data and a cudaMemcpyPeerAsync that pulls GPU 1's data directly to its -// device tmp; once both are done it records a single event and runs its -// reduction. GPU 1 waits on that event, does an H2D from the host buffer -// GPU 0 just filled, then runs its reduction. Five events total, no -// chunking — minimises the cross-device sync count that dominates the -// chunked copy path. -template -static bool ggml_cuda_ar_allreduce_peer_impl( - ggml_cuda_ar_pipeline * p, - ggml_backend_t * backends, - T * const buf[GGML_CUDA_MAX_DEVICES], - const bool compute[GGML_CUDA_MAX_DEVICES], - uint64_t reduce_id, - int64_t ne, - size_t nbytes, - const char * trace_label) { - GGML_ASSERT(p->n_devices == 2); - GGML_ASSERT(nbytes <= p->copy_bytes); - GGML_ASSERT(ne <= std::numeric_limits::max()); - - const int slot = ggml_cuda_ar_acquire_slot(p); - ggml_cuda_ar_trace_chunk(p, reduce_id, trace_label, 0, slot, 0, ne, nbytes, true); - - ggml_backend_cuda_context * cuda_ctx[2] = {}; - for (int i = 0; i < 2; ++i) { - cuda_ctx[i] = static_cast(backends[i]->context); - } - - // Inactive shards: zero on each device's compute stream so the wait below - // picks up the zero before any subsequent read of buf[i]. - for (int i = 0; i < 2; ++i) { - if (!compute[i]) { - ggml_cuda_set_device(p->devices[i]); - CUDA_CHECK(cudaMemsetAsync(buf[i], 0, nbytes, cuda_ctx[i]->stream())); - } - } - - // Each AR stream waits on its own compute stream (records ev.app per rank). - // Note: the cross-device wait on rank 1's compute is deferred to just - // before the PeerCopy below, so GPU 0's D2H can start as soon as its own - // compute is done — it doesn't depend on GPU 1. - for (int i = 0; i < 2; ++i) { - ggml_cuda_set_device(p->devices[i]); - ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); - } - - // GPU 0 stream: D2H(buf[0] -> host_large[0]) - // EventRecord(cpy[0]) — signals "host_large[0] is ready" - // WaitEvent(rank 1's compute) — only the PeerCopy needs this - // PeerCopy(buf[1] -> dev_tmp[0]) - // EventRecord(cpy[1]) — signals "buf[1] safe to overwrite" - // Reduction(buf[0] += dev_tmp[0]) - { - ggml_cuda_set_device(p->devices[0]); - - CUDA_CHECK(cudaMemcpyAsync(p->host_large[0], buf[0], nbytes, - cudaMemcpyDeviceToHost, p->streams[0])); - CUDA_CHECK(cudaEventRecord(p->ev_pool[0][slot].cpy[0], p->streams[0])); - - CUDA_CHECK(cudaStreamWaitEvent(p->streams[0], p->ev_pool[1][slot].app)); - CUDA_CHECK(cudaMemcpyPeerAsync(p->dev_tmp[0], p->devices[0], - buf[1], p->devices[1], - nbytes, p->streams[0])); - CUDA_CHECK(cudaEventRecord(p->ev_pool[0][slot].cpy[1], p->streams[0])); - - const int block_size = 256; - int n_blocks = (int) ((ne + block_size - 1) / block_size); - if (n_blocks > 1024) n_blocks = 1024; - ggml_cuda_ar_add_kernel<<streams[0]>>>( - buf[0], - reinterpret_cast(p->dev_tmp[0]), - (int) ne); - CUDA_CHECK(cudaGetLastError()); - } - - // GPU 1 stream: WaitEvent(cpy[0]) — D2H is done, host_large[0] ready - // H2D(host_large[0] -> dev_tmp[1]) — runs concurrently with GPU 0's PeerCopy - // WaitEvent(cpy[1]) — PeerCopy done, safe to write buf[1] - // Reduction(buf[1] += dev_tmp[1]) - { - ggml_cuda_set_device(p->devices[1]); - - CUDA_CHECK(cudaStreamWaitEvent(p->streams[1], p->ev_pool[0][slot].cpy[0])); - CUDA_CHECK(cudaMemcpyAsync(p->dev_tmp[1], p->host_large[0], nbytes, - cudaMemcpyHostToDevice, p->streams[1])); - CUDA_CHECK(cudaStreamWaitEvent(p->streams[1], p->ev_pool[0][slot].cpy[1])); - - const int block_size = 256; - int n_blocks = (int) ((ne + block_size - 1) / block_size); - if (n_blocks > 1024) n_blocks = 1024; - ggml_cuda_ar_add_kernel<<streams[1]>>>( - buf[1], - reinterpret_cast(p->dev_tmp[1]), - (int) ne); - CUDA_CHECK(cudaGetLastError()); - } - - // Per-device end events: AR stream → compute stream. - for (int i = 0; i < 2; ++i) { - ggml_cuda_set_device(p->devices[i]); - ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot, true); - } - - return true; -} - template static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_pipeline * p, @@ -1005,12 +896,9 @@ bool ggml_cuda_ar_allreduce( bool ok = true; if (use_copy_engine) { - const bool use_peer = p->use_peer; - const char * label = use_peer - ? (use_bf16 ? "peer-bf16" : "peer") - : (use_bf16 ? "copy_engine-bf16" : "copy_engine"); - const size_t copy_chunk_elems = use_peer ? (size_t) ne : (p->copy_chunk_bytes / type_size); - const size_t copy_chunks = use_peer ? 1 : ((nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes); + const char * label = use_bf16 ? "copy_engine-bf16" : "copy_engine"; + const size_t copy_chunk_elems = p->copy_chunk_bytes / type_size; + const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, copy_chunk_elems, copy_chunks); // After up-front BF16 conversion, the tmp buffers already hold the @@ -1025,17 +913,13 @@ bool ggml_cuda_ar_allreduce( case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); - ok = use_peer - ? ggml_cuda_ar_allreduce_peer_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label) - : ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + ok = ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); break; } case GGML_TYPE_BF16: { __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); - ok = use_peer - ? ggml_cuda_ar_allreduce_peer_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label) - : ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); + ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); break; } default: From 3e8b2639a9cd53330433bf459d51e3bdaadd7ded Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 18:41:30 -0700 Subject: [PATCH 38/81] add combined conversion/reduction kernel --- ggml/src/ggml-cuda/allreduce.cu | 96 +++++++++++++++++++++++---------- 1 file changed, 69 insertions(+), 27 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 7be4c1c5006..e192ee98b5c 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -229,15 +229,22 @@ static __global__ void ggml_cuda_ar_kernel( } } -template +// Combined load-convert-add kernel. The peer's contribution arrives as Tsrc +// (which may be a lower-precision type than Tdst when the BF16 round-trip is +// active). For bit-equivalence between the two GPUs, dst is first rounded +// through Tsrc's precision via a static_cast — peer already truncated its own +// value the same way before sending — so both sides perform identical +// arithmetic. When Tdst == Tsrc the round-trip cast is a no-op. +template static __global__ void ggml_cuda_ar_add_kernel( - T * __restrict__ dst, - const T * __restrict__ src, + Tdst * __restrict__ dst, + const Tsrc * __restrict__ src, int count) { const int tid = blockIdx.x * blockDim.x + threadIdx.x; const int nt = gridDim.x * blockDim.x; for (int i = tid; i < count; i += nt) { - dst[i] = dst[i] + src[i]; + const Tsrc d_low = static_cast(dst[i]); + dst[i] = static_cast(d_low) + static_cast(src[i]); } } @@ -742,11 +749,18 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- -template +// Asymmetric copy_impl: data sent over PCIe in Tsrc precision (one element of +// nbytes per ne element); accumulated locally into a Tdst buffer. When +// Tsrc == Tdst this is the original homogeneous reduction. When they differ +// (e.g. BF16 wire / F32 accumulator) the add kernel rounds dst through Tsrc +// for bit-equivalence between GPUs and we skip the otherwise-needed +// post-conversion entirely. +template static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, - T * const buf[GGML_CUDA_MAX_DEVICES], + Tsrc * const src_buf[GGML_CUDA_MAX_DEVICES], + Tdst * const dst_buf[GGML_CUDA_MAX_DEVICES], const bool compute[GGML_CUDA_MAX_DEVICES], uint64_t reduce_id, int64_t ne, @@ -771,7 +785,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); if (!compute[i]) { - CUDA_CHECK(cudaMemsetAsync(buf[i], 0, nbytes, p->streams[i])); + CUDA_CHECK(cudaMemsetAsync(src_buf[i], 0, nbytes, p->streams[i])); } for (size_t c = 0; c < copy_chunks; ++c) { @@ -781,12 +795,12 @@ static bool ggml_cuda_ar_allreduce_copy_impl( if (i == 0) { ggml_cuda_ar_trace_chunk( - p, reduce_id, trace_label, c, slot, offset / sizeof(T), - chunk_bytes / sizeof(T), chunk_bytes, c + 1 == copy_chunks); + p, reduce_id, trace_label, c, slot, offset / sizeof(Tsrc), + chunk_bytes / sizeof(Tsrc), chunk_bytes, c + 1 == copy_chunks); } CUDA_CHECK(cudaMemcpyAsync( - p->host_large[i] + offset, reinterpret_cast(buf[i]) + offset, chunk_bytes, + p->host_large[i] + offset, reinterpret_cast(src_buf[i]) + offset, chunk_bytes, cudaMemcpyDeviceToHost, p->streams[i])); CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); } @@ -814,9 +828,9 @@ static bool ggml_cuda_ar_allreduce_copy_impl( if (n_blocks > 1024) { n_blocks = 1024; } - ggml_cuda_ar_add_kernel<<streams[i]>>>( - buf[i], - reinterpret_cast(p->dev_tmp[i]), + ggml_cuda_ar_add_kernel<<streams[i]>>>( + dst_buf[i], + reinterpret_cast(p->dev_tmp[i]), (int) ne); CUDA_CHECK(cudaGetLastError()); @@ -877,6 +891,11 @@ bool ggml_cuda_ar_allreduce( to_bf16(tensors[i]->data, bf16_tmp[i].get(), ne, cuda_ctx->stream()); } else { CUDA_CHECK(cudaMemsetAsync(bf16_tmp[i].get(), 0, nbytes, cuda_ctx->stream())); + // The copy_engine path's combined add kernel reads/writes the + // F32 tensor data directly, so an inactive shard's accumulator + // must start at zero too. (The chunked-kernel path's + // post-conversion would also overwrite this with zero.) + CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, (size_t) ne * sizeof(float), cuda_ctx->stream())); } CUDA_CHECK(cudaGetLastError()); data_ptr[i] = bf16_tmp[i].get(); @@ -909,21 +928,40 @@ bool ggml_cuda_ar_allreduce( inner_compute[i] = use_bf16 ? true : compute_flag[i]; } - switch (kernel_type) { - case GGML_TYPE_F32: { - float * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); - ok = ggml_cuda_ar_allreduce_copy_impl(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); - break; + // Dispatch into copy_impl with explicit src/dst types. When use_bf16 + // is on, the wire type is BF16 (src = bf16_tmp) and the accumulator + // is F32 (dst = tensors[i]->data); the combined add kernel rounds dst + // through BF16 for bit-equivalence and writes F32 directly, so no + // post-conversion is needed. Otherwise src == dst (same native type). + if (use_bf16) { + GGML_ASSERT(kernel_type == GGML_TYPE_BF16); + __nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES]; + float * dst[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) { + src[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); + dst[i] = static_cast(tensors[i]->data); } - case GGML_TYPE_BF16: { - __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); - ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16>(p, backends, buf, inner_compute, reduce_id, ne, nbytes, label); - break; + ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, float>( + p, backends, src, dst, inner_compute, reduce_id, ne, nbytes, label); + } else { + switch (kernel_type) { + case GGML_TYPE_F32: { + float * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); + ok = ggml_cuda_ar_allreduce_copy_impl( + p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); + break; + } + case GGML_TYPE_BF16: { + __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); + ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, __nv_bfloat16>( + p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); + break; + } + default: + GGML_ASSERT(false); } - default: - GGML_ASSERT(false); } } else { const char * label = use_bf16 ? "kernel-bf16" : "kernel"; @@ -1000,7 +1038,11 @@ bool ggml_cuda_ar_allreduce( } } - if (use_bf16 && ok) { + // Post-conversion BF16 -> F32 is needed only for the chunked-kernel path, + // which leaves its result in bf16_tmp. The copy_engine path's combined + // add kernel writes the F32 result directly into tensors[i]->data, so + // we can skip the post-conversion entirely. + if (use_bf16 && ok && !use_copy_engine) { to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); for (int i = 0; i < n; ++i) { auto * cuda_ctx = static_cast(backends[i]->context); From 80faf56865fff4d1b9d87438739fdd68eb6646b2 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 21:53:01 -0700 Subject: [PATCH 39/81] add bf16 wire format for single kernel mode --- ggml/src/ggml-cuda/allreduce.cu | 231 ++++++++++++++++---------------- 1 file changed, 112 insertions(+), 119 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index e192ee98b5c..83313807ce0 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -82,59 +82,40 @@ struct ggml_cuda_ar_debug_ring { }; #endif // GGML_CUDA_AR_WATCHDOG -// --------------------------------------------------------------------------- -// Vectorised add helpers for Phase 3 reduction. All types use float4 -// (16 bytes) as the vector load unit for maximum PCIe throughput. -// --------------------------------------------------------------------------- -template -static __device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b); - -template <> -__device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b) { - return make_float4(a.x + b.x, a.y + b.y, a.z + b.z, a.w + b.w); -} - -template <> -__device__ __forceinline__ float4 ggml_cuda_ar_vec_add(float4 a, float4 b) { - float4 r; - half2 * ha = reinterpret_cast(&a); - half2 * hb = reinterpret_cast(&b); - half2 * hr = reinterpret_cast(&r); - #pragma unroll - for (int k = 0; k < 4; ++k) { hr[k] = ha[k] + hb[k]; } - return r; -} - -template <> -__device__ __forceinline__ float4 ggml_cuda_ar_vec_add<__nv_bfloat16>(float4 a, float4 b) { - float4 r; - __nv_bfloat162 * ba = reinterpret_cast<__nv_bfloat162 *>(&a); - __nv_bfloat162 * bb = reinterpret_cast<__nv_bfloat162 *>(&b); - __nv_bfloat162 * br = reinterpret_cast<__nv_bfloat162 *>(&r); - #pragma unroll - for (int k = 0; k < 4; ++k) { br[k] = ba[k] + bb[k]; } - return r; -} - -template +// Combined chunked-kernel AllReduce. sendbuf/recvbuf live in Tdst (the +// caller's tensor type); host_mine/host_other carry data in Twire (the +// on-wire type, possibly narrower than Tdst). +// +// Phase 1 reads Tdst from sendbuf, casts each element to Twire, and packs +// 16-byte vectors into host_mine — for Tdst=F32, Twire=BF16 this halves the +// host bytes written. +// +// Phase 3 reads 16-byte Twire vectors from host_other, casts each element to +// Tdst, then sums with the local sendbuf value (also rounded through Twire +// for bit-equivalence between GPUs since both sides truncate). When +// Tdst == Twire the casts are no-ops and behaviour matches the original +// homogeneous kernel. +template static __global__ void ggml_cuda_ar_kernel( - const T * __restrict__ sendbuf, - T * __restrict__ recvbuf, - T * __restrict__ host_mine, - const T * __restrict__ host_other, - int count, - int * arrival_mine, - int * arrival_other + const Tdst * __restrict__ sendbuf, + Tdst * __restrict__ recvbuf, + Twire * __restrict__ host_mine, + const Twire * __restrict__ host_other, + int count, + int * arrival_mine, + int * arrival_other #if GGML_CUDA_AR_WATCHDOG ,ggml_cuda_ar_debug_ring * ring, - int max_spin, - int rank, - int ar_slot + int max_spin, + int rank, + int ar_slot #endif ) { - // Number of elements of T per float4 vector (16 bytes). - constexpr int ELEMS_PER_VEC = 16 / sizeof(T); + // 16-byte vector unit for the wire type. Each phase-1 iter writes one + // vector to host memory; each phase-3 iter reads one and produces + // ELEMS_PER_VEC sums. + constexpr int ELEMS_PER_VEC = 16 / sizeof(Twire); #if GGML_CUDA_AR_WATCHDOG __shared__ int bail; @@ -150,15 +131,20 @@ static __global__ void ggml_cuda_ar_kernel( __syncthreads(); #endif - // Phase 1: vectorised D2H copy using float4 (16 bytes per load/store). + // Phase 1: cast sendbuf (Tdst) -> host_mine (Twire) and store as 16-byte vectors. { - const float4 * s4 = reinterpret_cast(sendbuf); - float4 * d4 = reinterpret_cast(host_mine); for (int i = tid; i < count_vec; i += nt) { - d4[i] = s4[i]; + const int off = i * ELEMS_PER_VEC; + Twire wire[ELEMS_PER_VEC]; + #pragma unroll + for (int k = 0; k < ELEMS_PER_VEC; ++k) { + wire[k] = static_cast(sendbuf[off + k]); + } + *reinterpret_cast(&host_mine[off]) = + *reinterpret_cast(wire); } if (tid < count - tail) { - host_mine[tail + tid] = sendbuf[tail + tid]; + host_mine[tail + tid] = static_cast(sendbuf[tail + tid]); } } @@ -215,16 +201,24 @@ static __global__ void ggml_cuda_ar_kernel( // Broadcast "peer has arrived" and acquire peer's host_other writes. __threadfence_system(); - // Phase 3: reduce. + // Phase 3: read peer's Twire vector, cast both sides through Twire for + // bit-equivalence, sum in Tdst precision, and write back to recvbuf. { - const float4 * s4 = reinterpret_cast(sendbuf); - const float4 * o4 = reinterpret_cast(host_other); - float4 * r4 = reinterpret_cast(recvbuf); for (int i = tid; i < count_vec; i += nt) { - r4[i] = ggml_cuda_ar_vec_add(s4[i], o4[i]); + const int off = i * ELEMS_PER_VEC; + Twire wire[ELEMS_PER_VEC]; + *reinterpret_cast(wire) = + *reinterpret_cast(&host_other[off]); + #pragma unroll + for (int k = 0; k < ELEMS_PER_VEC; ++k) { + const Twire d_low = static_cast(sendbuf[off + k]); + recvbuf[off + k] = static_cast(d_low) + static_cast(wire[k]); + } } if (tid < count - tail) { - recvbuf[tail + tid] = sendbuf[tail + tid] + host_other[tail + tid]; + const Twire d_low = static_cast(sendbuf[tail + tid]); + recvbuf[tail + tid] = + static_cast(d_low) + static_cast(host_other[tail + tid]); } } } @@ -877,10 +871,34 @@ bool ggml_cuda_ar_allreduce( compute_flag[i] = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; } - ggml_cuda_pool_alloc bf16_tmp[GGML_CUDA_MAX_DEVICES]; - void * data_ptr[GGML_CUDA_MAX_DEVICES]; + // Decide between copy-engine and chunked-kernel paths based on the working + // type's actual byte count. + const bool use_copy_engine = + p->copy_threshold > 0 && + nbytes >= p->copy_threshold && + nbytes <= p->copy_bytes; + // BF16 inactive-shard zeroing: when use_bf16 is on, the combined kernel + // (chunked-kernel path) and the combined add kernel (copy_engine path) + // both accumulate into the F32 tensor data directly, so an inactive + // shard's accumulator must start at zero. if (use_bf16) { + for (int i = 0; i < n; ++i) { + if (!compute_flag[i]) { + auto * cuda_ctx = static_cast(backends[i]->context); + ggml_cuda_set_device(p->devices[i]); + CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, (size_t) ne * sizeof(float), cuda_ctx->stream())); + } + } + } + + // Pre-convert F32 -> BF16 into bf16_tmp ONLY for the copy_engine + use_bf16 + // path; the chunked-kernel path's combined kernel does the conversion + // inline as it writes to host_buf. + ggml_cuda_pool_alloc bf16_tmp[GGML_CUDA_MAX_DEVICES]; + void * copy_src_ptr[GGML_CUDA_MAX_DEVICES] = {}; + + if (use_copy_engine && use_bf16) { to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); for (int i = 0; i < n; ++i) { auto * cuda_ctx = static_cast(backends[i]->context); @@ -891,28 +909,12 @@ bool ggml_cuda_ar_allreduce( to_bf16(tensors[i]->data, bf16_tmp[i].get(), ne, cuda_ctx->stream()); } else { CUDA_CHECK(cudaMemsetAsync(bf16_tmp[i].get(), 0, nbytes, cuda_ctx->stream())); - // The copy_engine path's combined add kernel reads/writes the - // F32 tensor data directly, so an inactive shard's accumulator - // must start at zero too. (The chunked-kernel path's - // post-conversion would also overwrite this with zero.) - CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, (size_t) ne * sizeof(float), cuda_ctx->stream())); } CUDA_CHECK(cudaGetLastError()); - data_ptr[i] = bf16_tmp[i].get(); - } - } else { - for (int i = 0; i < n; ++i) { - data_ptr[i] = tensors[i]->data; + copy_src_ptr[i] = bf16_tmp[i].get(); } } - // Decide between copy-engine and chunked-kernel paths based on the working - // type's actual byte count. - const bool use_copy_engine = - p->copy_threshold > 0 && - nbytes >= p->copy_threshold && - nbytes <= p->copy_bytes; - bool ok = true; if (use_copy_engine) { const char * label = use_bf16 ? "copy_engine-bf16" : "copy_engine"; @@ -938,7 +940,7 @@ bool ggml_cuda_ar_allreduce( __nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES]; float * dst[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) { - src[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); + src[i] = static_cast<__nv_bfloat16 *>(copy_src_ptr[i]); dst[i] = static_cast(tensors[i]->data); } ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, float>( @@ -947,14 +949,14 @@ bool ggml_cuda_ar_allreduce( switch (kernel_type) { case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast(data_ptr[i]); + for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); ok = ggml_cuda_ar_allreduce_copy_impl( p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); break; } case GGML_TYPE_BF16: { __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(data_ptr[i]); + for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(tensors[i]->data); ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, __nv_bfloat16>( p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); break; @@ -965,28 +967,27 @@ bool ggml_cuda_ar_allreduce( } } else { const char * label = use_bf16 ? "kernel-bf16" : "kernel"; + // host_buf carries Twire-typed data; max_chunk_elems is the count that + // fits in one host_buf at the wire size. const size_t max_chunk_elems = p->buf_bytes / type_size; const size_t chunks = ((size_t) ne + max_chunk_elems - 1) / max_chunk_elems; + const size_t input_type_size = ggml_type_size(input_type); ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, max_chunk_elems, chunks); - // Chunked-kernel path. Insert per-chunk kernels into each GPU's - // existing compute stream via events: - // record(app, compute_stream) — capture "upstream done" - // (incl. F32->BF16 conversion when applicable) - // wait(internal_stream, app) — internal stream defers until then - // launch one or more chunk kernels on internal_stream - // record(ker, internal_stream) — capture "final chunk done" - // wait(compute_stream, ker) — compute stream resumes - // (then runs BF16->F32 when applicable) + // Chunked-kernel path. The combined kernel reads Tdst (input_type) + // from tensors[i]->data, casts on-the-fly to Twire (kernel_type) for + // the host transfer, and accumulates the peer's Twire contribution + // back into tensors[i]->data — no pre/post-conversion needed. size_t chunk_index = 0; for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { const size_t remaining_elems = (size_t) (ne - chunk_start); const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; - const size_t chunk_bytes = chunk_elems * type_size; + const size_t chunk_wire_bytes = chunk_elems * type_size; + const size_t chunk_dst_bytes = chunk_elems * input_type_size; const int slot = ggml_cuda_ar_acquire_slot(p); const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; - ggml_cuda_ar_trace_chunk(p, reduce_id, label, chunk_index, slot, chunk_start, chunk_elems, chunk_bytes, last_chunk); + ggml_cuda_ar_trace_chunk(p, reduce_id, label, chunk_index, slot, chunk_start, chunk_elems, chunk_wire_bytes, last_chunk); for (int i = 0; i < n; ++i) { const int peer = 1 - i; // valid for n == 2 only @@ -997,12 +998,13 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_wait_for_compute(p, cuda_ctx, i, slot); } - char * data = static_cast(data_ptr[i]) + chunk_start * (int64_t) type_size; + char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) input_type_size; // Match NCCL/meta-backend semantics: inactive shards contribute - // zeros. On the BF16 path the tmp buffer was already zeroed. + // zeros. On the BF16 path the F32 tensor data was already + // zeroed up-front (above), so per-chunk zeroing isn't needed. if (!compute_flag[i] && !use_bf16) { - CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_bytes, p->streams[i])); + CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_dst_bytes, p->streams[i])); } #if GGML_CUDA_AR_WATCHDOG @@ -1011,22 +1013,27 @@ bool ggml_cuda_ar_allreduce( #define GGML_CUDA_AR_WDOG_EXTRA_ARGS #endif -#define LAUNCH_AR_KERNEL(T) \ - ggml_cuda_ar_kernel<<streams[i]>>>( \ - reinterpret_cast(data), \ - reinterpret_cast(data), \ - reinterpret_cast(p->host_buf[i]), \ - reinterpret_cast(p->host_buf[peer]), \ +#define LAUNCH_AR_KERNEL(Tdst, Twire) \ + ggml_cuda_ar_kernel<<streams[i]>>>( \ + reinterpret_cast(data), \ + reinterpret_cast(data), \ + reinterpret_cast(p->host_buf[i]), \ + reinterpret_cast(p->host_buf[peer]), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ ggml_cuda_ar_arrival_ptr(p, slot, peer) \ GGML_CUDA_AR_WDOG_EXTRA_ARGS) - switch (kernel_type) { - case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float); break; - case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half); break; - case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16); break; - default: GGML_ASSERT(false); + if (use_bf16) { + GGML_ASSERT(input_type == GGML_TYPE_F32); + LAUNCH_AR_KERNEL(float, __nv_bfloat16); + } else { + switch (input_type) { + case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float, float); break; + case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half, half); break; + case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16, __nv_bfloat16); break; + default: GGML_ASSERT(false); + } } #undef LAUNCH_AR_KERNEL @@ -1038,19 +1045,5 @@ bool ggml_cuda_ar_allreduce( } } - // Post-conversion BF16 -> F32 is needed only for the chunked-kernel path, - // which leaves its result in bf16_tmp. The copy_engine path's combined - // add kernel writes the F32 result directly into tensors[i]->data, so - // we can skip the post-conversion entirely. - if (use_bf16 && ok && !use_copy_engine) { - to_fp32_cuda_t to_fp32 = ggml_get_to_fp32_cuda(GGML_TYPE_BF16); - for (int i = 0; i < n; ++i) { - auto * cuda_ctx = static_cast(backends[i]->context); - ggml_cuda_set_device(p->devices[i]); - to_fp32(bf16_tmp[i].get(), (float *) tensors[i]->data, ne, cuda_ctx->stream()); - CUDA_CHECK(cudaGetLastError()); - } - } - return ok; } From 210165bed9c85d74b0856f67246dc37265883779 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 22:06:50 -0700 Subject: [PATCH 40/81] experimental on-stream small reduction kernel --- ggml/src/ggml-cuda/allreduce.cu | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 83313807ce0..d484b12c366 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -974,10 +974,12 @@ bool ggml_cuda_ar_allreduce( const size_t input_type_size = ggml_type_size(input_type); ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, max_chunk_elems, chunks); - // Chunked-kernel path. The combined kernel reads Tdst (input_type) - // from tensors[i]->data, casts on-the-fly to Twire (kernel_type) for - // the host transfer, and accumulates the peer's Twire contribution - // back into tensors[i]->data — no pre/post-conversion needed. + // Chunked-kernel path runs entirely on the caller's compute stream: + // since AR is a barrier here, same-stream ordering replaces the + // wait_for_compute / record_chunk_done event pairs and skips the + // cross-stream scheduling overhead that was hurting the small-tensor + // (tg) latency on the AR-stream variant. Only ev.ker is still + // recorded at end-of-AR for acquire_slot's pool-wraparound check. size_t chunk_index = 0; for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { const size_t remaining_elems = (size_t) (ne - chunk_start); @@ -993,10 +995,7 @@ bool ggml_cuda_ar_allreduce( const int peer = 1 - i; // valid for n == 2 only ggml_cuda_set_device(p->devices[i]); auto * cuda_ctx = static_cast(backends[i]->context); - - if (chunk_start == 0) { - ggml_cuda_ar_wait_for_compute(p, cuda_ctx, i, slot); - } + cudaStream_t stream = cuda_ctx->stream(); char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) input_type_size; @@ -1004,7 +1003,7 @@ bool ggml_cuda_ar_allreduce( // zeros. On the BF16 path the F32 tensor data was already // zeroed up-front (above), so per-chunk zeroing isn't needed. if (!compute_flag[i] && !use_bf16) { - CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_dst_bytes, p->streams[i])); + CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_dst_bytes, stream)); } #if GGML_CUDA_AR_WATCHDOG @@ -1014,7 +1013,7 @@ bool ggml_cuda_ar_allreduce( #endif #define LAUNCH_AR_KERNEL(Tdst, Twire) \ - ggml_cuda_ar_kernel<<streams[i]>>>( \ + ggml_cuda_ar_kernel<<>>( \ reinterpret_cast(data), \ reinterpret_cast(data), \ reinterpret_cast(p->host_buf[i]), \ @@ -1040,7 +1039,9 @@ bool ggml_cuda_ar_allreduce( #undef GGML_CUDA_AR_WDOG_EXTRA_ARGS CUDA_CHECK(cudaGetLastError()); - ggml_cuda_ar_record_chunk_done(p, cuda_ctx, i, slot, last_chunk); + if (last_chunk) { + CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].ker, stream)); + } } } } From fbc2bd7098b1ee7592f1bc89447be8441c1779c4 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 23:14:36 -0700 Subject: [PATCH 41/81] double buffer arrival slots, use token (incrementing) method --- ggml/src/ggml-cuda/allreduce.cu | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index d484b12c366..ac20037e0ce 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -248,7 +248,12 @@ static __global__ void ggml_cuda_ar_add_kernel( // Number of slots in the event / arrival ring. 128 is well above the actual // in-flight depth (single digits in practice) while keeping init cost low. -static constexpr int GGML_CUDA_AR_POOL_SIZE = 128; +// Two-slot ring is sufficient: lockstep guarantees the two GPUs are at most +// one AR (or chunk) apart, so slot[N%2] is always safe to reuse — peer has +// already consumed slot[N%2] from AR N-2 by the time we get to AR N. The +// slot wraparound's cudaEventSynchronize on ev.ker covers the host-side +// arrival reset against the prior AR's kernel. +static constexpr int GGML_CUDA_AR_POOL_SIZE = 2; // Maximum chunk size (bytes per GPU) handled by one internal kernel launch. // Larger tensors are reduced by issuing multiple chunked launches. From 3a41c7b33d53a77ec65020c93f78e300527ef941 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 27 Apr 2026 23:33:45 -0700 Subject: [PATCH 42/81] double buffer host_buf for small reductions --- ggml/src/ggml-cuda/allreduce.cu | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index ac20037e0ce..1ee9d98dda1 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -629,16 +629,19 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } memset(p->arrival, 0, arrival_bytes); - // Per-device pinned staging buffers. + // Per-device pinned staging buffers — POOL_SIZE-deep ring so the chunked- + // kernel can write the next slot's data while the peer is still reading + // the previous slot's. Indexed by (slot * buf_bytes) at the call site. p->buf_bytes = GGML_CUDA_AR_MAX_BYTES; + const size_t host_buf_total = (size_t) GGML_CUDA_AR_POOL_SIZE * p->buf_bytes; for (int i = 0; i < n_devices; ++i) { - if (cudaHostAlloc(&p->host_buf[i], p->buf_bytes, cudaHostAllocPortable) != cudaSuccess) { + if (cudaHostAlloc(&p->host_buf[i], host_buf_total, cudaHostAllocPortable) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for staging failed (%zu bytes)\n", - __func__, p->buf_bytes); + __func__, host_buf_total); ggml_cuda_ar_pipeline_free(p); return nullptr; } - memset(p->host_buf[i], 0, p->buf_bytes); + memset(p->host_buf[i], 0, host_buf_total); } // Prototype copy-engine path resources. Keep these deliberately large for @@ -1021,8 +1024,8 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_kernel<<>>( \ reinterpret_cast(data), \ reinterpret_cast(data), \ - reinterpret_cast(p->host_buf[i]), \ - reinterpret_cast(p->host_buf[peer]), \ + reinterpret_cast(p->host_buf[i] + (size_t) slot * p->buf_bytes), \ + reinterpret_cast(p->host_buf[peer] + (size_t) slot * p->buf_bytes), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ ggml_cuda_ar_arrival_ptr(p, slot, peer) \ From 330c14c9e73faaac934cdc720887d371227bcf4b Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 13:23:30 -0700 Subject: [PATCH 43/81] put in waits for use of host_mem in large reduction case (prevents stomping on in-use memory --- ggml/src/ggml-cuda/allreduce.cu | 35 +++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 1ee9d98dda1..f5249deeabd 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -304,6 +304,13 @@ struct ggml_cuda_ar_pipeline { cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking ggml_cuda_ar_event_slot *ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] + // Copy-engine: per-device "I finished reading my peer's host_large" + // event. Indexed by RECORDER device. Recorded same-device on streams[i] + // after stage 2's last H2D from host_large[peer]. Waited cross-device + // by peer's stage-1 stream before the next AR overwrites host_large[peer]. + cudaEvent_t host_large_read_done[GGML_CUDA_MAX_DEVICES]; + bool host_large_read_done_valid; + // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; @@ -579,7 +586,9 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->dev_tmp[i] = nullptr; p->streams[i] = nullptr; p->ev_pool[i] = nullptr; + p->host_large_read_done[i] = nullptr; } + p->host_large_read_done_valid = false; #if GGML_CUDA_AR_WATCHDOG for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { p->debug_ring[i] = nullptr; @@ -615,6 +624,13 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n return nullptr; } } + + if (cudaEventCreateWithFlags(&p->host_large_read_done[i], cudaEventDisableTiming) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaEventCreate for host_large_read_done failed for device %d\n", + __func__, p->devices[i]); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } } // Arrival ring: cache-line padded so each GPU's int is on its own line. @@ -733,6 +749,10 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { } delete[] p->ev_pool[i]; } + if (p->host_large_read_done[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaEventDestroy(p->host_large_read_done[i]); + } if (p->streams[i]) { ggml_cuda_set_device(p->devices[i]); cudaStreamDestroy(p->streams[i]); @@ -786,6 +806,15 @@ static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); + // Wait for peer's H2D from our host_large[i] (recorded in the + // previous AR's stage 2) to complete before we overwrite host_large[i]. + // host_large_read_done[peer] = peer finished reading host_large[i]. + // No-op on the first AR — no prior record exists. + if (p->host_large_read_done_valid) { + const int peer = 1 - i; + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->host_large_read_done[peer])); + } + if (!compute[i]) { CUDA_CHECK(cudaMemsetAsync(src_buf[i], 0, nbytes, p->streams[i])); } @@ -825,6 +854,11 @@ static bool ggml_cuda_ar_allreduce_copy_impl( cudaMemcpyHostToDevice, p->streams[i])); } + // Mark our reads of host_large[peer] complete so peer's next AR can + // safely overwrite it. Same-device record (event[i] on stream[i]); + // peer waits cross-device on event[i] before its next stage 1 D2H. + CUDA_CHECK(cudaEventRecord(p->host_large_read_done[i], p->streams[i])); + const int block_size = 256; int n_blocks = (int) ((ne + block_size - 1) / block_size); if (n_blocks > 1024) { @@ -838,6 +872,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot, true); } + p->host_large_read_done_valid = true; return true; } From 2b91d0253995e0f9591f221d4d167c9f4c4142a8 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 13:57:57 -0700 Subject: [PATCH 44/81] remove watchdog code --- ggml/src/ggml-cuda/allreduce.cu | 212 +------------------------------- 1 file changed, 2 insertions(+), 210 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index f5249deeabd..b128c06dd31 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -8,18 +8,6 @@ #include #include -// Set to 1 to enable the AllReduce spin-limit watchdog (development only). -// When enabled, the debug kernel bails out after GGML_CUDA_AR_MAX_SPIN -// iterations and writes a record to a per-GPU ring buffer that the -// background watchdog thread prints. -#define GGML_CUDA_AR_WATCHDOG 0 - -#if GGML_CUDA_AR_WATCHDOG -#include -#include -#include -#endif - // --------------------------------------------------------------------------- // Cross-GPU signal mechanism // @@ -53,35 +41,8 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { // The single-block configuration means __syncthreads() is sufficient for // intra-block coordination and we can use the cheaper non-cooperative launch. // 256 threads gives good occupancy while keeping register pressure low. -// -// When GGML_CUDA_AR_WATCHDOG is enabled, Phase 2 has a spin limit -// (max_spin). If the limit is reached the kernel writes a debug record to -// a per-GPU ring buffer in pinned host memory, then bails out — all threads -// exit the kernel immediately (Phase 3 is skipped). // --------------------------------------------------------------------------- -#if GGML_CUDA_AR_WATCHDOG -// One debug record written by the kernel on spin-limit bailout. -struct ggml_cuda_ar_debug_record { - int rank; // GPU rank (0 or 1) - int slot; // AllReduce pool slot - int spin_count; // spins before bailout - int arrival_mine; // readback of own arrival flag after signal_set - int arrival_other; // last value of peer's arrival flag - int count; // element count of the AllReduce call - int complete; // 1 = record fully written (set last, after fence) -}; - -static constexpr int GGML_CUDA_AR_RING_SIZE = 64; - -// Per-GPU ring buffer in pinned host memory. head is incremented by the -// GPU via atomicAdd; records[] is written by the GPU and read by the host. -struct ggml_cuda_ar_debug_ring { - int head; // next slot to write (GPU atomicAdd) - ggml_cuda_ar_debug_record records[GGML_CUDA_AR_RING_SIZE]; -}; -#endif // GGML_CUDA_AR_WATCHDOG - // Combined chunked-kernel AllReduce. sendbuf/recvbuf live in Tdst (the // caller's tensor type); host_mine/host_other carry data in Twire (the // on-wire type, possibly narrower than Tdst). @@ -103,34 +64,18 @@ static __global__ void ggml_cuda_ar_kernel( const Twire * __restrict__ host_other, int count, int * arrival_mine, - int * arrival_other -#if GGML_CUDA_AR_WATCHDOG - ,ggml_cuda_ar_debug_ring * ring, - int max_spin, - int rank, - int ar_slot -#endif - ) { + int * arrival_other) { // 16-byte vector unit for the wire type. Each phase-1 iter writes one // vector to host memory; each phase-3 iter reads one and produces // ELEMS_PER_VEC sums. constexpr int ELEMS_PER_VEC = 16 / sizeof(Twire); -#if GGML_CUDA_AR_WATCHDOG - __shared__ int bail; -#endif - const int tid = threadIdx.x; const int nt = blockDim.x; const int count_vec = count / ELEMS_PER_VEC; const int tail = count_vec * ELEMS_PER_VEC; -#if GGML_CUDA_AR_WATCHDOG - if (tid == 0) { bail = 0; } - __syncthreads(); -#endif - // Phase 1: cast sendbuf (Tdst) -> host_mine (Twire) and store as 16-byte vectors. { for (int i = tid; i < count_vec; i += nt) { @@ -158,45 +103,12 @@ static __global__ void ggml_cuda_ar_kernel( __threadfence_system(); // ensure the signal itself is visible across all GPUs -#if GGML_CUDA_AR_WATCHDOG - int writeback = ggml_cuda_ar_signal_get(arrival_mine); - int spin = 0; - int last = 0; - while ((last = ggml_cuda_ar_signal_get(arrival_other)) == 0) { - ++spin; - if (max_spin > 0 && spin >= max_spin) { - int ri = atomicAdd(&ring->head, 1) % GGML_CUDA_AR_RING_SIZE; - ggml_cuda_ar_debug_record * rec = &ring->records[ri]; - - rec->rank = rank; - rec->slot = ar_slot; - rec->spin_count = spin; - rec->arrival_mine = writeback; - rec->arrival_other = last; - rec->count = count; - - __threadfence_system(); - rec->complete = 1; - __threadfence_system(); - - bail = 1; - break; - } - __nanosleep(100); - } -#else while (ggml_cuda_ar_signal_get(arrival_other) == 0) { __nanosleep(100); } -#endif } __syncthreads(); -#if GGML_CUDA_AR_WATCHDOG - if (bail) { - return; - } -#endif // Broadcast "peer has arrived" and acquire peer's host_other writes. __threadfence_system(); @@ -275,11 +187,6 @@ static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = // preventing false-sharing stalls on the polling GPU. static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; -#if GGML_CUDA_AR_WATCHDOG -// Watchdog poll interval in milliseconds. -static constexpr int GGML_CUDA_AR_WDOG_POLL_MS = 1; -#endif - struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete cudaEvent_t cpy[GGML_CUDA_AR_COPY_MAX_CHUNKS] = {}; // copy-engine D2H chunks complete @@ -321,15 +228,6 @@ struct ggml_cuda_ar_pipeline { uint64_t trace_limit; uint64_t trace_chunk_limit; uint64_t trace_chunk_count; - -#if GGML_CUDA_AR_WATCHDOG - // Per-GPU debug ring buffers in pinned host memory. Written by the debug - // kernel on spin-limit bailout, read by the background watchdog thread. - ggml_cuda_ar_debug_ring * debug_ring[GGML_CUDA_MAX_DEVICES]; - int wdog_max_spin; // 0 = no limit (env: GGML_CUDA_AR_MAX_SPIN) - std::atomic wdog_stop{false}; - std::thread wdog_thr; -#endif }; // Return a pointer to the arrival int for (slot, rank). @@ -472,82 +370,6 @@ static void ggml_cuda_ar_record_chunk_done( } } -// --------------------------------------------------------------------------- -// Background watchdog thread — monitors per-GPU debug ring buffers for new -// bailout records. The kernel writes a record when it hits the spin limit; -// this thread polls the ring head counters every 1ms and prints any new -// complete records. Zero overhead on the dispatch path (no queue, no events). -// --------------------------------------------------------------------------- -#if GGML_CUDA_AR_WATCHDOG -static void ggml_cuda_ar_wdog_thread(ggml_cuda_ar_pipeline * p) { - int last_seen[GGML_CUDA_MAX_DEVICES] = {}; - - while (!p->wdog_stop.load(std::memory_order_relaxed)) { - for (int i = 0; i < p->n_devices; ++i) { - ggml_cuda_ar_debug_ring * ring = p->debug_ring[i]; - if (!ring) { continue; } - - int head = *(volatile int *)&ring->head; - while (last_seen[i] < head) { - int ri = last_seen[i] % GGML_CUDA_AR_RING_SIZE; - const ggml_cuda_ar_debug_record * rec = &ring->records[ri]; - - // Wait for the completion flag (kernel writes it last after fence). - if (*(volatile int *)&rec->complete) { - GGML_LOG_WARN("ggml_cuda_ar BAILOUT: gpu%d rank=%d slot=%d " - "spins=%d arrival_mine=%d arrival_other=%d count=%d\n", - p->devices[i], rec->rank, rec->slot, - rec->spin_count, rec->arrival_mine, - rec->arrival_other, rec->count); - last_seen[i]++; - } else { - break; // record not yet complete — check again next poll - } - } - } - std::this_thread::sleep_for(std::chrono::milliseconds(GGML_CUDA_AR_WDOG_POLL_MS)); - } -} - -static bool ggml_cuda_ar_wdog_init(ggml_cuda_ar_pipeline * p) { - for (int i = 0; i < p->n_devices; ++i) { - if (cudaHostAlloc(reinterpret_cast(&p->debug_ring[i]), - sizeof(ggml_cuda_ar_debug_ring), - cudaHostAllocPortable) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for debug ring failed on device %d\n", - __func__, p->devices[i]); - return false; - } - memset(p->debug_ring[i], 0, sizeof(ggml_cuda_ar_debug_ring)); - } - - const char * spin_env = getenv("GGML_CUDA_AR_MAX_SPIN"); - p->wdog_max_spin = (spin_env && spin_env[0]) ? atoi(spin_env) : 0; - GGML_LOG_INFO("%s: AR watchdog enabled — max_spin=%d " - "(set GGML_CUDA_AR_MAX_SPIN= to adjust)\n", - __func__, p->wdog_max_spin); - - p->wdog_stop.store(false); - p->wdog_thr = std::thread(ggml_cuda_ar_wdog_thread, p); - return true; -} - -static void ggml_cuda_ar_wdog_stop(ggml_cuda_ar_pipeline * p) { - p->wdog_stop.store(true); - if (p->wdog_thr.joinable()) { - p->wdog_thr.join(); - } -} - -static void ggml_cuda_ar_wdog_free(ggml_cuda_ar_pipeline * p) { - for (int i = 0; i < p->n_devices; ++i) { - if (p->debug_ring[i]) { - cudaFreeHost(p->debug_ring[i]); - } - } -} -#endif // GGML_CUDA_AR_WATCHDOG - // --------------------------------------------------------------------------- // Init / free // --------------------------------------------------------------------------- @@ -589,12 +411,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->host_large_read_done[i] = nullptr; } p->host_large_read_done_valid = false; -#if GGML_CUDA_AR_WATCHDOG - for (int i = 0; i < GGML_CUDA_MAX_DEVICES; ++i) { - p->debug_ring[i] = nullptr; - } - p->wdog_max_spin = 0; -#endif // Per-device streams and event pools. for (int i = 0; i < n_devices; ++i) { @@ -678,13 +494,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } } -#if GGML_CUDA_AR_WATCHDOG - if (!ggml_cuda_ar_wdog_init(p)) { - ggml_cuda_ar_pipeline_free(p); - return nullptr; - } -#endif - GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", __func__, n_devices, p->buf_bytes >> 10); @@ -713,12 +522,6 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { return; } -#if GGML_CUDA_AR_WATCHDOG - // Stop the watchdog thread first — it only reads pinned host memory, - // no GPU resources, so this is safe and returns within ~1ms. - ggml_cuda_ar_wdog_stop(p); -#endif - // Drain all in-flight kernels before tearing down resources. for (int i = 0; i < p->n_devices; ++i) { if (p->streams[i]) { @@ -761,9 +564,6 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { if (p->arrival) { cudaFreeHost(p->arrival); } -#if GGML_CUDA_AR_WATCHDOG - ggml_cuda_ar_wdog_free(p); -#endif delete p; } @@ -1049,12 +849,6 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_dst_bytes, stream)); } -#if GGML_CUDA_AR_WATCHDOG -#define GGML_CUDA_AR_WDOG_EXTRA_ARGS , p->debug_ring[i], p->wdog_max_spin, i, slot -#else -#define GGML_CUDA_AR_WDOG_EXTRA_ARGS -#endif - #define LAUNCH_AR_KERNEL(Tdst, Twire) \ ggml_cuda_ar_kernel<<>>( \ reinterpret_cast(data), \ @@ -1063,8 +857,7 @@ bool ggml_cuda_ar_allreduce( reinterpret_cast(p->host_buf[peer] + (size_t) slot * p->buf_bytes), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ - ggml_cuda_ar_arrival_ptr(p, slot, peer) \ - GGML_CUDA_AR_WDOG_EXTRA_ARGS) + ggml_cuda_ar_arrival_ptr(p, slot, peer)) if (use_bf16) { GGML_ASSERT(input_type == GGML_TYPE_F32); @@ -1079,7 +872,6 @@ bool ggml_cuda_ar_allreduce( } #undef LAUNCH_AR_KERNEL -#undef GGML_CUDA_AR_WDOG_EXTRA_ARGS CUDA_CHECK(cudaGetLastError()); if (last_chunk) { From ff4ed48daca347715bce5b06935cc9df271453a9 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 14:36:58 -0700 Subject: [PATCH 45/81] various cleanups / dead code removal --- ggml/src/ggml-cuda/allreduce.cu | 206 +++++--------------------------- 1 file changed, 27 insertions(+), 179 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index b128c06dd31..37573c437cd 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -158,20 +158,20 @@ static __global__ void ggml_cuda_ar_add_kernel( // Pipeline structure // --------------------------------------------------------------------------- -// Number of slots in the event / arrival ring. 128 is well above the actual -// in-flight depth (single digits in practice) while keeping init cost low. -// Two-slot ring is sufficient: lockstep guarantees the two GPUs are at most -// one AR (or chunk) apart, so slot[N%2] is always safe to reuse — peer has -// already consumed slot[N%2] from AR N-2 by the time we get to AR N. The -// slot wraparound's cudaEventSynchronize on ev.ker covers the host-side -// arrival reset against the prior AR's kernel. +// Number of slots in the event / arrival ring. Two slots is sufficient: +// lockstep guarantees the two GPUs are at most one AR (or chunk) apart, so +// slot[N%2] is always safe to reuse — peer has already consumed slot[N%2] +// from AR N-2 by the time we get to AR N. The slot wraparound's +// cudaEventSynchronize on ev.ker covers the host-side arrival reset against +// the prior AR's kernel. static constexpr int GGML_CUDA_AR_POOL_SIZE = 2; -// Maximum chunk size (bytes per GPU) handled by one internal kernel launch. +// Maximum chunk size (bytes per GPU) handled by one chunked-kernel launch. // Larger tensors are reduced by issuing multiple chunked launches. static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB -// Prototype copy-engine path for large F32 reductions. +// Copy-engine path: largest tensor accepted on this path; sets host_large / +// dev_tmp allocation size. static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB @@ -202,7 +202,6 @@ struct ggml_cuda_ar_pipeline { size_t copy_chunk_bytes; size_t bf16_threshold; // tensors >= this size (bytes) are reduced via FP32->BF16 round-trip; 0 disables uint64_t call_count; - uint64_t reduce_count; // Per-device resources. char * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging @@ -221,13 +220,6 @@ struct ggml_cuda_ar_pipeline { // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; - - // Temporary host-side tracing for prefill AllReduce analysis. - bool trace_enabled; - bool trace_chunks; - uint64_t trace_limit; - uint64_t trace_chunk_limit; - uint64_t trace_chunk_count; }; // Return a pointer to the arrival int for (slot, rank). @@ -236,12 +228,6 @@ static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, return reinterpret_cast(p->arrival + offset); } -static bool ggml_cuda_ar_env_enabled(const char * name) { - const char * value = getenv(name); - return value != nullptr && value[0] != '\0' && strcmp(value, "0") != 0 && - strcmp(value, "false") != 0 && strcmp(value, "FALSE") != 0; -} - static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) { const char * value = getenv(name); if (value == nullptr || value[0] == '\0') { @@ -253,88 +239,6 @@ static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) return end != value ? (uint64_t) parsed : default_value; } -static void ggml_cuda_ar_trace_call( - const ggml_cuda_ar_pipeline * p, - uint64_t reduce_id, - const char * path, - ggml_tensor ** tensors, - ggml_type type, - int64_t ne, - size_t nbytes, - size_t max_chunk_elems, - size_t chunks) { - if (!p->trace_enabled || reduce_id >= p->trace_limit) { - return; - } - - fprintf(stdout, - "GGML_CUDA_AR_TRACE call=%" PRIu64 - " path=%s name=\"%s\" type=%s ne=%" PRId64 " nbytes=%zu chunks=%zu" - " max_chunk_elems=%zu max_chunk_bytes=%zu" - " flags=[0x%x,0x%x] compute=[%d,%d]" - " data=[%p,%p]" - " ne0=[%" PRId64 ",%" PRId64 "] ne1=[%" PRId64 ",%" PRId64 "]" - " ne2=[%" PRId64 ",%" PRId64 "] ne3=[%" PRId64 ",%" PRId64 "]" - " nb0=[%zu,%zu] nb1=[%zu,%zu] nb2=[%zu,%zu] nb3=[%zu,%zu]\n", - reduce_id, - path, - tensors[0]->name, - ggml_type_name(type), - ne, - nbytes, - chunks, - max_chunk_elems, - max_chunk_elems * ggml_type_size(type), - tensors[0]->flags, - tensors[1]->flags, - (tensors[0]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0, - (tensors[1]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0, - tensors[0]->data, - tensors[1]->data, - tensors[0]->ne[0], tensors[1]->ne[0], - tensors[0]->ne[1], tensors[1]->ne[1], - tensors[0]->ne[2], tensors[1]->ne[2], - tensors[0]->ne[3], tensors[1]->ne[3], - (size_t) tensors[0]->nb[0], (size_t) tensors[1]->nb[0], - (size_t) tensors[0]->nb[1], (size_t) tensors[1]->nb[1], - (size_t) tensors[0]->nb[2], (size_t) tensors[1]->nb[2], - (size_t) tensors[0]->nb[3], (size_t) tensors[1]->nb[3]); - fflush(stdout); -} - -static void ggml_cuda_ar_trace_chunk( - ggml_cuda_ar_pipeline * p, - uint64_t reduce_id, - const char * path, - size_t chunk_index, - int slot, - int64_t chunk_start, - size_t chunk_elems, - size_t chunk_bytes, - bool last_chunk) { - if (!p->trace_enabled || !p->trace_chunks || - reduce_id >= p->trace_limit || - p->trace_chunk_count >= p->trace_chunk_limit) { - return; - } - - p->trace_chunk_count++; - fprintf(stdout, - "GGML_CUDA_AR_TRACE_CHUNK call=%" PRIu64 - " path=%s chunk=%zu slot=%d start=%" PRId64 - " elems=%zu bytes=%zu launches=%d last=%d\n", - reduce_id, - path, - chunk_index, - slot, - chunk_start, - chunk_elems, - chunk_bytes, - p->n_devices, - last_chunk); - fflush(stdout); -} - static int ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; @@ -362,12 +266,10 @@ static void ggml_cuda_ar_wait_for_compute( } static void ggml_cuda_ar_record_chunk_done( - ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot, bool last_chunk) { + ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot) { ggml_cuda_ar_event_slot & ev = p->ev_pool[rank][slot]; CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[rank])); - if (last_chunk) { - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); - } + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); } // --------------------------------------------------------------------------- @@ -376,41 +278,24 @@ static void ggml_cuda_ar_record_chunk_done( ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n_devices) { - if ((n_devices != 2) || (n_devices > GGML_CUDA_MAX_DEVICES)) { + if (n_devices != 2) { return nullptr; } auto * p = new ggml_cuda_ar_pipeline{}; - p->n_devices = n_devices; - p->buf_bytes = 0; - p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; - p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); - p->copy_chunk_bytes = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_CHUNK_BYTES", GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); + p->n_devices = n_devices; + p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; + p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); + p->copy_chunk_bytes = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_CHUNK_BYTES", GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); if (p->copy_chunk_bytes < GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN) { GGML_LOG_WARN("%s: GGML_CUDA_AR_COPY_CHUNK_BYTES=%zu below minimum %zu; clamping\n", __func__, p->copy_chunk_bytes, GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; } - p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default - p->call_count = 0; - p->reduce_count = 0; - p->arrival = nullptr; - p->trace_enabled = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE", 0) != 0 && - !ggml_cuda_ar_env_enabled("GGML_CUDA_AR_TRACE_DISABLE"); - p->trace_chunks = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_CHUNKS", 1) != 0; - p->trace_limit = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_LIMIT", 2048); - p->trace_chunk_limit = ggml_cuda_ar_env_u64("GGML_CUDA_AR_TRACE_CHUNK_LIMIT", 8192); - p->trace_chunk_count = 0; - for (int i = 0; i < n_devices; ++i) { - p->devices[i] = devices[i]; - p->host_buf[i] = nullptr; - p->host_large[i] = nullptr; - p->dev_tmp[i] = nullptr; - p->streams[i] = nullptr; - p->ev_pool[i] = nullptr; - p->host_large_read_done[i] = nullptr; + p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default + for (size_t i = 0; i < n_devices; ++i) { + p->devices[i] = devices[i]; } - p->host_large_read_done_valid = false; // Per-device streams and event pools. for (int i = 0; i < n_devices; ++i) { @@ -473,11 +358,10 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n ggml_cuda_ar_pipeline_free(p); return nullptr; } - memset(p->host_buf[i], 0, host_buf_total); } - // Prototype copy-engine path resources. Keep these deliberately large for - // now; memory footprint can be reduced after the bandwidth experiment. + // Copy-engine path: pinned host staging + device scratch, sized for the + // largest tensor we accept on this path (GGML_CUDA_AR_COPY_MAX_BYTES). for (int i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); if (cudaHostAlloc(&p->host_large[i], p->copy_bytes, cudaHostAllocPortable) != cudaSuccess) { @@ -497,22 +381,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " "%zu KB staging per GPU\n", __func__, n_devices, p->buf_bytes >> 10); - if (p->trace_enabled) { - fprintf(stdout, - "GGML_CUDA_AR_TRACE_INIT devices=%d staging_bytes=%zu pool=%d chunks=%d" - " copy_bytes=%zu copy_threshold=%zu copy_chunk_bytes=%zu" - " trace_limit=%" PRIu64 " chunk_limit=%" PRIu64 "\n", - p->n_devices, - p->buf_bytes, - GGML_CUDA_AR_POOL_SIZE, - p->trace_chunks, - p->copy_bytes, - p->copy_threshold, - p->copy_chunk_bytes, - p->trace_limit, - p->trace_chunk_limit); - fflush(stdout); - } return p; } @@ -584,10 +452,8 @@ static bool ggml_cuda_ar_allreduce_copy_impl( Tsrc * const src_buf[GGML_CUDA_MAX_DEVICES], Tdst * const dst_buf[GGML_CUDA_MAX_DEVICES], const bool compute[GGML_CUDA_MAX_DEVICES], - uint64_t reduce_id, int64_t ne, - size_t nbytes, - const char * trace_label) { + size_t nbytes) { GGML_ASSERT(p->n_devices == 2); GGML_ASSERT(nbytes <= p->copy_bytes); GGML_ASSERT(ne <= std::numeric_limits::max()); @@ -624,12 +490,6 @@ static bool ggml_cuda_ar_allreduce_copy_impl( const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? (nbytes - offset) : p->copy_chunk_bytes; - if (i == 0) { - ggml_cuda_ar_trace_chunk( - p, reduce_id, trace_label, c, slot, offset / sizeof(Tsrc), - chunk_bytes / sizeof(Tsrc), chunk_bytes, c + 1 == copy_chunks); - } - CUDA_CHECK(cudaMemcpyAsync( p->host_large[i] + offset, reinterpret_cast(src_buf[i]) + offset, chunk_bytes, cudaMemcpyDeviceToHost, p->streams[i])); @@ -670,7 +530,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( (int) ne); CUDA_CHECK(cudaGetLastError()); - ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot, true); + ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot); } p->host_large_read_done_valid = true; @@ -692,7 +552,6 @@ bool ggml_cuda_ar_allreduce( const int64_t ne = ggml_nelements(tensors[0]); GGML_ASSERT(ne > 0); - const uint64_t reduce_id = p->reduce_count++; const size_t input_nbytes = ggml_nbytes(tensors[0]); // BF16 round-trip: F32 inputs >= bf16_threshold are converted to BF16 for @@ -760,11 +619,6 @@ bool ggml_cuda_ar_allreduce( bool ok = true; if (use_copy_engine) { - const char * label = use_bf16 ? "copy_engine-bf16" : "copy_engine"; - const size_t copy_chunk_elems = p->copy_chunk_bytes / type_size; - const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; - ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, copy_chunk_elems, copy_chunks); - // After up-front BF16 conversion, the tmp buffers already hold the // (possibly zeroed-for-inactive) data, so the inner path can treat // every shard as compute. @@ -787,21 +641,21 @@ bool ggml_cuda_ar_allreduce( dst[i] = static_cast(tensors[i]->data); } ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, float>( - p, backends, src, dst, inner_compute, reduce_id, ne, nbytes, label); + p, backends, src, dst, inner_compute, ne, nbytes); } else { switch (kernel_type) { case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); ok = ggml_cuda_ar_allreduce_copy_impl( - p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); + p, backends, buf, buf, inner_compute, ne, nbytes); break; } case GGML_TYPE_BF16: { __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(tensors[i]->data); ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, __nv_bfloat16>( - p, backends, buf, buf, inner_compute, reduce_id, ne, nbytes, label); + p, backends, buf, buf, inner_compute, ne, nbytes); break; } default: @@ -809,13 +663,10 @@ bool ggml_cuda_ar_allreduce( } } } else { - const char * label = use_bf16 ? "kernel-bf16" : "kernel"; // host_buf carries Twire-typed data; max_chunk_elems is the count that // fits in one host_buf at the wire size. const size_t max_chunk_elems = p->buf_bytes / type_size; - const size_t chunks = ((size_t) ne + max_chunk_elems - 1) / max_chunk_elems; const size_t input_type_size = ggml_type_size(input_type); - ggml_cuda_ar_trace_call(p, reduce_id, label, tensors, kernel_type, ne, nbytes, max_chunk_elems, chunks); // Chunked-kernel path runs entirely on the caller's compute stream: // since AR is a barrier here, same-stream ordering replaces the @@ -823,16 +674,13 @@ bool ggml_cuda_ar_allreduce( // cross-stream scheduling overhead that was hurting the small-tensor // (tg) latency on the AR-stream variant. Only ev.ker is still // recorded at end-of-AR for acquire_slot's pool-wraparound check. - size_t chunk_index = 0; - for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems, ++chunk_index) { + for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems) { const size_t remaining_elems = (size_t) (ne - chunk_start); const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; - const size_t chunk_wire_bytes = chunk_elems * type_size; const size_t chunk_dst_bytes = chunk_elems * input_type_size; const int slot = ggml_cuda_ar_acquire_slot(p); const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; - ggml_cuda_ar_trace_chunk(p, reduce_id, label, chunk_index, slot, chunk_start, chunk_elems, chunk_wire_bytes, last_chunk); for (int i = 0; i < n; ++i) { const int peer = 1 - i; // valid for n == 2 only From 25ff620d76565d2f13a22e3a569afd26edb4dd17 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 15:17:56 -0700 Subject: [PATCH 46/81] fix fp16 mode --- ggml/src/ggml-cuda/allreduce.cu | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 37573c437cd..f20bdb097db 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -658,6 +658,13 @@ bool ggml_cuda_ar_allreduce( p, backends, buf, buf, inner_compute, ne, nbytes); break; } + case GGML_TYPE_F16: { + half * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); + ok = ggml_cuda_ar_allreduce_copy_impl( + p, backends, buf, buf, inner_compute, ne, nbytes); + break; + } default: GGML_ASSERT(false); } From 26c3d013f049ee82b28237a869aece7e9829500f Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 16:05:32 -0700 Subject: [PATCH 47/81] fix some comments/logging statements --- ggml/src/ggml-cuda/allreduce.cu | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index f20bdb097db..c7cbcf1c672 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -2,8 +2,6 @@ #include "convert.cuh" #include "ggml-impl.h" -#include -#include #include #include #include @@ -17,8 +15,9 @@ // __threadfence_system() provides the release ordering that makes the D2H // writes visible system-wide before the arrival flag is observed. // -// atomicAdd_system() is broken on RTX 5090 (hostNativeAtomicSupported = 0), -// so we use the volatile path throughout. +// atomicAdd_system() requires hostNativeAtomicSupported, which is unavailable +// on PCIe-attached consumer GPUs without NVLink, so the volatile path is the +// portable choice. // --------------------------------------------------------------------------- static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { @@ -379,8 +378,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " - "%zu KB staging per GPU\n", - __func__, n_devices, p->buf_bytes >> 10); + "%zu KB chunked-kernel staging + %zu MB copy-engine staging per GPU\n", + __func__, n_devices, p->buf_bytes >> 10, p->copy_bytes >> 20); return p; } From 83b9bacdccbab5d4787a54a5879fa0aad130b589 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 16:32:40 -0700 Subject: [PATCH 48/81] use increasing token scheme for arrival signals --- ggml/src/ggml-cuda/allreduce.cu | 58 ++++++++++++++++++++------------- 1 file changed, 35 insertions(+), 23 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index c7cbcf1c672..e389a4d500d 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -9,19 +9,27 @@ // --------------------------------------------------------------------------- // Cross-GPU signal mechanism // -// One int per (slot, rank) pair in pinned host memory: 0 = not arrived, -// 1 = arrived. There is exactly one writer (the owning GPU) and one reader -// (the peer), so we don't need atomics. A volatile store paired with -// __threadfence_system() provides the release ordering that makes the D2H -// writes visible system-wide before the arrival flag is observed. +// One int per (slot, rank) pair in pinned host memory. Each AR call writes a +// strictly increasing token (= the AR call number) into its own arrival int. +// The peer spins until its read of the other's arrival int equals the token +// it expects for this call — a mismatch means the peer hasn't arrived yet. +// Tokens never repeat over realistic call rates (32-bit int wraps in tens of +// days at thousands of ARs/sec), so arrival ints don't need to be reset +// between calls; we initialize once at pipeline init and let the values +// accumulate. +// +// There is exactly one writer (the owning GPU) and one reader (the peer), so +// we don't need atomics. A volatile store paired with __threadfence_system() +// provides the release ordering that makes the D2H writes visible system-wide +// before the arrival token is observed. // // atomicAdd_system() requires hostNativeAtomicSupported, which is unavailable // on PCIe-attached consumer GPUs without NVLink, so the volatile path is the // portable choice. // --------------------------------------------------------------------------- -static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p) { - *(volatile int *)p = 1; +static __device__ __forceinline__ void ggml_cuda_ar_signal_set(int * p, int token) { + *(volatile int *)p = token; } static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; @@ -34,7 +42,8 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { // // Phase 1 (all threads): copy sendbuf → host_mine via float4 loads. // __threadfence_system() commits writes to host. -// Phase 2 (thread 0): set arrival_mine = 1; spin on arrival_other == 1. +// Phase 2 (thread 0): write token to arrival_mine; spin until +// arrival_other == token. // Phase 3 (all threads): reduce: recvbuf[i] = sendbuf[i] + host_other[i]. // // The single-block configuration means __syncthreads() is sufficient for @@ -63,7 +72,8 @@ static __global__ void ggml_cuda_ar_kernel( const Twire * __restrict__ host_other, int count, int * arrival_mine, - int * arrival_other) { + int * arrival_other, + int token) { // 16-byte vector unit for the wire type. Each phase-1 iter writes one // vector to host memory; each phase-3 iter reads one and produces @@ -98,11 +108,11 @@ static __global__ void ggml_cuda_ar_kernel( // Phase 2: thread 0 signals arrival, then spins for the peer. if (tid == 0) { - ggml_cuda_ar_signal_set(arrival_mine); + ggml_cuda_ar_signal_set(arrival_mine, token); __threadfence_system(); // ensure the signal itself is visible across all GPUs - while (ggml_cuda_ar_signal_get(arrival_other) == 0) { + while (ggml_cuda_ar_signal_get(arrival_other) != token) { __nanosleep(100); } } @@ -160,9 +170,9 @@ static __global__ void ggml_cuda_ar_add_kernel( // Number of slots in the event / arrival ring. Two slots is sufficient: // lockstep guarantees the two GPUs are at most one AR (or chunk) apart, so // slot[N%2] is always safe to reuse — peer has already consumed slot[N%2] -// from AR N-2 by the time we get to AR N. The slot wraparound's -// cudaEventSynchronize on ev.ker covers the host-side arrival reset against -// the prior AR's kernel. +// from AR N-2 by the time we get to AR N. acquire_slot's +// cudaEventSynchronize on ev.ker for both devices makes that consumption +// explicit before we overwrite host_buf[slot] for the new AR. static constexpr int GGML_CUDA_AR_POOL_SIZE = 2; // Maximum chunk size (bytes per GPU) handled by one chunked-kernel launch. @@ -238,7 +248,12 @@ static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) return end != value ? (uint64_t) parsed : default_value; } -static int ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { +struct ggml_cuda_ar_slot_info { + int slot; + int token; +}; + +static ggml_cuda_ar_slot_info ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { const int slot = static_cast(p->call_count % GGML_CUDA_AR_POOL_SIZE); const bool pool_lapped = p->call_count >= GGML_CUDA_AR_POOL_SIZE; p->call_count++; @@ -250,11 +265,7 @@ static int ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * p) { } } - for (int i = 0; i < p->n_devices; ++i) { - *ggml_cuda_ar_arrival_ptr(p, slot, i) = 0; - } - - return slot; + return { slot, (int) p->call_count }; } static void ggml_cuda_ar_wait_for_compute( @@ -458,7 +469,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( GGML_ASSERT(ne <= std::numeric_limits::max()); GGML_ASSERT(p->copy_chunk_bytes > 0); - const int slot = ggml_cuda_ar_acquire_slot(p); + const int slot = ggml_cuda_ar_acquire_slot(p).slot; const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; GGML_ASSERT(copy_chunks <= GGML_CUDA_AR_COPY_MAX_CHUNKS); @@ -685,7 +696,7 @@ bool ggml_cuda_ar_allreduce( const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; const size_t chunk_dst_bytes = chunk_elems * input_type_size; - const int slot = ggml_cuda_ar_acquire_slot(p); + const auto [slot, token] = ggml_cuda_ar_acquire_slot(p); const bool last_chunk = chunk_start + (int64_t) chunk_elems == ne; for (int i = 0; i < n; ++i) { @@ -711,7 +722,8 @@ bool ggml_cuda_ar_allreduce( reinterpret_cast(p->host_buf[peer] + (size_t) slot * p->buf_bytes), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ - ggml_cuda_ar_arrival_ptr(p, slot, peer)) + ggml_cuda_ar_arrival_ptr(p, slot, peer), \ + token) if (use_bf16) { GGML_ASSERT(input_type == GGML_TYPE_F32); From 1fe1528616f0ddfa7d43a08da54065f2d15d7ef7 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 16:38:21 -0700 Subject: [PATCH 49/81] add top-level comment to allreduce.cu --- ggml/src/ggml-cuda/allreduce.cu | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index e389a4d500d..dfa4a0bd4f2 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -6,6 +6,30 @@ #include #include +// --------------------------------------------------------------------------- +// CUDA AllReduce for tensor-parallel inference across two GPUs. +// +// Provides a peer-to-peer, in-place sum reduction over matching tensors on +// two CUDA devices in the same process. Used by the tensor-split path +// alongside NCCL; targets setups without NVLink, where peer-to-peer transfers +// must go through pinned host memory over PCIe. +// +// Two reduction strategies are selected per call by tensor size: +// +// * Chunked-kernel path (small reductions): a single CUDA kernel both +// stages data through pinned host memory and performs the local sum. +// Cross-GPU synchronization happens *inside the kernel* (busy-wait on +// a host-memory flag), which keeps launch overhead low for the +// latency-sensitive token-generation case. +// +// * Copy-engine path (large reductions): the transfer is split into +// D2H + H2D cudaMemcpyAsync chunks driven by the GPU's copy engine, +// followed by a small device-side add kernel. Cross-GPU +// synchronization happens *outside the kernel*, via CUDA events +// between streams. This keeps the compute engine free while large +// transfers are in flight, which matters for prefill-sized tensors. +// --------------------------------------------------------------------------- + // --------------------------------------------------------------------------- // Cross-GPU signal mechanism // From 630a8005c314db2d040b5377887e195adac76642 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 16:40:42 -0700 Subject: [PATCH 50/81] improve top-level comment in allreduce.cu --- ggml/src/ggml-cuda/allreduce.cu | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index dfa4a0bd4f2..3df4aee8b58 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -9,10 +9,10 @@ // --------------------------------------------------------------------------- // CUDA AllReduce for tensor-parallel inference across two GPUs. // -// Provides a peer-to-peer, in-place sum reduction over matching tensors on -// two CUDA devices in the same process. Used by the tensor-split path -// alongside NCCL; targets setups without NVLink, where peer-to-peer transfers -// must go through pinned host memory over PCIe. +// Provides an in-place sum reduction over matching tensors on two CUDA +// devices in the same process. Used by the tensor-split path alongside +// NCCL; targets setups without NVLink, where data is exchanged between the +// GPUs by staging it through pinned host memory over PCIe. // // Two reduction strategies are selected per call by tensor size: // From 6d4b99839e4d9dd4517782d4e927595850059c8a Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 16:45:46 -0700 Subject: [PATCH 51/81] fix comments in ggml_cuda_ar_kernel --- ggml/src/ggml-cuda/allreduce.cu | 37 +++++++++++++++------------------ 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 3df4aee8b58..b4bcf2ab2fb 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -60,34 +60,31 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { } // --------------------------------------------------------------------------- -// Single-kernel AllReduce — 2 GPUs, supports float, half, and bfloat16. +// Chunked-kernel AllReduce — 2 GPUs, supports float, half, and bfloat16. // -// Both GPUs run this kernel simultaneously in independent streams. Each GPU: +// Both GPUs run this kernel simultaneously on independent streams. sendbuf +// and recvbuf live in Tdst (the caller's tensor type); host_mine / host_other +// carry data in Twire (the on-wire type, possibly narrower than Tdst — e.g. +// Tdst=F32 with Twire=BF16 halves the bytes pushed across PCIe). When +// Tdst == Twire the casts below are no-ops. // -// Phase 1 (all threads): copy sendbuf → host_mine via float4 loads. -// __threadfence_system() commits writes to host. -// Phase 2 (thread 0): write token to arrival_mine; spin until +// Each GPU runs three phases: +// +// Phase 1 (all threads): cast sendbuf (Tdst) → Twire and store as 16-byte +// vectors into host_mine. __threadfence_system() +// commits these writes to host memory. +// Phase 2 (thread 0): write token to arrival_mine; spin until // arrival_other == token. -// Phase 3 (all threads): reduce: recvbuf[i] = sendbuf[i] + host_other[i]. +// Phase 3 (all threads): read 16-byte Twire vectors from host_other, cast +// each element to Tdst, and sum with the local +// sendbuf value (also rounded through Twire so that +// both GPUs truncate identically — this guarantees +// bit-equivalent results across the two devices). // // The single-block configuration means __syncthreads() is sufficient for // intra-block coordination and we can use the cheaper non-cooperative launch. // 256 threads gives good occupancy while keeping register pressure low. // --------------------------------------------------------------------------- - -// Combined chunked-kernel AllReduce. sendbuf/recvbuf live in Tdst (the -// caller's tensor type); host_mine/host_other carry data in Twire (the -// on-wire type, possibly narrower than Tdst). -// -// Phase 1 reads Tdst from sendbuf, casts each element to Twire, and packs -// 16-byte vectors into host_mine — for Tdst=F32, Twire=BF16 this halves the -// host bytes written. -// -// Phase 3 reads 16-byte Twire vectors from host_other, casts each element to -// Tdst, then sums with the local sendbuf value (also rounded through Twire -// for bit-equivalence between GPUs since both sides truncate). When -// Tdst == Twire the casts are no-ops and behaviour matches the original -// homogeneous kernel. template static __global__ void ggml_cuda_ar_kernel( const Tdst * __restrict__ sendbuf, From 63608b7b0c44e4a535b6b8c1481796caf9e399f1 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 20:06:27 -0700 Subject: [PATCH 52/81] improve event handling for hostmem buffer usage tracking --- ggml/src/ggml-cuda/allreduce.cu | 53 +++++++++++++++++++++++++++++---- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index b4bcf2ab2fb..81fb62374f5 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -220,6 +220,7 @@ static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete cudaEvent_t cpy[GGML_CUDA_AR_COPY_MAX_CHUNKS] = {}; // copy-engine D2H chunks complete + cudaEvent_t h2d = nullptr; // copy-engine H2Ds complete (handoff AR stream → compute stream) cudaEvent_t ker = nullptr; // AllReduce kernel complete }; @@ -247,6 +248,13 @@ struct ggml_cuda_ar_pipeline { cudaEvent_t host_large_read_done[GGML_CUDA_MAX_DEVICES]; bool host_large_read_done_valid; + // Copy-engine: per-device "my add_kernel is done with dev_tmp" event. + // Recorded on the compute stream after each add_kernel; the AR stream + // waits on it before the next copy_impl's H2D overwrites dev_tmp. Lets us + // single-buffer dev_tmp despite add_kernel running on a separate stream. + cudaEvent_t dev_tmp_kernel_done[GGML_CUDA_MAX_DEVICES]; + bool dev_tmp_kernel_done_valid; + // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. // Use ggml_cuda_ar_arrival_ptr() to index. char * arrival; @@ -345,6 +353,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { bool ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].app, cudaEventDisableTiming) == cudaSuccess && + cudaEventCreateWithFlags(&p->ev_pool[i][s].h2d, cudaEventDisableTiming) == cudaSuccess && cudaEventCreateWithFlags(&p->ev_pool[i][s].ker, cudaEventDisableTiming) == cudaSuccess; for (int c = 0; ok && c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].cpy[c], cudaEventDisableTiming) == cudaSuccess; @@ -363,6 +372,12 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n ggml_cuda_ar_pipeline_free(p); return nullptr; } + if (cudaEventCreateWithFlags(&p->dev_tmp_kernel_done[i], cudaEventDisableTiming) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaEventCreate for dev_tmp_kernel_done failed for device %d\n", + __func__, p->devices[i]); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } } // Arrival ring: cache-line padded so each GPU's int is on its own line. @@ -393,6 +408,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n // Copy-engine path: pinned host staging + device scratch, sized for the // largest tensor we accept on this path (GGML_CUDA_AR_COPY_MAX_BYTES). + // dev_tmp is single-buffered; cross-AR safety is enforced by an explicit + // cross-stream wait in copy_impl on the prior AR's add_kernel-done event. for (int i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); if (cudaHostAlloc(&p->host_large[i], p->copy_bytes, cudaHostAllocPortable) != cudaSuccess) { @@ -447,6 +464,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { for (int c = 0; c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { if (p->ev_pool[i][s].cpy[c]) { cudaEventDestroy(p->ev_pool[i][s].cpy[c]); } } + if (p->ev_pool[i][s].h2d) { cudaEventDestroy(p->ev_pool[i][s].h2d); } if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } } delete[] p->ev_pool[i]; @@ -455,6 +473,10 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { ggml_cuda_set_device(p->devices[i]); cudaEventDestroy(p->host_large_read_done[i]); } + if (p->dev_tmp_kernel_done[i]) { + ggml_cuda_set_device(p->devices[i]); + cudaEventDestroy(p->dev_tmp_kernel_done[i]); + } if (p->streams[i]) { ggml_cuda_set_device(p->devices[i]); cudaStreamDestroy(p->streams[i]); @@ -529,11 +551,23 @@ static bool ggml_cuda_ar_allreduce_copy_impl( } // Stage 2: each GPU waits for each peer D2H chunk, pulls that chunk back to - // local scratch, then performs one device-local add over the assembled peer tensor. + // local device scratch (dev_tmp), then performs one device-local add over + // the assembled peer tensor. The H2Ds run on the AR stream (copy engine) + // and the add_kernel runs on the caller's compute stream, so the AR stream + // stays pure-copy and avoids an in-stream copy→compute engine switch every + // AR. dev_tmp is single-buffered: the AR stream waits cross-stream on the + // prior AR's add_kernel-done event before overwriting it. for (int i = 0; i < 2; ++i) { const int peer = 1 - i; ggml_cuda_set_device(p->devices[i]); + // Wait for the previous AR's add_kernel (on the compute stream) to + // finish reading dev_tmp before our H2D overwrites it. No-op on the + // first copy_impl call. + if (p->dev_tmp_kernel_done_valid) { + CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->dev_tmp_kernel_done[i])); + } + for (size_t c = 0; c < copy_chunks; ++c) { const size_t offset = c * p->copy_chunk_bytes; const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? @@ -546,24 +580,33 @@ static bool ggml_cuda_ar_allreduce_copy_impl( } // Mark our reads of host_large[peer] complete so peer's next AR can - // safely overwrite it. Same-device record (event[i] on stream[i]); - // peer waits cross-device on event[i] before its next stage 1 D2H. + // safely overwrite it. CUDA_CHECK(cudaEventRecord(p->host_large_read_done[i], p->streams[i])); + // Hand off from AR stream (copy engine) to compute stream: compute + // stream waits for all H2Ds to finish, then runs the add_kernel. + CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].h2d, p->streams[i])); + CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx[i]->stream(), p->ev_pool[i][slot].h2d)); + const int block_size = 256; int n_blocks = (int) ((ne + block_size - 1) / block_size); if (n_blocks > 1024) { n_blocks = 1024; } - ggml_cuda_ar_add_kernel<<streams[i]>>>( + ggml_cuda_ar_add_kernel<<stream()>>>( dst_buf[i], reinterpret_cast(p->dev_tmp[i]), (int) ne); CUDA_CHECK(cudaGetLastError()); - ggml_cuda_ar_record_chunk_done(p, cuda_ctx[i], i, slot); + // Record dev_tmp-released on the compute stream so the next copy_impl + // can wait for the kernel to finish before overwriting dev_tmp. Also + // record AR-done as ev.ker for acquire_slot's pool-wraparound sync. + CUDA_CHECK(cudaEventRecord(p->dev_tmp_kernel_done[i], cuda_ctx[i]->stream())); + CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].ker, cuda_ctx[i]->stream())); } p->host_large_read_done_valid = true; + p->dev_tmp_kernel_done_valid = true; return true; } From be783f9eff5d39fe3ed700eda94913b12038c651 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 28 Apr 2026 20:17:13 -0700 Subject: [PATCH 53/81] change ev_pool to fixed 2D array --- ggml/src/ggml-cuda/allreduce.cu | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 81fb62374f5..eb72c26b11c 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -239,7 +239,7 @@ struct ggml_cuda_ar_pipeline { char * host_large[GGML_CUDA_MAX_DEVICES]; // pinned staging for copy-engine path char * dev_tmp[GGML_CUDA_MAX_DEVICES]; // device scratch for copy-engine path cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking - ggml_cuda_ar_event_slot *ev_pool[GGML_CUDA_MAX_DEVICES]; // [device][slot] + ggml_cuda_ar_event_slot ev_pool[GGML_CUDA_MAX_DEVICES][GGML_CUDA_AR_POOL_SIZE]; // Copy-engine: per-device "I finished reading my peer's host_large" // event. Indexed by RECORDER device. Recorded same-device on streams[i] @@ -349,7 +349,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } p->streams[i] = stream; - p->ev_pool[i] = new ggml_cuda_ar_event_slot[GGML_CUDA_AR_POOL_SIZE](); for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { bool ok = cudaEventCreateWithFlags(&p->ev_pool[i][s].app, cudaEventDisableTiming) == cudaSuccess && @@ -457,17 +456,14 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { ggml_cuda_set_device(p->devices[i]); cudaFree(p->dev_tmp[i]); } - if (p->ev_pool[i]) { - ggml_cuda_set_device(p->devices[i]); - for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { - if (p->ev_pool[i][s].app) { cudaEventDestroy(p->ev_pool[i][s].app); } - for (int c = 0; c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { - if (p->ev_pool[i][s].cpy[c]) { cudaEventDestroy(p->ev_pool[i][s].cpy[c]); } - } - if (p->ev_pool[i][s].h2d) { cudaEventDestroy(p->ev_pool[i][s].h2d); } - if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } + ggml_cuda_set_device(p->devices[i]); + for (int s = 0; s < GGML_CUDA_AR_POOL_SIZE; ++s) { + if (p->ev_pool[i][s].app) { cudaEventDestroy(p->ev_pool[i][s].app); } + for (int c = 0; c < GGML_CUDA_AR_COPY_MAX_CHUNKS; ++c) { + if (p->ev_pool[i][s].cpy[c]) { cudaEventDestroy(p->ev_pool[i][s].cpy[c]); } } - delete[] p->ev_pool[i]; + if (p->ev_pool[i][s].h2d) { cudaEventDestroy(p->ev_pool[i][s].h2d); } + if (p->ev_pool[i][s].ker) { cudaEventDestroy(p->ev_pool[i][s].ker); } } if (p->host_large_read_done[i]) { ggml_cuda_set_device(p->devices[i]); From 56a87ee9b54dbb3b6755672de34e79a2b3b6aa80 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 29 Apr 2026 17:49:33 -0700 Subject: [PATCH 54/81] add chunked memcpy fallback for extra-large reductions (>32 MB) --- ggml/src/ggml-cuda/allreduce.cu | 76 +++++++++++++++++++++++---------- 1 file changed, 53 insertions(+), 23 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index eb72c26b11c..fb8acdc1819 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -2,6 +2,7 @@ #include "convert.cuh" #include "ggml-impl.h" +#include #include #include #include @@ -28,6 +29,8 @@ // synchronization happens *outside the kernel*, via CUDA events // between streams. This keeps the compute engine free while large // transfers are in flight, which matters for prefill-sized tensors. +// Reductions larger than the per-call inner cap are processed by an +// outer chunker that issues sequential inner calls. // --------------------------------------------------------------------------- // --------------------------------------------------------------------------- @@ -304,13 +307,6 @@ static void ggml_cuda_ar_wait_for_compute( CUDA_CHECK(cudaStreamWaitEvent(p->streams[rank], ev.app)); } -static void ggml_cuda_ar_record_chunk_done( - ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot) { - ggml_cuda_ar_event_slot & ev = p->ev_pool[rank][slot]; - CUDA_CHECK(cudaEventRecord(ev.ker, p->streams[rank])); - CUDA_CHECK(cudaStreamWaitEvent(cuda_ctx->stream(), ev.ker)); -} - // --------------------------------------------------------------------------- // Init / free // --------------------------------------------------------------------------- @@ -607,6 +603,40 @@ static bool ggml_cuda_ar_allreduce_copy_impl( return true; } +// Outer-level chunker: copy_impl handles up to copy_bytes per call (limited by +// the host_large / dev_tmp allocation size). When the full AR exceeds that, +// slice the tensor into copy_bytes-sized pieces and call copy_impl repeatedly. +// Each slice goes through its own stage 1 → stage 2 cycle and acquires its own +// slot, so cross-AR fences and pool wraparound work the same way as for any +// other sequence of small ARs. +template +static bool ggml_cuda_ar_allreduce_copy_outer( + ggml_cuda_ar_pipeline * p, + ggml_backend_t * backends, + Tsrc * const src_buf[GGML_CUDA_MAX_DEVICES], + Tdst * const dst_buf[GGML_CUDA_MAX_DEVICES], + const bool compute[GGML_CUDA_MAX_DEVICES], + int64_t ne) { + const int64_t outer_max_elems = (int64_t) (p->copy_bytes / sizeof(Tsrc)); + GGML_ASSERT(outer_max_elems > 0); + + bool ok = true; + for (int64_t outer_start = 0; outer_start < ne && ok; outer_start += outer_max_elems) { + const int64_t outer_ne = std::min(outer_max_elems, ne - outer_start); + const size_t outer_nbytes = (size_t) outer_ne * sizeof(Tsrc); + + Tsrc * src[GGML_CUDA_MAX_DEVICES]; + Tdst * dst[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < p->n_devices; ++i) { + src[i] = src_buf[i] + outer_start; + dst[i] = dst_buf[i] + outer_start; + } + ok = ggml_cuda_ar_allreduce_copy_impl( + p, backends, src, dst, compute, outer_ne, outer_nbytes); + } + return ok; +} + bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, @@ -644,11 +674,11 @@ bool ggml_cuda_ar_allreduce( } // Decide between copy-engine and chunked-kernel paths based on the working - // type's actual byte count. + // type's actual byte count. No upper bound: copy_outer slices reductions + // larger than copy_bytes into copy_bytes-sized pieces. const bool use_copy_engine = p->copy_threshold > 0 && - nbytes >= p->copy_threshold && - nbytes <= p->copy_bytes; + nbytes >= p->copy_threshold; // BF16 inactive-shard zeroing: when use_bf16 is on, the combined kernel // (chunked-kernel path) and the combined add kernel (copy_engine path) @@ -710,29 +740,29 @@ bool ggml_cuda_ar_allreduce( src[i] = static_cast<__nv_bfloat16 *>(copy_src_ptr[i]); dst[i] = static_cast(tensors[i]->data); } - ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, float>( - p, backends, src, dst, inner_compute, ne, nbytes); + ok = ggml_cuda_ar_allreduce_copy_outer<__nv_bfloat16, float>( + p, backends, src, dst, inner_compute, ne); } else { switch (kernel_type) { case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); - ok = ggml_cuda_ar_allreduce_copy_impl( - p, backends, buf, buf, inner_compute, ne, nbytes); + ok = ggml_cuda_ar_allreduce_copy_outer( + p, backends, buf, buf, inner_compute, ne); break; } case GGML_TYPE_BF16: { __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(tensors[i]->data); - ok = ggml_cuda_ar_allreduce_copy_impl<__nv_bfloat16, __nv_bfloat16>( - p, backends, buf, buf, inner_compute, ne, nbytes); + ok = ggml_cuda_ar_allreduce_copy_outer<__nv_bfloat16, __nv_bfloat16>( + p, backends, buf, buf, inner_compute, ne); break; } case GGML_TYPE_F16: { half * buf[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); - ok = ggml_cuda_ar_allreduce_copy_impl( - p, backends, buf, buf, inner_compute, ne, nbytes); + ok = ggml_cuda_ar_allreduce_copy_outer( + p, backends, buf, buf, inner_compute, ne); break; } default: @@ -746,11 +776,11 @@ bool ggml_cuda_ar_allreduce( const size_t input_type_size = ggml_type_size(input_type); // Chunked-kernel path runs entirely on the caller's compute stream: - // since AR is a barrier here, same-stream ordering replaces the - // wait_for_compute / record_chunk_done event pairs and skips the - // cross-stream scheduling overhead that was hurting the small-tensor - // (tg) latency on the AR-stream variant. Only ev.ker is still - // recorded at end-of-AR for acquire_slot's pool-wraparound check. + // since AR is a barrier here, same-stream ordering subsumes any + // cross-stream event handshake that the copy-engine path needs, and + // skips the cross-stream scheduling overhead that was hurting the + // small-tensor (tg) latency on the AR-stream variant. Only ev.ker is + // still recorded at end-of-AR for acquire_slot's pool-wraparound check. for (int64_t chunk_start = 0; chunk_start < ne; chunk_start += (int64_t) max_chunk_elems) { const size_t remaining_elems = (size_t) (ne - chunk_start); const size_t chunk_elems = remaining_elems < max_chunk_elems ? remaining_elems : max_chunk_elems; From 59c51b308383e1c36d5756daa5aa7bd8d3567929 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 29 Apr 2026 19:16:44 -0700 Subject: [PATCH 55/81] change thresholds for copy-engine path and bf16 demotion --- ggml/src/ggml-cuda/allreduce.cu | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index fb8acdc1819..7bb14989c53 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -206,7 +206,16 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB // Copy-engine path: largest tensor accepted on this path; sets host_large / // dev_tmp allocation size. static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB -static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB + +// AR wire size at which the copy-engine path beats the chunked-kernel path. +// Empirically determined: Linux dispatch overhead is low enough that even +// 128 KB ARs benefit from the copy engine, while on Windows the crossover +// sits around 1 MB. Override either via GGML_CUDA_AR_COPY_THRESHOLD. +#if defined(__linux__) +static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 128 * 1024; // 128 KB +#else +static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB +#endif static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB // Minimum chunk size the env-var override is allowed to set; this caps the // per-slot copy-event array. 256 KB → up to 128 chunks per 32 MB tensor. @@ -327,7 +336,10 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n __func__, p->copy_chunk_bytes, GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; } - p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 128 * 1024); // 128 KB default + // Default 1: BF16 round-trip is always on for F32 inputs (any non-zero + // ne). Set GGML_CUDA_AR_BF16_THRESHOLD=0 to disable, or to a larger + // byte threshold to opt out for small tensors. + p->bf16_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_BF16_THRESHOLD", 1); for (size_t i = 0; i < n_devices; ++i) { p->devices[i] = devices[i]; } From 28a71d301c36d1bbd47af95500577dfe8bb6917b Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 29 Apr 2026 20:59:52 -0700 Subject: [PATCH 56/81] multi-block kernel test --- ggml/src/ggml-cuda/allreduce.cu | 76 ++++++++++++++++++++------------- 1 file changed, 46 insertions(+), 30 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 7bb14989c53..288e9f66d6a 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -62,6 +62,16 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; } +// Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) +// ensures each GPU/block's arrival slot lives on its own line, preventing +// false-sharing stalls on the polling GPU. +static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; + +// Number of blocks the chunked-kernel launches with. Each block stripes a +// disjoint slice of the data and synchronizes through its own arrival-token +// slot so multiple SMs can pump PCIe stores in parallel. +static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; + // --------------------------------------------------------------------------- // Chunked-kernel AllReduce — 2 GPUs, supports float, half, and bfloat16. // @@ -84,9 +94,12 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { // both GPUs truncate identically — this guarantees // bit-equivalent results across the two devices). // -// The single-block configuration means __syncthreads() is sufficient for -// intra-block coordination and we can use the cheaper non-cooperative launch. -// 256 threads gives good occupancy while keeping register pressure low. +// Multi-block: blocks stripe vectors across (gridDim.x * blockDim.x) global +// threads to keep multiple SMs issuing PCIe stores in parallel. Each block +// has its own arrival-token slot (offset by blockIdx.x * ARRIVAL_STRIDE); +// thread 0 of each block signals/spins on that slot independently of other +// blocks. Tail elements (the leftover < ELEMS_PER_VEC at the end) are +// handled only by block 0 to avoid cross-block writes to the same slots. // --------------------------------------------------------------------------- template static __global__ void ggml_cuda_ar_kernel( @@ -103,15 +116,19 @@ static __global__ void ggml_cuda_ar_kernel( // vector to host memory; each phase-3 iter reads one and produces // ELEMS_PER_VEC sums. constexpr int ELEMS_PER_VEC = 16 / sizeof(Twire); + constexpr int ARRIVAL_INTS = (int)(GGML_CUDA_AR_ARRIVAL_STRIDE / sizeof(int)); const int tid = threadIdx.x; const int nt = blockDim.x; + const int bid = blockIdx.x; + const int gtid = bid * nt + tid; + const int gnt = gridDim.x * nt; const int count_vec = count / ELEMS_PER_VEC; const int tail = count_vec * ELEMS_PER_VEC; // Phase 1: cast sendbuf (Tdst) -> host_mine (Twire) and store as 16-byte vectors. { - for (int i = tid; i < count_vec; i += nt) { + for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; Twire wire[ELEMS_PER_VEC]; #pragma unroll @@ -121,35 +138,39 @@ static __global__ void ggml_cuda_ar_kernel( *reinterpret_cast(&host_mine[off]) = *reinterpret_cast(wire); } - if (tid < count - tail) { + if (bid == 0 && tid < count - tail) { host_mine[tail + tid] = static_cast(sendbuf[tail + tid]); } } - // Commit all host writes before signalling. + // Commit this block's host writes before signalling. __threadfence_system(); __syncthreads(); - // Phase 2: thread 0 signals arrival, then spins for the peer. + // Phase 2: thread 0 of each block signals on its own arrival slot, then + // spins for the matching slot from peer. Per-block tokens mean blocks + // proceed independently — no inter-block barrier needed. if (tid == 0) { - ggml_cuda_ar_signal_set(arrival_mine, token); + int * my_slot = arrival_mine + bid * ARRIVAL_INTS; + const int * other_slot = arrival_other + bid * ARRIVAL_INTS; - __threadfence_system(); // ensure the signal itself is visible across all GPUs + ggml_cuda_ar_signal_set(my_slot, token); + __threadfence_system(); // make our signal visible system-wide - while (ggml_cuda_ar_signal_get(arrival_other) != token) { + while (ggml_cuda_ar_signal_get(other_slot) != token) { __nanosleep(100); } } __syncthreads(); - // Broadcast "peer has arrived" and acquire peer's host_other writes. + // Acquire peer's host_other writes (this block's stripe of them). __threadfence_system(); // Phase 3: read peer's Twire vector, cast both sides through Twire for // bit-equivalence, sum in Tdst precision, and write back to recvbuf. { - for (int i = tid; i < count_vec; i += nt) { + for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; Twire wire[ELEMS_PER_VEC]; *reinterpret_cast(wire) = @@ -160,7 +181,7 @@ static __global__ void ggml_cuda_ar_kernel( recvbuf[off + k] = static_cast(d_low) + static_cast(wire[k]); } } - if (tid < count - tail) { + if (bid == 0 && tid < count - tail) { const Twire d_low = static_cast(sendbuf[tail + tid]); recvbuf[tail + tid] = static_cast(d_low) + static_cast(host_other[tail + tid]); @@ -207,15 +228,10 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB // dev_tmp allocation size. static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB -// AR wire size at which the copy-engine path beats the chunked-kernel path. -// Empirically determined: Linux dispatch overhead is low enough that even -// 128 KB ARs benefit from the copy engine, while on Windows the crossover -// sits around 1 MB. Override either via GGML_CUDA_AR_COPY_THRESHOLD. -#if defined(__linux__) -static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 128 * 1024; // 128 KB -#else -static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB -#endif +// EXPERIMENT: temporarily bumped to 16 MB so the multi-block chunked-kernel +// path handles all sub-16 MB ARs. Revert to the per-platform tuned values +// (Linux 128 KB, Windows 1 MB) when the experiment is done. +static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 16 * 1024 * 1024; // 16 MB static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB // Minimum chunk size the env-var override is allowed to set; this caps the // per-slot copy-event array. 256 KB → up to 128 chunks per 32 MB tensor. @@ -224,11 +240,6 @@ static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = static_cast((GGML_CUDA_AR_COPY_MAX_BYTES + GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN - 1) / GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); -// Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) -// ensures the arrival slots for the two GPUs never share a cache line, -// preventing false-sharing stalls on the polling GPU. -static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; - struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete cudaEvent_t cpy[GGML_CUDA_AR_COPY_MAX_CHUNKS] = {}; // copy-engine D2H chunks complete @@ -273,8 +284,12 @@ struct ggml_cuda_ar_pipeline { }; // Return a pointer to the arrival int for (slot, rank). +// Returns the base pointer for the (slot, rank) per-block token block. The +// kernel adds blockIdx.x * (ARRIVAL_STRIDE/sizeof(int)) internally to land on +// its own slot. static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, int rank) { - const size_t offset = ((size_t)slot * p->n_devices + rank) * GGML_CUDA_AR_ARRIVAL_STRIDE; + const size_t offset = ((size_t)slot * p->n_devices + rank) * + GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; return reinterpret_cast(p->arrival + offset); } @@ -389,7 +404,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n // Arrival ring: cache-line padded so each GPU's int is on its own line. const size_t arrival_bytes = - (size_t)GGML_CUDA_AR_POOL_SIZE * n_devices * GGML_CUDA_AR_ARRIVAL_STRIDE; + (size_t)GGML_CUDA_AR_POOL_SIZE * n_devices * + GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; if (cudaHostAlloc(reinterpret_cast(&p->arrival), arrival_bytes, cudaHostAllocPortable) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for arrival ring failed (%zu bytes)\n", @@ -817,7 +833,7 @@ bool ggml_cuda_ar_allreduce( } #define LAUNCH_AR_KERNEL(Tdst, Twire) \ - ggml_cuda_ar_kernel<<>>( \ + ggml_cuda_ar_kernel<<>>( \ reinterpret_cast(data), \ reinterpret_cast(data), \ reinterpret_cast(p->host_buf[i] + (size_t) slot * p->buf_bytes), \ From b35a0934ea1ce81d841d2f2ebd4a0d89df47d10b Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 29 Apr 2026 22:16:53 -0700 Subject: [PATCH 57/81] more fine-tuning for chukn-size, etc. --- ggml/src/ggml-cuda/allreduce.cu | 57 ++++++++++++++++++++++----------- 1 file changed, 39 insertions(+), 18 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 288e9f66d6a..d135a2b86cb 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -228,12 +228,17 @@ static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB // dev_tmp allocation size. static constexpr size_t GGML_CUDA_AR_COPY_MAX_BYTES = 32 * 1024 * 1024; // 32 MB -// EXPERIMENT: temporarily bumped to 16 MB so the multi-block chunked-kernel -// path handles all sub-16 MB ARs. Revert to the per-platform tuned values -// (Linux 128 KB, Windows 1 MB) when the experiment is done. -static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 16 * 1024 * 1024; // 16 MB -static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT = 2 * 1024 * 1024; // 2 MB -// Minimum chunk size the env-var override is allowed to set; this caps the +// AR wire size at which the copy-engine path takes over from the chunked- +// kernel path. Override via GGML_CUDA_AR_COPY_THRESHOLD. +static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 MB +// Per-call CE chunk-size heuristic: chunk_bytes = clamp(nbytes / 4, MIN, MAX). +// The /4 keeps ~4 chunks in flight at any moment (good D2H/H2D overlap with +// the peer); the clamps cover the cases where nbytes/4 is too small (per- +// memcpy fixed cost dominates) or too large (chunk-level pipelining stalls). +// Env var GGML_CUDA_AR_COPY_CHUNK_BYTES can override with a fixed value. +static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN = 512 * 1024; // 512 KB +static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX = 2 * 1024 * 1024; // 2 MB +// Absolute floor that an env-var override is allowed to set; this caps the // per-slot copy-event array. 256 KB → up to 128 chunks per 32 MB tensor. static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN = 256 * 1024; static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = @@ -324,6 +329,18 @@ static ggml_cuda_ar_slot_info ggml_cuda_ar_acquire_slot(ggml_cuda_ar_pipeline * return { slot, (int) p->call_count }; } +// Per-AR copy-engine chunk size: env-var override if set, else heuristic +// (clamp(nbytes/4, HEURISTIC_MIN, HEURISTIC_MAX)). +static size_t ggml_cuda_ar_chunk_bytes(const ggml_cuda_ar_pipeline * p, size_t nbytes) { + if (p->copy_chunk_bytes > 0) { + return p->copy_chunk_bytes; + } + size_t cb = nbytes / 4; + if (cb < GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN) cb = GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN; + if (cb > GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX) cb = GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX; + return cb; +} + static void ggml_cuda_ar_wait_for_compute( ggml_cuda_ar_pipeline * p, ggml_backend_cuda_context * cuda_ctx, int rank, int slot) { ggml_cuda_ar_event_slot & ev = p->ev_pool[rank][slot]; @@ -345,8 +362,10 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->n_devices = n_devices; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; p->copy_threshold = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_THRESHOLD", GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT); - p->copy_chunk_bytes = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_CHUNK_BYTES", GGML_CUDA_AR_COPY_CHUNK_BYTES_DEFAULT); - if (p->copy_chunk_bytes < GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN) { + // 0 = use the per-call heuristic (default). Non-zero env value forces a + // fixed chunk size for diagnostics, with a floor at COPY_CHUNK_BYTES_MIN. + p->copy_chunk_bytes = ggml_cuda_ar_env_u64("GGML_CUDA_AR_COPY_CHUNK_BYTES", 0); + if (p->copy_chunk_bytes > 0 && p->copy_chunk_bytes < GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN) { GGML_LOG_WARN("%s: GGML_CUDA_AR_COPY_CHUNK_BYTES=%zu below minimum %zu; clamping\n", __func__, p->copy_chunk_bytes, GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN); p->copy_chunk_bytes = GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN; @@ -530,10 +549,12 @@ static bool ggml_cuda_ar_allreduce_copy_impl( GGML_ASSERT(p->n_devices == 2); GGML_ASSERT(nbytes <= p->copy_bytes); GGML_ASSERT(ne <= std::numeric_limits::max()); - GGML_ASSERT(p->copy_chunk_bytes > 0); + + const size_t chunk_bytes = ggml_cuda_ar_chunk_bytes(p, nbytes); + GGML_ASSERT(chunk_bytes > 0); const int slot = ggml_cuda_ar_acquire_slot(p).slot; - const size_t copy_chunks = (nbytes + p->copy_chunk_bytes - 1) / p->copy_chunk_bytes; + const size_t copy_chunks = (nbytes + chunk_bytes - 1) / chunk_bytes; GGML_ASSERT(copy_chunks <= GGML_CUDA_AR_COPY_MAX_CHUNKS); ggml_backend_cuda_context * cuda_ctx[2] = {}; @@ -559,12 +580,12 @@ static bool ggml_cuda_ar_allreduce_copy_impl( } for (size_t c = 0; c < copy_chunks; ++c) { - const size_t offset = c * p->copy_chunk_bytes; - const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? - (nbytes - offset) : p->copy_chunk_bytes; + const size_t offset = c * chunk_bytes; + const size_t this_bytes = (nbytes - offset) < chunk_bytes ? + (nbytes - offset) : chunk_bytes; CUDA_CHECK(cudaMemcpyAsync( - p->host_large[i] + offset, reinterpret_cast(src_buf[i]) + offset, chunk_bytes, + p->host_large[i] + offset, reinterpret_cast(src_buf[i]) + offset, this_bytes, cudaMemcpyDeviceToHost, p->streams[i])); CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); } @@ -589,13 +610,13 @@ static bool ggml_cuda_ar_allreduce_copy_impl( } for (size_t c = 0; c < copy_chunks; ++c) { - const size_t offset = c * p->copy_chunk_bytes; - const size_t chunk_bytes = (nbytes - offset) < p->copy_chunk_bytes ? - (nbytes - offset) : p->copy_chunk_bytes; + const size_t offset = c * chunk_bytes; + const size_t this_bytes = (nbytes - offset) < chunk_bytes ? + (nbytes - offset) : chunk_bytes; CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy[c])); CUDA_CHECK(cudaMemcpyAsync( - p->dev_tmp[i] + offset, p->host_large[peer] + offset, chunk_bytes, + p->dev_tmp[i] + offset, p->host_large[peer] + offset, this_bytes, cudaMemcpyHostToDevice, p->streams[i])); } From 5d3df69bcda21bf9ccbf78058839bae9627b01e8 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 30 Apr 2026 16:46:08 -0700 Subject: [PATCH 58/81] various fixes for PR review --- ggml/src/ggml-cuda/allreduce.cu | 37 +++-- ggml/src/ggml-cuda/ggml-cuda.cu | 237 +++++++++++++++----------------- 2 files changed, 138 insertions(+), 136 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index d135a2b86cb..7d9590471bd 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -283,9 +283,14 @@ struct ggml_cuda_ar_pipeline { cudaEvent_t dev_tmp_kernel_done[GGML_CUDA_MAX_DEVICES]; bool dev_tmp_kernel_done_valid; - // Arrival ring: pinned, ARRIVAL_STRIDE bytes between adjacent ints. + // Arrival ring: ARRIVAL_STRIDE bytes between adjacent ints. Allocated as + // mapped pinned host memory so the device can read/write it directly. + // arrival_host is the cudaFreeHost handle; arrival_dev is the device-side + // pointer (from cudaHostGetDevicePointer) that the kernel and cudaMemset + // operate on. The CPU never touches either pointer's data. // Use ggml_cuda_ar_arrival_ptr() to index. - char * arrival; + char * arrival_host; + char * arrival_dev; }; // Return a pointer to the arrival int for (slot, rank). @@ -295,7 +300,7 @@ struct ggml_cuda_ar_pipeline { static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, int rank) { const size_t offset = ((size_t)slot * p->n_devices + rank) * GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; - return reinterpret_cast(p->arrival + offset); + return reinterpret_cast(p->arrival_dev + offset); } static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) { @@ -355,6 +360,8 @@ static void ggml_cuda_ar_wait_for_compute( ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n_devices) { if (n_devices != 2) { + GGML_LOG_DEBUG("%s: internal AllReduce only supports n_devices=2 (got %zu); " + "falling back\n", __func__, n_devices); return nullptr; } @@ -425,14 +432,28 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n const size_t arrival_bytes = (size_t)GGML_CUDA_AR_POOL_SIZE * n_devices * GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; - if (cudaHostAlloc(reinterpret_cast(&p->arrival), arrival_bytes, - cudaHostAllocPortable) != cudaSuccess) { + // Mapped + portable so cudaHostGetDevicePointer gives us a kernel-usable + // device pointer on every device — the CPU never touches the buffer. + if (cudaHostAlloc(reinterpret_cast(&p->arrival_host), arrival_bytes, + cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for arrival ring failed (%zu bytes)\n", __func__, arrival_bytes); ggml_cuda_ar_pipeline_free(p); return nullptr; } - memset(p->arrival, 0, arrival_bytes); + if (cudaHostGetDevicePointer(reinterpret_cast(&p->arrival_dev), + p->arrival_host, 0) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaHostGetDevicePointer for arrival ring failed\n", __func__); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } + ggml_cuda_set_device(p->devices[0]); + if (cudaMemset(p->arrival_dev, 0, arrival_bytes) != cudaSuccess) { + GGML_LOG_ERROR("%s: cudaMemset for arrival ring failed (%zu bytes)\n", + __func__, arrival_bytes); + ggml_cuda_ar_pipeline_free(p); + return nullptr; + } // Per-device pinned staging buffers — POOL_SIZE-deep ring so the chunked- // kernel can write the next slot's data while the peer is still reading @@ -521,8 +542,8 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaStreamDestroy(p->streams[i]); } } - if (p->arrival) { - cudaFreeHost(p->arrival); + if (p->arrival_host) { + cudaFreeHost(p->arrival_host); } delete p; } diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 93308bcfbe0..53b5c5bca93 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1146,14 +1146,17 @@ static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_inte // first use so NCCL's init/runtime quirks don't interfere with configurations // that never hit the fallback path. struct ggml_backend_cuda_comm_context { + using try_allreduce_fn = bool(*)(ggml_backend_cuda_comm_context *, struct ggml_tensor **); + std::vector backends; std::vector dev_ids; ggml_cuda_ar_pipeline * ar_pipeline = nullptr; - // NCCL is eligible when GGML_USE_NCCL is defined and the user did not - // force GGML_CUDA_ALLREDUCE=internal. Comms are initialised on first use. - bool nccl_eligible = false; + // Provider chosen at init time from GGML_CUDA_ALLREDUCE; called directly + // by the dispatch. One of try_allreduce_{internal,nccl,none,os}. + try_allreduce_fn try_allreduce = nullptr; + #ifdef GGML_USE_NCCL std::once_flag nccl_init_flag; bool nccl_init_ok = false; @@ -1170,85 +1173,6 @@ struct ggml_backend_cuda_comm_context { } }; -static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { - if (comm_ctx_v == nullptr) { - return; - } - delete static_cast(comm_ctx_v); -} - -// Create the comm context. -// -// GGML_CUDA_ALLREDUCE selects which provider(s) to enable: -// unset — try internal first, fall back to NCCL if compiled in, -// then to the meta-backend butterfly reduction. -// internal — internal only; on unsupported/failure, fall back to butterfly. -// nccl — NCCL only; on unsupported/failure, fall back to butterfly. -// none — skip the CUDA AllReduce entirely; always use butterfly. -// Returns nullptr when no CUDA provider is available, which causes the meta -// backend to run its generic butterfly reduction. -static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { - for (size_t i = 0; i < n_backends; i++) { - if (!ggml_backend_is_cuda(backends[i])) { - return nullptr; - } - } - - const char * env = getenv("GGML_CUDA_ALLREDUCE"); - const bool force_none = env && strcmp(env, "none") == 0; - const bool force_internal = env && strcmp(env, "internal") == 0; - const bool force_nccl = env && strcmp(env, "nccl") == 0; - if (env && *env && !force_none && !force_internal && !force_nccl) { - GGML_LOG_WARN("%s: unknown GGML_CUDA_ALLREDUCE value '%s', using default\n", __func__, env); - } - - if (force_none) { - GGML_LOG_INFO("%s: GGML_CUDA_ALLREDUCE=none; using meta-backend butterfly reduction\n", __func__); - return nullptr; - } - -#ifndef GGML_USE_NCCL - if (force_nccl) { - GGML_LOG_WARN("%s: GGML_CUDA_ALLREDUCE=nccl requested but NCCL not compiled in; using meta-backend butterfly reduction\n", __func__); - return nullptr; - } -#endif - - auto * ret = new ggml_backend_cuda_comm_context; - ret->backends.assign(backends, backends + n_backends); - ret->dev_ids.reserve(n_backends); - for (size_t i = 0; i < n_backends; i++) { - ret->dev_ids.push_back(static_cast(backends[i]->context)->device); - } - - // Try to allocate the internal pipeline unless the user forced NCCL. - if (!force_nccl) { - ret->ar_pipeline = ggml_cuda_ar_pipeline_init(ret->dev_ids.data(), n_backends); - if (ret->ar_pipeline == nullptr) { - // Clear any sticky CUDA error from the failed init so it can't - // leak into a later NCCL call. - (void) cudaGetLastError(); - if (force_internal) { - GGML_LOG_ERROR("%s: internal AllReduce pipeline init failed; falling back to butterfly\n", __func__); - } - } - } - -#ifdef GGML_USE_NCCL - ret->nccl_eligible = !force_internal; -#else - ret->nccl_eligible = false; -#endif - - // If nothing is usable, return nullptr so the meta backend uses butterfly. - if (ret->ar_pipeline == nullptr && !ret->nccl_eligible) { - delete ret; - return nullptr; - } - - return ret; -} - #ifdef GGML_USE_NCCL // AllReduce via NCCL. Reduces as FP32 for small tensors and BF16 for large // tensors (bandwidth-bound), then converts back to FP32. @@ -1326,71 +1250,65 @@ static bool ggml_backend_cuda_comm_allreduce_nccl( } #endif // GGML_USE_NCCL -enum ggml_cuda_comm_allreduce_result { - GGML_CUDA_COMM_ALLREDUCE_SUCCESS, - GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED, - GGML_CUDA_COMM_ALLREDUCE_FAILED, -}; - -static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_internal( +// Try the internal AllReduce. Returns true on success. Returns false when +// the pipeline is unavailable or the input is unsupported, so the caller +// falls through to the next provider. Tensor-shape errors are logged but +// still return false (the meta-backend butterfly will catch them). +static bool ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { if (comm_ctx->ar_pipeline == nullptr) { - GGML_LOG_DEBUG("%s: internal unsupported: pipeline unavailable\n", __func__); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } const size_t n_backends = comm_ctx->backends.size(); GGML_ASSERT(n_backends >= 1); GGML_ASSERT(tensors[0] != nullptr); - const int64_t ne = ggml_nelements(tensors[0]); + const int64_t ne = ggml_nelements(tensors[0]); const ggml_type type = tensors[0]->type; if (n_backends != 2) { GGML_LOG_DEBUG("%s: internal unsupported: n_backends=%zu\n", __func__, n_backends); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } - if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { GGML_LOG_DEBUG("%s: internal unsupported: type=%d\n", __func__, (int) type); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } if (ne == 0) { - return GGML_CUDA_COMM_ALLREDUCE_SUCCESS; + return true; } for (size_t i = 0; i < n_backends; ++i) { if (tensors[i] == nullptr) { GGML_LOG_ERROR("%s: internal failed: tensor[%zu] is null\n", __func__, i); - return GGML_CUDA_COMM_ALLREDUCE_FAILED; + return false; } if (ggml_nelements(tensors[i]) != ne || tensors[i]->type != type) { GGML_LOG_ERROR("%s: internal failed: tensor[%zu] ne=%" PRId64 " type=%d expected ne=%" PRId64 " type=%d\n", __func__, i, ggml_nelements(tensors[i]), (int) tensors[i]->type, ne, (int) type); - return GGML_CUDA_COMM_ALLREDUCE_FAILED; + return false; } if (!ggml_is_contiguously_allocated(tensors[i])) { GGML_LOG_DEBUG("%s: internal unsupported: tensor[%zu] is not contiguously allocated: ne=%" PRId64 " nbytes=%zu packed=%zu type=%d\n", __func__, i, ne, ggml_nbytes(tensors[i]), (size_t) ne * ggml_type_size(type) / ggml_blck_size(type), (int) type); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } if (((uintptr_t) tensors[i]->data & 0xF) != 0) { GGML_LOG_DEBUG("%s: internal unsupported: tensor[%zu] data pointer is not 16-byte aligned: %p type=%d ne=%" PRId64 "\n", __func__, i, tensors[i]->data, (int) type, ne); - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } } - return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors) - ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS - : GGML_CUDA_COMM_ALLREDUCE_FAILED; + return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); } #ifdef GGML_USE_NCCL // Lazily initialise NCCL communicators on first use. -// Returns true when comms are ready; false if init failed (dispatcher should skip NCCL). +// Returns true when comms are ready; false if init failed. static bool ggml_backend_cuda_comm_ensure_nccl(ggml_backend_cuda_comm_context * comm_ctx) { std::call_once(comm_ctx->nccl_init_flag, [&] { const size_t n = comm_ctx->dev_ids.size(); @@ -1406,44 +1324,107 @@ static bool ggml_backend_cuda_comm_ensure_nccl(ggml_backend_cuda_comm_context * return comm_ctx->nccl_init_ok; } -static ggml_cuda_comm_allreduce_result ggml_backend_cuda_comm_try_allreduce_nccl( +static bool ggml_backend_cuda_comm_try_allreduce_nccl( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { if (!ggml_backend_cuda_comm_ensure_nccl(comm_ctx)) { - return GGML_CUDA_COMM_ALLREDUCE_UNSUPPORTED; + return false; } - return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors) - ? GGML_CUDA_COMM_ALLREDUCE_SUCCESS - : GGML_CUDA_COMM_ALLREDUCE_FAILED; + return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); +} +#else +static bool ggml_backend_cuda_comm_try_allreduce_nccl( + ggml_backend_cuda_comm_context *, struct ggml_tensor **) { + return false; } #endif -// Dispatch order is fixed: internal first (if allocated), then NCCL (if -// eligible, lazily initialised on first call). If neither handles the tensor, -// return false so the meta backend runs its butterfly reduction. -static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { +// Platform-tuned default order. Each helper returns false if its provider is +// unavailable, so the short-circuit OR naturally falls through to the next. +#if defined(__linux__) +static bool ggml_backend_cuda_comm_try_allreduce_os( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + return ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors) + || ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); +} +#else +static bool ggml_backend_cuda_comm_try_allreduce_os( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + return ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors) + || ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); +} +#endif + +// "GGML_CUDA_ALLREDUCE=none": skip CUDA providers, fall back to butterfly. +static bool ggml_backend_cuda_comm_try_allreduce_none( + ggml_backend_cuda_comm_context *, struct ggml_tensor **) { + return false; +} + +static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { if (comm_ctx_v == nullptr) { - return false; + return; } - auto * comm_ctx = static_cast(comm_ctx_v); + delete static_cast(comm_ctx_v); +} - if (comm_ctx->ar_pipeline != nullptr) { - const ggml_cuda_comm_allreduce_result r = - ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); - if (r == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) return true; - if (r == GGML_CUDA_COMM_ALLREDUCE_FAILED) return false; - // UNSUPPORTED — fall through to NCCL (if eligible). +// Create the comm context. Internal AllReduce is allocated unconditionally +// (warning on failure). GGML_CUDA_ALLREDUCE is read here exactly once and +// used to pick the per-call provider function; runtime fallback to butterfly +// happens naturally if the chosen provider can't serve the call. +static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { + for (size_t i = 0; i < n_backends; i++) { + if (!ggml_backend_is_cuda(backends[i])) { + return nullptr; + } } -#ifdef GGML_USE_NCCL - if (comm_ctx->nccl_eligible) { - const ggml_cuda_comm_allreduce_result r = - ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); - if (r == GGML_CUDA_COMM_ALLREDUCE_SUCCESS) return true; - if (r == GGML_CUDA_COMM_ALLREDUCE_FAILED) return false; + auto * ret = new ggml_backend_cuda_comm_context; + ret->backends.assign(backends, backends + n_backends); + ret->dev_ids.reserve(n_backends); + for (size_t i = 0; i < n_backends; i++) { + ret->dev_ids.push_back(static_cast(backends[i]->context)->device); + } + + ret->ar_pipeline = ggml_cuda_ar_pipeline_init(ret->dev_ids.data(), n_backends); + if (ret->ar_pipeline == nullptr) { + // Clear any sticky CUDA error from the failed init so it can't leak + // into a later NCCL call. + (void) cudaGetLastError(); + GGML_LOG_WARN("%s: internal AllReduce pipeline init failed; " + "falling back to NCCL or butterfly\n", __func__); + } + + // Eager init and warning on targets where NCCL is the preferred provider +#if defined(GGML_USE_NCCL) && !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) && !defined(_WIN32) + if (!ggml_backend_cuda_comm_ensure_nccl(ret)) + { + static bool warning_printed = false; + if (!warning_printed) { + GGML_LOG_WARN("%s: NVIDIA Collective Communications Library (NCCL) is unavailable, " + "multi GPU performance will be suboptimal\n", __func__); + warning_printed = true; + } } #endif - return false; + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + if (env && strcmp(env, "internal") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal; + else if (env && strcmp(env, "nccl") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; + else if (env && strcmp(env, "none") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_none; + else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_os; + + return ret; +} + +// Top-level dispatch. Just calls the function pointer comm_init picked from +// GGML_CUDA_ALLREDUCE. Returns false to fall back to the meta backend's +// butterfly reduction. +static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { + if (comm_ctx_v == nullptr) { + return false; + } + auto * comm_ctx = static_cast(comm_ctx_v); + return comm_ctx->try_allreduce(comm_ctx, tensors); } ggml_backend_buffer_type_t ggml_backend_cuda_split_buffer_type(int main_device, const float * tensor_split) { From bfa0b69b3536492acc65988445f447c1c3f93230 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 30 Apr 2026 17:57:25 -0700 Subject: [PATCH 59/81] more PR fixes --- ggml/src/ggml-cuda/allreduce.cu | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 7d9590471bd..a5274895573 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -62,10 +62,10 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { return *(const volatile int *)p; } -// Byte spacing between adjacent arrival ints. 128 bytes (two cache lines) +// Byte spacing between adjacent arrival ints. 64 bytes (one cache line) // ensures each GPU/block's arrival slot lives on its own line, preventing // false-sharing stalls on the polling GPU. -static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 128; +static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 64; // Number of blocks the chunked-kernel launches with. Each block stripes a // disjoint slice of the data and synchronizes through its own arrival-token @@ -289,7 +289,7 @@ struct ggml_cuda_ar_pipeline { // pointer (from cudaHostGetDevicePointer) that the kernel and cudaMemset // operate on. The CPU never touches either pointer's data. // Use ggml_cuda_ar_arrival_ptr() to index. - char * arrival_host; + void * arrival_host; char * arrival_dev; }; @@ -340,10 +340,8 @@ static size_t ggml_cuda_ar_chunk_bytes(const ggml_cuda_ar_pipeline * p, size_t n if (p->copy_chunk_bytes > 0) { return p->copy_chunk_bytes; } - size_t cb = nbytes / 4; - if (cb < GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN) cb = GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN; - if (cb > GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX) cb = GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX; - return cb; + return std::min(GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX, + std::max(GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN, nbytes / 4)); } static void ggml_cuda_ar_wait_for_compute( @@ -434,7 +432,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; // Mapped + portable so cudaHostGetDevicePointer gives us a kernel-usable // device pointer on every device — the CPU never touches the buffer. - if (cudaHostAlloc(reinterpret_cast(&p->arrival_host), arrival_bytes, + if (cudaHostAlloc(&p->arrival_host, arrival_bytes, cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaHostAlloc for arrival ring failed (%zu bytes)\n", __func__, arrival_bytes); From 0f9bea36ed95b7d76ae90c826a92ff750049eb88 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 30 Apr 2026 18:17:38 -0700 Subject: [PATCH 60/81] fix semantics of all host mappings --- ggml/src/ggml-cuda/allreduce.cu | 95 +++++++++++++++++++-------------- 1 file changed, 55 insertions(+), 40 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index a5274895573..f62676f2339 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -252,6 +252,40 @@ struct ggml_cuda_ar_event_slot { cudaEvent_t ker = nullptr; // AllReduce kernel complete }; +// Mapped pinned host allocation: cudaHostAlloc + cudaHostGetDevicePointer +// in one place, with the host handle preserved for cudaFreeHost. Used where +// the CPU never touches the buffer — only the device reads/writes via the +// mapped device pointer. Required on systems where cudaDevAttrCanUseHost- +// PointerForRegisteredMem is 0 and the host pointer can't be used as a +// device pointer. +struct ggml_cuda_ar_host_mapping { + void * host = nullptr; // cudaFreeHost handle + char * dev = nullptr; // device-side pointer for kernels / cudaMemset / cudaMemcpyAsync + + cudaError_t alloc(size_t bytes) { + cudaError_t rc = cudaHostAlloc(&host, bytes, cudaHostAllocPortable | cudaHostAllocMapped); + if (rc != cudaSuccess) { + host = nullptr; + return rc; + } + rc = cudaHostGetDevicePointer(reinterpret_cast(&dev), host, 0); + if (rc != cudaSuccess) { + cudaFreeHost(host); + host = nullptr; + dev = nullptr; + } + return rc; + } + + void free() { + if (host) { + cudaFreeHost(host); + host = nullptr; + dev = nullptr; + } + } +}; + struct ggml_cuda_ar_pipeline { int n_devices; int devices[GGML_CUDA_MAX_DEVICES]; @@ -263,9 +297,9 @@ struct ggml_cuda_ar_pipeline { uint64_t call_count; // Per-device resources. - char * host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging - char * host_large[GGML_CUDA_MAX_DEVICES]; // pinned staging for copy-engine path - char * dev_tmp[GGML_CUDA_MAX_DEVICES]; // device scratch for copy-engine path + ggml_cuda_ar_host_mapping host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging (chunked-kernel) + ggml_cuda_ar_host_mapping host_large[GGML_CUDA_MAX_DEVICES]; // pinned staging (copy-engine) + char * dev_tmp[GGML_CUDA_MAX_DEVICES]; // device scratch for copy-engine path cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking ggml_cuda_ar_event_slot ev_pool[GGML_CUDA_MAX_DEVICES][GGML_CUDA_AR_POOL_SIZE]; @@ -283,14 +317,10 @@ struct ggml_cuda_ar_pipeline { cudaEvent_t dev_tmp_kernel_done[GGML_CUDA_MAX_DEVICES]; bool dev_tmp_kernel_done_valid; - // Arrival ring: ARRIVAL_STRIDE bytes between adjacent ints. Allocated as - // mapped pinned host memory so the device can read/write it directly. - // arrival_host is the cudaFreeHost handle; arrival_dev is the device-side - // pointer (from cudaHostGetDevicePointer) that the kernel and cudaMemset - // operate on. The CPU never touches either pointer's data. + // Arrival ring: ARRIVAL_STRIDE bytes between adjacent ints. Mapped pinned + // memory; CPU never reads/writes — only the kernel and cudaMemset. // Use ggml_cuda_ar_arrival_ptr() to index. - void * arrival_host; - char * arrival_dev; + ggml_cuda_ar_host_mapping arrival; }; // Return a pointer to the arrival int for (slot, rank). @@ -300,7 +330,7 @@ struct ggml_cuda_ar_pipeline { static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, int rank) { const size_t offset = ((size_t)slot * p->n_devices + rank) * GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; - return reinterpret_cast(p->arrival_dev + offset); + return reinterpret_cast(p->arrival.dev + offset); } static uint64_t ggml_cuda_ar_env_u64(const char * name, uint64_t default_value) { @@ -430,23 +460,14 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n const size_t arrival_bytes = (size_t)GGML_CUDA_AR_POOL_SIZE * n_devices * GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; - // Mapped + portable so cudaHostGetDevicePointer gives us a kernel-usable - // device pointer on every device — the CPU never touches the buffer. - if (cudaHostAlloc(&p->arrival_host, arrival_bytes, - cudaHostAllocPortable | cudaHostAllocMapped) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for arrival ring failed (%zu bytes)\n", + if (p->arrival.alloc(arrival_bytes) != cudaSuccess) { + GGML_LOG_ERROR("%s: alloc for arrival ring failed (%zu bytes)\n", __func__, arrival_bytes); ggml_cuda_ar_pipeline_free(p); return nullptr; } - if (cudaHostGetDevicePointer(reinterpret_cast(&p->arrival_dev), - p->arrival_host, 0) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostGetDevicePointer for arrival ring failed\n", __func__); - ggml_cuda_ar_pipeline_free(p); - return nullptr; - } ggml_cuda_set_device(p->devices[0]); - if (cudaMemset(p->arrival_dev, 0, arrival_bytes) != cudaSuccess) { + if (cudaMemset(p->arrival.dev, 0, arrival_bytes) != cudaSuccess) { GGML_LOG_ERROR("%s: cudaMemset for arrival ring failed (%zu bytes)\n", __func__, arrival_bytes); ggml_cuda_ar_pipeline_free(p); @@ -459,8 +480,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n p->buf_bytes = GGML_CUDA_AR_MAX_BYTES; const size_t host_buf_total = (size_t) GGML_CUDA_AR_POOL_SIZE * p->buf_bytes; for (int i = 0; i < n_devices; ++i) { - if (cudaHostAlloc(&p->host_buf[i], host_buf_total, cudaHostAllocPortable) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for staging failed (%zu bytes)\n", + if (p->host_buf[i].alloc(host_buf_total) != cudaSuccess) { + GGML_LOG_ERROR("%s: alloc for staging failed (%zu bytes)\n", __func__, host_buf_total); ggml_cuda_ar_pipeline_free(p); return nullptr; @@ -473,8 +494,8 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n // cross-stream wait in copy_impl on the prior AR's add_kernel-done event. for (int i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); - if (cudaHostAlloc(&p->host_large[i], p->copy_bytes, cudaHostAllocPortable) != cudaSuccess) { - GGML_LOG_ERROR("%s: cudaHostAlloc for large staging failed (%zu bytes)\n", + if (p->host_large[i].alloc(p->copy_bytes) != cudaSuccess) { + GGML_LOG_ERROR("%s: alloc for large staging failed (%zu bytes)\n", __func__, p->copy_bytes); ggml_cuda_ar_pipeline_free(p); return nullptr; @@ -508,12 +529,8 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { } for (int i = 0; i < p->n_devices; ++i) { - if (p->host_buf[i]) { - cudaFreeHost(p->host_buf[i]); - } - if (p->host_large[i]) { - cudaFreeHost(p->host_large[i]); - } + p->host_buf[i].free(); + p->host_large[i].free(); if (p->dev_tmp[i]) { ggml_cuda_set_device(p->devices[i]); cudaFree(p->dev_tmp[i]); @@ -540,9 +557,7 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { cudaStreamDestroy(p->streams[i]); } } - if (p->arrival_host) { - cudaFreeHost(p->arrival_host); - } + p->arrival.free(); delete p; } @@ -604,7 +619,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( (nbytes - offset) : chunk_bytes; CUDA_CHECK(cudaMemcpyAsync( - p->host_large[i] + offset, reinterpret_cast(src_buf[i]) + offset, this_bytes, + p->host_large[i].dev + offset, reinterpret_cast(src_buf[i]) + offset, this_bytes, cudaMemcpyDeviceToHost, p->streams[i])); CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); } @@ -635,7 +650,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy[c])); CUDA_CHECK(cudaMemcpyAsync( - p->dev_tmp[i] + offset, p->host_large[peer] + offset, this_bytes, + p->dev_tmp[i] + offset, p->host_large[peer].dev + offset, this_bytes, cudaMemcpyHostToDevice, p->streams[i])); } @@ -876,8 +891,8 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_kernel<<>>( \ reinterpret_cast(data), \ reinterpret_cast(data), \ - reinterpret_cast(p->host_buf[i] + (size_t) slot * p->buf_bytes), \ - reinterpret_cast(p->host_buf[peer] + (size_t) slot * p->buf_bytes), \ + reinterpret_cast(p->host_buf[i].dev + (size_t) slot * p->buf_bytes), \ + reinterpret_cast(p->host_buf[peer].dev + (size_t) slot * p->buf_bytes), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ ggml_cuda_ar_arrival_ptr(p, slot, peer), \ From 132934eebbddbca338fc47caf89dc7be19e69229 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 30 Apr 2026 18:26:02 -0700 Subject: [PATCH 61/81] require ampere+ --- ggml/src/ggml-cuda/allreduce.cu | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index f62676f2339..7caef7f5fe5 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -393,6 +393,18 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n return nullptr; } + // Require Ampere or newer on every participating device + const ggml_cuda_device_info & info = ggml_cuda_info(); + for (size_t i = 0; i < n_devices; ++i) { + const int cc = info.devices[devices[i]].cc; + if (cc < GGML_CUDA_CC_AMPERE) { + GGML_LOG_DEBUG("%s: internal AllReduce requires compute capability >= %d " + "(device %d has cc=%d); falling back\n", + __func__, GGML_CUDA_CC_AMPERE, devices[i], cc); + return nullptr; + } + } + auto * p = new ggml_cuda_ar_pipeline{}; p->n_devices = n_devices; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; From 1854ebc075bb4dc5c3cbf7afdf86c2f48c027b26 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 30 Apr 2026 18:41:23 -0700 Subject: [PATCH 62/81] small cleanups --- ggml/src/ggml-cuda/allreduce.cu | 6 ++---- ggml/src/ggml-cuda/ggml-cuda.cu | 7 ++----- 2 files changed, 4 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 7caef7f5fe5..a23637bded7 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -323,10 +323,8 @@ struct ggml_cuda_ar_pipeline { ggml_cuda_ar_host_mapping arrival; }; -// Return a pointer to the arrival int for (slot, rank). -// Returns the base pointer for the (slot, rank) per-block token block. The -// kernel adds blockIdx.x * (ARRIVAL_STRIDE/sizeof(int)) internally to land on -// its own slot. +// Base pointer for the (slot, rank) per-block token block. The kernel adds +// blockIdx.x * (ARRIVAL_STRIDE/sizeof(int)) internally to land on its own slot. static int * ggml_cuda_ar_arrival_ptr(const ggml_cuda_ar_pipeline * p, int slot, int rank) { const size_t offset = ((size_t)slot * p->n_devices + rank) * GGML_CUDA_AR_KERNEL_BLOCKS * GGML_CUDA_AR_ARRIVAL_STRIDE; diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 53b5c5bca93..d83da23c729 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1340,19 +1340,16 @@ static bool ggml_backend_cuda_comm_try_allreduce_nccl( // Platform-tuned default order. Each helper returns false if its provider is // unavailable, so the short-circuit OR naturally falls through to the next. -#if defined(__linux__) static bool ggml_backend_cuda_comm_try_allreduce_os( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { +#if defined(__linux__) return ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors) || ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); -} #else -static bool ggml_backend_cuda_comm_try_allreduce_os( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { return ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors) || ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); -} #endif +} // "GGML_CUDA_ALLREDUCE=none": skip CUDA providers, fall back to butterfly. static bool ggml_backend_cuda_comm_try_allreduce_none( From 6e9319c6a52a4c867b80700bcea8778ef1a9ea3c Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 4 May 2026 15:25:06 -0700 Subject: [PATCH 63/81] properly use host pointer for src/dst in cudaMemcpy calls --- ggml/src/ggml-cuda/allreduce.cu | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index a23637bded7..063b7db3589 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -259,11 +259,12 @@ struct ggml_cuda_ar_event_slot { // PointerForRegisteredMem is 0 and the host pointer can't be used as a // device pointer. struct ggml_cuda_ar_host_mapping { - void * host = nullptr; // cudaFreeHost handle - char * dev = nullptr; // device-side pointer for kernels / cudaMemset / cudaMemcpyAsync + uint8_t * host = nullptr; // cudaFreeHost handle; also the H-side ptr for cudaMemcpyAsync + uint8_t * dev = nullptr; // device-side pointer for kernels / cudaMemset cudaError_t alloc(size_t bytes) { - cudaError_t rc = cudaHostAlloc(&host, bytes, cudaHostAllocPortable | cudaHostAllocMapped); + cudaError_t rc = cudaHostAlloc(reinterpret_cast(&host), bytes, + cudaHostAllocPortable | cudaHostAllocMapped); if (rc != cudaSuccess) { host = nullptr; return rc; @@ -629,7 +630,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( (nbytes - offset) : chunk_bytes; CUDA_CHECK(cudaMemcpyAsync( - p->host_large[i].dev + offset, reinterpret_cast(src_buf[i]) + offset, this_bytes, + p->host_large[i].host + offset, reinterpret_cast(src_buf[i]) + offset, this_bytes, cudaMemcpyDeviceToHost, p->streams[i])); CUDA_CHECK(cudaEventRecord(p->ev_pool[i][slot].cpy[c], p->streams[i])); } @@ -660,7 +661,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->ev_pool[peer][slot].cpy[c])); CUDA_CHECK(cudaMemcpyAsync( - p->dev_tmp[i] + offset, p->host_large[peer].dev + offset, this_bytes, + p->dev_tmp[i] + offset, p->host_large[peer].host + offset, this_bytes, cudaMemcpyHostToDevice, p->streams[i])); } From a19d7113853ee5b3bcc5667134eff1a10826309e Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 4 May 2026 17:51:54 -0700 Subject: [PATCH 64/81] allreduce: lazy-init the internal pipeline on first use A config that lives entirely on NCCL never needs the chunked-kernel pipeline (host_buf, host_large, dev_tmp, streams, events, arrival ring). Defer pipeline creation to the first try_allreduce_internal call using the same std::call_once pattern as ensure_nccl, so those resources stay unallocated when only NCCL is in use. Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/ggml-cuda.cu | 33 ++++++++++++++++++++++++--------- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index d83da23c729..826977e77d6 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1151,6 +1151,10 @@ struct ggml_backend_cuda_comm_context { std::vector backends; std::vector dev_ids; + // Internal AR pipeline. Allocated lazily on first try_allreduce_internal + // call so a config that lives entirely on NCCL never spends VRAM / pinned + // host memory on it. Guarded by ar_pipeline_init_flag (std::call_once). + std::once_flag ar_pipeline_init_flag; ggml_cuda_ar_pipeline * ar_pipeline = nullptr; // Provider chosen at init time from GGML_CUDA_ALLREDUCE; called directly @@ -1250,13 +1254,29 @@ static bool ggml_backend_cuda_comm_allreduce_nccl( } #endif // GGML_USE_NCCL +// Lazily initialise the internal AR pipeline on first use. Returns true +// when the pipeline is ready; false if init failed (e.g. n_devices != 2 or +// pre-Ampere) — caller falls through to the next provider. +static bool ggml_backend_cuda_comm_ensure_internal(ggml_backend_cuda_comm_context * comm_ctx) { + std::call_once(comm_ctx->ar_pipeline_init_flag, [&] { + comm_ctx->ar_pipeline = ggml_cuda_ar_pipeline_init( + comm_ctx->dev_ids.data(), comm_ctx->dev_ids.size()); + if (comm_ctx->ar_pipeline == nullptr) { + // Clear any sticky CUDA error from the failed init so it can't + // leak into a later NCCL call. + (void) cudaGetLastError(); + } + }); + return comm_ctx->ar_pipeline != nullptr; +} + // Try the internal AllReduce. Returns true on success. Returns false when // the pipeline is unavailable or the input is unsupported, so the caller // falls through to the next provider. Tensor-shape errors are logged but // still return false (the meta-backend butterfly will catch them). static bool ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (comm_ctx->ar_pipeline == nullptr) { + if (!ggml_backend_cuda_comm_ensure_internal(comm_ctx)) { return false; } @@ -1382,14 +1402,9 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba ret->dev_ids.push_back(static_cast(backends[i]->context)->device); } - ret->ar_pipeline = ggml_cuda_ar_pipeline_init(ret->dev_ids.data(), n_backends); - if (ret->ar_pipeline == nullptr) { - // Clear any sticky CUDA error from the failed init so it can't leak - // into a later NCCL call. - (void) cudaGetLastError(); - GGML_LOG_WARN("%s: internal AllReduce pipeline init failed; " - "falling back to NCCL or butterfly\n", __func__); - } + // The internal AR pipeline is created on first try_allreduce_internal + // call (see ensure_internal) so a config that lives entirely on NCCL + // never spends resources on it. // Eager init and warning on targets where NCCL is the preferred provider #if defined(GGML_USE_NCCL) && !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) && !defined(_WIN32) From 5d1b22ab3d7ad6c953000f2dc939ef01408c6483 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Mon, 4 May 2026 20:15:22 -0700 Subject: [PATCH 65/81] allreduce: assert n_backends == 2 instead of soft-fallback ar_pipeline_init already requires n_devices == 2 and bails before any AR can get here, so by the time we reach try_allreduce_internal we know we have exactly two backends. Replace the runtime-debug-log fallback with a hard assert. Co-Authored-By: Claude Opus 4.7 (1M context) NCCL is in use. Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/ggml-cuda.cu | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 826977e77d6..e9233a3e963 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1287,10 +1287,7 @@ static bool ggml_backend_cuda_comm_try_allreduce_internal( const int64_t ne = ggml_nelements(tensors[0]); const ggml_type type = tensors[0]->type; - if (n_backends != 2) { - GGML_LOG_DEBUG("%s: internal unsupported: n_backends=%zu\n", __func__, n_backends); - return false; - } + GGML_ASSERT(n_backends == 2); if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { GGML_LOG_DEBUG("%s: internal unsupported: type=%d\n", __func__, (int) type); return false; From 727b141c02fd4c44c7fbf2ead638f50ed66091b9 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 5 May 2026 17:59:20 -0700 Subject: [PATCH 66/81] rework reduction provider selection. internal/nccl is OS dependent; most fallbacks are removed --- ggml/src/ggml-cuda/ggml-cuda.cu | 227 ++++++++++++++++++-------------- 1 file changed, 128 insertions(+), 99 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index e9233a3e963..03a9cf680e6 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -86,6 +86,9 @@ static_assert(sizeof(half) == sizeof(ggml_fp16_t), "wrong fp16 size"); +#define GGML_LOG_WARN_ONCE(str) \ + { static std::once_flag warn_flag; std::call_once(warn_flag, []() { GGML_LOG_WARN(str); }); } + [[noreturn]] void ggml_cuda_error(const char * stmt, const char * func, const char * file, int line, const char * msg) { int id = -1; // in case cudaGetDevice fails @@ -1151,19 +1154,16 @@ struct ggml_backend_cuda_comm_context { std::vector backends; std::vector dev_ids; - // Internal AR pipeline. Allocated lazily on first try_allreduce_internal - // call so a config that lives entirely on NCCL never spends VRAM / pinned - // host memory on it. Guarded by ar_pipeline_init_flag (std::call_once). - std::once_flag ar_pipeline_init_flag; - ggml_cuda_ar_pipeline * ar_pipeline = nullptr; - - // Provider chosen at init time from GGML_CUDA_ALLREDUCE; called directly - // by the dispatch. One of try_allreduce_{internal,nccl,none,os}. + // Set in comm_init to one of try_allreduce_{nccl, internal_strict, + // internal_lenient, butterfly} based on GGML_CUDA_ALLREDUCE and the + // platform. Each variant assumes the resources it needs were + // initialised in comm_init; nccl needs `comms`, both internal variants + // need `ar_pipeline`, butterfly needs nothing. try_allreduce_fn try_allreduce = nullptr; + ggml_cuda_ar_pipeline * ar_pipeline = nullptr; + #ifdef GGML_USE_NCCL - std::once_flag nccl_init_flag; - bool nccl_init_ok = false; std::vector comms; #endif @@ -1254,40 +1254,19 @@ static bool ggml_backend_cuda_comm_allreduce_nccl( } #endif // GGML_USE_NCCL -// Lazily initialise the internal AR pipeline on first use. Returns true -// when the pipeline is ready; false if init failed (e.g. n_devices != 2 or -// pre-Ampere) — caller falls through to the next provider. -static bool ggml_backend_cuda_comm_ensure_internal(ggml_backend_cuda_comm_context * comm_ctx) { - std::call_once(comm_ctx->ar_pipeline_init_flag, [&] { - comm_ctx->ar_pipeline = ggml_cuda_ar_pipeline_init( - comm_ctx->dev_ids.data(), comm_ctx->dev_ids.size()); - if (comm_ctx->ar_pipeline == nullptr) { - // Clear any sticky CUDA error from the failed init so it can't - // leak into a later NCCL call. - (void) cudaGetLastError(); - } - }); - return comm_ctx->ar_pipeline != nullptr; -} - -// Try the internal AllReduce. Returns true on success. Returns false when -// the pipeline is unavailable or the input is unsupported, so the caller -// falls through to the next provider. Tensor-shape errors are logged but -// still return false (the meta-backend butterfly will catch them). -static bool ggml_backend_cuda_comm_try_allreduce_internal( +// Run the internal AR pipeline. Returns false on unsupported / failed input +// — the caller decides whether to abort (env-forced) or fall back silently. +static bool ggml_backend_cuda_comm_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (!ggml_backend_cuda_comm_ensure_internal(comm_ctx)) { - return false; - } + GGML_ASSERT(comm_ctx->ar_pipeline != nullptr); const size_t n_backends = comm_ctx->backends.size(); - GGML_ASSERT(n_backends >= 1); + GGML_ASSERT(n_backends == 2); GGML_ASSERT(tensors[0] != nullptr); const int64_t ne = ggml_nelements(tensors[0]); const ggml_type type = tensors[0]->type; - GGML_ASSERT(n_backends == 2); if (type != GGML_TYPE_F32 && type != GGML_TYPE_F16 && type != GGML_TYPE_BF16) { GGML_LOG_DEBUG("%s: internal unsupported: type=%d\n", __func__, (int) type); return false; @@ -1323,53 +1302,50 @@ static bool ggml_backend_cuda_comm_try_allreduce_internal( return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); } -#ifdef GGML_USE_NCCL -// Lazily initialise NCCL communicators on first use. -// Returns true when comms are ready; false if init failed. -static bool ggml_backend_cuda_comm_ensure_nccl(ggml_backend_cuda_comm_context * comm_ctx) { - std::call_once(comm_ctx->nccl_init_flag, [&] { - const size_t n = comm_ctx->dev_ids.size(); - comm_ctx->comms.resize(n); - ncclResult_t rc = ncclCommInitAll(comm_ctx->comms.data(), (int) n, comm_ctx->dev_ids.data()); - if (rc != ncclSuccess) { - GGML_LOG_ERROR("%s: ncclCommInitAll failed: %s\n", __func__, ncclGetErrorString(rc)); - comm_ctx->comms.clear(); - return; - } - comm_ctx->nccl_init_ok = true; - }); - return comm_ctx->nccl_init_ok; -} +// --------------------------------------------------------------------------- +// try_allreduce variants — one per mode. All assume their required resource +// has already been initialised by comm_init. +// --------------------------------------------------------------------------- +// NCCL-only. Used for env=nccl on any platform AND for the Linux default. +// On NCCL-internal failure, ggml_backend_cuda_comm_allreduce_nccl aborts via +// NCCL_CHECK; we'll only get here on success. static bool ggml_backend_cuda_comm_try_allreduce_nccl( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (!ggml_backend_cuda_comm_ensure_nccl(comm_ctx)) { - return false; - } +#ifdef GGML_USE_NCCL + GGML_ASSERT(!comm_ctx->comms.empty()); return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); -} #else -static bool ggml_backend_cuda_comm_try_allreduce_nccl( - ggml_backend_cuda_comm_context *, struct ggml_tensor **) { - return false; -} + GGML_UNUSED(comm_ctx); GGML_UNUSED(tensors); + GGML_ABORT("try_allreduce_nccl unreachable: built without NCCL"); #endif +} -// Platform-tuned default order. Each helper returns false if its provider is -// unavailable, so the short-circuit OR naturally falls through to the next. -static bool ggml_backend_cuda_comm_try_allreduce_os( +// Internal-only (env=internal). Failure aborts so the user knows their +// requested mode is not viable. +static bool ggml_backend_cuda_comm_try_allreduce_internal_strict( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { -#if defined(__linux__) - return ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors) - || ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors); -#else - return ggml_backend_cuda_comm_try_allreduce_internal(comm_ctx, tensors) - || ggml_backend_cuda_comm_try_allreduce_nccl(comm_ctx, tensors); -#endif + if (ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors)) { + return true; + } + GGML_ABORT("GGML_CUDA_ALLREDUCE=internal: AR call failed (unsupported input). " + "Reset the environment variable to use the platform default."); +} + +// Internal with butterfly fallback. Used for the Windows default — internal +// is preferred but a return-false cleanly hits the meta-backend's butterfly. +static bool ggml_backend_cuda_comm_try_allreduce_internal_lenient( + ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { + if (ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors)) { + return true; + } + GGML_LOG_WARN_ONCE("internal AllReduce call failed; " + "meta-backend butterfly will be used for this and subsequent calls\n"); + return false; } -// "GGML_CUDA_ALLREDUCE=none": skip CUDA providers, fall back to butterfly. -static bool ggml_backend_cuda_comm_try_allreduce_none( +// Butterfly-only (env=none, or after a failed init for non-strict modes). +static bool ggml_backend_cuda_comm_try_allreduce_butterfly( ggml_backend_cuda_comm_context *, struct ggml_tensor **) { return false; } @@ -1381,10 +1357,36 @@ static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { delete static_cast(comm_ctx_v); } -// Create the comm context. Internal AllReduce is allocated unconditionally -// (warning on failure). GGML_CUDA_ALLREDUCE is read here exactly once and -// used to pick the per-call provider function; runtime fallback to butterfly -// happens naturally if the chosen provider can't serve the call. +// Resource initializers — return true on success. + +#ifdef GGML_USE_NCCL +static bool ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ctx) { + const size_t n = ctx->dev_ids.size(); + ctx->comms.resize(n); + ncclResult_t rc = ncclCommInitAll(ctx->comms.data(), (int) n, ctx->dev_ids.data()); + if (rc != ncclSuccess) { + ctx->comms.clear(); + GGML_LOG_ERROR("%s: ncclCommInitAll failed: %s\n", __func__, ncclGetErrorString(rc)); + return false; + } + return true; +} +#endif + +static bool ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context * ctx) { + ctx->ar_pipeline = ggml_cuda_ar_pipeline_init(ctx->dev_ids.data(), ctx->dev_ids.size()); + if (ctx->ar_pipeline == nullptr) { + // Clear sticky CUDA error from the failed init. + (void) cudaGetLastError(); + return false; + } + return true; +} + +// Pick the try_allreduce function pointer based on GGML_CUDA_ALLREDUCE / OS, +// then init the resource that pointer needs (NCCL or internal pipeline). +// Init failure aborts in every case — internal-lenient's "fall back to +// butterfly" applies to per-call failures, not init. static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { for (size_t i = 0; i < n_backends; i++) { if (!ggml_backend_is_cuda(backends[i])) { @@ -1399,35 +1401,62 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba ret->dev_ids.push_back(static_cast(backends[i]->context)->device); } - // The internal AR pipeline is created on first try_allreduce_internal - // call (see ensure_internal) so a config that lives entirely on NCCL - // never spends resources on it. + // 1. Pick the function pointer. + const char * env = getenv("GGML_CUDA_ALLREDUCE"); + const bool env_nccl = env && strcmp(env, "nccl") == 0; + const bool env_internal = env && strcmp(env, "internal") == 0; + const bool env_none = env && strcmp(env, "none") == 0; + + if (env_nccl) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; + else if (env_internal) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal_strict; + else if (env_none) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; +#if defined(_WIN32) + else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal_lenient; +#elif defined(__linux__) + else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; +#else + else GGML_ABORT("no AllReduce default for this platform; set GGML_CUDA_ALLREDUCE explicitly"); +#endif - // Eager init and warning on targets where NCCL is the preferred provider -#if defined(GGML_USE_NCCL) && !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) && !defined(_WIN32) - if (!ggml_backend_cuda_comm_ensure_nccl(ret)) - { - static bool warning_printed = false; - if (!warning_printed) { - GGML_LOG_WARN("%s: NVIDIA Collective Communications Library (NCCL) is unavailable, " - "multi GPU performance will be suboptimal\n", __func__); - warning_printed = true; + // 2. Init the matching resource. Strict modes (env-forced or Linux + // default = NCCL) abort on failure. The Windows-default lenient + // internal mode degrades to butterfly on init failure. Linux + // without NCCL compiled in degrades to butterfly with a warning. + if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_nccl) { +#ifdef GGML_USE_NCCL + if (!ggml_backend_cuda_comm_init_nccl(ret)) { + GGML_ABORT("NCCL init failed. Set GGML_CUDA_ALLREDUCE=internal or =none to bypass."); } - } +#else + if (env_nccl) { + GGML_ABORT("GGML_CUDA_ALLREDUCE=nccl requested but llama.cpp was not built with NCCL. " + "Recompile with -DGGML_CUDA_NCCL=ON or reset the environment variable."); + } + // Linux default with no NCCL compiled: warn and degrade to butterfly. + GGML_LOG_WARN_ONCE("NVIDIA Collective Communications Library (NCCL) is unavailable; " + "multi-GPU performance will be suboptimal. " + "Recompile with -DGGML_CUDA_NCCL=ON for best performance."); + ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; #endif - - const char * env = getenv("GGML_CUDA_ALLREDUCE"); - if (env && strcmp(env, "internal") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal; - else if (env && strcmp(env, "nccl") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; - else if (env && strcmp(env, "none") == 0) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_none; - else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_os; + } else if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_internal_strict) { + if (!ggml_backend_cuda_comm_init_internal(ret)) { + GGML_ABORT("internal AllReduce pipeline init failed (n_devices != 2 or pre-Ampere?). " + "Reset GGML_CUDA_ALLREDUCE to use the platform default."); + } + } else if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_internal_lenient) { + if (!ggml_backend_cuda_comm_init_internal(ret)) { + GGML_LOG_WARN_ONCE("internal AllReduce pipeline init failed (n_devices != 2 or pre-Ampere?); " + "meta-backend butterfly will be used"); + ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; + } + } + // else: butterfly, no init needed. return ret; } -// Top-level dispatch. Just calls the function pointer comm_init picked from -// GGML_CUDA_ALLREDUCE. Returns false to fall back to the meta backend's -// butterfly reduction. +// Top-level dispatch — calls the function pointer chosen by comm_init. +// Returns false to let the meta-backend's butterfly run. static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { if (comm_ctx_v == nullptr) { return false; From e85a2806fe628bf677df85f2c7e6666e806c01cf Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Tue, 5 May 2026 18:12:30 -0700 Subject: [PATCH 67/81] remove unneeded Turing arch check (llama.cpp doesn't even compile pre-Turing anyway) --- ggml/src/ggml-cuda/allreduce.cu | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 063b7db3589..27af6bbd5cc 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -392,18 +392,6 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n return nullptr; } - // Require Ampere or newer on every participating device - const ggml_cuda_device_info & info = ggml_cuda_info(); - for (size_t i = 0; i < n_devices; ++i) { - const int cc = info.devices[devices[i]].cc; - if (cc < GGML_CUDA_CC_AMPERE) { - GGML_LOG_DEBUG("%s: internal AllReduce requires compute capability >= %d " - "(device %d has cc=%d); falling back\n", - __func__, GGML_CUDA_CC_AMPERE, devices[i], cc); - return nullptr; - } - } - auto * p = new ggml_cuda_ar_pipeline{}; p->n_devices = n_devices; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; From cb3e52167273bd9aba08913d3d77fb271e1fce15 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 16:00:39 -0700 Subject: [PATCH 68/81] allreduce: ASCII-only comments and ggml_cuda_cast for value conversions Replace non-ASCII characters in comments (em dashes, right arrows) with ASCII equivalents (--, ->) so the source stays in the ggml/upstream norm. In the kernel-side code, replace static_cast/static_cast with ggml_cuda_cast<...> so the BF16 conversions go through the fast __float2bfloat16 / __bfloat162float intrinsics from convert.cuh. Pure pointer and integer casts stay as static_cast. Also drops two stray garbage tokens that snuck in from earlier merges (a duplicated 'return ok; }' tail in allreduce.cu and a leftover '_reg)' fragment in ggml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 50 ++++++++++++++++---------------- ggml/src/ggml-cuda/allreduce.cuh | 3 +- ggml/src/ggml-cuda/ggml-cuda.cu | 12 ++++---- 3 files changed, 33 insertions(+), 32 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 27af6bbd5cc..bc4ca414120 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -39,7 +39,7 @@ // One int per (slot, rank) pair in pinned host memory. Each AR call writes a // strictly increasing token (= the AR call number) into its own arrival int. // The peer spins until its read of the other's arrival int equals the token -// it expects for this call — a mismatch means the peer hasn't arrived yet. +// it expects for this call -- a mismatch means the peer hasn't arrived yet. // Tokens never repeat over realistic call rates (32-bit int wraps in tens of // days at thousands of ARs/sec), so arrival ints don't need to be reset // between calls; we initialize once at pipeline init and let the values @@ -73,17 +73,17 @@ static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 64; static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // --------------------------------------------------------------------------- -// Chunked-kernel AllReduce — 2 GPUs, supports float, half, and bfloat16. +// Chunked-kernel AllReduce -- 2 GPUs, supports float, half, and bfloat16. // // Both GPUs run this kernel simultaneously on independent streams. sendbuf // and recvbuf live in Tdst (the caller's tensor type); host_mine / host_other -// carry data in Twire (the on-wire type, possibly narrower than Tdst — e.g. +// carry data in Twire (the on-wire type, possibly narrower than Tdst -- e.g. // Tdst=F32 with Twire=BF16 halves the bytes pushed across PCIe). When // Tdst == Twire the casts below are no-ops. // // Each GPU runs three phases: // -// Phase 1 (all threads): cast sendbuf (Tdst) → Twire and store as 16-byte +// Phase 1 (all threads): cast sendbuf (Tdst) -> Twire and store as 16-byte // vectors into host_mine. __threadfence_system() // commits these writes to host memory. // Phase 2 (thread 0): write token to arrival_mine; spin until @@ -91,7 +91,7 @@ static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // Phase 3 (all threads): read 16-byte Twire vectors from host_other, cast // each element to Tdst, and sum with the local // sendbuf value (also rounded through Twire so that -// both GPUs truncate identically — this guarantees +// both GPUs truncate identically -- this guarantees // bit-equivalent results across the two devices). // // Multi-block: blocks stripe vectors across (gridDim.x * blockDim.x) global @@ -133,13 +133,13 @@ static __global__ void ggml_cuda_ar_kernel( Twire wire[ELEMS_PER_VEC]; #pragma unroll for (int k = 0; k < ELEMS_PER_VEC; ++k) { - wire[k] = static_cast(sendbuf[off + k]); + wire[k] = ggml_cuda_cast(sendbuf[off + k]); } *reinterpret_cast(&host_mine[off]) = *reinterpret_cast(wire); } if (bid == 0 && tid < count - tail) { - host_mine[tail + tid] = static_cast(sendbuf[tail + tid]); + host_mine[tail + tid] = ggml_cuda_cast(sendbuf[tail + tid]); } } @@ -149,7 +149,7 @@ static __global__ void ggml_cuda_ar_kernel( // Phase 2: thread 0 of each block signals on its own arrival slot, then // spins for the matching slot from peer. Per-block tokens mean blocks - // proceed independently — no inter-block barrier needed. + // proceed independently -- no inter-block barrier needed. if (tid == 0) { int * my_slot = arrival_mine + bid * ARRIVAL_INTS; const int * other_slot = arrival_other + bid * ARRIVAL_INTS; @@ -177,14 +177,14 @@ static __global__ void ggml_cuda_ar_kernel( *reinterpret_cast(&host_other[off]); #pragma unroll for (int k = 0; k < ELEMS_PER_VEC; ++k) { - const Twire d_low = static_cast(sendbuf[off + k]); - recvbuf[off + k] = static_cast(d_low) + static_cast(wire[k]); + const Twire d_low = ggml_cuda_cast(sendbuf[off + k]); + recvbuf[off + k] = ggml_cuda_cast(d_low) + ggml_cuda_cast(wire[k]); } } if (bid == 0 && tid < count - tail) { - const Twire d_low = static_cast(sendbuf[tail + tid]); + const Twire d_low = ggml_cuda_cast(sendbuf[tail + tid]); recvbuf[tail + tid] = - static_cast(d_low) + static_cast(host_other[tail + tid]); + ggml_cuda_cast(d_low) + ggml_cuda_cast(host_other[tail + tid]); } } } @@ -192,8 +192,8 @@ static __global__ void ggml_cuda_ar_kernel( // Combined load-convert-add kernel. The peer's contribution arrives as Tsrc // (which may be a lower-precision type than Tdst when the BF16 round-trip is // active). For bit-equivalence between the two GPUs, dst is first rounded -// through Tsrc's precision via a static_cast — peer already truncated its own -// value the same way before sending — so both sides perform identical +// through Tsrc's precision via ggml_cuda_cast -- peer already truncated its +// own value the same way before sending -- so both sides perform identical // arithmetic. When Tdst == Tsrc the round-trip cast is a no-op. template static __global__ void ggml_cuda_ar_add_kernel( @@ -203,8 +203,8 @@ static __global__ void ggml_cuda_ar_add_kernel( const int tid = blockIdx.x * blockDim.x + threadIdx.x; const int nt = gridDim.x * blockDim.x; for (int i = tid; i < count; i += nt) { - const Tsrc d_low = static_cast(dst[i]); - dst[i] = static_cast(d_low) + static_cast(src[i]); + const Tsrc d_low = ggml_cuda_cast(dst[i]); + dst[i] = ggml_cuda_cast(d_low) + ggml_cuda_cast(src[i]); } } @@ -214,7 +214,7 @@ static __global__ void ggml_cuda_ar_add_kernel( // Number of slots in the event / arrival ring. Two slots is sufficient: // lockstep guarantees the two GPUs are at most one AR (or chunk) apart, so -// slot[N%2] is always safe to reuse — peer has already consumed slot[N%2] +// slot[N%2] is always safe to reuse -- peer has already consumed slot[N%2] // from AR N-2 by the time we get to AR N. acquire_slot's // cudaEventSynchronize on ev.ker for both devices makes that consumption // explicit before we overwrite host_buf[slot] for the new AR. @@ -239,7 +239,7 @@ static constexpr size_t GGML_CUDA_AR_COPY_THRESHOLD_DEFAULT = 1024 * 1024; // 1 static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MIN = 512 * 1024; // 512 KB static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_HEURISTIC_MAX = 2 * 1024 * 1024; // 2 MB // Absolute floor that an env-var override is allowed to set; this caps the -// per-slot copy-event array. 256 KB → up to 128 chunks per 32 MB tensor. +// per-slot copy-event array. 256 KB -> up to 128 chunks per 32 MB tensor. static constexpr size_t GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN = 256 * 1024; static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = static_cast((GGML_CUDA_AR_COPY_MAX_BYTES + GGML_CUDA_AR_COPY_CHUNK_BYTES_MIN - 1) / @@ -248,13 +248,13 @@ static constexpr int GGML_CUDA_AR_COPY_MAX_CHUNKS = struct ggml_cuda_ar_event_slot { cudaEvent_t app = nullptr; // upstream computation complete cudaEvent_t cpy[GGML_CUDA_AR_COPY_MAX_CHUNKS] = {}; // copy-engine D2H chunks complete - cudaEvent_t h2d = nullptr; // copy-engine H2Ds complete (handoff AR stream → compute stream) + cudaEvent_t h2d = nullptr; // copy-engine H2Ds complete (handoff AR stream -> compute stream) cudaEvent_t ker = nullptr; // AllReduce kernel complete }; // Mapped pinned host allocation: cudaHostAlloc + cudaHostGetDevicePointer // in one place, with the host handle preserved for cudaFreeHost. Used where -// the CPU never touches the buffer — only the device reads/writes via the +// the CPU never touches the buffer -- only the device reads/writes via the // mapped device pointer. Required on systems where cudaDevAttrCanUseHost- // PointerForRegisteredMem is 0 and the host pointer can't be used as a // device pointer. @@ -319,7 +319,7 @@ struct ggml_cuda_ar_pipeline { bool dev_tmp_kernel_done_valid; // Arrival ring: ARRIVAL_STRIDE bytes between adjacent ints. Mapped pinned - // memory; CPU never reads/writes — only the kernel and cudaMemset. + // memory; CPU never reads/writes -- only the kernel and cudaMemset. // Use ggml_cuda_ar_arrival_ptr() to index. ggml_cuda_ar_host_mapping arrival; }; @@ -473,7 +473,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n return nullptr; } - // Per-device pinned staging buffers — POOL_SIZE-deep ring so the chunked- + // Per-device pinned staging buffers -- POOL_SIZE-deep ring so the chunked- // kernel can write the next slot's data while the peer is still reading // the previous slot's. Indexed by (slot * buf_bytes) at the call site. p->buf_bytes = GGML_CUDA_AR_MAX_BYTES; @@ -602,7 +602,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( // Wait for peer's H2D from our host_large[i] (recorded in the // previous AR's stage 2) to complete before we overwrite host_large[i]. // host_large_read_done[peer] = peer finished reading host_large[i]. - // No-op on the first AR — no prior record exists. + // No-op on the first AR -- no prior record exists. if (p->host_large_read_done_valid) { const int peer = 1 - i; CUDA_CHECK(cudaStreamWaitEvent(p->streams[i], p->host_large_read_done[peer])); @@ -628,7 +628,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( // local device scratch (dev_tmp), then performs one device-local add over // the assembled peer tensor. The H2Ds run on the AR stream (copy engine) // and the add_kernel runs on the caller's compute stream, so the AR stream - // stays pure-copy and avoids an in-stream copy→compute engine switch every + // stays pure-copy and avoids an in-stream copy->compute engine switch every // AR. dev_tmp is single-buffered: the AR stream waits cross-stream on the // prior AR's add_kernel-done event before overwriting it. for (int i = 0; i < 2; ++i) { @@ -688,7 +688,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( // Outer-level chunker: copy_impl handles up to copy_bytes per call (limited by // the host_large / dev_tmp allocation size). When the full AR exceeds that, // slice the tensor into copy_bytes-sized pieces and call copy_impl repeatedly. -// Each slice goes through its own stage 1 → stage 2 cycle and acquires its own +// Each slice goes through its own stage 1 -> stage 2 cycle and acquires its own // slot, so cross-AR fences and pool wraparound work the same way as for any // other sequence of small ARs. template diff --git a/ggml/src/ggml-cuda/allreduce.cuh b/ggml/src/ggml-cuda/allreduce.cuh index 1f7f9f41ba8..0f2c9518d5d 100644 --- a/ggml/src/ggml-cuda/allreduce.cuh +++ b/ggml/src/ggml-cuda/allreduce.cuh @@ -5,7 +5,7 @@ #include -// Opaque pipeline context — owns all pinned buffers, streams, and events. +// Opaque pipeline context -- owns all pinned buffers, streams, and events. struct ggml_cuda_ar_pipeline; // Allocate a pipeline for n_devices GPUs. @@ -26,3 +26,4 @@ bool ggml_cuda_ar_allreduce( ggml_cuda_ar_pipeline * pipeline, ggml_backend_t * backends, ggml_tensor ** tensors); + diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 03a9cf680e6..2e2947ecabd 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1255,7 +1255,7 @@ static bool ggml_backend_cuda_comm_allreduce_nccl( #endif // GGML_USE_NCCL // Run the internal AR pipeline. Returns false on unsupported / failed input -// — the caller decides whether to abort (env-forced) or fall back silently. +// -- the caller decides whether to abort (env-forced) or fall back silently. static bool ggml_backend_cuda_comm_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { GGML_ASSERT(comm_ctx->ar_pipeline != nullptr); @@ -1303,7 +1303,7 @@ static bool ggml_backend_cuda_comm_allreduce_internal( } // --------------------------------------------------------------------------- -// try_allreduce variants — one per mode. All assume their required resource +// try_allreduce variants -- one per mode. All assume their required resource // has already been initialised by comm_init. // --------------------------------------------------------------------------- @@ -1332,7 +1332,7 @@ static bool ggml_backend_cuda_comm_try_allreduce_internal_strict( "Reset the environment variable to use the platform default."); } -// Internal with butterfly fallback. Used for the Windows default — internal +// Internal with butterfly fallback. Used for the Windows default -- internal // is preferred but a return-false cleanly hits the meta-backend's butterfly. static bool ggml_backend_cuda_comm_try_allreduce_internal_lenient( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { @@ -1357,7 +1357,7 @@ static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { delete static_cast(comm_ctx_v); } -// Resource initializers — return true on success. +// Resource initializers -- return true on success. #ifdef GGML_USE_NCCL static bool ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ctx) { @@ -1385,7 +1385,7 @@ static bool ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context // Pick the try_allreduce function pointer based on GGML_CUDA_ALLREDUCE / OS, // then init the resource that pointer needs (NCCL or internal pipeline). -// Init failure aborts in every case — internal-lenient's "fall back to +// Init failure aborts in every case -- internal-lenient's "fall back to // butterfly" applies to per-call failures, not init. static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { for (size_t i = 0; i < n_backends; i++) { @@ -1455,7 +1455,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba return ret; } -// Top-level dispatch — calls the function pointer chosen by comm_init. +// Top-level dispatch -- calls the function pointer chosen by comm_init. // Returns false to let the meta-backend's butterfly run. static bool ggml_backend_cuda_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) { if (comm_ctx_v == nullptr) { From d11b178f7dddb49de436ee6d7348ab48e36324ec Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 16:37:45 -0700 Subject: [PATCH 69/81] allreduce: use ggml_cuda_memcpy_1 for the chunked-kernel vector copies The chunked kernel's two 16-byte register<->host transfers (Phase 1 store and Phase 3 load) used reinterpret_cast on both sides. Replace with ggml_cuda_memcpy_1, which is the canonical helper for this pattern and emits the same int4 LD/ST under the hood. Conformance passes; 5x reruns of 70b internal pp512 show 1832-1836 t/s, matching the prior matrix value of 1831 t/s -- no perf change as expected. Co-Authored-By: Claude Opus 4.7 (1M context) ok; }' tail in allreduce.cu and a leftover '_reg)' fragment in ggml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index bc4ca414120..f13a9355347 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -135,8 +135,7 @@ static __global__ void ggml_cuda_ar_kernel( for (int k = 0; k < ELEMS_PER_VEC; ++k) { wire[k] = ggml_cuda_cast(sendbuf[off + k]); } - *reinterpret_cast(&host_mine[off]) = - *reinterpret_cast(wire); + ggml_cuda_memcpy_1(&host_mine[off], wire); } if (bid == 0 && tid < count - tail) { host_mine[tail + tid] = ggml_cuda_cast(sendbuf[tail + tid]); @@ -173,8 +172,7 @@ static __global__ void ggml_cuda_ar_kernel( for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; Twire wire[ELEMS_PER_VEC]; - *reinterpret_cast(wire) = - *reinterpret_cast(&host_other[off]); + ggml_cuda_memcpy_1(wire, &host_other[off]); #pragma unroll for (int k = 0; k < ELEMS_PER_VEC; ++k) { const Twire d_low = ggml_cuda_cast(sendbuf[off + k]); From 7db79ffe9457bb5337ded1b61bc3275b505845f4 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 16:56:41 -0700 Subject: [PATCH 70/81] allreduce: assert cuda_ctx->device matches the pipeline's device Both ggml_cuda_ar_pipeline and ggml_backend_cuda_context carry the device they were created for; if they ever disagree, every cuda call that follows runs on the wrong device. Add GGML_ASSERT at each cuda_ctx retrieval site in the AR path so the misuse fails fast rather than silently corrupting. Also: rename __nv_bfloat16 -> nv_bfloat16 (typedef alias) for consistency with the rest of the file, and tighten one cudaGetLastError check to fire only after the to_bf16 call that can actually fail. Co-Authored-By: Claude Opus 4.7 (1M context) gml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index f13a9355347..37893dc857c 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -594,6 +594,7 @@ static bool ggml_cuda_ar_allreduce_copy_impl( for (int i = 0; i < 2; ++i) { ggml_cuda_set_device(p->devices[i]); cuda_ctx[i] = static_cast(backends[i]->context); + GGML_ASSERT(cuda_ctx[i]->device == p->devices[i]); ggml_cuda_ar_wait_for_compute(p, cuda_ctx[i], i, slot); @@ -768,6 +769,7 @@ bool ggml_cuda_ar_allreduce( for (int i = 0; i < n; ++i) { if (!compute_flag[i]) { auto * cuda_ctx = static_cast(backends[i]->context); + GGML_ASSERT(cuda_ctx->device == p->devices[i]); ggml_cuda_set_device(p->devices[i]); CUDA_CHECK(cudaMemsetAsync(tensors[i]->data, 0, (size_t) ne * sizeof(float), cuda_ctx->stream())); } @@ -784,15 +786,16 @@ bool ggml_cuda_ar_allreduce( to_bf16_cuda_t to_bf16 = ggml_get_to_bf16_cuda(GGML_TYPE_F32); for (int i = 0; i < n; ++i) { auto * cuda_ctx = static_cast(backends[i]->context); + GGML_ASSERT(cuda_ctx->device == p->devices[i]); bf16_tmp[i].pool = &cuda_ctx->pool(); bf16_tmp[i].alloc(ne); ggml_cuda_set_device(p->devices[i]); if (compute_flag[i]) { to_bf16(tensors[i]->data, bf16_tmp[i].get(), ne, cuda_ctx->stream()); + CUDA_CHECK(cudaGetLastError()); } else { CUDA_CHECK(cudaMemsetAsync(bf16_tmp[i].get(), 0, nbytes, cuda_ctx->stream())); } - CUDA_CHECK(cudaGetLastError()); copy_src_ptr[i] = bf16_tmp[i].get(); } } @@ -814,13 +817,13 @@ bool ggml_cuda_ar_allreduce( // post-conversion is needed. Otherwise src == dst (same native type). if (use_bf16) { GGML_ASSERT(kernel_type == GGML_TYPE_BF16); - __nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES]; - float * dst[GGML_CUDA_MAX_DEVICES]; + nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES]; + float * dst[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < n; ++i) { - src[i] = static_cast<__nv_bfloat16 *>(copy_src_ptr[i]); + src[i] = static_cast(copy_src_ptr[i]); dst[i] = static_cast(tensors[i]->data); } - ok = ggml_cuda_ar_allreduce_copy_outer<__nv_bfloat16, float>( + ok = ggml_cuda_ar_allreduce_copy_outer( p, backends, src, dst, inner_compute, ne); } else { switch (kernel_type) { @@ -832,9 +835,9 @@ bool ggml_cuda_ar_allreduce( break; } case GGML_TYPE_BF16: { - __nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast<__nv_bfloat16 *>(tensors[i]->data); - ok = ggml_cuda_ar_allreduce_copy_outer<__nv_bfloat16, __nv_bfloat16>( + nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; + for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); + ok = ggml_cuda_ar_allreduce_copy_outer( p, backends, buf, buf, inner_compute, ne); break; } @@ -873,6 +876,7 @@ bool ggml_cuda_ar_allreduce( const int peer = 1 - i; // valid for n == 2 only ggml_cuda_set_device(p->devices[i]); auto * cuda_ctx = static_cast(backends[i]->context); + GGML_ASSERT(cuda_ctx->device == p->devices[i]); cudaStream_t stream = cuda_ctx->stream(); char * data = static_cast(tensors[i]->data) + chunk_start * (int64_t) input_type_size; @@ -897,12 +901,12 @@ bool ggml_cuda_ar_allreduce( if (use_bf16) { GGML_ASSERT(input_type == GGML_TYPE_F32); - LAUNCH_AR_KERNEL(float, __nv_bfloat16); + LAUNCH_AR_KERNEL(float, nv_bfloat16); } else { switch (input_type) { - case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float, float); break; - case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half, half); break; - case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(__nv_bfloat16, __nv_bfloat16); break; + case GGML_TYPE_F32: LAUNCH_AR_KERNEL(float, float); break; + case GGML_TYPE_F16: LAUNCH_AR_KERNEL(half, half); break; + case GGML_TYPE_BF16: LAUNCH_AR_KERNEL(nv_bfloat16, nv_bfloat16); break; default: GGML_ASSERT(false); } } From 605f5e02b658d208cdf653add07844c2a14a957e Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 17:09:30 -0700 Subject: [PATCH 71/81] allreduce: expand one-liner for loops to braced bodies Code-style preference -- match the rest of the file by writing every for loop with the body on its own braced line. Three sites in the copy-engine typed dispatch. Co-Authored-By: Claude Opus 4.7 (1M context) in the AR path so the misuse fails fast rather than silently corrupting. Also: rename __nv_bfloat16 -> nv_bfloat16 (typedef alias) for consistency with the rest of the file, and tighten one cudaGetLastError check to fire only after the to_bf16 call that can actually fail. Co-Authored-By: Claude Opus 4.7 (1M context) gml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 37893dc857c..b8328826041 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -829,21 +829,27 @@ bool ggml_cuda_ar_allreduce( switch (kernel_type) { case GGML_TYPE_F32: { float * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); + for (int i = 0; i < n; ++i) { + buf[i] = static_cast(tensors[i]->data); + } ok = ggml_cuda_ar_allreduce_copy_outer( p, backends, buf, buf, inner_compute, ne); break; } case GGML_TYPE_BF16: { nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); + for (int i = 0; i < n; ++i) { + buf[i] = static_cast(tensors[i]->data); + } ok = ggml_cuda_ar_allreduce_copy_outer( p, backends, buf, buf, inner_compute, ne); break; } case GGML_TYPE_F16: { half * buf[GGML_CUDA_MAX_DEVICES]; - for (int i = 0; i < n; ++i) buf[i] = static_cast(tensors[i]->data); + for (int i = 0; i < n; ++i) { + buf[i] = static_cast(tensors[i]->data); + } ok = ggml_cuda_ar_allreduce_copy_outer( p, backends, buf, buf, inner_compute, ne); break; From 77e9b148d4411c80479504c09bbbbe06171df2e8 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 17:32:11 -0700 Subject: [PATCH 72/81] allreduce: rename template parameters Tdst/Twire/Tsrc -> T_dst/T_wire/T_src Code-style preference per PR review -- T_dst/T_wire/T_src is more consistent with surrounding code. Whole-word rename across all 58 sites in allreduce.cu (kernel definitions, internal uses, and comment text). Realigned the parameter columns in three function signatures whose T_src/T_dst lines shifted by 1 char relative to their non-templated neighbors. Co-Authored-By: Claude Opus 4.7 (1M context) to fire only after the to_bf16 call that can actually fail. Co-Authored-By: Claude Opus 4.7 (1M context) gml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 124 ++++++++++++++++---------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index b8328826041..12b04868111 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -76,21 +76,21 @@ static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // Chunked-kernel AllReduce -- 2 GPUs, supports float, half, and bfloat16. // // Both GPUs run this kernel simultaneously on independent streams. sendbuf -// and recvbuf live in Tdst (the caller's tensor type); host_mine / host_other -// carry data in Twire (the on-wire type, possibly narrower than Tdst -- e.g. -// Tdst=F32 with Twire=BF16 halves the bytes pushed across PCIe). When -// Tdst == Twire the casts below are no-ops. +// and recvbuf live in T_dst (the caller's tensor type); host_mine / host_other +// carry data in T_wire (the on-wire type, possibly narrower than T_dst -- e.g. +// T_dst=F32 with T_wire=BF16 halves the bytes pushed across PCIe). When +// T_dst == T_wire the casts below are no-ops. // // Each GPU runs three phases: // -// Phase 1 (all threads): cast sendbuf (Tdst) -> Twire and store as 16-byte +// Phase 1 (all threads): cast sendbuf (T_dst) -> T_wire and store as 16-byte // vectors into host_mine. __threadfence_system() // commits these writes to host memory. // Phase 2 (thread 0): write token to arrival_mine; spin until // arrival_other == token. -// Phase 3 (all threads): read 16-byte Twire vectors from host_other, cast -// each element to Tdst, and sum with the local -// sendbuf value (also rounded through Twire so that +// Phase 3 (all threads): read 16-byte T_wire vectors from host_other, cast +// each element to T_dst, and sum with the local +// sendbuf value (also rounded through T_wire so that // both GPUs truncate identically -- this guarantees // bit-equivalent results across the two devices). // @@ -101,21 +101,21 @@ static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // blocks. Tail elements (the leftover < ELEMS_PER_VEC at the end) are // handled only by block 0 to avoid cross-block writes to the same slots. // --------------------------------------------------------------------------- -template +template static __global__ void ggml_cuda_ar_kernel( - const Tdst * __restrict__ sendbuf, - Tdst * __restrict__ recvbuf, - Twire * __restrict__ host_mine, - const Twire * __restrict__ host_other, - int count, - int * arrival_mine, - int * arrival_other, - int token) { + const T_dst * __restrict__ sendbuf, + T_dst * __restrict__ recvbuf, + T_wire * __restrict__ host_mine, + const T_wire * __restrict__ host_other, + int count, + int * arrival_mine, + int * arrival_other, + int token) { // 16-byte vector unit for the wire type. Each phase-1 iter writes one // vector to host memory; each phase-3 iter reads one and produces // ELEMS_PER_VEC sums. - constexpr int ELEMS_PER_VEC = 16 / sizeof(Twire); + constexpr int ELEMS_PER_VEC = 16 / sizeof(T_wire); constexpr int ARRIVAL_INTS = (int)(GGML_CUDA_AR_ARRIVAL_STRIDE / sizeof(int)); const int tid = threadIdx.x; @@ -126,19 +126,19 @@ static __global__ void ggml_cuda_ar_kernel( const int count_vec = count / ELEMS_PER_VEC; const int tail = count_vec * ELEMS_PER_VEC; - // Phase 1: cast sendbuf (Tdst) -> host_mine (Twire) and store as 16-byte vectors. + // Phase 1: cast sendbuf (T_dst) -> host_mine (T_wire) and store as 16-byte vectors. { for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; - Twire wire[ELEMS_PER_VEC]; + T_wire wire[ELEMS_PER_VEC]; #pragma unroll for (int k = 0; k < ELEMS_PER_VEC; ++k) { - wire[k] = ggml_cuda_cast(sendbuf[off + k]); + wire[k] = ggml_cuda_cast(sendbuf[off + k]); } ggml_cuda_memcpy_1(&host_mine[off], wire); } if (bid == 0 && tid < count - tail) { - host_mine[tail + tid] = ggml_cuda_cast(sendbuf[tail + tid]); + host_mine[tail + tid] = ggml_cuda_cast(sendbuf[tail + tid]); } } @@ -166,43 +166,43 @@ static __global__ void ggml_cuda_ar_kernel( // Acquire peer's host_other writes (this block's stripe of them). __threadfence_system(); - // Phase 3: read peer's Twire vector, cast both sides through Twire for - // bit-equivalence, sum in Tdst precision, and write back to recvbuf. + // Phase 3: read peer's T_wire vector, cast both sides through T_wire for + // bit-equivalence, sum in T_dst precision, and write back to recvbuf. { for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; - Twire wire[ELEMS_PER_VEC]; + T_wire wire[ELEMS_PER_VEC]; ggml_cuda_memcpy_1(wire, &host_other[off]); #pragma unroll for (int k = 0; k < ELEMS_PER_VEC; ++k) { - const Twire d_low = ggml_cuda_cast(sendbuf[off + k]); - recvbuf[off + k] = ggml_cuda_cast(d_low) + ggml_cuda_cast(wire[k]); + const T_wire d_low = ggml_cuda_cast(sendbuf[off + k]); + recvbuf[off + k] = ggml_cuda_cast(d_low) + ggml_cuda_cast(wire[k]); } } if (bid == 0 && tid < count - tail) { - const Twire d_low = ggml_cuda_cast(sendbuf[tail + tid]); + const T_wire d_low = ggml_cuda_cast(sendbuf[tail + tid]); recvbuf[tail + tid] = - ggml_cuda_cast(d_low) + ggml_cuda_cast(host_other[tail + tid]); + ggml_cuda_cast(d_low) + ggml_cuda_cast(host_other[tail + tid]); } } } -// Combined load-convert-add kernel. The peer's contribution arrives as Tsrc -// (which may be a lower-precision type than Tdst when the BF16 round-trip is +// Combined load-convert-add kernel. The peer's contribution arrives as T_src +// (which may be a lower-precision type than T_dst when the BF16 round-trip is // active). For bit-equivalence between the two GPUs, dst is first rounded -// through Tsrc's precision via ggml_cuda_cast -- peer already truncated its +// through T_src's precision via ggml_cuda_cast -- peer already truncated its // own value the same way before sending -- so both sides perform identical -// arithmetic. When Tdst == Tsrc the round-trip cast is a no-op. -template +// arithmetic. When T_dst == T_src the round-trip cast is a no-op. +template static __global__ void ggml_cuda_ar_add_kernel( - Tdst * __restrict__ dst, - const Tsrc * __restrict__ src, + T_dst * __restrict__ dst, + const T_src * __restrict__ src, int count) { const int tid = blockIdx.x * blockDim.x + threadIdx.x; const int nt = gridDim.x * blockDim.x; for (int i = tid; i < count; i += nt) { - const Tsrc d_low = ggml_cuda_cast(dst[i]); - dst[i] = ggml_cuda_cast(d_low) + ggml_cuda_cast(src[i]); + const T_src d_low = ggml_cuda_cast(dst[i]); + dst[i] = ggml_cuda_cast(d_low) + ggml_cuda_cast(src[i]); } } @@ -562,18 +562,18 @@ void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline * p) { // Dispatch // --------------------------------------------------------------------------- -// Asymmetric copy_impl: data sent over PCIe in Tsrc precision (one element of -// nbytes per ne element); accumulated locally into a Tdst buffer. When -// Tsrc == Tdst this is the original homogeneous reduction. When they differ -// (e.g. BF16 wire / F32 accumulator) the add kernel rounds dst through Tsrc +// Asymmetric copy_impl: data sent over PCIe in T_src precision (one element of +// nbytes per ne element); accumulated locally into a T_dst buffer. When +// T_src == T_dst this is the original homogeneous reduction. When they differ +// (e.g. BF16 wire / F32 accumulator) the add kernel rounds dst through T_src // for bit-equivalence between GPUs and we skip the otherwise-needed // post-conversion entirely. -template +template static bool ggml_cuda_ar_allreduce_copy_impl( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, - Tsrc * const src_buf[GGML_CUDA_MAX_DEVICES], - Tdst * const dst_buf[GGML_CUDA_MAX_DEVICES], + T_src * const src_buf[GGML_CUDA_MAX_DEVICES], + T_dst * const dst_buf[GGML_CUDA_MAX_DEVICES], const bool compute[GGML_CUDA_MAX_DEVICES], int64_t ne, size_t nbytes) { @@ -666,9 +666,9 @@ static bool ggml_cuda_ar_allreduce_copy_impl( if (n_blocks > 1024) { n_blocks = 1024; } - ggml_cuda_ar_add_kernel<<stream()>>>( + ggml_cuda_ar_add_kernel<<stream()>>>( dst_buf[i], - reinterpret_cast(p->dev_tmp[i]), + reinterpret_cast(p->dev_tmp[i]), (int) ne); CUDA_CHECK(cudaGetLastError()); @@ -690,29 +690,29 @@ static bool ggml_cuda_ar_allreduce_copy_impl( // Each slice goes through its own stage 1 -> stage 2 cycle and acquires its own // slot, so cross-AR fences and pool wraparound work the same way as for any // other sequence of small ARs. -template +template static bool ggml_cuda_ar_allreduce_copy_outer( ggml_cuda_ar_pipeline * p, ggml_backend_t * backends, - Tsrc * const src_buf[GGML_CUDA_MAX_DEVICES], - Tdst * const dst_buf[GGML_CUDA_MAX_DEVICES], + T_src * const src_buf[GGML_CUDA_MAX_DEVICES], + T_dst * const dst_buf[GGML_CUDA_MAX_DEVICES], const bool compute[GGML_CUDA_MAX_DEVICES], int64_t ne) { - const int64_t outer_max_elems = (int64_t) (p->copy_bytes / sizeof(Tsrc)); + const int64_t outer_max_elems = (int64_t) (p->copy_bytes / sizeof(T_src)); GGML_ASSERT(outer_max_elems > 0); bool ok = true; for (int64_t outer_start = 0; outer_start < ne && ok; outer_start += outer_max_elems) { const int64_t outer_ne = std::min(outer_max_elems, ne - outer_start); - const size_t outer_nbytes = (size_t) outer_ne * sizeof(Tsrc); + const size_t outer_nbytes = (size_t) outer_ne * sizeof(T_src); - Tsrc * src[GGML_CUDA_MAX_DEVICES]; - Tdst * dst[GGML_CUDA_MAX_DEVICES]; + T_src * src[GGML_CUDA_MAX_DEVICES]; + T_dst * dst[GGML_CUDA_MAX_DEVICES]; for (int i = 0; i < p->n_devices; ++i) { src[i] = src_buf[i] + outer_start; dst[i] = dst_buf[i] + outer_start; } - ok = ggml_cuda_ar_allreduce_copy_impl( + ok = ggml_cuda_ar_allreduce_copy_impl( p, backends, src, dst, compute, outer_ne, outer_nbytes); } return ok; @@ -859,7 +859,7 @@ bool ggml_cuda_ar_allreduce( } } } else { - // host_buf carries Twire-typed data; max_chunk_elems is the count that + // host_buf carries T_wire-typed data; max_chunk_elems is the count that // fits in one host_buf at the wire size. const size_t max_chunk_elems = p->buf_bytes / type_size; const size_t input_type_size = ggml_type_size(input_type); @@ -894,12 +894,12 @@ bool ggml_cuda_ar_allreduce( CUDA_CHECK(cudaMemsetAsync(data, 0, chunk_dst_bytes, stream)); } -#define LAUNCH_AR_KERNEL(Tdst, Twire) \ - ggml_cuda_ar_kernel<<>>( \ - reinterpret_cast(data), \ - reinterpret_cast(data), \ - reinterpret_cast(p->host_buf[i].dev + (size_t) slot * p->buf_bytes), \ - reinterpret_cast(p->host_buf[peer].dev + (size_t) slot * p->buf_bytes), \ +#define LAUNCH_AR_KERNEL(T_dst, T_wire) \ + ggml_cuda_ar_kernel<<>>( \ + reinterpret_cast(data), \ + reinterpret_cast(data), \ + reinterpret_cast(p->host_buf[i].dev + (size_t) slot * p->buf_bytes), \ + reinterpret_cast(p->host_buf[peer].dev + (size_t) slot * p->buf_bytes), \ static_cast(chunk_elems), \ ggml_cuda_ar_arrival_ptr(p, slot, i), \ ggml_cuda_ar_arrival_ptr(p, slot, peer), \ From b5e18bcf9faca5a1cffc520a79286049c9ef07be Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 17:47:40 -0700 Subject: [PATCH 73/81] allreduce: drop hyphen in 'chunked-kernel' across comments Per PR review feedback -- 'chunked kernel' (no hyphen) reads more naturally in running prose, especially for ESL readers. Pure comment-only change; all 10 occurrences in allreduce.cu updated. Co-Authored-By: Claude Opus 4.7 (1M context) three function signatures whose T_src/T_dst lines shifted by 1 char relative to their non-templated neighbors. Co-Authored-By: Claude Opus 4.7 (1M context) to fire only after the to_bf16 call that can actually fail. Co-Authored-By: Claude Opus 4.7 (1M context) gml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 12b04868111..951f30fd78a 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -17,7 +17,7 @@ // // Two reduction strategies are selected per call by tensor size: // -// * Chunked-kernel path (small reductions): a single CUDA kernel both +// * Chunked kernel path (small reductions): a single CUDA kernel both // stages data through pinned host memory and performs the local sum. // Cross-GPU synchronization happens *inside the kernel* (busy-wait on // a host-memory flag), which keeps launch overhead low for the @@ -67,13 +67,13 @@ static __device__ __forceinline__ int ggml_cuda_ar_signal_get(const int * p) { // false-sharing stalls on the polling GPU. static constexpr size_t GGML_CUDA_AR_ARRIVAL_STRIDE = 64; -// Number of blocks the chunked-kernel launches with. Each block stripes a +// Number of blocks the chunked kernel launches with. Each block stripes a // disjoint slice of the data and synchronizes through its own arrival-token // slot so multiple SMs can pump PCIe stores in parallel. static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // --------------------------------------------------------------------------- -// Chunked-kernel AllReduce -- 2 GPUs, supports float, half, and bfloat16. +// Chunked kernel AllReduce -- 2 GPUs, supports float, half, and bfloat16. // // Both GPUs run this kernel simultaneously on independent streams. sendbuf // and recvbuf live in T_dst (the caller's tensor type); host_mine / host_other @@ -218,7 +218,7 @@ static __global__ void ggml_cuda_ar_add_kernel( // explicit before we overwrite host_buf[slot] for the new AR. static constexpr int GGML_CUDA_AR_POOL_SIZE = 2; -// Maximum chunk size (bytes per GPU) handled by one chunked-kernel launch. +// Maximum chunk size (bytes per GPU) handled by one chunked kernel launch. // Larger tensors are reduced by issuing multiple chunked launches. static constexpr size_t GGML_CUDA_AR_MAX_BYTES = 1024 * 1024; // 1 MB @@ -296,7 +296,7 @@ struct ggml_cuda_ar_pipeline { uint64_t call_count; // Per-device resources. - ggml_cuda_ar_host_mapping host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging (chunked-kernel) + ggml_cuda_ar_host_mapping host_buf[GGML_CUDA_MAX_DEVICES]; // pinned staging (chunked kernel) ggml_cuda_ar_host_mapping host_large[GGML_CUDA_MAX_DEVICES]; // pinned staging (copy-engine) char * dev_tmp[GGML_CUDA_MAX_DEVICES]; // device scratch for copy-engine path cudaStream_t streams[GGML_CUDA_MAX_DEVICES]; // non-blocking @@ -506,7 +506,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " - "%zu KB chunked-kernel staging + %zu MB copy-engine staging per GPU\n", + "%zu KB chunked kernel staging + %zu MB copy-engine staging per GPU\n", __func__, n_devices, p->buf_bytes >> 10, p->copy_bytes >> 20); return p; @@ -754,7 +754,7 @@ bool ggml_cuda_ar_allreduce( compute_flag[i] = (tensors[i]->flags & GGML_TENSOR_FLAG_COMPUTE) != 0; } - // Decide between copy-engine and chunked-kernel paths based on the working + // Decide between copy-engine and chunked kernel paths based on the working // type's actual byte count. No upper bound: copy_outer slices reductions // larger than copy_bytes into copy_bytes-sized pieces. const bool use_copy_engine = @@ -762,7 +762,7 @@ bool ggml_cuda_ar_allreduce( nbytes >= p->copy_threshold; // BF16 inactive-shard zeroing: when use_bf16 is on, the combined kernel - // (chunked-kernel path) and the combined add kernel (copy_engine path) + // (chunked kernel path) and the combined add kernel (copy_engine path) // both accumulate into the F32 tensor data directly, so an inactive // shard's accumulator must start at zero. if (use_bf16) { @@ -777,7 +777,7 @@ bool ggml_cuda_ar_allreduce( } // Pre-convert F32 -> BF16 into bf16_tmp ONLY for the copy_engine + use_bf16 - // path; the chunked-kernel path's combined kernel does the conversion + // path; the chunked kernel path's combined kernel does the conversion // inline as it writes to host_buf. ggml_cuda_pool_alloc bf16_tmp[GGML_CUDA_MAX_DEVICES]; void * copy_src_ptr[GGML_CUDA_MAX_DEVICES] = {}; @@ -864,7 +864,7 @@ bool ggml_cuda_ar_allreduce( const size_t max_chunk_elems = p->buf_bytes / type_size; const size_t input_type_size = ggml_type_size(input_type); - // Chunked-kernel path runs entirely on the caller's compute stream: + // Chunked kernel path runs entirely on the caller's compute stream: // since AR is a barrier here, same-stream ordering subsumes any // cross-stream event handshake that the copy-engine path needs, and // skips the cross-stream scheduling overhead that was hurting the From f4c8b2622adad41ff7ca2c32945d08ce28992918 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Wed, 6 May 2026 18:20:39 -0700 Subject: [PATCH 74/81] allreduce: use ggml_cuda_get_max_cpy_bytes() instead of hardcoded 16 The chunked kernel hardcoded a 16-byte vector unit; replace with the ggml_cuda_get_max_cpy_bytes() helper that fattn-common.cuh uses for the same purpose, so ELEMS_PER_VEC self-adjusts to the arch's widest single-instruction copy. Perf-neutral on supported targets (Volta+ returns 16). Co-Authored-By: Claude Opus 4.7 (1M context) hbors. Co-Authored-By: Claude Opus 4.7 (1M context) to fire only after the to_bf16 call that can actually fail. Co-Authored-By: Claude Opus 4.7 (1M context) gml-cuda.cu). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 951f30fd78a..8e40934202d 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -83,12 +83,13 @@ static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // // Each GPU runs three phases: // -// Phase 1 (all threads): cast sendbuf (T_dst) -> T_wire and store as 16-byte -// vectors into host_mine. __threadfence_system() -// commits these writes to host memory. +// Phase 1 (all threads): cast sendbuf (T_dst) -> T_wire and store as +// single-instruction-width vectors into host_mine. +// __threadfence_system() commits these writes to host +// memory. // Phase 2 (thread 0): write token to arrival_mine; spin until // arrival_other == token. -// Phase 3 (all threads): read 16-byte T_wire vectors from host_other, cast +// Phase 3 (all threads): read T_wire vectors from host_other, cast // each element to T_dst, and sum with the local // sendbuf value (also rounded through T_wire so that // both GPUs truncate identically -- this guarantees @@ -112,10 +113,10 @@ static __global__ void ggml_cuda_ar_kernel( int * arrival_other, int token) { - // 16-byte vector unit for the wire type. Each phase-1 iter writes one - // vector to host memory; each phase-3 iter reads one and produces - // ELEMS_PER_VEC sums. - constexpr int ELEMS_PER_VEC = 16 / sizeof(T_wire); + // Vector unit for the wire type, sized to the arch's widest single-instruction + // copy (16 B on Volta+). Each phase-1 iter writes one vector to host memory; + // each phase-3 iter reads one and produces ELEMS_PER_VEC sums. + constexpr int ELEMS_PER_VEC = ggml_cuda_get_max_cpy_bytes() / sizeof(T_wire); constexpr int ARRIVAL_INTS = (int)(GGML_CUDA_AR_ARRIVAL_STRIDE / sizeof(int)); const int tid = threadIdx.x; @@ -126,7 +127,7 @@ static __global__ void ggml_cuda_ar_kernel( const int count_vec = count / ELEMS_PER_VEC; const int tail = count_vec * ELEMS_PER_VEC; - // Phase 1: cast sendbuf (T_dst) -> host_mine (T_wire) and store as 16-byte vectors. + // Phase 1: cast sendbuf (T_dst) -> host_mine (T_wire) and store as vectors. { for (int i = gtid; i < count_vec; i += gnt) { const int off = i * ELEMS_PER_VEC; From 29b11cf515e2717e798c0ec5868cefe69563acb4 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 7 May 2026 18:27:59 -0700 Subject: [PATCH 75/81] ggml-cuda: PR review fixes -- annotate #endif, fix stale comment, assert nbytes alignment Three separate but minor changes from PR #22299 review feedback: 1. Annotate the five GGML_USE_NCCL #endif lines with the matching condition so the pairing is visible without scrolling back. 2. The comment block on ggml_backend_cuda_comm_context claimed NCCL is lazy-initialised; that was true at one point but the dispatch refactor (727b141c0) made both NCCL and the internal pipeline eager. Rewrite the comment to match current behaviour. 3. Assert in ggml_backend_cuda_comm_allreduce_internal that the tensor's byte size is a 16-byte multiple. The chunked-kernel issues full-width vector loads/stores, so this is a precondition; tensor-parallel splits of hidden-dim-multiples satisfy it trivially, but a hard assert turns any caller-side bug into a clear failure rather than UB. Co-Authored-By: Claude Opus 4.7 (1M context) device's new AR records its ev.ker -- otherwise the second device's wait sees the first device's just-recorded event (the in-flight new AR) and creates a circular dependency with the in-kernel peer signal. Two-pass dispatch (all waits, then all launches) avoids this. Bump POOL_SIZE 2 -> 8 (small memory cost, more breathing room for the GPU's view of the event chain) and add a runtime env override for the hybrid kernel chunk size (GGML_CUDA_AR_HYBRID_CHUNK_BYTES) for tuning. One-shot stderr diagnostic at first AR prints the chosen path + sizing. Result on 2x RTX 5090 Linux, 70b ub_sweep: ub=64 (1 MB AR): 913 -> 1036 t/s (+13.5% vs old, +1.8% vs NCCL) ub=128 (2 MB AR): 1056 -> 1181 (+11.9%, +3.7% vs NCCL) ub=256 (4 MB AR): 1212 -> 1424 (+17.5%, +3.5% vs NCCL) Internal now beats NCCL at every size (+1.8% to +15.6%), recovering all ground in the 1-4 MB regime that was previously a 10-12% loss. Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/ggml-cuda.cu | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 62e35ea704b..e88486442fb 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1144,10 +1144,10 @@ static const ggml_backend_buffer_type_i ggml_backend_cuda_split_buffer_type_inte // Communication context for multi-GPU AllReduce during tensor parallelism. // -// Created once per meta backend instance. The internal pipeline (if any) is -// allocated eagerly at init time; NCCL communicators are created lazily on -// first use so NCCL's init/runtime quirks don't interfere with configurations -// that never hit the fallback path. +// Created once per meta backend instance. Resources for the selected mode +// (NCCL communicators or the internal AllReduce pipeline) are initialised +// eagerly during comm_init so any init failure surfaces at startup rather +// than mid-run. struct ggml_backend_cuda_comm_context { using try_allreduce_fn = bool(*)(ggml_backend_cuda_comm_context *, struct ggml_tensor **); @@ -1165,14 +1165,14 @@ struct ggml_backend_cuda_comm_context { #ifdef GGML_USE_NCCL std::vector comms; -#endif +#endif // GGML_USE_NCCL ~ggml_backend_cuda_comm_context() { #ifdef GGML_USE_NCCL for (ncclComm_t comm : comms) { NCCL_CHECK(ncclCommDestroy(comm)); } -#endif +#endif // GGML_USE_NCCL ggml_cuda_ar_pipeline_free(ar_pipeline); } }; @@ -1276,6 +1276,13 @@ static bool ggml_backend_cuda_comm_allreduce_internal( return true; } + // Per-AR vector path requires the byte size to be a 16-byte multiple so + // the chunked-kernel can issue full-width vector loads/stores. In + // practice all tensors we hand off here come from tensor-parallel splits + // of hidden_dim-multiples and trivially satisfy this; a violation would + // indicate a caller-side bug. + GGML_ASSERT(((size_t) ne * ggml_type_size(type) & 0xF) == 0); + for (size_t i = 0; i < n_backends; ++i) { if (tensors[i] == nullptr) { GGML_LOG_ERROR("%s: internal failed: tensor[%zu] is null\n", __func__, i); @@ -1318,7 +1325,7 @@ static bool ggml_backend_cuda_comm_try_allreduce_nccl( #else GGML_UNUSED(comm_ctx); GGML_UNUSED(tensors); GGML_ABORT("try_allreduce_nccl unreachable: built without NCCL"); -#endif +#endif // GGML_USE_NCCL } // Internal-only (env=internal). Failure aborts so the user knows their @@ -1371,7 +1378,7 @@ static bool ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ct } return true; } -#endif +#endif // GGML_USE_NCCL static bool ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context * ctx) { ctx->ar_pipeline = ggml_cuda_ar_pipeline_init(ctx->dev_ids.data(), ctx->dev_ids.size()); @@ -1437,7 +1444,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba "multi-GPU performance will be suboptimal. " "Recompile with -DGGML_CUDA_NCCL=ON for best performance."); ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; -#endif +#endif // GGML_USE_NCCL } else if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_internal_strict) { if (!ggml_backend_cuda_comm_init_internal(ret)) { GGML_ABORT("internal AllReduce pipeline init failed (n_devices != 2 or pre-Ampere?). " From 6ae3093845ca185dedb85f5361893722ff28ac88 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 7 May 2026 20:01:22 -0700 Subject: [PATCH 76/81] simplify the init logic --- ggml/src/ggml-cuda/ggml-cuda.cu | 166 +++++++++++++------------------- 1 file changed, 66 insertions(+), 100 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index e88486442fb..fd6975707a7 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1154,11 +1154,11 @@ struct ggml_backend_cuda_comm_context { std::vector backends; std::vector dev_ids; - // Set in comm_init to one of try_allreduce_{nccl, internal_strict, - // internal_lenient, butterfly} based on GGML_CUDA_ALLREDUCE and the - // platform. Each variant assumes the resources it needs were - // initialised in comm_init; nccl needs `comms`, both internal variants - // need `ar_pipeline`, butterfly needs nothing. + // Set by the init chain (comm_init_{nccl, internal, none}) to one of + // try_allreduce_{nccl, internal, butterfly}. nccl needs `comms`, + // internal needs `ar_pipeline`, butterfly needs nothing. Per-call + // failures return false; the meta backend's generic implementation then + // handles that call. try_allreduce_fn try_allreduce = nullptr; ggml_cuda_ar_pipeline * ar_pipeline = nullptr; @@ -1310,17 +1310,14 @@ static bool ggml_backend_cuda_comm_allreduce_internal( } // --------------------------------------------------------------------------- -// try_allreduce variants -- one per mode. All assume their required resource -// has already been initialised by comm_init. +// Per-call dispatch -- three variants, one per backend. Each is set as +// comm_ctx->try_allreduce by the matching init step. Per-call failure +// returns false; the meta backend's generic implementation handles that call. // --------------------------------------------------------------------------- -// NCCL-only. Used for env=nccl on any platform AND for the Linux default. -// On NCCL-internal failure, ggml_backend_cuda_comm_allreduce_nccl aborts via -// NCCL_CHECK; we'll only get here on success. static bool ggml_backend_cuda_comm_try_allreduce_nccl( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { #ifdef GGML_USE_NCCL - GGML_ASSERT(!comm_ctx->comms.empty()); return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); #else GGML_UNUSED(comm_ctx); GGML_UNUSED(tensors); @@ -1328,30 +1325,11 @@ static bool ggml_backend_cuda_comm_try_allreduce_nccl( #endif // GGML_USE_NCCL } -// Internal-only (env=internal). Failure aborts so the user knows their -// requested mode is not viable. -static bool ggml_backend_cuda_comm_try_allreduce_internal_strict( +static bool ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors)) { - return true; - } - GGML_ABORT("GGML_CUDA_ALLREDUCE=internal: AR call failed (unsupported input). " - "Reset the environment variable to use the platform default."); + return ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors); } -// Internal with butterfly fallback. Used for the Windows default -- internal -// is preferred but a return-false cleanly hits the meta-backend's butterfly. -static bool ggml_backend_cuda_comm_try_allreduce_internal_lenient( - ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { - if (ggml_backend_cuda_comm_allreduce_internal(comm_ctx, tensors)) { - return true; - } - GGML_LOG_WARN_ONCE("internal AllReduce call failed; " - "meta-backend butterfly will be used for this and subsequent calls\n"); - return false; -} - -// Butterfly-only (env=none, or after a failed init for non-strict modes). static bool ggml_backend_cuda_comm_try_allreduce_butterfly( ggml_backend_cuda_comm_context *, struct ggml_tensor **) { return false; @@ -1364,36 +1342,53 @@ static void ggml_backend_cuda_comm_free(void * comm_ctx_v) { delete static_cast(comm_ctx_v); } -// Resource initializers -- return true on success. +// --------------------------------------------------------------------------- +// Init -- chained nccl -> internal -> none. Each step tries to bring up its +// resource; on failure it warns and recurses into the next step. +// --------------------------------------------------------------------------- +static void ggml_backend_cuda_comm_init_none(ggml_backend_cuda_comm_context * ret) { + ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; +} -#ifdef GGML_USE_NCCL -static bool ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ctx) { - const size_t n = ctx->dev_ids.size(); - ctx->comms.resize(n); - ncclResult_t rc = ncclCommInitAll(ctx->comms.data(), (int) n, ctx->dev_ids.data()); - if (rc != ncclSuccess) { - ctx->comms.clear(); - GGML_LOG_ERROR("%s: ncclCommInitAll failed: %s\n", __func__, ncclGetErrorString(rc)); - return false; +static void ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context * ret) { + ret->ar_pipeline = ggml_cuda_ar_pipeline_init(ret->dev_ids.data(), ret->dev_ids.size()); + if (ret->ar_pipeline) { + ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal; + return; } - return true; + + // Clear sticky CUDA error from the failed init. + (void) cudaGetLastError(); + GGML_LOG_WARN("internal AllReduce init failed (n_devices != 2 or pre-Ampere?); " + "falling back to meta-backend butterfly\n"); + ggml_backend_cuda_comm_init_none(ret); } -#endif // GGML_USE_NCCL -static bool ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context * ctx) { - ctx->ar_pipeline = ggml_cuda_ar_pipeline_init(ctx->dev_ids.data(), ctx->dev_ids.size()); - if (ctx->ar_pipeline == nullptr) { - // Clear sticky CUDA error from the failed init. - (void) cudaGetLastError(); - return false; +static void ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ret) { +#ifdef GGML_USE_NCCL + const size_t n = ret->dev_ids.size(); + ret->comms.resize(n); + ncclResult_t rc = ncclCommInitAll(ret->comms.data(), (int) n, ret->dev_ids.data()); + if (rc == ncclSuccess) { + ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; + return; } - return true; + + ret->comms.clear(); + GGML_LOG_WARN("NCCL init failed (%s); falling back to internal AllReduce\n", + ncclGetErrorString(rc)); +#else // GGML_USE_NCCL + GGML_LOG_WARN("NCCL not compiled in; falling back to internal AllReduce. " + "Recompile with -DGGML_CUDA_NCCL=ON for best multi-GPU performance.\n"); +#endif // GGML_USE_NCCL + + ggml_backend_cuda_comm_init_internal(ret); } -// Pick the try_allreduce function pointer based on GGML_CUDA_ALLREDUCE / OS, -// then init the resource that pointer needs (NCCL or internal pipeline). -// Init failure aborts in every case -- internal-lenient's "fall back to -// butterfly" applies to per-call failures, not init. +// Top-level init. Picks one of the three init paths based on +// GGML_CUDA_ALLREDUCE (or the platform default) and lets the chain handle +// any fallback. Unrecognised env values warn and fall through to the +// platform default. static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_backends) { for (size_t i = 0; i < n_backends; i++) { if (!ggml_backend_is_cuda(backends[i])) { @@ -1408,56 +1403,27 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba ret->dev_ids.push_back(static_cast(backends[i]->context)->device); } - // 1. Pick the function pointer. const char * env = getenv("GGML_CUDA_ALLREDUCE"); - const bool env_nccl = env && strcmp(env, "nccl") == 0; - const bool env_internal = env && strcmp(env, "internal") == 0; - const bool env_none = env && strcmp(env, "none") == 0; - - if (env_nccl) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; - else if (env_internal) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal_strict; - else if (env_none) ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; -#if defined(_WIN32) - else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_internal_lenient; -#elif defined(__linux__) - else ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_nccl; + if (!env) { + // Platform default: Linux uses NCCL, otherwise (generally Windows) internal +#if defined(__linux__) + ggml_backend_cuda_comm_init_nccl(ret); #else - else GGML_ABORT("no AllReduce default for this platform; set GGML_CUDA_ALLREDUCE explicitly"); + ggml_backend_cuda_comm_init_internal(ret); #endif - - // 2. Init the matching resource. Strict modes (env-forced or Linux - // default = NCCL) abort on failure. The Windows-default lenient - // internal mode degrades to butterfly on init failure. Linux - // without NCCL compiled in degrades to butterfly with a warning. - if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_nccl) { -#ifdef GGML_USE_NCCL - if (!ggml_backend_cuda_comm_init_nccl(ret)) { - GGML_ABORT("NCCL init failed. Set GGML_CUDA_ALLREDUCE=internal or =none to bypass."); - } -#else - if (env_nccl) { - GGML_ABORT("GGML_CUDA_ALLREDUCE=nccl requested but llama.cpp was not built with NCCL. " - "Recompile with -DGGML_CUDA_NCCL=ON or reset the environment variable."); - } - // Linux default with no NCCL compiled: warn and degrade to butterfly. - GGML_LOG_WARN_ONCE("NVIDIA Collective Communications Library (NCCL) is unavailable; " - "multi-GPU performance will be suboptimal. " - "Recompile with -DGGML_CUDA_NCCL=ON for best performance."); - ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; -#endif // GGML_USE_NCCL - } else if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_internal_strict) { - if (!ggml_backend_cuda_comm_init_internal(ret)) { - GGML_ABORT("internal AllReduce pipeline init failed (n_devices != 2 or pre-Ampere?). " - "Reset GGML_CUDA_ALLREDUCE to use the platform default."); - } - } else if (ret->try_allreduce == ggml_backend_cuda_comm_try_allreduce_internal_lenient) { - if (!ggml_backend_cuda_comm_init_internal(ret)) { - GGML_LOG_WARN_ONCE("internal AllReduce pipeline init failed (n_devices != 2 or pre-Ampere?); " - "meta-backend butterfly will be used"); - ret->try_allreduce = ggml_backend_cuda_comm_try_allreduce_butterfly; + } else { + std::string env_str(env); + if (env_str == "nccl") { + ggml_backend_cuda_comm_init_nccl(ret); + } else if (env_str == "internal") { + ggml_backend_cuda_comm_init_internal(ret); + } else if (env_str == "none") { + ggml_backend_cuda_comm_init_none(ret); + } else { + GGML_LOG_WARN("unknown GGML_CUDA_ALLREDUCE value: %s", env); + ggml_backend_cuda_comm_init_none(ret); } } - // else: butterfly, no init needed. return ret; } From d64b3b10e922a62207e67ab8c45b18b2e0458290 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Thu, 7 May 2026 20:15:55 -0700 Subject: [PATCH 77/81] address some other PR requests --- ggml/src/ggml-cuda/ggml-cuda.cu | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index fd6975707a7..314241abcbb 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1276,13 +1276,6 @@ static bool ggml_backend_cuda_comm_allreduce_internal( return true; } - // Per-AR vector path requires the byte size to be a 16-byte multiple so - // the chunked-kernel can issue full-width vector loads/stores. In - // practice all tensors we hand off here come from tensor-parallel splits - // of hidden_dim-multiples and trivially satisfy this; a violation would - // indicate a caller-side bug. - GGML_ASSERT(((size_t) ne * ggml_type_size(type) & 0xF) == 0); - for (size_t i = 0; i < n_backends; ++i) { if (tensors[i] == nullptr) { GGML_LOG_ERROR("%s: internal failed: tensor[%zu] is null\n", __func__, i); @@ -1304,6 +1297,7 @@ static bool ggml_backend_cuda_comm_allreduce_internal( __func__, i, tensors[i]->data, (int) type, ne); return false; } + GGML_ASSERT((ggml_nbytes(tensors[i]) & 0xF) == 0); } return ggml_cuda_ar_allreduce(comm_ctx->ar_pipeline, comm_ctx->backends.data(), tensors); @@ -1410,7 +1404,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba ggml_backend_cuda_comm_init_nccl(ret); #else ggml_backend_cuda_comm_init_internal(ret); -#endif +#endif // defined(__linux__) } else { std::string env_str(env); if (env_str == "nccl") { @@ -1420,7 +1414,7 @@ static void * ggml_backend_cuda_comm_init(ggml_backend_t * backends, size_t n_ba } else if (env_str == "none") { ggml_backend_cuda_comm_init_none(ret); } else { - GGML_LOG_WARN("unknown GGML_CUDA_ALLREDUCE value: %s", env); + GGML_LOG_WARN("unknown GGML_CUDA_ALLREDUCE value: %s\n", env); ggml_backend_cuda_comm_init_none(ret); } } From 2dd828a58fc8cf894d73846de859875608a41ec0 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Fri, 8 May 2026 19:34:33 -0700 Subject: [PATCH 78/81] ggml-cuda: stub internal AllReduce on HIP/MUSA, drop pre-Ampere mention, gate NCCL fallback warning on !HIP The internal AllReduce relies on cudaHostAllocPortable/Mapped, cudaHostGetDevicePointer, and __nanosleep -- none of which the HIP or MUSA shims expose -- so wrap the implementation in !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) and provide nullptr/no-op/false stubs in the #else branch. The dispatcher already treats a null pipeline as init failure and silently falls back to the meta backend's generic AllReduce, so HIP/MUSA builds compile clean and behave correctly without further call-site changes. PR review follow-ups: - drop "or pre-Ampere?" from the internal-init failure warning -- the kernel doesn't require Ampere or newer. - guard the "NCCL not compiled in" fallback warning behind !defined(GGML_USE_HIP); the suggestion to install NCCL only makes sense on NVIDIA builds. Co-Authored-By: Claude Opus 4.7 (1M context) hind, now +6-8% ahead at ub=1024-4096. Perplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 21 +++++++++++++++++++++ ggml/src/ggml-cuda/ggml-cuda.cu | 4 +++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index 8e40934202d..d8a6866eb1c 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -1,4 +1,7 @@ #include "allreduce.cuh" + +#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) + #include "convert.cuh" #include "ggml-impl.h" @@ -930,3 +933,21 @@ bool ggml_cuda_ar_allreduce( return ok; } + +#else // defined(GGML_USE_HIP) || defined(GGML_USE_MUSA) + +// HIP and MUSA lack the host-mapped pinned-memory APIs (cudaHostAllocPortable +// / cudaHostAllocMapped / cudaHostGetDevicePointer) and __nanosleep that this +// implementation relies on, so the internal AllReduce is a CUDA-only feature. +// The dispatcher in ggml-cuda.cu treats a nullptr pipeline as "init failed" +// and silently falls back to the meta backend's generic AllReduce. +ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int *, size_t) { + return nullptr; +} +void ggml_cuda_ar_pipeline_free(ggml_cuda_ar_pipeline *) { +} +bool ggml_cuda_ar_allreduce(ggml_cuda_ar_pipeline *, ggml_backend_t *, ggml_tensor **) { + return false; +} + +#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 314241abcbb..f309e04d116 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1353,7 +1353,7 @@ static void ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context // Clear sticky CUDA error from the failed init. (void) cudaGetLastError(); - GGML_LOG_WARN("internal AllReduce init failed (n_devices != 2 or pre-Ampere?); " + GGML_LOG_WARN("internal AllReduce init failed (n_devices != 2?); " "falling back to meta-backend butterfly\n"); ggml_backend_cuda_comm_init_none(ret); } @@ -1372,8 +1372,10 @@ static void ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * re GGML_LOG_WARN("NCCL init failed (%s); falling back to internal AllReduce\n", ncclGetErrorString(rc)); #else // GGML_USE_NCCL +#ifndef GGML_USE_HIP GGML_LOG_WARN("NCCL not compiled in; falling back to internal AllReduce. " "Recompile with -DGGML_CUDA_NCCL=ON for best multi-GPU performance.\n"); +#endif // !GGML_USE_HIP #endif // GGML_USE_NCCL ggml_backend_cuda_comm_init_internal(ret); From ced4924e3af6965af4d4ba2c075308034fadcbea Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Sat, 9 May 2026 14:27:19 -0700 Subject: [PATCH 79/81] allreduce: guard __nanosleep on Volta+ and reject pre-Volta devices at init __nanosleep is the only Volta-specific intrinsic in the kernel; wrap it in #if __CUDA_ARCH__ >= GGML_CUDA_CC_VOLTA / NO_DEVICE_CODE so the file still compiles cleanly when targeting older arches (the dispatcher's init check below ensures the kernel is never actually launched on pre-Volta). Add a per-device compute-capability check in pipeline_init that returns nullptr if any device is below sm70. The dispatcher already treats nullptr as init failure and silently falls back to the meta backend's generic AllReduce. Co-Authored-By: Claude Opus 4.7 (1M context) rom the internal-init failure warning -- the kernel doesn't require Ampere or newer. - guard the "NCCL not compiled in" fallback warning behind !defined(GGML_USE_HIP); the suggestion to install NCCL only makes sense on NVIDIA builds. Co-Authored-By: Claude Opus 4.7 (1M context) hind, now +6-8% ahead at ub=1024-4096. Perplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index d8a6866eb1c..d68a6c89230 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -161,7 +161,11 @@ static __global__ void ggml_cuda_ar_kernel( __threadfence_system(); // make our signal visible system-wide while (ggml_cuda_ar_signal_get(other_slot) != token) { +#if __CUDA_ARCH__ >= GGML_CUDA_CC_VOLTA __nanosleep(100); +#else + NO_DEVICE_CODE; +#endif // __CUDA_ARCH__ >= GGML_CUDA_CC_VOLTA } } @@ -394,6 +398,17 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n return nullptr; } + // The chunked kernel uses __nanosleep, which is sm70+ (Volta+). + for (size_t i = 0; i < n_devices; ++i) { + const int cc = ggml_cuda_info().devices[devices[i]].cc; + if (cc < GGML_CUDA_CC_VOLTA) { + GGML_LOG_DEBUG("%s: internal AllReduce requires compute capability >= %d " + "(device %d has cc=%d); falling back\n", + __func__, GGML_CUDA_CC_VOLTA, devices[i], cc); + return nullptr; + } + } + auto * p = new ggml_cuda_ar_pipeline{}; p->n_devices = n_devices; p->copy_bytes = GGML_CUDA_AR_COPY_MAX_BYTES; From db5516048b0a4aa556db6db1d77f35aceeb82e73 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Sat, 9 May 2026 16:36:09 -0700 Subject: [PATCH 80/81] allreduce: fix CI -Werror warnings (sign-compare, format, restrict alias, maybe-uninitialized) The CUDA CI builds with -Werror -Wsign-compare -Wformat -Wrestrict -Wmaybe-uninitialized. Address each: - n_devices is size_t; change `int i; i < n_devices` to size_t in the three init loops, and the matching GGML_LOG_INFO format from %d to %zu. - ggml_cuda_ar_kernel was launched with sendbuf == recvbuf (in-place reduction), so the __restrict__ qualifiers on those parameters were technically UB. Drop __restrict__ from sendbuf and recvbuf; an A/B sweep showed <0.6% perf delta (within noise) on Linux. - The buf/src/dst pointer arrays in ggml_cuda_ar_allreduce and the per-iteration arrays in ggml_cuda_ar_allreduce_copy_outer were declared with size GGML_CUDA_MAX_DEVICES but the loop only writes indices [0, n_devices); zero-initialise so the compiler sees the tail elements as defined. Co-Authored-By: Claude Opus 4.7 (1M context) now +6-8% ahead at ub=1024-4096. Perplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/allreduce.cu | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/ggml/src/ggml-cuda/allreduce.cu b/ggml/src/ggml-cuda/allreduce.cu index d68a6c89230..434689abd95 100644 --- a/ggml/src/ggml-cuda/allreduce.cu +++ b/ggml/src/ggml-cuda/allreduce.cu @@ -107,8 +107,8 @@ static constexpr int GGML_CUDA_AR_KERNEL_BLOCKS = 8; // --------------------------------------------------------------------------- template static __global__ void ggml_cuda_ar_kernel( - const T_dst * __restrict__ sendbuf, - T_dst * __restrict__ recvbuf, + const T_dst * sendbuf, + T_dst * recvbuf, T_wire * __restrict__ host_mine, const T_wire * __restrict__ host_other, int count, @@ -430,7 +430,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } // Per-device streams and event pools. - for (int i = 0; i < n_devices; ++i) { + for (size_t i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); cudaStream_t stream = nullptr; @@ -495,7 +495,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n // the previous slot's. Indexed by (slot * buf_bytes) at the call site. p->buf_bytes = GGML_CUDA_AR_MAX_BYTES; const size_t host_buf_total = (size_t) GGML_CUDA_AR_POOL_SIZE * p->buf_bytes; - for (int i = 0; i < n_devices; ++i) { + for (size_t i = 0; i < n_devices; ++i) { if (p->host_buf[i].alloc(host_buf_total) != cudaSuccess) { GGML_LOG_ERROR("%s: alloc for staging failed (%zu bytes)\n", __func__, host_buf_total); @@ -508,7 +508,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n // largest tensor we accept on this path (GGML_CUDA_AR_COPY_MAX_BYTES). // dev_tmp is single-buffered; cross-AR safety is enforced by an explicit // cross-stream wait in copy_impl on the prior AR's add_kernel-done event. - for (int i = 0; i < n_devices; ++i) { + for (size_t i = 0; i < n_devices; ++i) { ggml_cuda_set_device(p->devices[i]); if (p->host_large[i].alloc(p->copy_bytes) != cudaSuccess) { GGML_LOG_ERROR("%s: alloc for large staging failed (%zu bytes)\n", @@ -524,7 +524,7 @@ ggml_cuda_ar_pipeline * ggml_cuda_ar_pipeline_init(const int * devices, size_t n } } - GGML_LOG_INFO("%s: initialized AllReduce pipeline: %d GPUs, " + GGML_LOG_INFO("%s: initialized AllReduce pipeline: %zu GPUs, " "%zu KB chunked kernel staging + %zu MB copy-engine staging per GPU\n", __func__, n_devices, p->buf_bytes >> 10, p->copy_bytes >> 20); @@ -725,8 +725,8 @@ static bool ggml_cuda_ar_allreduce_copy_outer( const int64_t outer_ne = std::min(outer_max_elems, ne - outer_start); const size_t outer_nbytes = (size_t) outer_ne * sizeof(T_src); - T_src * src[GGML_CUDA_MAX_DEVICES]; - T_dst * dst[GGML_CUDA_MAX_DEVICES]; + T_src * src[GGML_CUDA_MAX_DEVICES] = {}; + T_dst * dst[GGML_CUDA_MAX_DEVICES] = {}; for (int i = 0; i < p->n_devices; ++i) { src[i] = src_buf[i] + outer_start; dst[i] = dst_buf[i] + outer_start; @@ -836,8 +836,8 @@ bool ggml_cuda_ar_allreduce( // post-conversion is needed. Otherwise src == dst (same native type). if (use_bf16) { GGML_ASSERT(kernel_type == GGML_TYPE_BF16); - nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES]; - float * dst[GGML_CUDA_MAX_DEVICES]; + nv_bfloat16 * src[GGML_CUDA_MAX_DEVICES] = {}; + float * dst[GGML_CUDA_MAX_DEVICES] = {}; for (int i = 0; i < n; ++i) { src[i] = static_cast(copy_src_ptr[i]); dst[i] = static_cast(tensors[i]->data); @@ -847,7 +847,7 @@ bool ggml_cuda_ar_allreduce( } else { switch (kernel_type) { case GGML_TYPE_F32: { - float * buf[GGML_CUDA_MAX_DEVICES]; + float * buf[GGML_CUDA_MAX_DEVICES] = {}; for (int i = 0; i < n; ++i) { buf[i] = static_cast(tensors[i]->data); } @@ -856,7 +856,7 @@ bool ggml_cuda_ar_allreduce( break; } case GGML_TYPE_BF16: { - nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES]; + nv_bfloat16 * buf[GGML_CUDA_MAX_DEVICES] = {}; for (int i = 0; i < n; ++i) { buf[i] = static_cast(tensors[i]->data); } @@ -865,7 +865,7 @@ bool ggml_cuda_ar_allreduce( break; } case GGML_TYPE_F16: { - half * buf[GGML_CUDA_MAX_DEVICES]; + half * buf[GGML_CUDA_MAX_DEVICES] = {}; for (int i = 0; i < n; ++i) { buf[i] = static_cast(tensors[i]->data); } From 8fd5f57f3d2e3d132394a76ce973e3c8349394a1 Mon Sep 17 00:00:00 2001 From: Scott Cutler Date: Sat, 9 May 2026 16:42:09 -0700 Subject: [PATCH 81/81] ggml-cuda: drop unused-function warning by guarding try_allreduce_nccl behind GGML_USE_NCCL The only call site (in init_nccl) is already inside #ifdef GGML_USE_NCCL, so the function is unreferenced in non-NCCL builds and trips nvcc's -Werror=unused-function check. Move the guard from inside the function body to around the entire definition. Co-Authored-By: Claude Opus 4.7 (1M context) ce reduction), so the __restrict__ qualifiers on those parameters were technically UB. Drop __restrict__ from sendbuf and recvbuf; an A/B sweep showed <0.6% perf delta (within noise) on Linux. - The buf/src/dst pointer arrays in ggml_cuda_ar_allreduce and the per-iteration arrays in ggml_cuda_ar_allreduce_copy_outer were declared with size GGML_CUDA_MAX_DEVICES but the loop only writes indices [0, n_devices); zero-initialise so the compiler sees the tail elements as defined. Co-Authored-By: Claude Opus 4.7 (1M context) now +6-8% ahead at ub=1024-4096. Perplexity (32 chunks) matches NCCL bit-for-bit (3.4044 vs 3.4043). Co-Authored-By: Claude Opus 4.7 (1M context) --- ggml/src/ggml-cuda/ggml-cuda.cu | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index f309e04d116..f367a9eff97 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -1309,15 +1309,12 @@ static bool ggml_backend_cuda_comm_allreduce_internal( // returns false; the meta backend's generic implementation handles that call. // --------------------------------------------------------------------------- +#ifdef GGML_USE_NCCL static bool ggml_backend_cuda_comm_try_allreduce_nccl( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) { -#ifdef GGML_USE_NCCL return ggml_backend_cuda_comm_allreduce_nccl(comm_ctx, tensors); -#else - GGML_UNUSED(comm_ctx); GGML_UNUSED(tensors); - GGML_ABORT("try_allreduce_nccl unreachable: built without NCCL"); -#endif // GGML_USE_NCCL } +#endif // GGML_USE_NCCL static bool ggml_backend_cuda_comm_try_allreduce_internal( ggml_backend_cuda_comm_context * comm_ctx, struct ggml_tensor ** tensors) {