internal AllReduce kernel for CUDA provider - #22299

Merged

JohannesGaessler merged 84 commits into

ggml-org:masterfrom

scutler-nv:scutler/internal-allreduce

May 10, 2026

Commits on Apr 22, 2026

ggml-cuda: add internal AllReduce provider for tensor parallelism

scutler-nv
and
claude
committed
4c11786
llama-bench: add --allreduce flag to select AllReduce provider

scutler-nv
and
claude
committed
52af99d
llama-bench: rename --allreduce to --reduction-provider / -rp

scutler-nv
and
claude
committed
8c0a791
llama-bench: pass WARN/ERROR log messages through in non-verbose mode

scutler-nv
and
claude
committed
10c4757
cmake: improve NCCL detection for source-tree builds, add static/dynamic switch

scutler-nv
and
claude
committed
2c1a1db
ggml-cuda: add AllReduce hang watchdog (GGML_CUDA_AR_WATCHDOG)

scutler-nv
and
claude
committed
c5207a4
ggml-cuda: fix intermittent AllReduce hang on Blackwell PCIe

scutler-nv
and
claude
committed
172cba2

Commits on Apr 23, 2026

ggml-cuda: fix watchdog shutdown ordering and pipeline_free drain

scutler-nv
and
claude
committed
3b584e1
ggml-cuda: replace event-based watchdog with per-GPU ring buffer

scutler-nv
and
claude
committed
860ee2a
fix: normalize line endings to LF (undo Windows CRLF conversion)

scutler-nv
and
claude
committed
433c35a
.gitattributes: force LF line endings to prevent Windows CRLF conversion

scutler-nv
and
claude
committed
bc8b080
ggml-cuda: move GGML_CUDA_AR_WATCHDOG from CMake option to local define

scutler-nv
and
claude
committed
8da7e74
unify kernel debug paths
scutler-nv
committed
372d408
use __threadfence_system explicitly (not in ggml_cuda_ar_signal_set)
scutler-nv
committed
5028250
preferentially use internal reduction for <=2 GPUs
scutler-nv
committed
cfdb0a6
templatize the main kernel to support fp16/bf16
scutler-nv
committed
239f20e
restore llama-bench.cpp changes
scutler-nv
committed
cfcaee3
revert CMakeLists changes
scutler-nv
committed
bb01919
remove notes from repo
scutler-nv
committed
ebc31bc
remove dead warmup code
scutler-nv
committed
014ad9f
fix comments
scutler-nv
committed
a6981b5
improve reduction provider fallback code
scutler-nv
committed
77c0e36
add messages for allreduce fallback
scutler-nv
committed
43743e5

Commits on Apr 24, 2026

rework reduction provider init to not call ncclCommInitAll if using the internal provider
scutler-nv
committed
2573b7b
fix case where a given tensor has not been computed
scutler-nv
committed
4d7736c
add chunked mode to the kernel for unlimited vector size
scutler-nv
committed
f046004
rework a few checks/fallbacks
scutler-nv
committed
fbcae51
Merge branch 'master' into dev/internal-allreduce
scutler-nv
committed
f3d321f
various small cleanups
scutler-nv
committed
7449c1d
allow disabling CUDA reductions completely (falling back to the non-CUDA butterfly mode)
scutler-nv
committed
2b91e21
simplify reduction provider selection
scutler-nv
committed
485ee23
minor simplifications
scutler-nv
committed
892b2e3
more cleanups/fixes
scutler-nv
committed
b34adf4

Commits on Apr 25, 2026

prototype alternate path for large reductions
scutler-nv
committed
b8caad3
chunked version of large reduction path
scutler-nv
committed
b0bdf83

Commits on Apr 27, 2026

use bf16 for large reductions
scutler-nv
committed
77b96e6

Commits on Apr 28, 2026

experimental reduction using cudaMemcpyPeerAsync (slightly slower)
scutler-nv
committed
0805e6a
revert experimental change
scutler-nv
committed
97eae7c
add combined conversion/reduction kernel
scutler-nv
committed
3e8b263
add bf16 wire format for single kernel mode
scutler-nv
committed
80faf56
experimental on-stream small reduction kernel
scutler-nv
committed
210165b
double buffer arrival slots, use token (incrementing) method
scutler-nv
committed
fbc2bd7
double buffer host_buf for small reductions
scutler-nv
committed
3a41c7b
put in waits for use of host_mem in large reduction case (prevents stomping on in-use memory
scutler-nv
committed
330c14c
remove watchdog code
scutler-nv
committed
2b91d02
various cleanups / dead code removal
scutler-nv
committed
ff4ed48
fix fp16 mode
scutler-nv
committed
25ff620
fix some comments/logging statements
scutler-nv
committed
26c3d01
use increasing token scheme for arrival signals
scutler-nv
committed
83b9bac
add top-level comment to allreduce.cu
scutler-nv
committed
1fe1528
improve top-level comment in allreduce.cu
scutler-nv
committed
630a800
fix comments in ggml_cuda_ar_kernel
scutler-nv
committed
6d4b998

Commits on Apr 29, 2026

improve event handling for hostmem buffer usage tracking
scutler-nv
committed
63608b7
change ev_pool to fixed 2D array
scutler-nv
committed
be783f9
Merge branch 'master' into dev/internal-allreduce
scutler-nv
committed
f4a4ce2

Commits on Apr 30, 2026

Commits on May 1, 2026

more PR fixes
scutler-nv
committed
bfa0b69
fix semantics of all host mappings
scutler-nv
committed
0f9bea3
require ampere+
scutler-nv
committed
132934e
small cleanups
scutler-nv
committed
1854ebc

Commits on May 4, 2026

properly use host pointer for src/dst in cudaMemcpy calls
scutler-nv
committed
6e9319c

Commits on May 5, 2026

allreduce: lazy-init the internal pipeline on first use

scutler-nv
and
claude
committed
a19d711
allreduce: assert n_backends == 2 instead of soft-fallback

scutler-nv
and
claude
committed
5d1b22a

Commits on May 6, 2026

Commits on May 7, 2026

allreduce: expand one-liner for loops to braced bodies

scutler-nv
and
claude
committed
605f5e0
allreduce: rename template parameters Tdst/Twire/Tsrc -> T_dst/T_wire/T_src

scutler-nv
and
claude
committed
77e9b14
allreduce: drop hyphen in 'chunked-kernel' across comments

scutler-nv
and
claude
committed
b5e18bc
allreduce: use ggml_cuda_get_max_cpy_bytes() instead of hardcoded 16

scutler-nv
and
claude
committed
f4c8b26
Merge remote-tracking branch 'origin/master' into dev/internal-allreduce
scutler-nv
committed
d816e47

Commits on May 8, 2026

ggml-cuda: PR review fixes -- annotate #endif, fix stale comment, assert nbytes alignment

scutler-nv
and
claude
committed
29b11cf
simplify the init logic
scutler-nv
committed
6ae3093
address some other PR requests
scutler-nv
committed
d64b3b1

Commits on May 9, 2026