CUDA: use radix-select TOP_K in the CUB fallback for wide rows

[Inovello]

Overview

When CUB has no DeviceTopK (every CCCL older than 3.2), ggml_cuda_op_top_k falls back to sorting every key of every row with a segmented radix sort and copying the first k. For wide rows the cost of the sort is large, the qwen4exp QSA indexer runs TOP_K over 4 rows x n_kv columns with k = 2051,and at 131k context that sort costs 5.1 ms per generated token which is about a third of the GPU time per token.

This PR makes the existing radix-select kernel (Until now this has been HIP-only) available to CUB builds without DeviceTopK and uses it in that fallback for rows of >= 8192 columns. The threshold was measured, it wasn't estimated or guessed. On an RTX 3090 (CUDA 12.0, CUB 2.0.1), the segmented sort was still faster at 4096 columns and the radix-select was faster from 8192 on. Builds that have CCCL >= 3.2 keep using DeviceTopK. HIP, MUSA and no-CUB builds are unchanged by this. GGML_CUDA_TOPK_ARGSORT=1 forces the old path so both can be compared within one binary.

Additional information

Correctness: test-backend-ops -o TOP_K passes 525/525 on both paths (268 cases at >= 8192 columns, up to 524,299 columns, k up to 9,999, 112 with ties).

Kernel only timings (test-backend-ops perf, TOP_K with k = 16, CUDA graphs off, GPU shared with a running server): on all 46 test shapes with rows of 8192 columns or more, radix-select beats the segmented sort by about 1.42x at worst and 2.25x in the median and up to 11x for 16-row batches. On 4096-column rows, the sort was still faster (24.1 us vs 38.9 us for radix-select), so the switch happens at 8192 columns rather than the 1024 proposed in the earlier unmerged #28366.

End to end (Qwen3.8-Flash-Next Q4_K_XL, 131k context, MTP draft, same binary, kill-switch A/B, two windows run in both orders): Nsight Systems traces show the top-k kernels dropping from 5.1 to 0.25 ms per generated token. Decode on cached 128-token continuations increased from 30.3 to 34.9 t/s, +13 to +18% on every one of six position-paired runs, and a fresh 128-token request after the prefill gained +10% as well (one pair). Prefill was unchanged as expected considering the sort does not run there. A separate production-sampling quality screen (480 requests at ~37k and ~119k context, three seeds, argsort path as the control) found no consistent quality regression and 9 to 12% higher decode throughput at ~119k on every seed.

Some caveats:

• Tie behaviour differs. The segmented sort always kept the lower index so its output was repeatable. Radix-select, like CUB's DeviceTopK keeps whichever of the exactly tied values it reaches first so the selected set can differ between runs when ties occur. Any tied choice is a valid top-k result and test-backend-ops accepts it. This is the same behaviour Eval bug: qwen4exp QSA indexer top-k selects a different cell set each run on CUDA (CUB DeviceTopK over tied block scores) #28497 describes for DeviceTopK.
• The 4096/8192 crossover was measured at k = 16 without CUDA graphs on one GPU model. Other k and other GPUs may cross elsewhere. The threshold of 8192 is conservative for the 3090.
• Not measured: MUSA and no-CUB builds (unchanged by this PR), other k at the boundary, an idle GPU.

This was measured on 2x RTX 3090 (sm_86), driver 595.84, CUDA 12.0.140, CUB 2.0.1, where CUB_TOP_K_AVAILABLE is undefined and this fallback is the active path.

Related earlier work: #28366 proposed the same dispatch with a 1024-column threshold and no measurements. It was closed unmerged.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. The code change was written by an AI assistant (Claude) under my direction and reviewed by me. All builds, tests, profiles and quality measurements were run by me on my own hardware. This description and the commit message are my own words.

[ggml-gh-bot]

Hi @Inovello, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[Inovello]

Closed PR #28223 until this one has been resolved to comply with the contribution guidelines. Thanks for flagging it! (Yes I am thanking a bot)