ggml: treat experts as cache residents during MoE offloading

[avifenesh]

Overview

When MoE weights are offloaded from host memory, the scheduler already copies only the experts used by the current GGML_OP_MUL_MAT_ID split. In long prompt-cache workloads, the same split-input staging tensor can be reused across turns, so many experts that were copied for earlier turns are still resident on the backend.

This change tracks resident MoE experts per scheduler tensor copy and treats the split-input staging tensor as an expert cache:

• tracks loaded expert IDs with a ggml_bitset_t
• computes missing_ids = used_ids & ~loaded_ids
• copies only missing contiguous expert ranges
• skips the split-input overwrite wait when no missing copy is needed
• invalidates resident state when n_expert, expert_size, source data, or destination staging data changes

The change is internal to ggml/src/ggml-backend.cpp. It does not add a user-facing flag, public API, or backend-specific implementation.

Additional information

Benchmark setup:

• Base: upstream/master at 64b38b561
• Hardware: NVIDIA GeForce RTX 5090 Laptop GPU, 23981 MiB VRAM
• Backend/build: CUDA 13.1, build-cuda13-clean/bin/llama-server
• Server mode: --cpu-moe
• Expert staging: default op offload enabled
• Context: -c 65536
• Prompt cache: enabled
• Workload: sequential medium-turn chat shape, 12 turns, about 5k new prompt tokens per turn, filling the prompt to about 60k tokens
• Generation: 64 tokens per request
• Warmup: none for this context-fill workload
• Reported throughput: processed prompt tokens/sec, so prompt-cache reuse does not inflate the speedup

Model	Max prompt tokens	Baseline duration	Patched duration	Speedup	Baseline avg latency	Patched avg latency	Baseline processed prompt tok/s	Patched processed prompt tok/s
Qwen3.6 35B A3B UD-Q4_K_M	61k	198.76s	44.91s	4.43x	16.49s	3.67s	307.83	1348.60
Gemma 4 26B A4B Q4_K_M	61k	152.69s	57.96s	2.63x	12.66s	4.77s	399.80	1041.12

Validation:

• git diff --check
• cmake --build build-cuda13-clean --target llama-server test-backend-ops -j 10
• build-cuda13-clean/bin/test-backend-ops test -o MUL_MAT_ID
  • CUDA0: 764/764 tests passed
  • Overall: 2/2 backends passed
• Error scan across the benchmark logs found no CUDA error, OOM, ASSERT, Traceback, abort, or similar failure strings.

Related upstream search:

• No exact duplicate found for resident MoE expert caching or CPU-MoE/op-offload staging reuse.
• Nearby file-overlap PRs to be aware of:
  • ggml: allow prefetching tensor overrides ggml-org/llama.cpp#21067
  • ggml: use dynamic allocation for split graph inputs ggml-org/llama.cpp#22789

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Codex assisted with local code review, benchmarking, cleanup, and preparing this fork-local staging PR. Before opening an upstream PR, I will manually review the patch and write/edit the final upstream PR text myself.

[gemini-code-assist]

Code Review

This pull request optimizes Mixture of Experts (MoE) offloading by tracking which experts are already resident on the backend, ensuring only missing experts are copied during graph splits. This is achieved by introducing a tracking structure within the scheduler and refactoring the copy logic. Review feedback recommends adding null checks for memory allocations to ensure the scheduler remains robust under memory pressure.

[gemini-code-assist]

The memory allocation for loaded->ids using calloc should be checked for failure, especially since expert_id_size could be large depending on the model configuration.

[gemini-code-assist]

The allocation of hv_tensor_moe_loaded using calloc should be checked for NULL to ensure the scheduler initialization is robust against memory pressure.