hexagon: support for device discovery and create sessions on demand - #27785
Conversation
github-actions
Bot
added
ggml
|
Tested on:
(test results from IQ10 are coming a bit later) |
…interfaces Co-authored-by: Marco Colombo <mcolombo@qti.qualcomm.com>
Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> Co-authored-by: Ehsan Bateni <ebateni@qti.qualcomm.com>
c208ad3 to
898e766
Compare
|
@max-krasnyansky Thanks for the update. We rebased our SA8797P changes onto the latest version of this PR and successfully validated it on the Android guest. Runtime FastRPC domain discovery now correctly identifies all four physical NSPs (nsp1000–nsp1003). Our previous local domain-discovery workaround is no longer required. Multi-NSP tensor split and streaming inference are both working correctly. We found one separate Hexagon issue that may be worth discussing. ggml_hexagon_supported_mul_mat() currently rejects quantized matrix multiplications when src0->ne[1] > 32768. Qwen3-VL has a 151936-row output head, so the LM head otherwise falls back to the CPU. We added an opt-in configurable row limit locally and verified that the larger LM-head matmul runs correctly on SA8797P. Would you prefer this limit to be exposed as a backend configuration option, or replaced with a capability/resource-based check? We can prepare this as a separate PR if useful. Thanks again for implementing the runtime domain discovery support. |
Yep. Confirmed internally as well. Thank you for the update.
Ha. I didn't even think of this in the multi-NPU context. |
max-krasnyansky
added
the
merge ready
|
@lhez please review/ack when you get a chance |
|
e: ggml_hexagon_supported_mul_mat() ne1 > 32768 → lm-head CPU fallback
|
|
@CreaV @GaTmaNnes |
|
hanks Max. This is very relevant to what I have been measuring on an SM8850 device with Qwen3.5-9B-D2-A-MTP. I have been profiling the Hexagon HTP path quite extensively, and one result seems particularly relevant to the solver/fusion changes you mention. On the same model and HTP configuration, I selectively requantized the attention weights from Q8_0 to Q4_0: model: 7.29 GiB → 5.07 GiB (~30% smaller) I also confirmed from the backend code that activations are still dynamically quantized to Q8_0_TILED for these matmuls, so the weight reduction does not eliminate the activation quantization/repack path. This makes me suspect that the remaining decode bottleneck is not simply the amount of weight data or raw MUL_MAT compute, but the surrounding execution pipeline: VTCM preparation, buffer management, op batching/queueing, synchronization and possibly fusion opportunities. Your #28202 changes to QKV/FFN fusion, the VTCM solver, OPBATCH/OPQUEUE sizing and VA buffer fragmentation therefore look particularly relevant. I am going to benchmark the same workload against #28202 and compare: MUL_MAT count → fused ops → DSP time → VTCM usage → DMA events → OPQUEUE/OPBATCH → wall-clock decode One additional observation: MXFP4 attention at essentially the same model size as Q4_0 gave ~6.6 tok/s versus ~5.9 tok/s for Q4_0 in one controlled run, despite very similar DMA event counts. So there may also be a kernel/data-layout effect beyond simple byte count. If you have any specific profiling counters or traces that would be useful for validating whether the new fusion/solver path is actually removing pipeline overhead on SM8850, I would be happy to collect them |
|
NEXA / GENIEX ENQUÊTE Date : Portée : HTP<->GPU Investigation de régression à l’exécution ======================================================================
L’enquête établit plusieurs faits indépendants.
SourceForge héberge un miroir exact du projet Nexa SDK et identifie explicitement https://github.com/NexaAI/nexa-sdk Le miroir contient des archives sources historiques et des paquets exécutables.
https://github.com/zhycheng614/nexa-sdk-executable Il décrit Nexa SDK comme supportant à la fois GGML et ONNX ainsi que plusieurs
https://github.com/qualcomm/nexa-sdk Cela est important car le dépôt historique NexaAI et la
7,29 Gio -> 5,07 Gio poids du modèle n’a PAS significativement amélioré le débit de décodage de l’horloge murale.
HTP0 : 7,68 t/s Cependant, ces mesures étaient thermiquement contaminées et
Il a rapporté : 7 254 incompatibilités plan/identité durant la diffusion du MTP. Fait crucial, une série ultérieure hors MTP a également produit : 2 662 désaccords LesÀ l’effet, le phénomène de décalage n’est PAS uniquement causé par le MTP.
Le diagnostic lui-même indique explicitement : « possible corruption silencieuse si le plan est réutilisé ». C’est un signal sérieux d’allocateur/validité du plan.
41 acceptés / 62 acceptés générés Cette stabilité est une preuve utile que le mécanisme du MTP lui-même fonctionne
Premier JZ A/B : Comparaison antérieure plus froide du JZ : La variation suit la température suffisamment fortement pour que ces différences
C’est une preuve solide que le chemin MTP contient un important composant de contrôle/orchestration côté
environ 38-40 °C au début . Par conséquent, l’état thermique est une variable expérimentale de première classe.
La bonne étape suivante est : RESTAURER / IDENTIFIER L’EXÉCUTION
|
================================================================================
|
|
Date: 2026-09-02 1. Executive SummaryThis campaign introduces a sparse-MoE control workload to the Qualcomm Hexagon HTP investigation. The purpose is to separate the cost of active matrix computation from the additional costs of MoE routing, expert dispatch, data movement, synchronization, and host/DSP orchestration during batch-1 decode. The tested workload was Marco-Nano-Instruct, converted from The clean result was: The dense Qwen reference from the same campaign was: The comparison is: Metric | Dense Qwen MTP1 | Marco-Nano sparse MoE -- | -- | -- End-to-end decode | 11.26 tok/s | 26.72 tok/s Time/token | 88.8 ms | 37.4 ms DSP MUL_MAT / token | ~65 ms | ~8.7 msThe lower confidence values intentionally reflect the need for direct event-level wall-clock attribution. 28. Preserved ArtifactsHost model directory: Primary files: Q4_0 size: Report: Validation: Raw PMU/profile logs are preserved for re-analysis. |
…gml-org#27785) * hex-devices: add support for lazy session allocation and cleanup dev interfaces Co-authored-by: Marco Colombo <mcolombo@qti.qualcomm.com> * hex-devices: support for runtime discovery of available NPU cores Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> Co-authored-by: Ehsan Bateni <ebateni@qti.qualcomm.com> * hex-devices: reject non-existing devices early during init --------- Co-authored-by: Marco Colombo <mcolombo@qti.qualcomm.com> Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> Co-authored-by: Ehsan Bateni <ebateni@qti.qualcomm.com>
…gml-org#27785) * hex-devices: add support for lazy session allocation and cleanup dev interfaces Co-authored-by: Marco Colombo <mcolombo@qti.qualcomm.com> * hex-devices: support for runtime discovery of available NPU cores Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> Co-authored-by: Ehsan Bateni <ebateni@qti.qualcomm.com> * hex-devices: reject non-existing devices early during init --------- Co-authored-by: Marco Colombo <mcolombo@qti.qualcomm.com> Co-authored-by: Alexander Lu <alexlu@qti.qualcomm.com> Co-authored-by: Ehsan Bateni <ebateni@qti.qualcomm.com>
5 participants
Overview
Minor overhaul of how
ggml-hexagoncreates devices and sessions.The original implementation created session early during backend init.
Since the number of supported sessions is limited it makes sense to create them only when actually needed for loading & running a model.
This change enables running
llama-serverin "router mode" without wasting available sessions.This update also adds support for NPU discovery at run-time on on devices that support newer API
(like the IQ10, and the latest Android phones).
If the device discover is not supported we fallback to the original hardcoded NPU/CDSP domain values.
Requirements