Skip to content

hexagon: add HTP backend support for GGML_OP_CLAMP - #25925

Closed
cqderek wants to merge 1 commit into
ggml-org:masterfrom
cqderek:hexagon-clamp-op
Closed

hexagon: add HTP backend support for GGML_OP_CLAMP#25925
cqderek wants to merge 1 commit into
ggml-org:masterfrom
cqderek:hexagon-clamp-op

Conversation

@cqderek

@cqderek cqderek commented Jul 20, 2026

Copy link
Copy Markdown

CLAMP was falling through to default:break in device_supports_op(), causing the scheduler to dispatch it to CPU on every call. On MoE models (e.g. Qwen3-30B-A3B) this triggers 224 CPU<->HTP FastRPC round-trips per forward pass due to per-layer expert routing weight normalization, adding significant fixed latency in the decode path.

Fix: wire GGML_OP_CLAMP through the existing op_unary framework, reusing the already-present hvx_clamp_scalar_f32() HVX kernel. No new HVX vector code is introduced; all changes are plumbing only.

Benchmark (Q4_0, SM8850 HTP, NDEV=1, -t 4, tg50 x5):
Before: 18.61 +/- 0.17 tok/s (224 CPU splits/graph)
After: 20.24 +/- 0.28 tok/s (0 CPU splits for CLAMP)
Gain: +10.4%"

Overview

GGML_OP_CLAMP was falling through to default: break in device_supports_op(), causing the scheduler to dispatch it to CPU on every call. On MoE models (e.g. Qwen3-30B-A3B) this triggers 224 CPU<->HTP FastRPC round-trips per forward pass due to per-layer expert routing weight normalization, adding significant fixed latency in the decode path.

Fix: wire GGML_OP_CLAMP through the existing op_unary framework, reusing the already-present hvx_clamp_scalar_f32() HVX kernel. No new HVX vector code is introduced; all changes are plumbing only.

Benchmark (Q4_0, SM8850 HTP, NDEV=1, -t 4, tg50 x5):

Before: 18.61 +/- 0.17 tok/s  (224 CPU splits/graph)
After:  20.24 +/- 0.28 tok/s  (0 CPU splits for CLAMP)
Gain:   +10.4%

## Additional information

Same fix benefits any model using `GGML_OP_CLAMP` on the HTP backend (e.g. Gemma 2/3 attention logit softcapping). Changes touch `ggml-hexagon.cpp` (op remap + `device_supports_op` routing), `htp-ops.h` (new `HTP_OP_CLAMP` enum), `htp/main.c` (dispatch), and `htp/unary-ops.c` (row-block and tiled clamp implementations, following the existing `SCALE`/`SQRT` pattern).

## Requirements

<!-- IMPORTANT: Please do NOT delete this section, otherwise your PR may be rejected -->

- I have read and agree with the [contributing guidelines](https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md)
- AI usage disclosure: <!-- mention: YES / NO - if yes, describe how AI was used -->

CLAMP was falling through to default:break in device_supports_op(),
causing the scheduler to dispatch it to CPU on every call. On MoE
models (e.g. Qwen3-30B-A3B) this triggers 224 CPU<->HTP FastRPC
round-trips per forward pass due to per-layer expert routing weight
normalization, adding significant fixed latency in the decode path.

Fix: wire GGML_OP_CLAMP through the existing op_unary framework,
reusing the already-present hvx_clamp_scalar_f32() HVX kernel.
No new HVX vector code is introduced; all changes are plumbing only.

Benchmark (Q4_0, SM8850 HTP, NDEV=1, -t 4, tg50 x5):
  Before: 18.61 +/- 0.17 tok/s  (224 CPU splits/graph)
  After:  20.24 +/- 0.28 tok/s  (0 CPU splits for CLAMP)
  Gain:   +10.4%"
@cqderek
cqderek requested a review from a team as a code owner July 20, 2026 13:31
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Jul 20, 2026
@ggml-gh-bot

ggml-gh-bot Bot commented Jul 20, 2026

Copy link
Copy Markdown

Hi @cqderek, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

  • PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

@cqderek

cqderek commented Jul 20, 2026

Copy link
Copy Markdown
Author

@max-krasnyansky could you please review this? This PR adds HTP backend support for GGML_OP_CLAMP.

@max-krasnyansky

Copy link
Copy Markdown
Member

@max-krasnyansky could you please review this? This PR adds HTP backend support for GGML_OP_CLAMP.

Please update the PR description (AI usage disclosure, etc).
Overall looks good.

@max-krasnyansky

Copy link
Copy Markdown
Member

@cqderek I realized that we already had this queued up internally #25934
I'm going to close this one. The implementation is basically the same since we already had hvx_clamp in the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Hexagon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants