From 4668af0771591f13215a256d3980e485d69b6650 Mon Sep 17 00:00:00 2001 From: Kyle Romero Date: Tue, 12 May 2026 15:15:28 -0700 Subject: [PATCH 1/5] Add QNN EP documentation --- docs/qnn.md | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100644 docs/qnn.md diff --git a/docs/qnn.md b/docs/qnn.md new file mode 100644 index 0000000000..0d2de7ec6f --- /dev/null +++ b/docs/qnn.md @@ -0,0 +1,65 @@ +# QNN Execution Provider + +ONNX Runtime GenAI supports running models on Qualcomm Snapdragon NPUs via the QNN Execution +Provider (EP). This enables hardware-accelerated LLM inference on Snapdragon-based devices on +Windows ARM64 and Linux ARM64. + +## Prerequisites + +**Packages** + +- `onnxruntime-genai` — the OGA runtime +- `onnxruntime-qnn` — the QNN EP plugin ([PyPI](https://pypi.org/project/onnxruntime-qnn/) / + [NuGet](https://www.nuget.org/packages/Microsoft.ML.OnnxRuntime.QNN)) + +**Model** + +QNN inference requires a model prepared for the QNN EP, packaged as an ONNX file containing +compiled QNN graph artifacts. The `genai_config.json` and model files are produced by the +[olive-recipes](https://github.com/microsoft/olive-recipes) pipeline targeting the QNN backend. + +## Usage + +The QNN EP is a plugin and must be registered before loading the model. Pass the path to the +`onnxruntime_providers_qnn` shared library from the `onnxruntime-qnn` package +(`onnxruntime_providers_qnn.dll` on Windows, `libonnxruntime_providers_qnn.so` on Linux): + +```python +import onnxruntime_genai as og + +# Register the QNN EP plugin before loading the model. +# The library is included in the onnxruntime-qnn package. +og.register_execution_provider_library( + "QNNExecutionProvider", + "/path/to/onnxruntime_providers_qnn.dll" # or libonnxruntime_providers_qnn.so on Linux +) + +# Load model. The genai_config.json from the olive-recipes pipeline already +# specifies the QNN provider, so no further config is needed in the common case. +# To override options, use og.Config directly: +# config = og.Config("/path/to/model") +# config.set_provider_option("QNN", "htp_performance_mode", "burst") +# model = og.Model(config) +model = og.Model("/path/to/model") +tokenizer = og.Tokenizer(model) + +params = og.GeneratorParams(model) +params.set_search_options(max_length=200) +generator = og.Generator(model, params) +generator.append_tokens(tokenizer.encode("What color is the sky?")) + +while not generator.is_done(): + generator.generate_next_token() + print(tokenizer.decode(generator.get_next_tokens()[0]), end="", flush=True) +print() +``` + +## Provider Options + +Options are set via `config.set_provider_option("QNN", key, value)`. + +| Option | Description | Values | +|---|---|---| +| `htp_performance_mode` | NPU power/performance profile | `burst`, `balanced`, `high_performance`, `power_saver` | +| `vtcm_mb` | VTCM allocation size in MB | `"8"`, `"16"`, etc. | +| `enable_htp_shared_memory_allocator` | Enable shared memory allocator for direct OGA/QNN tensor handoff | `"1"` to enable | From 122ce783d611dfe40246dddca94f2d5310d52b50 Mon Sep 17 00:00:00 2001 From: Kyle Romero Date: Tue, 12 May 2026 15:47:25 -0700 Subject: [PATCH 2/5] Add note on Genie API integration via DLC EPContext models --- docs/qnn.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/qnn.md b/docs/qnn.md index 0d2de7ec6f..f16c42050d 100644 --- a/docs/qnn.md +++ b/docs/qnn.md @@ -4,6 +4,10 @@ ONNX Runtime GenAI supports running models on Qualcomm Snapdragon NPUs via the Q Provider (EP). This enables hardware-accelerated LLM inference on Snapdragon-based devices on Windows ARM64 and Linux ARM64. +ONNX models containing an EPContext node with `ep_context_type: "dlc"` are routed directly +through the Genie API, enabling optimized LLM inference on the NPU. Models in this format are +produced by the [olive-recipes](https://github.com/microsoft/olive-recipes) pipeline. + ## Prerequisites **Packages** From c0df352dfd3f9359b6a3424abe64ef83678aad46 Mon Sep 17 00:00:00 2001 From: Kyle Romero Date: Wed, 13 May 2026 11:30:30 -0700 Subject: [PATCH 3/5] Address review comments: fix tokenizer decode, clarify Genie API, clarify EP naming - Use TokenizerStream for token-by-token decode (Tokenizer.decode expects an array, not a scalar) - Expand Genie API description: define it, note transparency to users, and clarify that only newer QAIRT-targeting Olive recipes produce the DLC EPContext type; older QNN recipes do not - Add comment clarifying QNNExecutionProvider (registration) vs QNN (options) --- docs/qnn.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/qnn.md b/docs/qnn.md index f16c42050d..92459f73ca 100644 --- a/docs/qnn.md +++ b/docs/qnn.md @@ -5,8 +5,12 @@ Provider (EP). This enables hardware-accelerated LLM inference on Snapdragon-bas Windows ARM64 and Linux ARM64. ONNX models containing an EPContext node with `ep_context_type: "dlc"` are routed directly -through the Genie API, enabling optimized LLM inference on the NPU. Models in this format are -produced by the [olive-recipes](https://github.com/microsoft/olive-recipes) pipeline. +through the Genie API — Qualcomm's optimized LLM inference runtime, part of the QAIRT SDK, +which provides accelerated token generation on the Snapdragon NPU. This routing is transparent +to OGA users; no additional configuration is required. Models in this format are produced by +newer QAIRT-targeting [olive-recipes](https://github.com/microsoft/olive-recipes) pipelines. +Older QNN-targeting recipes produce models without the DLC EPContext type and do not use the +Genie pathway. ## Prerequisites @@ -33,6 +37,8 @@ import onnxruntime_genai as og # Register the QNN EP plugin before loading the model. # The library is included in the onnxruntime-qnn package. +# Note: registration uses the full EP name "QNNExecutionProvider"; provider options +# (Config.set_provider_option / genai_config.json) use the short name "QNN". og.register_execution_provider_library( "QNNExecutionProvider", "/path/to/onnxruntime_providers_qnn.dll" # or libonnxruntime_providers_qnn.so on Linux @@ -46,6 +52,7 @@ og.register_execution_provider_library( # model = og.Model(config) model = og.Model("/path/to/model") tokenizer = og.Tokenizer(model) +tokenizer_stream = tokenizer.create_stream() params = og.GeneratorParams(model) params.set_search_options(max_length=200) @@ -54,7 +61,7 @@ generator.append_tokens(tokenizer.encode("What color is the sky?")) while not generator.is_done(): generator.generate_next_token() - print(tokenizer.decode(generator.get_next_tokens()[0]), end="", flush=True) + print(tokenizer_stream.decode(generator.get_next_tokens()[0]), end="", flush=True) print() ``` From 94ef3054a286908c4fd6f1f62ffb2dc4a54247a8 Mon Sep 17 00:00:00 2001 From: Kyle Romero Date: Thu, 14 May 2026 11:58:12 -0700 Subject: [PATCH 4/5] Refactor Genie paragraph to user-facing recommendation --- docs/qnn.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/docs/qnn.md b/docs/qnn.md index 92459f73ca..2db3ed5a0a 100644 --- a/docs/qnn.md +++ b/docs/qnn.md @@ -4,13 +4,10 @@ ONNX Runtime GenAI supports running models on Qualcomm Snapdragon NPUs via the Q Provider (EP). This enables hardware-accelerated LLM inference on Snapdragon-based devices on Windows ARM64 and Linux ARM64. -ONNX models containing an EPContext node with `ep_context_type: "dlc"` are routed directly -through the Genie API — Qualcomm's optimized LLM inference runtime, part of the QAIRT SDK, -which provides accelerated token generation on the Snapdragon NPU. This routing is transparent -to OGA users; no additional configuration is required. Models in this format are produced by -newer QAIRT-targeting [olive-recipes](https://github.com/microsoft/olive-recipes) pipelines. -Older QNN-targeting recipes produce models without the DLC EPContext type and do not use the -Genie pathway. +For best performance, use the newer QAIRT-targeting pipelines from +[olive-recipes](https://github.com/microsoft/olive-recipes), which optimize models for +accelerated NPU inference via the Genie runtime. Older QNN-targeting recipes produce +standard QNN models that also work but do not use this acceleration pathway. ## Prerequisites From b64ad3762c80a3d44880e2e19aa13d456c16495c Mon Sep 17 00:00:00 2001 From: Kyle Romero Date: Fri, 15 May 2026 16:31:45 -0700 Subject: [PATCH 5/5] Use onnxruntime_qnn helpers for EP registration in QNN docs Replace hardcoded EP name and library path with onnxruntime_qnn.get_ep_name() and onnxruntime_qnn.get_library_path(), which are platform-aware and self-documenting. --- docs/qnn.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/qnn.md b/docs/qnn.md index 2db3ed5a0a..578e302be2 100644 --- a/docs/qnn.md +++ b/docs/qnn.md @@ -25,20 +25,17 @@ compiled QNN graph artifacts. The `genai_config.json` and model files are produc ## Usage -The QNN EP is a plugin and must be registered before loading the model. Pass the path to the -`onnxruntime_providers_qnn` shared library from the `onnxruntime-qnn` package -(`onnxruntime_providers_qnn.dll` on Windows, `libonnxruntime_providers_qnn.so` on Linux): +The QNN EP is a plugin and must be registered before loading the model. The `onnxruntime-qnn` +package provides helpers to locate the library and EP name: ```python import onnxruntime_genai as og +import onnxruntime_qnn # Register the QNN EP plugin before loading the model. -# The library is included in the onnxruntime-qnn package. -# Note: registration uses the full EP name "QNNExecutionProvider"; provider options -# (Config.set_provider_option / genai_config.json) use the short name "QNN". og.register_execution_provider_library( - "QNNExecutionProvider", - "/path/to/onnxruntime_providers_qnn.dll" # or libonnxruntime_providers_qnn.so on Linux + onnxruntime_qnn.get_ep_name(), # "QNNExecutionProvider" + onnxruntime_qnn.get_library_path() # platform-aware path to the QNN EP shared library ) # Load model. The genai_config.json from the olive-recipes pipeline already