[QNN EP] Fix error messages being logged as VERBOSE instead of ERROR #24931

Copilot · 2025-06-02T18:36:13Z

Problem

QNN error messages were being logged at VERBOSE level instead of ERROR level, making them invisible unless verbose logging was enabled. Users would only see unhelpful generic error messages like:

Failed to finalize QNN graph. Error code: 1002 at location qnn_model.cc:167 FinalizeGraphs

But the actual detailed error messages from QNN were hidden in verbose logs:

tcm_migration.cc:2088:ERROR:Operator named q::*InputSlicePad (0x1654900000002) not sufficiently tiled to fit in TCM. Requires 12441600 bytes
graph_prepare.cc:2808:ERROR:Graph prepare TCM Migration action failed
graph_prepare.cc:2868:ERROR:Graph prepare failed during optimization with err: 17, Fatal Optimize

Root Cause

The QnnLogging callback function in qnn_backend_manager.cc was ignoring the level parameter from QNN and hardcoding all messages as kVERBOSE severity:

void QnnLogging(const char* format, QnnLog_Level_t level, uint64_t timestamp, va_list argument_parameter) {
  ORT_UNUSED_PARAMETER(level);  // ❌ Ignoring the actual log level
  // ...
  const auto severity = ::onnxruntime::logging::Severity::kVERBOSE;  // ❌ Hardcoded as VERBOSE

Solution

Modified the QnnLogging function to properly map QNN log levels to appropriate ORT severity levels:

QNN_LOG_LEVEL_ERROR → logging::Severity::kERROR ✅ Key fix
QNN_LOG_LEVEL_WARN → logging::Severity::kWARNING
QNN_LOG_LEVEL_INFO → logging::Severity::kINFO
QNN_LOG_LEVEL_VERBOSE/DEBUG → logging::Severity::kVERBOSE

Changes Made

Modified QnnLogging function: Removed hardcoded kVERBOSE and added proper level mapping
Added MapQNNLogLevelToOrtSeverity function: For potential future reuse
Minimal and surgical changes: Only 37 lines added, 2 removed

Impact

QNN error messages will now appear as ERROR-level logs in normal logging output, making debugging much easier for users without requiring verbose logging to be enabled.

Fixes #24876.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: vraspar <51386888+vraspar@users.noreply.github.com>

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

…NLogLevelToOrtSeverity static Co-authored-by: yuslepukhin <11303988+yuslepukhin@users.noreply.github.com>

HectorSVC · 2025-06-02T21:23:46Z

The existing code is by design. Please refer to my reply in the issue.

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

yuslepukhin · 2026-01-15T22:34:53Z

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline

azure-pipelines · 2026-01-15T22:35:14Z

Azure Pipelines successfully started running 4 pipeline(s).

yuslepukhin

yuslepukhin

yuslepukhin

yuslepukhin · 2026-01-16T21:49:58Z

/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline

azure-pipelines · 2026-01-16T21:50:17Z

Azure Pipelines successfully started running 4 pipeline(s).

revoking review

tianleiwu

This PR addresses a critical usability issue where QNN backend error messages were incorrectly logged at the VERBOSE level. This suppressed important error details unless verbose logging was enabled, making debugging difficult.

LGTM. The changes function exactly as described. The logic for mapping log levels is sound and will ensure QNN errors are visible by default.

…24931) ## Problem QNN error messages were being logged at VERBOSE level instead of ERROR level, making them invisible unless verbose logging was enabled. Users would only see unhelpful generic error messages like: ``` Failed to finalize QNN graph. Error code: 1002 at location qnn_model.cc:167 FinalizeGraphs ``` But the actual detailed error messages from QNN were hidden in verbose logs: ``` tcm_migration.cc:2088:ERROR:Operator named q::*InputSlicePad (0x1654900000002) not sufficiently tiled to fit in TCM. Requires 12441600 bytes graph_prepare.cc:2808:ERROR:Graph prepare TCM Migration action failed graph_prepare.cc:2868:ERROR:Graph prepare failed during optimization with err: 17, Fatal Optimize ``` ## Root Cause The `QnnLogging` callback function in `qnn_backend_manager.cc` was ignoring the `level` parameter from QNN and hardcoding all messages as `kVERBOSE` severity: ```cpp void QnnLogging(const char* format, QnnLog_Level_t level, uint64_t timestamp, va_list argument_parameter) { ORT_UNUSED_PARAMETER(level); // ❌ Ignoring the actual log level // ... const auto severity = ::onnxruntime::logging::Severity::kVERBOSE; // ❌ Hardcoded as VERBOSE ``` ## Solution Modified the `QnnLogging` function to properly map QNN log levels to appropriate ORT severity levels: - `QNN_LOG_LEVEL_ERROR` → `logging::Severity::kERROR` ✅ **Key fix** - `QNN_LOG_LEVEL_WARN` → `logging::Severity::kWARNING` - `QNN_LOG_LEVEL_INFO` → `logging::Severity::kINFO` - `QNN_LOG_LEVEL_VERBOSE/DEBUG` → `logging::Severity::kVERBOSE` ## Changes Made 1. **Modified `QnnLogging` function**: Removed hardcoded `kVERBOSE` and added proper level mapping 2. **Added `MapQNNLogLevelToOrtSeverity` function**: For potential future reuse 3. **Minimal and surgical changes**: Only 37 lines added, 2 removed ## Impact QNN error messages will now appear as ERROR-level logs in normal logging output, making debugging much easier for users without requiring verbose logging to be enabled. Fixes #24876. --- 💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more [Copilot coding agent tips](https://gh.io/copilot-coding-agent-tips) in the docs. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: vraspar <51386888+vraspar@users.noreply.github.com> Co-authored-by: yuslepukhin <11303988+yuslepukhin@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <dmitrism@microsoft.com>

| Commit | Commit Title | Author | | :--- | :--- | :--- | | `11dde2d9e` | [NV TensorRT RTX EP] Fix external tensorrt_plugins load path (#26814) | keshavv27 | | `080d96818` | Move model compatibility checks ahead of session initialization (#27037) | adrastogi | | `ec4f6bfa1` | [QNN EP] Fix error messages being logged as VERBOSE instead of ERROR (#24931) | Copilot | | `0432e7125` | perftest: support plugin eps for compile_ep_context (#27121) | Jaskaran Singh Nagi | | `727db0d3d` | Engine compatibility validity API implementation (#26774) | umangb-09 | | `27013522f` | Deprecate transformers model examples (#27156) | Jambay Kinley | | `f83d4d06e` | [QNN-EP] Implement file mapped weights feature (#26952) | quic-calvnguy | --------- Co-authored-by: keshavv27 <165012837+keshavv27@users.noreply.github.com> Co-authored-by: adrastogi <aditya.rastogi@microsoft.com> Co-authored-by: Aditya Rastogi <adityar@ntdev.microsoft.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: vraspar <51386888+vraspar@users.noreply.github.com> Co-authored-by: yuslepukhin <11303988+yuslepukhin@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <yuslepukhin@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Dmitri Smirnov <dmitrism@microsoft.com> Co-authored-by: Jaskaran Singh Nagi <jaskaran.singh.nagi@intel.com> Co-authored-by: umangb-09 <umangb@nvidia.com> Co-authored-by: Jambay Kinley <jambaykinley@microsoft.com> Co-authored-by: quic-calvnguy <quic_calvnguy@quicinc.com> Co-authored-by: quic_calvnguy <quic_calvnguy@quic_inc.com>

Initial plan for issue

593bdfa

Copilot AI assigned Copilot and vraspar Jun 2, 2025

Copilot started work on behalf of vraspar June 2, 2025 18:36 View session

Fix QNN error messages being logged as VERBOSE instead of ERROR

467abdd

Co-authored-by: vraspar <51386888+vraspar@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Error messages from QNN are turned into verbose level messages~~ Fix QNN error messages being logged as VERBOSE instead of ERROR Jun 2, 2025

Copilot AI requested a review from vraspar June 2, 2025 18:47

Copilot finished work on behalf of vraspar June 2, 2025 18:47

yuslepukhin previously requested changes Jun 2, 2025

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h Outdated Show resolved Hide resolved

Copilot started work on behalf of yuslepukhin June 2, 2025 19:34 View session

github-actions bot reviewed Jun 2, 2025

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Show resolved Hide resolved

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated Show resolved Hide resolved

Address code review feedback: eliminate duplicate code and make MapQN…

9d63cd8

…NLogLevelToOrtSeverity static Co-authored-by: yuslepukhin <11303988+yuslepukhin@users.noreply.github.com>

Copilot finished work on behalf of yuslepukhin June 2, 2025 19:41

jywu-msft added the ep:QNN issues related to QNN exeution provider label Jul 31, 2025

Merge branch 'main' into copilot/fix-24876

9969c9a

yuslepukhin marked this pull request as ready for review January 12, 2026 20:04

github-actions bot reviewed Jan 12, 2026

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated Show resolved Hide resolved

yuslepukhin and others added 3 commits January 12, 2026 12:11

Update onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

f005b5a

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

653e021

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Remove stray proto

9ac878c

yuslepukhin previously approved these changes Jan 15, 2026

View reviewed changes

yuslepukhin enabled auto-merge (squash) January 15, 2026 22:36

Merge branch 'main' into copilot/fix-24876

42c028c

yuslepukhin added release:1.24.0 and removed release:1.24.0 labels Jan 16, 2026

yuslepukhin added 2 commits January 16, 2026 11:54

Fix log level mapping

2d67d6b

Merge branch 'main' into copilot/fix-24876

96c1470

yuslepukhin dismissed their stale review via 96c1470 January 16, 2026 19:54

yuslepukhin previously approved these changes Jan 16, 2026

View reviewed changes

CI failure

9a9b309

yuslepukhin dismissed their stale review via 9a9b309 January 16, 2026 20:29

yuslepukhin previously approved these changes Jan 16, 2026

View reviewed changes

yuslepukhin added the release:1.24.0 label Jan 16, 2026

yuslepukhin changed the title ~~Fix QNN error messages being logged as VERBOSE instead of ERROR~~ [QNN EP] Fix error messages being logged as VERBOSE instead of ERROR Jan 16, 2026

yuslepukhin removed the release:1.24.0 label Jan 16, 2026

yuslepukhin disabled auto-merge January 16, 2026 22:28

yuslepukhin mentioned this pull request Jan 16, 2026

Error messages from QNN are turned into verbose level messages #24876

Closed

tirupath-qti approved these changes Jan 22, 2026

View reviewed changes

yuslepukhin added the release:1.24.0 label Jan 23, 2026

tianleiwu approved these changes Jan 23, 2026

View reviewed changes

tianleiwu merged commit ec4f6bf into main Jan 23, 2026
88 checks passed

tianleiwu deleted the copilot/fix-24876 branch January 23, 2026 18:42

tianleiwu mentioned this pull request Jan 27, 2026

ORT 1.24.0 release cherry pick round 3 #27168

Merged

tianleiwu added cherry-picked Cherry-picked for a cherrypicks branch and removed release:1.24.0 labels Jan 28, 2026

[QNN EP] Fix error messages being logged as VERBOSE instead of ERROR #24931

[QNN EP] Fix error messages being logged as VERBOSE instead of ERROR #24931

Conversation

Copilot AI commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Root Cause

Solution

Changes Made

Impact

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HectorSVC commented Jun 2, 2025

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuslepukhin commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

yuslepukhin commented Jan 16, 2026

Uh oh!

azure-pipelines bot commented Jan 16, 2026

Uh oh!

tianleiwu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Copilot AI commented Jun 2, 2025 •

edited

Loading