[QNN-EP] Implement file mapped weights feature #26952

quic-calvnguy · 2026-01-09T02:13:54Z

Description
Enables the file mapping of weights as well as the overall context bin. This feature is currently only enabled for ARM64 WIN devices

Motivation and Context
Currently, when reading the context bin, ORT allocates a large buffer on the heap. Assuming the same model is used, each ORT session will allocate a buffer for the context bin. This is incredibly wasteful when large models are used. Instead, WIN file mapping can be leveraged to map the context bin, then every time a context needs to be created with the context bin, the pointer to the context bin can be retrieved and used instead of some pre-allocated buffer, thus making QNN EP more memory-efficient. In the case of multiple ORT sessions, the context bin will only be loaded once for all sessions, increasing memory efficiency and overall initialization performance. This is very useful regarding the use of LLMs going forward.

yuslepukhin · 2026-01-13T19:00:25Z

The observation is not entirely true. ORT memory map external weights. You have an ability to request a weight as an ORT Value from the EP. If the weight is external it will be memory mapped.

See Graph::LoadExternalInitializerAsOrtValue. One can add this to the provider_wrapped_types.h to Graph and expose to DLL based EPs. Then you will get mapping for free.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

onnxruntime/core/providers/qnn/builder/qnn_file_mapping_callback_interface.h

yuslepukhin · 2026-01-14T18:55:59Z

I suggest not to implement QNN specific mapping, but re-use code in ORT.

yuslepukhin · 2026-01-14T20:36:42Z

Discussed offline, the EP maps initializers form the binary context, not from the external weights files.

- Create file mapping callback interface class - Android expected to have support in the future - Implement Windows callbacks in WindowsFileMapper - New option disable_file_mapped_weights - Feature is enabled by default with retry logic

yuslepukhin · 2026-01-15T00:00:52Z

onnxruntime/core/providers/qnn/qnn_execution_provider.cc

 #endif
  }

+  static const std::string DISABLE_FILE_MAPPED_WEIGHTS = "disable_file_mapped_weights";


static const std::string DISABLE_FILE_MAPPED_WEIGHTS

Is there a separate place where all constants are collected together?

yuslepukhin · 2026-01-15T00:02:08Z

onnxruntime/core/providers/qnn/qnn_execution_provider.cc

+    if ("1" == disable_file_mapped_weights_pos->second) {
+      enable_file_mapped_weights_ = false;
+    }
+    LOGS_DEFAULT(VERBOSE) << "User specified disable_file_mapped_weights: " << enable_file_mapped_weights_;


LOGS_DEFAULT(VERBOSE)

I believe we pass a logger to use to every EP so we can see which session emits which message. Pls, check if it is available.

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

yuslepukhin · 2026-01-15T00:09:20Z

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

+  ORT_RETURN_IF(!cache_file || !cache_file.good(), "Failed to retrieve context binary from: ", context_bin_filepath);
+
+  cache_file.seekg(0, cache_file.end);
+  size_t buffer_size = static_cast<size_t>(cache_file.tellg());


static_cast<size_t>

Would this require a safe cast? It may fail on 32-bit systems if that matters.

yuslepukhin · 2026-01-15T00:11:24Z

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

+    ORT_RETURN_IF_ERROR(ReadContextBinIfValid(context_bin_filepath, buffer_info, false));
+
+    size_t buffer_size = buffer_info.size;
+    ORT_RETURN_IF(buffer_size == 0, "Context bin has a size of 0 bytes: ", context_bin_filepath);


ORT_RETURN_IF(buffer

The size has already been validated inside the function and the error code was checked.

yuslepukhin · 2026-01-15T00:12:33Z

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

+  context_params_ptr_list.clear();
+  context_callbacks_list.clear();
+  context_paramsv2_list.clear();
+  context_params_list.clear();


Is this necessary? If it is, then the code is not exception safe.

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc

yuslepukhin · 2026-01-15T00:19:20Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+// At time of destruction. Usage of logger_ will not be available and will result in a seg fault
+WindowsFileMapper::~WindowsFileMapper() {
+  std::lock_guard<std::mutex> lock(map_mutex_);
+


Why do we need tis lock? Are we expecting multiple threads destroying the same object? Then we have bigger problems.

yuslepukhin · 2026-01-15T00:22:35Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+                                   OPEN_EXISTING,
+                                   FILE_ATTRIBUTE_NORMAL,
+                                   NULL);
+  ORT_RETURN_IF(file_handle == INVALID_HANDLE_VALUE,


file_handle

This handle is leaking on line 72. Check the wil library that has windows oriented smart pointers.
Also, you can close the file handle as soon as mapping handle is obtained, not need to cache it.

yuslepukhin · 2026-01-15T00:26:37Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+  std::lock_guard<std::mutex> lock(map_mutex_);
+  auto status = Status::OK();
+
+  auto bin_map_it = std::find_if(context_bin_to_mapping_handle_map_.begin(),


context_bin_to_mapping_handle_map_

Here and below. If this is a map, would not make sense to avoid sequential scan and use map find() method?

yuslepukhin · 2026-01-15T00:27:39Z

Please, avoid force pushes.

yuslepukhin · 2026-01-15T00:28:29Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+
+  HANDLE file_mapping_handle = bin_map_it->second;
+  auto mapping_it = std::find_if(mapping_handle_to_info_map_.begin(),
+                                 mapping_handle_to_info_map_.end(),


Hash tables exist to query elements in constant time, not O(n)

yuslepukhin · 2026-01-15T00:29:26Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+  LOGS(*logger_, INFO) << "Creating mapping pointer for " << bin_filepath;
+
+  std::lock_guard<std::mutex> lock(map_mutex_);
+  auto it = std::find_if(context_bin_to_mapping_handle_map_.begin(),


std::find_if

Ditto

yuslepukhin · 2026-01-15T00:30:40Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+
+  ORT_RETURN_IF(mapview_ptr == nullptr, "Failed to create mapping pointer for ", bin_filepath);
+
+  if (!context_bin_map_view_pointers_.insert(mapview_ptr).second) {


(!context_bin_map_view_pointers_.insert(mapview_ptr).second)

How is this even possible? You just created a new mapping that is guaranteed to have a unique pointer returned.

yuslepukhin

🕐

edgchen1 · 2026-01-15T00:59:34Z

/azp run Linux QNN CI Pipeline,Windows ARM64 QNN CI Pipeline

azure-pipelines · 2026-01-15T00:59:46Z

Azure Pipelines successfully started running 2 pipeline(s).

yuslepukhin · 2026-01-15T18:05:34Z

Please, comment on all Copilot review issues before resolving them.

onnxruntime/core/providers/qnn/builder/qnn_file_mapping_callback_interface.h

edgchen1 · 2026-01-15T22:23:00Z

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h

+  bool file_mapped_weights_enabled_ = false;
+
+#ifdef QNN_FILE_MAPPED_WEIGHTS_ENABLED
+  std::shared_ptr<FileMappingCallbackInterface> file_mapper_ = nullptr;


why is shared ownership needed?

edgchen1 · 2026-01-15T22:34:25Z

onnxruntime/core/providers/qnn/builder/qnn_windows_file_mapper.cc

+                                   OPEN_EXISTING,
+                                   FILE_ATTRIBUTE_NORMAL,
+                                   NULL);
+  ORT_RETURN_IF(file_handle == INVALID_HANDLE_VALUE,


general: it would be good to output more information about errors from Windows APIs, e.g., with ::GetLastError()

yuslepukhin requested a review from Copilot January 13, 2026 18:58

Copilot started reviewing on behalf of yuslepukhin January 13, 2026 18:59 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

quic_calvnguy added 3 commits January 14, 2026 13:56

[QNN-EP] Implement file mapped weights feature

ff2d8d5

- Create file mapping callback interface class - Android expected to have support in the future - Implement Windows callbacks in WindowsFileMapper - New option disable_file_mapped_weights - Feature is enabled by default with retry logic

Address PR comments

5a25c35

Remove unnecessary underscore

2e451ae

quic-calvnguy force-pushed the dev/calvnguy/file_mapped_weights branch from f55dc78 to 2e451ae Compare January 14, 2026 21:57

yuslepukhin reviewed Jan 15, 2026

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Show resolved Hide resolved

yuslepukhin reviewed Jan 15, 2026

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Show resolved Hide resolved

yuslepukhin reviewed Jan 15, 2026

View reviewed changes

onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Show resolved Hide resolved

yuslepukhin reviewed Jan 15, 2026

View reviewed changes

yuslepukhin requested changes Jan 15, 2026

View reviewed changes

edgchen1 reviewed Jan 15, 2026

View reviewed changes


		ORT_RETURN_IF(mapview_ptr == nullptr, "Failed to create mapping pointer for ", bin_filepath);

		if (!context_bin_map_view_pointers_.insert(mapview_ptr).second) {

[QNN-EP] Implement file mapped weights feature #26952

Are you sure you want to change the base?

[QNN-EP] Implement file mapped weights feature #26952

Conversation

quic-calvnguy commented Jan 9, 2026

Uh oh!

yuslepukhin commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yuslepukhin commented Jan 14, 2026

Uh oh!

yuslepukhin commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin commented Jan 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

edgchen1 commented Jan 15, 2026

Uh oh!

azure-pipelines bot commented Jan 15, 2026

Uh oh!

yuslepukhin commented Jan 15, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuslepukhin commented Jan 13, 2026 •

edited

Loading

yuslepukhin Jan 15, 2026 •

edited

Loading

yuslepukhin Jan 15, 2026 •

edited

Loading