Add enable_profiling API in RunOptions using Thread-Local Storage by xiaofeihan1 · Pull Request #26856 · microsoft/onnxruntime

xiaofeihan1 · 2025-12-23T09:04:00Z

Description

Support profiling for specific Run.

Similar to session-level profiling, developers can specify enable_profiling and profile_file_prefix in RunOptions. After the run completes, the resulting JSON file will be saved with the given profile_file_prefix + timestamp.

When profiling is enabled via RunOptions, it should ideally collect two types of events:

Profiler events
Used to calculate the total execution time of each operator.
Execution Provider (EP) profiler events
Used to measure GPU kernel execution time.

Unlike session-level, we need to ensure the collecting events is correct for multiple thread scenario.

For 1, this can be supported easily(sequential_executor.cc). We use a thread-local storage (TLS) variable, RunLevelState (defined in profiler.h), to maintain run-level profiling state for each thread.

For 2, each Execution Provider (EP) has its own profiler implementation, and each EP must ensure correct behavior under run-level profiling. This PR ensures that the WebGPU profiler works correctly with run-level profiling.

Scenario	Example	Expected Result
Concurrent runs on the same session with different run-level profiling settings	t1: `sess1.Run({ enable_profiling: true })` t2: `sess1.Run({ enable_profiling: false })` t3: `sess1.Run({ enable_profiling: true })`	Two trace JSON files are generated: one for `t1` and one for `t3`.
Run-level profiling enabled together with session-level profiling	`sess1 = OrtSession({ enable_profiling: true })` `sess1.Run({ enable_profiling: true })`	Two trace JSON files are generated: one corresponding to session-level profiling and one corresponding to run-level profiling.

Motivation and Context

Previously, profiling only for session level. Sometimes developer want to profile for specfic run . so the PR comes.

github-actions

You can commit the suggested changes from lintrunner.

onnxruntime/core/common/profiler_common.cc

github-actions

You can commit the suggested changes from lintrunner.

github-actions · 2026-01-04T09:45:31Z

onnxruntime/core/common/profiler_common.cc

+
+

Suggested change

qjia7 · 2026-01-04T09:29:27Z

onnxruntime/core/common/profiler.cc

+  if (file_path.empty()) {
+    run_level_state_.events.clear();
+    return std::string();
+  }


Move L201-L204 before collecting EP profiler events?

qjia7 · 2026-01-04T09:45:49Z

onnxruntime/core/session/inference_session.cc

@@ -3138,6 +3143,14 @@ Status InferenceSession::Run(const RunOptions& run_options,
  if (session_profiler_.IsEnabled()) {


Use if (session_profiler_.IsEnabled() || run_options.enable_profiling)?

qjia7 · 2026-01-04T10:07:52Z

onnxruntime/core/providers/webgpu/webgpu_profiler.h

+
+  // Run-level profiling support
+  void StartRunProfiling() override;
+  void EndRunProfiling(TimePoint start_time, onnxruntime::profiling::Events& events) override;


Can we reuse StartProfiling and EndProfiling no matter it's session level or run level? If it's session-level, just pass session-level's events to EndProfiling. Otherwise, pass run-level events to EndProfiling. Can it satisfy the requirement?

yuslepukhin · 2026-01-05T21:47:38Z

include/onnxruntime/core/common/gpu_profiler_common.h

    return true;
  }

+  virtual void StartRunProfiling() override {}


virtual vo

Is this necessary?

yuslepukhin · 2026-01-05T21:48:00Z

include/onnxruntime/core/common/profiler_common.h

+  virtual void StartRunProfiling() {}
+  virtual void EndRunProfiling(TimePoint start_time, Events& events) {
+    ORT_UNUSED_PARAMETER(start_time);
+    ORT_UNUSED_PARAMETER(events);


Suggest simply commenting out param names

yuslepukhin · 2026-01-05T21:54:34Z

onnxruntime/core/common/profiler.cc

+    out << "\"ts\" :" << rec.ts << ",";
+    out << R"("ph" : "X",)";
+    out << R"("name" :")" << rec.name << "\",";
+    out << "\"args\" : {";


Could this be a separate operator<<(std::ostream&) for the record or function?

yuslepukhin · 2026-01-05T21:55:02Z

onnxruntime/core/common/profiler_common.cc


+
+
 static constexpr int kMaxSymbolSize = 1024;


This file does not seem to have changes.

yuslepukhin · 2026-01-05T22:00:17Z

Here is the big question. Can we have things pertaining to RunOptions profiling stored within RunOptions and not TLS?
It may be possible to instantiate a separate profiler instance within RunOptions when enabled, then add an API to save to a file OR even retrieve in memory?
The in memory thing is optional, but having everything in RunOptions would be swell. Not having to do many changes in the profiler.

yuslepukhin

🕐

xiaofeihan1 · 2026-01-08T01:59:10Z

Here is the big question. Can we have things pertaining to RunOptions profiling stored within RunOptions and not TLS? It may be possible to instantiate a separate profiler instance within RunOptions when enabled, then add an API to save to a file OR even retrieve in memory? The in memory thing is optional, but having everything in RunOptions would be swell. Not having to do many changes in the profiler.

Thanks for the suggestion! I’m re-implementing this using your proposed solution. I’ll let you know once it’s done.

xiaofeihan1 · 2026-01-14T23:56:57Z

Close this PR because we will use another solution. See #26846

xiaofeihan1 closed this Jan 4, 2026

xiaofeihan1 reopened this Jan 4, 2026

xiaofeihan1 added 2 commits January 4, 2026 15:16

impl with TLS

781d77a

change to profile_file_prefix

559a8dd

xiaofeihan1 force-pushed the xiaofeihan/TLS_profiling branch from e8c9879 to 559a8dd Compare January 4, 2026 07:16

github-actions bot reviewed Jan 4, 2026

View reviewed changes

onnxruntime/core/common/profiler_common.cc Show resolved Hide resolved

fix pipeline

d75ee74

xiaofeihan1 force-pushed the xiaofeihan/TLS_profiling branch from 09efa64 to d75ee74 Compare January 4, 2026 09:43

github-actions bot reviewed Jan 4, 2026

View reviewed changes

onnxruntime/core/common/profiler_common.cc

Comment on lines +15 to +16

Copy link
Copy Markdown

Contributor

github-actions bot Jan 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

qjia7 reviewed Jan 4, 2026

View reviewed changes

yuslepukhin reviewed Jan 5, 2026

View reviewed changes

yuslepukhin requested changes Jan 5, 2026

View reviewed changes

yuslepukhin mentioned this pull request Jan 8, 2026

Add environment variable support for profiling configuration #26798

Open

xiaofeihan1 closed this Jan 14, 2026

		@@ -3138,6 +3143,14 @@ Status InferenceSession::Run(const RunOptions& run_options,
		if (session_profiler_.IsEnabled()) {

Conversation

xiaofeihan1 commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

qjia7 Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

qjia7 Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

qjia7 Jan 4, 2026

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

yuslepukhin Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

yuslepukhin commented Jan 5, 2026

Uh oh!

yuslepukhin left a comment

Choose a reason for hiding this comment

Uh oh!

xiaofeihan1 commented Jan 8, 2026

Uh oh!

xiaofeihan1 commented Jan 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xiaofeihan1 commented Dec 23, 2025 •

edited

Loading