Skip to content

Bugfix in tf32_hc_prenorm_gemm when SGLANG_ENABLE_JIT_DEEPGEMM=0 #29740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattyding wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Bugfix in tf32_hc_prenorm_gemm when SGLANG_ENABLE_JIT_DEEPGEMM=0 #29740

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4831300
Add local deep_gemm import in tf32_hc_prenorm_gemm so the HC-prenorm …
mattyding Jun 30, 2026
9898421
Merge branch 'main' into fix/hc-prenorm-deep-gemm-import
mattyding Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions python/sglang/srt/layers/deep_gemm_wrapper/entrypoint.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,8 @@ def tf32_hc_prenorm_gemm(
):
if x.shape[0] == 0:
return
import deep_gemm

deep_gemm.tf32_hc_prenorm_gemm(x, fn, out, sqrsum, num_splits=num_splits)
Comment on lines +245 to 247

@gemini-code-assist gemini-code-assist Bot Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

In high-performance LLM serving, tf32_hc_prenorm_gemm is executed on the hot path (potentially once per layer/step). Running import deep_gemm on every single invocation introduces unnecessary overhead from sys.modules lookups and import lock acquisition.

By declaring global deep_gemm and checking if it is already present in globals(), we can lazily import it on the first call and reuse the cached global reference on all subsequent calls. This reduces the overhead to a simple, extremely fast dictionary lookup.

Suggested change
import deep_gemm
deep_gemm.tf32_hc_prenorm_gemm(x, fn, out, sqrsum, num_splits=num_splits)
global deep_gemm
if "deep_gemm" not in globals():
import deep_gemm
deep_gemm.tf32_hc_prenorm_gemm(x, fn, out, sqrsum, num_splits=num_splits)

@mattyding mattyding Jun 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

global cache call saves ~29ns, negligable (measured)

i follow the pre-exisitng example set by https://github.com/sgl-project/sglang/blob/v0.5.14/python/sglang/srt/layers/mhc.py#L1446-L1449



Expand Down
Loading