Skip to content

[jit_kernel] Migrate copy kernel from sgl-kernel AOT to JIT - #19099

Closed
Johnsonms wants to merge 3 commits into
sgl-project:mainfrom
Johnsonms:jit-kernel-copy
Closed

Johnsonms wants to merge 3 commits into
sgl-project:mainfrom
Johnsonms:jit-kernel-copy

Conversation

@Johnsonms

@Johnsonms Johnsonms commented Feb 21, 2026

Copy link
Copy Markdown
Contributor

Motivation

#17865
copy_to_gpu_no_ce is a small utility kernel that copies a CPU int32 tensor to GPU without using the
copy engine, by packing data into a by-value kernel-argument struct. Migrating it from the AOT
sgl-kernel build to the JIT framework reduces build complexity and aligns with the ongoing effort to
move lightweight kernels to JIT.

Modifications

  • Add python/sglang/jit_kernel/csrc/elementwise/copy.cuh: CUDA kernel templated on kN, CPU/CUDA
    tensor validation via TVM FFI TensorMatcher
  • Add python/sglang/jit_kernel/copy.py: JIT Python wrapper, module cached per array size via
    @cache_once
  • Add python/sglang/jit_kernel/tests/test_copy.py: correctness test for N=64 and N=72
  • Add python/sglang/jit_kernel/benchmark/bench_copy.py: latency benchmark
  • Update deep_gemm.py: import copy_to_gpu_no_ce from sglang.jit_kernel.copy instead of
    sgl_kernel.elementwise
  • Remove sgl-kernel/csrc/elementwise/copy.cu and sgl-kernel/tests/test_copy.py
  • Remove AOT registration from CMakeLists.txt, common_extension.cc, sgl_kernel_ops.h,
    sgl_kernel/elementwise.py, and sgl_kernel/init.py

Accuracy Tests

python -m pytest python/sglang/jit_kernel/tests/test_copy.py -v
image

Benchmarking and Profiling

python python/sglang/jit_kernel/benchmark/bench_copy.py
image

Checklist

Review Process

  1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
  4. After green CI and required approvals, ask Merge Oncalls to merge.

Adds a JIT implementation of copy_to_gpu_no_ce alongside the existing
AOT sgl-kernel version for correctness and performance comparison.

- csrc/elementwise/copy.cuh: CUDA kernel templated on kN,
  packs CPU int32 data into a by-value kernel-arg struct to bypass the
  copy engine, validated with TVM FFI TensorMatcher
- copy.py: JIT Python wrapper, module cached per array size
- tests/test_copy.py: correctness test vs AOT for N=64,72
- benchmark/bench_copy.py: JIT vs AOT perf comparison

AOT implementation unchanged; Phase 2 will remove it.
…ase 2)

Remove the AOT implementation now that the JIT version is verified.

- Delete sgl-kernel/csrc/elementwise/copy.cu
- Remove from CMakeLists.txt, common_extension.cc, sgl_kernel_ops.h
- Remove from sgl_kernel/elementwise.py and __init__.py
- Delete sgl-kernel/tests/test_copy.py
- Update deep_gemm.py to import from sglang.jit_kernel.copy
- Simplify jit_kernel test and benchmark (no AOT reference needed)
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@Johnsonms Johnsonms changed the title [jit_kernel] Migrate copy_to_gpu_no_ce from sgl-kernel AOT to JIT [jit_kernel] Migrate copy kernel from sgl-kernel AOT to JIT Feb 21, 2026
@Johnsonms
Johnsonms marked this pull request as ready for review February 21, 2026 03:18
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

1 similar comment
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@hnyls2002

Copy link
Copy Markdown
Collaborator

Thanks @Johnsonms! This PR edits both python/sglang/jit_kernel/ and top-level sgl-kernel/, neither of which exists any more - they moved to python/sglang/kernels/jit/ and python/sglang/kernels/aot/ (#32072, #32648, #33400) - and the tracking issue #17865 is closed. copy_to_gpu_no_ce is still imported from the AOT wheel at python/sglang/srt/layers/moe/moe_runner/deep_gemm.py:96, so a fresh port would still be welcome. Closing as obsolete - please reopen if I've missed something.

@hnyls2002 hnyls2002 closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants