Conversation
Adds a JIT implementation of copy_to_gpu_no_ce alongside the existing AOT sgl-kernel version for correctness and performance comparison. - csrc/elementwise/copy.cuh: CUDA kernel templated on kN, packs CPU int32 data into a by-value kernel-arg struct to bypass the copy engine, validated with TVM FFI TensorMatcher - copy.py: JIT Python wrapper, module cached per array size - tests/test_copy.py: correctness test vs AOT for N=64,72 - benchmark/bench_copy.py: JIT vs AOT perf comparison AOT implementation unchanged; Phase 2 will remove it.
…ase 2) Remove the AOT implementation now that the JIT version is verified. - Delete sgl-kernel/csrc/elementwise/copy.cu - Remove from CMakeLists.txt, common_extension.cc, sgl_kernel_ops.h - Remove from sgl_kernel/elementwise.py and __init__.py - Delete sgl-kernel/tests/test_copy.py - Update deep_gemm.py to import from sglang.jit_kernel.copy - Simplify jit_kernel test and benchmark (no AOT reference needed)
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
1 similar comment
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Thanks @Johnsonms! This PR edits both |
Motivation
#17865
copy_to_gpu_no_ce is a small utility kernel that copies a CPU int32 tensor to GPU without using the
copy engine, by packing data into a by-value kernel-argument struct. Migrating it from the AOT
sgl-kernel build to the JIT framework reduces build complexity and aligns with the ongoing effort to
move lightweight kernels to JIT.
Modifications
tensor validation via TVM FFI TensorMatcher
@cache_once
sgl_kernel.elementwise
sgl_kernel/elementwise.py, and sgl_kernel/init.py
Accuracy Tests
python -m pytest python/sglang/jit_kernel/tests/test_copy.py -vBenchmarking and Profiling
python python/sglang/jit_kernel/benchmark/bench_copy.pyChecklist
Review Process
/tag-run-ci-label,/rerun-failed-ci,/tag-and-rerun-ci