torcho gemlite integration

[HDCharles]

Summary:

adds support for gemlite kernels

Test Plan:

python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 1024 --output 512 --json-model-override-args '{"architectures": ["TorchNativeLlamaForCausalLM"]}' --torchao-config gemlite-32-4-64 --dtype float16 --disable-cuda-graph

python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 32 --input 1024 --output 512 --json-model-override-args '{"architectures": ["TorchNativeLlamaForCausalLM"]}' --torchao-config gemlite-32-4-64 --dtype float16 --disable-cuda-graph

python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 1024 --output 512 --json-model-override-args '{"architectures": ["TorchNativeLlamaForCausalLM"]}' --enable-torch-compile --torchao-config gemlite-32-4-64 --dtype float16

python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 1 --input 1024 --output 512 --json-model-override-args '{"architectures": ["TorchNativeLlamaForCausalLM"]}' --enable-torch-compile --torchao-config gemlite-8-4-64 --dtype float16

Reviewers:

Subscribers:

Tasks:

Tags:

Motivation

This PR is to add support for teh torchao gemlite integration in SGLang for int4wo quantization, the motivation behind the work is that we expect these kernels to have better TTFT performance compared to the existing int4 integration which is optimized for non prefill performance.

Modifications

Added some new options to the torchao utils and added a place to store the gemlite cache after warmup

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[merrymercy]

supported by #2528

[zhyncs]

FYI I have temporarily removed it from the main branch due to some issues. I will add it back in the next version, as I need some time to figure out how dependency management can be more compatible.

[zhyncs]

Currently, if you want to use it, you can install it separately after installing SGLang.