Add integration with gemlite weight only quant

[jerryzh168]

Summary:
gemlite Only available with nightly torchao right now (or install from source)

Test Plan:

python3 -m sglang.bench_one_batch --model meta-llama/Llama-3.1-8B-Instruct --batch-size 1 --input 1024 --output 512 --json-model-override-args '{"architectures": ["TorchNativeLlamaForCausalLM"]}' --enable-torch-compile —torchao-config gemlite-4-64 --tp-size 1

Reviewers:

Subscribers:

Tasks:

Tags:

Motivation

Modifications

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[zhyncs]

Hi @jerryzh168 What is the release cycle of torchao? I can accept using the torchao nightly version, maybe you can try enabling it in the https://github.com/sgl-project/sglang/blob/main/python/pyproject.toml. What do you think? cc @merrymercy @Ying1123 @ispobock

[jerryzh168]

Hi @jerryzh168 What is the release cycle of torchao? I can accept using the torchao nightly version, maybe you can try enabling it in the main/python/pyproject.toml. What do you think? cc @merrymercy @Ying1123 @ispobock

we have ~ monthly releases, yeah depend on nightly version would be better for now, and we can update to a stable version a bit later I think

[jerryzh168]

I tried pip install torchao>=0.8.0.dev20241219 but it doesn't work, we probably need to use

pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu124

to install the nightly version, do we just want to add a version check here?

[jerryzh168]

@zhyncs I think we can land, it's fine to have this as an experimental feature for now I think, I added a print to ask people to use torchao nightly