From 0d3568938ff7a0e4d185c96f4da7304f382325e9 Mon Sep 17 00:00:00 2001 From: Qubitium Date: Tue, 24 Dec 2024 20:58:26 +0800 Subject: [PATCH 1/2] review: update docs --- docs/source/en/quantization/gptq.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/source/en/quantization/gptq.md b/docs/source/en/quantization/gptq.md index fb5b36580179..f762eed4020f 100644 --- a/docs/source/en/quantization/gptq.md +++ b/docs/source/en/quantization/gptq.md @@ -122,6 +122,14 @@ model = AutoModelForCausalLM.from_pretrained("{your_username}/opt-125m-gptq", de [Marlin](https://github.com/IST-DASLab/marlin) is a CUDA gptq kernel, 4-bit only, that is highly optimized for the Nvidia A100 GPU (Ampere) architecture where the the loading, dequantization, and execution of post-dequantized weights are highly parallelized offering a substantial inference improvement versus the original CUDA gptq kernel. Marlin is only available for quantized inference and does support model quantization. +Marlin inference can be activated via the `backend` property in `GPTQConfig` for GPTQModel: + +```pycon + +from transformers import AutoModelForCausalLM, GPTQConfig + +model = AutoModelForCausalLM.from_pretrained("{your_username}/opt-125m-gptq", device_map="auto", quantization_config=GPTQConfig(bits=4, backend="marlin")) +``` ## ExLlama From b3353ac1fea04ffcb40c4a423f1c97f60763bf2d Mon Sep 17 00:00:00 2001 From: Qubitium Date: Tue, 24 Dec 2024 21:00:14 +0800 Subject: [PATCH 2/2] fix typo --- docs/source/en/quantization/gptq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/quantization/gptq.md b/docs/source/en/quantization/gptq.md index f762eed4020f..fef2265dd9e1 100644 --- a/docs/source/en/quantization/gptq.md +++ b/docs/source/en/quantization/gptq.md @@ -124,7 +124,7 @@ model = AutoModelForCausalLM.from_pretrained("{your_username}/opt-125m-gptq", de Marlin inference can be activated via the `backend` property in `GPTQConfig` for GPTQModel: -```pycon +```py from transformers import AutoModelForCausalLM, GPTQConfig