Conversation
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
Summary of ChangesHello @kylesayrs, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request integrates the Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new example for Qwen3OmniMoeForConditionalGeneration, including a patch to support accelerated offloading. The example script demonstrates how to perform one-shot quantization with GPTQ and generate sample outputs. The changes are well-structured and the example is clear. My review includes a suggestion to improve the performance of the patch file by using more efficient tensor operations, and a comment on improving the clarity of the example script's save directory naming.
|
Related to #1673 as well |
|
Hi, I pass model.thinker to quantize, but model.save_pretrained saved a full bf16 model |
This is because in 'oneshot' pre_precess, 'model.save_pretrained' will be modified for saving compressed model. Passing model.thinker to oneshot means model.save_pretrained is not modified. Just add the code below before you save the model will fix the problem. add the following code before savefrom llmcompressor.transformers.compression.compressed_tensors_utils import modify_save_pretrained 
|
why does it save nothing？ # Apply AWQ quantization.
oneshot(
model=model.thinker,
processor=processor,
recipe=recipe,
dataset=ds,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
data_collator=data_collator,
)
# calibrated_model.save_pretrained(SAVE_DIR, save_compressed=True)
print("========== SAMPLE GENERATION ==============")
# dispatch_for_generation(model)
modify_save_pretrained(model)
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-G128"
model.save_pretrained(SAVE_DIR, save_compressed=True)
processor.save_pretrained(SAVE_DIR) |
I tried with no calibration quant, successfully saved compressed model. CODEimport requests from llmcompressor import oneshot model_id = "/mnt/home/model/Qwen3-Omni-30B-A3B-Instruct" scheme = "W4A16" recipe = [ oneshot( modify_save_pretrained(model) SAVE_DIR = model_id.rstrip("/").split("/")[-1] + "-thinker-" + scheme |
|
@Sekri0 Thanks, it worked |
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
kylesayrs
force-pushed
the
kylesayrs/qwen_omni
branch
from
47593d9 to
bbb3f7e
Compare
|
Documentation update |
brian-dellabetta
left a comment
brian-dellabetta
left a comment
There was a problem hiding this comment.
One comment regarding patch file placement
dsikka
left a comment
dsikka
left a comment
There was a problem hiding this comment.
Small nits. LGTM. Thanks!
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
|
Hi! Any way to quant qwen3 omni using awq? |
|
After quantifying with the script, I encounter the following error |
|
@kylesayrs the ignore layers solve this error: #2125 (comment) |
6 participants
Purpose
get_input_embeddingsnot auto‑handled for Qwen3OmniMoeForConditionalGeneration #1872Qwen3OmniMoeForConditionalGenerationChanges
model.thinkertooneshot, sincemodeldoes not implement a forward method (the thinker module is aPreTrainedModelthat contains all of the parameters worth quantizing)fast_pos_embed_interpolateto support accelerate offloadingimage_grid_thw, but leavepixel_valuesand other inputs unsqueezedTesting
output.wav