[Performance] Memoryless observers for weights

[kylesayrs]

Purpose

• Reduce memory usage when calibrating weights

Changes

• Always use memoryless_minmax and memoryless_mse when performing weight quantization

Testing

• lm-eval 🟢: https://github.com/neuralmagic/llm-compressor-testing/actions/runs/20797105387/job/59733406907

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[gemini-code-assist]

Summary of Changes

Hello @kylesayrs, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the quantization calibration mechanism by mandating the use of memoryless observers for weights. This adjustment is a direct consequence of discontinued training support for certain observer types, ensuring that the system consistently employs appropriate observers for inference. The update includes a modification to the initialize_observer function to automatically select the correct memoryless observer variant, thereby standardizing weight observation.

Highlights

• Enforced Memoryless Observers for Weights: The pull request introduces logic to ensure that only memoryless observers are used for weights during quantization. Specifically, if the base_name is 'weight' and the original observer type was 'static_minmax', 'minmax', or 'mse', it will now be automatically set to 'memoryless_minmax' or 'memoryless_mse' respectively.
• Docstring Clarification: The docstring for the initialize_observer function has been updated to explicitly state that it always initializes memoryless observers for weights, reflecting the new behavior.
• Streamlined Observer Initialization: The change simplifies the observer selection process by programmatically overriding the observer type for weights, aligning with the removal of training support for these observers.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request updates the initialize_observer function to enforce the use of memoryless observers for weights by replacing stateful observers like minmax and mse with their memoryless versions. This aligns with the goal of discontinuing support for training-time observer calibration for weights. My review includes a suggestion to improve the accuracy of the docstring and another to refactor the conditional logic for better readability and maintainability.

[brian-dellabetta]

Do we want to change the presets in CT as well? Seems strange to me that we have logic in presets, in QuantizationArgs validation, and here in LC as well

[kylesayrs]

@brian-dellabetta Yeah I agree, I'll change the presets too. I think this PR should still land though, since it also safeguards against users accidentally using a moving-average observer for weights, when in reality it won't do anything except use extra memory.

[kylesayrs]

vllm-project/compressed-tensors#540

@brian-dellabetta There's no need to wait for this, we can land these changes now

[dsikka]

Thanks!

[dsikka]

@kylesayrs are we concerned around algorithms beyond the QuantModifier requiring a non-memoryless observer for weights?

E.g Does the AutoRound modifier need to track values over time for weight updates?

[kylesayrs]

@dsikka Autoround does not use LC observers, as evidenced by how QuantizationMixin.start_calibration is never called.