Skip to content

Conversation

@kylesayrs
Copy link
Contributor

@kylesayrs kylesayrs commented Sep 30, 2024

What does this PR do?

This PR follows up on #31704 by allowing the compressed-tensors-nightly package to be used to load compressed tensors modules. This is done by relaxing installation requirements to only require the compressed-tensors module spec, but not the compressed-tensors package.

Compressed Tensors models cannot be loaded with nightly package

from transformers import AutoModel
model = AutoModel.from_pretrained("nm-testing/TinyLlama-1.1B-Chat-v1.0-W8A8_tensor_weight_static_per_tensor_act-e2e")
# ImportError: Using `compressed_tensors` quantized models requires the compressed-tensors library: `pip install compressed-tensors`

Compressed Tensors is reported as unavailable, despite the module being available through the nightly package

from transformers.utils.import_utils import is_compressed_tensors_available
is_compressed_tensors_available()
# False

After the changes, models can be imported with the nightly package and compressed tensors is reported as being available

Who can review?

@SunMarc @ArthurZucker
@dsikka @dhuangnm @andy-neuma
Anyone in the community is free to review the PR once the tests have passed.

@kylesayrs kylesayrs changed the title Allow for compressed_tensors-nightly Allow for nightly packages of compressed_tensors Sep 30, 2024
Copy link

@markurtz markurtz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These changes look correct to me. It brings up a separate consideration that we'll need at some point in the future, which is to gather the compressed tensors version so we can compare if there are breaking changes between the config format that is stored with the model and the compressed tensors that is loaded, especially with it loading nightly dev builds

Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@SunMarc SunMarc requested a review from LysandreJik October 1, 2024 14:47
@LysandreJik LysandreJik merged commit b09234c into huggingface:main Oct 2, 2024
@kylesayrs kylesayrs deleted the kylesayrs/support-nightly-packages branch October 2, 2024 15:01
BernardZach pushed a commit to BernardZach/transformers that referenced this pull request Dec 5, 2024
* only check spec

* correct typo in nightly package name
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants