Support loading hugging face checkpoint

[ulivne]

• Support loading checkpoint with INC

• load_cp explanation

• Add torch_dtype bf16 for the model

• Support getting quantized weight tensor

[yafshar]

@ulivne sounds like neural_compressor is missing from the requirements consider adding it!

[ulivne]

@ulivne sounds like neural_compressor is missing from the requirements consider adding it!

neural_compressor is installed automatically as part of habana software stack. it replaces habana_quantization_toolkit which was also not part of requirements.

[ulivne]

Removed test as it is uses a huggingface model, and we are not sure that we can use in our open source code (in terms of licence, intel regulations)

[regisss]

I think load_cp is not explicit enough. Transformers has load_in_4bit which I think would be better here.

@libinta You'll let me know when this PR can be merged.

[ulivne]

I think load_cp is not explicit enough. Transformers has load_in_4bit which I think would be better here.

@libinta You'll let me know when this PR can be merged.

this flag triggers neural compressor load api, which also supports loading of high precision models, and is planned to support loading of fp8 models in the future. it is correct that currently for gaudi only 4bit is suppoerted, however i think we should keep the load_cp name to avoid changing it in the future when we add loading of fp8 models.

[regisss]

I think load_cp is not explicit enough. Transformers has load_in_4bit which I think would be better here.
@libinta You'll let me know when this PR can be merged.

this flag triggers neural compressor load api, which also supports loading of high precision models, and is planned to support loading of fp8 models in the future. it is correct that currently for gaudi only 4bit is suppoerted, however i think we should keep the load_cp name to avoid changing it in the future when we add loading of fp8 models.

load_in_4_bit is just a proposal but honestly, if I'm a user and I don't know the arguments, load_cp doesn't tell me anything about what it does. I would prefer something more explicit like load_quantized_model or even better automatically detect if this is a 4/8-bit checkpoint.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[libinta]

one line missing here