[Serialization] add argument to pass shared tensors names to drop when saving

[hanouticelina]

Related to transformers#35080.

Currently, when saving a torch state dict with shared tensors, the key to keep is chosen alphabetically which might not be the desired behavior. This PR simply adds an optional state_dict_keys_to_discard argument to both save_torch_state_dict() and save_torch_model() that allows users to specify which keys should be discarded in priority when duplicates are found.
Note : The logic behind choosing which key should be discarded in priority is framework/model-specific and should be handled by the user. For example, for transformers.PreTrainedModel models, this is specified in the _tied_weights_keys attribute, example usage:

model = ... # A model inheriting from PreTrainedModel
save_torch_model(
    model,
    "path/to/folder",
    state_dict_keys_to_discard=model._tied_weights_keys,
)

or with save_torch_state_dict()

model = ... # A model inheriting from PreTrainedModel
save_torch_state_dict(
    model.state_dict(),
    "path/to/folder",
    state_dict_keys_to_discard=model._tied_weights_keys,
)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Nice ! Thanks for adding this !

[Wauplin]

Thank you !