bring back our demons: clean_up_tokenization_spaces

[ArthurZucker]

What does this PR do?

We already brought it back with:

        if clean_up_tokenization_spaces:
            # Call custom cleanup method if it exists (e.g., for CLVP's [SPACE] token replacement)
            if hasattr(self, "clean_up_tokenization") and callable(self.clean_up_tokenization):
                text = self.clean_up_tokenization(text)
            else:
                # Otherwise apply standard cleanup
                text = (
                    text.replace(" .", ".")
                    .replace(" ?", "?")
                    .replace(" !", "!")
                    .replace(" ,", ",")
                    .replace(" ' ", "'")
                    .replace(" n't", "n't")
                    .replace(" 'm", "'m")
                    .replace(" 's", "'s")
                    .replace(" 've", "'ve")
                    .replace(" 're", "'re")
                )

which basically always calls it.
We are doing this now because it allows to fix custom remote model that expect this to still exist.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[itazap]

nice thanks - we can also then rm this code then from tokenization_utils_tokenizers.py, toknization_python.py, and tokenization_mistral_common.py !

[hmellor]

This fixes the issues with remote models in vLLM.

Let's take @itazap's suggestion to deduplicate this BC code.

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: luke, plbart, wav2vec2