Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

3.0 Deprecation Warning: Remove deprecated tokenizers #8881

Closed
4 tasks done
ancalita opened this issue Jun 15, 2021 · 2 comments · Fixed by #9120
Closed
4 tasks done

3.0 Deprecation Warning: Remove deprecated tokenizers #8881

ancalita opened this issue Jun 15, 2021 · 2 comments · Fixed by #9120
Assignees
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework maintenance:rasa-3.0/deprecation

Comments

@ancalita
Copy link
Member

ancalita commented Jun 15, 2021

Background
This warning was found in:

  • rasa.nlu.tokenizers.convert_tokenizer:L27
  • rasa.nlu.tokenizers.lm_tokenizer.py:L21
"f""'{self.__class__.__name__}' is deprecated and ""
f""will be removed in the future. ""
f""It is recommended to use the '{WhitespaceTokenizer.__name__}' or ""
f""another {Tokenizer.__name__} instead."","

Definition of Done

  • resolve warnings in 3.0 version
  • Removed the deprecated tokenizers from the codebase.
  • Added a changelog entry for 3.0 saying we've removed the previously deprecated tokenizers
  • Removed the references in the docs
@ancalita ancalita added the area:rasa-oss 🎡 Anything related to the open source Rasa framework label Jun 15, 2021
@TyDunn TyDunn added this to the 3.0 Rasa Open Source milestone Jun 15, 2021
@samsucik
Copy link
Contributor

samsucik commented Jul 9, 2021

Are we actually removing these tokenisers in 3.0? 🤔

@kedz
Copy link
Contributor

kedz commented Jul 19, 2021

Also removing deprecated featurization related to the LanguageModelTokenizer originally from this issue #8879 :

  • rasa.nlu.utils.hugging_face.hf_transformers.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework maintenance:rasa-3.0/deprecation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants