fix whitespace LM tokenize issue

[howl-anderson]

Proposed changes:

• Skip token when its text can't be tokenized by LM tokenizer (for example, whitespace)

fixs #7406

Status (please check what you already did):

• added some tests for the functionality
• updated the documentation
• updated the changelog (please check changelog for instructions)
• reformat files using black (please check Readme for instructions)

[sara-tagger]

Thanks for submitting a pull request 🚀 @rctatman will take a look at it as soon as possible ✨

[tmbo]

looks good 👍 can you please add a test to the unit tests in tests/nlu/featurizers/test_lm_featurizer.py as well as a changelog file to changelog/?

[howl-anderson]

@tmbo Sure! I will add those features in later commits.

[howl-anderson]

@tmbo Please take a look

[tmbo]

thanks a lot for this whitespace fix - really cool 🚀

[rasabot]

Could not update branch. Most likely this is due to a merge conflict. Please update the branch manually and fix any issues.