feat(querylang): language support for term tokenization#6269

Merged

chewxy merged 2 commits intomasterfrom chewxy/termtokenizer

Sep 7, 2020

+51-3

Commits on Aug 25, 2020

Added support for term tokenization to tokenize by language
chewxy
committed
Cleaned up antipattern, added ideal behaviour in tests, but commented it out
chewxy
committed