Arabic Models

[Anas-Abdelhadi]

Hi there, I've checked your documentation and appears you only have support for two models (en). Following up on a prior question regarding Spanish, is there a tutorial to help developing a model for Arabic? I would appreciate it if you can help by pointing where to start and how to do it.
Thanks

[sanjayaksaxena]

Hello @Anas-Abdelhadi

Currently core tokenizer only supports latin & devanagari scripts; therefore it is not possible to build models for Arabic language(s).

If you are interested to contribute then we can together explore possibility of enhancing the tokenizer and work towards building the same.

Best,
Sanjaya

[Anas-Abdelhadi]

Hi @sanjayaksaxena,
Thanks for replying, yes I am definitely interested in contribution.
How should we proceed?

[sanjayaksaxena]

Thanks @Anas-Abdelhadi... sounds great. The winkNLP tokenizer is inspired by wink-tokenizer — request you to please look at it as that is a simpler version. We will have to figure out how to add capability to process Arabic scripts. You may also drop a line at [email protected].

Best,
Sanjaya