fix: correct the search logic for BertWordpieceTokenizer

[Apollon77]

Pull Request Template

PR Checklist

• I have run npm test locally and all tests are passing.
• I have added/updated tests for any new behavior.
• If this is a significant change, an issue has already been created where the problem / solution was discussed: [N/A, or add link to issue here]

PR Description

fixes #1192

I took he changes proposed in #1192, adjusted the tests to be successful and also added a test based on the description of the ticket (but honestly, @eric-lara @ericzon you need to state if the sematics of the issue is right ;-)) )
I also tried to add some more tests based on tze changes I understood so far. I hope that helps

Credits go to @juancavallotti

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[ericzon]

Taking a look on the example commented in #1192
"'vegan', the expected result is vega ##n, but you get v ##egan"
but I see in the tests:

This doesn't match, could you clarify which one is right?

[Apollon77]

Honestly... would be a question for @juancavallotti ... I just took over the Code and tried to finalize it and adjusted/added tests ... in fact this is the response we get from "His code changes" ... I can not judge that (see also notes in first post). Sorry

[juancavallotti]

@ericzon The split will depend on your vocab file. The algorithm should favor longer chunks of text from your vocab and start breaking down from there. Can you post the vocab file that you're using? That should help identify the issue.

[Apollon77]

https://github.com/axa-group/nlp.js/tree/master/packages/bert-tokenizer/dicts ... should it be

[juancavallotti]

If you see that file is cased and it doesn't have the token vega but Vega, the test I used was with an uncased vocabulary.