Does rapidfuzz work with other languages than English?

[lingvisa]

I am working with Asian languages like Japanese, Chinese. Does rapidfuzz work with non-ascill characters?

Thanks for clarification.

[maxbachmann]

Yes RapidFuzz works with any unicode characters.

Note however that many of the implementations have an optimized implementation for characters below 256 (extended ascii), since it can directly look those up in an array[256] which is not possible for the full unicode range. So when you run into performance problems it might help to transform the most common characters of the language to ascii characters.

[lingvisa]

In PhraseMatcher, usually one post takes only 1 ms to complete. Can you give me a rough idea of the complexity of 'triangle inequality' and how far away it is from trie match? In my application, I can not afford to spend much time on dictionary match.

[maxbachmann]

One example of an alternative are levenshtein automatons as described e.g. here: https://blog.mikemccandless.com/2011/03/lucenes-fuzzyquery-is-100-times-faster.html

However I have absolutely no clue how fast those would be in your use case.

[lingvisa]

A second thought for optimization is that, my dictionary size can be cut into half by limiting the term length >=6 chars, because short sequence is much more ambiguous so the precision would be very low. By this way, the 800K reduces to 400K. Could you benchmark again how long does it take to process one post using the code above? The reduced time should be more than half. Right?

[maxbachmann]

Could you benchmark again how long does it take to process one post using the code above?

In practice it will be a lot faster:

1. you have very different string lengths. RapidFuzz will skip many of those, since given your score_cutoff they can not be similar enough
2. shorter strings are faster to calculate

E.g. by selecting a random string length between 0 and 400 instead of making all strings length 400 I reach a lot faster run time. Here are the times for 1m elements with len 0-400
ascii = 233us -> 2.3ns per element
non-ascii = 277us -> 2.8ns per element

This should be really benchmarked with your actual data, since this largely depends on the string lengths your dealing with. You should be able to easily benchmark this yourself with process.extract.

[lingvisa]

Yes, I will definitely test it with my data in integration with Spacy and update here once I do it.