Enable fuzzy text matching in Matcher

[kwhumphreys]

Description

work in progress:

• use Levenshtein distance (polyleven implementation) to allow fuzzy matching of specific tokens in Matcher.
• FUZZY1, FUZZY2, ... FUZZY9 operators specify max string edit distance to allow a match.
• operators can apply to single tokens or sets, e.g. {"LOWER": {"FUZZY3: "string"}} or {"LOWER": {"FUZZY2": {"IN": ["string1", "string2"]}}}

Types of change

enhancement

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[kwhumphreys]

any Cython advice on how to avoid with gil at https://github.com/explosion/spaCy/pull/11359/files#diff-63f1a25e13e4c412a4decf97e30bf2157f1a0d800efb1d0b25679e1991a8b804R697 ?

[adrianeboyd]

Just a few initial notes:

• I think there are a lot of users who would like to have something like this, so this is a nice idea!

• we're pretty conservative about adding new dependencies to spacy, so we'll have to seriously consider whether we want to add rapidfuzz

• this feels like it ought to be an extra operator (similar to REGEX) rather than being a Matcher setting that's applied to a hard-coded subset of attributes

  pattern = [{"ORTH": {"FUZZY": "word"}}]

  Then you could apply it to any attributes that you want including custom extensions. What's trickier with the operator is a parameter for fuzziness.

[kwhumphreys]

Thanks @adrianeboyd. This plugin uses the notation you suggest https://spacy.io/universe/project/spaczz but in my evaluation it is much too slow to be practical, even with minimal use of the operator. Therefore I was trying for a more direct integration with the underlying rapidfuzz package.

Agreed that a user-specified list of attributes would be better than a hard-coded list, but I'd like to be able to specify this on Matcher init rather than on individual tokens. My use case is a large existing set of patterns which I'd like to optionally apply with fuzzy matching, without having to modify every pattern (but perhaps this could be done in a pattern pre-processing stage).

[adrianeboyd]

Have you tried implementing FUZZY within spacy's Matcher similar to REGEX? I think that spazz has some of its own python-only matcher implementation, so it's not just the extra operator that's causing the difference.

If you want to get a general idea of the speed difference just for the operator, you can try comparing {"ORTH": "the"} to {"ORTH": {"REGEX": "^the$"}}.

[kwhumphreys]

added a FUZZY attribute, as suggested (and as compatible with spaczz), but using a fixed fuzzy threshold value specified on init.

also added a parameter to specify a list of attributes on init, to be treated as fuzzy in all patterns, e.g. Matcher(en_vocab, fuzzy=85, fuzzy_attrs=["ORTH", "LOWER"]).

TODO: handle extension attributes
TODO: handle values in sets

[adrianeboyd]

We think this is a good idea and we discussed a bit internally how we'd like to support it:

• we would prefer not to add rapidfuzz as a dependency, but would instead prefer to directly include one edit distance / similarity implementation in spacy (possibly from rapidfuzz, or reimplemented ourselves if that makes more sense)
• we don't think that a top-level fuzzy setting for Matcher or EntityRuler makes sense because the appropriate fuzziness often depends so much on the exact strings / pattern / task and different patterns in the same Matcher would need different levels of fuzziness
• we're still not quite sure how to implement a fuzziness parameter for the operator; my initial clunky idea would be to support a small range of hard-coded operators like FUZZY75, FUZZY80, etc. but we're still thinking about the options/details here (ideas welcome!)
• it's possible there would be no way to support this in combination with set operators

[maxbachmann]

we would prefer not to add rapidfuzz as a dependency, but would instead prefer to directly include one edit distance / similarity implementation in spacy (possibly from rapidfuzz, or reimplemented ourselves if that makes more sense)

You can directly build against the C++ library https://github.com/maxbachmann/rapidfuzz-cpp.

[kwhumphreys]

• it's possible there would be no way to support this in combination with set operators

@adrianeboyd I just spent some time adding support for set operators. see https://github.com/explosion/spaCy/pull/11359/files#diff-d7740e9e8c9929c0b0e8e962c46a22e09263ef4891faceaa15415b2be94a5837R221-R263 but there probably isn't any way to use different fuzziness parameters for different set members.

[kwhumphreys]

• we're still not quite sure how to implement a fuzziness parameter for the operator; my initial clunky idea would be to support a small range of hard-coded operators like FUZZY75, FUZZY80, etc. but we're still thinking about the options/details here (ideas welcome!)

FUZZY1, FUZZY2, ... FUZZYN where N is the allowed Levenshtein string edit distance might be simpler than a 0-100 scale?

[adrianeboyd]

FUZZY1, FUZZY2, ... FUZZYN where N is the allowed Levenshtein string edit distance might be simpler than a 0-100 scale?

I like that idea! It is definitely concrete and easy for users to understand.

[kwhumphreys]

@adrianeboyd a small, fast, MIT-licensed implementation of Levenshtein distance here https://github.com/roy-ht/editdistance
Would this be acceptable to either import or copy into spaCy?

[adrianeboyd]

From looking at examples for the docs, I think my proposed default of 20% (floor) is way too low. It means that it allows 2 edits for any words from 1-14 characters, which isn't much of a useful range.

[adrianeboyd]

It also seemed kind of odd to just have 1..5, so I added 6..9, which gives you a total of ten buckets total to use for some very custom fuzzy_compare method.

[kwhumphreys]

From looking at examples for the docs, I think my proposed default of 20% (floor) is way too low. It means that it allows 2 edits for any words from 1-14 characters, which isn't much of a useful range.

My original function was trying to implement word length - 2 up to a maximum of 5. I don't think I've found any English word pairs where >5 edits is a useful match.

[adrianeboyd]

Having spacy.matcher.matcher.fuzzy_compare in the docs is kind of ugly, but I'm not sure we really want to intentionally export this higher. Unlike levenshtein, there's no particular reason for people to use this independently of the matcher.

[svlandeg]

This is looking great - thanks @kwhumphreys and @adrianeboyd for your work on this!

I had a few comments & suggestions, but overall it'll be great to have this in spaCy 3.5 🙂

[kwhumphreys]

@adrianeboyd note this usage, which I just came across today: https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness
it uses low/high token lengths, similarly to the min/max in the original fuzzy_compare function.