remove_accents option does not work properly anymore

[alainv62]

Since the removal of the unidecode library and its replacement with the unicodedata module in commit 4d517d1, the remove_accents option does not work properly anymore.
Eg: in French, 'référence' is replaced with 'rfrence'.
It seems that the normal form KC is here responsible as the normal form KD works fine with this example ('référence' is properly replaced with 'reference').

[esmeraldas63]

Have encountered same issues with Lithuanian and Latvian letters too

[bosd]

We need to fix this! Currently im travelling. So not much time to attend to this.
Does anyone have a suggestion for an alternative implementation or library?

[alainv62]

Why not just revert to the former solution optimized_str = unidecode(optimized_str) which was working fine?

[bosd]

Why not just revert to the former solution ?

Because of the mentioned license issue.

There are some good alternatives. Like built-ins or alternative libraries ( adding extra dependencies is not preferred).

[alainv62]

What about using the normal form KD ?

[rmilecki]

Why not just revert to the former solution optimized_str = unidecode(optimized_str) which was working fine?

See description in pull request that introduced that change: #436

And for the orginal report: #435

[bosd]

What about using the normal form KD ?

Yes, that seems to fix it..
small test:

>>> str1="référence"
>>> unicodedata.normalize('NFKD', str1).encode('ascii', 'ignore').decode('ascii')
'reference'

>>> str2="ä"
>>> unicodedata.normalize('NFKD', str2).encode('ascii', 'ignore').decode('ascii')
'a'

[bosd]

It is probably an good idea to add a test for the remove accents function.
Is there an example invoice to add to the tests?