Need APIs for unicode case mapping and case folding

[tarekgh]

@gafter commented on Mon May 11 2015

The Unicode function for determining the category of Unicode characters (System.Globalization.CharUnicodeInfo.GetUnicodeCategory(char)) appears to reflect a fairly recent version of the Unicode standard. However, the functions for performing Unicode case mapping (culture-insensitive uppercase and lowercase) appear to reflect Unicode version 1.0. This mismatch was a severe impediment for implementing spec-compliant case folding in the VB compiler.

There is also no support whatsoever for performing case folding according to the Unicode specification (simplified or otherwise). APIs in the platform to do this would help with any language that, like VB, wants to depend on the Unicode specification for the meaning of case-insensitive identifiers.

---

@jkotas commented on Mon May 11 2015

cc @ellismg

---

@ellismg commented on Mon May 11 2015

This is something I'd like to do. I think the open issue is how we support this on Windows (since I think that we can get Case Folding functionality from ICU).

In general, we've tried to move away from the framework itself shipping globalization data and used the OS provided APIs instead. Since Windows doesn't currently expose a way to do case folding, we'd have to figure out what to do.

---

@karelz commented on Fri Mar 17 2017

@tarekgh do we need a new API for this? If yes, we should move it to CoreFX. If not, let's remove "api addition" label.

[iSazonov]

Could we start designing the simple case folding API?

[tarekgh]

@iSazonov wouldn't the supported string normalization and casing APIs are not enough in your case?

[gafter]

@tarekgh What APIs are you suggesting approximate case folding?

[tarekgh]

We already have a casing APIs on TextInfo and String class. I understand we use the underlying OS for that and this is by design as we need to avoid carrying any globalization data as possible. what exactly the scenario that is not working for you?

[gafter]

The scenario is Case Folding as specified in the Unicode specification. See also

• http://unicode.org/faq/casemap_charprop.html
• http://www.unicode.org/versions/Unicode11.0.0/ch05.pdf#G21180 starting on page 240
• ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt

[tarekgh]

I know that, I am asking about what is your scenario you need the case folding and case mapping is not enough?

[gafter]

The scenario is case folding for the purposes of comparing identifiers in a case insensitive programming language according to the rules described in the Unicode specification. Case mapping is not enough; that is explained in some detail in the Unicode specification. There is no case folding API today.

[tarekgh]

Thanks @gafter

Considering we didn't get high demand on this functionality and we'll need to carry some Unicode data to be able to support it, I would suggest this functionality can be implemented in some NuGet package and doesn't have to be in the core for now. If we see there is more demand on that we can consider having it in core. we can use corefxlab if needed for hosting the code if needed.

[iSazonov]

Please see https://github.com/dotnet/corefx/issues/33047 and PowerShell/PowerShell#8120

In PowerShell/PowerShell#8120 I provide a plan. In short we hope to get performance benefits in PowerShell, RegEx and string comparisons in general.
In PowerShell we could win in engine because it use string comparisons everywhere. Also we could speed up Select-String cmdlet get parity with Unix grep and fastest Rust ripgrep utility. Ripgrep uses simple case folding. It show that we could get great win in CoreFX RegEx too.

On the week I prepared alfa code for simple case folding and while the results are encouraging.

[GrabYourPitchforks]

See also https://github.com/dotnet/corefx/issues/41333, which is a proposal to make string / char / Rune use these mappings by default. It doesn't involve any new API addition.

[GrabYourPitchforks]

If we did want to provide APIs for this, they'd probably look like the following:

public namespace System.Globalization
{
    public static class CharUnicodeInfo
    {
        // performs "simple" culture-unaware mapping to uppercase using CharUnicodeInfo backing data
        public static char ToUpperInvariant(char ch);
        public static int ToUpperInvariant(int codePoint);
        public static Rune ToUpperInvariant(Rune rune);

        // performs "simple" culture-unaware mapping to lowercase using CharUnicodeInfo backing data
        public static char ToLowerInvariant(char ch);
        public static int ToLowerInvariant(int codePoint);
        public static Rune ToLowerInvariant(Rune rune);

        // performs "simple" culture-unaware mapping to titlecase using CharUnicodeInfo backing data
        public static char ToTitleInvariant(char ch);
        public static int ToTitleInvariant(int codePoint);
        public static Rune ToTitleInvariant(Rune rune);

        // performs "simple" case-fold mapping using CharUnicodeInfo backing data
        public static char ToCaseFold(char ch);
        public static int ToCaseFold(int codePoint);
        public static Rune ToCaseFold(Rune rune);
    }
}

The behavior of all of these APIs is that they'd use the data that's already present in the code-behind for CharUnicodeInfo (generated from UnicodeData.txt and CaseFolding.txt). APIs which take int will throw if the input is outside the range [ U+0000..U+10FFFF ] (UTF-16 surrogate code points are allowed and will map to themselves). APIs which take char and Rune do not throw.

[tarekgh]

@GrabYourPitchforks would it make sense to move these functionality to char and string classes (and support string case folding too)?

[GrabYourPitchforks]

@tarekgh If we did https://github.com/dotnet/corefx/issues/41333, I think that would make sense. Otherwise we could end up with a mixture of ICU and NLS behaviors hanging off of string and char, and IMO it would be difficult to reason about.

[tarekgh]

I believe in the near future we'll use ICU across all platforms which will help not having this mixture issue.

[gafter]

The behavior of all of these APIs is that they'd use the data that's already present in the code-behind for CharUnicodeInfo (generated from UnicodeData.txt and CaseFolding.txt).

I don't understand. Unicode has no concept of an invariant mapping (i.e. one that that fails to follow the evolving Unicode standard).