CharUnicodeInfo.GetUnicodeCategory reports wrong categories

[tarekgh]

@drauch commented on Sat Oct 20 2018

The character 0x1C90 represents the GEORGIAN MTAVRULI CAPITAL LETTER AN and is according to the official Unicode documentation part of the general category Lu ("Letter, uppercase").

This is the entry from the official Unicode 11.0 database:

1C90;GEORGIAN MTAVRULI CAPITAL LETTER AN;Lu;0;L;;;;;N;;;;10D0;

However, the following .NET code:

char.GetUnicodeCategory('\x1C90')
// or
CharUnicodeInfo.GetUnicodeCategory('\x1C90')

both return UnicodeCategory.OtherNotAssigned for some reason.

What is the problem here? Do I have to install some specific .NET language pack so that the Unicode category is reported correctly? Is this a bug in the .NET Framework? I've tested it with up-to-date versions of the .NET Framework and .NET Core.

---

@drauch commented on Sat Oct 20 2018

Hmm, I've got the problem pointed out at https://stackoverflow.com/questions/52909494/why-does-char-getunicodecategory-report-wrong-categories -> your Unicode version is stuck at 5.0, even for .NET Core. That's a pity. I can guess why you don't want to update the information for char.GetUnicodeCategory() (backward compatibility), but CharUnicodeInfo has been introduced to return the most up-to-date information, hasn't it?

• Are you planning to upgrade your version of Unicode?
• Is there a compatibility switch to obtain up-to-date information?

Also the https://docs.microsoft.com/en-us/dotnet/api/system.globalization.charunicodeinfo.getunicodecategory page should document the fact that you're using Unicode 5.0.

---

@karelz commented on Mon Oct 22 2018

@tarekgh does it belong into CoreFX repo? Or is it easy to answer here? (GH issue mover is broken, so taking lazy approach before moving it manually)

---

@tarekgh commented on Mon Oct 22 2018

@drauch

.Net core is using Unicode 10 and we didn't moved to 11 yet. We can try to update that in .Net Core 3.0 as needed.

your Unicode version is stuck at 5.0,

This is not true. the doc https://docs.microsoft.com/en-us/dotnet/api/system.globalization.unicodecategory?redirectedfrom=MSDN&view=netframework-4.7.2#remarks is talking about the enumeration type and not about the supported version of the Unicode data we have in the .NET

Is there a compatibility switch to obtain up-to-date information?

We don't provide any comparability switch.

I'll move this issue to corefx

[drauch]

If I'd like to provide a pull request for this, what would I have to do? I checked https://github.com/dotnet/corefx/blob/master/src/Common/src/CoreLib/System/Globalization/CharUnicodeInfo.cs however InternalGetCategoryValue references a field s_pCategoryLevel1Index which is nowhere to be found (partial class, but a search in the whole repo does not bring up another match). Would gladly support this request if you tell me how to do so :)

[tarekgh]

@drauch thanks for offering help. I already opened a PR for that dotnet/coreclr#20529 and going to update our test.

[tarekgh]

dotnet/corefx#32968

[tarekgh]

it is done through the PR dotnet/coreclr#20529

[tarekgh]

reopened as it looks the file CharUnicodeInfoData.cs was not updated in the PR. I'll submit another PR for that.

[tarekgh]

dotnet/coreclr#20589 should be addressed this issue now.