[iOS][non-icu] Implement missing Locale and Casing functions #94038

mkhamoyan · 2023-10-26T13:11:29Z

Implement below functions

GetDefaultLocaleNameNative
IsPredefinedLocaleNative

Run all tests under System.Globalization.Tests and adjust them for hybrid mode

ghost · 2023-10-26T13:11:40Z

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Implement below functions

InitOrdinalCasingPageNative
GetDefaultLocaleNameNative
IsPredefinedLocaleNative

Run all tests under System.Globalization.Tests and adjust them for hybrid mode

Contributes to #80689
cc @SamMonoRT

Author:	mkhamoyan
Assignees:	mkhamoyan
Labels:	`area-System.Globalization`
Milestone:	-

ghost · 2023-10-26T13:11:50Z

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos
See info in area-owners.md if you want to be subscribed.

Issue Details

Implement below functions

InitOrdinalCasingPageNative
GetDefaultLocaleNameNative
IsPredefinedLocaleNative

Run all tests under System.Globalization.Tests and adjust them for hybrid mode

Contributes to #80689
cc @SamMonoRT

Author:	mkhamoyan
Assignees:	mkhamoyan
Labels:	`os-ios`
Milestone:	-

mkhamoyan · 2023-10-26T13:13:20Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-10-26T13:13:32Z

Azure Pipelines successfully started running 1 pipeline(s).

mdh1418

Are all of the tests that we are disabling for HybridGlobalizationOnOSX expected to not work?

mdh1418 · 2023-10-26T17:44:27Z

src/native/libs/System.Globalization.Native/pal_casing.m

@@ -149,4 +149,25 @@ int32_t GlobalizationNative_ChangeCaseInvariantNative(const uint16_t* lpSrc, int
    }
 }

+void GlobalizationNative_InitOrdinalCasingPageNative(int32_t pageNumber, UChar* pTarget)


I'm forgetting, does pal_casing.c get trimmed when we set HybridGlobalization to true?

Now it does not get trimmed in hybrid mode. But once we will not load ICU at all pal_casing.c will also not be loaded.

But once we will not load ICU at all pal_casing.c will also not be loaded.

What would this look like? Would it get trimmed out or would it be a file exclusion?
Since this and GlobalizationNative_InitOrdinalCasingPage have the exact same logic`, it would be nice if we found a way to not duplicate this code.

From

runtime/src/mono/mono/mini/CMakeLists.txt

Line 61 in f2e4ddf

pal_casing.c

and

runtime/src/native/libs/System.Globalization.Native/CMakeLists.txt

Line 61 in f2e4ddf

pal_casing.c

we will filter out for Apple platforms pal_casing.c

Our goal is to make hybrid globalization default mode for Apple platforms and in that case we will not need most of the ICU related functions (we will not need most of the pal_"name".c). I guess at the end to not have duplicates I can move all these kind of functions in some common place like pal_common.c

Were there other instances where the logic was exactly the same as the ICU function? This is the first that I noticed was exactly the same, but maybe if we anticipate more of this moving into a pal_common.c might be a good way to not duplicate code. Does this logic need the @autoreleasepool{} if its not allocating any memory?

No this was first function with exactly same logic. Moved it to pal_common.c .

src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.Unix.cs

src/libraries/Common/tests/TestUtilities/System/PlatformDetection.cs

mdh1418 · 2023-10-26T18:15:19Z

src/libraries/System.Globalization/tests/CompareInfo/CompareInfoTestsBase.cs

@@ -45,5 +45,6 @@ public class CompareInfoTestsBase
                                                              s_invariantCompare.Compare("\u3060", "\uFF80\uFF9E", CompareOptions.IgnoreKanaType | CompareOptions.IgnoreWidth | CompareOptions.IgnoreCase) == 0;

        protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm() => !PlatformDetection.IsHybridGlobalizationOnBrowser && IsNotWindowsKanaRegressedVersion();
+        protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalization() => IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm() && !PlatformDetection.IsHybridGlobalizationOnOSX;


Suggested change

protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalization() => IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm() && !PlatformDetection.IsHybridGlobalizationOnOSX;

protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalization() => IsNotWindowsKanaRegressedVersion() && PlatformDetection.IsNotHybridGlobalization();

Similar to the comment in PlatformDetection.cs

Since we have IsNotHybridGlobalizationOnBrowser, maybe we can update the !PlatformDetection.IsHybridGlobalizationOnBrowser to be PlatformDetection.IsNotHybridGlobalizationOnBrowser.

Also it would be nice to be consistent with the Browser vs Wasm. Would there be HybridGlobalization on Wasi?

In the future maybe there will be HybridGlobalization on WASI, but now all the implementation in WASM is done using browser API and there is no browser API in WASI.

Right, I would expect IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm in the future to be !PlatformDetection.IsHybridGlobalizationOnWasm && IsNotWindowsKanaRegressedVersion() (what I meant by consistent)

Where maybe PlatformDetection.IsHybridGlobalizationOnWasm = PlatformDetection.IsHybridGlobalizationOnBrowser || PlatformDetection.IsHybridGlobalizationOnWasi.

So to that pattern IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm -> IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnBrowser here would be more consistent but thats just a nit and can be addressed when HybridGlobalization is extended to Wasi.

src/libraries/System.Private.CoreLib/src/System/Globalization/CultureData.Icu.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CultureInfo.Unix.cs

mdh1418 · 2023-10-26T18:45:47Z

src/native/libs/System.Globalization.Native/pal_locale.m

+    @autoreleasepool
+    {
+        NSString *localeIdentifier = [NSString stringWithFormat:@"%s", localeName];
+        NSString *desiredLocaleIdentifier = [localeIdentifier stringByReplacingOccurrencesOfString:@"-" withString:@"_"];


Why do we replace all - with _? NSLocale availableLocaleIdentifiers returns a Locale ID which may include - for script designators right?

While debugging I noticed that availableLocaleIdentifiers doesn't have any Local ID containing -.

Would that mean that [NSLocale availableLocaleIdentifiers] doesnt contain Local IDs with _ on your system? Could it be different based on the OS version or on other devices?

Which desiredLocaleIdentifier contained a - that we needed to cast to _ to match the availableLocales?

Given that the docs specify there is a possibility of including -, because we only cast desiredLocaleIdentifier to only have _, it seems like there is a chance to miss a match.

Here are examples from availableLocaleIdentifiers in my system

I can see 949 Local IDs and none of them has -.
You are right this can be my system specific result. Will modify to not miss a match.

For all calls to IcuIsEnsurePredefinedLocaleName (seems like just CultureInfo.GetCultureInfo(string, bool) and CultureData.GetCultureData(string?, bool), it doesn't seem like there is any restriction or assertion on the input name.

If we see instances of the name passed to IcuIsEnsurePredefinedLocaleName being formatted as [language designator]-[region designator] (e.g. zh-HK), I'm wondering if any names containing a script designator would be formatted as [language designator]-[script desginator]-[region designator] (e.g. zh-Hans-HK). Looking at

runtime/src/libraries/System.Globalization/tests/CultureInfo/CultureInfoAll.cs

Line 622 in c1f4341

yield return new object[] { 0x0c04, new [] { "zh-hk" }, "zh-HK", "zho", "ZHH", "zh-Hant-HK", "zh-HK" };

, it seems that this is likely the case. Speaking of which we should double check if for zh-HK if the script should be Hans (the apple doc) or Hant (code snippet).

If the formatting will be passed as such, then neither the original input name nor the version where we replace - with _ will match the documented format for availableLocaleIdentifiers [language designator]-[script designator]_[region designator].

Perhaps we can check if the input string has three components, and if so, ensure its in the format of *-*_*. That being said, we should get a better idea of what the range of inputs we expect to this function. Maybe it would be sufficient to see if availableLocales contains either localeIdentifier or localeIdentifier.ReplaceLastOccurenceOf('-', '_').

Looks like that apple doc and what I get in availableLocaleIdentifiers are not same. E.g. zh-Hant_HK in availableLocaleIdentifiers is zh_Hant_HK, so if I change the implementation to have localeIdentifier.ReplaceLastOccurenceOf('-', '_') for the test case zh-Hans-HK it will return false but in reality it is in the availableLocaleIdentifiers but with the format [language designator]_[script designator]_[region designator] not like mentioned in the doc.

Thats unfortunate... is there a place with more up-to-date docs? Now I'm not sure if its consistent across all apple OS/version/etc. But if we do have a test that tests IcuIsEnsurePredefinedLocaleName I guess that can be ran on all of the apple platforms.

For now I think it might be fine to just go with your original logic that replaced all - with _. Otherwise we would need to just extract the components (sections separated by - and _)

From the code I found out that Apple calls to ICU for that information, see CFLocaleCopyAvailableLocaleIdentifiers here https://opensource.apple.com/source/CF/CF-1153.18/CFLocale.c.auto.html. uloc_getAvailable calls _canonicalize and in that function looks like all parts in LocalID are appended by '_' https://github.com/dotnet/icu/blob/dotnet/main/icu/icu4c/source/common/uloc.cpp#L1537

mkhamoyan · 2023-10-27T14:35:19Z

Are all of the tests that we are disabling for HybridGlobalizationOnOSX expected to not work?

Yes. These are cases which either were not possible (yet) to implement in hybrid mode, or behavioural changes after implementation

mkhamoyan · 2023-10-30T12:41:12Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-10-30T12:41:29Z

Azure Pipelines successfully started running 1 pipeline(s).

mkhamoyan · 2023-10-30T14:20:33Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-10-30T14:20:49Z

Azure Pipelines successfully started running 1 pipeline(s).

mdh1418 · 2023-10-30T16:56:11Z

These are cases which either were not possible (yet) to implement in hybrid mode, or behavioural changes after implementation

Can we document the behavior changes/impossible to implement, and if it falls under the "not yet possible", can we have an issue tracking that?

mkhamoyan · 2023-10-31T14:16:54Z

These are cases which either were not possible (yet) to implement in hybrid mode, or behavioural changes after implementation

Can we document the behavior changes/impossible to implement, and if it falls under the "not yet possible", can we have an issue tracking that?

Behaviour changes and things that are not supported are documented here https://github.com/dotnet/runtime/blob/main/docs/design/features/globalization-hybrid-mode.md .

Opened a ticket for the functionalities to investigate again if there is a possibility to implement #94216

mdh1418

After ensuring GlobalizationNative_IsPredefinedLocaleNative behavior is the same on all apple platforms for all localeName, looks good to me! Thanks!

ghost · 2023-10-31T16:10:21Z

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Implement below functions

InitOrdinalCasingPageNative
GetDefaultLocaleNameNative
IsPredefinedLocaleNative

Run all tests under System.Globalization.Tests and adjust them for hybrid mode

Contributes to #80689
cc @SamMonoRT

Author:	mkhamoyan
Assignees:	mkhamoyan
Labels:	`area-System.Globalization`, `os-ios`
Milestone:	-

mkhamoyan · 2023-10-31T16:27:05Z

/azp run runtime-extra-platforms

mkhamoyan · 2023-10-31T17:09:22Z

/azp run runtime-extra-platforms

azure-pipelines · 2023-10-31T17:09:47Z

Azure Pipelines successfully started running 1 pipeline(s).

mkhamoyan · 2023-11-01T10:16:44Z

Failures are not related.

Implement missing functions in locale

7eccecf

ghost assigned mkhamoyan Oct 26, 2023

dotnet-issue-labeler bot added the area-System.Globalization label Oct 26, 2023

mkhamoyan added os-ios Apple iOS and removed area-System.Globalization labels Oct 26, 2023

mkhamoyan requested review from steveisok, akoeplinger and mdh1418 October 26, 2023 13:12

mkhamoyan marked this pull request as ready for review October 26, 2023 13:13

runfoapp bot mentioned this pull request Oct 26, 2023

system.text.regularexpressions.tests work item #72834

Open

mdh1418 reviewed Oct 26, 2023

View reviewed changes

Merge branch 'main' into hyrbid_locale_functions

bf55d21

Changes requested by review

44cfd2c

This was referenced Oct 27, 2023

[wasm][debugger] Debugger tests getting randomly aborted on CI #89409

Open

The active test run was aborted. Reason: Test host process crashed dotnet/dnceng#451

Open

[wasm] Runtime tests' build gets terminated with exited with code 137 #94077

Closed

mkhamoyan added 2 commits October 30, 2023 13:28

Fix LocalD check and minor refactoring

0ea3dbc

Fix spacing

b3d7bd0

mkhamoyan requested a review from mdh1418 October 30, 2023 12:41

mkhamoyan added 2 commits October 30, 2023 14:33

Fix build failure

42ff127

Remove duplicates

2414ce9

This was referenced Oct 30, 2023

[mono] Tests failing with Failed to find compatible device: arm64-v8a #87267

Closed

[mono][tvos] OOM in System.IO.Tests.MemoryStreamTests #92467

Closed

mdh1418 approved these changes Oct 31, 2023

View reviewed changes

tarekgh added the area-System.Globalization label Oct 31, 2023

Remove duplicate by moving code to pal_common.c

c7c9ade

mkhamoyan requested review from vargaz, lambdageek and SamMonoRT as code owners October 31, 2023 16:25

Fix android build

daed1f5

build-analysis bot mentioned this pull request Oct 31, 2023

[browser][hybridGlobalization] WBT: Hybrid Globalization fails with AOT #94212

Closed

mkhamoyan requested a review from ilonatommy November 1, 2023 08:27

Merge branch 'main' into hyrbid_locale_functions

1f37040

mkhamoyan merged commit 68dde19 into dotnet:main Nov 1, 2023
185 of 188 checks passed

mkhamoyan deleted the hyrbid_locale_functions branch November 1, 2023 10:17

ghost locked as resolved and limited conversation to collaborators Dec 1, 2023

	protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalization() => IsNotWindowsKanaRegressedVersionAndNotHybridGlobalizationOnWasm() && !PlatformDetection.IsHybridGlobalizationOnOSX;
	protected static bool IsNotWindowsKanaRegressedVersionAndNotHybridGlobalization() => IsNotWindowsKanaRegressedVersion() && PlatformDetection.IsNotHybridGlobalization();

[iOS][non-icu] Implement missing Locale and Casing functions #94038

[iOS][non-icu] Implement missing Locale and Casing functions #94038

Conversation

mkhamoyan commented Oct 26, 2023 • edited Loading

ghost commented Oct 26, 2023

ghost commented Oct 26, 2023

mkhamoyan commented Oct 26, 2023

azure-pipelines bot commented Oct 26, 2023

mdh1418 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkhamoyan Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkhamoyan Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkhamoyan Oct 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkhamoyan Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

mkhamoyan commented Oct 27, 2023

mkhamoyan commented Oct 30, 2023

azure-pipelines bot commented Oct 30, 2023

mkhamoyan commented Oct 30, 2023

azure-pipelines bot commented Oct 30, 2023

mdh1418 commented Oct 30, 2023

mkhamoyan commented Oct 31, 2023

mdh1418 left a comment

Choose a reason for hiding this comment

ghost commented Oct 31, 2023

mkhamoyan commented Oct 31, 2023

mkhamoyan commented Oct 31, 2023

azure-pipelines bot commented Oct 31, 2023

mkhamoyan commented Nov 1, 2023

mkhamoyan commented Oct 26, 2023 •

edited

Loading

mkhamoyan Oct 31, 2023 •

edited

Loading

mkhamoyan Oct 31, 2023 •

edited

Loading

mkhamoyan Oct 27, 2023 •

edited

Loading

mkhamoyan Oct 31, 2023 •

edited

Loading