Limit `PNSE` by using Invariant `HashCode` in `HybridGlobalization` #96354

ilonatommy · 2023-12-29T12:15:58Z

Fixes paritally #95921.

We used to throw PNSE for all HashCode functions. The consequence was throwing PNSE for quite a few APIs, some of them crucial and far-connected with Globalization. With this PR the following APIs do not throw PNSE always (see the update in the doc for details):

- System.Collections.Hashtable.Add 
- System.Collections.Hashtable.GetHash 
- System.Collections.CaseInsensitiveHashCodeProvider.GetHashCode 
- System.Collections.Specialized.NameValueCollection.Add 
- System.Collections.Specialized.NameValueCollection.Remove 
- System.Collections.Specialized.NameValueCollection.Get 
- System.Collections.Specialized.NameValueCollection.Set 
- System.Collections.Specialized.NameObjectCollectionBase.BaseAdd 
- System.Collections.Specialized.NameObjectCollectionBase.BaseGet 
- System.Collections.Specialized.NameObjectCollectionBase.BaseRemove 
- System.Collections.Specialized.NameObjectCollectionBase.BaseSet 
- System.Collections.Specialized.OrderedDictionary.Add 
- System.Collections.Generic.HashSet 
- System.Collections.Generic.Dictionary

JS does not expose native, locale-sensitive hashing that we could use, so we decided to utilize Invariant version of functions instead with some limitations. Invariant hashing algorithm differs from ICU hashing with not ignoring empty unicode chars (e.g. zero width joiner), the strings get cleaned from them first.

Invariant locale - always supported with this PR by using invariant hashing method. This approach differs from the ICU approach where invariant culture was getting data from ICU4C.
Non-invariant locales with CompareOptions.None - use invariant hashing algo.
Non-invariant locales with CompareOptions.IgnoreCase - use JS-native toLower() and then invariant hashing algo.
Non-invariant locales with any other CompareOptions e.g. CompareOptions.IgnoreCase | CompareOptions.IgnoreSymbols - keep throwing PNSE, as it was without this PR.

It will be documented in the HG doc, so far I did not find an example where the behavior of APIs that threw PNSE so far differs between ICU and HG because of this change.

Out of scope (for follow-up PRs):

performance tests for methods using HashCode and performance improvements
SortKey - investigate similar change

ghost · 2023-12-29T12:16:05Z

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #95921.

Author:	ilonatommy
Assignees:	ilonatommy
Labels:	`arch-wasm`, `area-System.Globalization`
Milestone:	-

azure-pipelines · 2023-12-29T14:52:20Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2023-12-29T18:05:38Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-01-02T15:12:27Z

Azure Pipelines successfully started running 1 pipeline(s).

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.cs

azure-pipelines · 2024-01-02T17:09:55Z

Azure Pipelines successfully started running 1 pipeline(s).

...braries/System.Runtime/tests/System.Globalization.Extensions.Tests/GetStringComparerTests.cs

src/libraries/System.Runtime/tests/System.Runtime.Extensions.Tests/System/StringComparer.cs

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/StringGetHashCodeTests.cs

src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/RuneTests.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.Icu.cs

azure-pipelines · 2024-01-17T16:10:43Z

Azure Pipelines successfully started running 1 pipeline(s).

…ted for non-invariant cultures only with `IgnoreCase` or `None` options.

pavelsavara

Thanks for patience with my feedback.

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.WebAssembly.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.WebAssembly.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.Icu.cs

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.WebAssembly.cs

azure-pipelines · 2024-01-19T13:51:44Z

Azure Pipelines successfully started running 1 pipeline(s).

ilonatommy · 2024-01-19T16:21:01Z

/azp run runtime-wasm-libtests

azure-pipelines · 2024-01-19T16:21:12Z

Azure Pipelines successfully started running 1 pipeline(s).

ilonatommy · 2024-01-19T21:23:38Z

AOT lanes won't finish, they are generally broken. Failures are not related.

@matouskozak

…otnet#96354) * Unblock all tests. * SortKey is Invariant now. * Missing SortKey changes + fix HashCode. * Typo * Revert unbocking tests connected with CompareOptions PNSE. * Hashing uses Invariant mode -these tests should be skipped. * Add active issue. * feedback * Feedback * Better documentation. * Add new tests + sanitize string before invariant comparison. * Comment + more cases. * Clean CI. * Missing change for clean CI commit. * `SortKey` not supported for non-invariant cultures, `HashCode` supported for non-invariant cultures only with `IgnoreCase` or `None` options. * Feedback. * Fix build, add docs. * Feedback @matouskozak @pavelsavara * Added tests + fixed algo. * Block failing tests for a follow-up PR. * Add more details to PNSE. * Feedback - correct comment

* Moving the tests, cleaning up. * Removal. * Rename: fix build. * Fix order, apply feedback * Missing changes. * Fixed by #96354 * This test uses SortKey, should not be enabled. * Nit * These methods use `CompareOptions = IgnoreCase, IgnoreKanaType, IgnoreWidth`, should stay blocked.

ilonatommy added 2 commits December 29, 2023 12:14

Unblock all tests.

7f5d711

SortKey is Invariant now.

709da38

ilonatommy added arch-wasm WebAssembly architecture area-System.Globalization labels Dec 29, 2023

ilonatommy self-assigned this Dec 29, 2023

ilonatommy added 3 commits December 29, 2023 14:12

Missing SortKey changes + fix HashCode.

6e72858

Typo

2d7a037

Revert unbocking tests connected with CompareOptions PNSE.

b896c17

dotnet deleted a comment from azure-pipelines bot Dec 29, 2023

This was referenced Dec 29, 2023

System.Text.Json failing some large file tests #59678

Closed

[mono][tests] System.Runtime.Caching assert fails on linux-bionic #93106

Closed

[6.0] System.Random test failure: NextInt_AllValuesAreWithinSpecifiedRange #83421

Open

Hashing uses Invariant mode -these tests should be skipped.

de22e70

ilonatommy mentioned this pull request Jan 2, 2024

[browser] HybridGlobalization allows different hashes for strings that return true for Equals #96400

Closed

Add active issue.

5778cd0

ilonatommy requested a review from matouskozak January 2, 2024 15:12

ilonatommy requested review from mkhamoyan, SamMonoRT and tarekgh January 2, 2024 15:12

ilonatommy marked this pull request as ready for review January 2, 2024 15:13

stephentoub reviewed Jan 2, 2024

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Globalization/CompareInfo.cs Outdated Show resolved Hide resolved

feedback

d0666cd

build-analysis bot mentioned this pull request Jan 2, 2024

Checkout failure: "Git fetch failed with exit code 128" dotnet/arcade#9009

Open

2 tasks

pavelsavara approved these changes Jan 3, 2024

View reviewed changes

Feedback

65236ab

build-analysis bot mentioned this pull request Jan 17, 2024

BasicEventSourceTests.TestsManifestGeneration test failing in CI #97103

Closed

build-analysis bot mentioned this pull request Jan 17, 2024

Test failure - System.NullReferenceException in System.Threading.Lock.TryInitializeStatics #94728

Closed

SortKey not supported for non-invariant cultures, HashCode suppor…

9423bcc

…ted for non-invariant cultures only with `IgnoreCase` or `None` options.

pavelsavara reviewed Jan 18, 2024

View reviewed changes

ilonatommy changed the title ~~Limit PNSE by using Invariant SortKey in HybridGlobalization~~ Limit PNSE by using Invariant HashCode and SortKey in HybridGlobalization Jan 18, 2024

ilonatommy changed the title ~~Limit PNSE by using Invariant HashCode and SortKey in HybridGlobalization~~ Limit PNSE by using Invariant HashCode in HybridGlobalization Jan 18, 2024

ilonatommy added 4 commits January 18, 2024 11:34

Feedback.

2887914

Fix build, add docs.

a599f77

Feedback @matouskozak @pavelsavara

c9dc938

Merge branch 'main' into fix-95921

bfb12b3

build-analysis bot mentioned this pull request Jan 19, 2024

[browser][MT] various WebWorkerTest CI failures #96628

Closed

Added tests + fixed algo.

398d047

ilonatommy added 3 commits January 19, 2024 16:16

Block failing tests for a follow-up PR.

ee1f542

Add more details to PNSE.

10e9060

Feedback - correct comment

80d9d37

pavelsavara approved these changes Jan 19, 2024

View reviewed changes

build-analysis bot mentioned this pull request Jan 19, 2024

[browser][AOT] convert the LLVM value V to DTYPE #97217

Closed

ilonatommy merged commit 993b23f into dotnet:main Jan 19, 2024
181 of 193 checks passed

EgorBo mentioned this pull request Jan 20, 2024

Test failing: System.Globalization.Tests.CompareInfoHashCodeTests.CheckHashingOfSkippedChars #97241

Closed

ilonatommy mentioned this pull request Jan 22, 2024

Hash code tests cleanup #97299

Merged

ilonatommy added a commit to ilonatommy/runtime that referenced this pull request Jan 22, 2024

Fixed by dotnet#96354

91079fa

ilonatommy mentioned this pull request Jan 22, 2024

[browser] HybridGlobalization correct HashCode ranges of skipped unicodes #97351

Closed

github-actions bot locked and limited conversation to collaborators Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Limit `PNSE` by using Invariant `HashCode` in `HybridGlobalization` #96354

Limit `PNSE` by using Invariant `HashCode` in `HybridGlobalization` #96354

ilonatommy commented Dec 29, 2023 •

edited

Loading

ghost commented Dec 29, 2023

azure-pipelines bot commented Dec 29, 2023

azure-pipelines bot commented Dec 29, 2023

azure-pipelines bot commented Jan 2, 2024

azure-pipelines bot commented Jan 2, 2024

azure-pipelines bot commented Jan 17, 2024

pavelsavara left a comment

azure-pipelines bot commented Jan 19, 2024

ilonatommy commented Jan 19, 2024

azure-pipelines bot commented Jan 19, 2024

ilonatommy commented Jan 19, 2024

Limit PNSE by using Invariant HashCode in HybridGlobalization #96354

Limit PNSE by using Invariant HashCode in HybridGlobalization #96354

Conversation

ilonatommy commented Dec 29, 2023 • edited Loading

Out of scope (for follow-up PRs):

ghost commented Dec 29, 2023

azure-pipelines bot commented Dec 29, 2023

azure-pipelines bot commented Dec 29, 2023

azure-pipelines bot commented Jan 2, 2024

azure-pipelines bot commented Jan 2, 2024

azure-pipelines bot commented Jan 17, 2024

pavelsavara left a comment

Choose a reason for hiding this comment

azure-pipelines bot commented Jan 19, 2024

ilonatommy commented Jan 19, 2024

azure-pipelines bot commented Jan 19, 2024

ilonatommy commented Jan 19, 2024

Limit `PNSE` by using Invariant `HashCode` in `HybridGlobalization` #96354

Limit `PNSE` by using Invariant `HashCode` in `HybridGlobalization` #96354

ilonatommy commented Dec 29, 2023 •

edited

Loading