Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte #38653

Merged
merged 19 commits into from
Jul 20, 2020

Conversation

carlossanlop
Copy link
Member

@carlossanlop carlossanlop commented Jul 1, 2020

Contributes to #35035

Adds AdvSimd.Arm64 support for
System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar()
inside the file
runtime\src\libraries\System.Private.CoreLib\src\System\Text\Unicode\Utf8Utility.Validation.cs

The tests for this method live in:
runtime\src\libraries\System.Runtime\tests\System\Text\Unicode\Utf8UtilityTests.ValidateBytes.cs

I've been having difficulties testing this in my ARM device so I want to analyze the CI results.
I manually executed an additional "Libraries Test Run" pipeline to ensure arm64 is run in all platforms.

@ghost
Copy link

ghost commented Jul 1, 2020

Tagging subscribers to this area: @tannergooding
Notify danmosemsft if you want to be subscribed.

Copy link
Member

@GrabYourPitchforks GrabYourPitchforks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not familiar with advsimd yet so I can't comment on the intrinsics usage. But the overall control flow LGTM, modulo some comment cleanup. :)

Assuming AdvSimd.LoadVector128 allows unaligned reads, that advsimd implies little-endian, and MaxAcross(vector).ToScalar() is the arm64 equivalent of pmovmskb, the logic should be sound.

@carlossanlop carlossanlop marked this pull request as ready for review July 1, 2020 23:17
@carlossanlop
Copy link
Member Author

carlossanlop commented Jul 2, 2020

The Linux musl arm64 leg is failing in UTF8 unit tests, which might be related to my changes. I'm investigating.

  Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Utf8String.Experimental.Tests (found 177 of 186 test cases)
  Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 4)
Process terminated. Assertion failed.Process terminated. 
Assertion failed.
Buffer contained ill-formed UTF-8 data.
   at System.Utf8String.UnsafeCreateWithoutValidation(ReadOnlySpan`1 utf8Contents) in /_/src/libraries/System.Private.CoreLib/src/System/Utf8String.Construction.cs:line 510
   at System.Tests.Utf8TestUtilities.u8(String str) in /_/src/libraries/System.Utf8String.Experimental/tests/System/Utf8TestUtilities.cs:line 175
   at System.Tests.BoundedUtf8Span..ctor(ReadOnlySpan`1 utf16Data, PoisonPagePlacement placement) in /_/src/libraries/System.Utf8String.Experimental/tests/System/BoundedUtf8Span.cs:line 26
   at System.Tests.BoundedUtf8Span..ctor(String utf16Data, PoisonPagePlacement placement) in /_/src/libraries/System.Utf8String.Experimental/tests/System/BoundedUtf8Span.cs:line 21
   at System.Text.Tests.Utf8SpanTests.Trim(String input) in /_/src/libraries/System.Utf8String.Experimental/tests/System/Utf8SpanTests.Manipulation.cs:line 224

@kunalspathak
Copy link
Member

@carlossanlop - Did you measure the performance win we get from this?

@@ -15,6 +17,11 @@ namespace System.Text.Unicode
{
internal static unsafe partial class Utf8Utility
{
private static readonly Vector128<byte> s_mostSignficantBitMask = Vector128.Create((byte)0x80);
Copy link
Member

@kunalspathak kunalspathak Jul 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have seen in the past that referring static variable is slower on ARM64 because it has to call helper. Calling helper is done by loading its address which is 3 instructions + a helper call. Since you will be accessing it inside a loop, I would recommend to make these local variables. Here is a quick test:

[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static int AdvSimdMoveMask1(Vector128<byte> value)
{
    Debug.Assert(AdvSimd.Arm64.IsSupported);

    // extractedBits[i] = (value[i] & 0x80) == 0x80 & (1 << i);
    Vector128<byte> mostSignficantBitMask = s_mostSignficantBitMask;
    Vector128<byte> mostSignificantBitIsSet = AdvSimd.CompareEqual(AdvSimd.And(value, mostSignficantBitMask), mostSignficantBitMask);
    Vector128<byte> extractedBits = AdvSimd.And(mostSignificantBitIsSet, s_bitMask128);

    // self-pairwise add until all flags have moved to the first two bytes of the vector
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    return extractedBits.AsInt32().ToScalar();
}
Generated code for AdvSimdMoveMask1
; Assembly listing for method BitArrayTest.TestClass:Main(System.String[]):int
; Emitting BLENDED_CODE for generic ARM64 CPU - Windows
; optimized code
; fp based frame
; partially interruptible
; Final local variable assignments
;
;* V00 arg0         [V00    ] (  0,  0   )     ref  ->  zero-ref    class-hnd
;# V01 OutArgs      [V01    ] (  1,  1   )  lclBlk ( 0) [sp+0x00]   "OutgoingArgSpace"
;  V02 tmp1         [V02,T05] (  3,  3   )  simd16  ->  d16         HFA(simd16)  "Inline stloc first use temp"
;  V03 tmp2         [V03,T04] (  2,  4   )  simd16  ->   d8         HFA(simd16)  "Inlining Arg"
;  V04 tmp3         [V04,T01] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;  V05 tmp4         [V05,T02] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;  V06 tmp5         [V06,T03] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;* V07 cse0         [V07,T00] (  0,  0   )    long  ->  zero-ref    "CSE - aggressive"
;
; Lcl frame size = 0

G_M58909_IG01:
        A9BE7BFD          stp     fp, lr, [sp,#-32]!
        6D0127E8          stp     d8, d9, [sp,#16]
        910003FD          mov     fp, sp
                                                ;; bbWeight=1    PerfScore 2.50
G_M58909_IG02:
        4F00E5E8          movi    v8.16b, #0x0f
        D291BF00          movz    x0, #0x8df8
        F2BE2860          movk    x0, #0xf143 LSL #16
        F2CFFF60          movk    x0, #0x7ffb LSL #32
        52800021          mov     w1, #1
        6E084509          mov     v9.d[0], v8.d[1]
        97FF44AF          bl      CORINFO_HELP_GETSHARED_NONGCSTATIC_BASE
        D2858700          movz    x0, #0x2c38
        F2B8D1C0          movk    x0, #0xc68e LSL #16
        F2C03BC0          movk    x0, #478 LSL #32
        F9400000          ldr     x0, [x0]
        3CC08010          ldr     q16, [x0,#8]
        6E180528          mov     v8.d[1], v9.d[0]
        4E301D11          and     v17.16b, v8.16b, v16.16b
        6E308E30          cmeq    v16.16b, v17.16b, v16.16b
        D2858800          movz    x0, #0x2c40
        F2B8D1C0          movk    x0, #0xc68e LSL #16
        F2C03BC0          movk    x0, #478 LSL #32
        F9400000          ldr     x0, [x0]
        3CC08011          ldr     q17, [x0,#8]
        4E311E10          and     v16.16b, v16.16b, v17.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E042E00          smov    x0, v16.s[0]
                                                ;; bbWeight=1    PerfScore 29.50
G_M58909_IG03:
        6D4127E8          ldp     d8, d9, [sp,#16]
        A8C27BFD          ldp     fp, lr, [sp],#32
        D65F03C0          ret     lr
                                                ;; bbWeight=1    PerfScore 3.00

; Total bytes of code 124, prolog size 12, PerfScore 47.40, (MethodHash=ba8f19e2) for method BitArrayTest.TestClass:Main(System.String[]):int
; ============================================================

vs.

[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static int AdvSimdMoveMask2(Vector128<byte> value, Vector128<byte> mostSignficantBitMask, Vector128<byte> s_bitMask128)
{
    Debug.Assert(AdvSimd.Arm64.IsSupported);

    // extractedBits[i] = (value[i] & 0x80) == 0x80 & (1 << i);
    Vector128<byte> mostSignificantBitIsSet = AdvSimd.CompareEqual(AdvSimd.And(value, mostSignficantBitMask), mostSignficantBitMask);
    Vector128<byte> extractedBits = AdvSimd.And(mostSignificantBitIsSet, s_bitMask128);

    // self-pairwise add until all flags have moved to the first two bytes of the vector
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    return extractedBits.AsInt32().ToScalar();
}
Generated code for AdvSimdMoveMask2
; Assembly listing for method BitArrayTest.TestClass:Main(System.String[]):int
; Emitting BLENDED_CODE for generic ARM64 CPU - Windows
; optimized code
; fp based frame
; partially interruptible
; Final local variable assignments
;
;* V00 arg0         [V00    ] (  0,  0   )     ref  ->  zero-ref    class-hnd
;  V01 loc0         [V01,T04] (  3,  3   )  simd16  ->  d16         HFA(simd16)
;* V02 loc1         [V02    ] (  0,  0   )  simd16  ->  zero-ref    HFA(simd16)
;# V03 OutArgs      [V03    ] (  1,  1   )  lclBlk ( 0) [sp+0x00]   "OutgoingArgSpace"
;  V04 tmp1         [V04,T05] (  2,  2   )  simd16  ->  d17         HFA(simd16)
;  V05 tmp2         [V05,T03] (  2,  4   )  simd16  ->  d18         HFA(simd16)  "Inlining Arg"
;* V06 tmp3         [V06    ] (  0,  0   )  simd16  ->  zero-ref    HFA(simd16)  "Inlining Arg"
;* V07 tmp4         [V07    ] (  0,  0   )  simd16  ->  zero-ref    HFA(simd16)  "Inlining Arg"
;  V08 tmp5         [V08,T00] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;  V09 tmp6         [V09,T01] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;  V10 tmp7         [V10,T02] (  3,  6   )  simd16  ->  d16         HFA(simd16)  "dup spill"
;
; Lcl frame size = 0

G_M58909_IG01:
        A9BF7BFD          stp     fp, lr, [sp,#-16]!
        910003FD          mov     fp, sp
                                                ;; bbWeight=1    PerfScore 1.50
G_M58909_IG02:
        4F04E410          movi    v16.16b, #0x80
        D2804020          movz    x0, #513
        F2A10080          movk    x0, #0x804 LSL #16
        F2C40200          movk    x0, #0x2010 LSL #32
        F2F00800          movk    x0, #0x8040 LSL #48
        4E080C11          dup     v17.2d, x0
        4F00E5F2          movi    v18.16b, #0x0f
        4E301E52          and     v18.16b, v18.16b, v16.16b
        6E308E50          cmeq    v16.16b, v18.16b, v16.16b
        4E311E10          and     v16.16b, v16.16b, v17.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E30BE10          addp    v16.16b, v16.16b, v16.16b
        4E042E00          smov    x0, v16.s[0]
                                                ;; bbWeight=1    PerfScore 14.00
G_M58909_IG03:
        A8C17BFD          ldp     fp, lr, [sp],#16
        D65F03C0          ret     lr
                                                ;; bbWeight=1    PerfScore 2.00

; Total bytes of code 72, prolog size 8, PerfScore 24.70, (MethodHash=ba8f19e2) for method BitArrayTest.TestClass:Main(System.String[]):int
; ============================================================

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the reduced instruction count justify the added complexity of hoisting what amounts to implementation detail to the callsite?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For these hot paths, probably. But it also is something we should likely explicitly track as the JIT should realistically be hoisting this itself.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JIT won't hoist access to static variable because it is not an invariant. Different thread can change its value in middle of loop. See #35279 (comment)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps I'm missing something

This all changes for readonly statics since they can't be safely modified after their corresponding class constructors have completed.

If the JIT doesn't optimize readonly statics, then surely it should?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, readonly static optimize the helper call. I didn't realize that those are readonly static. I tried it on my local machine and looks like s_bitMask128 variable access is getting hoisted. However s_mostSignficantBitMask is not getting hoisted. Not saving it to a local variable Vector128<byte> mostSignficantBitMask = s_mostSignficantBitMask; makes its access hoisted.

[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static ulong AllStatic(Vector128<byte> value)
{
    Debug.Assert(AdvSimd.Arm64.IsSupported);

    Vector128<byte> mostSignificantBitIsSet = AdvSimd.CompareEqual(AdvSimd.And(value, s_mostSignficantBitMask), s_mostSignficantBitMask);
    Vector128<byte> extractedBits = AdvSimd.And(mostSignificantBitIsSet, s_bitMask128);
    // self-pairwise add until all flags have moved to the first two bytes of the vector
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    ulong result = extractedBits.AsUInt64().ToScalar();
    return result;
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static ulong OneStatic(Vector128<byte> value)
{
    Debug.Assert(AdvSimd.Arm64.IsSupported);

    Vector128<byte> mostSignficantBitMask = s_mostSignficantBitMask;
    Vector128<byte> mostSignificantBitIsSet = AdvSimd.CompareEqual(AdvSimd.And(value, s_mostSignficantBitMask), s_mostSignficantBitMask);
    Vector128<byte> extractedBits = AdvSimd.And(mostSignificantBitIsSet, s_bitMask128);
    // self-pairwise add until all flags have moved to the first two bytes of the vector
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    ulong result = extractedBits.AsUInt64().ToScalar();
    return result;
}
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private static ulong NoStatic(Vector128<byte> value, Vector128<byte> mostSignficantBitMask, Vector128<byte> bitMask128)
{
    Debug.Assert(AdvSimd.Arm64.IsSupported);

    Vector128<byte> mostSignificantBitIsSet = AdvSimd.CompareEqual(AdvSimd.And(value, mostSignficantBitMask), mostSignficantBitMask);
    Vector128<byte> extractedBits = AdvSimd.And(mostSignificantBitIsSet, bitMask128);
    // self-pairwise add until all flags have moved to the first two bytes of the vector
    extractedBits = AdvSimd.Arm64.AddPairwise(extractedBits, extractedBits);
    ulong result = extractedBits.AsUInt64().ToScalar();
    return result;
}

NoStatic vs. OneStatic

image

NoStatic vs. AllStatic

image

Summarize:
I need to look deeper on why OneStatic version doesn't CSE the load, but for now, probably don't store s_mostSignficantBitMask in local variable, but just use it directly in AdvSimd.And().

@briansull - FYI

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, thanks for the analysis!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the CSE decision might differ based on other factors in the method, you should compare the JIT code emitted with and without static variables and choose the optimal one. You can dump the JIT code by setting environment variable set COMPlus_JITDisasm=GetPointerToFirstInvalidByte.

@carlossanlop
Copy link
Member Author

@carlossanlop - Did you measure the performance win we get from this?

Not yet but I will.

@carlossanlop
Copy link
Member Author

carlossanlop commented Jul 10, 2020

I found a unit test that is failing in ARM64 without my changes. I reverted everything, built for arm64, ran the unit tests and this unit test still failed:

      System.Text.Unicode.Tests.Utf8UtilityTests.GetIndexOfFirstInvalidUtf8Sequence_WithLargeValidBuffers(
        input: "C3A921222324303132333435363738393A3B3C3D3E3F303132"..., expectedRuneCount: 38, expectedSurrogatePairCount: 0) [FAIL]
        Assert.Equal() Failure
        Expected: -1
        Actual:   12
        Stack Trace:
          /home/calope/runtime/src/libraries/System.Runtime/tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs(391,0): at System.Text.Unicode.Tests.Utf8UtilityTests.GetIndexOfFirstInvalidUtf8Sequence_Test_Core(Byte[] input, Int32 expectedRetVal, Int32 expectedRuneCount, Int32 expectedSurrogatePairCount)
          /home/calope/runtime/src/libraries/System.Runtime/tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs(341,0): at System.Text.Unicode.Tests.Utf8UtilityTests.GetIndexOfFirstInvalidUtf8Sequence_Test_Core(String inputHex, Int32 expectedRetVal, Int32 expectedRuneCount, Int32 expectedSurrogatePairCount)
          /home/calope/runtime/src/libraries/System.Runtime/tests/System/Text/Unicode/Utf8UtilityTests.ValidateBytes.cs(99,0): at System.Text.Unicode.Tests.Utf8UtilityTests.GetIndexOfFirstInvalidUtf8Sequence_WithLargeValidBuffers(String input, Int32 expectedRuneCount, Int32 expectedSurrogatePairCount)

@BruceForstall
Copy link
Member

Is that test run in any innerloop or outerloop arm64 runs in AzDO? (or is it intentionally disabled?)

@carlossanlop
Copy link
Member Author

Tests for last commit passed:

System.Runtime
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Runtime/tests/System.Runtime.Tests.csproj   
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET  
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll  
TestUtilities.Unicode -> /home/calope/runtime/artifacts/bin/TestUtilities.Unicode/net5.0-Debug/TestUtilities.Unicode.dll  
TestCollectibleAssembly -> /home/calope/runtime/artifacts/bin/TestCollectibleAssembly/netstandard2.0-Debug/TestCollectibleAssembly.dll  
TestLoadAssembly -> /home/calope/runtime/artifacts/bin/TestLoadAssembly/netstandard2.0-Debug/TestLoadAssembly.dll  
System.Reflection.TestModule -> /home/calope/runtime/artifacts/bin/System.Reflection.TestModule/netstandard2.0-Debug/System.Reflection.TestModule.dll  
System.TestStructs -> /home/calope/runtime/artifacts/bin/System.TestStructs/netstandard2.0-Debug/System.TestStructs.dll  
System.Runtime.Tests -> /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug/System.Runtime.Tests.dll  
----- start Thu Jul 16 16:28:52 PDT 2020 =============== To repro directly: =====================================================  
pushd /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug  
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing   
popd  
===========================================================================================================  
~/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug ~/runtime/src/libraries/System.Runtime/tests  
    Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)  
    Discovered:  System.Runtime.Tests (found 5478 of 5522 test cases)  
    Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 8)  
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [SKIP]  
        Condition(s) not met: "IsNotArm64Process"  
    System.Tests.DateTimeOffsetTests.ToLocalTime_MaxValue [SKIP]  
        Condition(s) not met: "IsMaxValuePositiveLocalOffset"  
    System.Tests.DateTimeOffsetTests.ToLocalTime_Ambiguous [SKIP]  
        Condition(s) not met: "IsPacificTime"  
    System.Tests.StringTests.IndexOf_SingleLetter(s: "Hello", target: '\0', startIndex: 0, count: 5, expected: -1) [SKIP]  
        Target \0 is not supported in ICU  
    System.Tests.StringTests.LastIndexOf_NullInStrings [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    System.Tests.StringTests.IndexOf_NullInStrings [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    System.Tests.ArgIteratorTests.ArgIterator_GetNextArgType [SKIP]  
        Condition(s) not met: "IsArgIteratorSupported"  
    System.Tests.ArgIteratorTests.ArgIterator_GetRemainingCount_GetNextArg [SKIP]  
        Condition(s) not met: "IsArgIteratorSupported"  
    System.Tests.StringTests.EndsWith_NullInStrings_NonOrdinal [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    System.Tests.StringTests.StartsWith_NullInStrings_NonOrdinal [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    System.Tests.StringTests.EndsWith_NullInStrings [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    System.Tests.StringTests.StartsWith_NullInStrings [SKIP]  
        Condition(s) not met: "IsNlsGlobalization"  
    Finished:    System.Runtime.Tests  
=== TEST EXECUTION SUMMARY ===  
    System.Runtime.Tests  Total: 35603, Errors: 0, Failed: 0, Skipped: 12, Time: 42.207s  
~/runtime/src/libraries/System.Runtime/tests  
----- end Thu Jul 16 16:29:38 PDT 2020 ----- exit code 0 ----------------------------------------------------------  
exit code 0 means Exited Successfully  
System.Text.Encoding
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj   
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET  
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll  
System.Text.Encoding.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug/System.Text.Encoding.Tests.dll  
----- start Thu Jul 16 16:29:52 PDT 2020 =============== To repro directly: =====================================================  
pushd /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug  
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing   
popd  
===========================================================================================================  
~/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug ~/runtime/src/libraries/System.Text.Encoding/tests  
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)  
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)  
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 8)  
    Finished:    System.Text.Encoding.Tests  
=== TEST EXECUTION SUMMARY ===  
    System.Text.Encoding.Tests  Total: 11039, Errors: 0, Failed: 0, Skipped: 0, Time: 31.565s  
~/runtime/src/libraries/System.Text.Encoding/tests  
----- end Thu Jul 16 16:30:26 PDT 2020 ----- exit code 0 ----------------------------------------------------------  
exit code 0 means Exited Successfully  
System.Utf8String.Experimental
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Utf8String.Experimental/tests/System.Utf8String.Experimental.Tests.csproj   
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET  
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll  
System.Utf8String.Experimental.Tests -> /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug/System.Utf8String.Experimental.Tests.dll  
----- start Thu Jul 16 16:30:40 PDT 2020 =============== To repro directly: =====================================================  
pushd /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug  
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Utf8String.Experimental.Tests.runtimeconfig.json --depsfile System.Utf8String.Experimental.Tests.deps.json xunit.console.dll System.Utf8String.Experimental.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing   
popd  
===========================================================================================================  
~/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug ~/runtime/src/libraries/System.Utf8String.Experimental/tests  
    Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)  
    Discovered:  System.Utf8String.Experimental.Tests (found 179 of 188 test cases)  
    Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 8)  
    Finished:    System.Utf8String.Experimental.Tests  
=== TEST EXECUTION SUMMARY ===  
    System.Utf8String.Experimental.Tests  Total: 380, Errors: 0, Failed: 0, Skipped: 0, Time: 0.674s  
~/runtime/src/libraries/System.Utf8String.Experimental/tests  
----- end Thu Jul 16 16:30:42 PDT 2020 ----- exit code 0 ----------------------------------------------------------  
exit code 0 means Exited Successfully  

@carlossanlop
Copy link
Member Author

carlossanlop commented Jul 17, 2020

The tests passed locally for the latest commit:

System.Runtime
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:Configuration=Release src/libraries/System.Runtime/tests/System.Runtime.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities.Unicode -> /home/calope/runtime/artifacts/bin/TestUtilities.Unicode/net5.0-Release/TestUtilities.Unicode.dll
TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
TestLoadAssembly -> /home/calope/runtime/artifacts/bin/TestLoadAssembly/netstandard2.0-Release/TestLoadAssembly.dll
System.Reflection.TestModule -> /home/calope/runtime/artifacts/bin/System.Reflection.TestModule/netstandard2.0-Release/System.Reflection.TestModule.dll
System.TestStructs -> /home/calope/runtime/artifacts/bin/System.TestStructs/netstandard2.0-Release/System.TestStructs.dll
TestCollectibleAssembly -> /home/calope/runtime/artifacts/bin/TestCollectibleAssembly/netstandard2.0-Release/TestCollectibleAssembly.dll
System.Runtime.Tests -> /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release/System.Runtime.Tests.dll
----- start Thu Jul 16 18:28:30 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release ~/runtime/src/libraries/System.Runtime/tests
    Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Runtime.Tests (found 5478 of 5522 test cases)
    Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 8)
    System.Tests.ArgIteratorTests.ArgIterator_GetNextArgType [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
    System.Tests.ArgIteratorTests.ArgIterator_GetRemainingCount_GetNextArg [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
    System.Tests.DateTimeOffsetTests.ToLocalTime_MaxValue [SKIP]
        Condition(s) not met: "IsMaxValuePositiveLocalOffset"
    System.Tests.StringTests.IndexOf_SingleLetter(s: "Hello", target: '\0', startIndex: 0, count: 5, expected: -1) [SKIP]
        Target \0 is not supported in ICU
    System.Tests.StringTests.LastIndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.IndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.DateTimeOffsetTests.ToLocalTime_Ambiguous [SKIP]
        Condition(s) not met: "IsPacificTime"
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [SKIP]
        Condition(s) not met: "IsNotArm64Process"
    System.Tests.StringTests.EndsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.StartsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.EndsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.StartsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    Finished:    System.Runtime.Tests
=== TEST EXECUTION SUMMARY ===
    System.Runtime.Tests  Total: 35577, Errors: 0, Failed: 0, Skipped: 12, Time: 14.086s
~/runtime/src/libraries/System.Runtime/tests
----- end Thu Jul 16 18:28:46 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully
System.Text.Encoding
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:Configuration=Release src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
System.Text.Encoding.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release/System.Text.Encoding.Tests.dll
----- start Thu Jul 16 18:30:52 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release ~/runtime/src/libraries/System.Text.Encoding/tests
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Text.Encoding.Tests
=== TEST EXECUTION SUMMARY ===
    System.Text.Encoding.Tests  Total: 11039, Errors: 0, Failed: 0, Skipped: 0, Time: 10.701s
~/runtime/src/libraries/System.Text.Encoding/tests
----- end Thu Jul 16 18:31:04 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully
System.Utf8String.Experimental
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:Configuration=Release src/libraries/System.Utf8String.Experimental/tests/System.Utf8String.Experimental.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
System.Utf8String.Experimental.Tests -> /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release/System.Utf8String.Experimental.Tests.dll
----- start Thu Jul 16 18:32:56 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Utf8String.Experimental.Tests.runtimeconfig.json --depsfile System.Utf8String.Experimental.Tests.deps.json xunit.console.dll System.Utf8String.Experimental.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release ~/runtime/src/libraries/System.Utf8String.Experimental/tests
    Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Utf8String.Experimental.Tests (found 179 of 188 test cases)
    Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Utf8String.Experimental.Tests
=== TEST EXECUTION SUMMARY ===
    System.Utf8String.Experimental.Tests  Total: 380, Errors: 0, Failed: 0, Skipped: 0, Time: 0.287s
~/runtime/src/libraries/System.Utf8String.Experimental/tests
----- end Thu Jul 16 18:32:57 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully

@carlossanlop
Copy link
Member Author

The Windows_NT net472 x86 and x64 Release build legs hit this:

      ---- System.PlatformNotSupportedException : Operation is not supported on this platform.
      Stack Trace:
        /_/src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.cs(133,0): at System.Text.Unicode.Utf8Utility.ValidateAndFixupUtf8String(Utf8String value)
        /_/src/libraries/System.Private.CoreLib/src/System/Utf8String.Construction.cs(125,0): at System.Utf8String.CreateFromRelaxed(ReadOnlySpan`1 buffer)
        /_/src/libraries/System.Utf8String.Experimental/tests/System/Utf8StringTests.Ctor.cs(275,0): at System.Tests.Utf8StringTests.Ctor_CreateFromRelaxed_Utf8()
        ----- Inner Stack Trace -----
        /_/src/libraries/System.Utf8String.Experimental/src/System/Runtime/Intrinsics/Intrinsics.Shims.cs(22,0): at System.Runtime.Intrinsics.Vector128.Create(UInt16 value)
        /_/src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Validation.cs(734,0): at System.Text.Unicode.Utf8Utility..cctor()

That job was executed in net472-Windows_NT-Release-x86-Windows.10.Amd64.Client19H1.Open. So why is it hitting that code?

@kunalspathak
Copy link
Member

That job was executed in net472-Windows_NT-Release-x86-Windows.10.Amd64.Client19H1.Open. So why is it hitting that code?

It seems to be hitting the shim method here that throws PlatformNotSupportedException. Is there some switch in some project file that you need to flip to make sure it doesn't pick the shimmed version?

@carlossanlop
Copy link
Member Author

We found the root cause for the net472 issue: I declared a static field in a class that is consumed by Utf8String.Experimental, which is a .NET Standard project, but the class belongs to System.Private.Corelib and because of that, anything that gets initialized in advance (due to being static) will end up calling its PNSE Shim. That's why that class had no other static fields, only methods. So what I did was instead of using a static field, I'm passing that value as an argument, like it was suggested before. cc @kunalspathak

@kunalspathak
Copy link
Member

We found the root cause for the net472 issue: I declared a static field in a class that is consumed by Utf8String.Experimental, which is a .NET Standard project, but the class belongs to System.Private.Corelib and because of that, anything that gets initialized in advance (due to being static) will end up calling its PNSE Shim. That's why that class had no other static fields, only methods. So what I did was instead of using a static field, I'm passing that value as an argument, like it was suggested before. cc @kunalspathak

The changes looks good to me.

@carlossanlop
Copy link
Member Author

We had a conversation offline and we decided we will collect the perf data after merging. @kunalspathak @jeffhandley

@carlossanlop
Copy link
Member Author

I posted the perf results in this comment: #39050 (comment)
It includes a comparison of SSE2 turned off vs on.
It shows the benchmarks run against each PR, and also the benchmarks run against all 3 PRs merged together.

@carlossanlop
Copy link
Member Author

@kunalspathak @echesakovMSFT I am going to address @GrabYourPitchforks 's last comments, but wanted to let you know that before I merge that commit, the CI failures seem unrelated to my changes, although they are consistent and pre-existing (i have been re-running the failed CI legs and they keep showing up).

Here are the two failures summarized, which I suspect will show up again after I merge the new commit:

Build libraries Windows_NT AllConfigurations x64 release

Results: https://dev.azure.com/dnceng/public/_build/results?buildId=736427&view=logs&j=e87d290e-7c43-5717-098e-deb8643034d5&t=699159aa-4b47-567c-6f4b-ee557432ed14

F:\workspace\_work\1\s\src\libraries\pkg\test\testPackages.proj(217,5): error : Validation Failed: Microsoft.Extensions.DependencyModel is harvesting assets from package version 3.1.5 which is not the latest for that package release. Latest package version from that release is 3.1.6. In order to fix this, run `dotnet msbuild Microsoft.Extensions.DependencyModel.pkgproj /t:UpdateHarvestVersionOnPackageIndex /p:UpdateStablePackageInfo=true`
##[error]src\libraries\pkg\test\testPackages.proj(217,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Validation Failed: Microsoft.Extensions.DependencyModel is harvesting assets from package version 3.1.5 which is not the latest for that package release. Latest package version from that release is 3.1.6. In order to fix this, run `dotnet msbuild Microsoft.Extensions.DependencyModel.pkgproj /t:UpdateHarvestVersionOnPackageIndex /p:UpdateStablePackageInfo=true`
Build wasm mono

Results: https://dev.azure.com/dnceng/public/_build/results?buildId=736427&view=logs&j=2c1bc6b5-3ed2-590f-6f2c-e4b87c1ec8e3&t=8dad6837-707b-544a-4fb6-dd541d5adf36&l=58

[FAIL] System.Linq.Parallel.Tests.ToLookupTests.ToLookup_AggregateException(labeled: ParallelEnumerable.Range, count: 1)
Assert.All() Failure: 2 out of 2 items in the collection did not pass.
[1]: Item: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler.
      ---> System.NullReferenceException: Object reference not set to an instance of an object.
        at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
        at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length)
        at System.Exception.CaptureDispatchState()
        at System.Runtime.ExceptionServices.ExceptionDispatchInfo..ctor(Exception exception)
        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(Exception source)
        at System.Threading.Tasks.TaskExceptionHolder.AddFaultException(Object exceptionObject)
        at System.Threading.Tasks.TaskExceptionHolder.Add(Object exceptionObject, Boolean representsCancellation)
        at System.Threading.Tasks.Task.AddException(Object exceptionObject, Boolean representsCancellation)
        at System.Threading.Tasks.Task.AddException(Object exceptionObject)
        at System.Threading.Tasks.Task.HandleException(Exception unhandledException)
        at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
        at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
        at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(Task task, Boolean taskWasPreviouslyQueued)
        at System.Threading.Tasks.TaskScheduler.TryRunInline(Task task, Boolean taskWasPreviouslyQueued)
        at System.Threading.Tasks.Task.InternalRunSynchronously(TaskScheduler scheduler, Boolean waitForCompletion)
        --- End of inner exception stack trace ---
        at System.Threading.Tasks.Task.InternalRunSynchronously(TaskScheduler scheduler, Boolean waitForCompletion)
        at System.Threading.Tasks.Task.RunSynchronously(TaskScheduler scheduler)
        at System.Linq.Parallel.QueryTask.RunSynchronously(TaskScheduler taskScheduler)
        at System.Linq.Parallel.SpoolingTask.<>c__DisplayClass0_0`2[[System.Linq.IGrouping`2[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Linq, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<SpoolStopAndGo>b__0()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    --- End of stack trace from previous location ---
        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
        at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
    Xunit.Sdk.IsTypeException: Assert.IsType() Failure
    Expected: System.Linq.Parallel.Tests.DeliberateTestException
    Actual:   System.Threading.Tasks.TaskSchedulerException
        at Xunit.Assert.IsType(Type expectedType, Object object)
        at Xunit.Assert.IsType[DeliberateTestException](Object object)
        at System.Linq.Parallel.Tests.AssertThrows.<>c__5`1[[System.Linq.Parallel.Tests.DeliberateTestException, System.Linq.Parallel.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].<Wrapped>b__5_0(Exception e)
        at Xunit.Assert.All[Exception](IEnumerable`1 collection, Action`1 action)
[0]: Item: System.Threading.Tasks.TaskSchedulerException: An exception was thrown by a TaskScheduler.
      ---> System.NullReferenceException: Object reference not set to an instance of an object.
        at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable)
        at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length)
        at System.Exception.CaptureDispatchState()
        at System.Runtime.ExceptionServices.ExceptionDispatchInfo..ctor(Exception exception)
        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Capture(Exception source)
        at System.Threading.Tasks.TaskExceptionHolder.AddFaultException(Object exceptionObject)
        at System.Threading.Tasks.TaskExceptionHolder.Add(Object exceptionObject, Boolean representsCancellation)
        at System.Threading.Tasks.Task.AddException(Object exceptionObject, Boolean representsCancellation)
        at System.Threading.Tasks.Task.AddException(Object exceptionObject)
        at System.Threading.Tasks.Task.HandleException(Exception unhandledException)
        at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
        at System.Threading.Tasks.Task.ExecuteEntryUnsafe(Thread threadPoolThread)
        at System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(Task task, Boolean taskWasPreviouslyQueued)
        at System.Threading.Tasks.TaskScheduler.TryRunInline(Task task, Boolean taskWasPreviouslyQueued)
        at System.Threading.Tasks.Task.InternalRunSynchronously(TaskScheduler scheduler, Boolean waitForCompletion)
        --- End of inner exception stack trace ---
        at System.Threading.Tasks.Task.InternalRunSynchronously(TaskScheduler scheduler, Boolean waitForCompletion)
        at System.Threading.Tasks.Task.RunSynchronously(TaskScheduler scheduler)
        at System.Linq.Parallel.QueryTask.RunSynchronously(TaskScheduler taskScheduler)
        at System.Linq.Parallel.SpoolingTask.<>c__DisplayClass0_0`2[[System.Linq.IGrouping`2[[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Linq, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.Int32, System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<SpoolStopAndGo>b__0()
        at System.Threading.Tasks.Task.InnerInvoke()
        at System.Threading.Tasks.Task.<>c.<.cctor>b__277_0(Object obj)
        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
    --- End of stack trace from previous location ---
        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
        at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
    Xunit.Sdk.IsTypeException: Assert.IsType() Failure
    Expected: System.Linq.Parallel.Tests.DeliberateTestException
    Actual:   System.Threading.Tasks.TaskSchedulerException
        at Xunit.Assert.IsType(Type expectedType, Object object)
        at Xunit.Assert.IsType[DeliberateTestException](Object object)
        at System.Linq.Parallel.Tests.AssertThrows.<>c__5`1[[System.Linq.Parallel.Tests.DeliberateTestException, System.Linq.Parallel.Tests, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].<Wrapped>b__5_0(Exception e)
        at Xunit.Assert.All[Exception](IEnumerable`1 collection, Action`1 action)
  at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

@carlossanlop
Copy link
Member Author

Unit tests passed locally for the latest commit. And because the change affects the Sse2 code too, I ran the unit tests in my x64 PC as well:

ARM64

System.Text.Json
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:TargetArchitecture=arm64 /p:Configuration=Release src/libraries/System.Text.Json/tests/System.Text.Json.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  System.Text.Json.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release/System.Text.Json.Tests.dll
  ----- start Mon Jul 20 12:08:29 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release ~/runtime/src/libraries/System.Text.Json/tests
    Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Json.Tests (found 1957 of 1991 test cases)
    Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Text.Json.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Json.Tests  Total: 8511, Errors: 0, Failed: 0, Skipped: 0, Time: 68.192s
  ~/runtime/src/libraries/System.Text.Json/tests
  ----- end Mon Jul 20 12:09:39 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully
System.Text.Encoding
calope@calopearm:~/runtime$ calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:TargetArchitecture=arm64 /p:Configuration=Release src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  System.Text.Encoding.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release/System.Text.Encoding.Tests.dll
  ----- start Mon Jul 20 12:11:10 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Release ~/runtime/src/libraries/System.Text.Encoding/tests
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Text.Encoding.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Encoding.Tests  Total: 11039, Errors: 0, Failed: 0, Skipped: 0, Time: 15.819s
  ~/runtime/src/libraries/System.Text.Encoding/tests
  ----- end Mon Jul 20 12:11:27 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully
System.Runtime
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:TargetArchitecture=arm64 /p:Configuration=Release src/libraries/System.Runtime/tests/System.Runtime.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  TestUtilities.Unicode -> /home/calope/runtime/artifacts/bin/TestUtilities.Unicode/net5.0-Release/TestUtilities.Unicode.dll
  TestCollectibleAssembly -> /home/calope/runtime/artifacts/bin/TestCollectibleAssembly/netstandard2.0-Release/TestCollectibleAssembly.dll
  TestLoadAssembly -> /home/calope/runtime/artifacts/bin/TestLoadAssembly/netstandard2.0-Release/TestLoadAssembly.dll
  System.TestStructs -> /home/calope/runtime/artifacts/bin/System.TestStructs/netstandard2.0-Release/System.TestStructs.dll
  System.Reflection.TestModule -> /home/calope/runtime/artifacts/bin/System.Reflection.TestModule/netstandard2.0-Release/System.Reflection.TestModule.dll
  System.Runtime.Tests -> /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release/System.Runtime.Tests.dll
  ----- start Mon Jul 20 12:11:48 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Release ~/runtime/src/libraries/System.Runtime/tests
    Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Runtime.Tests (found 5478 of 5522 test cases)
    Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 8)
      System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [SKIP]
        Condition(s) not met: "IsNotArm64Process"
      System.Tests.DateTimeOffsetTests.ToLocalTime_MaxValue [SKIP]
        Condition(s) not met: "IsMaxValuePositiveLocalOffset"
      System.Tests.ArgIteratorTests.ArgIterator_GetNextArgType [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
      System.Tests.ArgIteratorTests.ArgIterator_GetRemainingCount_GetNextArg [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
      System.Tests.DateTimeOffsetTests.ToLocalTime_Ambiguous [SKIP]
        Condition(s) not met: "IsPacificTime"
      System.Tests.StringTests.IndexOf_SingleLetter(s: "Hello", target: '\0', startIndex: 0, count: 5, expected: -1) [SKIP]
        Target \0 is not supported in ICU
      System.Tests.StringTests.LastIndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.IndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.EndsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.StartsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.EndsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.StartsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    Finished:    System.Runtime.Tests
  === TEST EXECUTION SUMMARY ===
    System.Runtime.Tests  Total: 35577, Errors: 0, Failed: 0, Skipped: 12, Time: 23.898s
  ~/runtime/src/libraries/System.Runtime/tests
  ----- end Mon Jul 20 12:12:15 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully
System.Utf8String.Experimental
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:TargetArchitecture=arm64 /p:Configuration=Release src/libraries/System.Utf8String.Experimental/tests/System.Utf8String.Experimental.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  System.Utf8String.Experimental.Tests -> /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release/System.Utf8String.Experimental.Tests.dll
  ----- start Mon Jul 20 12:12:32 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Utf8String.Experimental.Tests.runtimeconfig.json --depsfile System.Utf8String.Experimental.Tests.deps.json xunit.console.dll System.Utf8String.Experimental.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Release ~/runtime/src/libraries/System.Utf8String.Experimental/tests
    Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Utf8String.Experimental.Tests (found 179 of 188 test cases)
    Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Utf8String.Experimental.Tests
  === TEST EXECUTION SUMMARY ===
    System.Utf8String.Experimental.Tests  Total: 380, Errors: 0, Failed: 0, Skipped: 0, Time: 0.324s
  ~/runtime/src/libraries/System.Utf8String.Experimental/tests
  ----- end Mon Jul 20 12:12:33 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully
System.Formats.Asn1
calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:TargetArchitecture=arm64 /p:Configuration=Release src/libraries/System.Formats.Asn1/tests/System.Formats.Asn1.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  System.Formats.Asn1.Tests -> /home/calope/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release/System.Formats.Asn1.Tests.dll
  ----- start Mon Jul 20 12:19:07 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Formats.Asn1.Tests.runtimeconfig.json --depsfile System.Formats.Asn1.Tests.deps.json xunit.console.dll System.Formats.Asn1.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release ~/runtime/src/libraries/System.Formats.Asn1/tests
    Discovering: System.Formats.Asn1.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Asn1.Tests (found 527 test cases)
    Starting:    System.Formats.Asn1.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Formats.Asn1.Tests
  === TEST EXECUTION SUMMARY ===
    System.Formats.Asn1.Tests  Total: 3418, Errors: 0, Failed: 0, Skipped: 0, Time: 1.031s
  ~/runtime/src/libraries/System.Formats.Asn1/tests
  ----- end Mon Jul 20 12:19:09 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully

X64

System.Runtime
calope@CALOPEPC  D:\runtime   ARM-Utf8Utility.Validation ≣                                                 [12:04]
❯ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=x64 .\src\libraries\System.Runtime\tests\System.Runtime.Tests.csproj
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities.Unicode -> D:\runtime\artifacts\bin\TestUtilities.Unicode\net5.0-Release\TestUtilities.Unicode.dll
  TestUtilities -> D:\runtime\artifacts\bin\TestUtilities\net5.0-Release\TestUtilities.dll
  System.Reflection.TestModule -> D:\runtime\artifacts\bin\System.Reflection.TestModule\netstandard2.0-Release\System.Reflection.TestModule.dll
  System.TestStructs -> D:\runtime\artifacts\bin\System.TestStructs\netstandard2.0-Release\System.TestStructs.dll
  TestCollectibleAssembly -> D:\runtime\artifacts\bin\TestCollectibleAssembly\netstandard2.0-Release\TestCollectibleAssembly.dll
  TestLoadAssembly -> D:\runtime\artifacts\bin\TestLoadAssembly\netstandard2.0-Release\TestLoadAssembly.dll
  System.Runtime.Tests -> D:\runtime\artifacts\bin\System.Runtime.Tests\net5.0-Windows_NT-Release\System.Runtime.Tests.dll
  ----- start 2020-07-20 12:16:22.79 ===============  To repro directly: =====================================================
  pushd D:\runtime\artifacts\bin\System.Runtime.Tests\net5.0-Windows_NT-Release\
  "D:\runtime\artifacts\bin\testhost\net5.0-Windows_NT-Release-x64\dotnet.exe" exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Runtime.Tests (found 5479 of 5521 test cases)
    Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 12)
      System.Tests.DateTimeOffsetTests.ToLocalTime_MaxValue [SKIP]
        Condition(s) not met: "IsMaxValuePositiveLocalOffset"
      System.Text.Tests.RuneTests.Casing_Invariant [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.RuneTests.Casing_CultureAware [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.ArgIteratorTests.ArgIterator_Throws_PlatformNotSupportedException [SKIP]
        Condition(s) not met: "IsArgIteratorNotSupported"
      System.Tests.StringTests.IndexOf_SingleLetter(s: "Hello", target: '\0', startIndex: 0, count: 5, expected: -1) [SKIP]
        Target \0 is not supported in ICU
      System.Tests.StringTests.LastIndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.IndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.EndsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.StartsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.EndsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.StringTests.StartsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    Finished:    System.Runtime.Tests
  === TEST EXECUTION SUMMARY ===
    System.Runtime.Tests  Total: 35421, Errors: 0, Failed: 0, Skipped: 11, Time: 5.980s
  ----- end 2020-07-20 12:16:34.09 ----- exit code 0 ----------------------------------------------------------
System.Text.Encoding
calope@CALOPEPC  D:\runtime   ARM-Utf8Utility.Validation ≣                                                                                                                                   [12:16]
❯ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=x64 .\src\libraries\System.Text.Encoding\tests\System.Text.Encoding.Tests.csproj
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> D:\runtime\artifacts\bin\TestUtilities\net5.0-Release\TestUtilities.dll
  System.Text.Encoding.Tests -> D:\runtime\artifacts\bin\System.Text.Encoding.Tests\net5.0-Release\System.Text.Encoding.Tests.dll
  ----- start 2020-07-20 12:17:59.82 ===============  To repro directly: =====================================================
  pushd D:\runtime\artifacts\bin\System.Text.Encoding.Tests\net5.0-Release\
  "D:\runtime\artifacts\bin\testhost\net5.0-Windows_NT-Release-x64\dotnet.exe" exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 12)
    Finished:    System.Text.Encoding.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Encoding.Tests  Total: 11039, Errors: 0, Failed: 0, Skipped: 0, Time: 4.667s
  ----- end 2020-07-20 12:18:15.96 ----- exit code 0 ----------------------------------------------------------
System.Text.Json
calope@CALOPEPC  D:\runtime   ARM-Utf8Utility.Validation ≣                                                                                                                                   [12:18]
❯ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=x64 .\src\libraries\System.Text.Json\tests\System.Text.Json.Tests.csproj
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> D:\runtime\artifacts\bin\TestUtilities\net5.0-Release\TestUtilities.dll
  System.Text.Json.Tests -> D:\runtime\artifacts\bin\System.Text.Json.Tests\net5.0-Release\System.Text.Json.Tests.dll
  ----- start 2020-07-20 12:18:30.53 ===============  To repro directly: =====================================================
  pushd D:\runtime\artifacts\bin\System.Text.Json.Tests\net5.0-Release\
  "D:\runtime\artifacts\bin\testhost\net5.0-Windows_NT-Release-x64\dotnet.exe" exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Json.Tests (found 1957 of 1991 test cases)
    Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 12)
    Finished:    System.Text.Json.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Json.Tests  Total: 8511, Errors: 0, Failed: 0, Skipped: 0, Time: 10.408s
  ----- end 2020-07-20 12:18:42.85 ----- exit code 0 ----------------------------------------------------------
System.Formats.Asn1
calope@CALOPEPC  D:\runtime   ARM-Utf8Utility.Validation ≣                                                                                                                                   [12:18]
❯ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=x64 .\src\libraries\System.Formats.Asn1\tests\System.Formats.Asn1.Tests.csproj
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> D:\runtime\artifacts\bin\TestUtilities\net5.0-Release\TestUtilities.dll
  System.Formats.Asn1.Tests -> D:\runtime\artifacts\bin\System.Formats.Asn1.Tests\net5.0-Release\System.Formats.Asn1.Tests.dll
  ----- start 2020-07-20 12:18:51.60 ===============  To repro directly: =====================================================
  pushd D:\runtime\artifacts\bin\System.Formats.Asn1.Tests\net5.0-Release\
  "D:\runtime\artifacts\bin\testhost\net5.0-Windows_NT-Release-x64\dotnet.exe" exec --runtimeconfig System.Formats.Asn1.Tests.runtimeconfig.json --depsfile System.Formats.Asn1.Tests.deps.json xunit.console.dll System.Formats.Asn1.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Asn1.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Asn1.Tests (found 527 test cases)
    Starting:    System.Formats.Asn1.Tests (parallel test collections = on, max threads = 12)
    Finished:    System.Formats.Asn1.Tests
  === TEST EXECUTION SUMMARY ===
    System.Formats.Asn1.Tests  Total: 3418, Errors: 0, Failed: 0, Skipped: 0, Time: 0.335s
  ----- end 2020-07-20 12:18:53.33 ----- exit code 0 ----------------------------------------------------------
System.Utf8String.Experimental
calope@CALOPEPC  D:\runtime   ARM-Utf8Utility.Validation ≣                                                                                                                                   [12:18]
❯ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=x64 .\src\libraries\System.Utf8String.Experimental\tests\System.Utf8String.Experimental.Tests.csproj
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> D:\runtime\artifacts\bin\TestUtilities\net5.0-Release\TestUtilities.dll
  System.Utf8String.Experimental.Tests -> D:\runtime\artifacts\bin\System.Utf8String.Experimental.Tests\net5.0-Release\System.Utf8String.Experimental.Tests.dll
  ----- start 2020-07-20 12:19:18.16 ===============  To repro directly: =====================================================
  pushd D:\runtime\artifacts\bin\System.Utf8String.Experimental.Tests\net5.0-Release\
  "D:\runtime\artifacts\bin\testhost\net5.0-Windows_NT-Release-x64\dotnet.exe" exec --runtimeconfig System.Utf8String.Experimental.Tests.runtimeconfig.json --depsfile System.Utf8String.Experimental.Tests.deps.json xunit.console.dll System.Utf8String.Experimental.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Utf8String.Experimental.Tests (found 188 test cases)
    Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 12)
      System.Tests.Utf8StringTests.GetHashCode_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.Utf8StringTests.TryFind_Rune_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.Utf8SpanTests.GetHashCode_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.Utf8StringTests.TryFind_Char_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.Utf8SpanTests.TryFind_Char_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Tests.Utf8StringTests.TryFind_Utf8String_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.Utf8SpanTests.TryFind_Utf8Span_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.Utf8SpanTests.Equals_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
      System.Text.Tests.Utf8SpanTests.TryFind_Rune_WithComparison [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    Finished:    System.Utf8String.Experimental.Tests
  === TEST EXECUTION SUMMARY ===
    System.Utf8String.Experimental.Tests  Total: 389, Errors: 0, Failed: 0, Skipped: 9, Time: 0.165s
  ----- end 2020-07-20 12:19:19.80 ----- exit code 0 ----------------------------------------------------------

// bump our input counter by that amount, and resume processing from the
// "the first byte is no longer ASCII" portion of the main loop.
// We should not expect a total number of zeroes equal or larger than 16.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not expect a total number of zeroes equal or larger than 16. [](start = 27, length = 70)

minor: I would re-phrase the comment to "Make sure that pInputBuffer is not advanced by more than 15 positions."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that's ok with you, I would like to address this in another PR so that I don't reset the CI (it's taking a really long time to finish).
I have another similar TODO from the other PR (modifying a comment). We can collect similar requests and address them separately.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. No problem.

@kunalspathak
Copy link
Member

Changes LGTM

@carlossanlop
Copy link
Member Author

This is ready to merge.

One repeated unrelated failure:

Libraries Build Windows_NT allConfigurations x64 Release:

Results: https://dev.azure.com/dnceng/public/_build/results?buildId=738082&view=logs&j=b39d9e17-137d-57db-fe6a-b5d7d766b2bb&t=b9e617d9-bc19-56ff-a2c0-d3a10041f0bc

src\libraries\pkg\test\testPackages.proj(217,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Validation Failed: Microsoft.Extensions.DependencyModel is harvesting assets from package version 3.1.5 which is not the latest for that package release. Latest package version from that release is 3.1.6. In order to fix this, run `dotnet msbuild Microsoft.Extensions.DependencyModel.pkgproj /t:UpdateHarvestVersionOnPackageIndex /p:UpdateStablePackageInfo=true`

The WASM leg is stuck with the yellow dot next to it (says it's still running). There seems to be a delay between the time a CI leg finishes and the time it gets reported in the PR CI results (I asked @ViktorHofer and he confirmed this is a known issue)

The actual WASM leg already finished and it was successful: https://dev.azure.com/dnceng/public/_build/results?buildId=738082&view=logs&j=108d2c4a-8a62-5a58-8dad-8e1042acc93c&t=4a4d9b63-088e-49f3-b7e4-699b664f7a06

@carlossanlop carlossanlop merged commit 9d88a94 into dotnet:master Jul 20, 2020
tannergooding pushed a commit to tannergooding/runtime that referenced this pull request Jul 21, 2020
…InvalidByte (dotnet#38653)

* AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte

* Move comment to the top, add shims.

* Little endian checks

* Use custom MoveMask method for AdvSimd

* Address suggestions to improve the AdvSimdMoveMask method

* Define initialMask outside MoveMask method

* UInt64 in Arm64MoveMask

* Add unit test case to verify intrinsics improvement

* Avoid casting to smaller integer type

* Typo and comment

* Use ShiftRightArithmetic instead of CompareEqual + And.
Remove test case causing other unit tests to fail.

* Use AddPairwise version of GetNotAsciiBytes

* Add missing shims causing Linux build to fail

* Simplify GetNonAsciiBytes to only one AddPairwise call, shorter bitmask

* Respect data type returned by masking method

* Address suggestions - assert trailingzerocount and bring back uint mask

* Trailing zeroes in AdvSimd need to be divided by 4, and total number should not be larger than 16

* Avoid declaring static field which causes PNSE in Utf8String.Experimental (S.P.Corelib code is used for being NetStandard)

* Prefer using nuint for BitConverter.TrailingZeroCount
tannergooding added a commit that referenced this pull request Jul 22, 2020
…#39738)

* AdvSimd support for System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar (#39050)

* AdvSimd support for System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar

* Move using directive outside #if.
Improve Arm64MoveMask.

* Change overloads

* UIn64 in Arm64MoveMask

* Build error implicit conversion fix

* Rename method and use simpler version

* Use ShiftRightArithmetic instead of CompareEqual + And.

* Remove unnecessary comment

* Add missing shims causing Linux build to fail

* AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 (#39041)

* AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8

* Readd using to prevent build failure.
Add AdvSimd equivalent operation to TestZ.

* Inverted condition

* Address IsSupported order, improve use ExtractNarrowingSaturated usage

* Rename source to result, second argument utf16Data

* Improve CompareTest

* Add shims causing failures in Linux

* Use unsigned version of ExtractNarrowingSaturate, avoid using MinAcross and use MaxPairwise instead

* Missing support check for Sse2.X64

* Add missing case for AdvSimd

* Use MinPairwise for short

* AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte (#38653)

* AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte

* Move comment to the top, add shims.

* Little endian checks

* Use custom MoveMask method for AdvSimd

* Address suggestions to improve the AdvSimdMoveMask method

* Define initialMask outside MoveMask method

* UInt64 in Arm64MoveMask

* Add unit test case to verify intrinsics improvement

* Avoid casting to smaller integer type

* Typo and comment

* Use ShiftRightArithmetic instead of CompareEqual + And.
Remove test case causing other unit tests to fail.

* Use AddPairwise version of GetNotAsciiBytes

* Add missing shims causing Linux build to fail

* Simplify GetNonAsciiBytes to only one AddPairwise call, shorter bitmask

* Respect data type returned by masking method

* Address suggestions - assert trailingzerocount and bring back uint mask

* Trailing zeroes in AdvSimd need to be divided by 4, and total number should not be larger than 16

* Avoid declaring static field which causes PNSE in Utf8String.Experimental (S.P.Corelib code is used for being NetStandard)

* Prefer using nuint for BitConverter.TrailingZeroCount

* Fix build failure in net472 debug AdvSimd Utf16Utility (#39652)

Co-authored-by: Carlos Sanchez Lopez <[email protected]>
@carlossanlop carlossanlop deleted the ARM-Utf8Utility.Validation branch July 29, 2020 16:35
Jacksondr5 pushed a commit to Jacksondr5/runtime that referenced this pull request Aug 10, 2020
…InvalidByte (dotnet#38653)

* AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte

* Move comment to the top, add shims.

* Little endian checks

* Use custom MoveMask method for AdvSimd

* Address suggestions to improve the AdvSimdMoveMask method

* Define initialMask outside MoveMask method

* UInt64 in Arm64MoveMask

* Add unit test case to verify intrinsics improvement

* Avoid casting to smaller integer type

* Typo and comment

* Use ShiftRightArithmetic instead of CompareEqual + And.
Remove test case causing other unit tests to fail.

* Use AddPairwise version of GetNotAsciiBytes

* Add missing shims causing Linux build to fail

* Simplify GetNonAsciiBytes to only one AddPairwise call, shorter bitmask

* Respect data type returned by masking method

* Address suggestions - assert trailingzerocount and bring back uint mask

* Trailing zeroes in AdvSimd need to be divided by 4, and total number should not be larger than 16

* Avoid declaring static field which causes PNSE in Utf8String.Experimental (S.P.Corelib code is used for being NetStandard)

* Prefer using nuint for BitConverter.TrailingZeroCount
@ghost ghost locked as resolved and limited conversation to collaborators Dec 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-arm64 area-System.Runtime.Intrinsics utf8-impact Potentially impacts UTF-8 support in the runtime
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants