Bad codegen when EVEX embedded broadcast is combined with embedded masking on some bitwise instructions

[saucecontrol]

Example:

using System;
using System.Runtime.Intrinsics;
using System.Runtime.CompilerServices;

[MethodImpl(MethodImplOptions.NoInlining)]
static Vector128<uint> BadAnd(Vector128<ulong> v, Vector128<uint> w) =>
    Vector128.ConditionalSelect(
        Vector128.GreaterThan(w, Vector128<uint>.Zero),
        (v & Vector128<uint>.One.AsUInt64()).AsUInt32(),
        Vector128<uint>.Zero
    );

Console.WriteLine(BadAnd(Vector128<ulong>.AllBitsSet, Vector128.Create(0u, 0, 2, 2)));

Output on net9.0

<0, 0, 1, 1>

Output on main

<0, 0, 0, 0>

Codegen for BadAnd

vmovups  xmm0, xmmword ptr [r8]
vxorps   xmm1, xmm1, xmm1
vpcmpud  k1, xmm0, xmm1, 6
vmovups  xmm0, xmmword ptr [rdx]
vpandq   xmm0 {k1}{z}, xmm0, qword ptr [reloc @RWD00] {1to2} ; bad mask size
vmovups  xmmword ptr [rcx], xmm0

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[jakobbotsch]

cc @dotnet/jit-contrib