Improve the emitter support for APX instructions

[tannergooding]

Currently the emitter support for APX instructions is a bit complex and it can make the logic hard to follow.

In particular, APX comes with two basic types of encoding changes:

1. The 2-byte REX2 prefix
2. Extensions to the 4-byte EVEX prefix

For the former, we generally have flagged the relevant instructions in instrsxarch.h with a new Encoding_REX2 flag. This is simple to recognize and fits in with the existing model used for Encoding_VEX and Encoding_EVEX.

The extended EVEX prefix, however, is quite a bit more complex and doesn't cleanly integrate with the existing IsEvexEncodableInstruction and/or TakesEvexPrefix support. Rather there is a separate TakesApxExtendedEvexPrefix query and no IsApxExtendedEvexEncodableInstruction. This leads to a lot of duplicated checks and logic between the APX Extended EVEX and regular EVEX spaces.

I believe it would be beneficial to have some separate Encoding_* flag (potentially Encoding_APX) to differentiate the EVEX support that is APX only, rather than using the existing Encoding_EVEX flag. We should then likely continue using the regular TakesEvexPrefix and IsEvexEncodableInstruction paths for all EVEX usage, including APX Extended EVEX, ensuring that the latter does the appropriate UseApxEncoding() check for such instructions. -- This is notably what we do for cases like AVX-VNNI and AVX-IFMA which can have VEX-only, EVEX-only, or VEX+EVEX support.

This should reduce the total number of checks being done and keep the encoding logic contained to the relevant code paths. It should also make it simpler to do things like forcing APX usage or lighting up APX in other profitable scenarios, not just those that require features like NDD or extended register support.

[tannergooding]

CC. @dotnet/intel

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[tannergooding]

I encountered this while working on #117288, which was attempting to take advantage of compressed displacement in more scenarios and reduce the cost around it.

-- This was a relevant example because while the VEX encoding is profitable for existing cases that can be disp8, it is 1-byte larger than the EVEX encoding for cases that can use compressed displacement. I expect that APX will have similar examples where we will want to quickly and efficiently perform such checks and so we want the "can this be EVEX encoded" and other checks to be as cheap and normalized as possible

[Ruihan-Yin]

Hi @tannergooding, I can take this task and work on it, please assign me if you are okay with it, thanks!