mishegos: Construct VEX/XOP/EVEX opcodes

[aengelke]

The VEX/XOP/EVEX "prefixes" actually behave like being part of the opcode (the opcode byte must immediately follow the "prefix"). Explicitly constructing these opcodes significantly increases the likelihood of correctly hitting vector instructions.

This is interesting not only for coverage, but also for the plenty of UD constraints in AVX-512.

I hope I didn't miss any occasions where implicit assumptions on the opcode size are made.

[woodruffw]

Thanks for adding support for these! I'll take a closer look in a bit.

[woodruffw]

Thanks for adding this! There is indeed an implicit assumption about opcode size:

/* An x86 instruction's opcode is no longer than 3 bytes.
 */
typedef struct __attribute__((packed)) {
  uint8_t len;
  uint8_t op[3];
} opcode;
static_assert(sizeof(opcode) == 4, "opcode should be 4 bytes");

...so that opcode struct needs to be lengthened to accommodate these changes.

[woodruffw]

JFYI: I haven't forgotten this! I'll have some time to review it again today or tomorrow.

[woodruffw]

Sorry for the delay here. I just merged your performance refactor, so this needs deconflicting. The changes themselves LGTM, though!