s390x: Fix instruction encoding and disassembly format bugs

[uweigand]

• Fix encoding of the AHY instruction.
• Fix disassembly format of FIEBR, FIDBR, and LEDBRA instructions.

CC @cfallin @elliottt - this fixes s390x issues detected here: #5780

[afonso360]

Hey, I'm trying to understand why fuzzgen didn't catch this when you ran it a few weeks ago. This encoding error only triggers when we try to add a 16bit value directly from memory to a 32 bit register right?

I think fuzzgen can already generate code like that, although we only ever load from the stack, does s390x select a different encoding in those cases?

[uweigand]

Hey, I'm trying to understand why fuzzgen didn't catch this when you ran it a few weeks ago. This encoding error only triggers when we try to add a 16bit value directly from memory to a 32 bit register right?

Yes, and in addition the memory access must use a displacement of 4096 or larger. For smaller displacements, the (correctly encoded) AH instruction is used instead of AHY. This is probably a rare case - not sure if the fuzzer tries to enforce large displacements.

[afonso360]

Oh! That explains it, We constrain the stack size quite a lot. We should probably increase that. Thanks!

Edit: Just looked it up, our max stack size is 1024, which is really low.

[jameysharp]

I believe you that this is the correct encoding. 👍