Generate opcode metadata from bytecodes.c instead of opcode.py

[iritkatriel]

We would ideally have bytecodes.c as the single source of truth about opcodes. So opcode.py and the code generated from it should be replaced by alternatives from the cases_generator, if we can.

Linked PRs

• gh-105481: add flags to each instr in the opcode metadata table, to replace opcode.hasarg/hasname/hasconst #105482
• gh-105481: add pseudo-instructions to the cases DSL #105506
• gh-105481: Fix types and bug for pseudos #105788
• gh-105481: add HAS_JUMP flag to opcode metadata #105791
• gh-105481: remove HAS_ARG, HAS_CONST, IS_JUMP_OPCODE, IS_PSEUDO_OPCODE and replace by their new versions #105865
• gh-105481: generate _specializations and _specialized_instructions from bytecodes.c #105913
• gh-105481: refactor instr flag related code into a new InstructionFlags class #105950
• gh-105481: move Python/opcode_metadata.h to Include/internal/pycore_opcode_metadata.h #106673
• gh-105481: expose opcode metadata via the _opcode module #106688
• gh-105481: Generate the opcode lists in dis from data extracted from bytecodes.c #106758
• gh-105481: do not auto-generate pycore_intrinsics.h #106913
• gh-105481: add OPCODE_IS_INSTRUMENTED (generated from bytecodes.c) to replace MIN_INSTRUMENTED_OPCODE (defined in opcode.py) #107276
• gh-105481: remove dependency of _inline_cache_entries on opname #107339
• gh-105481: the ENABLE_SPECIALIZATION flag does not need to be generated by the build script, or exposed in opcode.py #107534
• gh-105481: combine regen-opcode-targets with regen-opcode to avoid calculating the specialized opcodes in two places #107540
• gh-105481: simplify definition of pseudo ops in Lib/opcode.py #107561
• GH-105481: Mark more files as generated #107598
• gh-105481: split opcode_ids.h out of opcode.h so that it can be generated separately #107866
• gh-105481: reduce repetition in opcode metadata generation code #107942
• gh-105481: generate op IDs from bytecode.c instead of hard coding them in opcode.py #107971
• gh-105481: generate _PyOpcode_Deopt and _PyOpcode_OpName in generate_cases instead of generate_opcode_h #108025
• gh-105481: fix out of date comment #108079
• gh-105481: opcode.h is no longer generated during the build #108080
• gh-105481: remove regen-opcode. Generated _PyOpcode_Caches in regen-cases. #108367

[iritkatriel]

@gvanrossum The old macros HAS_ARG and HAS_CONST are exposed in Include/opcode.h (and have been for a long time). Does this mean that before we can remove them, we need to expose the opcode_metadata.h file there as well? We should also mark all of this as unstable API (how do we do that?)

[gvanrossum]

Neither HAS_ARG nor HAS_CONST is mentioned in the docs at all. I am of the opinion that in this case this implies they are not public, and anyone using them should not be surprised if they disappear. (It would be more of an issue if they turned into lies.)

opcode.h is mentioned only once in the docs, as the source of truth for dis.py. (It is also mentioned twice in Misc/NEWS.d/, but that's really just a changelog.)

I do think we ought to provide similar functionality to debuggers.

Info about declaring unstable APIs comes from PEP 649, which links to https://devguide.python.org/developer-workflow/c-api/#c-api. It looks like you must use the PyUnstable_ prefix (e.g. PyUnstable_BytecodeHasArg), and it ought to be in a .h file under Include/cpython/.

[iritkatriel]

I do think we ought to provide similar functionality to debuggers.

Do you mean to expose the stuff in opcode_metadata.h file through some unstable c api?

[gvanrossum]

Yeah, to the extent that it’s useful, in function form (so the data structure is not public).

Or we can wait until someone asks.

[iritkatriel]

There is a usage of HAS_ARG in Modules/_opcode.c. We could remove it (send oparg -1 if None was given and do this check inside Python/ code). Otherwise we need to move opcode_metadata.h from Python/ to Include/internal.

[gvanrossum]

I would just set it to 0 when it's None or missing. Honestly it's always bothered me that you must know ahead of time whether an arg is needed or not.

[vstinner]

@encukou wrote "Unstable API needs documentation and tests."

I reopen this issue and close gh-107149.

[iritkatriel]

I'm not sure it was the right move to rename _PyUnstable_GetUnaryIntrinsicName to PyUnstable_GetUnaryIntrinsicName, etc.

I created them as private because I did not intend to document them as a public API (we need a reason to do that). I put Unstable in the name because it's one of those things that can change all the time. If we don't want the _PyUnstable_ prefix then let's remove the Unstable and leave the _ until we have a reason/decision to make these part of the public C API.

[iritkatriel]

I'm moving this discussion back to #107149, and closing this, because this PR was about a large change, and #107149 is about the naming.