GH-98831: "Generate" the interpreter (#98830)

The switch cases (really TARGET(opcode) macros) have been moved from ceval.c to generated_cases.c.h. That file is generated from instruction definitions in bytecodes.c (which impersonates a C file so the C code it contains can be edited without custom support in e.g. VS Code).

The code generator lives in Tools/cases_generator (it has a README.md explaining how it works). The DSL used to describe the instructions is a work in progress, described in https://github.com/faster-cpython/ideas/blob/main/3.12/interpreter_definition.md.

This is surely a work-in-progress. An easy next step could be auto-generating super-instructions.

**IMPORTANT: Merge Conflicts**

If you get a merge conflict for instruction implementations in ceval.c, your best bet is to port your changes to bytecodes.c. That file looks almost the same as the original cases, except instead of `TARGET(NAME)` it uses `inst(NAME)`, and the trailing `DISPATCH()` call is omitted (the code generator adds it automatically).

.gitattributes

@@ -82,6 +82,7 @@ Parser/parser.c                                     generated
 Parser/token.c                                      generated
 Programs/test_frozenmain.h                          generated
 Python/Python-ast.c                                 generated
+Python/generated_cases.c.h                          generated
 Python/opcode_targets.h                             generated
 Python/stdlib_module_names.h                        generated
 Tools/peg_generator/pegen/grammar_parser.py         generated

Makefile.pre.in

@@ -1445,7 +1445,19 @@ regen-opcode-targets:
 		$(srcdir)/Python/opcode_targets.h.new
 	$(UPDATE_FILE) $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/opcode_targets.h.new
 
-Python/ceval.o: $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/condvar.h
+.PHONY: regen-cases
+regen-cases:
+	# Regenerate Python/generated_cases.c.h from Python/bytecodes.c
+	# using Tools/cases_generator/generate_cases.py
+	PYTHONPATH=$(srcdir)/Tools/cases_generator \
+	$(PYTHON_FOR_REGEN) \
+	    $(srcdir)/Tools/cases_generator/generate_cases.py \
+		-i $(srcdir)/Python/bytecodes.c \
+		-o $(srcdir)/Python/generated_cases.c.h.new
+	$(UPDATE_FILE) $(srcdir)/Python/generated_cases.c.h $(srcdir)/Python/generated_cases.c.h.new
+
+Python/ceval.o: $(srcdir)/Python/opcode_targets.h $(srcdir)/Python/condvar.h $(srcdir)/Python/generated_cases.c.h
+
 
 Python/frozen.o: $(FROZEN_FILES_OUT)
 

Misc/NEWS.d/next/Build/2022-10-28-22-24-26.gh-issue-98831.IXRCRX.rst

@@ -0,0 +1 @@
+We have new tooling, in ``Tools/cases_generator``, to generate the interpreter switch from a list of opcode definitions.