GH-98831: "Generate" the interpreter #98830

gvanrossum · 2022-10-28T22:00:34Z

Overview

This is the first stage of a project to generate the interpreter, in particular the cases of the main switch. The references below explain the why. This PR introduces the following:

Two new files in the Python subdirectory, bytecodes.c and generated_cases.c.h. (UPDATE: cases.h -> generated_cases.c.h.)
- bytecodes.c contains instruction definitions. These almost look like the old switch cases, but instead of TARGET(INAME) { ... } they have inst(INAME, stack_effect) { ... }, and the DISPATCH() macro call at the end of the block can be omitted. (The stack effect is not yet used by the generator, and it is not always correct.) At the top and bottom of bytecodes.c is some dummy C code that makes look like valid C code to editors like VS Code.
- generated_cases.c.h is a generated file containing the switch cases. It is currently identical to the switch cases in ceval.c.
A new bunch of tooling, in Tools/cases_generator. This includes:
- ~~A temporary script, extract_cases.py, that extracts the cases from ceval.c and writes them, in the form of instruction definitions, to bytecodes.c.~~
- A script, generate_cases.py, that reads the instruction definitions from bytecodes.c and regenerates the cases, writing them to generated_cases.c.h.
- A parser for a subset of C that turns blocks of C code into an AST, so we can do transformations at the AST level. (There is currently one example of this -- the function always_exits() determines recursively whether a block is guaranteed to exit, using a custom definition of "exit" that includes macro calls like DISPATCH() and GO_TO_INSTRUCTION().)
Changes to ceval.c so that it uses #include "generated_cases.c.h". For now I've left the original switch cases in ceval.c, inside #else ... #endif, so that we can compare the original switch cases to the generated switch cases to validate the toolchain. However, I plan to rip these out either once the PR is (nearly) approved, or perhaps in a quick follow-up PR.
A new target in Makefile.pre.in to auto-generate generated_cases.c.h from bytecodes.c, using make regen-cases.
- Currently missing is similar tooling on Windows. For now, you can only regenerate the cases on UNIX. They are checked into the repo, so Windows should still compile.

References

Metadata

Issue: Generate the interpreter #98831

This doesn't work yet, just a checkpoint. I'm going to bed now. It's going to be a long slog.

(Don't do anything with them yet.)

(This is a checkpoint, doesn't work yet.)

https://gist.github.com/markshannon/db7ab649440b5af765451bb77c7dba34#gistcomment-4334862

- Change family syntax to `family(NAME) = {OP1, OP2, ...};` - Some cleanups in parser.py and generate_cases.py - Fixes to bytecodes.c to reduce red wiggles in VS Code

gvanrossum

I'm pushing the new version in a few minutes, after merging two tough commits from main. The pushed version will once again have the cases in ceval.c, I'll remove those later (right before merging).

gvanrossum · 2022-11-02T23:09:44Z

Python/bytecodes.c

+#define SET_TOP(v)        (stack_pointer[-1] = (v))
+#define GETLOCAL(i)     (frame->localsplus[i])
+
+#define inst(name, stack_effect) case name:


Good point. I added some more dummy arguments and labels, fixed the inst() macro, and I'm just copying all #includes from ceval.c...

gvanrossum · 2022-11-02T23:13:11Z

Python/bytecodes.c

+#define GETLOCAL(i)     (frame->localsplus[i])
+
+#define inst(name, stack_effect) case name:
+#define family(name) static int family_##name


Ah, that was a last-minute change, I had a + separator there first. I'll put it inside curlies and make the macro define an unsized array.

Tools/cases_generator/parser.py

Tools/cases_generator/generate_cases.py

gvanrossum · 2022-11-02T23:50:49Z

I have made the requested changes; please review again.

bedevere-bot · 2022-11-02T23:50:53Z

Thanks for making the requested changes!

@brandtbucher: please review the changes made to this pull request.

gvanrossum · 2022-11-02T23:51:17Z

(Wait, I haven't pushed yet. In a minute.)

gvanrossum · 2022-11-02T23:52:44Z

Now it's ready.

bedevere-bot · 2022-11-03T00:39:38Z

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 7a327aa 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

If you get a merge conflict, talk to Guido

gvanrossum · 2022-11-03T00:53:58Z

I have one more commit (removing the cases from ceval.c) and then I'll merge. If you get a merge conflict after that, re-apply your changes to the instructions in ceval.c to bytecodes.c, and re-run make regen-cases.

gvanrossum · 2022-11-03T03:58:04Z

All buildbots passed except the three s390 ones. I'm not going to wait for those, I'm pushing the final commit (fddc08c) and once the regular CI passes I'll merge. Hopefully before midnight.

Debt I owe from PR GH-98830.

gvanrossum added 30 commits October 11, 2022 16:47

New script to generate the ceval switch

089ab0c

This doesn't work yet, just a checkpoint. I'm going to bed now. It's going to be a long slog.

Add 'End regular instructions' marker

bba5b94

Extract to a format closer to interpreter_definition.md

99b0058

Switch to generating new DSL syntax

2939830

Indent output; Always generate stack effect syntax; use argparse

65abfc4

Check jump flag; tweak begin and end of the script a bit

51dbe28

Fix incorrect output for ??--??

a238f69

Script to generate cases.h from bytecodes.inst

2d5b1b5

Write bytecodes.inst into Python directory by default

ae042be

Initial bytecodes.inst

4315621

Initial cases.h

8c80bfd

Changes to ceval.c to use cases.h

7301203

Changes to Makefile.pre.in to generate cases.h

d9ac40b

Move #endif around

a5bff0c

Write families to .inst file

8c13c55

(Don't do anything with them yet.)

Derive stack effect for super-instructions and specializations

34c84bf

Regenerated bytecodes.inst with fewer errors

ff70895

Support array stack effects

78f8591

Array stack effects are now in bytecodes.inst

a29b815

Add Mark's C lexer

688bb5d

Start writing a real parser

7def42b

(This is a checkpoint, doesn't work yet.)

Move cases infrastructure to its own directory

8fae259

Pristine copy of lexer from Mark's gist

b998a97

https://gist.github.com/markshannon/db7ab649440b5af765451bb77c7dba34#gistcomment-4334862

Fix tiny bits in lexer.py

c44adad

Move c_lexer.c to lexer.c

d0c4398

Complete parser (not using it yet):

06a3afa

Complete the parser and use it

33540af

Use a dataclass for Token; move stuff around to fix tokens

1990b61

Make a nice syntax error in lexer.py

690d4c5

More handy stuff for raising syntax errors

b3cd440

gvanrossum added 2 commits November 2, 2022 16:43

Update from Brandt's code review

d8e6db8

- Change family syntax to `family(NAME) = {OP1, OP2, ...};` - Some cleanups in parser.py and generate_cases.py - Fixes to bytecodes.c to reduce red wiggles in VS Code

Merge remote-tracking branch 'origin/main' into generate-ceval-switch

8a312ca

gvanrossum commented Nov 2, 2022

View reviewed changes

gvanrossum requested a review from brandtbucher November 2, 2022 23:50

bedevere-bot added awaiting change review and removed awaiting changes labels Nov 2, 2022

Update generated files after merging from main

7a327aa

gvanrossum force-pushed the generate-ceval-switch branch from 8c273bd to 7a327aa Compare November 2, 2022 23:52

brandtbucher approved these changes Nov 3, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Nov 3, 2022

gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 3, 2022

Remove the redundant cases from ceval.c

fddc08c

If you get a merge conflict, talk to Guido

gvanrossum added the DO-NOT-MERGE label Nov 3, 2022

gvanrossum merged commit 41bc101 into python:main Nov 3, 2022

bedevere-bot removed the awaiting merge label Nov 3, 2022

This was referenced Nov 3, 2022

Make sure regen-cases are in regen-all #99048

Closed

gh-99048: add regen-cases to regen-all in Makefile #99049

Closed

gvanrossum mentioned this pull request Nov 3, 2022

GH-98831: Remove redundant extract_cases.py script #99065

Merged

gvanrossum removed the DO-NOT-MERGE label Nov 3, 2022

gvanrossum added a commit that referenced this pull request Nov 3, 2022

GH-98831: Remove redundant extract_cases.py script (GH-99065)

916af11

Debt I owe from PR GH-98830.

gvanrossum deleted the generate-ceval-switch branch December 8, 2022 22:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-98831: "Generate" the interpreter #98830

GH-98831: "Generate" the interpreter #98830

gvanrossum commented Oct 28, 2022 •

edited

Loading

gvanrossum left a comment

gvanrossum Nov 2, 2022

gvanrossum Nov 2, 2022

gvanrossum commented Nov 2, 2022

bedevere-bot commented Nov 2, 2022

gvanrossum commented Nov 2, 2022

gvanrossum commented Nov 2, 2022

bedevere-bot commented Nov 3, 2022

gvanrossum commented Nov 3, 2022

gvanrossum commented Nov 3, 2022

GH-98831: "Generate" the interpreter #98830

GH-98831: "Generate" the interpreter #98830

Conversation

gvanrossum commented Oct 28, 2022 • edited Loading

Overview

References

Metadata

gvanrossum left a comment

Choose a reason for hiding this comment

gvanrossum Nov 2, 2022

Choose a reason for hiding this comment

gvanrossum Nov 2, 2022

Choose a reason for hiding this comment

gvanrossum commented Nov 2, 2022

bedevere-bot commented Nov 2, 2022

gvanrossum commented Nov 2, 2022

gvanrossum commented Nov 2, 2022

bedevere-bot commented Nov 3, 2022

gvanrossum commented Nov 3, 2022

gvanrossum commented Nov 3, 2022

gvanrossum commented Oct 28, 2022 •

edited

Loading