Compiling with Mypyc

[nickdrozd]

Mypyc is a compiler that comes with Mypy. It uses type annotations to convert Python modules to C extensions. This doesn't sound like something that could possibly work, but in some cases it does, and it can provide a substantial speedup. It might be worth looking into with Coverage.

There are some valid and common Python idioms that Mypyc won't accept, so pursuing this will require some design changes. For example, there is a function called actual_path, defined like this:

if env.WINDOWS:
    def actual_path(path: str) -> str:
        ...
else:
    def actual_path(path: str) -> str:
        ...

But Mypyc won't accept conditionally defined functions or classes. So this would need to be reworked.

There are some deeper problems that don't have obvious workarounds. Some modules might just have to be skipped.

Then there is the issue of getting the compiled extensions to interact with the SQL extension. I know nothing about that.

Despite all that, the following invocation succeeds:

mypyc coverage --exclude cmdline --exclude html --exclude python --exclude plugin_support --exclude plugin --exclude parser --exclude env --exclude control --exclude pytracer --exclude phystokens --exclude tracer --exclude files

I don't know how to make it run, but it does compile without any errors. So there is some reason for hope that this could actually work.

[nedbat]

I wonder about the benefits to this work? The most performance-sensitive part of coverage.py is the trace function, and that is written in C. These days, the best way to improve the speed is to use the new PEP 669 tracing facility instead of sys.settrace. So I'm not sure it's worth it to twist the code for mypyc. Is there some way to quantify what we might gain?

[nickdrozd]

The main benefit would be speed. Even if the bulk of time is already spent in a C extension, it may be possible to significantly reduce time spent in the Python parts. Python is shockingly slow, even when you take into account that Python is slow. Personally I would love it if Coverage were even 10% faster.

Another benefit possibly is that it can lead to clearer code. You describe it as "twist the code for mypyc", but really it's more like untwisting the code. There might be latent bugs, ambiguities, type puns, or other irregularities that get exposed through satisyfing the compiler. Clarifying dynamic funny business, etc. Dynamic funny business can be great fun! But it isn't straightforward and it usually isn't fast either.

Plus, the Coverage code is already checked by Mypy, so using Mypyc is just one more step in that direction. And many of the modules already compile. I could be totally wrong, but my feeling is that it would not take a ton of work to get tangible benefits.

Here is a blog post that discusses compiling the Black formatter.