Make LLDB compatible with the Python Limited C API

[JDevlieghere]

Currently, LLDB uses the Python private or unstable API. This means that LLDB only works with the exact same version of Python that LLDB was compiled with. See https://docs.python.org/3/c-api/stable.html for more details.

Previously, we were limited by SWIG, which generated code that wasn't part of the limited API. As of version 4.2, SWIG generates code that's compatible with the Python 3.4 stable API and later.

Currently, the minimum Python version is 3.8. Based on a quick attempt to define Py_LIMITED_API to 0x03080000, we'll need to find replacements for the following identifiers:

• PyImport_Inittab (Guarded by LLDB_USE_LIBEDIT_READLINE_COMPAT_MODULE)
• PyThreadState (Guarded by !LLDB_USE_PYTHON_SET_INTERRUPT)
• PyUnicode_AsUTF8AndSize (Stable ABI since version 3.10)
• PyConfig
• _Py_IsFinalizing (Stable ABI since version 3.13)
• Py_buffer (Stable ABI since version 3.11)
• PyBUF_SIMPLE
• PyBUF_READ
• PyObject_Print
• PyGILState_Check
• _PyObject_Dump
• PyRun_SimpleString
• PyRun_String
• PyBytes_AS_STRING
• PyList_GET_SIZE
• PyTuple_GET_SIZE

This issue tracks the work to migrate over to only using the limited C API in LLDB.

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Jonas Devlieghere (JDevlieghere)

Currently, LLDB uses the Python private or unstable API. This [means](https://lldb.llvm.org/resources/caveats.html#python) that LLDB only works with the exact same version of Python that LLDB was compiled with. See https://docs.python.org/3/c-api/stable.html for more details.

Previously, we were limited by SWIG, which generated code that wasn't part of the limited API. As of version 4.2, SWIG generates code that's compatible with the Python 3.4 stable API and later.

Currently, the minimum Python version is 3.8. Based on a quick attempt to define Py_LIMITED_API to 0x03080000, we'll need to find replacements for the following identifiers:

• _Py_IsFinalizing
• _PyObject_Dump
• Py_buffer
• PyBUF_READ
• PyBUF_SIMPLE
• PyBytes_AS_STRING
• PyGILState_Check
• PyList_GET_SIZE
• PyRun_SimpleString
• PyRun_String
• PyTuple_GET_SIZE
• PyUnicode_AsUTF8AndSize

This issue tracks the work to migrate over to only using the limited C API in LLDB.

[JDevlieghere]

Some of these will automatically go away if we bump the minimum supported Python version. For example, Py_buffer and its accessors are part of the limited API as of Python 3.11 and Py_IsFinalizing was added in 3.13. On the flip side, that usually indicates that there are no good equivalents in older versions.

[bulbazord]

You mentioned this in your description, but I think it's important to clarify that this will make SWIG 4.2 the minimum version of SWIG for LLDB development.

[JDevlieghere]

You mentioned this in your description, but I think it's important to clarify that this will make SWIG 4.2 the minimum version of SWIG for LLDB development.

Not quite. You're right that you need SWIG or later to build an LLDB that only uses the stable API. However, this issue tracks preparing LLDB to work with the stable API. The project policy decision can be made later and orthogonally. For example, we could keep the minimum version and only build against the stable API if you're using a new enough SWIG.