3.9.0a5.rst

Disallow control characters in hostnames in http.client, addressing :cve:`2019-18348`. Such potentially malicious header injection URLs now cause a InvalidURL to be raised.

Optimize pending calls in multithreaded applications. If a thread different than the main thread schedules a pending call (:c:func:`Py_AddPendingCall`), the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending calls which cannot be executed. Only the main thread can execute pending calls.

Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread executes pending calls.

Port _weakref extension module to multiphase initialization (PEP 489).

Port _collections module to multiphase initialization (PEP 489).

Optimize signal handling in multithreaded applications. If a thread different than the main thread gets a signal, the bytecode evaluation loop is no longer interrupted at each bytecode instruction to check for pending signals which cannot be handled. Only the main thread of the main interpreter can handle signals.

Previously, the bytecode evaluation loop was interrupted at each instruction until the main thread handles signals.

If :c:func:`Py_AddPendingCall` is called in a subinterpreter, the function is now scheduled to be called from the subinterpreter, rather than being called from the main interpreter. Each subinterpreter now has its own list of scheduled calls.

Port _heapq module to multiphase initialization.

Port itertools module to multiphase initialization (PEP 489).

Speed up calls to frozenset() by using the PEP 590 vectorcall calling convention. Patch by Donghee Na.

subinterpreters: Move _PyRuntimeState.ceval.tracing_possible to PyInterpreterState.ceval.tracing_possible: each interpreter now has its own variable.

Speed up calls to set() by using the PEP 590 vectorcall calling convention. Patch by Donghee Na.

Port _statistics module to multiphase initialization (PEP 489).

Use inline function to replace extension modules' get_module_state macros.

Correctly raise SyntaxError if await is used inside non-async functions and PyCF_ALLOW_TOP_LEVEL_AWAIT is set (like in the asyncio REPL). Patch by Pablo Galindo.

Allow executing asynchronous comprehensions on the top level when the PyCF_ALLOW_TOP_LEVEL_AWAIT flag is given. Patch by Batuhan Taskaya.

Speed up calls to tuple() by using the PEP 590 vectorcall calling convention. Patch by Donghee Na.

Changed list overallocation strategy. It no longer overallocates if the new size is closer to overallocated size than to the old size and adds padding.

Update Unicode database to Unicode version 13.0.0.

Clear the frames of daemon threads earlier during the Python shutdown to call objects destructors. So "unclosed file" resource warnings are now emitted for daemon threads in a more reliable way.

Fix a bug that was causing incomplete results when calling pathlib.Path.glob in the presence of symlinks that point to files where the user does not have read access. Patch by Pablo Galindo and Matt Wozniski.

Fix :c:func:`PyEval_RestoreThread` random crash at exit with daemon threads. It now accesses the _PyRuntime variable directly instead of using tstate->interp->runtime, since tstate can be a dangling pointer after :c:func:`Py_Finalize` has been called. Moreover, the daemon thread now exits before trying to take the GIL.

Fix a possible :exc:`SystemError` in math.{atan2,copysign,remainder}() when the first argument cannot be converted to a :class:`float`. Patch by Zackery Spytz.

Fix race condition where threads created by PyGILState_Ensure() could get a duplicate id.

This affects consumers of tstate->id like the contextvar caching machinery, which could return invalid cached objects under heavy thread load (observed in embedded scenarios).

Fixed a crash due to incorrect handling of weak references in collections.OrderedDict classes. Patch by Pablo Galindo.

Port audioop extension module to multiphase initialization (PEP 489).

Relax :term:`decorator` grammar restrictions to allow any valid expression (PEP 614).

Tweak import deadlock detection code to not deadlock itself.

Port _locale extension module to multiphase initialization (PEP 489).

Optimize :c:func:`PyUnicode_AsUTF8` and :c:func:`PyUnicode_AsUTF8AndSize` slightly when they need to create internal UTF-8 cache.

Fix unparsing of ext slices with no items (foo[:,]). Patch by Batuhan Taskaya.

Do not optimize annotations if 'from __future__ import annotations' is used. Patch by Pablo Galindo.

Using :data:`NotImplemented` in a boolean context has been deprecated. Patch contributed by Josh Rosenberg.

Don't leak environment variable __PYVENV_LAUNCHER__ into the interpreter session on macOS.

Add :class:`zipfile.Path` to __all__ in the :mod:`zipfile` module.

Improved error messages for validation of ast.Constant nodes. Patch by Batuhan Taskaya.

__module__ of the AST node classes is now set to "ast" instead of "_ast". Added docstrings for dummy AST node classes and deprecated attributes.

:func:`uuid.getnode` now skips IPv6 addresses with the same string length than a MAC address (17 characters): only use MAC addresses.

Deprecated ast.AugLoad and ast.AugStore node classes because they are no longer used.

Ensure bin/python3.# is always present in virtual environments on POSIX platforms - by Anthony Sottile.

Deprecated ast.Param node class because it's no longer used. Patch by Batuhan Taskaya.

Ensure all workers exit when finalizing a :class:`multiprocessing.Pool` implicitly via the module finalization handlers of multiprocessing. This fixes a deadlock situation that can be experienced when the Pool is not properly finalized via the context manager or a call to multiprocessing.Pool.terminate. Patch by Batuhan Taskaya and Pablo Galindo.

sys.settrace(), sys.setprofile() and _lsprof.Profiler.enable() now properly report :c:func:`PySys_Audit` error if "sys.setprofile" or "sys.settrace" audit event is denied.

AIX: Fix _aix_support module when the subprocess is not available, when building Python from scratch. It now uses new private _bootsubprocess module, rather than having two implementations depending if subprocess is available or not. So _aix_support.aix_platform() result is now the same if subprocess is available or not.

:class:`collections.OrderedDict` now implements | and |= (PEP 584).

The column name found in sqlite3.Cursor.description is now truncated on the first '[' only if the PARSE_COLNAMES option is set.

Ensure :attr:`unittest.mock.AsyncMock.await_args_list` has call objects in the order of awaited arguments instead of using :attr:`unittest.mock.Mock.call_args` which has the last value of the call. Patch by Karthikeyan Singaravelan.

Updated :data:`os.environ` and :data:`os.environb` to support PEP 584's merge (|) and update (|=) operators.

The ensurepip module now invokes pip via the runpy module. Hence it is no longer tightly coupled with the internal API of the bundled pip version, allowing easier updates to a newer pip version both internally and for distributors.

Fix the :meth:`random.Random.seed` method when a :class:`bool` is passed as the seed.

More reliable use of os.scandir() in Path.glob(). It no longer emits a ResourceWarning when interrupted.

:mod:`multiprocessing` now supports abstract socket addresses (if abstract sockets are supported in the running platform). When creating arbitrary addresses (like when default-constructing :class:`multiprocessing.connection.Listener` objects) abstract sockets are preferred to avoid the case when the temporary-file-generated address is too large for an AF_UNIX socket address. Patch by Pablo Galindo.

:func:`ast.dump` no longer outputs optional fields and attributes with default values. The default values for optional fields and attributes of AST nodes are now set as class attributes (e.g. Constant.kind is set to None).

Fixed :func:`ast.unparse` for extended slices containing a single element (e.g. a[i:j,]). Remove redundant tuples when index with a tuple (e.g. a[i, j]).

Fix :mod:`json.tool` to catch :exc:`BrokenPipeError`. Patch by Donghee Na.

Avoid a possible "RuntimeError: dictionary changed size during iteration" from :func:`inspect.getmodule` when it tried to loop through :data:`sys.modules`.

Revert "bpo-37330: open() no longer accept 'U' in file mode". The "U" mode of open() is kept in Python 3.9 to ease transition from Python 2.7, but will be removed in Python 3.10.

The hosts method on 32-bit prefix length IPv4Networks and 128-bit prefix IPv6Networks now returns a list containing the single Address instead of an empty list.

Add getConnection method to logging HTTPHandler to enable custom connections.

Reimplement distutils.spawn.spawn function with the :mod:`subprocess` module.

Add --without-decimal-contextvar build option. This enables a thread-local rather than a coroutine local context.

:class:`collections.defaultdict` now implements | (PEP 584).

Fix runpy.run_path() when using pathlike objects

Change inspect.Signature.parameters back to collections.OrderedDict. This was changed to dict in Python 3.9.0a4.

Refactor queue_manager in :class:`concurrent.futures.ProcessPoolExecutor` to make it easier to maintain.

Fix AttributeError when calling get_stack on a PyAsyncGenObject Task

The :func:`compileall.compile_dir` function's ddir parameter and the compileall command line flag -d no longer write the wrong pathname to the generated pyc file for submodules beneath the root of the directory tree being compiled. This fixes a regression introduced with Python 3.5.

:class:`types.MappingProxyType` objects now support the merge (|) operator from PEP 584.

The :mod:`importlib` module now ignores the :envvar:`PYTHONCASEOK` environment variable when the :option:`-E` or :option:`-I` command line options are being used.

Remove :meth:`tempfile.SpooledTemporaryFile.softspace` as files no longer have the softspace attribute in Python 3. Patch by Shantanu.

Improve pathlib.Path compatibility on zipfile.Path and correct performance degradation as found in zipp 3.0.

Keep ASDL signatures in the docstrings for AST nodes. Patch by Batuhan Taskaya

Deprecated ast.Suite node class because it's no longer used. Patch by Batuhan Taskaya.

Add thread_name_prefix to default asyncio executor

Fix handling of header in :class:`urllib.request.AbstractDigestAuthHandler` when the optional qop parameter is not present.

HTTP status codes 103 EARLY_HINTS and 425 TOO_EARLY are added to :class:`http.HTTPStatus`. Patch by Donghee Na.

Adding HTTP status 418 "I'm a Teapot" to HTTPStatus in http library. Patch by Ross Rhodes.

Remove default value from attrs parameter of :meth:`xml.etree.ElementTree.TreeBuilder.start` for consistency between Python and C implementations.

Open issue in the BPO indicated a desire to make the implementation of codecs.open() at parity with io.open(), which implements a try/except to assure file stream gets closed before an exception is raised.

Added starred expressions support to return and yield statements for lib2to3. Patch by Vlad Emelianov.

When using minidom module to generate XML documents the ability to add Standalone Document Declaration is added. All the changes are made to generate a document in compliance with Extensible Markup Language (XML) 1.0 (Fifth Edition) W3C Recommendation (available here: https://www.w3.org/TR/xml/#sec-prolog-dtd).

Add support for scoped IPv6 addresses to :mod:`ipaddress`. Patch by Oleksandr Pavliuk.

Simplified AST for subscription. Simple indices are now represented by their value, extended slices are represented as tuples. :mod:`ast` classes Index and ExtSlice are considered deprecated and will be removed in future Python versions. In the meantime, Index(value) now returns a value itself, ExtSlice(slices) returns Tuple(slices, Load()).

Updated the Language Reference for PEP 572.

Change 'string' to 'specification' in format doc.

The language reference no longer restricts default class namespaces to dicts only.

Fix misleading documentation about mixed-type numeric comparisons.

Update :mod:`token` documentation to reflect additions in Python 3.8

Changed operand name of MAKE_FUNCTION from argc to flags for module :mod:`dis`

test_gdb now skips tests if it detects that gdb failed to read debug information because the Python binary is optimized.

test_site.test_startup_imports() is now skipped if a path of :data:`sys.path` contains a .pth file.

Do not fail test_shutil test_chown test when uid or gid of user cannot be resolved to a name.

test_subprocess.test_user() now skips the test on an user name if the user name doesn't exist. For example, skip the test if the user "nobody" doesn't exist on Linux.

Fix build with DTrace but without additional DFLAGS.

setup.py now uses a basic implementation of the :mod:`subprocess` module if the :mod:`subprocess` module is not available: before required C extension modules are built.

Add --with-platlibdir option to the configure script: name of the platform-specific library directory, stored in the new :data:`sys.platlibdir` attribute. It is used to build the path of platform-specific extension modules and the path of the standard library. It is equal to "lib" on most platforms. On Fedora and SuSE, it is equal to "lib64" on 64-bit platforms. Patch by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner.

Ensures the required :file:`vcruntime140.dll` is included in install packages.

Avoid hang when computer is hibernated whilst waiting for a mutex (for lock-related objects from :mod:`threading`) around 49-day uptime.

distutils will no longer statically link :file:`vcruntime140.dll` when a redistributable version is unavailable. All future releases of CPython will include a copy of this DLL to ensure distributed extensions can continue to load.

Update Windows builds to use SQLite 3.31.1

Update Windows release build machines to Visual Studio 2019 (MSVC 14.2).

Package for nuget.org now includes repository reference and bundled icon image.

Update macOS builds to use SQLite 3.31.1

For 'Go to Line', use a Query box subclass with IDLE standard behavior and improved error checking.

Since clicking to get an IDLE context menu moves the cursor, any text selection should be and now is cleared.

Edit "Go to line" now clears any selection, preventing accidental deletion. It also updates Ln and Col on the status bar.

Selecting code context lines no longer causes a jump.

Port python-gdb.py to FreeBSD. python-gdb.py now checks for "take_gil" function name to check if a frame tries to acquire the GIL, instead of checking for "pthread_cond_timedwait" which is specific to Linux and can be a different condition than the GIL.

Added support to fix getproxies in the :mod:`!lib2to3.fixes.fix_urllib` module. Patch by José Roberto Meza Cabrera.

Add :c:func:`PyModule_AddType` helper function: add a type to a module. Patch by Donghee Na.

Remove _PyRuntime.getframe hook and remove _PyThreadState_GetFrame macro which was an alias to _PyRuntime.getframe. They were only exposed by the internal C API. Remove also PyThreadFrameGetter type.

Add :c:func:`PyThreadState_GetFrame` function: get the current frame of a Python thread state.

Add _PyArg_NoKwnames helper function. Patch by Donghee Na.

Add :c:func:`PyThreadState_GetInterpreter`: get the interpreter of a Python thread state.

Add :c:func:`PyInterpreterState_Get` function to the limited C API.

If :c:func:`PySys_Audit` fails in :c:func:`PyEval_SetProfile` or :c:func:`PyEval_SetTrace`, log the error as an unraisable exception.

Move the static inline function flavor of Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() to the internal C API: they access PyThreadState attributes. The limited C API provides regular functions which hide implementation details.

Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macro no longer access PyThreadState attributes, but call new private _PyTrash_begin() and _PyTrash_end() functions which hide implementation details.

:c:func:`PyDescr_NewMethod` and :c:func:`PyCFunction_NewEx` now include the method name in the SystemError "bad call flags" error message to ease debug.

Deprecated :c:func:`!PyEval_InitThreads` and :c:func:`!PyEval_ThreadsInitialized`. Calling :c:func:`!PyEval_InitThreads` now does nothing.

:c:macro:`Py_UNREACHABLE` is now implemented with __builtin_unreachable() and analogs in release mode.

:c:func:`PyNumber_ToBase` now raises a :exc:`SystemError` instead of crashing when called with invalid base.

The :c:func:`Py_FatalError` function is replaced with a macro which logs automatically the name of the current function, unless the Py_LIMITED_API macro is defined.

Extension modules: :c:member:`~PyModuleDef.m_traverse`, :c:member:`~PyModuleDef.m_clear` and :c:member:`~PyModuleDef.m_free` functions of :c:type:`PyModuleDef` are no longer called if the module state was requested but is not allocated yet. This is the case immediately after the module is created and before the module is executed (:c:data:`Py_mod_exec` function). More precisely, these functions are not called if :c:member:`~PyModuleDef.m_size` is greater than 0 and the module state (as returned by :c:func:`PyModule_GetState`) is NULL.

Extension modules without module state (m_size <= 0) are not affected.

Fixed segfault in Py_BuildValue() called with a format containing "#" and undefined PY_SSIZE_T_CLEAN whwn an exception is set.

Add a private API to get and set the frame evaluation function: add :c:func:`_PyInterpreterState_GetEvalFrameFunc` and :c:func:`_PyInterpreterState_SetEvalFrameFunc` C functions. The :c:type:`_PyFrameEvalFunction` function type now takes a tstate parameter.