Python News: What's New From February 2022?

Python 3.11 Alpha 5 Released

While the final release of Python 3.11 is planned for October 2022, which is still months ahead, you can already check out what’s coming. According to the development and release schedule described in PEP 664, Python 3.11 is now in the alpha phase of its release cycle, which is meant to collect early feedback from users like you. At the beginning of February, 3.11.0a5, the fifth of seven planned alpha releases, became available for testing.

To play around with an alpha release of Python, you can grab the corresponding Docker image from Docker Hub, install an alternative Python interpreter with pyenv, or build the CPython source code using a compiler tool. The source code method lets you clone CPython’s repository from GitHub and check out a bleeding-edge snapshot without waiting for an alpha release.

If you’d like to explore some of the most exciting features coming to Python 3.11, then make sure you can run the alpha release of the interpreter. You’ll find the commands to download and run Python 3.11.0a5 below:

So what’s all the hype about this alpha release? There are several big and small improvements, but for now, let’s focus on some of the Python 3.11 highlights!

PEP 657: Error Locations in Tracebacks

Python 3.10 has already greatly improved its error messages. By pinpointing the root cause, providing context, and even suggesting fixes, error messages have become more human-friendly and helpful for Python beginners. Python 3.11 takes error messaging one step further to improve the debugging experience and expose an API for code analysis tools.

Sometimes, a single line of code can contain multiple instructions or a complex expression, which would be hard to debug in older Python versions:

$ cat test.py
x, y, z = 1, 2, 0
w = x / y / z

$ python3.10 test.py
Traceback (most recent call last):
  File "/home/realpython/test.py", line 2, in <module>
    w = x / y / z
ZeroDivisionError: float division by zero

$ python3.11a5 test.py
  File "/home/realpython/test.py", line 2, in <module>
    w = x / y / z
        ~~~~~~^~~
ZeroDivisionError: float division by zero

Here, one of the variables causes a zero division error. Python 3.10 tells you about the problem, but it doesn’t indicate the culprit. In Python 3.11, the traceback will include visual feedback about the exact location on a line that raised an exception. You’ll also have a programmatic way of getting that same information for tools.

Note that some of these enhanced tracebacks won’t work for code evaluated on the fly in the Python REPL, because the tracebacks require pre-compiled bytecode to keep track of the source lines:

Python 3.11.0a5 (main, Mar  1 2022, 10:05:02) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x, y, z = 1, 2, 0
>>> w = x / y / z
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: float division by zero

For more information on error locations in tracebacks, see PEP 657.

PEP 654: Exception Groups and except*

Python 3.11 will bring a new exception type called the exception group, which will allow for combining several exceptions into one container. The primary purpose of exception groups is to simplify error handling in concurrent code, especially Async IO, which has traditionally been verbose and messy. But there are other use cases, like grouping multiple exceptions raised during data validation.

Because the exception group is going to be a standard exception, you’ll still be able to handle it using the regular try … except clause. At the same time, there will be a new except* syntax for cherry-picking a particular member of an exception group while re-raising other members that you don’t currently want to deal with. If you catch a regular exception with except*, then Python will bundle it in a temporary exception group for you.

Here’s a somewhat contrived example of an exception group in action:

>>> try:
...     raise ExceptionGroup(
...         "Validation exceptions", (
...             ValueError("Invalid email"),
...             TypeError("Age must be a number"),
...             KeyError("No such country code"),
...         )
...     )
... except* (ValueError, TypeError) as subgroup:
...     print(subgroup.exceptions)
... except* KeyError as subgroup:
...     print(subgroup.exceptions)
...
(ValueError('Invalid email'), TypeError('Age must be a number'))
(KeyError('No such country code'),)

Unlike a regular except clause, the new syntax, except*, doesn’t stop when it finds a matching exception type but continues matching onward. This lets you handle more than one exception at a time. Another difference is that you end up with a subgroup of filtered exceptions that you can access through the .exceptions attribute.

Exception groups will have nice, hierarchical tracebacks, too:

  + Exception Group Traceback (most recent call last):
  |   File "/home/realpython/test.py", line 2, in <module>
  |     raise ExceptionGroup(
  |     ^^^^^^^^^^^^^^^^^^^^^
  | ExceptionGroup: Validation exceptions
  +-+---------------- 1 ----------------
    | ValueError: Invalid email
    +---------------- 2 ----------------
    | TypeError: Age must be a number
    +---------------- 3 ----------------
    | KeyError: 'No such country code'
    +------------------------------------

For more information on exception groups and except*, see PEP 654.

PEP 673: Self Type

Even though the release notes of Python 3.11 Alpha 5 mention this new feature, it hasn’t actually been merged to the main branch before tagging the release point in the repository’s history. If you’d like to get a taste of the Self type anyway, then use the latest commit in the main branch of CPython.

In short, you can use Self as a type hint, for example to annotate a method returning an instance of the class in which it was defined. There are many special methods in Python that return self. Apart from that, implementing fluent interface or data structures with cross-references, such as a tree or a linked list, would also benefit from this new type hint.

Below is a more concrete example that illustrates the usefulness of Self. In particular, due to the circular reference problem, you can’t annotate an attribute or a method with the class that you’re defining:

>>> from dataclasses import dataclass
>>> @dataclass
... class TreeNode:
...     parent: TreeNode
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 3, in TreeNode
NameError: name 'TreeNode' is not defined

Until now, there were two work-arounds for this error:

1. Use a string literal like "TreeNode", which most type checkers recognize
2. Define a type variable with the base class as the bound

Both methods share the same drawback of having to update the code when your class name changes. In Python 3.11, you’ll have another, more elegant, meaningful, and name-independent way to communicate the same intent:

Python 3.11.0a5+ (heads/main:b35603532b, Mar  3 2022, 19:32:06) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from dataclasses import dataclass
>>> from typing import Self
>>> @dataclass
... class TreeNode:
...     parent: Self
...
>>> TreeNode.__annotations__
{'parent': typing.Self}

For more information on the Self type, see PEP 673.

PEP 646: Variadic Generics

Again, although the Python 3.11 Alpha 5 release notes prominently feature variadic generics, they were still in very early development at the time of this article’s publication. The relevant code hasn’t even been merged into the main branch of the CPython repository. That said, if you insist on getting your hands dirty, then check out the draft implementation residing in a forked repository.

Generic types, or generics for short, help achieve better type safety. They’re a way of specifying types parameterized with other types, which might be concrete types like int or str:

fruits: set[str] = {"apple", "banana", "orange"}

Such a declaration restricts the type of set elements to strings so that a type checker would consider any attempt to add any other type an error. You can also denote the generic parameters with abstract symbols representing placeholders for those types. In this case, you’ve been able to define a custom type variable in earlier Python versions with TypeVar, like so:

from typing import TypeVar

T = TypeVar("T", int, float, str)

def add(a: T, b: T) -> T:
    return a + b

This would enforce that add() be called with arguments of the same type, T, which must be int, float, or str. A type checker would reject an attempt to call add() with two incompatible types.

Now, Python 3.11 will come with variadic generics in the form of TypeVarTuple, which targets a very specific use case found in scientific libraries like NumPy and dealing with multi-dimensional arrays. With variadic generics, you’ll be able to define the shape of an array by parameterizing its type with a variable number of placeholder types:

from typing import Generic, TypeVarTuple

Ts = TypeVarTuple("Ts")

class DatabaseTable(Generic[*Ts]):
    def insert(self, row: tuple[*Ts]) -> None:
        ...

users: DatabaseTable[int, str, str] = DatabaseTable()
users.insert((1, "Joe", "Doe"))
users.insert((2, "Jane", "Doe"))

roles: DatabaseTable[str, str] = DatabaseTable()
users.insert(("a2099b0f-c614-4d8d-a195-0330b919ff7b", "user"))
users.insert(("ea35ce1f-2a0f-48bc-bf4a-c555a6a63c4f", "admin"))

For more information on variadic generics, see PEP 646.

Performance Optimizations

Python 3.11 will become noticeably faster. You can experience the difference for yourself by running a short code snippet in the terminal using the old and the new interpreter. Here’s a quick benchmark of calculating the thirty-fifth element of the Fibonacci sequence, deliberately implemented with recursion to simulate a challenging task for the computer:

$ SETUP='fib = lambda n: 1 if n < 2 else fib(n - 1) + fib(n - 2)'

$ python3.10 -m timeit -s "$SETUP" 'fib(35)'
1 loop, best of 5: 3.16 sec per loop

$ python3.11a5 -m timeit -s "$SETUP" 'fib(35)'
1 loop, best of 5: 1.96 sec per loop

As you can see, it takes slightly over three seconds for Python 3.10 to complete the computation and less than two seconds for Python 3.11 Alpha 5. That’s a whopping increase in performance! Naturally, the difference will vary depending on the task at hand, and you should generally expect less impressive numbers on average.

For more information on the performance optimizations, see the faster-cpython project on GitHub.