How to Kill a Python Thread

I'm often asked how to kill a background thread, and the answer to this question makes a lot of people unhappy: threads cannot be killed. In this article I'm going to show you two options we have in Python to terminate threads.

A Threaded Example

To make this article more useful, let's use a simple example. Below is the complete code, which you can copy & paste and run on your computer with a name such as thread.py:

import random
import threading
import time

def bg_thread():
    for i in range(1, 30):
        print(f'{i} of 30 iterations...')
        time.sleep(random.random())  # do some work...

    print(f'{i} iterations completed before exiting.')

th = threading.Thread(target=bg_thread)
th.start()
th.join()

The only requirement for this application is Python, so there is no need to install any packages. You can start this application as follows:

$ python thread.py

Start the application and let it print a few lines. Before it gets to 30 lines, press Ctrl-C to interrupt it and note what happens:

~ $ python thread.py
1 of 30 iterations...
2 of 30 iterations...
3 of 30 iterations...
4 of 30 iterations...
5 of 30 iterations...
6 of 30 iterations...
7 of 30 iterations...
^CTraceback (most recent call last):
  File "thread.py", line 14, in <module>
    th.join()
  File "/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
  File "/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
8 of 30 iterations...
9 of 30 iterations...
10 of 30 iterations...
11 of 30 iterations...
12 of 30 iterations...
13 of 30 iterations...
^CException ignored in: <module 'threading' from '/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py'>
Traceback (most recent call last):
  File "/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 1388, in _shutdown
    lock.acquire()
KeyboardInterrupt:

In the above run, I pressed Ctrl-C when the application reached the 7th iteration. At this point the main thread of the application raised the KeyboardInterrupt exception and wanted to exit, but the background thread did not comply and kept running. At the 13th iteration I pressed Ctrl-C a second time, and this time the application did exit.

Strange, isn't it? The problem is that Python has some logic that runs right before the process exits that is dedicated to wait for any background threads that are not configured as daemon threads to end before actually returning control to the operating system.

So the process received the interrupt signal in its main thread and was ready to exit, but first it needed to wait for the background thread to end. But this thread does not know anything about interrupting, all the thread knows is that it needs to complete 30 iterations before ending.

The wait mechanism that Python uses during exit has a provision to abort when a second interrupt signal is received. This is why a second Ctrl-C ends the process immediately.

As I mentioned in the introduction, threads cannot be killed, so what do you do? In the following sections I'll show you two options you have in Python to make the thread end in a timely matter.

Daemon Threads

I mentioned above that before Python exits, it waits for any threads that are not daemon threads. So what is a daemon thread? You may think I'm playing word games with you, but really the definition of a daemon thread is exactly that, a thread that does not block the Python interpreter from exiting.

How do you make a thread be a daemon thread? All thread objects have a daemon property. You can set this property to True before starting the thread, and then that thread will be considered a daemon thread.

Here is the example application from above, changed so that the background thread is a daemon thread:

import random
import threading
import time

def bg_thread():
    for i in range(1, 30):
        print(f'{i} of 30 iterations...')
        time.sleep(random.random())  # do some work...

    print(f'{i} iterations completed before exiting.')

th = threading.Thread(target=bg_thread)
th.daemon = True
th.start()
th.join()

Modify the application as indicated above, run it again, and try to interrupt it:

~ $ python x.py
1 of 30 iterations...
2 of 30 iterations...
3 of 30 iterations...
4 of 30 iterations...
5 of 30 iterations...
6 of 30 iterations...
^CTraceback (most recent call last):
  File "thread.py", line 15, in <module>
    th.join()
  File "/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 1011, in join
    self._wait_for_tstate_lock()
  File "/Users/mgrinberg/.pyenv/versions/3.8.6/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

This time the first Ctrl-C causes the process to exit immediately.

So what happens to the thread? The thread continues to run as if nothing happened right until the Python process terminates and returns to the operating system. At this point the thread just ceases to exist. You may think that this is effectively a way to kill the thread, but consider that to kill threads in this way you have to kill the process as well.

Python Events

Using daemon threads is an easy way to avoid having to handle an unexpected interruption in a multithreaded program, but this is a trick that only works in the particular situation of the process exiting. Unfortunately there are times when an application may want to end a thread without having to kill itself. Also, some threads may need to perform clean up work before they exit, and daemon threads do not allow that.

So what other options are there? Since it is not possible to force the thread to end, the only other option is to add logic to it that voluntarily exits when requested. There are several ways to implement this type of solution, but one that I particularly like is to use an Event object.

The Event class is provided in the threading module of the Python standard library. You can create an event object by instantiating the class:

exit_event = threading.Event()

An event object can be in one of two states: set or not set. After creation, the event is not set. To change the event state to set, you can call the set() method. To find out if an event is set or not, you can use the is_set() method, which returns True if the event is set or False if not. It is also possible to "wait" for the event using the wait() method. A wait operation blocks until the event is set, with an optional timeout.

The idea is to set the event at the time when the thread needs to exit. The thread then needs to check the state of the event as often as it can, usually inside a loop, and handle its own termination when it finds that the event has been set.

For the example shown above a good solution is to add a signal handler that catches the Ctrl-C interruption, and instead of exiting abruptly, just set the event and let the thread end gracefully.

Below is a possible implementation of this solution:

import random
import signal
import threading
import time

exit_event = threading.Event()


def bg_thread():
    for i in range(1, 30):
        print(f'{i} of 30 iterations...')
        time.sleep(random.random())  # do some work...

        if exit_event.is_set():
            break

    print(f'{i} iterations completed before exiting.')


def signal_handler(signum, frame):
    exit_event.set()


signal.signal(signal.SIGINT, signal_handler)
th = threading.Thread(target=bg_thread)
th.start()
th.join()

If you try to interrupt this version of the application, everything looks much nicer:

~ $ python thread.py
1 of 30 iterations...
2 of 30 iterations...
3 of 30 iterations...
4 of 30 iterations...
5 of 30 iterations...
6 of 30 iterations...
7 of 30 iterations...
^C7 iterations completed before exiting.

Note how the interruption was handled gracefully and the thread was able to run the code that appears after the loop. This technique is very useful when the thread needs to close file handles or database connections before it exits. Being able to run clean up code before the thread exits is sometimes necessary to avoid leaks of resources.

I mentioned above that event objects can also be waited on. Consider the thread loop in the example above:

    for i in range(1, 30):
        print(f'{i} of 30 iterations...')
        time.sleep(random.random())

        if exit_event.is_set():
            break

In each iteration, there is a call to time.sleep(), which will block the thread. If the exit event is set while the thread is sleeping then it cannot check the state of the event, so there is going to be a small delay before the thread is able to exit.

In cases like this one, where there is sleeping, it is more efficient to combine the sleep with the check of the event object by using the wait() method:

    for i in range(1, 30):
        print(f'{i} of 30 iterations...')
        if event.wait(timeout=random.random()):
            break

This solution effectively gives you an "interruptible" sleep, because if the event is set while the thread is stuck in the middle of the call to wait() then the wait will return immediately.

Conclusion

Did you know about event objects in Python? They are one of the simpler synchronization primitives and can be used not only as exit signals but in many other situations in which a thread needs to wait for some external condition to occur.

Do you want to learn another cool technique that uses event objects? My How to Make Python Wait article shows how to use an event to wait for a thread to end while showing a progress indicator!