Threading and forking combined

Mixing multiprocessing and threading is generally problematic and a recipe for deadlocks.

The following code was entered in 2016 at https://bugs.python.org/issue27422 in the Python bug tracker:

import multiprocessing

from concurrent.futures import ThreadPoolExecutor


def run(arg):
    print(f"starting {arg}")
    p = multiprocessing.Process(target=print, args=("running", arg))
    p.start()
    p.join()
    print(f"finished {arg}")


if __name__ == "__main__":
    n = 16
    tests = range(n)
    with ThreadPoolExecutor(n) as pool:
        for _r in pool.map(run, tests):
            pass

starting 0starting 1

starting 2
starting 3
starting 4
starting 5
starting 6
starting 7
starting 8
starting 9
starting 10
starting 11
starting 12
starting 13
starting 14
starting 15
running 1
running 0
finished 1
finished 0
running 8
running 3
running 14
finished 8
finished 3
finished 14
running 2
running 7
finished 7
finished 2
running 10
running 6
finished 10
running 5
finished 6
finished 5
running 4
finished 4
running 11
running 12
running 9
finished 11
finished 12
finished 9
running 13
finished 13
running 15
finished 15

Usually, threading is recommended after the fork, not before. Otherwise, the locks used when executing the threads are duplicated across the processes. If one of these processes dies with a lock, all other processes with this lock are deadlocked.