Call Python Multiprocessing module from C/C++ cause adding new processes infinitely

Question

I am working on a project with an unusual framework under Windows system: c++ main() calls Python Multiprocessing function via C API. The framework works well without Multiprocessing. Once Multiprocessing module is used (even just 1 process), the program keeps adding new c++ main() exe processes. The Python script standalone works well with multiprocessing module used. I think it is related to the implementation of multiprocessing module; searching all though the internet find no related information. Could anyone provide some hint? Thank you!

start.py

import time
import multiprocessing as mp



def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleeping...')
    
def benchmark():
    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    p1.start()
    #p2.start()
    p1.join()
    #p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')
 
if __name__ == '__main__':
    benchmark()

callPythonFromCpp.cpp

#include <Windows.h>
#include <iostream>
#include <string>
#include <Python.h>


using namespace std;

void CallPython(string PythonModuleName, string PythonFunctionName)
{
    char* funcname = new char[PythonFunctionName.length() + 1];
    strcpy_s(funcname, PythonFunctionName.length() + 1, PythonFunctionName.c_str());

    char* modname = new char[PythonModuleName.length() + 1];
    strcpy_s(modname, PythonModuleName.length() + 1, PythonModuleName.c_str());

    
    // Initialize the Python interpreter 
    Py_Initialize();

    PyObject* my_module = PyImport_ImportModule(modname);

    PyObject* my_function = PyObject_GetAttrString(my_module, funcname);

    // Call a callable Python object callable, with arguments given by the tuple args. 
    // If no arguments are needed, then args can be NULL.

    PyObject* my_result = PyObject_CallObject(my_function, NULL);

    // Undo all initializations made by Py_Initialize() and subsequent use of Python/C API functions, 
    // and destroy all sub-interpreters (see Py_NewInterpreter() below) that were created and not yet 
    // destroyed since the last call to Py_Initialize(). Ideally, this frees all memory allocated by the Python interpreter.
    Py_Finalize();

    delete[] funcname;
    delete[] modname;
}
int main()
{
    CallPython("start", "benchmark");
    system("pause");
    return 0;
}

enter image description here

1. Call single-process coded Python from C++ main(), it works well;
2. Call multiprocessing Python from C++ main(), it keeps adding new process, Python code is not running;
3. Run multiprocessing Python script standalone, it works well parallelly.

Updated: print sys.executable and os.getpid() in python code; When p1.start() is called, a new process of C++ main() is called recursively. I think it is related to how does multiprocessing identify the main and child processes. e.g, Python standalone has "if name == 'main'", which is essential to use multiprocessing module.

Python update:

def benchmark():
    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    print('sys.executable: ' + sys.executable + '\n')
    print('os pid: ' + str(os.getpid()) + '\n')
    p1.start()
    #p2.start()
    #p1.join()
    #p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')

Terminal output:

sys.executable: C:\Users\xxx\source\repos\callPythonFromCpp\x64\Release\callPythonFromCpp.exe

os pid: 25408

Finished in 0.08 second(s)
sys.executable: C:\Users\xxx\source\repos\callPythonFromCpp\x64\Release\callPythonFromCpp.exe

os pid: 32332

Finished in 0.01 second(s)
sys.executable: C:\Users\xxx\source\repos\callPythonFromCpp\x64\Release\callPythonFromCpp.exe

os pid: 9456

Finished in 0.01 second(s)
sys.executable: C:\Users\xxx\source\repos\callPythonFromCpp\x64\Release\callPythonFromCpp.exe

os pid: 8944

Update 2023-11-23 Following this post Embedded python: multiprocessing not working my program works now. C++ code keeps the same; Python code as below:

import time
import multiprocessing as mp



def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleeping...')
    
def benchmark():
    sys.argv = [r'C:\path_to_this\start.py']
    mp.set_executable(r'C:\path_to_Python_install\python.exe')

    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')
 
if __name__ == '__main__':
    benchmark()

Answer 1

Following this post Embedded python: Embedded python: multiprocessing not working my program works now. C++ code remains the same; Python code needs 2 more lines to set up sys.argv and multiprocessing.set_executable.

I think it is caused by the implementation of the multiprocessing module: by default, it assumes the multiprocessing.set_executable is python.exe so that multiple independent python.exe instances will be called;

however, when I use a C++ main() to call the multiprocessing codes in Python script, the multiprocessing module treats the C++ main() as the executable. It starts multiple C++ exes, then more C++ exes will be called recursively, in the order of CPUs. My 24-core machine freezes much faster than the 8-core machine.

Below is the working Python code:

import time
import multiprocessing as mp

def do_something():
    print('Sleeping 1 second...')
    time.sleep(1)
    print('Done Sleeping...')

def benchmark():
    sys.argv = [r'C:\path_to_this\start.py']
    mp.set_executable(r'C:\path_to_Python_install\python.exe')

    start = time.perf_counter()

    p1 = mp.Process(target=do_something)
    p2 = mp.Process(target=do_something)
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    finish = time.perf_counter()

    print(f'Finished in {round(finish - start,2)} second(s)')

There is another potential problem: if multiprocessing code exists in Python codes, we cannot cythonize all Python codes; there must be a .py file on disk for the multiprocessing module to parse. See this post: Trouble transitioning to multiprocessing python code to cython

Update: from the Cython official document, we "should treat multiprocessing from an embedded Cython executable as unsupported."

https://cython.readthedocs.io/en/latest/src/tutorial/embedding.html