Proper usage of SIGTERM and SystemExit

107 Views Asked by At

I am writing a function which trains some models in a loop, but if I pass a SIGTERM signal, it completes the training of current model and then exits the code. I have written it like this

async def func1():
    obj = modelobj()
    def termination_handler(signum, frame):
        obj.train_final_model()
        print("Training done !")
        raise SystemExit
    try:
        signal.signal(signal.SIGTERM, termination_handler)
        obj.train_models()
        await asyncio.sleep(1)
    except SystemExit:
        print("All models trained !")
    except Exception as e:
        print(e)

Please note, the object and its methods are used as placeholders and not actual methods since the code being developed is confidential.

Also to note, I am running this code using gunicorn with uvicorn workers as worker-class so that I can terminate a particular PID using the command kill -s TERM <pid>

# FastAPI app
@app.post("/models")
async def model_training():
    asyncio.ensure_future(func1())
    await asyncio.sleep(1)
    return "OK"

Now, when this program runs, sometimes it gives the desired output, sometimes it does not print the print statement inside termination_handler, sometimes it throws error saying it did not find some model, and in some cases it trains models even after receiving the SIGTERM signal. What is it that I am doing wrong here?

0

There are 0 best solutions below