I am writing a function which trains some models in a loop, but if I pass a SIGTERM signal, it completes the training of current model and then exits the code. I have written it like this
async def func1():
obj = modelobj()
def termination_handler(signum, frame):
obj.train_final_model()
print("Training done !")
raise SystemExit
try:
signal.signal(signal.SIGTERM, termination_handler)
obj.train_models()
await asyncio.sleep(1)
except SystemExit:
print("All models trained !")
except Exception as e:
print(e)
Please note, the object and its methods are used as placeholders and not actual methods since the code being developed is confidential.
Also to note, I am running this code using gunicorn with uvicorn workers as worker-class so that I can terminate a particular PID using the command kill -s TERM <pid>
# FastAPI app
@app.post("/models")
async def model_training():
asyncio.ensure_future(func1())
await asyncio.sleep(1)
return "OK"
Now, when this program runs, sometimes it gives the desired output, sometimes it does not print the print statement inside termination_handler, sometimes it throws error saying it did not find some model, and in some cases it trains models even after receiving the SIGTERM signal. What is it that I am doing wrong here?