I'm running a python deduping application using the dedupe package. I've deployed the same as an API using flask and gunicorn. I'm running the application on a linux server with 128GB RAM and 40 core configuration. With a data size of 900000, the CPU resources get throttled very fast. Since it get's throttled very fast, many times the OS itself is killing the application. When the memory consumption is at 50%, the CPU consumption is at full throttle.
The gunicorn command I'm running to deploy is
gunicorn main:app -b 0.0.0.0:XXXX --timeout=1200 -w 30
How do I get around this CPU utilization?