Processes don't terminate when I call Python script using crontab

Question

I have a script called yarn_monitor.py. When I run it, the program executes correctly and when I look at the running processes using ps -u myname, everything is clear.

But when I run yarn_monitor.py using cron:

* * * * * /home/me/projects/yarn_monitor/yarn_monitor.py

I see several processes that don't quit. Here they are repeated twice:

19337 ?        00:00:00 yarn_monitor.py
19338 ?        00:00:03 java
19418 ?        00:00:00 sendmail
19419 ?        00:00:00 postdrop
20043 ?        00:00:00 yarn_monitor.py
20046 ?        00:00:02 java
20199 ?        00:00:00 sendmail
20200 ?        00:00:00 postdrop

Eventually, as I let cron keep running the job, I get a Java GC out of memory error.

As far as I know I'm not using any Java in my process. Here are my imports:

from __future__ import print_function
import os
import sys
import re
import time
from pysnmp.entity.rfc3413.oneliner import ntforg
from pysnmp.proto import rfc1902

Any idea what could be happening? Or ways to keep running this job while killing previously created processes?

Answer 1

It looks like the close isn't working correctly when run through cron, but it works fine when called manually.

Sounds to me like yarn is expecting some environment variables to be set which aren't set when run as a cron job. Try the following to debug this:

* * * * * yarn application -list > /home/me/error_log 2>&1 

Now wait 1 minute and look into /home/me/error_log and see what it's complaining about. This will give you a hint on what you need to do to fix your environment.

Answer 2

It is possible that the bash_profile of cron tab does not have all the environment variables to use the correct path. Try doing something like

***** source ~/.bashrc && python abc.py

Or

Mention the absolute path for the Python source like /use/lib/Python 2.7/Python abc.py