incrond processes with shell script only exit if script exit code is 1?

Question

Configuration

I have incrond 0.5.12 on CentOS 7.6 configured as follows in /etc/incron.d/example:

/var/tmp/dir IN_CREATE sh /root/incron_script.sh $@/$#

My /root/incron_script.sh simply contains the following: echo "$@" >> /tmp/incrond_log.log

What this means is that, when I create a file in var/tmp/dir, the file full path is appended to /tmp/incrond_log.log. That's it.

Problem definition

The problem is basically that, if incrond is configured to call a shell script, processes are being created and are not being stopped unless that shell script exits with something other than 0. What I'm looking at is the output of systemctl status incrond (or ps aux | grep ..., same thing).

So below, for example, there are 2 created processes.

[root@server ~]# systemctl status incrond
● incrond.service - Inotify System Scheduler
   Loaded: loaded (/usr/lib/systemd/system/incrond.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-12-11 13:39:55 +03; 11min ago
  Process: 16746 ExecStart=/usr/sbin/incrond (code=exited, status=0/SUCCESS)
 Main PID: 16747 (incrond)
    Tasks: 498
   Memory: 5.9M
   CGroup: /system.slice/incrond.service
           ├─13687 /usr/sbin/incrond
           ├─13747 /usr/sbin/incrond

Testing

We create 5 files, check if their names were appended to the log (incrond is working) and check how many processes are being spawned.

mkdir -p /var/tmp/dir
rm -f /var/tmp/dir/*
echo -n > /tmp/incrond_log.log
systemctl restart incrond
for i in $(seq 1 5);
do
    touch /var/tmp/dir/a$i.txt
    sleep 0.5
    tail -n1 /tmp/incrond_log.log
    systemctl status incrond | grep /usr/sbin/incrond | wc -l
done

Expected result

I would expect incrond to fork a process for every file created at this directory but to exit immediately after since there's not much to do really. If the log shows that the file path is in the log file, this means that the incrond process should have stopped since it did its job. By default, there were 2 processes in systemctl status incrond, so the expected result of the command is:

/var/tmp/dir/a1.txt
2
/var/tmp/dir/a2.txt
2
/var/tmp/dir/a3.txt
2
/var/tmp/dir/a4.txt
2
/var/tmp/dir/a5.txt
2

Actual result

The actual result is:

/var/tmp/dir/a1.txt
3
/var/tmp/dir/a2.txt
4
/var/tmp/dir/a3.txt
5
/var/tmp/dir/a4.txt
6
/var/tmp/dir/a5.txt
7

Diagnosis

The problem is manifesting as zombie processes:

root      1540  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      1551  0.0  0.0  12784   672 ?        S    19:49   0:00 /usr/sbin/incrond
root      1553  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      1566  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      1576  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      2339  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      2348  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      2351  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      2355  0.0  0.0  12784   224 ?        S    19:49   0:00 /usr/sbin/incrond
root      5471  0.0  0.0      0     0 ?        Z    19:17   0:00 [incrond] <defunct>
root      5480  0.0  0.0      0     0 ?        Z    19:17   0:00 [incrond] <defunct>
root      5483  0.0  0.0      0     0 ?        Z    19:17   0:00 [incrond] <defunct>
root      5561  0.0  0.0      0     0 ?        Z    19:17   0:00 [incrond] <defunct>
root      8012  0.0  0.0      0     0 ?        Z    19:12   0:00 [incrond] <defunct>
root      8023  0.0  0.0      0     0 ?        Z    19:12   0:00 [incrond] <defunct>
root      8025  0.0  0.0      0     0 ?        Z    19:12   0:00 [incrond] <defunct>
root      8148  0.0  0.0      0     0 ?        Z    19:12   0:00 [incrond] <defunct>

This is as far as I can inspect. I don't know how to look into this further.

The fix

If, instead of a regular exit, I exit 1, processes exit properly. So my /root/incron_script becomes: echo "$@" >> /tmp/incrond_log.log && exit 1. My status now looks like:

[root@server ~]# systemctl status incrond
● incrond.service - Inotify System Scheduler
   Loaded: loaded (/usr/lib/systemd/system/incrond.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-12-11 14:09:04 +03; 16s ago
  Process: 7882 ExecStart=/usr/sbin/incrond (code=exited, status=0/SUCCESS)
 Main PID: 7888 (incrond)
    Tasks: 6
   Memory: 220.0K
   CGroup: /system.slice/incrond.service
           └─7888 /usr/sbin/incrond

Dec 11 14:09:09 server.example.com incrond[7888]: PATH (/var/tmp/dir) FILE (a1.txt) EVENT (IN_CREATE)
Dec 11 14:09:09 server.example.com incrond[7888]: (system::example) CMD (sh /root/incron_script.sh /var/tmp/dir/a1.txt )
Dec 11 14:09:10 server.example.com incrond[7888]: PATH (/var/tmp/dir) FILE (a2.txt) EVENT (IN_CREATE)
Dec 11 14:09:10 server.example.com incrond[7888]: (system::example) CMD (sh /root/incron_script.sh /var/tmp/dir/a2.txt )
Dec 11 14:09:10 server.example.com incrond[7888]: PATH (/var/tmp/dir) FILE (a3.txt) EVENT (IN_CREATE)
Dec 11 14:09:10 server.example.com incrond[7888]: (system::example) CMD (sh /root/incron_script.sh /var/tmp/dir/a3.txt )
Dec 11 14:09:11 server.example.com incrond[7888]: PATH (/var/tmp/dir) FILE (a4.txt) EVENT (IN_CREATE)
Dec 11 14:09:11 server.example.com incrond[7888]: (system::example) CMD (sh /root/incron_script.sh /var/tmp/dir/a4.txt )
Dec 11 14:09:11 server.example.com incrond[7888]: PATH (/var/tmp/dir) FILE (a5.txt) EVENT (IN_CREATE)
Dec 11 14:09:11 server.example.com incrond[7888]: (system::example) CMD (sh /root/incron_script.sh /var/tmp/dir/a5.txt )

Question

So is this the expected behavior then? Why does exit 0 keep the process alive while exit 1 doesn't? Where is this documented? Any suggestions on how I can debug this further?

Updates

• 2018-12-12: added diagnosis (zombie threads)

Answer 1

This seems to be part of a larger issue with incron 0.5.12 (incron/issues/52, incron/issues/53)