Process continues to hold file after file deletion

Question

I created a log file by running the iostat command to a text file, and ran the command in the background using nohup.

#nohup iostat -xm 5 > /z/logfile.txt &

Later on, I created a cronjob that runs every ten minutes doing the same as above, after I realized my process was being killed by a reboot. I've also setup log-rotation as below:

/z/logfile.txt {
        size 20M
        rotate 0
        create 0644 root root
        missingok
        notifempty

} 

Now I have realized that the logfile.txt gets deleted but the iostat command keeps pointing at deleted files as shown by the lsof -n | grep deleted command. There the disk space is not freed.

How can I make sure the files are rotated and thereafter iostat points to the newly created file, freeing up disk space?

Any ideas how to set it up correctly?

Answer 1

One solution would be to write a program that will read from iostat, write to the output file, and accept a signal to reopen the file. For example, if you did: iostat -xm 5 | log-daemon /z/logfile.txt where log-daemon is a simple script like:

#!/bin/bash
echo $$ > /var/run/log-daemon
exec > $1
trap 'exec > $1' SIGHUP
read line
while test $? -le 0; do 
        echo $line
        read line
done

Then add a postrotate clause in the logrotate config to send a HUP to the log-daemon:

postrotate
               /usr/bin/kill -HUP $(cat /var/run/log-daemon)

Answer 2

Would pointing your cronjob iostat command at a softlink not work?

ln -s /z/logfile.txt iostat_link.txt
nohup iostat -xm 5 > /z/iostat_link.txt &

I haven't used logrotate before, but I tested this by manually changing the file in the background while I had this running:

#Make the files
touch afile1.txt
ln -s afile1.txt file.txt

#Kick off loop
for i in {1..1000};do echo "running still $i" >> file.txt;sleep 3;done &

[localhost (2017-05-15 20:30:55) IP: 26.176 ~]# cat afile1.txt
running still 7
running still 8
running still 9

#Change the file out from under the loop
mv afile1.txt afile1.txt.backup;touch afile1.txt

[localhost (2017-05-15 20:31:21) IP: 26.176 ~]# cat afile1.txt
running still 15
running still 16
running still 17

Answer 3

Check if file system where log is recording is full. If you have such a case find and kill process or reboot server in worst case.