Null in the beginning of file after rotating podman containers logs

17 Views Asked by At

We are using logrotate to rotate our podman container logs.

Our configurations look like below.

/home/user/share/logs/*/*.log {
rotate 24
hourly
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
copytruncate
missingok
notifempty
}

using the above configurations we can rotate the logs successfully, but when we keep this running for 2-3 days then we see null characters at the beginning of the file and we are not able to open the logs. This only happens after 2-3 days before that everything works as expected.

After the error Log file looks like this

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

After doing some research we found out the reason behind it is-

When rotating it maintains the same writing offset and starts writing from the middle of the new file. Everything before that offset is marked as NULL, as the beginning of the file is empty.*

We updated our configuration and tried the below configurations:

#1- Empty the content manually

/home/user/share/logs/*/*.log {
rotate 24
hourly
copy
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
missingok
notifempty
nosharedscripts
postrotate
        cat /dev/null > "$1"
endscript
}

This shows the same null character error.

#2- Tried creating new file instead of copy and then as container to start writing in new file.

/home/user/share/logs/*/*.log {
rotate 24
hourly
create
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
missingok
notifempty
nosharedscripts
postrotate
        postrotate
        filepath="$1"
        basename="${filepath##*/}"
        container="${basename%.*}"
        echo "$container"
        podman exec $container sh -c "kill -USR1 1"
endscript

with this, the podman containers stop writing logs to the file and no new logs are found.

#3- removing the null characters from the file.

/home/user/share/logs/*/*.log {
rotate 24
hourly
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
copytruncate
missingok
notifempty
nosharedscripts
postrotate
        sleep 15
        temp_file=$(mktemp)
        sed 's/\x00//g' "$1" > "$temp_file"
        cat "$temp_file" > "$1"
        rm "$temp_file"
endscript
}

Nothing is working for me, not sure what I am missing here. Please help and Thanks in Advance.

0

There are 0 best solutions below