We are using logrotate to rotate our podman container logs.
Our configurations look like below.
/home/user/share/logs/*/*.log {
rotate 24
hourly
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
copytruncate
missingok
notifempty
}
using the above configurations we can rotate the logs successfully, but when we keep this running for 2-3 days then we see null characters at the beginning of the file and we are not able to open the logs. This only happens after 2-3 days before that everything works as expected.
After the error Log file looks like this
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
After doing some research we found out the reason behind it is-
When rotating it maintains the same writing offset and starts writing from the middle of the new file. Everything before that offset is marked as NULL, as the beginning of the file is empty.*
We updated our configuration and tried the below configurations:
#1- Empty the content manually
/home/user/share/logs/*/*.log {
rotate 24
hourly
copy
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
missingok
notifempty
nosharedscripts
postrotate
cat /dev/null > "$1"
endscript
}
This shows the same null character error.
#2- Tried creating new file instead of copy and then as container to start writing in new file.
/home/user/share/logs/*/*.log {
rotate 24
hourly
create
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
missingok
notifempty
nosharedscripts
postrotate
postrotate
filepath="$1"
basename="${filepath##*/}"
container="${basename%.*}"
echo "$container"
podman exec $container sh -c "kill -USR1 1"
endscript
with this, the podman containers stop writing logs to the file and no new logs are found.
#3- removing the null characters from the file.
/home/user/share/logs/*/*.log {
rotate 24
hourly
compress
dateext
dateformat .%Y-%m-%d_%H-%M-%S
copytruncate
missingok
notifempty
nosharedscripts
postrotate
sleep 15
temp_file=$(mktemp)
sed 's/\x00//g' "$1" > "$temp_file"
cat "$temp_file" > "$1"
rm "$temp_file"
endscript
}
Nothing is working for me, not sure what I am missing here. Please help and Thanks in Advance.