I've implemented a Fluentd sidecar to rotate logfiles stored on a Kubernetes cluster.
The container uses the following config to run logrotate every 15 minutes on a cronjob.
*/15 * * * * /usr/sbin/logrotate -v /etc/logrotate.conf
/etc/logrotate.conf below
/var/lib/docker/containers/*/*.log
{
size 10M
missingok
notifempty
copytruncate
rotate 5
su root root
}
This appears to be succesfully rotating logfiles when they reach roughly 10M, however logfiles are not being purged once the .5 logfile is rolled. It's possible some are, but some are reaching .9. Filesystem below:
0 Apr 20 21:48 ./6270fe6c6875e47272ce4d5c4cbd5766ce934c79878f50166211803f78e862cb/6270fe6c6875e47272ce4d5c4cbd5766ce934c79878f50166211803f78e862cb-json.log
569 Apr 20 21:48 ./6270fe6c6875e47272ce4d5c4cbd5766ce934c79878f50166211803f78e862cb/6270fe6c6875e47272ce4d5c4cbd5766ce934c79878f50166211803f78e862cb-json.log.1
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.9
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.8
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.7
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.6
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.5
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.4
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.3
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.2
9.5M Apr 20 22:00 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log.1
787.1K Apr 20 22:08 ./e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de/e21792c56001e343291e64927dcfc8dd5be3449b0c347c41211aa450ba0ce7de-json.log
0 Apr 20 21:48 ./bbec8c11ef419591efdbe6850144cac5bfb77222e8024447f9b15422d3e2b0ca/bbec8c11ef419591efdbe6850144cac5bfb77222e8024447f9b15422d3e2b0ca-json.log
146.4K Apr 20 21:48 ./bbec8c11ef419591efdbe6850144cac5bfb77222e8024447f9b15422d3e2b0ca/bbec8c11ef419591efdbe6850144cac5bfb77222e8024447f9b15422d3e2b0ca-json.log.1
8.0M Apr 20 22:08 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log
7.0M Apr 20 21:58 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.5
7.1M Apr 20 21:59 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.4
7.1M Apr 20 22:00 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.3
7.1M Apr 20 22:00 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.2
7.2M Apr 20 22:00 ./33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.1
0 Apr 20 10:46 ./f683a00adabfc6ea9b4edfd19c336d3bd2e0e5747fb8b49713ffb5e002f4266e/f683a00adabfc6ea9b4edfd19c336d3bd2e0e5747fb8b49713ffb5e002f4266e-json.log
I don't know if perhaps I'm using an invalid configuration? I don't need to keep the logs for long, they're shipped to AWS CloudWatch almost as quickly as they're written - I just need the nodes to not consume disk space.
Log output (trimmed)...
reading config file /etc/logrotate.conf
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state
Handling 1 logs
rotating pattern: /var/lib/docker/containers/*/*.log
7340032 bytes (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/lib/docker/containers/052e7eaaab7bfe74e77a2e43f314a0dd219ddfd998ad670cab4936fa2ee16937/052e7eaaab7bfe74e77a2e43f314a0dd219ddfd998ad670cab4936fa2ee16937-json.log
Now: 2021-04-21 10:45
Last rotated at 2021-04-21 08:45
log does not need rotating (log size is below the 'size' threshold)
...
...
rotating log /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log, log->rotateCount is 5
dateext suffix '-20210421'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.5 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.6 (rotatecount 5, logstart 1, i 5),
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.4 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.5 (rotatecount 5, logstart 1, i 4),
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.3 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.4 (rotatecount 5, logstart 1, i 3),
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.2 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.3 (rotatecount 5, logstart 1, i 2),
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.1 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.2 (rotatecount 5, logstart 1, i 1),
renaming /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.0 to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.1 (rotatecount 5, logstart 1, i 0),
old log /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.0 does not exist
copying /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log to /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.1
truncating /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log
removing old log /var/lib/docker/containers/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e/33f2d4ad5090b8699ace2942e398b64707074b2aed28000887214f4594ce165e-json.log.6