Forum

Hi,

We had some runaway logs on an ubuntu cluster and traced it back to Hiawatha's rotate behavior. Apparently when logrotate runs with the default logrotate config, it never stops writing to the original file. The new access.log stays at 0 while the older log continues to grow (even after compression! eep) leaving compressed logs corrupt and uncompressed logs incorrect. If left long enough, the logfile is destroyed by logrotate, but hiawatha continues logging to it creating the invisible ever-growing file!

This can be fixed with restarting hiawatha when logrotate runs, but is not ideal. 2 better options.

1) Implement graceful reload and add a post-rotate action that performs the reload. Works, but depends on initscripts (different mechanism will be needed for systemd)

or

2) Add copytruncate to the default logrotate config. This is what we're doing, seems like the easiest and can be done with repackaging instead of code.

Also, logrotate should rotate all logs that hiawatha produces by default. Current package only rotates access.log and we had slow death by error.log

Hiawatha version: Hiawatha v9.11, cache, IPv6, Monitor, reverse proxy, SSL (1.3.9), Tomahawk, URL toolkit, XSLT
Operating System: Ubuntu 14.04.1 LTS

For anyone else using this in production, we used,

/etc/logrotate.d/hiawatha

/var/log/hiawatha/*.log {
        copytruncate
        weekly
        compress
        delaycompress
        rotate 52
        missingok
        create 640 www-data www-data
        sharedscripts
}

For now it's in an ansible playbook, but it would be nice if hiawatha came ready to go from the .deb!

To make Hiawatha rotate its current logfiles, send the HUP signal to its process. Add the following to your logrotate configuration.

/usr/sbin/killall -HUP hiawatha

I'll make sure this is present in the .deb of the next release.

Thanks, Hugo! You+Hiawatha are the best!