Log File Rotation Problems

The traditional syslogd only allows logging to a specific file, such as /var/log/messages. To avoid filling disks, log files are usually rotated with a utility such as logrotate(8) or newsyslog(8). However, the standard rotation methods have problems, including edge cases where logs may be lost, trouble referencing prior logs at some future point, and poor behavior should the log rotation fail or system crash for long periods of time.

These rotation problems apply to other applications, such as Apache or Squid, though syslogd log files will be the primary focus here, and syslog-ng the recommended solution.

Send `HUP` Signal and Hope

The usual manner of log file rotation involves a race condition between the log rotate program and syslogd, where the log rotate program uses several system calls to move aside the old logs and then notify syslogd. This design flaw can be eliminated by having the logging daemon control log file rotation itself.

Other applications do not support the HUP signal method. In this case, a copy and truncate method is used on log files. If the application writes more logs between the copy and HUP signal or truncate operations, log entries will be silently discarded.

The syslog-ng logging daemon includes several solutions that eliminate rotation problems: log file templates, so logs go to a file based on the current date (avoiding rotation entirely), or sending the logs into a database (old logs can be archive or dropped from the primary table).

Conveyer Belt Rotation

Complicated Backups | Hard to Reference Logs

Most log file rotation scripts increment numbers in the names of all the previous log files, and then save the current maillog to maillog.1 (or maillog.0, under some implementations). Older log files eventually fall off the end of the conveyer belt after a certain number of rotations have been done. This method works well in some situations, but imposes several problems.

Complicated Backups

Backup utilities or file synchronization software such as rsync will be forced to backup all the log files following each rotation, as the renames have changed the contents of all the log files by name. Reprogramming the backup software to be aware of the rotation policy would be complicated and error prone, but might be possible by tracking the filesystem inode number, and noting a change in the filename.

In contrast, if the log files are saved to a filename by date by syslog-ng, then backup software such as rsync will be able to skip previously copied files, and only transfer what has changed in the currently open log files.

Hard to Reference Logs

Between daily versus weekly rotation schedules, early morning log file rotation times, and other implementation differences, tracking down an old log file entry is a time consuming task, as one must poke around to find what file the data has been moved to, and repeat this process each time the log entry needs to be looked up.

For example, with conveyer belt rotation, a log entry from two days ago might be found in:

messages - weekly rotation, not yet rotated. Depends on the day of the week.
messages.1 - weekly rotation, on certain days of the week, or where the files are rotated to messages.0 before messages.1.
messages.2 - daily rotation, via previous messages.1 rotation, or messages.1, if the first file was messages.0.
messages - any sort of rotation, except the system crashed over the holiday weekend, and now the rotation is several days out of sync, and may need manual intervention by an administrator to ensure the logs for the correct time period are rotated into the correct files.

In other words, there is no easy way to calculate where a specific log will be located at any point in time under traditional log file rotation, without knowing the particulars of the log rotation implementation and configuration, and whether the system has been running or log rotation actually working.

Also, if log rotation fails, but the logging continues, additional work may be required by administrators if subsequent log processing assumes the log files had be properly rotated according to site policy. This assumes the log rotation failure is even detected in a timely fashion.

Solutions

One solution: ensure the logging daemon—such as syslog-ng—rotates the log files itself, or better yet, writes to log files that do not need rotation:

destination everything {
file("/var/log/archive/$R_YEAR/$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n")
template_escape(no)
);
};

Another solution: httplog. This utility can rotate logs into time and hostname based files, and maintain a symbolic link to the current logfile. For example, to rotate output from a shell command with a new logfile created each minute: