Central Logging for Unix

messages messages.11 messages.15 messages.4 messages.8
messages.0 messages.12 messages.16 messages.5 messages.9
messages.1 messages.13 messages.2 messages.6
messages.10 messages.14 messages.3 messages.7

We can do better! Because…

Hope is on the way!

New protocol work

LogAnalysis Mailing List - discussion, flamewars, & pointers
Short term: consider syslog-ng.

Advantages of `syslog-ng`

http://sial.org/howto/logging/syslog-ng/

Hostname retention (or chaining).
Custom Transmission Control Protocol (TCP) logging.
Configuration much more featureful: syslog-ng.conf

destination mail {
file("/var/log/archive/mail/$YEAR/$MONTH/$YEAR-$MONTH-$DAY"
template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n") template_escape(no) ); };

Log format customization: year, timezone, etc.

2004-04-15T15:06:50-0700 <daemon.info> server.example.org sshd[58101]: Connection↵
closed by 192.0.2.151

Easy insert to database.

Disadvantages of `syslog-ng`

Extra work to install, maintain.
Custom logpaths: custom tools needed.
Log analysis tools assume syslogd format.

<$FACILITY.$PRIORITY> okay.
$ISODATE more problematic.

Reveals buggy client implementations.

Buggy Client Logs

SSH on Solaris, Gnome log problems:

2004-08-12T12:41:45-0800 <auth.info> ' from 192.0.2.151
2004-08-12T12:42:08-0800 <user.warning> client.example.org (username-1234): ↵
Error releasing lockfile: Failed to link '/afs/example/home/username/.gconfd/…

Kluge around with syslog-ng.conf, swatch rules.
Recommend developers use proper syslog(3) libraries.

Log Analysis

Don't run as superuser!

Setup logs account.
Log readers in logs group.
Run analysis as logs.

Learn to love Regular Expressions.

/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d-\d{4} <[^.]+\.[^>]+> \S+ clamd(?:\[\d+\])?: /

Need active review, realtime checks, and periodic reporting.

Active Review

Watch logfiles for problems with tail(1).

tail -f /var/log/messages

Markup for easier reading with colorize or Colorifer.
Could also use swatch for ongoing review.
Other tips?

Custom Script: `eet`

Like tee(1), but different.
Follows multiple files.

$ ls /var/log/htpd/
access_log error_log httpd.pid proxy ssl_engine_log
$ eet /var/log/httpd/*_log | colorize

Custom Script: `ptail`

Applies Perl expressions to input data.
Exclude or limit ongoing logdata seen.

$ ptail /var/log/everything 'print unless m/(\.(debug|info|notice)> |modprobe)/'

$ ptail /var/log/messages 'print if /named|testhost/'

How to write Perl one liners.

Cannot watch logs all the time…

Realtime Notification

/var/log/everything - rotated daily

Look for problems (permission denied)?
RAID failures, other critical events.
swatch daemon: hot-potato.conf.

# 3ware logs
watchfor /(?i)3w-xxxx.+no longer fault tolerant/
mail=root,subject=LW warn: disk 3ware RAID not fault tolerant
throttle 1:00:00,use=regex

# this uses MIMEDefang MDLOG entries to spot viruses from "local" subnets
watchfor /MDLOG,[^,]+,virus,[^,]+,(?:10\.0\.(?:11|12)|192\.0\.2)/
mail=root:admin-alert,subject=LW warn: mail site malware source
throttle 30:00,use=regex

Notification Tips

Custom tag for job completion.

# for reporting things via email
watchfor /(?i)custom-notify: /
mail=root,subject=LW info: misc custom notify message
throttle 15:00,use=regex

$ long-running-job ; logger "custom-notify: long job done"

Alert on logging change.

watchfor /logwatch restart/
mail=root,subject=LW info: log logwatch restart
throttle 15:00,use=regex

Periodic Reporting

swatch: unknown entries reporting.

ignore /\.info> \S+ clamd(?:\[\d+\])?: /
ignore /\.notice> \S+ clamd(?:\[\d+\])?: clamd (startup|shutdown) succeeded/

watchfor /clamd\[\d+\]:.+Unable to open file or directory/
echo
throttle 1:00:00,use=regex

watchfor /./
echo

Many other similar tools, logsurfer, logwatch, etc.
Many other reports possible.

Custom Script: `unseen`

Lists perviously unseen files.
No reliance on rotation event.

#!/bin/sh

unseen -w 30d -T -s ~/share/unseen.logs /var/log/archive | \
while read logfile; do

swatch --config-file=everything.conf --examine=$logfile | \
mail -s "swatch check: file=`basename $logfile`" `id -un`

done

Custom Script: `redress`

Marks up text for Regular Expression use.
Ongoing updating of swatch, other configurations.

$ redress | pbcopy
example.log: (user-500) file=/var/log/maillog
$ echo `pbpaste`
example\.log: $user-500$ file=\/var\/log\/maillog

New script, could do more…