Thoughts on how to manage central logs for the Seattle SAGE Group. Jeremy Mates 2004-08-12 The author disclaims all copyrights and releases this document into the public domain. $Id: central-logging.xml,v 1.10 2004/08/12 23:20:28 jmates Exp $

What is Logging?

ktrace logger -it "this is a log message" kdump -f ktrace.out

What is Logged?

Too much, too little, and everything between (± bugs).

What do you want?

It depends!

Educational environment: mainly Linux.

210 systems logging to central loghost.

35 servers

30 printers

85 cluster hosts

rest desktops, miscellaneous

loghost, log proxies for private cluster, build networks.

Easy to get started with syslogd:

*.info @loghost.example.org

Logfile analysis gains

Detect problems before users!

Detect unknown problems?

Security Benefits

Remote Duplication

Correlation

Limit access to loghost.

Time syncronization.

NTPNetwork Time Protocol.

Or, syslog-ng option:

options { use_time_recvd(yes); };

Modest loghost hardware needs (210 clients):

Pentium II 350, 128M RAM, 60G IDE disk.

175,000 messages/day (mean over last seven days).

Gigabyte/month data uncompressed.

http://sial.org/howto/logging/syslogd-problems/

Limited configuration possible: /etc/syslog.conf.

*.err;kern.debug;auth.notice;mail.crit /dev/console *.info;kern.debug;mail.none /var/log/messages mail.info /var/log/maillog *.emerg *

No retention of hostnames across proxies.

Only UDPUser Datagram Protocol supported.

Logfiles also problematic…

No year, timezone, facility, nor priority in logfile.

Apr 15 15:06:50 server sshd[58101]: Connection closed by 192.0.2.151

Hard to find rotated logs, hard to rsync.

messages messages.11 messages.15 messages.4 messages.8 messages.0 messages.12 messages.16 messages.5 messages.9 messages.1 messages.13 messages.2 messages.6 messages.10 messages.14 messages.3 messages.7 We can do better! Because…

New protocol work

http://www.syslog.cc/ietf/protocol.html

http://www.ietf.org/html.charters/syslog-charter.html

LogAnalysis Mailing List - discussion, flamewars, & pointers

Short term: consider syslog-ng.

http://sial.org/howto/logging/syslog-ng/

Hostname retention (or chaining).

Custom TCPTransmission Control Protocol logging.

Configuration much more featureful: syslog-ng.conf

destination mail { file("/var/log/archive/mail/$YEAR/$MONTH/$YEAR-$MONTH-$DAY" template("$ISODATE <$FACILITY.$PRIORITY> $HOST $MSG\n") template_escape(no) ); };

Log format customization: year, timezone, etc.

2004-04-15T15:06:50-0700 <daemon.info> server.example.org sshd[58101]: Connection closed by 192.0.2.151

Easy insert to database.

Extra work to install, maintain.

Custom logpaths: custom tools needed.

Log analysis tools assume syslogd format.

<$FACILITY.$PRIORITY> okay.

$ISODATE more problematic.

Reveals buggy client implementations.

SSH on Solaris, Gnome log problems:

2004-08-12T12:41:45-0800 <auth.info> ' from 192.0.2.151 2004-08-12T12:42:08-0800 <user.warning> client.example.org (username-1234): Error releasing lockfile: Failed to link '/afs/example/home/username/.gconfd/…

Kluge around with syslog-ng.conf, swatch rules.

Recommend developers use proper syslog libraries.

Don't run as superuser!

Setup logs account.

Log readers in logs group.

Run analysis as logs.

Learn to love Regular Expressions.

/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d-\d{4} <[^.]+\.[^>]+> \S+ clamd(?:\[\d+\])?: /

Need active review, realtime checks, and periodic reporting.

Watch logfiles for problems with tail.

tail -f /var/log/messages

Markup for easier reading with colorize or Colorifer.

Could also use swatch for ongoing review.

Other tips?

Like tee, but different.

Follows multiple files.

ls /var/log/htpd/ access_log error_log httpd.pid proxy ssl_engine_log eet /var/log/httpd/*_log | colorize

Applies Perl expressions to input data.

Exclude or limit ongoing logdata seen.

ptail /var/log/everything 'print unless m/(\.(debug|info|notice)> |modprobe)/' ptail /var/log/messages 'print if /named|testhost/'

How to write Perl one liners.

Cannot watch logs all the time…

/var/log/everything - rotated daily

Look for problems (permission denied)?

RAID failures, other critical events.

swatch daemon: hot-potato.conf.

# 3ware logs watchfor /(?i)3w-xxxx.+no longer fault tolerant/ mail=root,subject=LW warn: disk 3ware RAID not fault tolerant throttle 1:00:00,use=regex # this uses MIMEDefang MDLOG entries to spot viruses from "local" subnets watchfor /MDLOG,[^,]+,virus,[^,]+,(?:10\.0\.(?:11|12)|192\.0\.2)/ mail=root:admin-alert,subject=LW warn: mail site malware source throttle 30:00,use=regex

Custom tag for job completion.

# for reporting things via email watchfor /(?i)custom-notify: / mail=root,subject=LW info: misc custom notify message throttle 15:00,use=regex long-running-job ; logger "custom-notify: long job done"

Alert on logging change.

watchfor /logwatch restart/ mail=root,subject=LW info: log logwatch restart throttle 15:00,use=regex

swatch: unknown entries reporting.

ignore /\.info> \S+ clamd(?:\[\d+\])?: / ignore /\.notice> \S+ clamd(?:\[\d+\])?: clamd (startup|shutdown) succeeded/ watchfor /clamd\[\d+\]:.+Unable to open file or directory/ echo throttle 1:00:00,use=regex watchfor /./ echo

Many other similar tools, logsurfer, logwatch, etc.

Many other reports possible.

Lists perviously unseen files.

No reliance on rotation event.

#!/bin/sh unseen -w 30d -T -s ~/share/unseen.logs /var/log/archive | \ while read logfile; do swatch --config-file=everything.conf --examine=$logfile | \ mail -s "swatch check: file=`basename $logfile`" `id -un` done

Marks up text for Regular Expression use.

Ongoing updating of swatch, other configurations.

redress | pbcopy example.log: (user-500) file=/var/log/maillog echo `pbpaste` example\.log: \(user-500\) file=\/var\/log\/maillog

New script, could do more…