Part 16 - QmailAnalog w/qlogtools & qms-analog

Qmailanalog performs some basic log analysis on those qmail log files and then outputs them to a desired location. In my case, I run qmailanalog every night and output the results to e-mail. Along with qmailanalog, we're going to install the "qlogtools" package. Qlogtools, as its name implies, provides an array of tools which can be used to analyze the qmail logs. We're going to use one of the qlogtool packages, tai64n2tai, to convert the timestamps on the log files from a machine readable format to a human readable format which will come to us when we get the finished report. After we've installed both Qmailstats and Qlogtools, we will create a script which you can run on a nightly basis to generate e-mail stats. The script will also incorporation qms-analog, which we installed with qmail-scanner previously. The qms-analog output will give add qmail-scanner stats to our nightly report.
First, let's install qmailanalog...
cd /downloads/qmailrocks/
tar zxvf qmailanalog-0.70.tar.gz
cd qmailanalog-0.70

RH 9/RHEL/Fedora/Slackware users: You will need to patch qmailanalog with an additional errno patch: patch < /downloads/qmailrocks/patches/0.70-errno.patch

make && make setup check
That's it. Qmailanalog is installed!
Now let's install qlogtools...
cd /downloads/qmailrocks/
tar zxvf qlogtools-3.1.tar.gz
cd qlogtools-3.1

RH 9/RHEL/Fedora/Slackware users: You will need to patch qlogtools with an additional errno patch: patch < /downloads/qmailrocks/patches/qlogtools_errno.patch

mkdir /usr/local/man (if directory already exists, you're good to go)
make
./installer
OK. The qlogtools library of tools should now be installed.
Now we will implement a script to run Qmailanalog and then you can hook that script into the server's crontab to get stats generated every night.
The script below is a solid script that sends an email to the server administrator with both the qmailanalog output as well as qms-analog's readout of qmail-scanner's activities. Pretty sweet, huh?
cp /downloads/qmailrocks/qms-analog-0.4.2/qmailstats /var/qmail/bin
vi /var/qmail/bin/qmailstats

#!/bin/sh
## qms-analog and qmailanalog invocation script
##
## Note: For better readability of the nightly stats email, set your email
## client font to a fixed width font - then all the columns line up
## very nicely.
##
PATH=/usr/local/qmailanalog/bin:/var/qmail/bin:/bin:/usr/bin:/usr/local/bin
QMAILSTATS="/tmp/q.$$"
EMAILMSG="/tmp/qms.$$"
umask 077
DATE=`date +'%D'`

## prepare qmail log entries for qmailanalog routines
cat /var/log/qmail/qmail-send/* /var/log/qmail/qmail-pop3d/* /var/log/qmail/qmail-smtpd/* | tai64n2tai | awk '{$1=substr($1,1,index($1,".")+6);print}' | matchup > $QMAILSTATS 5>/dev/null

## build the email message header
echo "To: your_postmaster@yourdomain.com" > $EMAILMSG
echo "From: your_postmaster@yourdomain.com" >> $EMAILMSG
echo "Subject: Nightly Qmail Stats Report for $DATE" >> $EMAILMSG
echo "" >> $EMAILMSG
echo "" >> $EMAILMSG

## qms-analog invocation
#
# USAGE: qms-analog hours-of-history
#
# hours-of-history (0 - n) hours of history to collect
# 0 => all records
# sort-key (optional) sort key for account statistics
# msgbw (default) msg bandwidth - successful msgs
# alpha alphanumeric by account name
# virus number of viruses received
# saavg Spamassassin avg score
# sadet Spamassassin msgs detected
#
# Examples:
# "qms-analog 24" - use only records within the last 24 hours,
# sort by msg bandwidth
# "qms-analog 168" - use only records within the last 7 days,
# sort by msg bandwidth
# "qms-analog 0" - use all records, sort by msg bandwidth
# "qms-analog 0 alpha" - use all records, sort alphabetically
# "qms-analog 0 saavg" - use all records, sort by Spam average score
#
# Note: Add or remove statistical time frames to suit your preference -
# "last 24 hours" and "all records" are uncommented below by default.
##
#### Last 24 hours
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~ L a s t 2 4 H o u r s ~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
cat /var/spool/qmailscan/qms-events.log | qms-analog 24 >> $EMAILMSG
####
#### Last 7 days
#echo "" >> $EMAILMSG
#echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
#echo "~~~~~~~~~~~~~~~~~~~~~~~~~~ L a s t 7 D a y s ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
#cat /var/spool/qmailscan/qms-events.log | qms-analog 168 >> $EMAILMSG
####
#### Last 30 days
#echo "" >> $EMAILMSG
#echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
#echo "~~~~~~~~~~~~~~~~~~~~~~~~~~ L a s t 3 0 D a y s ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
#cat /var/spool/qmailscan/qms-events.log | qms-analog 5040 >> $EMAILMSG
####
#### All records in qms-events.log
echo "" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~ A l l R e c o r d s ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
cat /var/spool/qmailscan/qms-events.log | qms-analog 0 >> $EMAILMSG
####

## qmailanalog invocation
echo "" >> $EMAILMSG
echo "" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
zoverall < $QMAILSTATS >> $EMAILMSG
echo "" >> $EMAILMSG
echo "" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
zfailures < $QMAILSTATS >> $EMAILMSG
echo "" >> $EMAILMSG
echo "" >> $EMAILMSG
echo "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" >> $EMAILMSG
zdeferrals < $QMAILSTATS >> $EMAILMSG
echo "" >> $EMAILMSG

## pipe the message into qmail-inject
cat $EMAILMSG | qmail-inject

## delete temp files
rm -f $QMAILSTATS
rm -f $EMAILMSG
Now set the script executable...
chmod 750 /var/qmail/bin/qmailstats
Now run the script...
/var/qmail/bin/qmailstats
Check your email and you should get a report with some pretty cool stuff in it! Your report should look something like this.
OK, if the qmailstats script is working, you will now want to create a crontab entry to run this script every night.
So, as the "root" user let's set up a cron entry...
crontab -e
0 3 * * * /var/qmail/bin/qmailstats 1>/dev/null 2>/dev/null
Save and exit from the crontab editor and you should be all set. The above entry will run the qmailstats script every night at 3:00A