Logcheck mail locations, related scripts and other mail locations on postfix server

Log mail location on postfixserver

logadmin account

1. Normal logcheck mail from computer nodes (logcheck@nxxxx) are accessed at /home/logadmin/Maildir/.computer_node/

2. Normal logcheck mail from computerdb (logcheck@hera) are accessed at /home/logadmin/Maildir/.hera/

3. Normal logcheck mail from Linux file server (logcheck@dxx) are accessed at /home/logadmin/Maildir/.data_server/

4. Normal logcheck mail from headnodes (logcheck@hx) are accessed at /home/logadmin/Maildir/.headnodes/

5. Normal logcheck mail from Postfixserver itself (logcheck@postfixserver) are accessed at /home/logadmin/Maildir

6. Logcheck mail from computer nodes with= "[logcheck] Reboot"= as Subject are accessed at /home/logadmin/Maildir/.reboot

7. At the present, the mail to outside-cooling@aei.mpg.de are accessed at /home/logadmin/Maildir/.ddc4000

8. Other exceptional mails to logadmin@postfixserver are accessed at /home/logadmin/Maildir

loki account

1. Smart error mail are accessed at accessed at /home/loki/Maildir/.smart_error and forwarded to management list

2. mail to posmaster@nxxxx, logcheck@nxxxx and root@nxxxx are accessed /home/loki/Maildir/.computer_node

3. mail to posmaster@dxx, logcheck@dxx and root@dxx are accessed /home/loki/Maildir/.data_server

4. mail to posmaster@hera, logcheck@hera and root@hera are accessed /home/loki/Maildir/.hera

5. mail to posmaster@hx, logcheck@hx and root@hx are accessed /home/loki/Maildir/.headnodes

6. All other mail to root@postfixserver are accessed /home/loki/Maildir/

Where to check all the not-normal-logcheck mail

To check all the not-normal-logcheck mails, you have to work through all the following place:

/home/loki/Maildir
/home/loki/Maildir/.computer_node
/home/loki/Maildir/.data_server
/home/loki/Maildir/.hera
/home/loki/Maildir/.headnodes
/home/loki/Maildir/.smart_error

/home/logadmin/Maildir
/home/logadmin/Maildir/.ddc4000

At the moment, there is a cron jobs, /etc/cron.d/delete_portmap_mail_from_loki, running everyday at 9:30am to remove the mails related with those following message under /home/loki/Maildir.

No such map auto.hosts. Reason: Can't communicate with portmapper
No such map auto.home. Reason: Can't communicate with portmapper
No such map auto.hosts. Reason: Can't communicate with portmapper

/usr/local/bin/make_snmp_file.sh: line 53: printf: -r: invalid option
printf: usage: printf [-v var] format [arguments]

Subject: Cron  printf ”#################################\n# Do not edit!\n# 
This file is generated by crontab using the NIS map grid-mapfile\n################
#########\n\n”> $file; /usr/bin/ypcat -k grid-mapfile >> $file
/bin/sh: /etc/grid-security/grid-mapfile: No such file or directory
/bin/sh: /etc/grid-security/grid-mapfile: No such file or directory

Scripts on Postfixserver

1. nodesummary.pl

This program is to summarize the log mail from computer nodes. It runs everyday at 6:20,12:20 and 18:20 through a cron file located at /etc/cron.d/mailsummary.

It reads the log mails (located at /home/logadmin/Maildir/.computer/new) sent from computer nodes 5:00-6:05,11:00-12:05 and 17:00-18:05, creates a summary file at /home/logadmin/mailsummary/summary-year-date-time (eg: summary-2008-05-16-05), then send the summary to the atlas_admin@aei.mpg.de.

Note: log mails from computer nodes are possible arrive slightly after 6:00, 12:00 and 18:00 (a few minutes later), due to the random delay setting of the logcheck cron job.

2. delete_node_mail

This program is to delete the computer nodes' log mails which are more than 3 days old. It runs at 1:00am everyday through a cron file located at /etc/cron.d/delete_node_mail.

3. fileserversummary.pl

This program is to summarize the log mail from file servers. It runs everyday at 7:00,13:00 and 19:00 through a cron file located at /etc/cron.d/fileserversummary.

It reads the log mails (located at /home/logadmin/Maildir/.date_server/new) sent from Linux file servers at 5:00,11:00 and 17:00, creates a summary file at /home/logadmin/fileserver/summary-year-date-time (eg: summary-2008-05-16-05), then send the summary to the management list.

4. headnodesummary.pl

This program is to summarize the log mail from headnodes. It runs everyday at 7:30,13:30 and 19:30 through a cron file located at /etc/cron.d/headnodesummary.

It reads the log mails (located at /home/logadmin/Maildir/.headnodes/new) sent from headnodes at 5:00,11:00 and 17:00, creates a summary file at /home/logadmin/headnodesummary/summary-year-date-time (eg: summary-2008-05-16-05), then send the summary to the management list.

5. search_node_mail.pl

It is a small program which could be used to search the logcheck mail (at /home/logadmin/Maidir/.computer_node/new) from a specific node at a specific time. Run it as:

./search_node_mail yyyy-mm-dd hh nxxxx, it will display the mail on the standard output.

6. How to combine logs in the nodes' summary mail? (Similar rules apply to fileserversummary.pl and headnodesummary.pl)

For example, in the file of summary-2008-08-21-05, you have got hundreds of logs like the followings.

automount: mount(nfs): host n0003/distributed: lookup failure
automount: mount(nfs): host n0003: lookup failure
automount: mount(nfs): host d03/distributed: lookup failure
automount: mount(nfs): host d03: lookup failure
...
automount: mount(nfs): host n1003/distributed: lookup failure
automount: mount(nfs): host n1003: lookup failure

You prefer keep this kind of information but summarize them into one line in the summary file. To do so, use regex, add the following lines in the file /home/logadmin/log_combined

#logNo
automount: mount\(nfs\): host (n[0-9]+|d[0-9]+)(/distributed)?: lookup failure
automount: mount(nfs): host n* or d* or n*/distributed or d*/distributed: lookup failure

7. How to filter/skip logs in the nodes' summary mail? (Similar rules apply to fileserversummary.pl and headnodesummary.pl)

For example, in the file of summary-2008-08-21-05, you have got hundreds of normal logs like the followings.

pam_rhosts_auth: allowed to root@n0101.atlas.local as root
...
pam_rhosts_auth: allowed to root@n1200.atlas.local as root

You don't need those kind of normal logs appeared in the file of summary-2008-08-21-05 and future summary. You should update logcheck rule on failenny at ignore.d.server/logcheck-local-rules and fai softupdate the logcheck rule on all the nodes. But If you cannot do this for some time being, another alternative quick way is to add one line like the following in the file /home/logadmin/log_skipped.

pam_rhosts_auth: allowed to root@n[0-9]{4}.atlas.local as root"

Note: Remember to remove those logs in the file /home/logadmin/log_skipped after you have added them in the ignore.d.server/logcheck-local-rules on failenny and updated through fai softupdate on all the nodes.